Jev Applied: Scoring experiment quality with a decision model

TL;DR: We are building an experiment quality score into GrowthBook, and some of the checks need judgment: is the hypothesis good, do the goal metrics measure what it predicts, etc. Instead of asking a chat model for a grade, we split each check into small yes/no questions and send them to Jev, a decision model from TypeSafe. In our test, Jev took about a tenth of a second per experiment. At about 600 input tokens per pass, 10,000 scoring passes would cost around a quarter.

"Test new homepage." "The CEO wants to try the blue header." Entries like these show up in the hypothesis field of almost every experimentation program. A clear hypothesis is what turns an experiment result into something you can learn from. If you do not say what you are changing, what you expect to happen, and why, then a win is hard to learn from and a loss is impossible to learn from.

The problem is scale. A team running a few experiments a month can review every hypothesis by hand. A team running hundreds cannot, and the problems compound when you add more and more signals of a ‘quality’ experiment.

We are adding an experiment quality score to GrowthBook to help with this. Judging whether a hypothesis is good, and whether the metrics match it, takes reading comprehension. This post covers how we built those two checks with a decision model, what went wrong along the way, and how it compared to a chat model.

Why not just ask an LLM?

The obvious approach is to send the quality checks to a chat model and ask for a score from 0 to 1 with a reason. That is how our first version worked, and it works reasonably well. LLMs are good at reading a sentence and telling you whether it makes sense.

But it is an awkward fit for a check that should run every time someone edits an experiment. A chat model takes a second or more per call, costs enough that orgs need usage limits, and returns a paragraph and a score that can drift between runs. A score that moves when nothing changed is confusing, and people will stop trusting it.

What we actually wanted was a decision our code could use directly.

What a decision model is

Jev is the first release in what TypeSafe calls System One models. You give it some state (here, the hypothesis and the goal metric names to name a few) and a set of questions with fixed answer types, such as yes/no or a scale with named levels. It answers every question in one pass and returns a probability for each answer instead of generating text. Because the valid answers are fixed in advance, there is nothing to parse and no free text for the model to make up.

TypeSafe is upfront that this is the wrong tool for chat, code, or anything that needs a written explanation. For a classification task like ours, that is exactly the trade you want.

Break one big question into small ones

The most useful design choice had nothing to do with which model we used. We split the quality score into many different checks and ratings. For example, instead of asking "how good is this hypothesis?", we ask three narrow questions:

• Does it name a change? A new design, copy, flow, price or feature, rather than only a goal.
• Does it predict an effect? A measurable outcome with a direction, such as more signups or fewer drop-offs, rather than "a better experience."
• Does it give a reason? Why the change should cause the effect.

The score is the average of the three answers, combined in our code rather than the model's.

Small questions make the score explain itself. "Redesigning the homepage will increase signups" names a change and predicts an effect, but gives no reason, so it loses about a third of its points and the check says exactly that: "The hypothesis doesn't explain why." The experiment's Quality tab then lists the fixes that would raise the score the most, with the points each one adds. A single grade from a chat model can tell you a hypothesis is weak. It is much harder for it to tell you consistently which part is missing.

How it compared

To test our AI quality scoring, we created test cases. With hypothesis checking, for example, we hand-labelled 30 hypotheses, ranging from good ("Pre-filling the shipping address from the user's last order will increase completed orders, since retyping the address is where most users drop off.") to the ones at the top of this post, and paired some good hypotheses with the wrong metrics. We then ran each through Jev and through a chat model, gpt-5.4-mini, given the same prompt our existing AI checks use.

Jev charges only for input tokens. At $0.042 per million input tokens, 10,000 scoring passes would cost roughly $0.25 if each pass used about 600 input tokens. Re-running a check uses additional tokens.

We used these hand-labelled examples to see how each model’s judgments lined up with ours. We also pinned the Jev version so a new release wouldn’t silently change the model we use.

What fast and cheap changes

When a check costs a fraction of a cent and takes about a tenth of a second in our test, it’s practical to run it whenever the experiment changes. We designed the AI checks to re-run about a minute after any edit to the hypothesis or goal metrics, on drafts as well as running and stopped experiments. That moves the feedback from a post-mortem to the moment someone is writing the hypothesis, which is when it is most useful.

Not everything moved over. Custom AI checks, where an organization writes its own open-ended rule, still may use a chat model, because those questions are not known in advance. We actually use Jev to see if the AI check can be done with Jev or require an LLM.

When to try a decision model

If an LLM task in your product is really a classification, it is worth trying a model built for classification. Split the judgment into the smallest questions you can, combine the answers in code, and put any rule you can state plainly in code as well. Before trusting any judge, label some examples by hand and see how often it agrees with you.

Experiment Quality scoring is coming soon for GrowthBook Enterprise customers. If you want early access, or have opinions on what a good hypothesis looks like, we would love to hear from you.

Design your next experiment

Put a hypothesis to the test and measure the results with GrowthBook’s A/B testing tools.

Start for Free