How to run experiments with Claude

AI coding tools can help you create an experiment. They cannot decide what the experiment means.

Claude Code can inspect code, add a treatment path, wire a flag, and write tests faster than most teams can move through a dashboard. That speed is useful only if the experiment has a clear hypothesis, stable assignment, and metrics that answer the actual product question.

This guide shows how to run an experiment with Claude Code and GrowthBook without turning AI-generated code into a measurement shortcut.

Start with the decision, not the variant

The prompt should begin with the decision you want to make:

We need to decide whether the new onboarding checklist improves activation.
Create the implementation behind a GrowthBook flag.
Use the existing onboarding flow as the control.
Do not remove the old path.
Add tests for control and treatment.
Return an experiment spec with hypothesis, primary metric, guardrails, and rollout plan.

That prompt gives Claude useful work. It also prevents the common failure mode: an agent creates a variant before anyone has agreed what evidence would make the variant worth shipping.

The implementation details below are grounded in Claude Code documentation, Claude Code commands, Claude MCP connector docs, Model Context Protocol introduction, Claude Code review discussion, and Hacker News discussion of Claude Code feature flags. For GrowthBook-specific steps, use the GrowthBook MCP docs, feature flag docs, and feature flag experiment docs.

GrowthBook's feature flag experiment docs are the bridge between implementation and measurement. A flag controls assignment. An experiment measures whether the treatment changes a metric.

Define the experiment spec before editing files

Ask Claude for a short spec first. It should include:

This is not process theater. It protects the experiment from becoming a demo of whatever the agent happened to implement.

For AI-generated product changes, guardrails should usually include at least one engineering signal and one product signal. Error rate, latency, and task completion are more useful than a single click metric when the change affects core behavior.

Use GrowthBook MCP to create the experiment objects

With the GrowthBook MCP Server connected, Claude can call tools for feature flags, experiments, attributes, metrics, SDK connections, and docs search. That gives the agent live access to the control plane instead of forcing it to describe dashboard steps from memory.

The practical prompt looks like this:

Create a GrowthBook experiment for the new onboarding checklist.
Link it to the new_onboarding_checklist flag.
Use false as control and true as treatment.
Use activation_rate as the primary metric if it exists.
Add signup_completion and error_rate as guardrails if available.
If a metric is missing, stop and tell me what needs to be created.

The final sentence matters. Agents should not invent metrics. If the metric does not exist, the right answer is a gap report, not a fake experiment.

Implement assignment through a feature flag

The code should evaluate a GrowthBook flag and keep both branches available:

const showChecklist = useFeatureIsOn("new_onboarding_checklist");

return showChecklist ? (
<GuidedOnboardingChecklist user={user} />
) : (
<CurrentOnboarding user={user} />
);

This is the cleanest implementation shape for most product experiments because rollout and experiment assignment stay in GrowthBook while application code stays readable.

For server-side experiments, keep assignment close to the decision point:

const variant = growthbook.getFeatureValue("pricing_page_layout", "control");

if (variant === "compact") {
return renderCompactPricingPage(request);
}

return renderCurrentPricingPage(request);

Use a string or JSON flag when the experiment needs more than true or false. Use a boolean flag when the treatment is a single on/off branch.

Add QA before you add traffic

Experiment code has two correctness requirements. The treatment has to work, and the control has to stay untouched. That second requirement is easy to miss when an AI tool edits shared components.

Ask Claude to run a review with a narrow brief:

Review the experiment implementation.
Confirm assignment is stable.
Confirm the control path still behaves like the original code.
Confirm the treatment path emits the same required analytics events.
Check for test coverage in both variants.
Do not change files.

Then run the normal test suite. If the repository has visual tests, smoke tests, or contract tests, run those too. Feature flag experiments fail when the instrumentation is wrong as often as when the UI is wrong.

Watch the right metrics while the experiment runs

GrowthBook's experimentation product page and metrics docs emphasize metric definitions because the experiment is only as trustworthy as the measurement. For AI-assisted changes, the primary metric should map to user value, not agent output.

Good primary metrics:

• Activation for onboarding changes.
• Qualified signup for pricing-page changes.
• Task completion for AI assistants.
• Retention or return rate for workflow changes.

Good guardrails:

• Error rate.
• Latency.
• Support contact rate.
• Refund or cancellation rate.
• Downstream conversion.

If Claude suggests a metric that is easy to instrument but not tied to the decision, reject it. The AI tool should help implement measurement, not redefine success around whatever it can find.

Decide before you ship

The experiment should end with one of three actions:

• Ship the treatment and remove the temporary branch.
• Keep iterating because the evidence is inconclusive or mixed.
• Roll back because the treatment harmed the primary metric or guardrails.

Ask Claude to summarize the result, but keep the decision accountable:

Summarize the GrowthBook experiment result for engineering review.
Include primary metric direction, guardrail status, sample ratio issues, and suggested follow-up.
Do not recommend shipping unless the predefined decision rule is met.

That prompt keeps the agent anchored to the experiment plan instead of turning the result into a sales pitch for its own code.

Where GrowthBook fits

Claude Code accelerates implementation. GrowthBook keeps experimentation honest. The combination works when each system has the right job: the AI tool edits code and explains changes, while GrowthBook handles flags, assignment, metrics, guardrails, and results.

If you want a first experiment, choose a small feature with a real user outcome. Create the flag, keep the fallback, define the metric, and let Claude help with the code. Then let the experiment decide.

A complete experiment prompt

Use a two-part prompt. First ask for a plan:

Plan a GrowthBook-backed experiment for this feature.
Include hypothesis, control, treatment, primary metric, guardrails, assignment unit, required code changes, and tests.
Do not edit files yet.

Then ask for implementation:

Implement the approved experiment plan.
Keep the control path intact.
Use the existing GrowthBook SDK client.
Add tests for both variants.
Do not enable production targeting.
Return assumptions and changed files.

Separating plan and implementation gives reviewers a chance to catch weak measurement before code exists. It also gives Claude a more precise target.

What to do when the result is inconclusive

AI-generated experiments will not always produce a clean win or loss. An inconclusive result is still useful if it narrows the next question. Ask:

• Was the treatment visible enough to affect behavior?
• Was the primary metric too far downstream?
• Did guardrails move even if the primary metric did not?
• Did segment-level results suggest a narrower audience?
• Did the agent change more than one meaningful variable?

Do not ask Claude to rescue the experiment by adding post-hoc metrics until the team has stated why those metrics answer the original question. GrowthBook can support retroactive metric inspection, but retroactive analysis should be labeled as exploratory.

After the decision

After the result, use the agent for cleanup:

The experiment decision is complete.
If treatment shipped, remove the control path and temporary flag checks.
If treatment did not ship, remove the treatment path.
Keep analytics events needed for ongoing measurement.
Return a cleanup diff summary.

This is where many teams lose the benefit of feature flags. The flag created safety during rollout, but it becomes technical debt if nobody removes or graduates it after the decision.

Assignment details the agent should not guess

Before traffic enters the experiment, ask Claude to return the assignment assumptions explicitly:

List the experiment assignment assumptions.
Include assignment unit, audience eligibility, sticky bucketing behavior, excluded users, and how QA users can force each variant.
Do not change configuration.

This catches several subtle problems. An experiment assigned by user ID behaves differently from one assigned by account ID. A pricing experiment may need account-level assignment so one company does not see inconsistent plans. A logged-out onboarding experiment may need anonymous IDs that later reconcile to users. If the agent cannot explain assignment, the team is not ready to trust the result.

GrowthBook should remain the source of truth for assignment and exposure. The code should evaluate the flag, render the right branch, and emit required events. It should not create its own randomization layer just because that is easy for the agent to implement.

Instrumentation review

AI-generated experiments often look correct in the UI while measurement is broken. Add an instrumentation review before launch:

• Does the control path emit the same required events it emitted before?
• Does the treatment path emit comparable events?
• Are event names stable across variants?
• Are metric windows appropriate for the behavior?
• Are guardrails available before traffic starts?
• Does the experiment exclude staff, test accounts, or internal traffic when needed?

Ask Claude to inspect analytics calls and GrowthBook metric references together. If it only reviews React components, it may miss the event layer. If it only reviews the dashboard configuration, it may miss code paths that never fire the event.

Result readout template

When the experiment ends, do not ask the agent "should we ship?" Ask for a structured readout:

Prepare an experiment readout.
Include hypothesis, primary metric, guardrails, sample ratio checks, segments worth reviewing, implementation caveats, and cleanup recommendation.
Separate observed results from recommendations.

That wording keeps the readout useful for product, engineering, and data review. The agent can summarize, but the decision should still trace back to the rule the team wrote before launch.

Cleanup after the experiment

The cleanup task depends on the result:

This is a good place to use Claude because cleanup is code-heavy and easy to defer. Give the agent the decision, the flag key, and the desired final state. Do not ask it to infer the decision from a chart screenshot.

Practical implementation playbook

Treat run experiments with claude as a workflow, not a one-off prompt. The teams that get the most value from AI coding tools do three things consistently: they narrow the task, keep release state in GrowthBook, and make the final output reviewable by humans who were not in the original chat.

Step 1: Write the release contract

Before code changes, define the contract in a few sentences:

Change:
User-visible behavior:
GrowthBook flag:
Default value:
Fallback:
Owner:
Primary metric:
Guardrails:
Cleanup condition:

The agent can help fill missing fields, but it should not silently invent them. If the primary metric or owner is missing, the right next step is a gap report.

Step 2: Inspect before editing

Ask the tool to inspect existing patterns first:

• Where does the app initialize the GrowthBook SDK?
• Which helper functions or hooks evaluate flags?
• How are tests written for feature-flagged behavior?
• Where do analytics events fire?
• Which files are likely to change?

This keeps the first implementation from creating a second GrowthBook client, duplicating SDK logic, or placing a flag check too high in the component tree.

Step 3: Make one controlled change

The first pass should be deliberately small. Create or reference one flag, preserve one fallback, and test both states. If the agent discovers a useful refactor, capture it as a separate follow-up. Do not let a flag implementation turn into a cleanup project, design-system rewrite, and analytics migration in the same pull request.

Step 4: Review for release risk

Use the agent again, but switch it into review mode:

Review this AI-assisted change for release risk.
Check flag key consistency, fallback behavior, test coverage, analytics parity, targeting assumptions, and cleanup notes.
List blockers first.
Do not modify files.

This prompt usually finds a different class of issues than normal linting. It looks for the ways a change can be technically valid but operationally unsafe.

Step 5: Expose gradually

Deployment and release should stay separate. Deploy the code with the default safe value. Use GrowthBook to expose the change to internal users, a beta segment, a percentage rollout, or an experiment audience. If the change needs evidence, connect it to an experiment before broad exposure.

Step 6: Close the loop

The workflow is not done when the flag turns on. It is done when one of these outcomes happens:

• The treatment ships and temporary branching logic is removed.
• The treatment loses and the new path is removed.
• The test is inconclusive and the next iteration is defined.
• The rollout is paused because a guardrail or QA issue appeared.

Ask the agent for cleanup only after the human-owned decision is clear.

Example review packet

A good final packet should fit in a pull request comment:

• GrowthBook object: new_checkout_summary flag and linked experiment.
• Default: false for all production users.
• Fallback: existing checkout summary component.
• Treatment: new summary component with revised copy and layout.
• Tests: unit tests for off/on states and one integration smoke test.
• Metrics: checkout completion as primary, error rate and latency as guardrails.
• Rollout: internal staff, beta accounts, then experiment traffic.
• Cleanup: remove losing branch or graduate winning branch after decision.

This packet is not bureaucracy. It is the artifact that lets someone review the work without trusting a chat transcript.