Experiments

How to run A/B tests in Sanity

A graphic of a bar chart with an arrow pointing upward.

Publishing two headlines is easy. Making sure the right visitor sees the right headline—and measuring that experience—is the experiment.

Sanity gives content teams a structured place to author alternatives. Your application still needs to choose one, preserve that choice for the relevant visitor, render it correctly, and connect the experience to an outcome. Publishing headline B for everyone next week creates a before-and-after comparison, with traffic and timing differences mixed into the result.

This guide builds a small server-rendered content test using Sanity, GrowthBook, and a TypeScript Next.js App Router application. Editors own two headline fields. GrowthBook assigns visitors to a string-valued feature experiment. The application renders the selected headline and reports an exposure after the browser mounts the page.

The example deliberately uses ordinary Sanity fields. An existing Sanity integration tutorial demonstrates a personalization plugin, but its official repository is archived. An existing installation may still be useful; adopting it now requires a maintenance decision. A small explicit schema keeps this walkthrough independent of that dependency.

Define the experiment before changing the schema

Give each system a clear responsibility

Sanity stores the content alternatives and their publication state. GrowthBook supplies the assignment rules and experiment analysis. Next.js applies the rules to the current visitor and renders the selected content. Your analytics pipeline carries exposure and outcome events into the data source used for analysis.

Keep those responsibilities separate even if the interfaces make them feel connected. A Sanity document revision is not an experiment assignment. A flag evaluation is not necessarily a page view. A button click is not automatically a completed signup. Naming each boundary makes the first broken assumption easier to find.

Start with a concrete hypothesis: replacing a feature-focused offer headline with an outcome-focused headline will increase completed trial signups per eligible visitor. Keep the offer, destination, and other page content consistent if the headline is the intended intervention. If you change the whole page, describe the treatment as a page package rather than attributing the result to one sentence.

Choose the assignment unit and primary metric before implementation. This example uses an anonymous browser identifier. An account-level buying journey may need an account identifier instead. A user who moves between anonymous and authenticated states needs a deliberate identity policy; switching hash identifiers during an experiment can change their assignment.

Estimate whether the available traffic can answer the question in a useful period. Start from the current completed-signup rate, the smallest improvement that would change your decision, and the traffic eligible for this specific offer. Total site visits can exaggerate the usable sample when only a small share reaches the tested surface. If the required duration is impractical, consider a more consequential intervention or a higher-traffic surface rather than replacing completed signups with an easy click metric simply to obtain a result.

Record what would stop the test for operational reasons, separately from the statistical decision. A missing content variant, broken signup destination, or substantial variation-specific delivery failure requires investigation even when the observed signup rate looks favorable. An experiment should teach you about the intended experience, not reward a branch that quietly excludes difficult cases.

Establish the implementation contract

Use one set of identifiers throughout the setup:

ItemValue in this walkthroughResponsibility
Sanity document typeofferPageHolds editable headline alternatives
Sanity document IDlanding-offerSelects this published offer
GrowthBook feature keysanity-offer-copyReturns control or variant
Experiment tracking keysanity-offer-copy-v1Identifies the experiment in events
Assignment attributeidUses the first-party gbid cookie

Feature values select your content. Experiment and variation IDs identify assignments for analysis. Do not assume the variation ID is the same as the string value: a result can have a variation key such as 0 while returning the content value control.

The application should already have a working App Router layout, TypeScript configuration, and initialized Sanity Studio. Use a supported Node version for your installed framework and packages. Install the content client and GrowthBook SDK inside the application, then record the resolved versions in the lockfile:

npm install @sanity/client @growthbook/growthbook server-only
npm ls @sanity/client @growthbook/growthbook next react

These examples use current GrowthBookClient and Sanity client interfaces. Keep the lockfile with the implementation so future upgrades can be checked against the same rendering and attribution tests.

Make the rendering assumptions explicit

This walkthrough uses request-time rendering with Cache Components disabled. Merge this property into your existing configuration rather than replacing unrelated settings:

// next.config.ts
import type { NextConfig } from "next";

const nextConfig: NextConfig = { cacheComponents: false };
export default nextConfig;

Restart the application after changing configuration. An application already using Cache Components needs a design that keeps request-specific identity outside shared cache boundaries; copying a route-level option into that different model is insufficient.

Add these application environment variables, substituting your own project and SDK connection values:

# .env.local
SANITY_PROJECT_ID=your-project-id
SANITY_DATASET=your-public-test-dataset
GROWTHBOOK_API_HOST=https://cdn.growthbook.io
GROWTHBOOK_CLIENT_KEY=sdk-your-connection-key

The example reads public published content without a Sanity token. A private dataset needs a server-side read credential and appropriate access controls. Never turn a draft-reading credential into a public environment variable just to make a browser request work. An SDK connection key is also different from an administrative GrowthBook API credential.

Publish two variants in Sanity

Add validated fields to the content model

Create the document type in your Studio's schema directory. The ordinary Sanity schema helpers express the editor-facing contract without introducing a plugin-specific field format:

// schemaTypes/offerPage.ts
import { defineField, defineType } from "sanity";

export const offerPage = defineType({
  name: "offerPage",
  title: "Offer page",
  type: "document",
  fields: [
    defineField({
      name: "controlHeadline",
      title: "Control headline",
      type: "string",
      validation: (rule) => rule.required().max(120),
    }),
    defineField({
      name: "variantHeadline",
      title: "Variant headline",
      type: "string",
      validation: (rule) => rule.required().max(120),
    }),
  ],
});

Register offerPage in the schema array your Studio configuration actually imports. For a new minimal schema module, that registration is:

// schemaTypes/index.ts
import { offerPage } from "./offerPage";
export const schemaTypes = [offerPage];

For an existing Studio, append the imported type to its existing array. Do not remove the other document types. Confirm the new type appears in Studio, both fields are required, and long headlines produce a validation error.

Create a test document with ID landing-offer, fill both fields, and publish it. If your Studio creates a random document ID, use that actual ID consistently in the query parameter below instead. The displayed document title and its _id are different concepts; verify the ID with your content inspection tool. Studio validation improves the editing experience but does not constrain every API write, so the application must validate the stored fields too.

Use obviously different temporary headlines during wiring, then replace them with the approved experimental copy before launch. Distinct diagnostic text makes it easier to discover an inverted mapping than two subtle variants that look almost identical.

For example, the published test document should have this shape. Enter the two headline values through Studio; the JSON is a reference for inspecting the stored document, not a command that publishes it:

{
  "_id": "landing-offer",
  "_type": "offerPage",
  "controlHeadline": "Project tools for distributed teams",
  "variantHeadline": "Keep your next project moving"
}

Sanity supplies revision and timestamp metadata when the document is stored. Verify the published query returns the same ID and authored values before connecting the experiment.

Treat the headline pair as one editorial unit. Publishing control while treatment is still a draft leaves the application without the complete set it needs for enrollment. For a larger content package, store or reference the title, supporting text, image, and destination together so the application selects a coherent package. Independently selecting each field can create combinations nobody reviewed and a different experimental question from the one in the brief.

Keep ordinary navigation and essential offer terms consistent unless they are part of the treatment. A text experiment should not accidentally send the two groups to different signup flows. Have the content owner inspect both complete journeys, including mobile wrapping and long translations, rather than approving isolated strings in the editor.

Query published content with a fixed API version

Create the server-only content module. GROQ parameters keep the document identifier separate from the query text. An explicit published perspective excludes in-flight unpublished changes:

// lib/offer-content.ts
import "server-only";
import { createClient } from "@sanity/client";

export type Offer = {
  _id: string;
  _rev: string;
  controlHeadline: string;
  variantHeadline?: string;
};

const client = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: process.env.SANITY_DATASET!,
  apiVersion: "2026-03-01",
  perspective: "published",
  useCdn: false,
});

export async function getOffer(): Promise<Offer | null> {
  return client.fetch<Offer | null>(
    '*[_type == "offerPage" && _id == $id][0]{_id, _rev, controlHeadline, variantHeadline}',
    { id: "landing-offer" },
    { cache: "no-store" },
  );
}

Confirm the query returns the published document and both expected strings. Keep the API date pinned; using today's date on every request can silently change the API contract later. TypeScript describes the expected result but does not validate existing stored data, which is why the rendering code also checks required content.

The uncached fetch is a simple correctness baseline, not the final performance strategy for every site. Sanity's Next.js client guidance distinguishes CDN use, revalidation, and preview needs. You can cache shared published content once its invalidation behavior is understood; avoid caching a visitor's chosen rendered page under a key shared with everyone else.

Keep preview changes out of the live treatment

A preview should help an editor inspect an alternative without enrolling that preview session in the live test. Keep draft access, forced assignments, and preview routing outside the production analytics stream. Check that an unpublished change is visible in the editor's preview but absent from a normal published query.

Record the published revision used when the experiment starts. A new edit to either headline changes the treatment definition even if the flag key and allocation remain the same. Freeze the tested copy or create an explicitly reviewed new phase; otherwise the readout combines materially different experiences under one label.

Check your experiment design

Open the experiment-design workflow to turn your hypothesis, assignment unit, and metrics into a reviewable specification. Configure API access as directed and review the plan before launch.

Open Experiment Design Instructions

Assign a visitor before rendering

Configure a string-valued feature experiment

In GrowthBook, create the string feature sanity-offer-copy with default value control. Create and link an experiment with tracking key sanity-offer-copy-v1, values control and variant, and the intended traffic allocation. Set the hash attribute to id, select the correct SDK environment, and add the primary goal and guardrails.

The experiment configuration and feature-flag experiment rules determine whether evaluation enrolls the visitor. A default or forced feature value is useful for debugging, but it does not automatically create the same assignment event as an active experiment rule.

Before exposing real visitors, verify the SDK connection returns the intended feature in the intended environment. A correctly spelled feature in another project or environment can still leave this application on the fallback value. Keep the experiment stopped or restricted to your test audience while validating the integration.

Establish the identifier before the page request

Next.js cookies can be read in a Server Component, but outgoing cookie writes belong in a Route Handler or server function. Use a small bootstrap route to establish the anonymous identifier, then redirect to the offer:

// app/experiment-session/route.ts
import { randomUUID } from "node:crypto";
import { NextRequest, NextResponse } from "next/server";

export const runtime = "nodejs";

export function GET(request: NextRequest) {
  const response = NextResponse.redirect(new URL("/offer?session=1", request.url));
  if (!request.cookies.get("gbid")?.value) {
    response.cookies.set("gbid", randomUUID(), {
      httpOnly: true,
      secure: process.env.NODE_ENV === "production",
      sameSite: "lax",
      path: "/",
      maxAge: 60 * 60 * 24 * 180,
    });
  }
  response.headers.set("Cache-Control", "private, no-store");
  return response;
}

Visit this route directly during initial testing and confirm the response sets one cookie and redirects to /offer. A repeat request should retain the same ID. This is a technical identity example; fit the identifier's lifetime and use to your application's consent and identity policy.

The query marker records one bootstrap attempt. If cookies are unavailable, the page below uses a non-enrolled baseline instead of redirecting again. Do not assign a fresh random ID on every render, because the same person could otherwise cross between variations and contaminate the comparison.

Reuse feature data while keeping visitor context separate

The current GrowthBook Node SDK supports a shared client with per-request scoped instances. Create the client once per process and keep the visitor's attributes in the scoped context:

// lib/growthbook.ts
import "server-only";
import { GrowthBookClient } from "@growthbook/growthbook";

const client = new GrowthBookClient({
  apiHost: process.env.GROWTHBOOK_API_HOST,
  clientKey: process.env.GROWTHBOOK_CLIENT_KEY,
});
export async function getGrowthBook() {
  const result = await client.init({ timeout: 2000 });
  if (!result.success) {
    console.error("GrowthBook payload unavailable", result.source);
    return null;
  }
  return client;
}

Calling init from the request helper lets the SDK use its shared feature cache and refresh behavior instead of retaining one initial promise forever. A successful cache response can still reflect the SDK's freshness policy. For stricter operational requirements, configure and test polling or streaming appropriate to your deployment; do not assume a timer survives a serverless invocation. Verify how a stopped experiment reaches a warm instance.

The offer page fetches content before evaluating the feature, so a missing document cannot create an assignment event for an unrenderable offer. The request-local callback queues experiment identifiers for the browser boundary:

// app/offer/page.tsx
import { cookies } from "next/headers";
import { notFound, redirect } from "next/navigation";
import { getOffer } from "../../lib/offer-content";
import { getGrowthBook } from "../../lib/growthbook";
import { ExperimentView } from "./experiment-view";

export const runtime = "nodejs";
export const dynamic = "force-dynamic";

export default async function OfferPage({ searchParams }: {
  searchParams: Promise<{ session?: string }>;
}) {
  const id = (await cookies()).get("gbid")?.value;
  if (!id && (await searchParams).session !== "1") redirect("/experiment-session");
  const offer = await getOffer();
  if (!offer || typeof offer.controlHeadline !== "string" || !offer.controlHeadline.trim()) notFound();

  const assignments: { experiment_id: string; variation_id: string }[] = [];
  const validVariant = typeof offer.variantHeadline === "string" && !!offer.variantHeadline.trim();
  if (!validVariant) console.error("Offer variant content is invalid", offer._rev);
  const client = id && validVariant ? await getGrowthBook() : null;
  const scoped = client?.createScopedInstance({
      attributes: { id },
      trackingCallback: (experiment, result) => {
        if (result.value !== "control" && result.value !== "variant") {
          console.error("Invalid offer assignment value", {
            experiment_id: experiment.key, variation_id: result.key,
            user_id: id, assigned_value: result.value,
          });
          return;
        }
        assignments.push({ experiment_id: experiment.key, variation_id: result.key });
      },
    });
  const selected = scoped?.getFeatureValue<string>("sanity-offer-copy", "control") ?? "control";
  const actual = selected === "variant" && offer.variantHeadline?.trim()
    ? "variant" : "control";
  if (actual !== selected) console.error("Unexpected offer feature value", selected);
  const headline = actual === "variant" ? offer.variantHeadline! : offer.controlHeadline;
  const deliveredAssignments = actual === selected ? assignments : [];

  return (
    <main data-content-variant={actual}>
      <h1>{headline}</h1>
      <a href="/signup">Start a trial</a>
      <ExperimentView assignments={deliveredAssignments}
        contentRevision={offer._rev} contentVariant={actual} />
    </main>
  );
}

The /signup destination represents your existing application flow; implement or replace it before testing completed signups. If treatment content is missing, the page serves control outside enrollment and logs a configuration problem. If payload loading fails, it also stays outside enrollment. Investigate these conditions before interpreting an experiment: a pleasant fallback must not hide that part of the intended population received no experimental treatment.

Validate the assigned value inside the callback as well as the selected rendering value. A malformed experiment value such as null can resolve to the SDK fallback while still generating an assignment callback. The diagnostic preserves its experiment, variation, and identity; it must not become a view of a valid control treatment. Repair the invalid configuration and investigate the affected interval rather than treating filtered events as a trustworthy comparison.

Validating the complete pair before evaluation makes that content eligibility check independent of the eventual assigned arm. Failures after assignment require a different response. Preserve assignment and delivery-error diagnostics, examine whether failure rates differ by arm, and halt or repair the test when delivery is unreliable. Suppressing failed treatment views and analyzing only successful displays can select a favorable subset; it is not a clean way to rescue an experiment.

Record the experience that reached the browser

Defer the view event until the content mounts

A server render can result from prefetching or a navigation the visitor never completes. Use the SDK's deferred-tracking pattern to carry request-local assignments into a mounted client component. This example sends a same-origin diagnostic event with the actual content revision:

// app/offer/experiment-view.tsx
"use client";
import { useEffect, useRef } from "react";

type Assignment = { experiment_id: string; variation_id: string };

export function ExperimentView({ assignments, contentRevision, contentVariant }: {
  assignments: Assignment[]; contentRevision: string; contentVariant: string;
}) {
  const sent = useRef(new Set<string>());
  useEffect(() => {
    for (const assignment of assignments) {
      const key = `${assignment.experiment_id}:${assignment.variation_id}:${contentRevision}`;
      if (sent.current.has(key)) continue;
      sent.current.add(key);
      const accepted = navigator.sendBeacon("/api/experiment-view", new Blob([
        JSON.stringify({ ...assignment, content_revision: contentRevision,
          content_variant: contentVariant, event_id: crypto.randomUUID() }),
      ], { type: "application/json" }));
      if (!accepted) console.warn("Experiment view was not queued");
    }
  }, [assignments, contentRevision, contentVariant]);
  return null;
}

The ref prevents repeated effect execution within the mounted component from sending the same assignment twice. It is not a warehouse-wide deduplication strategy. A remount can send another event, network delivery can fail, and a queued beacon is not an ingestion receipt. A durable production collector should deduplicate retries by event ID and preserve the exposure policy your analysis expects.

Mounting establishes that client execution reached the rendered route. It does not prove the headline was visible in the viewport or read by a person. If the experiment concerns a component below the fold, define the appropriate visibility trigger separately. Keep the definition symmetric across variations.

Inspect a real request without inventing an analytics integration

Add the Route Handler below as a local diagnostic sink. It gets the identity from the same-origin cookie instead of trusting a user ID supplied by JavaScript:

// app/api/experiment-view/route.ts
import { NextRequest } from "next/server";

export async function POST(request: NextRequest) {
  const id = request.cookies.get("gbid")?.value;
  if (!id) return new Response("Missing identity", { status: 400 });
  let event: Record<string, unknown>;
  try { event = await request.json(); }
  catch { return new Response("Invalid JSON", { status: 400 }); }
  const fields = ["experiment_id", "variation_id", "content_revision", "content_variant", "event_id"];
  if (!event || typeof event !== "object" || fields.some((field) =>
    typeof event[field] !== "string" || !(event[field] as string).trim() ||
    (event[field] as string).length > 200)) {
    return new Response("Invalid event", { status: 400 });
  }
  console.log(JSON.stringify({ event: "experiment_view", user_id: id,
    received_at: new Date().toISOString(), ...Object.fromEntries(fields.map((f) => [f, event[f]])) }));
  return new Response(null, { status: 204 });
}

Open /offer in a browser, inspect the POST in the network panel, and confirm the local server prints the expected experiment key, variation key, document revision, and identity. A command-line HTML fetch alone should not produce this mounted-client event.

Before a real experiment, replace the console sink with your existing durable analytics ingestion path. Add its normal validation, abuse controls, delivery monitoring, retention rules, and retry behavior. Browser-provided events are observations, not an authorization mechanism. Do not use an experiment event to grant access or trust an arbitrary client-supplied field for a billing decision.

Integrate consent and tracker readiness into the same exposure policy for both arms. If your production tracker initializes asynchronously, queue the event until that supported tracker is ready rather than discarding whichever arm happened to mount first. Monitor the difference between server assignments, client events, accepted collector requests, and warehouse rows. Those counts describe different boundaries and need not be identical, but unexplained variation-specific losses can invalidate the result.

Join exposures to outcomes using the same unit

The data-source assignment query needs the experiment key, variation identifier, timestamp, and assignment unit in the warehouse schema you actually use. Map the diagnostic fields deliberately; a log line does not create that table or configure GrowthBook for you.

Verify an ingested exposure and a completed trial signup can be joined through the same identifier or a tested identity mapping. If signup replaces an anonymous cookie with an account ID, record the relationship rather than treating the two unrelated strings as equivalent. Inspect timestamps and duplicate handling as well as the existence of the events.

Define the signup outcome with Fact Tables and metrics. Keep the primary denominator tied to the eligible exposed population. A headline may increase button clicks while bringing in fewer completed or qualified signups, so retain the downstream business outcome and relevant guardrails instead of declaring the click rate sufficient.

Verify the route and operate the test

Check identity, content, and allocation separately

Run the application locally and establish a cookie jar. These commands inspect the HTTP route and selected HTML; they do not execute the client exposure component:

curl -i -c sanity-test-cookies.txt http://localhost:3000/experiment-session
curl -s -b sanity-test-cookies.txt http://localhost:3000/offer
curl -s -b sanity-test-cookies.txt http://localhost:3000/offer

The first response should redirect and set gbid. Both subsequent responses should show the same data-content-variant and matching headline while the experiment configuration remains unchanged. Keep the cookie jar private to the local test and remove it afterward.

Run the application's static checks and production build after integrating the files:

npx tsc --noEmit
npm run build

Resolve errors against the framework and package versions in your lockfile. A passing type check verifies interfaces, not live identity continuity or event ingestion. Start the production build using your application's normal script and repeat the browser and cookie checks there before testing the hosted deployment.

Use multiple isolated browser profiles or controlled test IDs to exercise both variations. A single identifier repeatedly receiving control is expected under deterministic assignment; it does not establish that treatment is broken. Conversely, manually forcing a feature value proves the rendering branch, not the randomized enrollment or exposure pipeline.

Test a missing treatment headline, missing document, stopped experiment, unavailable SDK payload, and unpublished content edit. The expected outcomes differ: a missing document should not produce a view; a missing treatment should not be mislabeled as treatment; a stopped experiment should reach the intended baseline after the configured payload refresh.

Inspect the deployed cache and event path

Repeat the checks on the actual deployment. Development mode can hide caching and prefetch behavior that changes under a production build or hosting layer. Inspect response headers, the rendered headline, browser navigation, and the events that arrive in the warehouse. Confirm that a response generated for one visitor cannot be reused as another visitor's assigned page.

Shared content caching can be valuable once correctness is established. Cache the complete published variant set or include every selection dependency in the appropriate cache key; keep request identity and evaluation outside shared state. Coordinate document publication with revalidation so a new revision does not remain partially deployed across regions or application instances.

Use a launch receipt that connects each boundary to evidence:

BoundaryEvidence to inspectBlock launch when
PublicationPublished query returns approved alternatives and revisionDraft or missing content reaches the live route
AssignmentSame identifier is stable; test audience exercises both valuesIdentity changes per request or environment is wrong
RenderingActual headline matches selected content keyHTML is shared across incompatible assignments
ExposureMounted route produces correctly attributed warehouse rowsPrefetches become views or fallback becomes treatment
OutcomeCompleted signup joins to the intended unitAttribution disappears at authentication or conversion

Inspect sample allocation and instrumentation before interpreting lift. A mismatch can reflect targeting, bot exclusion, missing events, or a broken branch. It is a reason to investigate the experiment population, not a reason to edit the result until the groups look balanced.

Finish with a durable content decision

Agree on duration, outcome maturity, meaningful effect size, and a decision framework before reading the results. An imprecise result may leave the copy decision unresolved. Absence of statistical significance does not prove that the variants perform identically.

When the decision is made, stop enrollment through the experiment configuration, verify the change reaches deployed instances, and publish the chosen content as the normal baseline. Keep the experiment record and the exact content revision with the learning. If you remove the feature branch, review the rendering and tracking code together so a retired test does not keep emitting misleading events.

Give the content owner a clear operating rule: which fields are frozen during a test, who can publish a revised treatment, and who checks the downstream measurement. With those rules in place, the next experiment can reuse the integration while keeping each new hypothesis and content version distinct.

Put your content test into practice

Set up GrowthBook for your Sanity content experiments, connect the outcomes you already track, and use a consistent process to decide what to publish next.

Get Started With GrowthBook

Table of Contents

Related Articles

See All Articles
Experiments

T-statistic explained: What the t-value tells you about your results

Nov 13, 2026
x
min read

An experiment report shows a t-value of 2.4. Is the product change large? Is it beneficial? Does it have a 95% chance of working? That single number cannot answer those questions.

A t-statistic compares an estimated quantity with a hypothesized value, using the estimate’s standard error as the measuring stick. It compresses the size of the observed departure and the precision of the estimate into one signed number. Understanding the two parts is more useful than memorizing a cutoff.

For experiment readers, the essential habit is to unpack the t-value before interpreting it. Establish what was subtracted, which uncertainty estimate sits underneath it, and what reference distribution applies. Then return to the effect in product units and the decision it needs to support.

Read t as a difference divided by uncertainty

Identify the numerator and its direction

The general structure is:

t = (estimated quantity - hypothesized value) / estimated standard error

In an independent treatment–control comparison, the estimated quantity might be the treatment mean minus the control mean. With a null difference of zero, a positive t means the estimated treatment mean is higher. Reverse the subtraction order and the sign reverses.

“Higher” does not necessarily mean “better.” More purchases may be desirable; more errors usually are not. A negative latency difference can support an improvement, while a positive error-rate difference can indicate harm. Read the direction in the metric’s own terms before assigning a business meaning to the sign.

The hypothesized value also matters. The usual zero-difference comparison asks whether the population mean difference is zero. Testing another specified difference changes the numerator. Neither calculation independently establishes an equivalence or non-inferiority claim; those require the appropriate hypotheses, margins and decision procedure.

Distinguish standard error from standard deviation

An observation standard deviation describes spread among observed unit outcomes. A standard error describes uncertainty in an estimator, such as a sample mean or a difference between means. They can share units while answering different questions.

For an ordinary mean of independent, identically distributed observations with finite variance, the estimated standard error is the sample standard deviation divided by the square root of the sample size. The standard-error calculation therefore changes when the number of independent observations changes, even if their measured spread stays the same.

This is why dividing a mean difference by the observation standard deviation does not give the usual two-sample t-statistic. It omits how precisely the means were estimated. The distinction between standard error and standard deviation is the first denominator check to make when a t-value looks surprising.

Unit-free does not mean effect size

Both numerator and standard error have the same measurement units, which cancel in their ratio. Converting a revenue analysis from dollars to cents multiplies both by 100 and leaves t unchanged. That is a useful consistency check for a positive linear change of units.

However, t does not describe the size of the product effect independently of precision. An observed difference of one dollar can have a small or large t-value depending on the standard error. Comparing t-values across two experiments is therefore not a reliable way to rank their business impact.

The statistic measures departure from the null in estimated standard-error units. Keep the raw difference beside it so the unit-free summary does not displace the quantity the team actually needs.

The calculation also needs a usable, positive estimated standard error. A zero or undefined denominator is a reason to inspect the data and model, rather than manufacture a finite t-value or call the result certain. Software may reject such inputs; for example, the R t-test reference documents an error for effectively constant input data.

Match the denominator to the study design

Different designs estimate different uncertainty

The formula’s short form hides a substantial choice: how was the standard error calculated? Three common cases illustrate why the design must come first.

DesignEstimated standard errorReference degrees of freedom
Ordinary one-sample meanSample SD divided by √nn − 1 under the classical model
Ordinary paired mean differenceSD of within-pair differences divided by √number of pairsNumber of independent complete pairs − 1 under the classical model
Independent Welch comparison√(s² treatment / n treatment + s² control / n control)Welch–Satterthwaite approximation using both group sizes and variances

In the paired case, the relevant spread is the spread of differences within pairs. Substituting the two arms’ separate standard deviations discards the relationship the design uses. The paired t calculation treats the differences as its observations; the independent information comes from the pairs.

For two independent groups, Welch’s calculation does not require equal population variances. The pooled Student calculation adds that assumption and uses a different variance estimate. The Welch-versus-Student comparison helps distinguish those cases. A label reading only “t-test” can conceal the choice.

Separate a reference model from an arithmetic ratio

You can calculate a difference divided by an estimated standard error without proving that the resulting statistic has the reference distribution you intend to use. Exact classical one-sample and paired t inference relies on the corresponding independent normal-observation model; in the paired case, this concerns the differences.

Welch’s reference distribution is an approximation with estimated degrees of freedom. It is not generally exact even for unequal-variance normal populations. Non-normal outcomes can support useful large-sample mean inference under appropriate conditions, but there is no sample-size threshold that makes every heavy tail or dependence problem disappear.

If accounts are randomized, individual events from one account are not automatically independent units. A standard error that ignores the account structure may understate uncertainty. More rows in an exported file do not by themselves justify a smaller denominator or larger degrees of freedom.

Plan for more precise estimates

Explore variance-reduction methods and their tradeoffs before your next test. Match each method to the outcome you intend to estimate.

Explore Variance Reduction

Hold the observed effect fixed and change precision

Work through an independent mean comparison

Consider a hypothetical fixed-horizon experiment with one completed revenue outcome per independent user. Control has mean revenue of $10; treatment has $11. Each arm has sample standard deviation $4 and 100 users. The target is the arithmetic mean difference, treatment minus control, and the null difference is zero.

The estimated effect is $1 per user. Using the independent unequal-variance standard error, square each group’s standard deviation, divide by its sample size, add those terms and take the square root. Here that is √(16/100 + 16/100), or about $0.5657.

Divide the $1 difference by $0.5657 to get t ≈ 1.7678. The estimated difference lies about 1.77 estimated standard errors above the zero-difference null. The sample standard deviation is still $4; the $0.5657 figure measures uncertainty in the difference between the two estimated means.

For these equal sample sizes and equal observed variances, the Welch–Satterthwaite calculation gives 198 degrees of freedom. Under the conventional Welch approximation, the two-sided p-value is approximately 0.07864 and the 95% confidence interval for the mean difference is approximately −$0.1155 to $2.1155.

Compare two hypothetical studies

Now consider a separate hypothetical study with the same group means and standard deviations, but 400 independent users per arm. The observed $1 difference has not grown. Its estimated standard error is smaller.

Summary100 users per arm400 users per arm
Observed mean difference$1$1
Sample SD in each arm$4$4
Estimated SE of the difference$0.5657$0.2828
t-statistic1.76783.5355
Welch reference df198798
Conventional two-sided p-value0.078640.0004304
Conventional 95% CI for the difference[−$0.1155, $2.1155][$0.4448, $1.5552]

Quadrupling each group’s sample size halves the estimated standard error when these observed standard deviations stay fixed. It doubles t for the same observed effect. The larger statistic reflects greater estimated precision here, not a larger observed revenue gain.

These are teaching summaries for separate studies. They are not an instruction to keep extending one experiment until significance appears. In a real extension, means and variances can change, and the stopping rule affects how the inference should be calibrated. An observed t-value also does not directly tell you the power of a future study.

Translate t using the appropriate reference distribution

Degrees of freedom and tails determine the threshold

A t-value needs a reference distribution before you can obtain a p-value or critical value. The Student t family includes a degrees-of-freedom parameter; smaller values generally give heavier tails. The same absolute statistic can therefore produce different tail probabilities at different df.

For a prespecified two-sided 5% test, the relevant critical value is the 97.5th percentile of the reference t distribution. It approaches about 1.96 as df becomes large, but 1.96 is not a universal t cutoff. Nor does every test use n − 1. Degrees of freedom depend on the calculation and may be fractional for Welch.

A one-sided alternative uses a specified tail and a different critical value. A negative statistic is not evidence in favor of an upper-tail alternative simply because its absolute value is large. Choose the direction before seeing outcomes; taking whichever tail looks favorable changes the decision rule.

A p-value conditions on the null model

For a conventional two-sided t-test, the p-value is the probability, under the null model and assumptions, of a statistic at least as extreme in either direction as the observed t. It is not the probability that the null is true, that chance caused the result, or that a future experiment will succeed.

The ASA’s p-value principles also distinguish statistical significance from effect magnitude and practical importance. A threshold crossing does not determine whether the expected benefit pays for the change. If you need the broader conditional-probability interpretation, the t-test p-value guide develops it separately.

Read the interval in the original units

The corresponding conventional interval is the estimated difference plus or minus the relevant critical t-value times its estimated standard error. For the same two-sided procedure at 5%, a 95% interval excluding zero agrees with rejecting a zero-difference null. That agreement requires compatible test, interval and adjustment settings.

Confidence describes the repeated-use coverage of the method under its assumptions. It does not assign a 95% posterior probability to the fixed true effect lying inside the particular computed interval. The confidence-interval construction makes the relationship between standard error, critical value and width explicit.

In the first study, the interval includes a small loss and a substantial gain. In the second, it excludes zero but still spans different levels of business value. A minimum worthwhile effect helps interpret that range. Neither a large p-value nor an interval containing zero proves equality; equivalence requires an analysis designed for that claim.

Audit the inputs before acting on a t-value

Trace an unexpected statistic to its components

Start with the estimated effect and null value. Confirm the subtraction order, currency or time units, eligible population and observation window. A swapped group order explains a sign change; it does not explain a change in a correctly calculated two-sided p-value.

Then inspect the denominator. Check group sizes and sample variances at the actual independent unit. Look for duplicated joins, repeated events counted as independent observations, incomplete follow-up and treatment-dependent missingness. The arithmetic can be internally consistent while the data fail to represent the intended experiment.

Inspect the analysis settings too. A ratio metric, regression adjustment or variance-reduction method can have a standard error different from the simple independent-mean formula above. Do not attempt to reproduce a dashboard by substituting top-line traffic counts into the wrong calculation. GrowthBook’s statistical effect calculations describe its estimators and frequentist reference methods, including distinct metric types.

Keep the decision rule attached to the evidence

The worked examples use conventional fixed-horizon inference. Repeatedly inspecting that calculation and stopping at a favorable threshold changes its false-positive behavior. A planned sequential analysis is a different inferential procedure; GrowthBook documents an asymptotic confidence-sequence approach with its own settings. It does not repair incorrect units or missing outcomes.

Likewise, selecting the largest t among many metrics, variants or segments creates a testing problem beyond a single comparison. Preserve the planned comparisons and applicable corrections rather than treating the most impressive number as an isolated result.

A useful readout names the estimated effect, interval, test, df, direction and sample sizes, followed by the measurement checks and decision threshold. The team can then assess what the evidence supports and whether that effect is worth shipping. Use the t-value to understand precision relative to the null; keep the launch decision grounded in effects on users and the business.

Review the result in context

Open the official experiment-analysis workflow to inspect saved results, health checks and analysis settings with configured API access. It can refresh a result snapshot when needed; stopping or rollout is separate.

View the Analysis Workflow
Experiments

How to run A/B tests in Strapi

Nov 13, 2026
x
min read

A published Strapi document is content. An A/B test also needs an assignment, a delivered experience, and an outcome you can attribute.

The integration becomes unreliable when those pieces disagree. An older example may read data.attributes, while your Strapi 5 endpoint returns flat fields. A numeric database ID may be mistaken for a document ID. An internal service call may return a draft even though a similar REST request defaults to published content.

This guide builds an explicit Strapi 5 content test using a small server-rendered Node application and GrowthBook. It validates the complete published content pair before enrolling a visitor, preserves a stable browser identifier, and records an exposure only after browser execution reaches the selected page.

The code also fingerprints the reviewed strings so a later edit can be rejected before assignment. The local event collector demonstrates the measurement boundary; a real experiment still needs your existing signup flow, durable analytics delivery and verified warehouse joins.

Define the Strapi 5 delivery contract

Pick a content model you can inspect

The existing GrowthBook Strapi integration guide demonstrates Dynamic Zone components selected through a frontend SDK. The reference here uses two ordinary string fields and server-side evaluation. It does not require a separately installed Strapi experimentation plugin or depend on an unverified plugin compatibility claim.

Start with a hypothesis about a meaningful outcome: an outcome-focused headline will increase completed trial signups per eligible offer-page visitor. Keep the offer and signup destination consistent if headline copy is the intervention. If the change includes a larger component package, describe that package as the treatment.

A plain-text pair makes the first integration easier to inspect. You can verify the exact response, render the selected value safely, and compare the approved strings. When extending the model to components or Dynamic Zones, validate their full structure and use a reviewed variation mapping. Reordering an array must not silently redefine which experience belongs to control.

For example, an editorial interface might place an alternative card first because it is easier to find, while the frontend still interprets index zero as control. The content can remain valid JSON and render perfectly while the experiment labels have effectively switched. Use explicit variation keys or a constrained, reviewed ordering contract when extending the simple pair. Check the rendered package against the experiment mapping, not just that every component has a title.

Use the document API that your code actually calls

Strapi 5 REST responses expose normal content fields directly under data. The document endpoint uses documentId; the numeric id in a response is not the identifier this guide puts in the URL. Some plugin endpoints have different response conventions, so this contract applies to an ordinary generated collection endpoint.

The production request explicitly sets REST status to published. This is also the REST default when Draft & Publish is enabled. The internal Document Service API defaults differently, to draft. Do not transfer that internal default—or an older v4 parameter—into this REST implementation.

Keep locale explicit as well. The requested document and locale together identify the content version being delivered. This guide checks the returned locale instead of borrowing another CMS's fallback behavior or assuming a valid document ID means every translation exists.

The document concept groups localized and draft/published variations around a document identifier. That does not make the document identifier an immutable snapshot of the text. Keep source identity, publication state and approved content as separate fields in the experiment record. Changing one is a deployment decision even when the others remain familiar.

Define assignment and measurement before launch

Use one stable assignment unit. The example creates an anonymous first-party browser ID; account-level interactions may require account assignment instead. Plan the anonymous-to-authenticated transition so signup measurement does not lose the identifier or enroll one person under several unrelated IDs.

Choose a primary business outcome, a small diagnostic funnel and relevant guardrails. A headline can increase button clicks without increasing completed or qualified signups. Estimate eligible traffic and the effect that would change your decision before starting a test that cannot resolve the question in a useful period.

Separate content ownership from experiment operation. An editor approves the published pair, an engineer owns delivery and fallback behavior, and the analytics owner verifies exposure and outcome data. Name who can change the tested content and how a change starts a new reviewed comparison.

Model, publish and inspect the content pair

Create a localized collection in development

Use an existing Strapi 5 test application with Draft & Publish enabled for the collection. Configure internationalization and the intended locale if the content is localized. The Content-type Builder belongs to the development workflow; deploy schema changes through your normal application process.

Create a collection named Experiment page with singular API ID experiment-page and plural API ID experiment-pages. A minimal schema at src/api/experiment-page/content-types/experiment-page/schema.json is:

{
  "kind": "collectionType",
  "collectionName": "experiment_pages",
  "info": {
    "singularName": "experiment-page",
    "pluralName": "experiment-pages",
    "displayName": "Experiment page"
  },
  "options": {"draftAndPublish": true},
  "pluginOptions": {"i18n": {"localized": true}},
  "attributes": {
    "controlHeadline": {
      "type": "string",
      "required": true,
      "pluginOptions": {"i18n": {"localized": true}}
    },
    "variantHeadline": {
      "type": "string",
      "required": true,
      "pluginOptions": {"i18n": {"localized": true}}
    }
  }
}

For an existing model, merge the relevant changes rather than replacing its other fields. Restart or rebuild according to your development workflow, create a document, enter both approved headlines and publish the intended locale. Copy its actual documentId from the API response or your document inspection workflow.

Review model changes and content changes through their appropriate paths. Adding a field affects application schema and deployment; editing a headline changes a content entry. Give the editor clear field descriptions so control means the approved baseline and variant means the approved experimental alternative. A required field helps authoring, but the renderer still validates the actual response from the deployed endpoint.

With internationalized content, inspect the specific localized pair that will be eligible for the experiment. Translating control but leaving treatment in an unintended language changes more than the hypothesis describes. Keep locale targeting and content approval aligned, and decide in advance whether results across languages belong to one planned analysis or separate comparisons.

Because both alternatives are plain strings, this request needs no relation population. Components, relations and media require an explicit population strategy when added later; populate=* is not a substitute for defining the treatment's dependencies and permissions.

Use a read token on the rendering server

Create a scoped Content API token for the renderer. A read-only token supports find and findOne; a Custom token can narrow the relevant endpoint access. Keep it on the application server. You do not need to expose an admin token or broadly enable public-role access for this authenticated server request.

Read-only describes allowed operations, not a universal guarantee that a caller can never request drafts. Fix production status and locale in trusted code. Keep editor preview routes and any draft-capable configuration separate from live enrollment and analytics.

When extending the endpoint with a language switcher, select from an approved locale set rather than forwarding every browser query parameter to Strapi. A request parameter named status should not turn the production renderer into a draft-content proxy. Keep the collection path and publication choice fixed, and validate the returned document and locale even after constructing a valid request.

Check permissions on the actual route you use. A working admin interface does not prove the rendering token can read the generated Content API endpoint. Conversely, an access problem is not a reason to grant broad public access to the entire model. Confirm the token's scope and the endpoint response independently of experiment assignment.

Create a reference Node project, or integrate the files into your existing application. Use a modern Node runtime compatible with the locked dependencies, such as Node 22 or later:

npm init -y
npm install express@5 cookie-parser @growthbook/growthbook@2.0.0
mkdir public

Add server configuration. STRAPI_ORIGIN is the CMS origin, including a port when needed, without /admin or /api appended:

# .env.local
STRAPI_ORIGIN=http://localhost:1337
STRAPI_DOCUMENT_ID=your-document-id
STRAPI_LOCALE=en
STRAPI_READ_TOKEN=your-read-token
STRAPI_EXPECTED_SHA256=
GROWTHBOOK_CLIENT_KEY=sdk-your-test-connection-key
PORT=3000

Use the appropriate HTTPS origin outside your local development setup. Store credentials through your deployment's configuration mechanism, not in public HTML or query parameters.

Validate the published response and reviewed fingerprint

The following reader fetches one document by documentId, with explicit locale selection. It rejects a missing, draft, wrong-locale or malformed pair before the SDK can assign anyone:

// strapi-content.mjs
import { createHash } from "node:crypto";

export async function getPublishedPair() {
  for (const name of ["STRAPI_ORIGIN", "STRAPI_DOCUMENT_ID", "STRAPI_LOCALE", "STRAPI_READ_TOKEN"]) {
    if (!process.env[name]) throw new Error(`Missing ${name}`);
  }
  const origin = new URL(process.env.STRAPI_ORIGIN);
  if (!["http:", "https:"].includes(origin.protocol) || origin.pathname !== "/" ||
      origin.search || origin.hash || origin.username || origin.password) throw new Error("Invalid CMS origin");
  const documentId = process.env.STRAPI_DOCUMENT_ID;
  const locale = process.env.STRAPI_LOCALE;
  const url = new URL(`/api/experiment-pages/${encodeURIComponent(documentId)}`, origin);
  url.searchParams.set("status", "published");
  url.searchParams.set("locale", locale);
  const response = await fetch(url, { headers: { Authorization: `Bearer ${process.env.STRAPI_READ_TOKEN}` },
    signal: AbortSignal.timeout(3000) });
  if (!response.ok) throw new Error(`Strapi delivery status ${response.status}`);
  const payload = await response.json();
  const data = payload?.data;
  if (!data || Array.isArray(data) || data.documentId !== documentId || data.locale !== locale ||
      typeof data.publishedAt !== "string" || !Number.isFinite(Date.parse(data.publishedAt)) ||
      typeof data.controlHeadline !== "string" || !data.controlHeadline.trim() ||
      typeof data.variantHeadline !== "string" || !data.variantHeadline.trim()) {
    throw new Error("Invalid published Strapi content pair");
  }
  const canonical = JSON.stringify([origin.origin, documentId, locale,
    data.controlHeadline, data.variantHeadline]);
  const sha256 = createHash("sha256").update(canonical).digest("hex");
  const expected = process.env.STRAPI_EXPECTED_SHA256;
  if (expected && (!/^[0-9a-f]{64}$/i.test(expected) || expected.toLowerCase() !== sha256)) {
    throw new Error("Strapi content differs from the reviewed pair");
  }
  return { copy: { control: data.controlHeadline, variant: data.variantHeadline },
    source: { instance: origin.origin, documentId, locale, publishedAt: data.publishedAt, sha256 } };
}

Leave the optional expected hash blank for the first read-only inspection, with Strapi running and the document published. Run:

node --env-file=.env.local --input-type=module -e "import {getPublishedPair} from './strapi-content.mjs'; console.log(await getPublishedPair());"

Confirm the actual approved pair, source origin, document and locale. Set STRAPI_EXPECTED_SHA256 to the returned digest before enrollment. The SHA-256 calculation fingerprints those exact strings and source dimensions. It does not retrieve an older Strapi version or grant access to anything.

documentId, updatedAt and publishedAt are useful context, but they do not freeze a treatment. The fingerprint rejects changes to the reviewed pair before assignment. Identical strings can retain the same digest after republication; publication metadata is recorded separately. Extend the fingerprint when additional content becomes part of the treatment.

Treat updating the expected digest as a review step. First inspect the new delivered pair, confirm why it changed, and decide whether the current experiment can continue under a documented phase or needs a new experiment. Copying the latest digest merely to clear an unavailable response defeats the check. Preserve the approved strings alongside the digest so a reviewer can understand the change without reconstructing it from a hash.

The digest covers the two headline strings and source dimensions in this implementation. It does not cover an unrelated image, price, signup destination or application deployment. If those become part of the intervention, extend the reviewed package and its validation. A useful content contract is explicit about both its coverage and the changes that require another review.

Review the experiment before launch

Open the experiment-design workflow to turn your content hypothesis, assignment unit, and metrics into a reviewable specification. Configure API access as directed and review the plan before launch.

Open Experiment Design Instructions

Render the assigned page with request-local context

Link the content values to an experiment

Create a string feature strapi-offer-copy, defaulting to control. Link an experiment with tracking key strapi-offer-copy-v1, mapping the two string values to the intended variations. Set hash attribute id, the correct SDK environment, targeting and goals in the experiment configuration.

The content value chooses a string from the pair. The experiment and variation identifiers describe the assignment for analysis. Preserve experiment.key and result.key; do not substitute the headline or assume a variation key must equal the content value.

The GrowthBook Node SDK can share feature payload data while keeping each visitor's context in a scoped instance. The server initializes through the request path and checks success, using the SDK's caching and refresh policy rather than retaining one startup promise forever. Test stopping and refreshing on warm instances; a shared cache is not an instant global update guarantee.

Keep the browser identity stable and the output safe

Create the server below. The existing /signup application flow is a prerequisite: implement or replace that link before measuring completed signups. The collector is a local diagnostic endpoint until connected to durable analytics.

// server.mjs
import express from "express";
import cookieParser from "cookie-parser";
import { randomUUID } from "node:crypto";
import { fileURLToPath } from "node:url";
import { GrowthBookClient } from "@growthbook/growthbook";
import { getPublishedPair } from "./strapi-content.mjs";

const app = express();
app.use(cookieParser());
app.use(express.static(fileURLToPath(new URL("./public/", import.meta.url))));
const experimentKey = "strapi-offer-copy-v1";
const gb = new GrowthBookClient({ apiHost: "https://cdn.growthbook.io",
  clientKey: process.env.GROWTHBOOK_CLIENT_KEY });
const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
const identity = (req) => typeof req.cookies.gb_strapi_id === "string" && uuid.test(req.cookies.gb_strapi_id)
  ? req.cookies.gb_strapi_id : null;
const html = (value) => value.replace(/[&<>"']/g,
  (char) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[char]);
const jsonData = (value) => JSON.stringify(value).replace(/</g, "\\u003c");

app.get("/experiment-session", (req, res) => {
  res.set("Cache-Control", "private, no-store");
  if (!identity(req)) res.cookie("gb_strapi_id", randomUUID(), { httpOnly: true,
    secure: process.env.NODE_ENV === "production", sameSite: "lax", path: "/",
    maxAge: 180 * 24 * 60 * 60 * 1000 });
  res.redirect(303, "/offer?session=1");
});

app.get("/offer", async (req, res) => {
  res.set("Cache-Control", "private, no-store");
  const id = identity(req);
  if (!id && req.query.session !== "1") return res.redirect(303, "/experiment-session");
  try {
    const content = await getPublishedPair();
    const assignments = [];
    let selected = "control";
    if (id) {
      const initialized = await gb.init({ timeout: 2000 });
      if (initialized.success) {
        const scoped = gb.createScopedInstance({ attributes: { id, locale: content.source.locale },
          trackingCallback: (experiment, result) => {
            if (experiment.key !== experimentKey) return;
            if (result.value !== "control" && result.value !== "variant") {
              console.error("Invalid Strapi assignment", { experiment_id: experiment.key,
                variation_id: result.key, user_id: id, value: result.value });
              return;
            }
            assignments.push({ experiment_id: experiment.key, variation_id: result.key, value: result.value });
          } });
        const result = scoped.evalFeature("strapi-offer-copy");
        if (result.value === "control" || result.value === "variant") selected = result.value;
        else console.error("Invalid Strapi feature value", result.value);
      } else console.error("Feature payload unavailable", initialized.source);
    }
    const context = { assignments: assignments.filter((item) => item.value === selected),
      content_variant: selected, source: content.source };
    res.type("html").send(`<!doctype html><html lang="${html(content.source.locale)}">
      <head><meta charset="utf-8"><title>Trial offer</title></head><body>
      <main data-content-variant="${selected}"><h1>${html(content.copy[selected])}</h1>
      <a href="/signup">Start a trial</a></main>
      <script type="application/json" id="strapi-exposure">${jsonData(context)}</script>
      <script src="/experiment-view.js" defer></script></body></html>`);
  } catch (error) {
    console.error("Strapi offer unavailable", error.message);
    res.status(503).type("html").send("<h1>This offer is temporarily unavailable</h1>");
  }
});

app.post("/experiment-view", express.json({ limit: "4kb" }), (req, res) => {
  const id = identity(req);
  const event = req.body;
  const source = event?.source;
  const fields = ["experiment_id", "variation_id", "event_id", "content_variant"];
  const sourceFields = ["instance", "documentId", "locale", "publishedAt", "sha256"];
  if (!id || !event || typeof event !== "object" ||
      fields.some((key) => typeof event[key] !== "string" || !event[key].trim() || event[key].length > 200) ||
      event.experiment_id !== experimentKey || !["control", "variant"].includes(event.content_variant) ||
      !source || typeof source !== "object" || sourceFields.some((key) =>
        typeof source[key] !== "string" || !source[key].trim() || source[key].length > 300) ||
      !/^[0-9a-f]{64}$/i.test(source.sha256) || !Number.isFinite(Date.parse(source.publishedAt))) {
    return res.sendStatus(400);
  }
  console.log(JSON.stringify({ event: "experiment_view", user_id: id,
    received_at: new Date().toISOString(), ...Object.fromEntries(fields.map((key) => [key, event[key]])),
    source: Object.fromEntries(sourceFields.map((key) => [key, source[key]])) }));
  res.sendStatus(204);
});

app.listen(Number(process.env.PORT || 3000));

The cookie parser can produce non-string values, so identity is validated before assignment. Express cookie maxAge uses milliseconds. The redirect establishes the cookie before the offer request; its one-attempt marker avoids looping when persistence fails. Adapt identity and consent behavior to the application's existing policy.

The headline is escaped as HTML text. The JSON data element separately escapes less-than signs so a closing-script sequence cannot terminate it. These are different output contexts. Do not use this plain-string renderer as a general HTML or Rich Text renderer.

Keep failure behavior visible

A missing document, wrong locale, draft payload or changed fingerprint fails before enrollment. A failed feature-payload initialization can retain valid control content outside enrollment. Unexpected raw assignment values are logged and do not become misleading control-view events.

Fetching both alternatives in one validated payload makes this eligibility check independent of the assigned arm. A different architecture that first assigns treatment and then fetches only its content has another failure problem: a missing treatment resource can selectively remove treatment visitors from observed exposures. If larger packages require that architecture, preserve assignment and delivery failures explicitly and design the analysis population before launch. Do not infer that a missing view means the visitor never belonged to the comparison.

Preserve diagnostics and investigate the affected interval. Filtering failed treatment deliveries from analysis can select the comparison population after assignment. A safe display fallback does not make a broken experiment trustworthy; repair or halt it when delivery or configuration is unreliable.

The personalized response uses private, no-store. Shared caching of the complete validated pair is a separate choice from caching a visitor's selected HTML. Verify the actual reverse proxy and CDN behavior rather than assuming a response header alone describes every cache in the deployment.

When adding a shared content cache, include the CMS instance, document, locale, publication selection and reviewed fingerprint in the appropriate cache design. A cached English pair should not satisfy a request for a different locale, and a previously approved pair should not conceal a broken publication process indefinitely. Test invalidation and recovery using the same deployment configuration as the experiment.

The server selects one headline before producing HTML, so the browser does not need to rewrite that headline to implement this treatment. Measure the server response path under realistic load nonetheless: content fetching, payload initialization and application work still consume time. Keep loading and error guardrails appropriate to the page rather than promising that server-side evaluation has no performance cost.

Connect browser exposure to the outcome

Record a defined display boundary

The callback during evaluation queues assignments; it is not proof that HTML reached a browser. Use the deferred-tracking pattern to carry compact metadata into a client script that runs after parsing the page:

// public/experiment-view.js
(() => {
  const node = document.getElementById("strapi-exposure");
  if (!node || node.dataset.sent === "1") return;
  let context;
  try { context = JSON.parse(node.textContent); }
  catch { console.error("Invalid Strapi exposure data"); return; }
  if (!Array.isArray(context?.assignments)) return;
  node.dataset.sent = "1";
  for (const assignment of context.assignments) {
    if (!assignment || typeof assignment !== "object" ||
        !["control", "variant"].includes(context.content_variant) ||
        assignment.value !== context.content_variant || typeof assignment.experiment_id !== "string" ||
        typeof assignment.variation_id !== "string") continue;
    const event = { experiment_id: assignment.experiment_id, variation_id: assignment.variation_id,
      content_variant: context.content_variant, source: context.source, event_id: crypto.randomUUID() };
    if (!navigator.sendBeacon("/experiment-view", new Blob([JSON.stringify(event)],
      { type: "application/json" }))) console.warn("Strapi exposure was not queued");
  }
})();

This boundary establishes that browser execution reached the rendered page. It does not prove viewport visibility or reading. Use a symmetric visibility condition if that is the intended exposure definition. A successful beacon queue operation also does not prove network acceptance or durable ingestion.

The data-element marker prevents duplicate execution within one document. Reloads can produce additional displays, and collector retries need their own event-ID policy. Define whether analysis uses first exposure or another consistent rule, then make the assignment query match that definition.

Replace local logging with verified ingestion

The diagnostic collector reads identity from the HttpOnly cookie, validates the event/source fields and discards unexpected fields. It still receives browser-supplied observations. Do not use that metadata as an authorization or billing decision, and do not treat a console row as a warehouse record.

Connect your supported analytics ingestion with appropriate validation, origin controls, delivery monitoring, retries and deduplication. Apply consent and tracker-readiness behavior consistently across both arms. Compare assignments, browser events, accepted requests and warehouse rows when investigating loss; those counts represent different boundaries.

Configure the GrowthBook data source to read experiment key, variation ID, timestamp and assignment unit from the actual stored schema. Preserve the document/locale/hash provenance in the event or a linked experiment record so the content behind a result remains identifiable.

Validate that mapping with one controlled event before increasing traffic. Compare the browser request, accepted collector fields and resulting warehouse row. Field names can change through an analytics transformation, timestamps may be stored in different zones, and a custom identifier can be dropped by a tag or export. Finding the row and joining its outcome is stronger evidence than seeing a successful HTTP response from the local collector.

Join the business outcome to the same identifier

Define completed signup or another chosen outcome with Fact Tables and metrics. Verify that it joins to the browser assignment identifier or a tested authenticated-identity mapping. A signup on another origin does not automatically carry this cookie.

Inspect an actual controlled exposure and outcome together, including timestamps, duplicate handling and follow-up window. Recent visitors may not yet have had enough time to complete the outcome. Keep diagnostic clicks and qualified conversions separate so faster but less useful signals do not replace the intended decision.

Verify publication and maintain a clean test

Check HTTP behavior before interpreting results

With the reviewed hash configured, run the application and establish a local cookie jar:

node --env-file=.env.local server.mjs
curl -i -c strapi-test-cookies.txt http://localhost:3000/experiment-session
curl -s -b strapi-test-cookies.txt http://localhost:3000/offer
curl -s -b strapi-test-cookies.txt http://localhost:3000/offer

Expect a fixed redirect and consistent selected content for the same identifier while configuration remains stable. These requests inspect server HTML; they do not execute the browser exposure script. Use a real browser to inspect the beacon and collector, then verify the actual analytics and warehouse path separately.

Exercise both arms with controlled identities. A forced value can prove a rendering branch, but it does not prove ordinary randomized enrollment. Keep preview, forced and employee QA traffic outside production analysis.

Persistence is also narrower than sticky assignment across arbitrary configuration changes. The same browser ID can be used consistently while a changed experiment key, rule or allocation alters the experience. Freeze the intended configuration during the comparison, record deliberate changes, and use the platform's supported bucketing behavior when your requirements extend beyond a stable identifier.

Test the modern contract and its failure cases

Test conditionExpected behaviorWhat to investigate
v4-style nested responseReject missing flat fieldsMigration compatibility and parser version
Numeric row ID used as document IDNo successful content displayActual documentId and generated endpoint
Draft or wrong-locale payloadReject before assignmentStatus, localization and preview configuration
Reviewed string changesHash mismatch before enrollmentPublication approval and treatment version
SDK, cookie or event failureExplicit baseline/unavailable/diagnostic pathIdentity continuity and delivery loss

Also test HTML-significant characters and a closing-script sequence in approved QA content. They should remain text, and the data element should remain intact. Repeat cache and identity checks on the actual deployment: local success does not establish proxy behavior, public rendering or warehouse attribution.

Include a schema deployment in the regression check when field names or content structure change. A correct new schema in development can coexist with older published documents or a renderer expecting the previous shape. Verify the deployed model, the delivered payload and the rendering contract together. Keep a clear unavailable/diagnostic result for incompatible content rather than disguising it as a valid control observation.

Close the loop with an owned content decision

Before interpreting lift, investigate allocation anomalies, missing data and variation-specific failures. Agree on duration, outcome maturity, meaningful effect and a decision framework before seeing the result. An imprecise estimate can leave the content choice unresolved, and nonsignificance does not prove equivalence.

When the test ends, stop enrollment, verify payload refresh in deployed instances and publish the chosen headline as ordinary content. Preserve the approved pair, source document/locale, fingerprint and metric definition with the learning. Remove retired assignment and tracking branches through review so future tests begin with a clear owner and a clean content contract.

Coordinate the content promotion with the fingerprint gate. Turning the winning headline into the new control changes the reviewed pair, so approve that resulting baseline and its new digest deliberately. Test the publication/configuration sequence in staging; do not clear a mismatch by copying a new hash without checking what it represents. The final baseline should work independently of the retired experiment rule and its exposure code.

Run a clear content comparison

Connect published Strapi content to consistent assignments and the business outcomes your team already measures. Start with a small test and verify the complete delivery and measurement path.

Get Started With GrowthBook
Experiments

Selection bias: What randomization can and can’t prevent

Nov 12, 2026
x
min read

Your randomizer can work correctly while the final experiment comparison becomes unreliable. People can enter different reporting paths, lose their identifiers, or disappear from outcome collection after assignment. The result still looks like an A/B test, but the rows left in the analysis may no longer support the question you meant to answer.

Random assignment is a powerful protection against selection into treatment. It does not guarantee representative recruitment, complete follow-up, or valid post-treatment filters. Understanding that boundary helps you distinguish an allocation problem from a problem observing the people you allocated.

The most consequential question is often what randomization lets you learn when some outcomes remain unknown. A larger sample or a more sophisticated statistical engine cannot supply those missing facts without additional evidence or assumptions.

Understand the protection randomization provides

Assignment should not depend on potential outcomes

Selection bias concerns systematic distortions introduced by how observations enter or remain in the comparison relevant to your question. The selection-bias definition emphasizes the relationship between the studied group and the population of interest. Terminology varies across fields, so name the actual selection process rather than assuming the label explains it.

With a common assignment probability, proper random assignment makes treatment allocation independent of the units’ potential outcomes. When assignment probabilities vary across prespecified strata, that independence is conditional on those strata. Product teams do not have to identify every motivation, prior preference, or unmeasured characteristic before assigning users. The allocation mechanism avoids deliberately routing more promising users into treatment and less promising users into control.

This protection concerns the units eligible for that randomization. It also relies on the implemented experiment matching the design: the assigned treatment must be well defined, identity must support the assignment, outcomes must be measured appropriately, and interference between units must be addressed where relevant.

Balance is an expectation, not a perfect copy

Randomization does not create two identical lists of people. A finite experiment can have chance differences in baseline behavior even when the mechanism is sound. The methodological basis of randomized comparisons is comparability on average under the design, rather than exact equality of every measured or unmeasured characteristic in every sample.

Investigate evidence of implementation failure, but do not repeatedly rerun allocation until the observed outcome looks attractive. Prespecified blocking or stratification can be legitimate design choices; changing allocation in response to favorable results is a different procedure.

Randomization also does not mean the observed difference equals the true effect exactly. Sampling or randomization uncertainty remains even with complete measurement. A confidence interval addresses uncertainty under its assumptions; it does not certify that recruitment, joins, filters, or outcome observation were valid.

There is another limit to what is observed. Each unit receives one assigned experience, so you do not observe that same unit’s outcome under the alternative at the same time. Randomization supports an aggregate comparison under the design; it does not reveal an individual causal effect for every user. Keeping this distinction clear prevents a clean experiment average from becoming an unsupported claim that every customer benefits.

Name the treatment effect you want to estimate. The effect of offering an experience and the effect among people who actually use it are different questions. Random allocation directly supports the first comparison under its assumptions; a user’s decision to adopt is not another random allocation.

Define the population and unit the design covers

Random assignment is different from random sampling

Assignment decides which experience an enrolled unit receives. Sampling concerns which units enter the study from a population. Randomizing a volunteer beta group can support a comparison inside that beta, while leaving a separate question about whether its result applies to customers who did not volunteer.

A narrow study is not automatically a failed study. An Android-only feature may appropriately be tested among eligible Android users. A launch claim about all platforms needs additional support. State the population, setting, and time period in the decision before debating whether the sample is representative enough.

The broader selection-versus-sampling comparison explains these categories and post-treatment subgroup examples. For an experiment review, the immediate task is to identify which population your assignment actually covers and whether the final analysis preserves it.

Choose identity and eligibility before treatment changes them

Eligibility should describe who can meaningfully receive the tested experience, using information that the treatment cannot already have changed. A supported device or an account state measured at entry can define a relevant population. Successfully using the new experience is usually an outcome, not an equivalent entry criterion.

Choose the randomization unit to match the intervention. Assigning individual employees to different collaboration settings may create interference when they work together. Assigning accounts can address some shared-experience requirements, but changes the number of independent units and the analysis needed. Randomization-unit design guidance makes identity stability and network effects part of the experiment design, not merely SDK configuration.

In implementation, preserve the assigned unit and variation through the outcome join. A device identifier that disappears after login can create missing outcomes rather than genuine non-conversions. A changing identity can also expose one person to multiple variations. Record the identity and timing assumptions with the experiment configuration, then verify the delivered data follows them.

Define the denominator first

Use the KPI playbook to distinguish binary, continuous, and ratio metrics, and connect each measurement to the experiment decision it is meant to support.

Read the KPI Playbook

Missing outcomes can defeat an apparently favorable result

Complete-case rates describe the people still observed

Suppose a hypothetical experiment assigns 100 units to each arm. The binary outcome window has ended for everyone, but a collection problem leaves some outcomes unknown. There are 18 known successes in each arm:

Outcome availabilityControlTreatment
Assigned units100100
Units with observed binary outcomes9060
Known successes among observed outcomes1818
Units with unknown outcomes1040
Success rate among observed outcomes20%30%

The complete-case comparison is 18/90 versus 18/60, an observed difference of +10 percentage points. It compares the units whose outcomes were available. It does not establish the success-rate difference for all 100 assigned units in each arm.

The larger missing fraction in treatment is a warning about observation. It is not proof of a particular direction or magnitude of bias. Equal missing fractions would not settle the issue either: equally many missing users could have different outcomes or different reasons for being unobserved.

Missingness does not inevitably create bias in every analysis. Purely random loss can mainly reduce precision; other missingness processes can select people with systematically different outcomes. The missing-value count alone does not identify which process occurred. Record the reasons for loss and what was known about affected units before treatment, rather than treating an arbitrary percentage threshold as a universal certificate of validity.

Intention-to-treat analysis with missing outcomes requires a defensible approach to those unknown outcomes. Keeping everyone’s original assignment preserves the target comparison, but the words “intention to treat” do not recover information that was never observed. Automatically assigning every missing value zero imposes an outcome assumption.

Ask what the unknown values could change

Because the example’s outcome is binary, we can calculate simple logical completion ranges. If none of control’s ten unknown outcomes were successes, its full-cohort sample rate would be 18%. If all ten were successes, it would be 28%.

Treatment has forty unknown outcomes, so its corresponding range is 18% to 58%. Across possible completions of these missing entries, the all-assigned sample rate difference could run from −10 percentage points to +40 points.

These are logical missing-outcome completion bounds, not a confidence interval and not exact bounds on a population causal effect. They hold the observed entries fixed and vary the unknown binary values. Even if every missing outcome were recovered, inference about an underlying effect would retain uncertainty from the experiment’s design.

The exercise reveals how little the complete-case +10-point result establishes without assumptions about observation. Filling all missing values with zero happens to produce 18% in both arms here. That zero difference is one completion, not a demonstrated absence of an effect.

Attrition analysis can use sensitivity checks to make missing-data assumptions visible. A useful next step is to recover outcomes from reliable records, investigate why collection failed, and assess conclusions under plausible alternatives. The most favorable completion is not a substitute for that investigation.

Preserve the comparison after assignment

Different exclusions require different repairs

Distinguish incomplete observation from a known zero. An account that was fully observed and did not purchase can legitimately contribute zero to a purchase metric. An account that no longer joins to the purchase system has an unknown outcome unless other evidence establishes what happened.

Telemetry-loss research shows why lost records can affect power and bias comparisons. Increasing traffic through the same broken collection path need not fix the problem. It may narrow the reported interval around a distorted complete-case comparison.

Inspect the metric and Fact Table definitions, including identifiers, join types, filters, timestamp alignment, and outcome availability. A left join followed by a filter on an outcome-table field can silently become an exclusion rule. A metric name does not reveal which people survived that rule.

Post-treatment filtering is another mechanism. Restricting analysis to purchasers or successful feature users can select different people in each arm when treatment changes those behaviors. Identical SQL applied to both arms does not make selected membership independent of treatment. Use such a view for a clearly limited descriptive purpose or an appropriate causal analysis; do not assume it inherits the original randomized comparison.

Triggering needs a causal and implementation argument

Analyzing a relevant triggered population can be defensible when the trigger identifies comparable eligibility across assignments. The criterion is not simply whether logging happens before or after the assignment timestamp. Ask whether the tested treatment can affect who satisfies the condition or whether that condition is recorded.

GrowthBook’s activation-metric settings explicitly warn that activation should be independent of assignment. Where a feature exists only in treatment, a carefully designed counterfactual logging approach can record who would have encountered it in control. That requires equivalent logic and validation, not just a treatment-only click event with a new name.

Passing a sample ratio mismatch check is useful evidence about allocation counts, not a guarantee that every inclusion rule is valid. Selection can change the composition of two equally sized groups. Review reasons for missingness and baseline characteristics alongside counts, rather than forcing arm sizes to match by discarding extra observations.

No statistical repair is universal. Weighting and imputation require assumptions and relevant observed information. A group absent from the data cannot be studied merely by assigning it a larger weight. A missingness mechanism related to unobserved outcomes cannot be declared harmless because a model returned a precise estimate.

For example, an imputation model may assume observed baseline information is sufficient to account for who has missing outcomes. That assumption needs a substantive argument; the unobserved values themselves cannot be inspected to verify it. Compare plausible departures from the assumption and report whether the practical decision survives. Do not choose the missing-data model because it produces the preferred launch recommendation.

Report the evidence randomization actually preserved

Track assignment separately from outcome observation

Maintain counts for eligible units, assigned units, recorded exposures, observed outcomes, and the final analysis population. Explain losses between stages by arm and by relevant baseline groups. The ledger should distinguish a known outcome from unavailable measurement, and an original exclusion from a later analytic restriction.

The data-quality requirements for online experiments include completeness, consistent assignment records, reliable joins, and timing. An allocation check alone covers only part of that path. A clean A/A test can reveal existing problems but cannot prove a future treatment will leave exposure and outcome collection unchanged.

Preserve the query or metric revision behind the result. If the team repairs a join or recovers delayed records, state whether the correction reconstructs the intended original population. If it does not, explain the resulting scope and whether a new run is needed. Avoid relabeling a selected survivor cohort as the original experiment.

Design outcome observation before increasing exposure. For an important transaction metric, a reliable backend record may allow recovery when a client event fails, provided identity and timing still join correctly. Verify that the recovery method is available in both arms and measures the intended outcome. A second event stream is useful evidence only when its own coverage and meaning are understood.

Make missing-data assumptions part of the launch decision

Alongside the experiment results, report outcome availability and the assumptions used for missing values. Transparent randomized-trial reporting treats participant flow, exclusions, and missing-data handling as information needed to judge a result. The same statistical discipline is useful for product experimentation when the participants are accounts rather than clinical-trial patients.

For the hypothetical experiment, the immediate conclusion is that the favorable complete-case result does not identify the all-assigned sample comparison without more information. The next action is to investigate observation, recover outcomes where possible, and evaluate the remaining uncertainty. It is not to declare a winner from 20% versus 30%, or a tie from zero-filled missing outcomes.

Randomization gives you a defensible starting contrast. Preserving it requires that the data and analysis continue to describe the units and outcomes the decision concerns. Before launching from a randomized result, be able to explain not only how people received a variation, but also whose outcomes are known, whose are missing, and what your conclusion assumes about the difference.

Inspect the metric definition

Open the read-only metric search workflow to find a fact metric and inspect its aggregation, filters, identifiers, and associated fact table. Running the workflow requires configured GrowthBook API access; the instructions do not execute an audit when opened.

View the Metric Search Workflow

Ready to ship faster?

No credit card required. Start with feature flags, experimentation, and product analytics—free.

Simplified white illustration of a right angle ruler or carpenter's square tool.White checkmark symbol with a scattered pixelated effect around its edges on a transparent background.