What is dark launching? How to test features in production safely

A dark launch lets production exercise new code before customers depend on its output.

Dark launching means deploying a feature or changed backend behavior to production while keeping it hidden from the general user population. The code runs in the real environment, often on real requests, but its output is discarded, logged for comparison, or shown only to an internal cohort. The public experience continues using the existing path.

The purpose is to answer technical questions that staging cannot answer reliably. Can the new service handle the production traffic mix? Does a recommendation model respond within the latency budget? Are permissions, dependencies, and observability configured correctly? What does the feature cost at real volume?

A dark launch reduces uncertainty before public exposure. It does not eliminate risk. Hidden code can still consume capacity, read sensitive data, call downstream services, produce writes, or slow the visible request path. A safe dark launch has an explicit boundary, controlled side effects, version-specific telemetry, and a fast way to stop.

What dark launching actually means

Martin Fowler's definition of dark launching focuses on calling a new or changed backend behavior from existing users without those users being able to tell. That framing distinguishes a dark launch from merely deploying disabled code.

Consider a new product-recommendation service:

• The current service still decides what customers see.
• Production requests also invoke the candidate recommendation service.
• The candidate response is recorded for latency, error, cost, and quality analysis.
• Customers do not see the candidate recommendations.
• The team can stop the hidden call without redeploying.

The new code is receiving realistic inputs and exercising production dependencies, yet it is not authoritative. That last condition is critical. If the candidate response determines prices, writes irreversible state, or changes the user interface, the release is no longer fully dark for the affected users.

Teams sometimes use "dark launch" more broadly for any unannounced or limited release. The exact label matters less than documenting three facts:

1. Who or what invokes the new path?
2. Can its output affect users or shared state?
3. Which control disables it?

Those answers tell operators what the launch can validate and what harm it can cause.

Common dark-launch patterns

Dark launching is an outcome, not a single implementation. The right pattern depends on the system and the question being tested.

Deploy behind an off-by-default feature flag

The simplest pattern is to ship the new code behind a release flag that resolves to the existing behavior by default. The production artifact contains both paths, but ordinary users cannot enter the new one.

This validates packaging, startup, dependency loading, and compatibility with the production runtime. It does not exercise the new path until someone targets an internal identity, a synthetic request, or a controlled percentage.

GrowthBook's guide to feature flags in CI/CD describes how flags separate code deployment from feature release. The key operational benefit is reversibility: changing flag configuration can stop the path faster than rebuilding and redeploying the application.

Target employees or test accounts

Teams can enable a hidden feature for named employees, QA accounts, development tenants, or requests with a protected attribute. This is often called dogfooding or an internal release.

Internal targeting tests the complete experience, including the UI, and can reveal integration and usability problems. It remains a dark launch relative to customers, but it is not invisible to the internal cohort. Test accounts need realistic permissions and data shapes; otherwise a successful internal test may say little about production customers.

Never rely on a query parameter or client-side value that an external user can guess. Evaluate sensitive targeting rules on a trusted server and log why the request qualified.

Mirror or shadow production traffic

Traffic mirroring copies real requests to a candidate service while the current service remains authoritative. Istio describes mirrored requests as out-of-band, fire-and-forget traffic whose responses are discarded. Google Cloud load balancing similarly supports mirroring a percentage of requests to test a backend.

Shadow traffic can expose rare payloads, production traffic distributions, performance bottlenecks, and dependency behavior that synthetic tests miss. It also multiplies work. A 100% mirror can nearly double processing for that request path and may double paid API calls if the candidate is not isolated.

Sanitize or exclude data that the candidate environment is not authorized to process. Mirroring also requires special handling for requests that create side effects; replaying a payment, email, mutation, or queue message can produce real harm even when its response is discarded.

Compute a hidden result in the live request path

Another pattern invokes both implementations synchronously, returns the old result, and compares the new result in the background. This is useful for search ranking, fraud rules, pricing calculations, recommendation systems, and AI outputs.

The candidate call must not extend visible latency beyond the service budget. Use strict timeouts, concurrency limits, sampling, and circuit breakers. If the hidden computation shares the same CPU, database pool, or rate-limited dependency, it can degrade the current experience without ever rendering a new UI.

Control every stage of a dark launch

Use GrowthBook feature flags to keep new behavior off by default, target internal cohorts, and expand exposure without another deployment.

Explore Feature Flags

What a dark launch can and cannot prove

A dark launch is strong evidence for some questions and weak evidence for others.

The last distinction prevents a common mistake. A technically healthy recommendation engine may produce worse recommendations. An AI assistant may stay within its latency budget yet reduce task completion. A dark launch can verify readiness to expose the feature; it usually cannot establish that the exposure is valuable.

After technical checks pass, use a controlled experiment when the product decision depends on causal impact. GrowthBook's experimentation platform can use the same feature flag as the assignment layer, connecting exposure to goal and guardrail metrics.

A safe dark-launch process

Treat a dark launch as a production change with a narrower output boundary, not as a test that is exempt from release controls.

1. Write the learning question

Choose the uncertainty that justifies production execution. Examples include:

• Whether p95 latency remains below 150 milliseconds at expected volume.
• Whether the new parser handles the full production event distribution.
• Whether a model stays within a cost budget per eligible request.
• Whether candidate and current decisions disagree in important segments.
• Whether a new dependency remains reliable during peak traffic.

A specific question determines the traffic sample, duration, instrumentation, and exit criteria. "See how it goes" leads to indefinite hidden systems with no decision.

2. Map every side effect

Trace reads, writes, network calls, messages, cache mutations, logs, and billing consequences. Classify each action:

• Safe and read-only.
• Safe only for synthetic or internal identities.
• Idempotent with a dedicated key.
• Redirected to an isolated sink.
• Prohibited during the dark phase.

Do not assume an ignored response means an ignored effect. A mirrored POST can still charge a card. A candidate worker can still acknowledge a queue message. An AI request can still send data to a third-party model provider. Use read-only modes, stubbed integrations, isolated topics, and explicit authorization boundaries.

3. Create a kill switch and safe default

Wrap the entry point in a flag or routing rule whose default preserves the existing path. Define:

• The owner and on-call contact.
• Who can change production configuration.
• The default when flag data is unavailable.
• How quickly configuration reaches every instance.
• Which dashboard and alert identify the dark path.
• The command or action that stops it.

Test the off switch under load. A flag that stops new calls may not cancel queued jobs or in-flight requests, so document how residual work drains.

4. Instrument the candidate separately

Attach a version, flag, or route attribute to logs, traces, metrics, and sampled outputs. Monitor:

• Invocation count and eligibility.
• Errors, timeouts, retries, and fallbacks.
• Latency percentiles and queue delay.
• CPU, memory, concurrency, and connection pools.
• Dependency rate limits and failure rates.
• Cost per invocation and total incremental cost.
• Output validity, disagreement, or quality scores.
• Writes or side effects attempted and blocked.

OpenTelemetry's signal model gives teams a common way to connect metrics, logs, and traces. For a dark launch, the practical requirement is attribution: an incident responder must be able to separate candidate work from the current production path.

5. Expand exposure in stages

Start with synthetic requests or internal accounts, then move to a small representative sample. Increase only after the relevant observation window passes.

A possible sequence is:

1. Production deployment with no candidate invocation.
2. Protected synthetic probes.
3. Named internal users or test tenants.
4. One percent of eligible read-only requests.
5. Ten percent, then a larger shadow sample.
6. Technical readiness decision.

Sampling must be stable when comparisons depend on a user or account. For stateful workflows, keep the whole workflow on the same path rather than mirroring isolated steps that lack context.

6. Compare, decide, and remove the dark path

Define the decision before collecting data: proceed to an internal experience, start a canary, run an experiment, revise the implementation, or stop.

If the feature proceeds, preserve the release flag while the public rollout expands. If it does not, remove candidate traffic and clean up unused infrastructure. Do not leave dark code running indefinitely; it continues consuming capacity and expands the system's operational and security surface.

Dark launch, canary, blue-green, and experimentation

These techniques can form a progression:

• Dark launch: production exercises the code, but customers do not depend on its output.
• Canary release: a small real cohort receives the new behavior.
• Progressive rollout: exposure expands through controlled stages.
• Experiment: eligible users are assigned to variants so the team can estimate outcome impact.
• Full release: the chosen behavior becomes the default and temporary controls are removed.

A blue-green deployment operates at the environment layer and can host any of these exposure policies. The green environment might first receive shadow traffic, then a small canary, and finally all traffic. Feature flags operate inside the application, so they can target a particular behavior even after the infrastructure cutover is complete.

Each stage answers a different question. Dark launch asks whether the system can run. Canary asks whether limited real exposure is operationally safe. Experimentation asks whether the behavior causes the desired outcome. A full launch is the decision to make that behavior normal.

Dark-launch risks and practical safeguards

The "dark" label can create false confidence. Review these risks explicitly:

• Accidental exposure: Protect server-side rules, verify default values, and add automated tests for unauthorized identities.
• Duplicate side effects: Block mutations, use idempotency keys, and isolate external integrations.
• Capacity contention: Set concurrency and rate limits so candidate work cannot starve the current path.
• Privacy expansion: Confirm that the candidate and its vendors are authorized to process the mirrored fields.
• Misleading comparisons: Account for timeouts, missing context, and nonrepresentative internal accounts.
• Unbounded cost: Track incremental compute, storage, model tokens, and third-party calls.
• Flag debt: Assign an owner and removal date when the flag is created.

Feature toggles add code paths that must coexist. Fowler's feature-toggle guidance recommends treating release toggles as transitional. GrowthBook's feature flag best practices similarly make ownership, observability, and cleanup part of the release rather than post-launch housekeeping.

Move from hidden execution to deliberate evidence

A successful dark launch is not one that stays invisible forever. It is one that reduces a defined technical uncertainty without exposing users to an unproven output or allowing the candidate to damage shared systems.

Start with a precise learning question. Map side effects, create a tested off switch, instrument the candidate separately, and expand the sample only when its safeguards hold. Then choose the next stage deliberately: internal use, canary, experiment, full rollout, or removal.

GrowthBook can provide the flag and targeting layer across that progression. To keep a new path off by default, release it to controlled cohorts, and connect later exposure to experiment results, get started with GrowthBook.