Feature flags in microservices: Patterns and best practices

Feature flags can reduce the blast radius of a microservice release, but only when services remain correct without perfectly synchronized flag state.

Microservices let teams deploy and scale components independently. A feature that spans several services complicates that independence: the API may support a new contract before the frontend uses it, a worker may process both data formats during migration, and a recommendation service may run two algorithms for an experiment.

Feature flags provide runtime control over those transitions. They can enable a new path for employees, route a small percentage of accounts, mirror traffic to a candidate service, or stop a failing integration. They can also create distributed configuration dependencies, inconsistent decisions, and a graph of flags no team understands.

The right design treats flags as bounded controls around independently compatible services—not as a distributed transaction that makes all services change behavior at the same instant.

Why microservices change feature flag design

In a monolith, one process may evaluate a flag and choose a complete code path. In a distributed workflow:

• Several services can evaluate related controls.
• Instances receive configuration updates at different times.
• A request may outlive a flag change.
• Async messages may be processed minutes later.
• Old and new service versions run together.
• Retries can reach different replicas.
• User, account, and service identities may not align.
• Partial failure is normal.

The design goal is not global instantaneous consistency. It is predictable behavior under configuration lag, network failure, mixed versions, and retries.

A Reddit discussion about flags across frontend, backend, and microservices illustrates the ownership question: distinct capabilities often deserve distinct controls, and backend support can be enabled before a frontend exposes it. Another discussion warns that a requirement for every service to stay perfectly synchronized may indicate a distributed monolith.

Pattern 1: Evaluate at the owning service

The service that owns the business decision should usually evaluate the flag.

If the checkout service chooses between payment orchestration paths, it evaluates that decision using account or request context. Downstream payment adapters receive an explicit command; they do not independently decide whether the account belongs to the new checkout.

Benefits:

• One authoritative decision.
• Fewer inconsistent evaluations.
• Clear ownership and tests.
• Smaller targeting-data surface.
• Easier exposure logging.

Keep evaluation close to the decision without burying it deep inside unrelated logic. Wrap the SDK with a domain-facing interface such as checkoutPolicy.useNewOrchestration(account).

Pattern 2: Propagate the decision

When several services must behave consistently for one operation, evaluate once and propagate an immutable decision or version in the request, message, or workflow state.

Useful fields include:

• Experiment and variation identifier.
• Feature behavior version.
• Assignment unit.
• Trace or correlation identifier.
• Assignment timestamp.

Do not propagate an unrestricted map of every flag. That couples services to a global configuration and can expose sensitive rules. Pass the minimum business-relevant decision.

For asynchronous workflows, persist the decision with the job. Re-evaluating hours later can send one transaction through two variants and contaminate an experiment.

Pattern 3: Use independent service flags for independent changes

A customer-facing capability may require several deployment steps:

1. Add backward-compatible API fields.
2. Deploy a new worker that can process both formats.
3. Begin dual writing.
4. Backfill data.
5. Enable treatment reads.
6. Expose the frontend.
7. Remove old fields.

Use separate flags when steps have different owners, risks, or rollback conditions. Name them according to the technical behavior, not only the initiative.

Avoid exposing users until backend compatibility is confirmed. A frontend flag can be the last release control, while backend flags manage migration and capacity.

Dependencies should be shallow and documented. If every service has a prerequisite graph that mirrors the entire project plan, flags are substituting for architecture.

Pattern 4: Local evaluation with distributed configuration

GrowthBook feature flags are evaluated locally in application SDKs after a feature-definition payload is loaded. This removes a network call from each decision and lets services use cached configuration during a temporary control-plane outage.

Local evaluation fits high-throughput microservices, but configuration distribution needs engineering:

• Polling or streaming interval.
• CDN, proxy, or relay topology.
• Cache persistence and maximum age.
• Payload size.
• Startup readiness.
• Version reporting.
• Regional propagation.

Services will briefly hold different versions. Design changes to remain valid during that window. Record configuration version in traces so incidents can identify what each service evaluated.

Pattern 5: Remote evaluation for sensitive central policy

Remote evaluation sends context to an evaluator and receives a variation. It keeps rules and segments centralized and can prevent client or service payloads from containing sensitive policy.

Use it when rule confidentiality or centralized enforcement outweighs the runtime dependency. Engineer:

• Strict timeouts.
• Bounded retries.
• Circuit breakers.
• Local caches.
• Regional capacity.
• Fallback policy.
• Attribute minimization.

Do not put a remote call on a hot path without measuring tail latency and failure amplification. A flag service outage should not become a fleet-wide retry storm.

Pattern 6: Sidecar or relay proxy

A local or regional proxy can fetch configuration once and serve several application instances. It may centralize caching, reduce outbound connections, or provide a language-neutral interface.

The proxy is production infrastructure. Define readiness, horizontal scaling, version compatibility, certificate rotation, telemetry, and disaster recovery. If every service depends on one proxy instance, the design has created a single point of failure.

The sidecar and gateway patterns in the Azure Architecture Center provide broader context for placing shared concerns alongside or in front of services.

Pattern 7: Shadow traffic and dark computation

Send a copy of requests or data to a candidate service without using its response for the user. Compare output, latency, errors, and resource consumption before active routing.

Shadowing is useful for service replacements, search, recommendations, fraud systems, and model changes. It still creates cost and may duplicate side effects. The shadow path must not charge a card, send an email, mutate customer state, or emit misleading business events.

Use sanitized or authorized data and apply the same privacy controls as production. A flag can control shadow percentage and provide a kill switch.

Pattern 8: Experiment assignment at a stable boundary

For a product experiment, assign at the highest stable unit that matches the experience—often account or user. Propagate the variation through participating services and log one meaningful exposure.

Do not let each microservice hash the same user independently unless configuration, hashing, namespace, and attributes are guaranteed identical. Even then, one authoritative assignment is easier to audit.

GrowthBook feature flag experiments connect SDK assignment with warehouse-native analysis. Outcome data can include conversion, retention, revenue, latency, errors, or infrastructure cost from several services.

An operational canary asks whether the system remains healthy. A controlled experiment asks whether the treatment causes a better product outcome. Use both when necessary.

Consistency without distributed transactions

Design backward-compatible APIs

Add fields before requiring them. Accept old and new request formats during the transition. Ignore unknown fields. Version semantics when compatibility cannot be preserved.

Flags should select compatible behavior, not make incompatible deployments appear atomic. AWS guidance on feature flags for independently released micro-frontends similarly warns against blocking independent releases and describes shared flag decisions during rendering.

Make retries idempotent

A request assigned to a treatment may be retried after configuration changes. Persist the behavior version with the idempotency key or workflow state. Ensure repeated processing does not duplicate side effects.

Separate rollout from data migration

Use expand-and-contract:

• Expand schemas and deploy tolerant readers.
• Begin safe dual writes.
• Backfill and validate.
• Switch reads under a flag.
• Monitor.
• Stop old writes.
• Contract schemas after rollback is no longer required.

A flag can reverse reads quickly, but it cannot undo incompatible data writes automatically.

Expect configuration skew

Set a maximum tolerated propagation delay and test it. If a workflow cannot survive two services seeing different configuration versions for that period, pass the decision explicitly or redesign the contract.

Ownership and governance

Assign flags to domains

Organize projects, environments, tags, and permissions around service or product ownership. A central platform team provides infrastructure and policy; domain teams own behavior and cleanup.

Every flag should include owner, type, fallback, risk, review date, and code references. Cross-service flags need one accountable release owner plus participating service owners.

Restrict production changes

Separate read, development edit, production edit, approval, and administration permissions. High-risk routing, payment, authorization, and data-migration flags may require a second approver.

GrowthBook approval workflows and audit features support governed configuration changes. Self-hosting does not remove the need for identity, role, and audit design.

Keep flags out of secret stores

Flags are dynamic, visible operational controls; secrets require confidentiality and different access patterns. A recent practitioner question about moving flags out of a secret vault captures the problem: storing nonsecret operational configuration with credentials can make ordinary release visibility and change control difficult.

Never put actual secrets in a flag payload.

Observability

Instrument both the flag system and feature behavior.

Flag infrastructure

Monitor:

• Configuration age by service and region.
• Poll or stream failures.
• Cache load and persistence.
• SDK initialization errors.
• Evaluation errors and fallback count.
• Payload size and download latency.
• Version distribution across instances.

Feature behavior

Tag traces or structured logs with a privacy-safe variation and configuration version. The OpenTelemetry feature flag semantic conventions provide standard event and attribute concepts.

Avoid high-cardinality attributes and personal data. Sample routine evaluations, but preserve enough data to debug an incident or reconcile experiment exposure.

Guardrails

Monitor error rate, latency, saturation, queue depth, timeouts, retries, cost, and business outcomes. GrowthBook Safe Rollouts can connect staged exposure to guardrail metrics.

Define thresholds before ramping and state who can stop or advance the release.

Testing strategy

Unit tests

Test service behavior for every supported value and fallback. Keep evaluation wrappers small so domain logic can be tested without a live control plane.

Contract tests

Test old and new service versions, payloads, events, and schemas. A flag transition often exposes compatibility failures that unit tests miss.

Integration tests

Run workflows with combinations that can exist during deployment and rollback. Do not attempt every combination of all global flags; scope tests to documented dependencies.

Failure tests

Start without configuration, interrupt updates, return malformed values, let caches age, and roll back while requests are active. Test the kill switch on a production-like environment.

Experiment QA

Verify assignment distribution, identity, exposure timing, metric joins, and sample ratio mismatch. One service should not emit exposure while another silently uses control behavior.

Common anti-patterns

Every service evaluates the same user flag

Minor context or configuration differences produce inconsistent behavior. Evaluate once at the owner and propagate the decision.

A flag coordinates an incompatible big-bang release

If all services must switch simultaneously, the design has not achieved independent deployability. Add compatibility layers and sequence the rollout.

Global configuration is passed everywhere

Services become coupled to flags they do not own, and sensitive rules spread across the system. Pass domain decisions, not the entire flag universe.

Permanent branches remain undocumented

Release and experiment flags should be removed. Operational and permission flags can be long-lived, but need owners, tests, and review.

A rollback cannot reverse side effects

Turning off a flag does not unsend emails, undo writes, or restore schemas. Design compensating actions and migration rollback separately.

Capacity planning and incident response

A flag platform becomes part of the delivery system even when evaluation occurs locally. Plan its capacity and failure behavior with the same care as service discovery, identity, and telemetry.

Model configuration fan-out

The control plane usually serves changes far less often than applications evaluate flags, but a large fleet can create synchronized bursts. Thousands of instances may start after a regional deployment, reconnect after an outage, or poll at the same interval.

Measure:

• Connected SDK clients and configuration requests per region.
• Payload size as flags, segments, and rules grow.
• Cache hit rate at relays, proxies, or CDNs.
• Reconnection behavior and retry jitter.
• Time for a critical kill switch to reach 50%, 95%, and 100% of instances.
• Memory and CPU cost of evaluation in representative services.

Add randomized polling, exponential backoff, regional caches, and bounded payloads where appropriate. Scaling the evaluator without scaling the database or configuration origin simply moves the bottleneck.

Define service-class defaults

One global fallback policy is rarely correct. Document behavior by flag class:

Defaults are application decisions, not SDK decoration. Review them with the service owner and test them without network access or cached configuration.

Build an incident playbook

An actionable playbook should answer:

1. How do responders determine whether the control plane, distribution layer, SDK, or a particular flag is failing?
2. Which services and regions have stale or missing configuration?
3. Can the risky feature be disabled from the control plane, or is an application deployment required?
4. Which queues, workflows, or in-flight requests continue after the flag changes?
5. How is configuration frozen or restored to a known version?
6. Who can approve an emergency change, and where is the audit record?

Practice at least three scenarios: the flag platform is unavailable, a bad rule is distributed successfully, and a service interprets a valid value incorrectly. These require different responses. Failover helps the first; configuration rollback helps the second; an application rollback or compensating change may be required for the third.

Control concurrent rollouts

Microservice teams can independently change application versions, infrastructure, and flags. Without coordination, several treatments may affect the same request path and make regressions difficult to attribute.

Attach release and configuration versions to traces, maintain a view of active high-risk rollouts, and pause unrelated changes during incidents. This does not require a central committee for every flag. It requires enough shared visibility for an on-call engineer to reconstruct the state of a distributed request.

Budget for flag lifecycle work

Every temporary flag adds alternate paths to testing, observability, and incident reasoning. Include cleanup in the original delivery estimate and track age, owner, last evaluation, and code references. Remove the losing branch only after all relevant service versions have stopped referencing it.

For cross-service initiatives, sequence cleanup just like rollout: remove downstream dependencies, stop old events or fields, verify consumers, and then delete upstream controls. Deleting the control-plane record first can strand old instances on unintended defaults.

An adoption plan

1. Select one service-owned feature.
2. Define flag owner, fallback, and cleanup.
3. Install a GrowthBook server SDK.
4. Create separate development and production environments.
5. Wrap evaluation in a domain interface.
6. Test disabled, enabled, missing-config, and stale-cache states.
7. Deploy treatment code disabled.
8. Expose employees and verify traces.
9. Ramp gradually with operational guardrails.
10. Run a controlled experiment if product impact matters.
11. Record the decision and remove temporary branches.

After the first workflow succeeds, standardize context schema, SDK versions, telemetry, approval policy, and templates. Avoid rolling out a platform to every service before the team knows which practices it wants to standardize.

Preserve service autonomy

Feature flags help microservices release independently when the decision has a clear owner, services tolerate mixed versions, and configuration skew is expected. They hurt when teams use them to simulate an atomic switch across incompatible services.

Design for one authoritative decision, propagate it when consistency matters, use separate controls for independent migration steps, monitor configuration versions, and clean up temporary branches. GrowthBook provides open-source feature flags, local SDK evaluation, progressive delivery, and experiments in the same system. Start for free with one service and prove failure behavior before scaling reliably across additional critical production services.