Mobile Experimentation Platform
Explore the design of mobile experimentation platforms that address unique challenges like delayed app updates and version fragmentation. Understand end-to-end architectures, client-side evaluation, governance systems to avoid experiment conflicts, and durable telemetry pipelines that enable safe, large-scale mobile feature testing without impacting user experience.
Mobile experimentation at scale introduces constraints that don’t exist in server-driven systems. A single misconfigured feature flag can impact millions of users, yet fixing it is not always immediate due to app store release cycles, version fragmentation, and limited control over deployed binaries. What appears to be a simple configuration issue quickly becomes an operational challenge.
This is the core tension of mobile experimentation. Unlike web systems, where experiments can be adjusted in real time, mobile environments require architectures that work around delayed updates, inconsistent client versions, and unreliable connectivity.
A mobile experimentation platform addresses these constraints by decoupling experiment control from app releases, enabling on-device evaluation, and ensuring reliable data collection. This lesson explores how such systems are designed, covering configuration delivery, client-side assignment, governance mechanisms, and resilient telemetry pipelines.
End-to-End Architecture
A robust mobile experimentation platform operates as a distributed, closed-loop system across the control plane, distribution edge, and client nodes. This architecture ensures that experiments are evaluated safely and consistently without degrading the user experience or requiring a binary release for every change.
The architecture is divided into three primary functional domains that facilitate continuous feedback:
The control plane (preparation): In the experiment management console, product owners define the experiment parameters. The config compilation service then transforms these high-level rules into a serialized manifest.
The distribution layer (delivery): The manifest is distributed via a CDN edge layer. By using edge caching and TTL-based invalidation, the platform can reach millions of devices globally with sub-second latency, shielding origin servers from traffic spikes during app launches.
The client and analytics plane (evaluation and feedback): The on-device evaluation engine inside the mobile app parses the manifest to perform deterministic variant assignment. As the user interacts with the feature, events are queued in a durable telemetry pipeline, which are eventually processed by the statistical analysis engine to complete the cycle.
Establishing this end-to-end flow requires a highly available distribution layer to deliver experiment rules to a fragmented device population without incurring performance costs.
Config delivery and CDN-backed distribution
The delivery pipeline begins in the management console, where experiments, targeting predicates, and traffic allocations are defined. A compilation service aggregates active experiments into a single versioned JSON or protocol buffer manifest. This manifest is signed and pushed to a CDN origin.
Two invalidation strategies keep devices current:
TTL-based expiry: Handles routine refreshes during app foregrounding or background cycles.
Push-based purge: Involves invalidating CDN edge caches via silent push notifications (e.g., FCM/APNs) for emergency kill-switch scenarios.
The client fetches the manifest during cold starts, foreground resumes, or upon receiving urgent update notifications. If the CDN is unreachable, the client falls back to a locally persisted disk copy or a minimal seed manifest baked into the binary at build time. This ensures the app remains functional even on a first-time install without connectivity.
This CDN-backed approach means config delivery scales with CDN capacity, not backend compute. Millions of devices can fetch updated configs within seconds of a publish event, without a single request hitting your origin servers during normal operation.
With config delivery in place, the next question is how the device decides which variant a user sees.
On-device assignment and layered evaluation
On-device evaluation eliminates the latency and reliability risks of server-side round-trips. When a manifest is loaded, the engine evaluates experiments locally.
The client computes a bucket using the following formula:
This produces a value between 0 and 9999. This bucket maps to a variant based on traffic ranges defined in the manifest. Because the hash is deterministic, the user receives a consistent assignment across sessions without server interaction.
Before bucketing, the engine evaluates
Experiments are organized into layers, where each layer represents a mutually exclusive traffic partition. Within a layer, a user can belong to only one experiment. Across layers, the same user can participate in multiple experiments simultaneously because each layer uses an independent hash salt, producing uncorrelated bucket assignments.
The evaluation engine also unifies feature flags and remote config under a single pass. An experiment variant can toggle a feature flag or override a remote config value, so the device resolves all three systems in one evaluation cycle.
The following flowchart demonstrates this evaluation logic.
Note: The .hashValue in production should use a stable hashing algorithm like MurmurHash3 or SHA-256 truncated to an integer, because Swift’s built-in hashValue is randomized across process launches starting in Swift 4.2.
With assignment mechanics covered, the next challenge is preventing experiments from colliding with each other.
Governance and mutual exclusion layers
Running hundreds of concurrent experiments creates a real risk of interaction effects, where two experiments modify the same checkout button or the same network retry logic simultaneously, making it impossible to attribute metric changes to either one.
The governance system addresses this through several mechanisms:
Mutual exclusion layers: Each layer owns a slice of total traffic, and experiments within a layer share that slice exclusively. A user bucketed into experiment A within layer 1 cannot simultaneously be bucketed into experiment B in the same layer.
Namespace registry: The management console maintains a registry of code paths and UI surfaces. When a PM creates an experiment targeting the checkout flow, the registry enforces that no other active experiment in the same layer touches that namespace.
Release-health guardrails: Automated monitors track crash rates,
ANR rate Application Not Responding is a metric measuring how often an Android app's main thread is blocked for more than 5 seconds, causing the system to display a "not responding" dialog. Traffic segmentation validation: Pre-experiment checks verify that each user segment (new users, power users, specific locales) meets minimum sample size requirements to maintain statistical significance.
Attention: Mutual exclusion layers only prevent conflicts within the experimentation platform. If a team ships a hardcoded change in a binary release that touches the same UI surface as an active experiment, the layer system cannot detect or prevent the collision. Coordinate with your release process.
Governance ensures experiments run safely. The next step is capturing the data that tells you whether they worked.
Durable telemetry and analytics integration
The telemetry pipeline closes the experimentation loop by connecting on-device user behavior back to the statistical analysis engine.
On-device event collection: Every user action is tagged with the active experiment assignments, including experimentid, variantid, and assignment_timestamp. Events are batched in a local SQLite queue and uploaded opportunistically when the device has connectivity. This ensures durability even in offline-first scenarios where a user might complete an entire session without a network connection.
Server-side ingestion and analysis: On the backend, events land in a partitioned Kafka topic, are enriched with server-side dimensions such as subscription tier and user cohort, and are written to a columnar data warehouse like BigQuery or Snowflake. The
statistical analysis engine A backend service that computes per-variant metrics, applies hypothesis testing methods such as sequential testing or fixed-horizon tests, and determines whether observed differences between variants are statistically significant.
A critical nuance often overlooked involves outdated mobile clients. Devices running old app versions that do not support newer experiments can pollute the control group if not properly filtered. The telemetry pipeline must tag every event with the manifest version so the analysis engine can exclude stale clients.
With telemetry flowing reliably, the remaining question is how the system handles scale and failure gracefully.
Scalability and fault tolerance trade-offs
CDN-backed config delivery is eventually consistent. A window of seconds to minutes exists where some devices have the new manifest and others do not. For most experiments, this is acceptable, but synchronized launches require a different approach. The manifest can embed a server-side activation timestamp, and the on-device engine delays evaluation until the local clock passes that timestamp, coordinating rollout across devices.
From a CAP theorem perspective, the on-device evaluation engine favors availability and partition tolerance. It works offline by reading from the cached manifest, sacrificing consistency because a user might briefly see a stale assignment until the next fetch succeeds.
As experiments grow to hundreds, manifest size can bloat and slow evaluation. Delta updates, where the client fetches only the diff from its current manifest version, and per-platform manifest splitting reduce payload size and parsing time.
Practical tip: Monitor your manifest size as a release metric. Set an alert if it exceeds a threshold (for example, 500 KB compressed) and investigate whether expired experiments are being properly cleaned up.
Test Your Knowledge!
A mobile experimentation platform uses deterministic hashing with the formula hash(user_id + experiment_salt) % 10000 for bucketing. Why does each experiment layer use a different salt value?
To reduce the computational cost of hashing on low-end devices
To make the hash output unpredictable so that users cannot guess their variant
To allow the server to override the client-side assignment when needed
To ensure that a user's bucket assignment in one layer is independent of their assignment in another layer, preventing correlated assignments across experiments
Now let’s trace the full life cycle of a single experiment from creation to conclusion.
Putting it all together
A PM creates an experiment in the management console, defining variants, targeting rules, and a mutual exclusion layer. The config compiler produces a new manifest version. The CDN distributes it globally within seconds. Millions of devices fetch the manifest, and the on-device evaluation engine assigns each eligible user to a variant using deterministic hashing within their assigned layer. As users interact with the app, telemetry events tagged with experiment assignments flow through the durable pipeline into the data warehouse. The statistical analysis engine computes per-variant metrics and declares a winner. Throughout this entire life cycle, release-health guardrails monitor crash rates and stand ready to kill a bad variant in minutes, not days.
The platform’s resilience comes from treating the mobile device as an autonomous evaluation node that operates independently of the backend, while the backend provides the intelligence layer for analysis and governance. This architecture enables safe experimentation at massive scale across fragmented mobile ecosystems.