How X3 decides
Why not "the winner so far"?
Checking a Bayesian "probability to be best" every day and shipping at 95 % finds a "winner" in about one of five experiments where nothing differs at all (simulated: 21–23 % false wins). With one look at the end it is about 5 %. X3 therefore never acts on interim P(best): the numbers are hidden until the decision, and the only automatic actions during the run use always-valid statistics that stay correct however often you look.
The rule (x3-rule-v1)
- Pre-registration at scheduling: hypothesis, one primary metric, guardrails, unit, arms, targeting, runtime, cooldown, fallback, extensions and auto-apply are frozen (SHA-256 hash).
- Planner (last 28 days): baseline at the exposure point, visitors per week, detectable effect (MDE) at 2/4/6/8 weeks, expected conversions per arm, which guardrails are checkable. It sizes for two-sided α = 0.05 and 80 % power – a little stricter than the decision, on purpose.
- During the run (hourly): data hygiene (test traffic, bots, failed verification – a visit whose network changed between the assignment and the first page view is verified by the signed receipt of the assignment and counts normally), sample ratio check (SRM: exact test from 30 units, χ² from 200, alarm at p < 0.001), tracking gaps (only when ≥ 10 exposures per day are expected; otherwise only project-wide outages count), harm stops per variant and guardrail (two-stream mSPRT, α = 0.01, only from 100 units and 10 events per arm) and the overwhelming early win (same test, from day 14 and 50 % of the planned units; it ends enrolment and waits for the cooldown so late guardrails can mature).
- Decision once after the cooldown: Beta(1,1) posteriors, P(best) per arm (closed form for two arms, numerical integration for three or four). A variant is applied when P(best) ≥ 0.96, no guardrail shows harm, the split is healthy, every arm has ≥ 10 conversions and ≥ 50 % of the planned units were reached.
Outcomes
| Outcome | When | Applied |
|---|---|---|
winner | the rule holds for a variant | the variant |
control_best | the control is clearly better (P ≥ 0.96) | control |
equivalent | every arm's expected loss ≤ 1 % with the planned units reached | the fallback / simpler arm |
inconclusive | none of the above – the text names the effect size that would have been visible | the pre-registered fallback |
data_problem | SRM, > 3 outage days or > 5 % failed verification (and at least 5 such units) | control, never automatically |
stopped_harm | every variant was harm-stopped | control |
Fewer than 50 % of the planned units is not a data problem: the outcome is inconclusive
(underpowered) and the fallback applies. The effect shown ("erwarteter Effekt") is shrunk toward
zero (winner's curse at low power); intervals are 95 % credible intervals.
Extensions
- Traffic extension (pre-registered, ≤ 4 weeks): taken automatically when the projection says fewer than half of the planned units – it depends only on counts, never on results.
- Outcome extension (pre-registered): one second look at the end when inconclusive; both looks then need P(best) ≥ 0.975.
- Manual extension while running: once, admins only, with a reason – it raises the threshold to 0.975 and is shown as an addendum to the pre-registration.
Nicht-schlechter-Test (low traffic)
For a change you would make anyway (UX best practice) when superiority cannot be shown in reasonable time: the fallback is the variant, which is kept unless the control is clearly better (P(best_control) ≥ 0.90 with at least 10 conversions in total) or a harm stop fires. Honest strength at about 36 visit-days per arm and 29 % conversion: a variant that halves conversions is still kept in about 4 of 10 runs – the safeguard catches collapses, not moderate harm. The result states "Geschäftsentscheidung mit Absicherung, kein Nachweis einer Verbesserung" and the largest drop that would have gone unnoticed.
Guardrails and proxy metrics
A guardrail is ok, harm or not_checkable – below 10 events per arm, with an interval wider than
the margin (−10 % by default) or, for qualified leads, when fewer than 80 % of the leads were
reviewed. not_checkable is never shown as "kein Schaden"; it names the drop that could go
unnoticed. When the primary is a proxy (click, goal, funnel step, relevance) and a guardrail is not
checkable, the switch waits for a confirmation unless the pre-registration acknowledged it
(proxy_unchecked_ok).
Counting
The primary is counted per exposed unit, from the first exposure until the end of the cooldown,
intention-to-treat (a unit stays in the arm it saw first). Leads follow the same counting as
everywhere in X3: no spam, a repeat of the same person within the journey look-back does not count –
restricted to the identifiers every arm collects (dedupe_on). Units that were already leads before
they saw the experiment are left out. Attribution models never change experiment results.