Experiments (A/B tests)
An experiment shows two to four versions ("arms") of one part of your website to randomly chosen visitors and measures which one reaches your goal more often. X3 experiments are built for honest answers on small and large websites:
- Every experiment has a runtime. You choose it up front (whole weeks, 2–12). At the end X3 decides once, with a pre-registered rule, and switches every visitor to the better arm within about two minutes – no deploy needed. If the data cannot decide, the pre-registered fallback applies and the result says so in plain words.
- No peeking traps. P(best) and "winner" are hidden until the decision. During the run only two things can happen automatically: a variant that clearly hurts is stopped (harm stop), and an overwhelming early win ends the run early – both with always-valid statistics. Details: How X3 decides.
- Privacy by default. Cookieless: the same visitor sees the same arm for one day without any device storage. Longer stickiness only with consent. See Units and privacy.
- Claude-first. "Teste auf meiner Website, ob …" – Claude plans the sample size honestly, creates the experiment, writes both code paths with the SDK, verifies them, starts after your deploy, watches the data quality and cleans up the losing code afterwards. See Experiments with Claude.
Concepts
| Term | Meaning |
|---|---|
| Flag key | Stable id used in code: x3.variant("quiz-contact-split"). Unique per project, never reused. Public. |
| Arm | One version. control is always the first arm – today's site. Arm keys and optional params (button text, video id) are public. |
| Area | What is changed (quiz.contact, hero.cta). Experiments of one area are mutually exclusive. |
| Exposure | The moment a visitor actually sees a difference – experiment_view, sent by variant() or expose(). Only exposed visitors are analysed. |
| Unit | What is randomised and counted: visit-day (cookieless), visitor (consented id) or page load. |
| Primary metric | Exactly one decision metric: lead, qualified lead, conversion, goal, funnel step, event or relevance – in shop projects also purchase or revenue per unit. |
| Guardrails | Up to three metrics that must not get worse (harm stops, no rollout of a harmful arm). A proxy primary (click, funnel step, goal) needs lead as a guardrail. |
| Runtime and cooldown | Whole weeks; the cooldown lets late conversions (e.g. qualified leads) arrive. |
| Pre-registration | Hypothesis, arms, metrics, unit, targeting, runtime and rule are frozen at scheduling; the SHA-256 hash is shown. |
Lifecycle
| Status | What visitors get |
|---|---|
Draft (draft) | control; signed QA links preview every arm |
Scheduled (scheduled) | control, no exposure – but code calls are recorded: "Code live erkannt" |
Running (running) | their arm; exposures are counted |
Paused (paused) | control, nothing new counted (earlier visitors keep counting) |
Cooldown (cooldown) | still their arm; visitors new after the end are not analysed |
Decided (decided) | control until someone rolls out (auto-apply off, price tests, data problems) |
Rolled out (rolled_out) | everyone the applied arm, no exposures; "Zurück auf Kontrolle" undoes it |
Cleaned up (cleaned_up) | the code no longer calls the key; X3 answers the applied arm for 90 more days |
| Stopped / archived | control / read-only |
A start needs two proofs from the live site: the website calls the flag (deployed code) and every arm was seen in a QA visit during the last 7 days. Use the signed QA links of the experiment (test traffic, never counted).
What "switch to the better version" means
At the decision X3 stores the outcome – winner, control_best, equivalent, inconclusive,
data_problem or stopped_harm – and the arm to apply. With auto-apply (the default) the public
client config now answers that arm for every visitor; code still contains both paths until you
clean up. A data problem never switches automatically, price and offer
tests always wait for a click, and every rollout can be undone.