A/B Test Results
Variants with lift, a confidence interval and a two-proportion z-test — plus the peeking warning, because stopping at the first significant reading is the failure that actually happens.
src/components/ui/ab-test-results.tsxAbout
From the component's own source, where it was written.
Experiment variants with lift, a confidence interval, and an honest verdict.
**It shows the interval, not just the number.** "+12% lift" is not a result; "+12%, 95% CI [−3%, +27%]" is, and it says the opposite — that the experiment has not concluded. A dashboard that prints the point estimate alone manufactures certainty, and people ship on it.
**The maths.** A two-proportion z-test with a pooled standard error for the p-value, and an unpooled standard error for the interval — the standard pair, and the reason the interval can straddle zero while p sits just under 0.05. The interval is then divided by the control rate so it is on the same scale as the lift beside it; an absolute-difference interval printed next to a relative lift is read as bounding that lift, and it does not. The normal approximation needs roughly ten conversions in each arm, so below that the verdict is withheld rather than computed on data too thin to carry it.
**It warns about peeking, because that is the real-world failure.** Checking an experiment repeatedly and stopping at the first significant reading inflates the false-positive rate far above the nominal 5% — a fixed-horizon test read continuously is wrong roughly a third of the time. If a sample-size target is given and has not been reached, this says so plainly.
It reports a *statistical* result about one metric. Whether that metric is the one that matters, whether the assignment was actually random, and whether the effect is worth the change are not things any component can tell you.
Install
npx astralyx-ui add ab-test-results
Usage
import { AbTestResults } from '@/components/ui/ab-test-results'
<AbTestResults variants={variants} targetSample={20000} />Composer
Change the props and the source below follows.
Landing page headline
| Variant | Visitors | Conversion | Lift | 95% CI on lift | p |
|---|---|---|---|---|---|
| Control — current copycontrol | 18,420 | 6.00%(1105) | — | — | — |
| B — shorter headline | 18,390 | 6.75%(1241) | +12.5% | [+4.2%, +20.8%] | 0.003 |
| C — pricing above fold | 18,505 | 6.15%(1138) | +2.5% | [-5.6%, +10.6%] | 0.544 |
Below the planned 20,000 per arm. Stopping at the first significant reading inflates the false-positive rate well past 5% — treat anything here as provisional.
<AbTestResults
variants={variants}
alpha={0.05}
targetSample={20000}
/>Examples
Three variants, one significant
<AbTestResults variants={variants} title="Landing page headline" targetSample={20000} />API
Generated from the source, so it cannot fall behind the code. One table per exported component, then hooks, functions and types.
AbTestResults
Extends Omit<ComponentProps<'div'>, 'title'>.
| Prop | Type | Default | Description |
|---|---|---|---|
| variants | Variant[] | required | { id, name, visitors, conversions, control? }. Exactly one should be the baseline. |
| title | ReactNode | — | — |
| alpha | number | 0.05 | Two-sided significance threshold. |
| targetSample | number | — | Enables the peeking warning. A fixed-horizon test read continuously is wrong far more often than its nominal alpha. |
| metricLabel | string | 'Conversion' | — |
| emptyLabel | string | 'No variants.' | — |
| label | string | 'Experiment results' | — |
| scale | relative, like the lift | — | The z-test gives an interval for the absolute difference in rates. Printed beside a relative lift it gets read as bounding that lift, which it does not — so it is divided by the control rate (the first-order delta-method interval), and the two columns are in the same units. |
Types
Variant
Experiment variants with lift, a confidence interval, and an honest verdict.
| Field | Type | Description |
|---|---|---|
| id | string | — |
| name | ReactNode | — |
| visitors | number | Users in this arm. |
| conversions | number | — |
| control? | boolean | Exactly one variant should be the baseline. |
Behaviour
| Topic | Detail | Description |
|---|---|---|
| the interval | shown, not hidden | "+12% lift" is not a result. "+12%, 95% CI [−3%, +27%]" is, and it says the opposite. Printing the point estimate alone manufactures certainty people ship on. |
| the maths | two-proportion z | Pooled standard error for the p-value, unpooled for the interval — the standard pair, and why the interval can straddle zero while p sits just under 0.05. |
| thin data | no verdict | The normal approximation needs roughly ten conversions and ten non-conversions per arm. Below that it says "too few" rather than computing on data too thin to carry it. |
| what it cannot tell you | stated | Whether the metric is the one that matters, whether assignment was random, and whether the effect is worth the change. |