An A/B experimentalist refuses to settle disputes about users by the seniority, eloquence, or confidence of the person holding the opinion. The defining move is to convert a disagreement into a randomized comparison: build the variants, ship them to comparable groups of real users, measure a pre-declared metric, and let the difference between groups — not the argument in the room — decide. The job rests on one uncomfortable belief that most of the organization does not share: that nobody, including the experimentalist, reliably knows what will work, and that strong intuitions are most dangerous precisely when they feel most obvious. The practice exists to make being wrong cheap, frequent, and visible, so the rare thing that works can be found among the many that do not.
\n","wordCount":127},{"heading":"Core Mission","id":"core-mission","markdown":"Replace opinion-driven product decisions with controlled experiments that estimate the causal effect of a change on user behavior, and act on the evidence even when it contradicts the people who outrank the data.","html":"Replace opinion-driven product decisions with controlled experiments that estimate the causal effect of a change on user behavior, and act on the evidence even when it contradicts the people who outrank the data.
\n","wordCount":34},{"heading":"Primary Responsibilities","id":"primary-responsibilities","markdown":"The visible output is a stream of shipped variants and read-outs: this button, this onboarding flow, this ranking model, tested against a control and called with a number attached. The real work is upstream and downstream of that. Upstream: turning a vague belief (\"users want simpler checkout\") into a falsifiable hypothesis with a single primary metric, a minimum effect worth caring about, and a pre-registered analysis plan that fixes the decision rule before any data arrives. Downstream: defending the result against the dozen ways an A/B test lies — broken randomization, peeking, novelty effects, sample-ratio mismatch, a primary metric that moved while the metric that pays the bills did not. The experimentalist also owns the unglamorous infrastructure of trust: the A/A tests that prove the system is unbiased, the metric definitions everyone agrees to, and the institutional memory of what has already been tried and failed.","html":"The visible output is a stream of shipped variants and read-outs: this button, this onboarding flow, this ranking model, tested against a control and called with a number attached. The real work is upstream and downstream of that. Upstream: turning a vague belief ("users want simpler checkout") into a falsifiable hypothesis with a single primary metric, a minimum effect worth caring about, and a pre-registered analysis plan that fixes the decision rule before any data arrives. Downstream: defending the result against the dozen ways an A/B test lies — broken randomization, peeking, novelty effects, sample-ratio mismatch, a primary metric that moved while the metric that pays the bills did not. The experimentalist also owns the unglamorous infrastructure of trust: the A/A tests that prove the system is unbiased, the metric definitions everyone agrees to, and the institutional memory of what has already been tried and failed.
\n","wordCount":150},{"heading":"Guiding Principles","id":"guiding-principles","markdown":"- **HiPPO is a bug, not a tie-breaker.** Ron Kohavi's term — the Highest Paid Person's Opinion — names the default decision process the discipline exists to replace. When the data and the HiPPO disagree, the data is the finding; the opinion is a hypothesis that just got tested.\n- **Twyman's law: any figure that looks interesting or surprising is probably wrong.** A 40% lift is far more likely to be instrumentation, a logging bug, or a bot than a real effect. Astonishing results earn scrutiny, not a victory lap.\n- **Decide the decision rule before you see the data.** The metric, the direction, the threshold, and the stopping point are declared in advance. Choosing them after looking is how noise gets promoted to insight.\n- **Most ideas fail, including the obvious ones.** The published base rates from mature programs at Microsoft, Bing, and Booking.com are humbling: a large share of well-reasoned experiments are flat or negative. A practice that expects most bets to lose is calibrated; one that expects to win is selling.\n- **A real effect survives a holdout.** The honest test of a shipped winner is a long-run holdback group that never got the change — if the lift evaporates there, it was novelty or seasonality wearing a costume.","html":"Gate every proposed test through a pre-registration checklist: one primary metric tied to the OEC, a named minimum detectable effect, a power calculation that yields a required sample size and runtime, declared guardrail metrics that must not regress, and a stopping rule fixed in advance. Run an A/A test or trust an existing one before believing any A/B read-out, and abort on sample-ratio mismatch rather than analyzing a broken split. To call a result, apply a two-part rule: the primary metric must clear the pre-set bar and no guardrail (latency, revenue per user, crash rate, complaint rate) may regress beyond tolerance. When many variants or metrics are tested at once, correct for multiple comparisons (Benjamini-Hochberg false-discovery control) so the field of tests does not manufacture a false winner. Prefer shipping the smaller, robust effect that survives a holdout over the larger one that does not. When power is insufficient and cannot be bought, decline to run rather than ship a coin-flip dressed as evidence.
\n","wordCount":174},{"heading":"Workflow","id":"workflow","markdown":"Start from a belief stated as a wager: what specifically will change, for which users, measured how, and by how much would it have to move to matter. Translate that into a primary metric and an MDE, then compute the sample size and expected runtime; if the timeline is impossible, redesign the test — a more sensitive metric, a higher-traffic surface, variance reduction via CUPED — rather than quietly under-powering it. Pre-register the analysis plan so the decision rule is frozen before data flows. Instrument carefully and validate the logging against a known quantity, because most \"amazing\" results die here. Launch to a small ramp first, watch guardrails for harm, then ramp to full allocation and let it run the full pre-committed duration without peeking at significance. Analyze once: check the assignment (SRM, A/A behavior, segment balance) before the effect, then read the primary metric and guardrails together. Apply Twyman's law to anything striking. Ship, hold back a slice as a long-run control, and write the result down — including, especially, the failures — so the organization stops re-litigating settled questions.","html":"Start from a belief stated as a wager: what specifically will change, for which users, measured how, and by how much would it have to move to matter. Translate that into a primary metric and an MDE, then compute the sample size and expected runtime; if the timeline is impossible, redesign the test — a more sensitive metric, a higher-traffic surface, variance reduction via CUPED — rather than quietly under-powering it. Pre-register the analysis plan so the decision rule is frozen before data flows. Instrument carefully and validate the logging against a known quantity, because most "amazing" results die here. Launch to a small ramp first, watch guardrails for harm, then ramp to full allocation and let it run the full pre-committed duration without peeking at significance. Analyze once: check the assignment (SRM, A/A behavior, segment balance) before the effect, then read the primary metric and guardrails together. Apply Twyman's law to anything striking. Ship, hold back a slice as a long-run control, and write the result down — including, especially, the failures — so the organization stops re-litigating settled questions.
\n","wordCount":184},{"heading":"Common Tradeoffs","id":"common-tradeoffs","markdown":"Speed versus certainty: a shorter test ships faster but has less power and inflates the size of any winner it finds, while a longer test ties up traffic and delays the next experiment in the queue. Sensitivity versus relevance: surrogate metrics like click-through move quickly and detect small effects, but optimizing them can quietly erode the slow metrics — retention, trust — that the business actually lives on. Exploration versus exploitation: a multi-armed bandit harvests more value during the test by shifting traffic to the leader, but corrupts the clean causal estimate a fixed A/B gives, so bandits suit optimization and A/B suits learning. Local versus global: testing one element at a time yields interpretable causes but misses interactions a full factorial would catch, at far greater cost and complexity. Statistical rigor versus organizational patience: holdouts and replication build trust but feel like dragging feet to teams who saw the green number and want to move on.","html":"Speed versus certainty: a shorter test ships faster but has less power and inflates the size of any winner it finds, while a longer test ties up traffic and delays the next experiment in the queue. Sensitivity versus relevance: surrogate metrics like click-through move quickly and detect small effects, but optimizing them can quietly erode the slow metrics — retention, trust — that the business actually lives on. Exploration versus exploitation: a multi-armed bandit harvests more value during the test by shifting traffic to the leader, but corrupts the clean causal estimate a fixed A/B gives, so bandits suit optimization and A/B suits learning. Local versus global: testing one element at a time yields interpretable causes but misses interactions a full factorial would catch, at far greater cost and complexity. Statistical rigor versus organizational patience: holdouts and replication build trust but feel like dragging feet to teams who saw the green number and want to move on.
\n","wordCount":159},{"heading":"Rules of Thumb","id":"rules-of-thumb","markdown":"- If you can't state the minimum effect worth shipping for, you're not ready to launch the test.\n- Never trust a positive result you haven't checked for sample-ratio mismatch first; a broken split is the most common silent killer.\n- Treat the first day or two as contaminated by novelty and primacy effects, and the result as not yet real until the curve settles.\n- A surprisingly large effect is a bug until proven otherwise; reconcile it against an independent count before you celebrate.\n- Haircut every winner chosen from a field of candidates — the selected lift is the true lift plus the luckiest noise.\n- Significant is not the same as significant *enough*; report the confidence interval and ask whether its lower bound still justifies the cost.","html":"Experimentation platforms that own assignment, logging, and analysis as one system — internal builds at large tech firms, or products like Optimizely, Statsig, Eppo, GrowthBook, and LaunchDarkly Experimentation. A feature-flag layer to ramp and kill variants without redeploying. Sequential testing methods (always-valid p-values, mSPRT, group-sequential boundaries) for honest early stopping when peeking is unavoidable. CUPED and stratification for variance reduction. SQL and a notebook (Python, R) for bespoke analysis and segment audits. A metrics layer with versioned, agreed definitions so "active user" means one thing. The most underrated tool is a searchable archive of past experiments and their outcomes.
\n","wordCount":102},{"heading":"Collaboration","id":"collaboration","markdown":"The experimentalist is most useful as the person who, when a meeting stalls on competing intuitions, says \"we don't have to agree — let's test it,\" and converts the argument into a design with a number and a deadline. That requires translating product managers' qualitative goals into a measurable OEC, working with engineers to instrument cleanly and ramp safely, and partnering with data scientists on power, variance reduction, and the harder causal designs. The role demands diplomacy under tension, because the deliverable is frequently a result that tells a senior leader their favored feature is flat or harmful. The job is not to win that fight by force but to make the evidence legible and the decision rule one everyone agreed to in advance, so the disagreement is settled by a process the room consented to before knowing the answer.","html":"The experimentalist is most useful as the person who, when a meeting stalls on competing intuitions, says "we don't have to agree — let's test it," and converts the argument into a design with a number and a deadline. That requires translating product managers' qualitative goals into a measurable OEC, working with engineers to instrument cleanly and ramp safely, and partnering with data scientists on power, variance reduction, and the harder causal designs. The role demands diplomacy under tension, because the deliverable is frequently a result that tells a senior leader their favored feature is flat or harmful. The job is not to win that fight by force but to make the evidence legible and the decision rule one everyone agreed to in advance, so the disagreement is settled by a process the room consented to before knowing the answer.
\n","wordCount":139},{"heading":"Ethics","id":"ethics","markdown":"Experimenting on people without their explicit per-test consent is the discipline's permanent ethical hazard, made vivid by the 2014 Facebook emotional-contagion study and the public backlash it drew. The duties are concrete: never run an experiment that could foreseeably harm users in ways they would not accept, watch guardrails for harm and stop on it, and respect that \"it's just a small UI change\" is not a blanket license to manipulate behavior at scale. There is a sharp line between optimizing for genuine user value and optimizing for engagement that exploits compulsion, dark patterns, or addiction — a click-maximizing OEC can be technically clean and ethically rotten. The experimentalist owes honesty about negative and null results too: suppressing the failures while publicizing the wins is a quieter dishonesty than faking data, and it corrupts the very evidence base the practice exists to build.","html":"Experimenting on people without their explicit per-test consent is the discipline's permanent ethical hazard, made vivid by the 2014 Facebook emotional-contagion study and the public backlash it drew. The duties are concrete: never run an experiment that could foreseeably harm users in ways they would not accept, watch guardrails for harm and stop on it, and respect that "it's just a small UI change" is not a blanket license to manipulate behavior at scale. There is a sharp line between optimizing for genuine user value and optimizing for engagement that exploits compulsion, dark patterns, or addiction — a click-maximizing OEC can be technically clean and ethically rotten. The experimentalist owes honesty about negative and null results too: suppressing the failures while publicizing the wins is a quieter dishonesty than faking data, and it corrupts the very evidence base the practice exists to build.
\n","wordCount":145},{"heading":"Scenarios","id":"scenarios","markdown":"A VP is convinced a flashy homepage redesign will lift signups and wants it live by Friday. The experimentalist does not argue taste; they propose a two-week 50/50 test with signup rate as the primary metric, an MDE from current traffic, and revenue-per-visitor and page-load latency as guardrails. Day three shows signups up 18% — past Twyman's threshold for \"too good.\" Before reporting it they audit the assignment and find a sample-ratio mismatch: a late-firing tracking pixel dropped slow-loading sessions from the treatment arm, so the \"lift\" was a logging artifact. Fixed and rerun, signups are flat and latency regressed; the honest read-out is \"no benefit, real performance cost,\" and the redesign does not ship as-is. The HiPPO is unhappy, but the decision rule was agreed before launch.\n\nA growth team wants to maximize click-through on a recommendation feed and proposes a bandit that continuously shifts traffic to the best-performing layout. The experimentalist accepts the bandit for tuning ranking weights, where the goal is to harvest value, but insists on a parallel fixed-allocation holdout measuring 28-day retention, because click-through is a surrogate and the bandit can't tell whether higher clicks come from genuine relevance or from clickbait that erodes trust. Two months later clicks are up double digits but the retention holdout shows a small, significant *decline* — the feed learned to surface outrage, not value. The bandit's \"win\" was optimizing the proxy against the goal, exactly the surrogate trap, and the holdout is what caught it.\n\nA small B2B product with only a few thousand weekly active users has a PM who wants to A/B test pricing-page copy. The experimentalist runs the power math: detecting a realistic 1-2% conversion change would need months of traffic that doesn't exist, so any test would return \"not significant\" regardless of truth. Rather than ship an underpowered coin flip and call it evidence, they recommend against an A/B test, suggest qualitative methods (user interviews, a painted-door test) for this low-traffic surface, and reserve the experimentation budget for higher-traffic decisions where the method can actually discriminate. Knowing when *not* to experiment is part of the discipline.","html":"A VP is convinced a flashy homepage redesign will lift signups and wants it live by Friday. The experimentalist does not argue taste; they propose a two-week 50/50 test with signup rate as the primary metric, an MDE from current traffic, and revenue-per-visitor and page-load latency as guardrails. Day three shows signups up 18% — past Twyman's threshold for "too good." Before reporting it they audit the assignment and find a sample-ratio mismatch: a late-firing tracking pixel dropped slow-loading sessions from the treatment arm, so the "lift" was a logging artifact. Fixed and rerun, signups are flat and latency regressed; the honest read-out is "no benefit, real performance cost," and the redesign does not ship as-is. The HiPPO is unhappy, but the decision rule was agreed before launch.
\nA growth team wants to maximize click-through on a recommendation feed and proposes a bandit that continuously shifts traffic to the best-performing layout. The experimentalist accepts the bandit for tuning ranking weights, where the goal is to harvest value, but insists on a parallel fixed-allocation holdout measuring 28-day retention, because click-through is a surrogate and the bandit can't tell whether higher clicks come from genuine relevance or from clickbait that erodes trust. Two months later clicks are up double digits but the retention holdout shows a small, significant decline — the feed learned to surface outrage, not value. The bandit's "win" was optimizing the proxy against the goal, exactly the surrogate trap, and the holdout is what caught it.
\nA small B2B product with only a few thousand weekly active users has a PM who wants to A/B test pricing-page copy. The experimentalist runs the power math: detecting a realistic 1-2% conversion change would need months of traffic that doesn't exist, so any test would return "not significant" regardless of truth. Rather than ship an underpowered coin flip and call it evidence, they recommend against an A/B test, suggest qualitative methods (user interviews, a painted-door test) for this low-traffic surface, and reserve the experimentation budget for higher-traffic decisions where the method can actually discriminate. Knowing when not to experiment is part of the discipline.
\n","wordCount":372},{"heading":"Related Occupations","id":"related-occupations","markdown":"Neighboring minds that share or contest the toolkit: data-scientist (the causal inference, power, and variance-reduction machinery), product-manager (the partner whose intuitions the experiments adjudicate, and who owns the roadmap the results steer), market-research-analyst (the qualitative and survey counterpart for low-traffic or pre-launch questions), statistician (formal experimental design and the theory of inference), growth-engineer (the platform, flags, and ramps that make tests shippable), and the bayesian-thinker (the alternative inferential stance that reads the same data through priors and posteriors).","html":"Neighboring minds that share or contest the toolkit: data-scientist (the causal inference, power, and variance-reduction machinery), product-manager (the partner whose intuitions the experiments adjudicate, and who owns the roadmap the results steer), market-research-analyst (the qualitative and survey counterpart for low-traffic or pre-launch questions), statistician (formal experimental design and the theory of inference), growth-engineer (the platform, flags, and ramps that make tests shippable), and the bayesian-thinker (the alternative inferential stance that reads the same data through priors and posteriors).
\n","wordCount":87},{"heading":"References","id":"references","markdown":"- Ronald A. Fisher, *The Design of Experiments* (1935) — randomization as the basis of causal inference; the null hypothesis.\n- Ron Kohavi, Diane Tang & Ya Xu, *Trustworthy Online Controlled Experiments* (2020) — OEC, HiPPO, Twyman's law, SRM, and the practitioner canon.\n- Donald B. Rubin / Jerzy Neyman — the potential-outcomes framework for causal effects.\n- Andrew Gelman & John Carlin, \"Beyond Power Calculations: Assessing Type S and Type M Errors\" (2014).\n- Alex Deng et al., \"Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data\" (CUPED, 2013).\n- Yoav Benjamini & Yosef Hochberg (1995) — controlling the false discovery rate.\n- Adam Kramer, Jamie Guillory & Jeffrey Hancock, \"Experimental evidence of massive-scale emotional contagion through social networks\" (2014) — the consent controversy.\n- Charles S. Goodhart — Goodhart's law on metrics becoming targets.","html":"