Experimentation & Causal Decision Making

The situation

A commercial pricing change offered meaningful upside, but it also introduced conversion and customer-risk downside. Comparing two group averages would have been easy; deciding whether the change was credible, safe and worth rolling out was the harder problem.

The business needed to know whether the experiment was behaving correctly, whether guardrails were deteriorating, whether there was enough evidence to act, and what risk it would accept by making the decision. Measurement had to support that judgement rather than substitute for it.

Design backwards from the rollout decision

The experiment began with a deterministic assignment rule and a pre-defined primary commercial metric. Sample-ratio mismatch checks verified that allocation behaved as intended before anyone interpreted the result. Conversion and customer-risk measures were treated as guardrails, not footnotes added after seeing an attractive headline number.

Confidence intervals made the range of plausible commercial effects visible. A minimum worthwhile effect created a decision threshold, while futility and stopping reasoning prevented the team from waiting indefinitely for certainty the experiment could not provide. This kept statistical discipline connected to the cost of rolling out, holding, or gathering more evidence.

1. Validate
2. Estimate
3. Check risk
4. Decide

Put uncertainty next to the decision

The decision view brought the estimated effect, commercial threshold and guardrails together. That made it possible to discuss the evidence in plain terms: the most likely outcome, the downside still consistent with the data, and whether that residual risk was acceptable.

The recommendation

Rollout should follow only when assignment checks pass, the credible range of commercial impact clears the decision threshold, and guardrails remain within the business’s stated tolerance. If the likely upside is too small, stop for futility; if downside risk remains material, hold or gather more evidence rather than hiding uncertainty behind a binary “significant” label.

This gave stakeholders a decision they could defend: not simply that one variant won, but why the expected upside justified the remaining risk and how performance would be monitored after rollout.