The team picked the idea with the highest RICE score. After launch, clicks went up, and the report already says "win", although payments are still pending. Then at the next meeting it turns out the new variant has more refunds.
Write down your decision first, then read the analysis. The numbers in the problems are made up. This is a written self-check: what you end up with is a calculation and a plan that someone else can recheck.
Case: two ideas in the queue
The team is scoring two changes to registration. Reach is the unique customers the change will touch in a quarter. Impact is on a shared scale, Confidence is written as a fraction, and Effort is everyone's work in person-months.
| Idea | Reach | Impact | Confidence | Effort |
|---|---|---|---|---|
| A: simplify the registration step | 1,000 | 1 | 0.8 | 2 |
| B: add a completed example | 500 | 2 | 0.5 | 1 |
Calculate RICE as Reach × Impact × Confidence / Effort. Explain what it means when one row beats the other. Then change the conditions: B needs part of the work from A first. How does that change the choice?
Analysis: priority rests on estimates
A comes out at 1,000 × 1 × 0.8 / 2 = 400. B: 500 × 2 × 0.5 / 1 = 500. On the assumptions the team wrote down, B ranks higher. The gap shows expected benefit per unit of work in the chosen model. There is no measured effect behind it: nobody has built the change yet.
Intercom's RICE method combines reach, impact, confidence and effort. To compare ideas, you need a shared horizon and reach unit, clear grounds for the estimates and the full scope of work. Confidence of 0.5 weights how sure you are of the estimates; it is not a measured 50% chance of success.
If B needs part of A done first, that dependency belongs in B's Effort, unless the team already counted it. You compare packages of work that can actually be delivered. You cannot treat B as a cheap standalone task and at the same time pretend the development it depends on is already finished.
A weak estimate can be tested before a big build. For example, find out whether people understand the completed example and whether it helps them through the hard step. That check sharpens an assumption, but the commercial result still has to be measured. This is exactly what hypothesis prioritisation is for: everyone can see the grounds for a choice, and they can be revised.
Effort sits in the denominator of RICE: more work means a lower score. Ease in ICE scores simplicity, and the higher it is, the higher the score, so it works the other way round. That is why the team's table records the chosen formula and scales; otherwise one word, "priority", hides different calculations.
Case: clicks are declared a win
Before launch, the team chose the share of customers with a paid order as the primary metric, with payment errors as the guardrail. Clicks rose in the first days, but the agreed payment window has not closed. Madiyar, the team lead, proposes making clicks the primary metric and ending the test.
What can be reported now? What should the plan have contained before launch? And what should the team do if it finds a mistake in the protocol?
Analysis: keep the decision rule
Right now you have a result for an intermediate action and no answer about completed payments. The original metric and the open observation window stay in the log; clicks are recorded next to them, but no decision is made on them. If you switch the metric after looking at the data, the experiment starts answering a different question.
The decision rule before the result
Before launch
Audience, assignment unit and method
Primary metric and guardrails
Outcome window and completion rule
Current observation
Clicks rose; payments are still pending
Record the intermediate action separately
Result
Retain the original metric and open window
Wait for evidence on completed payments
Switching the metric after seeing the data makes the experiment answer a different question.
The plan records the audience, the assignment unit, the assignment method, the primary metric, guardrails, the outcome window and the completion rule. Sample size depends on the baseline rate and on the change the decision hinges on. The final reading date and what to do about a shortfall are set before the start.
The coffee reorder reminder already has a plan like this. To tell 25% from 33% at 5% significance and 80% power, you need 473 people per group, 946 in total. About 100 newcomers a month agree to receive messages, which is 3.33 people a day: 285 days of recruitment and another 60 days of waiting, with the final reading on day 345. That is the coffee shop's test; it has nothing to do with the registration example in the RICE table.
In it, even IDs get the reminder and odd IDs stay in the holdout. The group is assigned before the flow does anything, and this method only works if ID parity is unrelated to channel, date and branch. If it is related, you need a different random assignment method before launch. The comparison keeps every assigned participant, including those who never read the message.
Interim readings are planned for roughly months five and eight, when the windows of the first three and six months of recruitment close: about 150 and 300 people per group. They do not move the decision date, whatever they show. If a group has fewer than 473 participants with a closed window by day 345, the report records the shortfall and the uncertainty. This plan does not allow stretching the test until a convenient result turns up.
A measurement fault gets its own record: what broke, which data are affected and which new check is needed. The corrected protocol gets its own version. Renaming a column does not turn old data into support for a new hypothesis. You can stop the test on a customer risk defined in advance, but record it as a protective decision, without declaring a winner.
Case: more paid orders, less contribution
In another shop experiment, 1,000 buyers were randomly assigned to A and 1,000 to B. Each person counts for at most one order. Recruitment periods and observation windows are the same, and payments and refunds have run through the agreed period. Before launch, the team decided not to expand a variant if refunds exceed 8% of paid orders.
| Measure | A | B |
|---|---|---|
| Paid orders before refunds | 100 | 110 |
| Fully refunded orders | 5 | 15 |
| Order price | 10,000 ₸ | 10,000 ₸ |
Each order that stays leaves 4,000 ₸ of contribution after variable product and fulfilment costs. On a full refund, the buyer gets the money back, the goods return to stock and their cost is restored. But the shop loses 1,000 ₸ on everything in that order that cannot be recovered, including fulfilment and refund handling. The case has no other non-recoverable costs for a refunded order; that is an assumption of the task. Advertising, development and fixed costs are left out for now.
Calculate the payment rate, the refund rate and each group's contribution after refund costs. Is having more payments enough to roll B out to everyone?
Analysis: the guardrail is part of the result
The payment rate is 100 / 1,000 = 10% for A and 110 / 1,000 = 11% for B. The observed gap is 1 percentage point. How reliable it is has to be assessed with the planned method: percentages alone are not enough to declare a winner.
Refunds: 5 / 100 = 5% and 15 / 110, about 13.6%. B is over the 8% limit set in advance, so under that rule expansion is put off. If the cause is in the tracking, fix the data first. If the refunds are real, look into the product, the promise and the purchase journey, and only then decide how to change the variant.
After refunds, each group keeps 95 orders. A's contribution: 95 × 4,000 - 5 × 1,000 = 375,000 ₸. B's: 95 × 4,000 - 15 × 1,000 = 365,000 ₸. B has more payments but, within the task's boundaries, 10,000 ₸ less contribution. And that is not yet the project's full profit: the excluded costs get added next.
The log will hold both metrics, the financial calculation, the breached guardrail and the decision not to expand the current variant. A "payment win" label would hide the data that change the decision.
Exercise: write your own experiment protocol
Pick one hypothesis from your project. Write down the change, the mechanism through which it should work and the metric, then the audience, assignment, sample, window, guardrails and the final-reading rule. Add the costs and a formula for the commercial result. And describe the outcome in which the data will not be enough for a conclusion.
In a finished protocol, a colleague can see exactly what will change and which observation will settle the decision. If the hypothesis promises purchases but the plan measures only clicks, the link needs work. If the duration depends on when the hoped-for gap appears, you have no completion rule. A HADI cycle ends with a conclusion from the data you got, and "we do not know enough, here is the next check" counts as one.