The page for an analytics course got a shorter form: the optional budget question was removed. Leads went from 50 to 80. You want to keep the new form, the sales manager complains about lead quality, and the owner asks about payments: ten in the old group, eight in the new one. Everyone defends their own number, because before launch nobody wrote down which one the decision would rest on.
It is the same course as in the end-to-end analytics lesson: the price is 25,000 ₸ and contribution per sale is 15,000 ₸. We invented the form test for this lesson, and its figures are made up. HADI (Hypothesis, Action, Data, Insights) makes you connect the question, the change, the data and the decision before launch. Sber's guide uses the same expansion.
Leads and payments in the same groups
| Metric | Form A | Form B |
|---|---|---|
| Visitors | 500 | 500 |
| Leads | 50 | 80 |
| Payments | 10 | 8 |
Two fewer payments are not enough to call B harmful.
H: the hypothesis explains the obstacle
"Remove the field" is a task for the developer. A hypothesis explains why that will help the buyer. For the course, it might read:
People do not know in advance how much they are willing to spend on training, and they abandon the form at the budget question. If we remove it from the first form, more visitors will finish the enquiry, lead quality will not drop, and the share of visitors who pay will grow.
That record has a cause, a change and an expected result. Look for the cause in enquiries first, and watch how people fill in the form. If visitors complain about an error in the phone field but skip the budget question without trouble, the hypothesis has nothing to stand on and you need a different explanation.
A change is ready for testing when it is clear who it is for and what that person will see. Here, one field in the first form changes. Course price, ads, the offer and the manager's work stay the same. Remove the price from the page along with the field, and you can no longer say which of the two changes produced the result.
A: a plan you can launch
Eligible new visitors are split between forms A and B at the same time and at random. The variant sticks to the visitor: come back a couple of days later and you see the same form. Staff and test submissions are excluded under a rule written before launch. Check how a visitor links to an enquiry and a payment on your own test orders, or at the end of the test you will find payments with nothing to attach them to.
The primary metric is the share of assigned visitors who paid for the course within 30 days of entering the test. We chose the 30-day window for the example. On a real project it comes from the data: how long this course usually takes from a first visit to payment. The manager rates lead quality the same way in both groups: do the programme, format and price suit the person.
Lead count shows whether the field affected completion. Qualified leads show whom the change brought in. Complaints, refunds and the manager's time are guardrail metrics: they cap what the improvement may cost. If there are more leads but the sales team spends all day calling people the course does not suit, that is a test result too.
Before launch, the card records group sizes, the minimum useful effect, analysis method, the end of recruitment and the readout date. Get a rough sample size for comparing two proportions from the A/B test planner. The 500 visitors per group in the example below are there to explain the table; for your course the sample size is worked out from scratch.
D: keep one comparison base throughout
Every visitor in the table has had a full 30-day window. Repeat enquiries from one person are merged. Payments are tied to the variant the person got on entering the test, and cancellations and refunds are excluded under one rule for both groups.
| Metric | Form A | Form B |
|---|---|---|
| Assigned visitors | 500 | 500 |
| Visitors with an enquiry | 50 | 80 |
| Visitors with a qualified enquiry | 30 | 32 |
| Visitors with a payment | 10 | 8 |
| Share with an enquiry | 10% | 16% |
| Share with a qualified enquiry | 6% | 6.4% |
| Share with a payment | 2% | 1.6% |
| Revenue at 25,000 ₸ | 250,000 ₸ | 200,000 ₸ |
| Contribution at 15,000 ₸ per sale | 150,000 ₸ | 120,000 ₸ |
There are 30 more leads but only two more qualified ones. The lead rate rose by 6 percentage points, which is plus 60% on the previous 10%. The share of paying visitors in this data fell by 0.4 points. The money rows restate the same payments, with ad spend and the cost of changing the form not yet taken off.
The sales manager will find the qualified share of leads handy: 30 / 50 = 60% for A and 32 / 80 = 40% for B. But the main comparison keeps every assigned visitor in the denominator. The form has already changed who submits an enquiry. Divide only by the people who did, and you compare two groups made up of different people.
Before estimating the effect, the analyst reconciles group assignment, duplicate IDs, payment links and how many days each participant has been in the test. If some group B payments got lost on the way from the CRM, the data is restored first. Statistics run over those gaps measure the difference between reports, not the difference between forms.
I: answer the original question
From this table you can write: "B produced more leads with almost the same number of qualified ones. The observed payment rate is 1.6% against 2% for A. We estimate the effect on payments with the method chosen before launch, including its uncertainty." A difference of two payments is too small to pick a winner. Read a gap of one or two customers in the creative review with the same caution.
Calling B harmful because of two missing payments is just as premature. If the test ended as planned and the estimated range covers both growth and decline, the result is recorded as inconclusive. You can then keep the old form and look more closely at the new enquiries. A new test gets a new plan. Adding visitors to the old test until the result looks convenient is not allowed: keep peeking and stop at a lucky moment, and sooner or later a "winner" turns up even where there is no difference.
The test log keeps the original hypothesis, both forms, the audience, dates, metrics, final table, data problems and the owner's decision. Keeping the old form is a perfectly good decision: the team does not grow a flow of weak leads with no visible benefit. The next cycle tests a new explanation.
A cycle does not have to fit into a week
The length of a cycle is set by the outcome window and the flow of people; the sprint calendar has nothing to do with it. Course payments take 30 days to come in, while a reminder test measured by second orders within 60 days stretches over many months: the calculation is in the retention test. Promise a client "a confirmed hypothesis every week" and you will not get more customers, only a temptation to close tests early.
Exercise: complete the cycle before launch
Take a change to a form, a message or a product step. Write down what gets in the person's way, who the change is for, the single element you are changing, the primary metric with its window, and the guardrail metrics. Add group sizes, the assignment rule, the end of recruitment and the readout date, and the decision owner. Next to that, note what the team will do if the result turns out useful, harmful or inconclusive.
Then review this message from the course team's chat: "Two weeks after the start, form B has 40 leads and form A has 25. Switching everyone to B." Which decision does the written plan allow at this point?
The primary metric is payment within 30 days of entering the test, and after 14 days that window has not closed for any participant. Leads speak only to form completion, and in the final table more leads did not bring more payments. Under the plan, the team keeps recruiting, monitors group assignment, duplicates and the manager's time, and reads the result on the scheduled date. If a guardrail threshold is hit, for example on complaints, the test stops under that rule, and no winner is declared.
Video: HADI in practice
A video from the Marketing WHOyarketing channel. As you watch, check it against your card: does the card name the decision the team is collecting data for?