Self-check: cohorts, money and useful product outcomes

Azamat, who owns an online shop in Almaty, opens a report: revenue is up, the share of repeat buyers is down, and the newest cohort shows zero. "Revenue is growing," he says, "so let's raise the budget and rebuild the website urgently while we are at it." As the analyst, you need to work out what has actually changed and which decision the numbers can support.

This is a written self-check: keep your calculations and explanations, then compare them with the analysis. The numbers are made up; repeat purchases are counted within 30 days.

Case: three cohorts from one shop

In May, June and July, 100 people made their first purchase each month. Each person belongs to one cohort only, by the month of their first payment. Retention here is the share of buyers with at least one further paid, non-refunded order within 30 days of their first payment. A buyer with several repeat orders adds one person to the numerator.

At the reporting date, everyone from May and June has had the full window. In May, 30 people bought again; in June, 20. Some July buyers have not had their 30 days yet, but the analyst still entered 0% for July.

Which values can go into the report for completed cohorts? How should July be shown? And separately: can you say the gap between May and June appeared because the product got worse?

Analysis: the same age and the same action

May: 30 / 100 = 30%. June: 20 / 100 = 20%. The gap is 10 percentage points. July needs an incomplete-window label, the cut-off date and the date when the last buyer's window will close. Zero would mean that all 30 days had passed and nobody came back, and July's 30 days are not over yet.

While the cohort matures, the analysis does not stop. You keep watching orders and errors as usual, and for an early comparison you take the same short horizon for every group. Give it a different name from the final 30-day metric and keep the two apart: one column should never hold results for buyers of different ages.

If you take only the July buyers whose window has already closed, you get part of the cohort. You can show it as a separate row with its own denominator, but you cannot label it as the result for all 100 people. Those who bought early in July may have come from other campaigns or bought other products.

The gap between completed cohorts is an observation. To explain it, you need the buyer mix, channels, categories, stock, delivery terms and the history of changes in tracking. Cohort segmentation helps you see what changed: who the buyers are, or how they behave within groups. To blame a new product version, two neighbouring months are not enough; you need a comparison that separates its effect from everything else.

The verb in the definition matters too. We count a paid repeat purchase within the window. Opening an app, returning exactly on the last day and buying after the window answer different questions.

Case: revenue rises while fewer buyers return

Now add the economics of the same two completed cohorts. Each buyer's first purchase brought 10,000 ₸ of revenue, and the goods and delivery cost 6,000 ₸ in variable costs. Everyone who came back placed exactly one further order within the window.

Every order in the calculation is paid, with no refunds. Advertising and fixed costs are left out of the table. Work out each cohort's revenue from first and repeat purchases, then contribution after variable costs. What will you tell Azamat, for whom revenue growth is reason enough to raise the budget?

Analysis: separate revenue from contribution

First purchases give each cohort 100 × 10,000 = 1,000,000 ₸ of revenue and the same contribution: 100 × (10,000 - 6,000) = 400,000 ₸.

May's repeat orders add 30 × 10,000 = 300,000 ₸ of revenue and 30 × 4,000 = 120,000 ₸ of contribution. June's bring 20 × 20,000 = 400,000 ₸ of revenue, but only 20 × 4,000 = 80,000 ₸ of contribution: variable costs grew along with order value.

Revenue is 100,000 ₸ higher; contribution is 40,000 ₸ lower. Fewer people came back, but they spent more, and turnover went up. Contribution did not benefit. Why the order value rose is still to be found out: the products and order terms may have changed.

These are cohort totals over one horizon, not the shop's whole revenue for a calendar month. It is also too early to call them profit: advertising, fixed costs and other items are missing. To decide how much to spend on acquisition, you need the cost of a new buyer, later repeat purchases and confidence that the shop can handle the extra orders. Raising the budget just because revenue grew is as unfounded as switching campaigns off because of one overall chart.

Your answer to Azamat can be short: "Among completed cohorts, 30-day repeat purchasing fell from 30% to 20%. Revenue rose thanks to more expensive repeat orders, but contribution after variable costs dropped by 40,000 ₸. We will break the change down by product and channel and add acquisition costs. We will assess July once its window closes."

Case: sign-ups are called customer value

Another product helps people book services. The team proposes "accounts created" as its North Star Metric. Yet anyone can register without choosing a service or a time.

Suggest a measure closer to the task the person actually gets done, say where its confirmation will come from, and which quality guardrails to put next to it. What should you do if the service knows a booking was created but never gets a reliable "visit took place" status?

Analysis: a metric needs a boundary of observation

The candidate depends on what job the product does. If the service helps people book, count people with a successfully created booking, excluding tests, duplicates and cancellations by a rule chosen in advance. If the job includes the service itself, a confirmed visit is closer to it. But you can count that only where the status is really collected.

When visit data is missing, the metric keeps its plain name: "created bookings". Next to it, show what share of bookings has a confirmation, and no-shows wherever they are recorded. If you treat a missing status as a successful visit, the result will look better than it is.

The main product metric still has neighbours: booking errors, cancellations, time to complete the task, complaints and retention. The number of actions is easy to inflate without any benefit to people, so guardrail metrics are kept alongside it.

Exercise: prepare one line of your own report

Pick a metric from your own project and describe it: unit, event, numerator, denominator, window, source, refresh date and owner. Attach a calculation on the data you have, without personal details. Mark incomplete windows, name the decision the metric supports, and the data that decision still lacks.

A colleague should be able to recalculate the finished line and read it the same way you do. If "return" actually hides an app login while the decision is about repeat payment, the definition needs fixing. If a fresh cohort shows zero, its window needs a label. In a dashboard built for decisions, a deviation comes with a next step: who checks the data, in which breakdown, and when they report back.