Dauren is an analyst working with a fitness club in Almaty: leads come in from Instagram and 2GIS, and managers call them back. Dauren is rebuilding a purchase forecast as of 1 September. The extract already contains a payment from 10 September, and the model has picked it up as a feature. On historical data it finds buyers almost perfectly. But on 1 September that payment did not exist, so the model will never repeat this result in production.
The dates, amounts and results in this lesson are illustrative. A marketing director on a project like this has two questions to answer. Does the model select people well when it only gets what was known at decision time? And does the action that follows the forecast actually help? They are different questions: a model can be excellent at finding people who would buy anyway, and then a discount for them just cuts revenue from purchases that would have happened without it.
Data available at forecast time
A purchase forecast for 1 September
Known on 1 September
Past visits and purchases
Check that the data was in the system on 1 September
Future payment
10 September
Do not use as a feature for the 1 September forecast
A historical extract containing future payment leaks data. The production system did not know it then.
Define the event and decision time
"Find good customers" is not a target, because you cannot check it. You can check something else: payment within 30 days of the lead. Then name the prediction time and the action that follows: a call queue, an offer of help, a stock plan. Without an action, a high score stays a number in a report.
Keep three things for every record: the reference point, what was known on that date, and the outcome after it. Yesterday's lead has not had its 30 days yet. Labelling it "did not buy" just because there is no payment today is a mistake: the person may pay next week. Assess quality on records whose observation window has closed.
Decide in advance what counts as the outcome. Payment, revenue net of refunds and contribution after variable costs are three different numbers. If Dauren trained the model to find leads, it is too early to call it a profit forecast. First connect those leads to the outcome the business needs, and test that connection on data.
The selection threshold also depends on how much people can handle. The club's managers can call 100 leads, so compare the quality of those 100 under the same working procedure. A rule that hands managers noticeably more records may win on call volume alone, and you will mistake effort for selection quality.
Rebuild what was known before the prediction
The future sneaks into a model through unremarkable fields. Current CRM status, total purchases and the deal's closing reason all contain events that had not happened at decision time. Putting an old date on today's customer row does not give you a historical snapshot: the data in the row is still today's.
For every field, find two dates: when the event happened and when the record appeared in the system. A manager may enter yesterday's call only this evening. The call happened, but a forecast made in the morning would not have seen it. If you keep a history of when each piece of data arrived, you can reproduce the conditions the model will work in.
Google's crash course on monitoring ML systems covers both outcome leakage into features and the gaps between training and serving. In practice Dauren's job is simpler to state: assemble exactly the set of fields the system will have before the call.
For evaluation, hold back data the model never saw in training and that resembles its future work. With a time-based split, training records must have their outcomes in before the boundary, and feature preparation stays out of the evaluation portion. Think about who the model will score, too: new people or customers you already know. An evaluation done on a different group answers a different question.
Calculate precision and recall for the selected group
The evaluation sample has 1,000 leads, and 50 of them paid. A model that always answers "will not buy" is right 950 times. Accuracy, the share of correct answers, comes to 950 / 1,000 = 95%, and not a single buyer makes it into the call queue.
Dauren's model selected 100 leads, and 20 of them bought. Now you can see both who was found and who was missed:
| Question | Calculation | Result |
|---|---|---|
| What share of those selected bought? | 20 / 100 | Precision = 20% |
| What share of all buyers did we find? | 20 / 50 | Recall = 40% |
| How many selected leads did not buy? | 100 - 20 | 80 |
| How many buyers remained outside the priority list? | 50 - 20 | 30 |
Classification metrics are read together. A selected group can be very precise and still leave most buyers out. Pick the threshold knowing what each error costs, how many calls can be made and what the selection is for.
A score of 80 is not an 80% probability. To read it as a probability you need a calibration check: whether the stated probabilities match how often the outcome actually happens on suitable data. Without one, the score can rank records, provided that ranking has been tested. Do not put a percentage on it.
Compare the model with a simple rule. For example, the queue could be ordered by a field the managers already see in the CRM. Assess both on comparable data with the same workload. A complex model is worth maintaining only if the process needs its result and it pays for its upkeep.
Price the action separately from selection
Suppose handling one priority lead costs the club 500 ₸ of a manager's time at an internal rate. It is a number for the calculation and says nothing about market rates. 100 leads cost 100 × 500 = 50,000 ₸. Take the contribution of an additional purchase, after variable costs but before contact and model costs, as 5,000 ₸.
Covering that time takes 50,000 / 5,000 = 10 additional purchases. The 50,000 ₸ values managers' hours; there are no new payouts behind it. Building and maintaining the model are not included, and the full decision adds them. And the 20 buyers the model found are not 20 additional purchases: some of them would have bought without a call.
A purchase propensity forecast says how likely a purchase is under current conditions. The effect of a call or a discount is how much behaviour changed because of the action. To measure it, split a suitable audience at random into people who get a call and people who do not, or find another sound causal comparison. Contribution after contact and offer costs is calculated on its own.
Even with good selection the action can be unnecessary. If the discount goes to people ready to buy a membership at full price, the club gives away part of its contribution and gets no extra purchases. So the report shows two measures side by side: forecast quality and the result of a tested action. The first does not replace the second.
Decide in advance what happens on errors and data changes
A forecast needs an owner, a shelf life and a clear next step for the manager. A low score is the model's estimate and says nothing about a person's worth. Agree on what happens to a lead that misses the queue: what help the person still gets, such as a WhatsApp reply or a later call.
Keep the model version, the inputs, the forecast and what the manager did. A field goes missing, a source changes, an update is late: react straight away, before wrong decisions pile up. Keep a fallback for that: a manual queue or the earlier validated rule.
After launch, compare quality on new leads whose window has closed and watch the managers' workload. A new product or acquisition channel can change the mix of leads. A good historical score is a starting point, and you check it against current work regularly.
Exercise: explain 30 buyers out of 100
Another sample has 1,000 leads and 60 buyers. The model selected 100 leads, and 30 of them bought. Calculate precision and recall and write down what is missing to judge the value of a call. Then do the same for your own project: record the prediction time, the target event and a field through which the future could accidentally get into the model.
Precision here is 30 / 100 = 30% and recall is 30 / 60 = 50%. Another 30 buyers never made it into the queue. These numbers describe selection quality only. To credit purchases to the manager, you need to know how similar people would have behaved without a call, and what the calls cost.
If your project has leakage, the first fix is to collect the features that were available in time and assess the forecast again. If selection is good but the value of a call is unknown, the next step is to test the action itself. The rollout decision rests on both parts: the selection and the effect.