The owner of an online coffee shop in Almaty is watching a demo of a new analytics platform. The screen shows revenue, a funnel and a button for starting a test. His question is simpler: where in this report is paid order O42 for 15,000 ₸, and which page variant did the buyer see? The platform's sales rep suggests paying for the subscription now and discussing the integration later.
Start the choice with that order. You have already met the shop and O42 when sending a paid purchase from the server. The comparison figures below are illustrative. You will walk the purchase path on test records and work out what it costs to maintain two different setups.
Write the requirement before meeting the vendor
The shop wants to compare two checkout variants. Each eligible visitor gets a variant before buying and keeps it under the experiment's rules. As the analyst, you need results for everyone who was assigned a variant, including people who bought nothing. The payment list gives you the conversion numerator, but you cannot rebuild the denominator from it.
The requirement fits in one sentence: "For each participant and experiment, recover the assigned variant, the exposure, subsequent orders and payments within the chosen window." Add who owns the data and how to export it. With that sentence you come to the demo with your own task, and the sales rep has to show it solved instead of touring every feature.
Linking records in a test
Checkout
Variant B
- Payment record
- O42, 15,000 ₸, KZT
Result row
- Order
O42
- Variant
B
- Payment
Counted
The participant needs a separate ID. Group comparison needs all assigned participants, including non-buyers.
Follow O42 from assignment to the report
An Order ID is the order number. The participant needs an ID of their own: a person can place several orders, and an assigned participant may place none. If you keep only O42 and a "B" label, you cannot compare the groups, because the people who did not buy are simply missing from the data.
In the test environment, the team walks this chain:
| Step | What must remain in the data |
|---|---|
| Participant receives a variant | Participant ID, experiment ID, variant and assignment time |
| Page variant is shown | Exposure record under the rule chosen in advance |
| Order O42 is created | Link to the participant and the right experiment |
| Payment is confirmed | O42, amount 15,000 ₸, currency KZT and payment time |
| Analytics receives the data | Receipt time and one counted payment |
| Refund is received | Separate transaction and a result updated under the metric's rule |
Assignment and exposure can happen at different times. The plan decides in advance who goes into the main analysis. Keeping only buyers, or only people who reached a convenient step, means fitting the groups to the result.
To test delivery, send the payment record a second time. O42 should still count 15,000 ₸, even though the delivery log shows two attempts. Contact, deal and payment are linked by different keys: you cannot find a purchase number from the buyer's contact.
Label test records and keep them apart from live ones. Otherwise, during acceptance they end up in the financial report or reach the ad account as real conversions. One successful order confirms only the normal path; failures and the other variants are still ahead.
Give each part of the setup a job
In a small project, a spreadsheet, a CRM and a small handler cover part of the work. With more sources, you may need a warehouse and an experiment system. Company size tells you little until you have a list of jobs.
| Job | What to ask the vendor to show |
|---|---|
| Events and behaviour | Source records, a segment and the required unit of analysis |
| Sales | Order, confirmed payment and correction history |
| Variant allocation | Stable assignment and an exposure log |
| Exchange between systems | Retries, delays, failures and recovery |
| Messages | Unsubscribe, exclusion after an order and frequency limits |
| Reporting | Metric definitions and a reproducible calculation |
One product can cover several rows. But every row has a person responsible for it: an analytics vendor will not become the owner of the payment status in your CRM.
A feature flag can switch a variant on. But for the comparison to show that it was the variant that worked, it also needs proper assignment, an experiment unit, metrics and an analysis plan. If the page is one thing on Monday and another on Friday, the variant's effect gets mixed up with everything else that changed that week.
The GrowthBook experiment results documentation describes checks for group balance and other data problems. In the demo, ask to see a warning on a deliberately broken export. That tells you which checks work in the proposed configuration and which data you will have to connect yourself. If the system has no payments, the platform's name will not add them.
Continue acceptance testing with failures
After the normal path, start breaking things: take the test receiver offline, delay a payment, change the order in which records arrive. In the log, write what happened, how it was noticed and what the owner did.
O42 has two timestamps: when the buyer paid and when the information reached the report. If the payment reached analytics late, the purchase must not quietly move into another period. Time zone and reporting window are part of the metric definition.
Test device switching the way the vendor promises to recognise one person across devices. If the platform cannot link an anonymous phone to a laptop, say so in the result description. Do not merge records by similar names to make the table reconcile: that folds different people into one buyer.
Messages need their own test: the person withdrew consent, has already ordered or has hit the frequency limit. The system should skip the send and record why. A current-state check before sending is part of acceptance for the whole setup.
Get data you can leave with
Ask for an export of a small set: participants, assignments, available events, orders and metric definitions. From it, another specialist should arrive at the same participant and payment counts. A screenshot of the final conversion rate will not do that.
Before buying, find out whether the fields and history you need are there, what permissions and API limits apply, and how to export the data after the subscription ends. Take the exact terms from the commercial proposal and the chosen plan's documentation. Exporting the contact list does not carry over experiment history, so ask about that directly.
Session recordings need acceptance too. Type test text into the masked fields and make sure it does not show in the recording. Check that access permissions work and that retention and deletion are set up. Do not enter real passwords or payment details for this. Test the configuration you chose, and repeat the check when new fields appear in the form after an update.
Put subscriptions and labour in one budget
In our comparison, the first setup costs 30,000 ₸ a month in services and needs 20 hours of upkeep. The second costs 90,000 ₸ and needs four hours. For this budget, the team values an hour of work at 5,000 ₸. These are project estimates, not tied to any service's pricing.
| Cost | First setup | Second setup |
|---|---|---|
| Monthly services | 30,000 ₸ | 90,000 ₸ |
| Maintenance | 20 × 5,000 = 100,000 ₸ | 4 × 5,000 = 20,000 ₸ |
| Monthly total | 130,000 ₸ | 110,000 ₸ |
| One-off implementation | 60,000 ₸ | 180,000 ₸ |
| Implementation and six months of operation | 840,000 ₸ | 840,000 ₸ |
Every month the second setup is 20,000 ₸ cheaper, but its implementation costs 120,000 ₸ more. Under these assumptions, the difference pays back in six months. If the project ends sooner, that argument for the second setup disappears.
Put training, data migration, storage, support and a future exit into both estimates. Check what is already counted, or the same work is easy to add twice. When volume grows, recalculate the unit you actually pay for: events, contacts, messages or computation. Measure upkeep hours in the pilot: a vendor's promise is not a measurement.
Exercise: choose after the pilot
Build a job table for your client and run one order through both setups. Keep the evidence so another specialist can recheck it.
0 of 6 done
Suppose the pilot shows the first setup needs only 12 hours a month. Then it costs 30,000 + 12 × 5,000 = 90,000 ₸. The second still costs 110,000 ₸ and more to implement. If both meet the requirements equally, the labour saving that held up the old calculation is gone. To buy the second setup anyway, you need another reason, and that one has to be confirmed too.
And if neither setup could link the participant to O42, the dashboard choice waits until the data is fixed. The client gets more from a documented gap and a costed repair than from a new interface with the old mistake.
What to read while preparing the choice
Brian Balfour's principles, process and tactics help describe the work before you pick a tool. Bring your O42 scenario, export requirements and budget to the vendor meeting: they are what you accept or reject the proposal on.