The owner of an online appliance store in Almaty received a polished digest of the week's reviews: three main themes, percentages and an explanation of why deliveries were late. The marketer, Akmaral, opened the source texts and saw that customers had only written about the delay itself. The model had made up the cause. The report looked like a diagnosis, although nobody had even started looking into delivery.
The store needs a weekly digest of review themes, and verified observations should reach the people responsible for them. The setup: AI proposes groupings and a draft, Akmaral checks the meaning, an ordinary formula counts the records, and the head of the relevant area, delivery or support, decides what to investigate next. All numbers in this lesson are made up.
An AI pilot for review analysis
Source reviews
Retain texts for meaning checks
Draft themes
AI proposes grouping and a draft
Verified digest
Akmaral checks the meaning; a formula counts the records
Decision by the area head
Pass on observations and choose what to investigate
Customers reported a delay. The team has not investigated its cause yet.
Define a result someone can accept
The report should state the period, the number of reviews included, the themes, the review IDs behind each theme, the frequencies and short observations. If a customer did not give a reason for a complaint, say so: "Cause to be investigated". Write guesses about causes under the observation, as a list of questions to check.
As the process owner, Akmaral sets the boundary: this is a digest of reviews from one agreed source, and it says nothing about all customers. People who write a review already differ from those who stay silent. If a theme appears in half of the reviews, that does not mean half of all customers are unhappy about it.
Sign-off can be checked point by point. Every theme rests on source records. A quote keeps their meaning. The count can be reproduced from the IDs. An unverified cause does not become a fact. If even one point fails, smooth writing does not make up for it.
Sort out the input first
Fix the source, the start and end of the period, which records are included and how duplicates are removed. If the same message lands in the export twice, its frequency must not double. For two different messages from one person, pick the rule in advance: are you counting messages, enquiries or customers? In our example, the unit is a review with a stable ID.
Keep the original text, its language and the link to the source in a working environment approved for this data. Give the model only what the task needs. Agree on access and data scope before launch: if the task is only to find themes, the model has no use for names, phone numbers or order details.
A long review, an empty record and a message about three problems at once must not vanish quietly. If you exclude a record, note the reason and how many such records there were. Otherwise a new tool will look more accurate than the old one simply because it stopped handling difficult texts.
Split the roles of model, rules and person
| Step | Who does it | What to check |
|---|---|---|
| Selection and exact duplicate removal | Agreed rules | Period, source and number of exclusions |
| Theme proposals | Model following instructions | Each theme's link to source IDs |
| Counting | Formula or script on checked IDs | Uniqueness and denominator |
| Sign-off on meaning | Akmaral | Quotes, limitations, no new facts |
| Passing on observations | Akmaral as digest owner | Who received it and the next step |
Automatic file loading can come later. First show that the team produces a useful digest that can be checked. If the categories are fuzzy, moving data without a person in the loop will not fix them.
In the instructions, allow a review to have several themes and allow the answer "unclear". The review "The order came late, but an employee helped sort it out" is about both delivery time and support. A single "negative" label loses half the meaning. And "late because there weren't enough couriers" adds a cause that is not in the text.
Count reviews, not matched phrases
The checked set has 20 unique reviews. Delivery comes up in 12, packaging in eight, support in six. One review can fall into several themes, but within a theme its ID counts once.
| Theme | Unique reviews with the theme | Share of 20 reviews |
|---|---|---|
| Delivery | 12 | 60% |
| Packaging | 8 | 40% |
| Support | 6 | 30% |
The shares add up to 130% because the themes overlap. There is no need to force them to 100% by dividing by 26 mentions: that gives a different measure, the share of mentions, and it no longer answers "how many reviews are about delivery". R01, for example, may sit in both delivery and support, but in the overall review count it is one record.
Theme frequency says nothing about tone. Support can be praised or criticised. If that matters for the decision, add a tone rule, test it and keep the original phrase next to the rating. Counting six mentions of support as six complaints without reading them is a mistake.
Prepare an evaluation set
Collect varied reviews: short and long, clear and ambiguous, with several themes and with too little information. If customers write in Russian and Kazakh, you need a reviewer who understands both. For ambiguous cases, write down in advance which answers are acceptable: there is no need to demand a word-for-word match with a reference.
OpenAI's evaluation guide suggests grounding evaluation in real tasks and comparing automated scores with human review. For a review digest, that means: build a list of typical errors and properties of a good answer, and compare new instructions against the old ones with it.
Keep tuning examples and evaluation inputs apart. If you change the request after every answer on the same twenty reviews, better quality on them tells you nothing about next week. After tuning, freeze the version and run it on new reviews. Repeat some inputs: that shows whether the answer is stable.
In the initial evaluation set, 18 of 20 reviews were tagged without errors. In two, the model added a cause for the complaint. That gives 90% fully accepted answers, and it applies to this set only. Before building the final table, Akmaral checked all 20 records and fixed both errors. The theme table above is already the result after that sign-off.
Count the path to release
Suppose a manual weekly digest takes three hours. In the new process, 30 minutes go to preparing the input, ten minutes to the draft, an hour to review and twenty minutes to corrections. Two hours in total. The measurement is illustrative: same volume, same accepted quality, rejected attempts included.
Over four releases a month, manual work takes twelve hours and the new process eight. At an internal rate of 5,000 ₸ an hour, that is 60,000 ₸ and 40,000 ₸ of team time. We picked the rate for the calculation; it does not describe a market price for an hour.
On top of the time per release, maintenance takes an hour a month, another 5,000 ₸. The tool has 10,000 ₸ a month set aside, not tied to any service's pricing. Under this valuation the new process costs 40,000 + 5,000 + 10,000 = 55,000 ₸. That is 5,000 ₸ a month less than manual work.
Initial setup took another three hours, or 15,000 ₸. So the first month comes to 70,000 ₸ against 60,000 ₸ for manual work: it pays for the setup, and the saving starts in the second month. Record real payments and the value of time on different lines.
Assign an owner and a path for failures
Akmaral, who releases the digest, needs the source records, the criteria and the right to stop it going out. In a log she keeps the instruction version, the tool, review time, errors and corrections. If quality drops, the log shows what changed: the input or the process.
If the file has not arrived, some records went missing or nobody is available to check the meaning, the report does not become ready just because it is Monday. Decide in advance who does the digest by hand, who warns the recipient and who restores the process. If failures keep happening, revisit the scope and the method.
Expanding the pilot starts with a new task that can also be checked. Sending a draft to an editor and sending conclusions to customers are different actions. Publishing or changing a budget needs its own permissions and its own sign-off conditions, and a successful review grouping never tested them.
Exercise: what do you know after the pilot?
Take a weekly process of your own that you want to hand to AI, partly or fully. Write down the counting unit, source data, sign-off criteria, owner, total working time and new cash costs. Mark which step stays with a person and how they will check the result.
Then work through the store's case. The team shows that 18 of 20 reviews were tagged without errors and proposes sending future reports without review. It also thinks the 130% in the table is an error and wants to divide every theme by 26. What do you tell them, and when does the initial setup pay for itself on these numbers?
Two answers contained causes that were not in the reviews, so 90% is no reason to drop the review. The instructions need fixing and a fresh run on new inputs. The sum of the shares is not an error either: the themes overlap, and each share answers how many of the twenty reviews mention that theme. Dividing by 26 changes what the measure means. Keep the denominator and explain the overlap under the table.
Under the illustrative valuation, the repeat process is 5,000 ₸ a month cheaper than manual work. Three months of that difference cover the 15,000 ₸ of initial setup, if volume, quality and effort stay the same. Over three months both options cost 180,000 ₸: manual 3 × 60,000, new 15,000 + 3 × 55,000. The valuations even out. In cash, the new process costs 30,000 ₸ more over three months: salaries stay the same and the tool is paid for. The calculation does not license dropping Akmaral's review.
Your process record is ready when a colleague can reproduce the count from it and see who signs off the release. No source data or no time for review means nobody is controlling quality yet. Labour and payments merged into one unexplained total need splitting before you ask for a budget. Close these gaps before expanding the pilot.