Review AI quality
A weekly routine for catching bad automated replies before your customers do.
Automation without review drifts. Thirty minutes a week is enough.
The weekly pass
1. Read ten automated replies at random. Not the ones that got complaints — random ones. You are looking for answers that were confident, plausible, and wrong. Those never generate a complaint; the customer just quietly leaves.
2. Read every low-confidence escalation. Each is the AI telling you something is missing. If the answer is documented and it still escalated, the article is not direct enough. If it is not documented, write it.
3. Check reopened tickets. A resolved ticket the customer reopened is the clearest signal of an answer that did not land. Reopen rate is the most honest quality metric you have.
4. Scan for repetition. The same question appearing over and over means a knowledge gap, a confusing product page, or a policy customers cannot find.
What to act on
| Finding | Action |
|---|---|
| Confidently wrong answer | Fix the article, then raise the threshold |
| Escalated a documented question | Rewrite the article to answer in the first sentence |
| Same question repeatedly | Add an article; consider fixing the source of confusion |
| High reopen rate on one intent | Set that intent to Assist while you investigate |
Metrics worth watching
- Automated resolution rate — share of tickets the AI closed without a human. Rising is good, but only alongside a flat reopen rate.
- Reopen rate — the honest counterweight. Rising means you are closing tickets that were not solved.
- Handoff rate — falling means the knowledge base is improving.
- First response time — should drop sharply once automation is on.
Never read the automation rate alone. It is trivially improved by lowering your threshold, at the cost of exactly the thing you care about.
After an incident
If a bad answer went out at volume:
- Set the affected intent to Escalate immediately
- Lower the hourly burst limit
- Fix the underlying article
- Return the intent to Assist, watch a day, then to Auto
The burst limit is what makes this recoverable. See Tune Autopilot thresholds.
Related