Tune Autopilot thresholds
Pick a confidence threshold and burst limit you can defend, and know when to move them.
On this page
The two numbers
Reply threshold — minimum classifier confidence before the AI answers on its own. Below it, the conversation goes to a human.
Hourly burst limit — the maximum automated replies sent in any one hour.
A staged rollout
Do not start at full automation. This sequence takes about two weeks and surfaces problems while they are still cheap.
| Stage | Threshold | Autopilot | What you're learning |
|---|---|---|---|
| Week 1 | 95 | Off | Are the drafts any good? |
| Week 2 | 95 | On | Does high-confidence automation hold up? |
| Week 3 | 90 | On | Where does quality start to slip? |
| Steady | 85–90 | On | — |
Below about 80, you are automating answers the system itself is telling you it is unsure about. Very few teams should go there.
Why the burst limit exists
It is a blast radius control, not a performance setting.
If a knowledge base article is wrong, or a policy changed and nobody updated the AI, the burst limit decides whether twelve customers or four hundred get the wrong answer before someone notices. Set it to roughly what your team would handle in an hour manually. Raise it only once you have gone a few weeks without a bad automated reply.
Per-intent tuning beats one global number
A single threshold applies the same caution to "what are your hours" and "my order is three weeks late". Those deserve different treatment.
Set low-risk intents to Auto and everything sensitive to Assist. You will usually get better results from a moderate global threshold with careful per-intent modes than from one very high number.
When to change a setting
Change it when you have evidence, not on a schedule:
- Raise the threshold if you find AI replies that were confidently wrong
- Lower it if humans are handling a pile of questions the knowledge base clearly answers
- Raise the burst limit after a clean stretch with no bad replies
- Lower it immediately after any incident, while you investigate
What tuning cannot fix
If the AI escalates constantly at a high threshold, the problem is usually an empty or vague knowledge base, not the threshold. Lowering the number then just converts escalations into wrong answers. Fix the knowledge first — see Train the AI.
Related