Conversation quality review
How Coworker conversations are reviewed automatically, the sampling share and low-usage settings, what it costs, and what is and is not stored.
Conversation quality review reads a sample of each Coworker's finished interactions and labels each one as helped, struggled or unclear. The totals appear on the Signals card of the Coworker's details page, so a Supervisor deciding on a promotion has evidence beyond thumbs. See Coworker details.
What gets reviewed#
An hourly job looks at interactions from the last 30 days for Coworkers in the Interview, Internship or Apprenticeship phase. An interaction is eligible when it has finished: the person's message, the Coworker's reply and the person's follow-up. A sample of the eligible ones is reviewed. The sample is fixed for each interaction, so running the job again never re-draws it, and raising the percentage only adds interactions.
The review sees the person's request, the Coworker's whole answer including a note of each tool it used and whether that tool succeeded (never the tool's output), and the person's next message.
Skipped on purpose:
- Interactions in conversations started by triggers or other automatic work.
- Interactions that open with an automatic system message rather than a person's request.
- Interactions where the person already gave a thumbs-up or thumbs-down to any reply in the exchange. Their own rating stands.
- Interactions whose conversation content has since been removed by your retention settings. They are passed over, not counted as failures. Reviews read the stored conversation, so a retention period shorter than 30 days shortens what can be reviewed to what still exists.
- Interactions whose review has failed three times.
Who can see the results#
The Signals card is visible to the Coworker's Supervisor and to people with the admin or
executive ability. It is not shown to anyone else, and nothing on it is broken down by person.
Only a Supervisor sees Review new interactions. Settings are admin-only.
The Supervisor can also open the interactions behind the struggled counts: each reason on the card links to the Feedback to work through list on the Coworker's guided review, filtered to that reason. The Supervisor reads them with the same access they already have to their Coworker's conversations, and marks each one as Fixed, Could not reproduce, Transient error, Not fixable, Not a problem or Other. Admins and executives see aggregates only. The list shows the date, the reason and the duty, and no names. A row whose conversation was removed by your retention settings is marked Content no longer available; the Supervisor can still mark it.
Struggled interactions also appear as themes in the weekly feedback synthesis, tagged Automatic review. Each synthesis run takes in at most the 60 newest of them.
Turn it on or off#
Open Settings, then the Administration tab, and select Manage on the Features card. Conversation quality review has an Off and On switch. It is on by default.
The switch controls the hourly schedule only. With it off, nothing is reviewed automatically, but a Supervisor can still select Review new interactions on a Coworker's details page.
The environment variable WI_QUALITY_REVIEW_SCHEDULER=none removes the hourly schedule for
every company, regardless of the switch; the default is dbos. See the
Configuration reference.
Sampling share#
While Conversation quality review is On, its card on the same page also shows Share reviewed (percent), a whole number from 0 to 100. It is the share of eligible interactions that are reviewed. The default is 25. A higher share gives a sharper read and costs more. 0 turns automatic review off for the company, including Review new interactions, which then tells the Supervisor that automatic review is switched off. Select Save to apply.
Lowering the share does not shrink a run already in progress; the next run uses the new value.
What it costs#
Each review is one model call, with a second one when the first reply cannot be used. It uses your Default small model, or the company default model if that is not set (see Default models). With neither set, a manual review reports that automatic review isn't set up and the schedule skips your company.
Each review sends only that one interaction, trimmed to a few thousand tokens. Limits keep spend bounded regardless of the percentage: at most 40 reviews per Coworker in any rolling week and at most 200 per run for the company. A failed attempt counts against the weekly limit.
The cost appears under the system job quality-review in the System jobs table of
Usage and cost, and also under the Coworker it was for.
Low-usage warning settings#
Under Promotion gates, the Low-usage warning row (select Configure) sets when promotion shows a Little recent usage warning. The warning appears in the advance dialog for a Coworker leaving Internship or Apprenticeship. It never blocks the move.
| Field | Default | Meaning |
|---|---|---|
| Minimum interactions | 5 | Warn when the Coworker had fewer interactions than this in the look-back period. |
| Look-back (days) | 30 | How many days back interactions are counted, 1 to 365. If the Coworker entered its current phase more recently, only the time since then is counted. |
| Idle days | 14 | Warn when nobody has used the Coworker for this many days, 1 to 365. |
Select Save in the dialog to apply. See also Promotion gates.
Use in Value estimates#
The daily Value estimates read these verdicts: a conversation that struggled is counted at a discount, and every other conversation, including all those not reviewed, is counted in full.
What is stored#
Only labels are stored for each reviewed interaction: helped, struggled or unclear; for a struggled one, a reason (the person corrected a mistake, asked again another way, got an incomplete reply, gave up, got an error, or other); and which of the Coworker's Primary Responsibilities it matched, if any. The conversation text is read from the stored conversation when needed and is never stored in the results. It is sent to your company's configured model provider, exactly as in chat. Because only labels are kept, results survive conversation retention.
How to read the numbers#
- Minimum samples. A helped rate needs at least 20 reviewed interactions that were judged helped or struggled; unclear ones do not count toward it. A duty's rate needs 5. Below that, the card shows the number reviewed and no percentage. Thumbs-up rate needs 5 ratings and the tool failure rate needs 10 tool calls.
- Finished interactions only. An interaction is reviewed once it is finished, so one the person simply left unfinished is not reviewed. This under-counts successes and struggles equally, so treat the helped rate as a trend rather than an exact figure. An interaction where the person never followed up has weak evidence and is often labelled unclear.
- Critical errors are failed chat replies whose cause a retry would not normally clear. Failures such as a busy or unreachable service, a timeout or a rate limit are counted separately as temporary errors.
- Tool failures count a Coworker's top-level tool calls that failed, with the tools that failed most. A saved script counts once, under its own name. The calls a script makes inside itself are not counted individually.
- Model opinion. Verdicts are a model's judgment of the person's follow-up message; ambiguous cases are marked unclear. Review a sample of the interactions behind the counts before relying on them.