The Reviewer Who Isn't There: Testing AI Oversight Promises Before You Sign
Photo: Constanceofficiel, CC BY-SA 4.0, via Wikimedia Commons
The phrase arrives late in most sales conversations, usually just after someone has described an automated decision and just before someone else has asked whether it is safe. Human in the loop, the vendor says, and the room relaxes. It is a comforting formulation. It is also, in most proposals, an assertion without an address, a name, a timetable or a fallback. For a UK SME signing a twelve-month agreement, the useful question is not whether a human is in the loop, but whether that human would still be in it on the second Tuesday of August, when half the operations team is away and the queue is at its longest.
A phrase doing two jobs at once
It is worth being fair to the vendors. In regulated settings, oversight language exists because the alternatives are worse: fully unattended automation, with no one accountable when an output is wrong. The ICO's guidance on automated decision-making, and the UK GDPR provisions that sit behind it, place real weight on meaningful human involvement rather than nominal sign-off — a distinction that has consequences well beyond the compliance team.
But that same phrase is also doing commercial work. It reassures a buying committee without committing anyone to a shift pattern. Two suppliers can use identical wording and mean entirely different things: one runs a review desk staffed through the working day; the other has a product manager who looks at a sample once a month. Both will tell you there is a human in the loop.
The remedy is not scepticism for its own sake. It is specificity, gathered early enough to matter.
Four questions, asked in a fixed order
Oversight claims tend to collapse when you push on four particulars.
Who reviews? Not a job title in a diagram, but a role that exists on an org chart, with a manager and a rota. Ask whether the reviewer is employed by the vendor, subcontracted, or offshore, and whether the same person also has a sales or delivery target that might compete with the review.
What do they see? A reviewer with the model's output and nothing else is not reviewing anything — they are rubber-stamping. Meaningful involvement usually requires the source input, the model's reasoning or confidence signal, and enough customer context to judge whether the output is sensible.
How fast? An oversight process measured in days is a quality-assurance process, not a loop. If the automated action has already taken effect by the time the reviewer looks, the human is auditing history, not gating behaviour.
What happens when they are unavailable? This is the question that separates a process from a promise. A named individual with no deputy is a single point of failure wearing a compliance badge.
The exercise: ninety minutes, one afternoon
You do not need a red-team budget to test these answers. A tabletop session with operations, finance and whoever owns the contract will usually do. Invite the vendor to walk through one realistic scenario end to end — a flagged transaction, a misclassified document, a generated customer response — and record what they say.
| Stage | Question to ask | Evidence to request |
|---|---|---|
| Trigger | What causes this item to be queued for review? | A sample alert or queue screenshot |
| Reviewer | Who picks it up, in which time zone, on which rota? | A role description or rota excerpt |
| Information | What appears on their screen? | A redacted example of the review view |
| Decision | What can they do — approve, edit, reject, escalate? | Documented permissions |
| Timing | What is the target and actual turnaround? | Recent performance data |
| Absence | Who covers leave, sickness and bank holidays? | Named backup role, not person |
| Record | What is written down, and for how long? | Log format and retention period |
| Review | Who reads the logs, and how often? | A sample monthly report |
The value of the exercise lies less in the answers than in the hesitations. A vendor who can produce a redacted review screen and a rota within the hour is describing a real operation. One who offers to come back to you is describing an intention.
The log is the proof
Oversight claims are only as good as the record they leave. Ask to see the audit log before you need it — not the marketing screenshot, but the structure: does each entry capture the item, the model output, the reviewer identity, the timestamp of the decision, and any edit made? Can it be exported in a format your own team can read without a vendor engineer present?
Two practical checks follow. First, retention: a log kept for thirty days will not help you investigate a complaint that surfaces two months later, and the ICO expects records proportionate to the risk. Second, access: if only the vendor can retrieve the log, you are reliant on their goodwill at precisely the moment goodwill is scarcest.
Escalation is a rota problem, not a policy problem
Most escalation paths are written as if nothing ever goes wrong on a Friday. Test yours against the calendar rather than the policy document. Who is on call over the Christmas shutdown? What is the defined response when the queue exceeds a threshold — does the system pause, widen the review, or keep going and hope? Is there a documented step-in arrangement if the vendor's review capacity drops?
A useful discipline is to name roles rather than individuals in every document, then keep the rota separately. The contract should not need amending when someone resigns.
Wording that keeps the claim honest
A handful of clauses do most of the work. Require a named reviewing role with a documented deputy. Set a review turnaround that is shorter than the time taken for the automated action to become irreversible, or require the action to wait. Oblige the vendor to retain review logs for an agreed period and to provide them on request within a stated number of working days. Require notice of any material change to the review process — including automation of the review itself — and reserve a right to renegotiate if the change reduces oversight. Finally, add a service credit or exit trigger tied to oversight performance, so the commitment has a consequence.
None of this is exotic. It is the ordinary apparatus of a service relationship, applied to a phrase that has so far escaped it.
Proportionality, and the point of the exercise
Not every AI tool warrants a review desk. A drafting assistant that suggests email wording needs lighter treatment than a system that scores credit applications. The test is whether an error would be visible to a customer, costly to reverse, or difficult to explain to a regulator.
Where it would be, the human in the loop should be traceable to a person, a screen, a clock and a calendar. That is a modest standard. It is also the difference between oversight you can demonstrate and oversight you merely inherited from a slide.
