The Audit Is a Retrieval Exam

Confidence: hypothesis. The failure modes below are drawn from published inspection findings and widely reported industry patterns, not firsthand deployment. The buildable pattern at the end is not shipped. Argue with it.

You built a retrieval system. You evaluated it against your own test set — the questions you thought to ask, phrased the way you phrase them. It scored well. It shipped. Then a real user asked a question you never wrote down, phrased in a way your eval set never covered, and the system either missed the right document entirely or retrieved three different versions of the same fact from three different sources and returned all of them with equal confidence, letting the user pick.

Nobody on your team saw that failure until the user reported it. That is the ordinary, forgivable version of the problem. Now imagine the user is a government auditor, the “report” is a public finding with your organization's name on it, and you get exactly one attempt per question, live, in a conference room, with someone from your own team answering out loud in real time while you watch.

That is what a regulatory inspection is. It is a retrieval benchmark you do not control, run by an adversary who writes excellent queries, scored once, in public.

The clinical why, for outsiders

In clinical drug development, a sponsor (the company running the trial) eventually has to sit an inspector down — from the FDA, the EMA, or an equivalent body — and answer questions about how the trial was actually run. Show me the consent records. Show me this patient's dose change and everything that led to it. Show me how this problem was found, escalated, and fixed. The standard against which they're judged is called GCP, Good Clinical Practice — the rulebook for how a trial's data and decisions have to be documented. The evidence they're expected to produce lives in the TMF, the Trial Master File — in principle, the complete paper trail of the study's life.