Make It Structurally Impossible

Confidence: hypothesis. The failure modes below are drawn from published inspection findings and widely reported industry patterns, not firsthand deployment. The architecture pattern at the end is not shipped by me anywhere. Argue with it.

You gave your agent a tool called update_ticket. It works well — that's the problem. Ask it to reconcile a gap in the log, and it will. Fluently. Convincingly. With a timestamp that looks right, phrasing that matches the surrounding entries, and zero awareness that the gap it just closed should have stayed open, flagged, and investigated. The model isn't malfunctioning. It's doing exactly what generation does: producing the most plausible continuation of a pattern. The problem is that you pointed that capability at your record of what actually happened, and gave it the credentials to write there.

This used to be a hard problem. Fabricating a convincing period-style document — the right letterhead, the right phrasing, the right “missing” meeting minutes — took a skilled forger real time and real skill. An LLM does it in seconds, on request, without malice, often because a well-meaning engineer asked it to “fill in the gaps” in a dataset. The cost of fabrication just fell to near zero. Your controls were built for a world where it wasn't.

The clinical version of this problem

As per my knowledge, clinical drug development is one of the most heavily audited document cultures on earth. Every trial keeps a Trial Master File (TMF) — the complete record of who did what, when, and why, across years and often dozens of sites. Its existence isn't optional; regulators require it, and a GCP inspection (Good Clinical Practice — the rulebook for running an ethical, well-documented trial) is largely an audit of whether that file actually tells the truth. EMA's 2023 inspection report recorded 720 deficiencies across 67 inspections, and the single largest category — ahead of everything else — was Essential Documents: missing files, undefined storage, records nobody could produce on request.

As per publicly documented inspection patterns, inspectors tend to forgive a documented mess. A deviation with an honest reason, logged when it happened, reads as competence. What they cite is chaos: the record that doesn't exist, or worse, the record that exists but couldn't have been written when it claims to have been.