If You Prompt Until You Like the Answer, It Isn't an Assessment

Confidence: hypothesis. The failures below are drawn from published research on language-model behavior, not firsthand deployment. The pattern at the end is an argument, not a shipped system. Argue with it.

A team wants a second opinion on their own forecast. So they ask a model to assess it, independently. The model comes back lower than they hoped. They adjust the prompt, add some context, run it again. Still a bit low. They reframe the question, feed in the supporting data, run it once more. This time the number matches what they wanted all along. They write it up as an independent assessment.

Nothing in that story involves lying. Every run was real. But the result is not an assessment. It is the team's own preferred answer, wearing a neutrality badge it did not earn.

The clinical why

A probability of success decides whether a program is funded. The team presenting it is the team whose program depends on it, so their honest estimate drifts upward. Everyone knows this, which is why governance wants an independent check on the number.

Now put an AI model in the role of that independent check. It looks perfect. It has no career riding on the outcome. It has read more precedent than any human reviewer. But there is a catch that matters more than any of that: the team being checked controls the prompt. And a model that you can re-run, reword, and re-feed until it agrees with you is not checking you. It is following you.