A test framework for prompts

Define a repeatable way to judge whether a prompt that turns athlete data into scored attributes is accurate, consistent and honest when data is missing.

A response that merely looks right is not evidence. The same prompt can invent a statistic, overvalue volume over efficiency or give different answers to the same case, and an edit to the prompt can fix one problem and cause another.

The framework defines eight criteria, each scored from 1 to 5: accuracy, adherence to the data, consistency, completeness, justification, hallucination, score coherence and format. It uses ten test cases covering simple, complex, ambiguous, extreme and reliability situations (for example, many shots with low efficiency, missing data, contradictory indicators and a request for information that was never provided), each meant to run three times with the same input. Failures are linked to a hypothesis about the prompt, the prompt is rewritten as explicit, checkable rules, and the same cases run again. The cycle is: test, find failures, analyze causes, fix the prompt, retest and validate.

No results are published for this experiment. The tables in the original document are an illustrative scenario, written to show how failures would be identified and traced, not a record of measured runs.

It is a method, not a measurement: there is no published evidence about a real prompt. Ten cases with three runs each are a small sample, the 1-to-5 scores are assigned by a person, and a clean retest does not guarantee that new errors will not appear, so the cycle has to be repeated whenever the prompt, the model or the indicators change.

Turning vague instructions into rules that can be checked is what makes a fix testable. The framework puts missing data and unanswerable requests among the cases because that is where a model is most tempted to invent something.