Median of three samples for stable scores

Find out whether any mode makes the scores for the same input reliable from one run to the next.

The same input can produce different scores from the same model, even at temperature 0, which undermines a score that people might act on.

Each of six athlete cases was answered eight times per mode (temperature 0 with and without a seed, and the median of three samples at two temperatures), and the score vectors were compared. Only the deterministic validation was applied. The experiment ran twice, with 48 answers per mode and run.

No mode made the scores reliably stable: the case that varied changed between runs, and the same mode went from a largest spread of 1 point to 8 between two executions. The seed did not help consistently. The median vote was the only mode in which every answer was valid, because it discards invalid samples.

Six cases and two runs are a small sample. The variation comes from the model, so tuning these settings cannot remove it. The median costs three times the generation. Computing the scores in code from the cited indicators would remove the variation, but it would turn qualitative judgments into a formula, so it was not done.

A negative result is still a result: it settled the decision to keep the consensus available but off by default. Consistency is something to measure, not to assume.