Grounded answers over reports (RAG)

Answer questions about scouting reports using only what the reports say, and make it visible how much of each answer is actually supported.

A model asked about reports it has not read invents plausible details: numbers, sources and actions. Citing a source does not prove that the source says it.

Report PDFs are turned into text, translated from Spanish to Brazilian Portuguese, split into chunks and stored with embeddings in SQLite. A question retrieves the five most relevant chunks, ranked by a hybrid score that combines embedding similarity with keyword overlap and discards weak matches. The prompt requires a source after each statement and runs at temperature 0. Every answer then ends with a grounding report: a lexical check of how many statements, numbers and cited sources appear in the retrieved text, followed by a semantic check made by a second model call.

In the evaluation, a question supported by the reports was answered with the semantic check passing, and questions the reports could not answer (a past tournament winner, a player's salary) were declined as expected. One question written afterwards failed: retrieval ranked the right report first but filled the other four slots with excerpts about other athletes, and the model declined. Only a handful of RAG questions were tested, so this shows behavior, not a failure rate.

The index has no athlete metadata to filter on, so a question about one athlete can retrieve excerpts about others. PDF extraction does not use OCR, so scanned reports may yield no text. The relevance score ranks excerpts; it does not measure confidence or truth. The lexical check is not semantic, and answers are not compared with official statistics.

A citation proves nothing by itself: what helped was showing, next to the answer, what could and could not be traced. A retrieval failure looks like a model failure, so the two are worth measuring apart.