AI LabExploring the intelligence behind the code.
Experiments, research and ideas exploring the possibilities of artificial intelligence.
The philosophy
Explore
Investigate models and emerging technologies.
Experiment
Test ideas, architectures and techniques.
Understand
Learn from results, failures and discoveries.
Experiments & Research
I'm studying AI systems development and management, and I use study projects to explore language models. Each experiment states what I wanted to find out, how I approached it, what I measured and where it falls short. They are learning projects, not products.
EXP / 001
Featured experiment
Proof of conceptIn developmentGrounded answers over reports (RAG)
Answering questions about scouting reports using only retrieved excerpts, and showing how much of each answer can be traced back to them.
Model: gpt-4o-mini, OpenAI embeddings
Technologies
- TypeScript
- Node.js
- OpenAI API
- SQLite
- RAG
EXP / 002
Proof of conceptIn developmentValidating model output with rules and a judge model
Checking what a language model says with rules in code and a second model that has to quote the evidence.
Model: gpt-4o-mini
Technologies
- TypeScript
- Node.js
- OpenAI API
- SQLite
EXP / 003
StudyConcludedMedian of three samples for stable scores
Testing whether answering with the median of three samples makes a model's scores stable.
Model: gpt-4o-mini
Technologies
- TypeScript
- Node.js
- OpenAI API
EXP / 004
StudyConcludedAnalyst personas built with prompts
Designing analyst personas with prompts, conversation history and simple input and output checks.
Model: gpt-4o-mini
Technologies
- TypeScript
- OpenAI API
- SQLite
EXP / 005
StudyConcludedChoosing an approach and a model
Comparing rules, machine learning, language models and a hybrid design, then three models, to decide how to automate football performance analysis.
EXP / 006
StudyConcludedA test framework for prompts
A method to test a prompt like software: fixed cases, repeated runs, scored criteria, failures traced to causes and a retest after each fix.
EXP / 007
StudyConcludedFine-tuning a small model to classify reviews (LoRA)
Fine-tuning a 1.5-billion-parameter model with LoRA to classify short football performance reviews, and what the evaluation does and does not show.
Model: Qwen2.5-1.5B-Instruct, LoRA adapter
EXP / 008
StudyPlanningRondo
A platform to create and run experts: AI agents specialized in the functions of a football club, fed by connected data sources, with answers traceable to them. The technical foundation exists; product features do not.
Research log
The experiments in the order they appear in the lab, with what each one taught me. There are no dates: this is a path, not a calendar.
EXP / 001
Grounded answers over reports (RAG)
Read the experiment — Grounded answers over reports (RAG)Context: Answering questions about scouting reports using only retrieved excerpts, and showing how much of each answer can be traced back to them.
What I learned: A citation proves nothing by itself: what helped was showing, next to the answer, what could and could not be traced. A retrieval failure looks like a model failure, so the two are worth measuring apart.
EXP / 002
Validating model output with rules and a judge model
Read the experiment — Validating model output with rules and a judge modelContext: Checking what a language model says with rules in code and a second model that has to quote the evidence.
What I learned: Measure with cases written after the fixes, or the numbers flatter the work. A verifier should not be trusted blindly: requiring a quote made it checkable. The check costs an extra request per answer, the price of fewer unsupported statements.
EXP / 003
Median of three samples for stable scores
Read the experiment — Median of three samples for stable scoresContext: Testing whether answering with the median of three samples makes a model's scores stable.
What I learned: A negative result is still a result: it settled the decision to keep the consensus available but off by default. Consistency is something to measure, not to assume.
EXP / 004
Analyst personas built with prompts
Read the experiment — Analyst personas built with promptsContext: Designing analyst personas with prompts, conversation history and simple input and output checks.
What I learned: Without tests or measurements there is no way to tell whether a persona follows its instructions, which is the gap the validation and evaluation experiments address.
EXP / 005
Choosing an approach and a model
Read the experiment — Choosing an approach and a modelContext: Comparing rules, machine learning, language models and a hybrid design, then three models, to decide how to automate football performance analysis.
What I learned: Writing the trade-offs down makes the choice open to discussion. The experiment on validating model output is one way of applying the hybrid idea: rules and a verifier around the model, because the model alone does not guarantee consistency.
EXP / 006
A test framework for prompts
Read the experiment — A test framework for promptsContext: A method to test a prompt like software: fixed cases, repeated runs, scored criteria, failures traced to causes and a retest after each fix.
What I learned: Turning vague instructions into rules that can be checked is what makes a fix testable. The framework puts missing data and unanswerable requests among the cases because that is where a model is most tempted to invent something.
EXP / 007
Fine-tuning a small model to classify reviews (LoRA)
Read the experiment — Fine-tuning a small model to classify reviews (LoRA)Context: Fine-tuning a 1.5-billion-parameter model with LoRA to classify short football performance reviews, and what the evaluation does and does not show.
What I learned: A higher count is not evidence until the test data is kept apart from training: here the same file feeds both, so the table cannot say whether the adapter generalizes. Counting format failures separately from labelling errors, and looking at which examples drifted, taught more than the headline number. Next time I would split the data first, freeze the test set and repeat with other seeds.