Case Study
LLM-as-Judge Evaluation Framework | 1,360 Scored in 18s
A Solana AI hackathon had 1,360 submissions and several prize categories, one of them worth $500,000. Reading all of that by hand is weeks of work, and by the time you finish, the judging is inconsistent because nobody scores project 1,300 the way they scored project 12.
I built an LLM-as-judge framework that scored the whole batch in about 18 seconds and shortlisted the top three in each category. Every result carries a numeric score and a written justification for it.
How it works
One engine, different rubrics
Each prize category defines its own criteria, so the grand prize and the more specialized awards are not judged against the same yardstick. The engine takes the rubric as input rather than baking it in, which is why the same code handles all of them.
Upload the submissions, and results come back grouped by category with scores and reasoning attached to each shortlisted project.
Why the justification matters
A score on its own is not much use. If a project loses, somebody will ask why, and "the model said 78" is not an answer.
So every score comes with a written explanation of how the project met or missed the criteria. That is also what makes the whole thing auditable, which is the same reason this pattern works as a quality gate inside other LLM applications rather than only for hackathons.
Making it fast and affordable
Eighteen seconds for 1,360 projects comes down to concurrency rather than the model. I built the scoring pipeline, the structured output parsing that keeps every result machine-readable, and the concurrency layer that keeps all those calls fast without the cost getting silly.
It has since been reused as a benchmark and evaluation gate on other work, which was not the original brief but turned out to be the more useful part.
More screenshots
