// WRITING · 2026.03.02
Evaluating llms on quantitative tasks
benchmarks lie. building small, honest evals for math-heavy and finance-heavy workloads.
Public benchmarks are optimized against the moment they are published. By the time a model tops a leaderboard, that leaderboard has leaked into training data. For anything quantitative, you have to build your own evals — small, private, and honest.
Why generic benchmarks fail here
A model that scores well on grade-school math can still fail on:
- multi-step numerical reasoning where an early rounding error compounds,
- unit and currency handling, where being almost right is being wrong,
- problems where the correct answer is “there is not enough information.”
Building an honest eval
- Keep it small. Fifty carefully chosen problems you understand beat five thousand you have never read.
- Grade the reasoning, not just the number. A right answer for a wrong reason will not survive contact with new inputs.
- Hold it back. The moment an eval is public, it stops measuring the thing you care about.
Benchmarks tell you who is winning a game. Evals tell you whether your system works. Only one of those pays the bills.