Blog
Benchmarks, in the open
What happens when we put small fine-tuned models against the LLMs they're meant to replace, including the parts that don't go our way.
A fine-tuned 2B model read receipts more accurately than Gemini Flash-Lite, at about the same cost
We fine-tuned Gemma 2 2B to turn receipt text into JSON and compared it with Gemini 3.5 Flash-Lite on 185 held-out receipts: 70.3% against 63.2% of receipts fully right. Served with vLLM on one GPU, it cost $0.52 per 1,000 receipts against $0.58, with a p95 latency of 3.4 seconds against 15.3. The numbers, the mistakes, and why serving changed the cost threefold.
A fine-tuned 2B model beat Gemini Flash-Lite by 7 points on 77 banking intents
We fine-tuned Gemma 2 2B on Banking77 and compared it with Gemini 3.5 Flash-Lite on 270 held-out customer messages: 85.2% against 78.2% accuracy, a gap that holds at 95% confidence. The full numbers, where the small model still loses, and what we haven't measured yet.