A fine-tuned 2B model read receipts more accurately than Gemini Flash-Lite, at about the same cost

Why this test

Our first test was classification: read a message, pick one of 77 labels. Plenty of LLM traffic is a different job: read a document and fill in a fixed set of fields. Invoices, receipts, claims, order emails. The output is longer, every field can be wrong on its own, and the JSON has to parse every time.

We used CORD v2, a public dataset of receipts from shops and restaurants in Indonesia (CC BY 4.0). Each receipt comes with its text and labelled fields: line items, subtotal, tax, service charge, total, cash, change and card. Two things to know before you read the numbers. This is Indonesian shop receipts, not scanned US or European invoices. And the input is the dataset's clean, human-checked text, not the raw output of an OCR engine, so both models had an easier job than they would on real scans.

What we did

• Built 484 training receipts and a separate file of 300 held-out receipts, after removing 52 receipts whose text was identical to another one, so no held-out receipt also appears in training.
• Fine-tuned unsloth/gemma-2-2b-it with LoRA (300 steps of 4 examples) using SynqTune's structured extraction template, on one g4dn.xlarge GPU in our own AWS account. On held-out receipts, perplexity fell from 4.04 for the base model to 1.38.
• Served the fine-tuned model with vLLM on one g5.xlarge (an NVIDIA A10G), answering one receipt at a time.
• Sent 200 of the held-out receipts to both models. Gemini got a production-style prompt: the JSON schema below, the rules for each field, and JSON-only output.
• Scored both with SynqTune's comparison report: accuracy per field with 95% bootstrap intervals, receipts with every field right, whether the JSON parsed and matched the schema, latency, and cost from each side's real prices.

Both models had to produce this shape:

{
  "items": [{ "name": "...", "quantity": "...", "price": "..." }],
  "subtotal": "...", "tax": "...", "service_charge": "...", "discount": "...",
  "total": "...", "cash": "...", "change": "...", "card": "..."
}

Line items are scored item by item: same number of items, and each item's name, quantity and price must match one in the reference, in any order. A receipt with seven items and one wrong price counts as wrong on items.

Results

185 receipts were compared. 15 requests to Gemini failed, and those receipts were left out of both models' scores, so both are measured on exactly the same receipts.

More accurate, by a small margin per field and a bigger one per receipt. Two points of field accuracy sounds small, and the two 95% intervals almost touch, so treat that gap as likely rather than settled. The gap that matters for a pipeline is whole receipts: 70.3% against 63.2% came out with nothing to fix. On 39 receipts only Gemini made a mistake; on 24 only the fine-tuned model did. The first measurement, on 268 receipts, showed the same pattern: 67.9% against 60.1%.

Both produced valid JSON every time. Every answer from both models parsed and matched the schema, so neither needed a retry loop.

Per field, the two are close. The fine-tuned model was ahead or level on most fields and slightly behind on service charge:

Speed and cost: serving decides it

A receipt's JSON is about 200 tokens, so how fast the model generates decides both latency and cost. We measured the same model two ways:

The A10G costs about twice as much an hour as the T4, but vLLM answers about six times faster on it, so each receipt costs a third as much. The fine-tuned model's cost is the GPU's on-demand price times its measured time per receipt. Gemini's comes from its token use, about 239 input and 203 output tokens per receipt, at $0.30 and $2.50 per million tokens.

Latency is mixed. Gemini is half a second faster at the median. But one receipt in twenty took Gemini more than 15 seconds, against 3.4 for the fine-tuned model. If your pipeline waits on the slowest request, that tail matters more than the median.

Cost is roughly even, not a big saving. $0.52 against $0.58 is about 10% cheaper, and it assumes the GPU is busy all the time. Our numbers come from one request at a time; vLLM can answer many at once on the same GPU, which should cut the cost per receipt further, but we haven't measured that yet. At low volume, a GPU that sits idle costs more than paying per request.

Where it gets receipts wrong

A few receipts Gemini got right and the fine-tuned model didn't:

JPR SUS U 12,727
L C MESES 20,000
PLASTIK 33 0
SUDTOTAL 32,727
PB1 10% 10.000%
AMOUNT 3,273
3.00xITEMS
TOTAL 36,000
CASH 50,000
CHANGE 14,000

1x 52.000 52.000
Nasi CampurBali
1x 22.000 22.000
Lemon
SubTTL 74.000
TAX 7.400
CASH 81.400

1 MIE UP BSR C 1 36,000
INC(+) 3,600
Total 39,600
Payment 0
Payment#Less 39,600

Two of the three are the same habit: it fills in what a receipt usually says instead of what this one prints. Most training receipts have a total and a subtotal, so it supplies them. More training examples of receipts with fields missing are the obvious fix. Knowing that PB1 is a tax is local knowledge a larger model brings and 484 receipts don't fully teach.

Some receipts both models miss are labelling calls. On one, the receipt shows 10 tofu balls at 6.800 each for 68.000; both models gave the unit price, and the label wants the line total.

What's next

Measure the fine-tuned model answering many receipts at once, so the cost reflects a busy production setup. Then rerun on raw OCR text instead of clean text, and with more training examples of receipts that leave fields out. We'll update this post when we have those numbers.