Loading…

Receipt extraction, served with vLLM: Gemma 2 2B against Gemini Flash-Lite | SynqTune

This run scored a saved model again, without training: the model from the "Receipt extraction: fine-tuning" sample, served with vLLM and answering one receipt at a time.

Fine-tuned model: unsloth/gemma-2-2b-it
Compared with: gemini-3.5-flash-lite
GPU: g5.xlarge
Run time: 32m 15s
Held-out perplexity: 1.38 (base 4.04)

Instance setup

g5.xlarge

Succeeded

instance:: g5.xlarge
disk:: 100 GB

Load data

Succeeded

Default settings

Saved model

saved_model

Succeeded

base model:: unsloth/gemma-2-2b-it
task instruction:: Extract the fields defined by this JSON schema from the input. Reply with a single JSON object only, using null for fields that are not present. Schema: { "type": "object", "required": ["items", "subtotal", "tax", "service_charge", "discount", "total", "cash", "change", "card"], "properties": { "items": {"type": "array"}, "subtotal": {"type": ["string", "null"]}, "tax": {"type": ["string", "null"]}, "service_charge": {"type": ["string", "null"]}, "discount": {"type": ["string", "null"]}, "total": {"type": ["string", "null"]}, "cash": {"type": ["string", "null"]}, "change": {"type": ["string", "null"]}, "card": {"type": ["string", "null"]} } }

Evaluate

eval_holdout

Succeeded

holdout %:: 10
max samples:: 100

React Flow

Training loss

1 steps

Comparison with your current LLM

Scored for structured extraction · 185 held-out examples · tolerance 2 points · set 280989e5

Ready to replace

15 incumbent request(s) failed and were left out of both scores.

Metric	Fine-tuned	gemini · gemini-3.5-flash-lite
Field accuracy (95% interval)	95.7% (94.7–96.7)	93.9% (92.3–95.4)
All fields correct	70.3%	63.2%
Valid JSON	100.0%	100.0%
Matches the JSON schema	100.0%	100.0%
Latency p50 · lower is better	1604 ms	1139 ms
Latency p95 · lower is better	3380 ms	15264 ms
Cost per 1,000 requests · lower is better	$0.5240	$0.5788

Estimated savings: $10.96 per month at your request volume.

Fine-tuned cost measured single request at a time; batched serving is cheaper.

Fine-tuned model served with vLLM.

Requests sent several at a time

The same examples, sent to the fine-tuned model N at a time, as a server is used in production. Cost is the GPU's hourly price over the examples answered per hour.

At a time	Per second	Latency p50	Latency p95	Cost per 1,000
1	0.54	1613 ms	3411 ms	$0.5209
8	3.89	1696 ms	3484 ms	$0.0718
32	10.36	2094 ms	4261 ms	$0.0270

Accuracy by field

Field	Examples	Fine-tuned	Incumbent
items	185	80.5%	75.7%
subtotal	185	95.7%	94.6%
tax	185	97.8%	96.8%
service_charge	185	98.4%	98.9%
discount	185	97.8%	95.7%
total	185	97.8%	93.0%
cash	185	97.3%	97.3%
change	185	98.4%	97.8%
card	185	97.8%	95.7%

Disagreements

Fine-tuned worse than incumbent: 24 · Both imperfect: 25 · Incumbent worse than fine-tuned: 39