A fine-tuned 2B model beat Gemini Flash-Lite by 7 points on 77 banking intents
The short version
On 270 held-out customer messages, a Gemma 2 2B model fine-tuned on Banking77 got 85.2% of intents right. Gemini 3.5 Flash-Lite, given the same instruction, got 78.2%. The 95% interval for the gap is +1.9 to +11.9 points, so the fine-tuned model is ahead, not just tied, and our report's verdict is “ready to replace”. It still loses on a few near-neighbour intents, and we haven't measured Gemini's cost yet.
Why this test
A lot of what companies send to GPT, Claude and Gemini is narrow and repetitive: read a customer message, pick the right category, route it. That's a job a small model trained on that one task should be able to do as well as a frontier model, for a fraction of the cost, on hardware the company controls. SynqTune exists to test that claim on your own data and give you a verdict you can show to whoever approves the switch.
So we ran it on ourselves first. Banking77 is a public dataset of banking customer messages, each labelled with one of 77 intents, from card_arrival to pending_top_up. Many of the intents are near neighbours, which makes it a fair stand-in for a support-routing task.
What we did
- Fine-tuned
unsloth/gemma-2-2b-itwith LoRA (rank 16, 300 steps of 4 examples) using SynqTune's classification template, on oneg4dn.xlargeGPU in our own AWS account. The whole run, from instance start to evaluated model, took 81 minutes. - Held out 10% of the data, which the model never trained on. On it, perplexity fell from 11.45 for the base model to 1.06.
- Sent 300 of the held-out messages to
gemini-3.5-flash-lite, with the same task instruction the fine-tuned model gets, which lists the allowed intents. This is the comparison a team actually faces: a prompted API model against a small model trained on their data. - Scored both with SynqTune's comparison report: accuracy with 95% bootstrap intervals, an interval for the difference, macro-F1, accuracy per intent, and the examples where either model was wrong.
Results
270 examples were compared. 30 requests to Gemini failed, and those examples were left out of both models' scores, so both are measured on exactly the same messages.
| Metric | Gemma 2 2B, fine-tuned | Gemini 3.5 Flash-Lite |
|---|---|---|
| Accuracy95% interval 80.4–89.6% and 73.7–83.0% | 85.2% | 78.2% |
| Macro-F1 | 0.838 | 0.767 |
| Answers that weren't a single valid label | 0.4% (1) | 0% |
| Examples only this model got right | 36 | 17 |
| Latency, median | 696 ms | 652 ms |
| Latency, p95 | 1,115 ms | 1,371 ms |
| Cost per 1,000 requestssee below | $0.106 | Not measured |
The gap is real at this sample size. The fine-tuned model is 7.0 points ahead, and the 95% interval for that difference, +1.9 to +11.9 points, doesn't include zero. With 49 examples last time, a tie was all we could say.
They fail differently. The fine-tuned model got 36 messages right that Gemini missed, and Gemini got 17 right that the fine-tuned model missed. Both missed 23. The one answer that wasn't a valid label came from the fine-tuned model: asked “can i deposit a check here?”, it invented cheque_or_cash_deposit instead of top_up_by_cash_or_cheque.
Latency is a wash. Gemini answered a little faster at the median, 652 ms against 696 ms, and the fine-tuned model had the shorter tail, 1,115 ms against 1,371 ms at p95. The fine-tuned model answered one request at a time on a single entry-level GPU; Gemini's numbers are network round trips to Google.
Where the small model still loses
Most of the fine-tuned model's misses were near-neighbour intents. These are messages Gemini got right:
| Customer message | Correct intent | Fine-tuned model said |
|---|---|---|
| “I contacted the seller for a direct refund last week but he still hasn't give me my money.” | Refund_not_showing_up | request_refund |
| “What do I do if my top-up has not gone through?” | pending_top_up | top_up_failed |
| “Is there restrictions on where I can use my card?” | card_acceptance | country_support |
| “how long does it take to get my card i am in a rush” | card_delivery_estimate | card_arrival |
| “I think I have an extra transfer charge.” | transfer_fee_charged | extra_charge_on_statement |
The same confusions show up per intent. The fine-tuned model got pending_top_up and card_acceptance right half the time where Gemini got all of them, and Refund_not_showing_up 60% of the time against Gemini's 100%. “Pending” versus “failed” was a weak spot in the 49-example run too. Both runs trained on the same data, so that was expected; adding examples for these intents is the obvious next step.
Where Gemini loses
Gemini's weak spots were different intents:
| Intent | Examples | Fine-tuned | Gemini |
|---|---|---|---|
| balance_not_updated_after_bank_transfer | 8 | 88% | 50% |
| apple_pay_or_google_pay | 7 | 100% | 57% |
| declined_cash_withdrawal | 7 | 100% | 57% |
| failed_transfer | 5 | 100% | 40% |
Each of these has only 5 to 8 examples, so treat any single row with care. Together they show what a prompt alone struggles with: telling apart intents whose names sound alike but that this bank separates on purpose, which is exactly what training on the bank's own labels teaches.
Some of the 23 messages both models missed look like labelling calls more than model errors. “Why is the transaction to my friend still pending?” is labelled transfer_not_received_by_recipient; both models said pending_transfer.
What we haven't measured yet
Gemini's cost. The fine-tuned model's $0.106 per 1,000 requests comes from the GPU's on-demand price ($0.526 an hour) and its measured time per request, answering one request at a time; batching would make it cheaper. Gemini's cost needs its token prices entered in the run, and we left them empty again. That's our mistake, and there is no savings figure in this post until we fix it.
Other datasets and models. This is one public dataset, one base model and one API model. A frontier model, or Gemini with a few examples in its prompt, may close some of the gap. Your own traffic is the test that counts.
What went wrong on the way
Getting a comparison we trust took several runs, and each failure changed SynqTune:
- The first run failed outright: the Gemini model we had configured had been retired, and every request came back 404. SynqTune now sends one test request when a comparison run launches, before any GPU starts, so a retired model or a key with no credits costs a fraction of a cent instead of a GPU run. You can turn that off for any run.
- The second run scored Gemini at 0%, because it was sent the bare customer message with no instruction and replied like a chatbot instead of naming an intent. Worse, our report called the fine-tuned model “ready”. A comparison where one side can't be scored is now reported as an error, not a verdict, and Gemini gets the task instruction by default.
- The third run, the 49-example one, counted Gemini's rate-limit waits and retries in its latency, which made Gemini look far slower than it is. Latency is now timed on the request that succeeds, which is why this post can report it.
Each of these makes a benchmark look better than it is. If you're evaluating a switch away from an LLM API, check for them in whatever tool you use, including ours.
What's next
A rerun with Gemini's token prices, so the report shows cost per 1,000 requests for both models and a monthly saving, and more training examples for the intents the fine-tuned model confuses. We'll update this post again when it's done.
Have a classification or routing workload on an LLM API?
We're looking for a few teams to run this comparison on their own data, in their own AWS account, with our help. You get the report; we learn what it takes to switch.