A fine-tuned 2B model matched Grok 4 Fast at routing consumer complaints, in 75 ms instead of 5.6 seconds

Why this test

Banks, lenders and fintechs get complaints by email, chat and web form, and someone has to send each one to the right team. Many now use an LLM API for that first step. It's a narrow, high-volume job with a fixed list of answers: the kind of call a small model trained on your own history should be able to take over.

The US Consumer Financial Protection Bureau publishes every complaint it receives, with the consumer's own description and the product they filed it under, as a public-domain database. The descriptions are real, long and messy, and personal details are already redacted as XXXX. We used 9 of its products:

What we did

• Took complaints from one slice of the database that have a written description, dropped duplicates and very short ones, and cut long descriptions to their first 300 words. We balanced the products: 450 training complaints and 34 held-out complaints for each.
• Fine-tuned unsloth/gemma-2-2b-it with LoRA on the 4,050 training complaints, using SynqTune's classification template, on one g5.xlarge (A10G) GPU in our own AWS account. On held-out text, perplexity fell from 43.06 for the base model to 5.36.
• Sent 200 held-out complaints to grok-4-fast through xAI's API, with the same task instruction the fine-tuned model gets, which lists the 9 allowed products. The fine-tuned model, served with vLLM, answered the same 200.
• Scored both with SynqTune's comparison report: accuracy with 95% bootstrap intervals, macro-F1, accuracy per product, latency, cost, and every complaint either model got wrong. The comparison took 17 minutes, using the model already saved from the training run.

Results

All 200 requests to Grok succeeded, so both models are scored on the same 200 complaints.

Accuracy is a tie. 1.5 points apart, with intervals that overlap almost entirely. 200 complaints aren't enough to tell which is better overall, and the report says so.

But they aren't the same model. Each got about 15 to 18 complaints right that the other missed, and the split by product, below, is lopsided in both directions. A tie in the headline number hides that.

Speed and cost

The answer is one label, so the fine-tuned model is quick: 75 ms at the median and 93 ms at p95, one request at a time. Grok 4 Fast took 5.6 seconds at the median and 15 seconds at p95 to produce the same few tokens. Called as grok-4-fast, the model reasons before answering, which likely accounts for most of the wait. xAI also offers a non-reasoning variant, which should be faster and cheaper; we haven't tested it yet.

Grok's $0.179 per 1,000 comes from its token use, about 441 input and 2 output tokens per complaint, at the $0.40 and $1.00 per million tokens we set for the run. It counts only the tokens in the answer; reasoning tokens, which xAI also bills, weren't included, so Grok's real cost is somewhat higher. The fine-tuned model's cost is the GPU's on-demand price times its measured time per complaint, and it falls as vLLM answers more requests at once:

As before, the low numbers assume the GPU is kept busy. A support queue that gets a complaint a minute would leave it idle, and per-request pricing wins at that volume. A team routing tens of thousands a day, or re-routing a backlog, is where one GPU pays for itself.

Where each model wins

Accuracy by product. The report lists the five where the fine-tuned model did worst; the last row is everything else:

The small model loses clearly on vehicle loans, credit cards and bank accounts, and wins by a wide margin on debt collection, mortgages, money transfers and personal loans taken together: 82 of 93 against 67. Each row has around 20 complaints, so single rows are rough, but the pattern is consistent.

These are complaints Grok got right and the fine-tuned model didn't:

Every one of these went to money transfer. Fintech debit cards, gift cards, a Chime dispute: when money moves through an app, the small model reaches for money_transfer. That's a learned shortcut, and it costs it the credit card and bank account rows.

Some of these labels are arguable. The product on a CFPB complaint is the one the consumer picked when they filed it, not one an analyst assigned. Whether a prepaid gift card is a credit card or a Chime dispute is about a bank account is a judgment call, and consumers make it inconsistently. On your own data, where your team assigns the labels, this kind of noise is usually lower.

The fix is more training examples of app-based cards and accounts filed under credit card and bank account, so the model stops treating “app” as “money transfer”.

What went wrong on the way

• We first compared this model with Gemini 3.5 Flash-Lite. The first full run failed at its last step, the comparison, and the trained model wasn't saved. A failed comparison now keeps the model. On the rerun the model was saved, but 94 of 306 requests to Gemini timed out, too many for the comparison to count. Our Gemini caller now retries timeouts.
• We had just added xAI's Grok to SynqTune, for comparisons and for grading training data, so we ran the comparison again, without retraining, against Grok 4 Fast. A Gemini rerun is still to come.
• We compared on 200 of the 306 held-out complaints. That's why the intervals are wide; the next run will use all of them.

What's next

A retrain with more card and account examples, compared on all 306 held-out complaints against Grok 4 Fast, its non-reasoning variant and Gemini Flash-Lite, with reasoning tokens counted in Grok's cost. We'll update this post when it's done.