A fine-tuned 2B model matched Gemini Flash-Lite at labelling contract clauses, for less than half the cost
92.4%
Gemma 2 2B, fine-tuned
90.8%
Gemini 3.5 Flash-Lite
$0.021 per 1,000 clauses against $0.047.
The short version
On 249 clauses from contracts it never saw, a Gemma 2 2B model fine-tuned to name a clause's type got 92.4% right. Gemini 3.5 Flash-Lite, given the same instruction and the same 10 types, got 90.8%. That's a tie at this sample size, not a win. Served with vLLM on one GPU, the fine-tuned model answered in 70 ms at the median and cost $0.021 per 1,000 clauses, against Gemini's $0.047. Our report's verdict is “close”, because the small model is clearly weaker on one type, exclusivity.
Why this test
Legal and procurement teams send a lot of contract text to LLMs for one narrow job: find the clauses that matter and say what kind they are, so a reviewer can jump to the liability cap or the change-of-control terms. The language is formal and repetitive, and every company has its own list of clause types it cares about. That's the kind of task where a small model trained on your own labels should hold its own.
CUAD, the Contract Understanding Atticus Dataset from The Atticus Project, is a public set of 510 commercial contracts annotated by lawyers. We took 10 of its clause types, the ones most contract reviews look for:
governing_lawanti_assignmentcap_on_liabilityaudit_rightsinsurancelicense_grantexclusivitynon_competetermination_for_conveniencechange_of_controlWhat we did
- Split CUAD by contract, not by clause: 85% of the contracts for training, 15% held out. Every test clause comes from a contract the model never saw, which is how it would be used. Clauses the lawyers tagged with two types were left out, because they have no single right answer.
- Fine-tuned
unsloth/gemma-2-2b-itwith LoRA on 3,775 labelled clauses, using SynqTune's classification template, on oneg5.xlarge(A10G) GPU in our own AWS account. On held-out text, perplexity fell from 81.05 for the base model to 2.40. - Sent 300 held-out clauses to
gemini-3.5-flash-litewith the same task instruction the fine-tuned model gets, which lists the 10 allowed types, and served the fine-tuned model with vLLM for the same 300. - Scored both with SynqTune's comparison report: accuracy with 95% bootstrap intervals, macro-F1, accuracy per type, latency, cost, and every clause where either model was wrong. Training, evaluation and the comparison took 73 minutes end to end.
Results
51 requests to Gemini failed, and those clauses were left out of both models' scores, so both are measured on the same 249 clauses. The report also found 19 held-out clauses whose exact text appears in a training contract too (standard boilerplate), and left them out so the model isn't scored on text it memorised.
| Metric | Gemma 2 2B, fine-tuned | Gemini 3.5 Flash-Lite |
|---|---|---|
| Accuracy95% interval 89.2–95.6% and 87.6–94.4% | 92.4% | 90.8% |
| Macro-F1 | 0.905 | 0.875 |
| Answers that weren't a single valid label | 0% | 0% |
| Clauses only this model got right | 11 | 7 |
| Latency, mediansee below | 70 ms | 14.3 s |
| Latency, p95 | 98 ms | 25.9 s |
| Cost per 1,000 clausesone request at a time | $0.021 | $0.047 |
Accuracy is a tie. The fine-tuned model is 1.6 points ahead, but the two intervals overlap almost entirely. The honest reading is that a 2B model trained on 3,775 clauses does this job about as well as Gemini Flash-Lite, not better. Macro-F1, which weights every type equally, leans the same way: 0.905 against 0.875.
Neither model invents labels. Every answer from both was exactly one of the 10 types.
Speed and cost
The answer is a single label, a few tokens, so the fine-tuned model is quick: 70 ms at the median and under 100 ms at p95, one request at a time. Gemini's 14.3 seconds at the median is far slower than we measured on Banking77 a few days earlier (652 ms). It was a bad day for the API: the same slowness is why 51 requests failed. Treat Gemini's latency here as what we saw, not as typical.
Cost doesn't depend on that. Gemini's $0.047 per 1,000 comes from its token use, about 130 input and 3 output tokens per clause, at $0.30 and $2.50 per million. The fine-tuned model's cost is the GPU's on-demand price times its measured time per clause, and vLLM can answer many requests at once on the same GPU:
| Requests at once | Clauses per second | p95 latency | Per 1,000 clauses |
|---|---|---|---|
| 1 | 13 | 98 ms | $0.0207 |
| 8 | 92 | 110 ms | $0.0030 |
| 32 | 242 | 153 ms | $0.0012 |
With 32 requests in flight, one A10G labels 242 clauses a second, and the cost falls to $0.0012 per 1,000, about 40 times cheaper than Gemini. That only holds when the GPU is kept busy. A contract review tool that labels a few hundred clauses an hour would leave it idle most of the time, and then paying per request is cheaper. Batch jobs, like labelling a back catalogue of contracts, are where this pays off.
Where the small model still loses
Accuracy by type, for the five types where the fine-tuned model did worst:
| Clause type | Clauses | Fine-tuned | Gemini |
|---|---|---|---|
| exclusivity | 16 | 81% | 94% |
| change_of_control | 20 | 65% | 70% |
| license_grant | 47 | 87% | 89% |
| audit_rights | 40 | 98% | 98% |
| insurance | 32 | 100% | 100% |
These are clauses Gemini got right and the fine-tuned model didn't:
| Clause | Correct type | Fine-tuned model said |
|---|---|---|
| “During the term of this Agreement, Sabco will not represent or accept as a primary, secondary, or associate sponsor any other company reasonably deemed to be competitive with Prolong products.” | exclusivity | non_compete |
| “Subject to Articles 5.1 and 5.2, the Parties acknowledge and agree that CERES shall have the exclusive right, with the right to grant sublicenses to this right, to use and commercially exploit JOINT INTELLECTUAL PROPERTY…” | license_grant | exclusivity |
| “You consent to our communication with any party we deem necessary about the Hotel in order for us to evaluate the proposed Change of Ownership Transfer.” | change_of_control | audit_rights |
| “Before expiration of the Term, either Party may terminate this Agreement upon: (i) any material breach of the Agreement by the other Party… (ii) ten (10) business days written notice… or (iii)…” | change_of_control | termination_for_convenience |
It leans on surface words. A licence that grants an “exclusive right” became exclusivity, and a promise not to sponsor competitors became non_compete. Those types do overlap in meaning, and the lawyers who labelled CUAD drew the line in a particular place. Gemini followed that line more often. Exclusivity has only 16 test clauses, so the gap is 2 clauses, but the direction matches the mistakes.
Change of control is hard for both. 65% against 70%. A change-of-control clause is often one item in a list of termination triggers, like the last example above, and the label depends on a single sub-clause the model has to find. Both models missed 12 clauses in total.
The fix is the usual one: more training examples for exclusivity and change of control, especially the ones that sit next to licence and termination language.
What went wrong on the way
- The first attempt served the model with plain Hugging Face generation. It would have measured the wrong thing: nobody serves a model in production that way. We cancelled it and relaunched with vLLM, which is now an option you choose when you launch a run.
- The relaunch on an A10G failed before training: the template asked for 16-bit floats on a GPU that wanted bfloat16. The template now picks whichever the GPU supports.
- 51 Gemini requests timed out and were dropped. Our caller retried rate limits and server errors but not timeouts. It retries those too now, so the next comparison should lose far fewer.
What's next
A rerun with more exclusivity and change-of-control examples, on more held-out clauses so the interval is narrow enough to say whether the small model is actually ahead. We'll update this post when it's done.
Labelling contracts or documents with an LLM API?
We're looking for a few teams to run this comparison on their own data, in their own AWS account, with our help. You get the report; we learn what it takes to switch.