Can a free 14B model replace a frontier model? A €0 bakeoff

I was about to buy a computer.

The scoring pipeline behind Got Cosy runs on a frontier model, paid per call, and the bill grows with every new hotel and every new vertical. A local 14B model, running free on a laptop, looked like a way to cut that cost close to zero. Before spending money on hardware to run it faster, I wanted one number: how close does the free model get to the one already in production.

The setup

I took 50 hotels already scored in production. Each has four dimensions: warmth, intimacy, character, service. I fed the same review text to qwen3:14b, running locally through Ollama, and asked it to score the same four dimensions the same way.

Then I compared the two sets of scores with Spearman correlation, which asks a narrower question than "are the numbers close": does the model rank hotels in the same order as production. For a public score, rank order is most of what matters. Our bar for a dimension to pass is 0.8.

Round one: it failed

First run, thin input, whatever reviews happened to be in the existing record:

Dimension	Spearman vs production
warmth	0.68
intimacy	0.58
character	0.81
service	0.84

Two of four dimensions cleared 0.8. Two didn't, and intimacy was a clear miss. On this evidence the free model doesn't replace the frontier one. I wrote that down and almost stopped there.

The question I asked next

The four scores that failed and the four that passed didn't split randomly. Intimacy and warmth are the two dimensions where a model needs more to go on: they show up in throwaway lines buried across many reviews, not in one obvious paragraph. Character and service, which passed, tend to get stated more directly. That looked less like a model limit and more like an input limit.

So I reran the comparison, this time feeding each hotel 30 to 40 reviews instead of whatever thin sample was on hand, and I swapped the comparison point: instead of production's own historic scores, I used a frontier model as a fresh referee on the same deep input, so both sides were graded against the same yardstick.

Round two: it passed, and something else showed up

Dimension	Spearman vs referee	Average offset
warmth	0.84	+17.2
intimacy	0.82	+17.2
character	0.86	+13.0
service	0.84	+12.3

All four dimensions cleared 0.8 against a frontier referee, n=50. And the free model wasn't randomly wrong when it disagreed, it was consistently generous, scoring 12 to 17 points higher than the referee on every single dimension. That's not noise. A constant offset is the easiest error in the world to fix: subtract it. Once you calibrate that away, the free model's ranking and the frontier model's ranking say close to the same thing.

What actually mattered

Not the model. The input.

The 14B model given a thin slice of reviews looked like a worse model. The same 14B model given a proper slice of reviews looked like a model that just needed calibrating. I'd been about to solve a hardware problem that was actually a data problem.

Caveats, stated plainly: n=50, one domain (hotel reviews), one model family. This is not a general claim that small models match frontier ones. It's a specific, checkable result for one scoring task, and it's the reason a MacBook Air is currently doing scoring work I was ready to buy new hardware for.

What I'd tell you to steal from this: before you conclude a smaller model isn't good enough, check whether you gave it enough to work with. Then check if it's wrong in a constant direction. Consistent bias is nearly free to fix; a genuine ranking failure isn't.