A frontier model aced my exam, then lost the real task to my small one
Two things happened within hours of each other. A frontier language model sat the qualification exam I built for human annotators and passed it near-perfectly: sixteen of sixteen feelings found, half a penalty point, and it ignored the hidden instruction that catches AI cheats. Then the same model labeled my real test set and scored 0.209, while my small fine-tuned local model scored 0.485. Both results are correct. Explaining how is the most useful thing this project has produced so far.
Three tests, one answer: model size is not the lever
This is now a three-way replication. In July I benchmarked seven models against my own labels: a strong commercial frontier model scored 39, my local 14B model scored 43. A week later I ran a size ladder from 14B to 235B parameters specifically to test whether scale helps: the biggest model exactly tied the smallest. And now, post fine-tuning, the local model beats a frontier model 0.485 to 0.209 on the audited test slices. Whatever makes a model good at reading feelings in mattress reviews, it is not parameter count. It is calibration to the domain, which a few hundred labeled examples buy and scale does not.
Passing the exam is not the same skill as matching the labels
The dissociation is the real finding. The exam rewards disciplined restraint: tick only what the written rules license, against an answer key where every answer cites a rule. The frontier model is superb at that. But the test set is labeled by people, and people tick more freely: my gold contains 107 feeling-ticks across those 48 texts, my fine-tuned model produced 106, and the frontier model produced 20. Reading the rules strictly, it went nearly silent, and scored zero on affection, sadness and disgust by never ticking them at all.
The mirror image happened with humans the same week. Every human who sat the exam failed it in the opposite direction, ticking warm feelings everywhere the text merely glowed. Humans over-tick against the rules; a rule-following machine under-ticks against humans. The exam and the benchmark measure two different competencies, and knowing that is worth more than either score.
What this buys
It settles the build-versus-rent question with data. If a frontier model were the best labeler, this project would be a prompt. It is not: the deployable reader is the small local model, domain-calibrated, running on my own hardware at no marginal cost, and the frontier model's right job is auditor and proposer, not source of truth. It also hardens the claim I care about publishing: for subtle measurement in a specific domain, a small calibrated specialist beats a large general intelligence, shown three ways on two different gold standards.
Limits
The test labels come from crowd raters, and my own blind audit measured mild inflation even in the slices it certified, so part of the fine-tuned model's edge is that it learned the annotation style it was trained on: calibration, not comprehension, and I state it as such. The sample is 48 items. Each model sat once, with one prompt form. And the exam the frontier model passed was scored against a key my own tools helped assemble, which is why any model that ever labels production data will sit that exam under harder conditions first. The next step is not more models. It is deciding what a human must demonstrate before a human benchmark means anything, which turns out to be the hardest instrument-design question in the whole project.