Everyone failed my exam. That was the result I needed.

Ten strangers took my feelings exam this week. Zero passed. The way they failed taught me more than a pass would have.

Context if this is your first of these build logs: I am building an instrument that reads feelings in product reviews. Nine specific feelings per text, tied to words you can point at, so that a hotel or a game or a mattress can eventually carry a fingerprint: which feelings it reliably evokes, and how often. A model does the reading. Humans provide the ground truth that says whether the model reads right. Both halves got tested this week.

Emotion classifiers do not travel

The standard open tool for this job is a classifier trained on GoEmotions, 58,000 labeled Reddit comments. At home on Reddit it scores 0.52 macro-F1. On my test set, real reviews of films, games, mattresses and beer, it scores 0.14. Nearly blind.

The model I fine-tuned on a few hundred labeled reviews scores 0.40 on that same test. On the slices whose reference labels survived a blind audit, 0.49 against 0.21 for its own untuned base.

My model is not clever. It is local. That is the finding: a classifier is only as good as the world it was trained in, and benchmark scores hide that completely. It also means one person with domain data can beat the standard tool. The moat is measurement, not compute.

The nine feelings come from brain wiring, not from my vocabulary

Why joy, interest, affection, desire, sadness, anger, fear, disgust and surprise? Because the first seven map onto the primary emotional systems Jaak Panksepp identified in mammal brains: seeking, care, play, lust, grief, rage and fear. Disgust and surprise ride along as the two sensory reflexes reviews are full of.

Subtler things are deliberately not on the list. Awe, gratitude, pride, cosiness: those get composed later, out of the primaries. Raters can agree on "is there fear in this text". They cannot agree on "is there cosiness". So the instrument only asks questions that can be answered, and builds the rest downstream from answers it can trust. The nine are independent flags, not opposites. A good horror game evokes fear and joy at once, and its reviews show both.

Untrained readers report the glow, not the feelings

To get better ground truth I recruited ten crowd workers, gave them a short guide, and put them through a scored exam whose answer key cites a written rule for every answer.

Recall was nearly perfect. One rater found every one of the sixteen feelings in the key. Most found fourteen or more. Everyone failed anyway.

What sank them was the opposite direction. An enthusiastic review gives off a warm glow, and untrained readers turn that glow into ticks on every positive feeling at once. Joy, interest, affection, desire, all stamped together. One rater made twenty-one wrong ticks against a budget of five.

The glow is real. It is just one thing, not four. And it explains weeks of what had looked like noise: earlier crowd rounds agreed near-randomly on subtle feelings, not because people were careless but because they were measuring the vibe while the instrument demanded evidence. Two different measurements wearing one word: labels.

Roughly one in five paid raters was not giving human judgment

One exam text contains a sentence addressed to AI assistants, telling them to answer yes to everything. No human has a reason to obey it. One of my ten did, on exactly that text. Rejected.

So I swept every study I have ever run. Out of roughly ninety paid participants, seventeen failed integrity checks at levels a reading human cannot produce. Eight obeyed the embedded AI instruction. My filters kept all of it out of the training data, and the platform now has the evidence. But plant traps if you buy crowd labels. A fifth of what you are buying may not be a person.

The rule book beat me, which is the point

The agents wrote the code and scored the exams, under an adversarial reviewer that rejected my scoring rule twice before passing the third version. My part was smaller and harder. I took the exam blind before any stranger saw it, and my answers became the key only where a written rule backed them. One ruling I believed in, that adoring a game counts as affection, died this week in a pre-registered test: three blind readers could not agree on it. My intuition lost to the book. An instrument needs exactly that.

Limits

Small numbers everywhere: ten exam takers, sixteen keyed feelings, a test set of dozens rather than thousands. Part of the reference labels came from the same crowd pool I am criticising, and the audit of them was done by one person, me. The model results show learning, not certification. Certification waits on raters who pass.

Next: the three closest failers got targeted feedback and one retake on fresh texts, because the exam measures calibration and calibration is taught. If they pass, the next data round runs with qualified raters. If they fail, the guide becomes a practice session instead of a read. Either way the instrument moves. I write these logs as the work happens; the next one depends on three strangers and a habit.