I taught a model to predict 'would buy again' and then checked it against 1,473 real answers
Most review sites ask you to guess whether a reviewer is happy. I wanted to stop guessing and check.
Steam gives every review a ground truth. Underneath the text, each reviewer also clicks a thumb: recommended or not recommended. That thumb is the real answer. Nobody has to infer it. So Steam is a rare place to test a specific claim: can you read review text alone and correctly deduce whether the person would buy the thing again, without ever seeing their thumb.
The setup
I classified 1,473 real reviews from text alone, no thumb visible to the model, using short excerpts rather than full reviews. Then I unmasked the thumbs and scored the deductions against them.
Two numbers matter more than the rest: how often the deduction was right overall, and how often it caught the people who were actually unhappy.
The results
Accuracy came in at .905. A dumb baseline that just guesses "recommended" every time, because most Steam reviews are positive, gets .864. So the deduction beat the lazy baseline, but not by a landslide.
The number I cared about more was negative-class recall: of the real thumbs-down, how many did the text-only deduction catch. That came out to .870, 174 of 200. Most unhappy players write like unhappy players.
Precision on the negative call was weaker: .604. Out of every "this person probably wouldn't buy again" call, a chunk were wrong. In a confusion of 1,473 reviews, that was 26 false positives and 114 false negatives.
Where it breaks
The misses aren't random. They cluster around a specific pattern: reviews that complain for three paragraphs and then click recommend anyway. Steam has a whole subculture of this. "This game sent me to the hospital, would recommend." Sarcasm and grievance-with-affection read like regret to a model that's parsing tone, but the human still clicked yes. That's not a model bug so much as a mismatch between what language usually means and what one platform's culture does with it.
The two-ledger rule
This is why I never report one number. I report two, always, side by side: the explicit rate, meaning what people directly say when asked, and the inferred rate, meaning what the language across the whole set implies. The explicit subset is small and skews toward people motivated enough to answer directly, so it's a biased sample. The inferred rate covers everyone, including the people who never state an opinion outright. Neither number gets to stand in for the other.
The caveat
This is one platform's culture, tested against one label. Steam reviewers write in a specific register: sarcastic, grievance-heavy, quick to complain and still recommend. A different domain with less irony, product reviews for something plain like a mattress, might get a cleaner precision number, or might not. I don't know yet. That has to be measured fresh in each domain, not assumed from this one.
What I'd tell you to steal from this: if you're deducing intent from language, find one place where you can check your deduction against a real answer first, and report the stated number and the inferred number separately instead of blending them into one.