← Back to blog

    My anger detector had never seen a hospital review. It got 20 of 21 right.

    This morning I sat 21 hospital and clinic reviews that my anger model had flagged as angry. The model was trained on software reviews only. It had never seen a hospital review. I marked 20 of the 21 as anger. The one it got wrong was a calm account of waiting five hours and getting no medicine. Neglect, with no heat in the words. The model heard heat that was not there.

    That is the first time a feeling other than joy has cleared my bar on a domain the model was never trained on. The bar is 17 of 21 yes on a blind read, mine, with nothing on the page about what the model believed.

    What cleared and what did not

    Joy holds seven domains. Anger now holds one, health. The same anger model read 21 finance reviews the same way this morning and I marked 15. That is a miss. Interest has seven domains waiting for me to sit. Desire has its first deck. Five feelings, affection, sadness, fear, disgust and surprise, have no model yet.

    The five dark feelings were not waiting in a queue. They were refusing.

    Yesterday's handover said the paid reads for those five feelings were stuck behind other jobs in a queue that runs one job at a time. I repeated that to myself this morning. Then an agent read the code. Fifteen queue rows had stopped at the first step, a small teacher exam, because that step carried a hard cap of 20 dollars against a ledger already at 72. The cap dated from when the ledger stood at zero, and nobody raised it. A second bug sat behind the first. The paid read never wrote its own output file, so even a read that finished would have left the chain unable to continue. Both are fixed. Waiting on the queue would never have found either.

    The lesson I wrote down: a queue that is not moving is not always blocked. Read the log of the row at the head of it before explaining the delay.

    One job at a time is the law on my desk, so the reads got their own lane

    The job runner on my machine runs one job at a time because two jobs at once crashed it twice in July. The paid reads do not use the machine at all. They are network calls to a model in a data centre. So the reads now run in a second lane beside the runner, one read at a time, with their own budget. I gave it 72 dollars. That buys 16 domain reads at 5.24 dollars per 100,000 reviews. The first read, 56,922 reviews of places in Texas, ran at about 540 reviews a minute.

    What a miss sets in motion now

    Until today a miss on any domain except software led nowhere. The retrain path only existed for software. Now a miss on any domain rebuilds one training set per feeling from every review I have judged for that feeling, across all domains, and trains one new model. Each new model gets its own versioned folder, so the old one stays as evidence. Then every domain gets a fresh deck from the new model, numbered so the old deck and my verdicts on it stay in the record.

    The first of these is training right now: anger, 3,522 training rows, 523 of them reviews I judged myself, on a rented graphics card at 1.19 dollars an hour with a 4 dollar cap.

    The rented box took an old address

    That box lost twenty minutes and about 40 cents before it started. The rental company gave it an IP address that an earlier rental had used. My laptop still remembered the old machine's key for that address, so it refused every connection as a possible attack. Correct behaviour from the security side, and a stall from mine. The fix was to delete three lines from a file. It took two relaunches because I skipped that step the first time.

    The board I sit from is now a kanban

    New on the left: first decks for a domain, the ones that can turn a cell green. Then v1 and v2 for the second and third attempt after a miss. Then recall decks, then adjudications, then everything else. One card per domain, always the newest deck. A deck I clear stays green for the day and drops off tomorrow. I asked for it at noon. An agent had it built by two.

    Limits

    One judge, me, and one clear on one domain. Twenty of 21 on a blind read is above my bar and still a single deck of 21. The next anger deck on health will be drawn from the retrained model and may not hold. The finance miss at 15 of 21 is unexplained until I sit its adjudication. The five dark feelings get models only once their reads finish, roughly a day and a half of reading at the current pace. Money is a real limit. The account held 98 dollars this morning, split 72 to reads and 16 to boxes, and that covers 16 of the 34 reads queued.

    Next

    Sit the twelve open decks. Watch the anger retrain land and see whether health holds and finance moves. Let the reads lane work through the dark feelings, places first, since those are the shortest pools.