My plain-Swedish model scored 88 percent on its own exam and 22 percent on real text
Last Friday I started a side project: paste a Swedish authority letter, get it back in klarspråk, every change shown next to the rule that caused it. By Saturday it was live at klarspraka.se, running on a model I trained on the Mac Studio on my desk. In between, the model scored 88 percent on the exam I had built for it and 22 percent on the first real text it saw. That gap is the story.
Why plain Swedish, and why no big model
Swedish law, språklagen 2009:600, 11 §, says public bodies must write language that is "vårdat, enkelt och begripligt": cared for, simple, understandable. There is a rule book for it: Språkrådet's Myndigheternas skrivregler, Länsstyrelsen's Klarspråk guide, Isof's Klarspråkshjälpen. I turned those sources into a catalogue of 313 rules. 176 of them a program can find and fix on its own: a word, a form, a construction. 113 need a rewrite. 24 are judgment calls, which the tool shows the user as a question and never makes on its own.
The text an authority would paste is case text. So one rule stood before the first line of code: no text ever reaches a third-party model. Open weights only, on hardware I control, inside the EU. That ruled out the obvious shortcut of calling a frontier model, for the service and for making training data alike.
Training data made by breaking good text
I had no expert-written pairs and no budget for them. A set of 2,000 to 5,000 pairs from klarspråk editors would cost half a million to a million kronor. What I had was public text that already follows the rules: the agencies' own web pages. A crawler, polite at one request a second, fetched 1,935 pages from Försäkringskassan, Skatteverket, Pensionsmyndigheten, 1177, Konsumentverket and Krisinformation. The linter kept 34,274 paragraphs that broke no rule.
Then a program made them worse. It took each clean paragraph and planted one to three labelled faults: a nominalisation, a passive, a "skall", a law reference moved to the front. The spoiled version became the input and the original the target. One pair in ten was left untouched so the model would learn to leave good text alone. About 50,000 pairs, no human label and no big model anywhere.
The untuned models lost to a linter
Before training anything I ran six open models on a sealed set with the linter in front of them. Two scores: did the output keep every number, date, name, law reference and negation (checker ok), and did it match the original paragraph exactly.
| Model, untuned | Checker ok | Exact |
|---|---|---|
| EuroLLM-9B | 77% | 30% |
| Qwen2.5-7B | 75% | 15% |
| Mistral Small 3.2, 4-bit | 57% | 7% |
| Qwen3-32B, 4-bit | 58% | 3% |
| Qwen2.5-32B, 4-bit | 53% | 3% |
| Qwen3-30B-A3B, 4-bit | 48% | 3% |
None of them beat the linter alone. EuroLLM-9B, from an EU consortium under Apache 2.0, was the best base and became the one I trained.
A loop that trains, scores, and keeps or reverts
Each training run was a LoRA on the Studio with mlx-lm, rank 32, 600 iterations, 45 to 60 minutes for the 9B. A loop ran recipe after recipe against a frozen set of 155 items and kept a change only if it beat the best by more than the noise band. Exact match went from 19 percent untuned to 72 percent for the first recipe to 89 percent for the best. Two runs that looked better, double the learning rate and twice the iterations, landed inside the noise and were reverted. Seventeen recipes in all. Training on the Studio costs nothing but time, which is why the loop could afford to be picky.
88 percent on the exam, 22 percent on real text
Then the sealed exam, 1,322 items the loop had never seen: checker 99.7 percent, exact 88 percent. And 27 pairs of real text, written by people, before and after a klarspråk edit: 22 percent exact. The untuned base had scored 37 percent on those. My model returned 19 of the 27 unchanged.
The reason was in the data. The generator almost never produced the two faults real editors fix most: a passive to make active, and a law reference to move last. Of 25,244 training rows, 119 carried a passive and 11 a law-first. The model had learned, correctly, that a passive is almost never something to touch. A diagnostic set of 59 items confirmed it. The model left 97 percent of the passives and 100 percent of the law-first items as they were.
Fixing the generator and retraining took passives from 0 to 48 percent exact, 61 percent on the 4-bit copy that serves the site, with the loop score unchanged at 89. Law-first is still at zero. Eleven rows is not a lesson.
Two models, one stream
A 9B at 4 bits answers a one-page letter in about five seconds. Too slow to feel like typing. So the same recipe and data trained EuroLLM-1.7B as well, nine minutes per run. At 8 bits it holds 86.5 percent exact at 102 tokens a second and does the letter in two. The page streams the small model's rewrite from the first third of a second and lets the 9B replace it paragraph by paragraph as it finishes. Speculative decoding, the small model as a draft for the large one, measured no gain, so it is two models, not one trick.
On passives the small model is weaker: 21 percent where the 9B reaches 48. It types, the 9B decides.
What the text touches on the way
The site is a static page on a Hetzner box. The model button sends the text through an SSH tunnel to the Studio in Malmö and back. The Node server there writes the text nowhere: not to disk, not to a log. The access line is time, path, status, bytes and milliseconds. Caddy strips IP addresses and request headers from its own log. A test asserts that no input text appears in any log. If the tunnel is down, the page hides the button.
The safety net sits after the model. If a rewrite changed a number, a date, a name, a law reference or a negation, that paragraph is kept as it was. Quotes, personnummer and diarienummer are never touched and show in grey. Every change the user sees carries its rule id.
Limits
The judge is a program, not a person. "Exact match" means the model reproduced the paragraph the generator had spoiled, which proves it undoes planted faults, not that an editor would prefer its output. The human test is 27 pairs, and the model lost it. No authority has used it on real case text. The page is served by a German host and the model runs on a computer in my home, which meets my EU rule and is not a datacentre. The rule book sources are copyrighted, and the question of permission to build on them is open. Three and a half days and 62 commits, most of them with an AI agent at the keyboard, is a prototype.
Next
Enough law-first examples to move that number off zero. A second human test set that is not 27 pairs. Then a pilot with one authority, which is the only test that counts.