Why it wins
Why this entry holds up.
In math and code the answer is right or wrong. The program runs or it does not. So every claim on this site is one a stranger can check rather than one they take on faith. Including the claim I would rather not make. The adapted model lost its head to head, 46 against 54, and the reason is written down next to the number.
The core argument
Math and code are checkable. So the win is provable.
A symbolic engine decides whether a math answer is correct. A real test suite decides whether a program works. No opinion in the loop. Anyone who runs the same check lands on the same result, which makes the improvement a fact, not a feeling.
The score comes from a deterministic check. Run it again and you get the same number. The claim travels with the proof.
- ✓A correct answer is decided by sympy, not by a grader's taste.
- ✓A working program is decided by tests that execute and pass.
- ✓Anyone can rerun the eval and land on the same result.
The score comes from a judgement about how the output reads. Two careful judges can disagree, and a rerun can drift.
- ~Quality is rated by preference, which moves with the rater.
- ~The same output can score differently on a second pass.
- ~The claim is hard to hand to a stranger and have them confirm.
When the metric is a check, not a vote, a better number is a better model, and you can prove it on demand.
Verified, and the exact share
13,260 of 17,000. Not all of them, and I say which.
The same checks that make the category provable are the ones that built the dataset. 78.0 percent of the published rows carry mechanical verification. The rest do not, and the data says which is which.
9,104 math rows, gold checked
Each math row is checked against an independent gold answer three ways: exact, numeric, and symbolic with SymPy. All three have to agree. An algebraically equal form still counts, a wrong one drops.
4,156 code rows, executed
Each code row runs against a real unit test suite in a sandboxed subprocess. It is kept only if every test passes, so a green check means the program ran, not that it looked right.
3,740 rows carry no check, and say so
The general slice is unverified ballast, kept so a fine tune does not lose broad instruction following. It carries verify_level "-" in the data, so anyone can drop it. Counting it as verified would have been the easy lie.
What checkable looks like
The model returns 10/28. The reference is 5/14. A string match would reject it. The symbolic check sees they are the same number and keeps the row. The decision is a fact, reproducible by anyone who runs the line.
Decontamination, and a mistake I caught
My build originally decontaminated against files that turned out to be the train splits of GSM8K and MBPP, which means the real benchmarks had never been checked at all. I found that and fixed it. The shipped dataset is decontaminated by 13 gram overlap against the genuine held out sets below. Two contaminated rows turned up and were removed, both GSM8K train rows that overlap GSM8K test, which is an artefact of GSM8K rather than of this pipeline. MBPP and HumanEval were only ever decontamination targets, so no row comes from either.
- GSM8Ktest1,319 rows
- MBPPtest, task_id 11 to 510500 rows
- HumanEvalfull164 rows
Near duplicates by MinHash at Jaccard 0.6 or above: 0. Exact duplicate prompts: 0. Empty fields: 0.
Open and reproducible
Every claim has an artifact behind it.
- 01The dataset ships open, with its provenance in the data. 17,000 rows on Hugging Face and Kaggle under CC-BY-SA-4.0. The source column names the upstream corpus on every row, all 17,000 of them, across 7 public corpora.
- 02The build pipeline ships with it. Nine scripts under build/ in the dataset repo: pool expansion, three domain builders, two verifiers, hardening and assembly. The file is the output of code you can read.
- 03The weights ship, and the base they need resolves. The LoRA adapter is 43,856,576 bytes on Hugging Face and mirrored on Kaggle. adapter_config.json names google/gemma-4-31B-it, the card names the same, and the Hub returns 200 for it.
- 04The result ships, including the losing part. The adapted model scored 46 against the base model's 54. That is in the model card, in the machine readable metadata, and on this site.
Released open
✓ liveThe honest line
The adapted model scored 46 against the base model's 54. It lost.
Saying so is the entire point. 29 of 59 optimizer steps came in over the clip threshold of 2, peaking at 677.21 against a median of 1.97, so on half the run the optimizer took fixed size strides in noisy directions. A run I did on the same platform peaked at 0.72 and never clipped once. On top of that, a 31B instruction tuned base is already strong at boxed math answers and Python that passes its tests, so I left myself almost no headroom. Data quality sets a ceiling on what a fine tune can teach. It does not create headroom that was never there.
Every figure in that paragraph is in trainer_state.json inside the published weights, one download from anyone who wants to check it. I would rather publish this than a number I cannot account for.
See it for yourself.
Run a problem through the solver and watch the check decide, or read the method to see how every answer is verified before it counts.