VerdictOpen the solver

Open release

Here are the receipts.

The dataset, the scripts that built it, the trained weights, and the result of the run. Where every row came from and how it was checked. Four commands you can run yourself to confirm the counts. And the training result, which was a regression, reported with the reason rather than buried.

What is inside

17,000 rows, and where each one came from.

Six columns, all populated, zero missing values anywhere. Two of them do the real work. source names the corpus a row came from. verify_level says how that row was checked, or admits that it was not. Adaption ingested 17,586 rows from this corpus for the training run.

  • rows published0
  • columns, 0 missing0
  • independently verified0.0 percent
  • named source corpora0

column shape, as published

problem str the problem statement worked_solution str step by step solution source str the upstream corpus, on all 17,000 rows domain str "math" | "code" | "general" difficulty str "hard" | "medium" | "-" verify_level str "independent" | "executed" | "-"

Provenance, by source

17,000 of 17,000 attributed

Seven public corpora, each named in the rows it produced. The licences are what the upstream publishers state. Check them yourself before you rely on this table, because upstream terms change and mine is a snapshot.

SourceRowsSharePublisher licenceDomain
OpenMathReasoning4,38025.8%CC-BY-4.0math
OpenCodeInstruct3,77022.2%CC-BY-4.0code
OpenMathInstruct-23,24819.1%CC-BY-4.0math
Tulu-33,10818.3%ODC-BYgeneral
OpenR1-Math1,4768.7%Apache-2.0math
Dolly-15k6323.7%CC-BY-SA-3.0general
code_contests3862.3%Apache-2.0 / CC-BY-4.0code
Total17,000100.0%CC-BY-SA-4.0 aggregateall

Verification, by level

13,260 of 17,000 verified

13,260 of 17,000 rows carry independent verification. That is 78.0 percent. The other 3,740 are general rows and they carry none. I could have called the whole file verified and nobody would have checked. Owning the gap is what makes the 78 percent worth anything.

  • verify_level: independent9,104 rows

    Math. The answer is checked against an independent gold standard three ways: exact, numeric, and symbolic with SymPy. All three must agree.

  • verify_level: executed4,156 rows

    Code. The solution is run against its unit tests in a sandboxed subprocess. Every test must pass or the row is dropped.

  • verify_level: -3,740 rows

    General. Not verified. This is anti forgetting ballast, kept so a fine tune does not lose broad instruction following. It is labelled in the data so you can filter it out.

Check it yourself

Do not take my word for it. Run these.

Four calls. No account, no key, no login. Each one settles a number I have put on this page, and the output you should get back is printed next to it. If I have something wrong, one of these catches me.

  1. 01

    The dataset has 17,000 rows and 6 columns.

    Hugging Face computes this from the parquet file itself. It is not a number I typed into a card.

    runnable

    the command

    curl -s "https://datasets-server.huggingface.co/size\ ?dataset=manifesta/verified-math-code-17k" \ | python -c "import sys,json; s=json.load(sys.stdin)['size']['configs'][0]; \ print(s['num_rows'], s['num_columns'])"

    what it returns

    17000 6

    If the published file ever stops matching the row count I state, this command says so before I do.

  2. 02

    13,260 of 17,000 rows carry verification, and 0 values are missing.

    The same service counts every distinct value in every string column, source and verify_level included.

    runnable

    the command

    curl -s "https://datasets-server.huggingface.co/statistics\ ?dataset=manifesta/verified-math-code-17k&config=default&split=train"

    what it returns

    verify_level independent 9104 · executed 4156 · "-" 3740 source OpenMathReasoning 4380 · OpenCodeInstruct 3770 OpenMathInstruct-2 3248 · Tulu-3 3108 OpenR1-Math 1476 · Dolly-15k 632 code_contests 386 nan_count 0 on all six columns

    9,104 plus 4,156 is 13,260 of 17,000, which is 78.0 percent. The other 3,740 are general rows that carry no verification. They are marked in the data, not hidden in it.

  3. 03

    The weights exist and download.

    An empty weights repo looks identical to a full one from the outside. Ask the CDN for the file size.

    runnable

    the command

    curl -sIL https://huggingface.co/manifesta\ /adaption_verified_math_code_instruct\ /resolve/main/adapter_model.safetensors \ | grep -i "^HTTP\|^content-length"

    what it returns

    HTTP/1.1 302 Found HTTP/1.1 200 OK content-length: 43856576

    43,856,576 bytes of LoRA adapter. trainer_state.json sits beside it in the same repo, and that is where every training number below comes from.

  4. 04

    The declared base model resolves.

    adapter_config.json names a base model. That name has to be a repo that exists, or nothing can load the adapter at all.

    runnable

    the command

    curl -s -o /dev/null -w "%{http_code}\n" \ https://huggingface.co/api/models/google/gemma-4-31B-it

    what it returns

    200

    adapter_config.json says base_model_name_or_path is google/gemma-4-31B-it. The model card says the same thing. The Hub returns 200. All three agree.

Rather click than type. The raw endpoints: size, statistics, adapter_config.json, and the base model.

The training result

The adapted model lost.

AutoScientist scored it against the base, head to head. Adapted 46. Base 54. It went backwards by 8 points and there is no reading of that which makes it a win. So here is what happened, from trainer_state.json inside the published weights.

adapted46regression
against
base, gemma-4-31B-it54stronger

why it went backwards

  1. 01

    The run was unstable.

    29 of the 59 optimizer steps came in over the clip threshold of 2. The median norm was 1.97 and the peak was 677.21. Clipping at 2 when the true norm is 677 keeps the direction and throws the magnitude away, so on half the steps the optimizer stopped following the loss surface and started taking equal length strides in noisy directions.

  2. 02

    59 steps is a nudge, not a fine tune.

    At that step count the weights barely leave where they started. Most of what the eval scored was still the base model, plus a half applied update.

  3. 03

    The base was already good at this.

    gemma-4-31B-it is a 31B instruction tuned model. Boxed math answers and Python that passes its tests are exactly what a frontier instruction tune spends its effort on. I picked a domain where the base had almost nothing left to learn.

gradient norms, from the run state

max_grad_norm 2 optimizer steps 59 peak gradient norm 677.21 (step 38) median gradient norm 1.97 steps above 1.0 39 of 59 (66%) steps above the clip 29 of 59 (49%) # a stable run on the same platform, same threshold peak gradient norm 0.72 steps above the clip 0 of 34

The second block is a chart QA run I did on the same platform, same clip threshold. It peaked at 0.72 and never clipped once in 34 steps. That is what a stable run looks like. Put the two side by side and the maths run is not close.

The data quality pass, separately

not the headline

Adaptive Data scores the corpus itself, before and after its own pass. That measures the data, not the trained model, which is why it sits down here and not at the top. The corpus went in at 8.0 out of 10, the highest starting point of anything I built for this, and came out at 8.2. Very little room to move when you start that high.

MetricBeforeAfter
Quality score8.08.2
GradeBB
Percentile15.317.8

What the run actually taught me

Across every run I did for this, the win rate tracked how weak the base model was in the target domain. Not how big the dataset was, not how clean. Data quality sets a ceiling on what a fine tune can teach. It does not create headroom that was never there.

I would rather publish this than a number I cannot account for. Every figure above is one download away from anyone who wants to check it. Read the full model card.

Reproduce

The dataset is the output of a pipeline, and the pipeline ships.

Nine scripts, published in the dataset repo under build/. This is not a hand curated pile with a story attached to it. Run them in this order and you get the file.

All nine, the audit evidence, and a verifier that rechecks every number on this page against the live published files, are in one public repo: A1VARA5/verified-math-code-17k. verify.py is standard library only, so there is no install step and no account. It runs 23 checks, and the same 23 run on a daily schedule, which means if anything on this page stops being true the badge goes red without me touching it.

$ git clone https://github.com/A1VARA5/verified-math-code-17k $ cd verified-math-code-17k $ python verify.py ... 23 passed, 0 failed
the order they run in9 scripts, all published
$ python build/expand_pools.py # candidate pools, per domain $ python build/build_math.py # math candidates + gold answers $ python build/build_code.py # code candidates + unit tests $ python build/build_general.py # the 3,740 row general slice $ python build/verify_core.py # math: exact + numeric + symbolic $ python build/verify_code.py # code: execute the tests $ python build/harden.py # contract, dedup, decontaminate $ python build/assemble_v3.py # 17,000 rows + build_manifest.json
  1. 01

    Expand the candidate pools

    build/expand_pools.py

    Pulls candidates from the seven upstream corpora and builds a working pool per domain. Most of what enters here does not survive.

  2. 02

    Build the math candidates

    build/build_math.py

    Shapes math rows into the problem and worked_solution contract, and attaches the independent gold answer each row will be checked against.

  3. 03

    Build the code candidates

    build/build_code.py

    Shapes code rows and attaches the unit tests each solution will have to pass.

  4. 04

    Build the general slice

    build/build_general.py

    Assembles the 3,740 row anti forgetting slice. This one is deliberately not verified, and it is labelled that way in the output.

  5. 05

    Verify the math

    build/verify_core.py

    Checks every math answer against its gold standard three ways: exact, numeric, and symbolic with SymPy. All three have to agree or the row is dropped. 9,104 rows survive as verify_level independent.

  6. 06

    Execute the code

    build/verify_code.py

    Runs every code solution against its unit tests in a sandboxed subprocess. Any failing test drops the row. 4,156 rows survive as verify_level executed.

  7. 07

    Harden, dedupe, decontaminate

    build/harden.py

    Enforces the output contract, removes duplicates, and decontaminates by 13 gram overlap against the genuine held out splits of GSM8K test, MBPP test and HumanEval.

  8. 08

    Assemble and write the manifest

    build/assemble_v3.py

    Writes the final 17,000 rows and the machine readable build manifest: counts by domain, by source, by verification level and by difficulty.

  9. 09

    Shared helpers

    build/common.py

    Not a stage. The IO, normalisation and licence bookkeeping the other eight import.

then run it yourself

from datasets import load_dataset ds = load_dataset( "manifesta/verified-math-code-17k", split="train", ) len(ds) # 17000 # keep only the mechanically verified rows ds.filter(lambda r: r["verify_level"] != "-") # # 13260 # drop the share alike slice ds.filter(lambda r: r["source"] != "Dolly-15k") # # 16368

The training side is reproducible too. The blueprint used for the Adaption run ships as extras/BLUEPRINT.md, and the counts the build produced ship as extras/build_manifest.json. The run itself was done on Adaption, and the platform is documented at docs.adaptionlabs.ai.

Licence

CC-BY-SA-4.0, and 632 rows are the reason.

One slice forces it. 632 rows come from Dolly-15k, which is CC-BY-SA-3.0 and carries a share alike obligation. Share alike in one source propagates to the whole release, so the honest aggregate is CC-BY-SA-4.0, even though the other six sources are attribution only or looser.

If share alike does not work for you, one filter fixes it. Drop source == "Dolly-15k" and 16,368 rows remain, attribution only. All 632 sit in the unverified general slice, so dropping them costs you no verified data.

# attribution only subset, 16,368 rows ds = ds.filter(lambda r: r["source"] != "Dolly-15k") # all 632 removed rows were: # domain general # verify_level "-" (unverified anyway)

Check these licences against the upstream publishers before you rely on them. The table above is what those projects state today and terms change. The per source breakdown is in the dataset card and the source column carries it on every row, so you can work out the licence position of any subset you build.

7 sources named17,000 of 17,000 attributed13,260 verified3,740 unverified, labelledshare alike, from 632 rows