Method

How Verdict is built.

Curate real problems. Check every answer before it enters the data. Adapt on what survives. The recipe is simple to state and hard to fake. This page walks the whole pipeline, from raw rows to the training result, and the training result was a regression.

01The data 02Verification 03Difficulty 04Adaptation 05The result

01 · The data

17,000 rows, six columns, nothing missing.

Real problems and real solutions, curated from seven named public corpora. Each row carries a worked solution, the path to the answer rather than the answer alone. Two of the six columns are the ones that make the release checkable: source and verify_level.

Composition, exact

0 math rows, gold checked, verify_level independent
0 code rows, tests executed, verify_level executed
0 general rows, unverified, kept to prevent forgetting
0 total, with zero missing values in any column

The bar to get in

Checked math against an independent gold standard, code run against its tests
Attributed the upstream corpus is named on all 17,000 rows
Deduped zero exact duplicate prompts, zero near duplicates at MinHash Jaccard 0.6
Decontaminated 13 gram overlap against GSM8K test, MBPP test and HumanEval

one row, before it is allowed in✓ verified

{
  "problem":
    "A bag holds 5 red and 3 blue marbles.
     Two are drawn without replacement.
     What is the probability both are red?",

  "worked_solution":
    "There are C(8,2) = 28 equally likely pairs.
     Red pairs number C(5,2) = 10.
     So P(both red) = 10 / 28 = 5 / 14.
     The final answer is $\boxed{5/14}$.",

  "source":       "OpenMathReasoning",
  "domain":       "math",
  "difficulty":   "medium",
  "verify_level": "independent"
}

02 · Verification, the core

13,260 rows are checked. The other 3,740 say so.

Every math row is checked against an independent gold answer three ways: exact, numeric, and symbolic with SymPy. Every code row runs against its unit tests in a sandboxed subprocess. A wrong answer never becomes training signal. The 3,740 general rows carry no check at all, and they say so in the data. They carry verify_level: "-", so you can drop them in one line.

math · sympy equivalence✓ kept

The answer is parsed and compared symbolically, so different forms of the same value match. 5/14 and 10/28 are equal, a string check would miss that.

# answer matched, not string compared
from sympy import simplify, sympify

candidate = sympify("5/14")
gold      = sympify("0.357142857...")

simplify(candidate - gold) == 0
#  →  True   keep this row

# "10/28" and "5/14" are also equal:
simplify(sympify("10/28") - candidate) == 0
#  →  True

code · running asserts✓ kept

The solution runs against its tests. If a single assert fails, the row is dropped. Code that does not work does not teach the model to write code that works.

# solution is executed against real tests
def two_sum(nums, target):
    seen = {}
    for i, n in enumerate(nums):
        if target - n in seen:
            return [seen[target - n], i]
        seen[n] = i

assert two_sum([2, 7, 11, 15], 9) == [0, 1]
assert two_sum([3, 2, 4], 6)     == [1, 2]
assert two_sum([3, 3], 6)        == [0, 1]
#  all pass  →  keep this row

The filter, in action

Curation time rejection sampling. The filter runs once, when the dataset is built, not at inference. Wrong answers and broken code are dropped on sight. What reaches training has already been proven correct by a tool, SymPy for math and a real interpreter for code. The counts below are from the published file, and the Hugging Face statistics endpoint will confirm them without taking my word for it.

✓ kept✗ dropped

$ python build/verify_core.py    # math
$ python build/verify_code.py    # code

verify_level in the shipped file:

  independent   9,104   math, gold checked three ways
  executed      4,156   code, unit tests run and passed
  "-"           3,740   general, not verified, labelled

  verified     13,260 of 17,000   (78.0%)

# a large majority of scanned candidates never
# made it this far. that is the cost of the rule.

Why this is the defensibility backbone

A claim about a dataset is worth whatever its file can prove. 13,260 rows carry a verification level in the data itself. 17,000 carry a named source. Both are readable with one curl call. The claim and the proof travel together, so nothing here rests on trust.

03 · Difficulty

There is no easy band. Nothing easy made the set.

A verified set can still be too easy. All warm ups, and the model learns to look right without learning to think. The difficulty column in the published file has three values and easy is not one of them. Median row length is 1,514 characters and the 90th percentile is 24,974, and that long tail is genuine multi step competition reasoning rather than padding.

the difficulty column, counted over 17,000 rows0 hard

medium

hard

unlabelled, "-"

Not an illustration. These are the values in the shipped file. The 3,740 unlabelled rows are the general slice. No difficulty grade, because there is no verification on them either.

04 · Adaptation

Adapt on what survived, with Adaptive Data.

The verified dataset was adapted with Adaptive Data by Adaption. 17,586 rows were ingested from this corpus. The run maps problem to the prompt and worked_solution to the completion, keeps the length detailed, and leaves prompt rephrasing off because rephrasing would invalidate answers that were checked against the exact wording.

1
Column mapping map
prompt is problem, completion is worked_solution. Those are the real column names in the published file, so the mapping is copy and paste.
2
Prompt rephrasing stays off off
The verification is tied to the exact wording of the problem. Rephrase the prompt and the checked answer may no longer be the answer, so the release tells downstream users to leave that recipe off.
3
Deduplication dedup
Near duplicate rows are dropped so the model does not over weight repeated patterns. The shipped file already reports zero exact duplicates and zero near duplicates at Jaccard 0.6.
4
Detailed length length
brand_controls length is set to detailed, preserving the worked steps rather than collapsing to short answers.

datasets.runrun complete

# dataset id 6ad1a83f-b806-4a20-ba6d-239029d30711
# 17,586 rows ingested from the 17,000 row corpus

client.datasets.run(
    dataset_id = "6ad1a83f-b806-4a20-ba6d-239029d30711",
    columns = {
        "prompt":     "problem",          # the question
        "completion": "worked_solution",  # the checked solution
    },
    options = {
        "deduplication":    True,   # drop near duplicate rows
        "prompt_rephrase":  False,  # rephrasing breaks the answers
    },
    brand_controls = {
        "length": "detailed",       # keep the worked steps
    },
)

05 · The result

The adapted model scored 46. The base scored 54.

It went backwards by 8 points. That is the number the run produced and it goes here as it came out. The loss fell the whole way, so the model was learning something. It still lost the head to head, so what it learned did not beat what the base already knew.

the run, as reported

AutoScientist, head to head, blind

  adapted   46
  base      54          gemma-4-31B-it
  delta     -8          a regression

trainer_state.json, from the published weights

  optimizer steps        59
  max_grad_norm          2
  peak gradient norm     677.21   (step 38)
  median gradient norm   1.97
  steps above 1.0        39 of 59
  steps above the clip   29 of 59

  train loss   3.482  ->  1.238
  eval  loss   2.366  ->  1.220

Adapted

lost the head to head

Base

gemma-4-31B-it, already strong here

regression, reported

Two things compound. 29 of 59 optimizer steps came in over the clip threshold of 2, peaking at 677.21 against a median of 1.97, so on half the run the optimizer took fixed size strides in noisy directions. And a 31B instruction tuned base is already good at boxed math answers and Python that passes its tests. I left myself almost no headroom. Every figure above is in trainer_state.json, inside the published weights.

See verification live →Read the full result