How assert(llm) decides what counts as correct

1. Datasets

Five sets ship with the tool. Every item carries an expected answer written before any model saw it, and the type of that expected answer decides which grader runs. All five are at version 1.0.

You can also write or paste your own prompts in the browser. They are scored by exactly the same rules, and they never leave the tab.

Item fields

• expected — the answer. Its shape picks the grader when type is absent.
• fn, arity — required function name and parameter count, for code items.
• require — substrings the answer must contain, for example a SQL clause.
• forbid — constructs that fail the item, for example sorted( when the task is to implement the sort.
• aliases — other spellings accepted for a short answer.

Changelog

• 2026-10 · v1.0 — five sets frozen at the counts above.
• 2026-10 — hard set added: five problems, each answer checked twice independently.
• 2026-10 — sql set added with the require mechanism for clause-level checks.
• 2026-10 — coding grew to 20 items and gained a global rule forbidding LLM SDK imports.

A change to any item is a change to the set version. Results from different versions are not comparable, which is why the version is shown on every run.

2. Scoring

Each answer is scored on its own, by a rule that does not involve a model. The same reply always produces the same score. An item scores 1 or 0; the pass threshold is 0.7, so with binary graders a pass means the rule matched.

Multiple choice (mcq)

The grader pulls the letters out of the reply and compares them as a set with the expected letters. It reads an answer whether the model writes b, B) or a sentence ending in the letter. Two rules earn their keep here: a trailing article — “the answer is a” — is not read as option a, and a letter must stand on a word boundary, so a refusal containing the word “can” does not count as choosing c.

Code (code)

The grader checks the shape of the answer, not its runtime. Nothing is executed, in your browser or anywhere else:

1. the required function name and parameter count are present;
2. the body returns or yields something;
3. every require string appears;
4. no forbid construct appears, and no globally forbidden one either.

The global rule is a single pattern: an answer that imports anthropic, openai, groq or google.generativeai fails. It exists because a model asked for a CSV parser answered by importing an SDK and delegating the parsing to another model — an answer that passed every item-level check while doing none of the work.

Short answer (short)

The expected phrase has to appear inside the reply, on word boundaries, after both sides are normalised: Unicode folded to a canonical form, case dropped, whitespace collapsed, and digit grouping removed so that 299 792, 299,792 and 299792 are the same number. Listed aliases are accepted too.

3. Confidence intervals

A run here is 15 to 50 questions. On that few, a single percentage is close to meaningless on its own, so every accuracy is shown with a Wilson 95% interval beside it.

        p̂ + z²/2n                z         ⎧ p̂(1-p̂)   z²  ⎫
centre = ───────────   margin = ───────  √ ⎨ ─────── + ──── ⎬
         1 + z²/n               1 + z²/n   ⎩    n      4n²  ⎭

p̂ = correct / n      n = scored answers      z = 1.96  (95%, two-sided)

Why Wilson and not the normal approximation

The textbook interval, p̂ ± z·√(p̂(1-p̂)/n), assumes a large sample and a proportion away from the ends. A 15-question run breaks both. At 14 out of 15 it puts the upper bound above 100%, and at 15 out of 15 it collapses to zero width — claiming certainty from fifteen answers. Wilson stays inside 0–100% and stays wide when the sample is small, which is the honest answer.

The last two rows are the argument for longer runs. The accuracy is identical; only the interval changes.

4. Statistical ties

A model card carries a ≈ statistical tie flag when its interval overlaps the leader's. The flag is on the challenger, never on the leader, and it never appears when only one model ran.

What it means: on this many questions, the difference between the two could be sampling noise. It is not a claim that the models are equally good — only that this run cannot separate them. The remedy is more questions, not a different reading of the same ones.

5. LLM-as-judge

A judge is optional. When you pick one, it reads every answer and returns a score from 0 to 5 with a one-line reason. It is advisory: the judge never changes pass or fail. The verdict always comes from the deterministic grader, and the judge's score sits next to it.

The prompt

You are evaluating an LLM's answer. Score it 0-5 and explain in one sentence.

Question: {prompt}
Expected answer: {expected}
Model's answer: {actual}

Respond in this exact format:
SCORE: [0-5]
REASON: [one sentence]

Known biases, and what is done about them

• Self-enhancement. A model tends to rate its own output higher. A model picked as a test subject cannot also be the judge — the tool refuses the combination.
• Length preference. Judges reward longer answers. Because the judge cannot change pass or fail, this bias costs you nothing on the verdict; it only shows up in the advisory average.
• Refusal to score. A judge sometimes declines or answers off-format. That answer is recorded with no score instead of being treated as a zero, so a parsing failure does not look like a bad answer.
• Disagreement between judges. You can pick several. Each is averaged separately and shown separately, so two judges that disagree stay visible as two numbers rather than being blended into one.

6. Cost and tokens

The tool reports tokens in and tokens out, summed from the usage figures each provider returns with its response. It does not convert those into money, and no screen shows a price.

The reason is that a number in dollars would be a guess. Prices differ per provider, per model and per tier; they change without notice; and a run through an OpenRouter key is billed differently from the same model called directly. Reporting tokens means reporting what was actually measured.

7. Limitations and known issues

Reasons to distrust a number from this tool, in roughly the order they matter.

Contamination

These questions are public. Public questions end up in training data, and a model that has seen an item is not being tested on it, it is being asked to recall it. A held-out set — questions never published, used only for the ranking — is the standard defence, and this tool does not have one yet. Treat high scores on the public sets as an upper bound.

Sample size

Fifteen to fifty questions is a small sample, and the intervals in section 3 show how small. Most differences between frontier models on these sets will come out as a tie. That is the honest result, not a defect in the measurement.

The ISTQB set is not official

The items are written against Foundation-level syllabus topics in the style of the exam. They are not the official ISTQB sample exam, are not endorsed by ISTQB, and should not be used as exam preparation. The name describes the subject matter, not the provenance.

Graders check shape, not behaviour

Code answers are never executed. A function with the right name, the right parameter count, a return statement and no forbidden construct passes — even if the logic inside is wrong. This catches the failure modes worth catching at this scale and misses everything else; a passing code item means “plausibly shaped”, not “correct”.

Short answers can match by accident

Phrase matching inside a long reply will occasionally reward an answer that contains the right words for the wrong reason. The mitigation is keeping expected answers to a key term, which narrows but does not close the gap.

One run is one sample

Models are non-deterministic. The same questions, the same model and the same settings can give a different score an hour later. Nothing here is averaged across runs.

The judge is uncalibrated

See section 5. Its biases are known and unmeasured on these sets.