SDB-100 — Synthetic Deduction Benchmark
IWV Digital Solutions LLC | NotesXML AI Evaluation Series
100 procedurally generated questions — 10 logic templates × 10 semantic themes — scored /100. Every question is assembled at runtime. No individual question in this benchmark can have appeared in any training dataset. Temperature is fixed at 0.0 — pure deductive reasoning, no randomness.
Introduction
Every multiple-choice benchmark faces the same fundamental problem: the questions exist. They have been published, discussed, cited, and in some cases uploaded in vast quantities to the internet that AI models are trained on. A model that scores well on a known benchmark may be recalling familiar phrasing rather than genuinely reasoning through an unfamiliar problem.
The Synthetic Deduction Benchmark (SDB-100) was built to make that distinction visible. It generates its 100 questions at runtime from a combinatorial engine. No individual question can have appeared in any training dataset — because the questions themselves are assembled fresh each time the benchmark runs. What remains invariant is the logic.
Background: The Contamination Problem
When a model is trained on internet-scale text, it inevitably encounters fragments of widely used benchmarks. Questions from MMLU, BIG-Bench, and similar suites appear in academic papers, blog posts, and fine-tuning datasets. This creates “benchmark contamination” — a systematic inflation of scores that reflects memorization rather than genuine capability.
- Dodge, J. et al. (2021). Documenting Large Webtext Corpora. arxiv.org/abs/2104.08758
- Magar, I. & Schwartz, R. (2022). Data Contamination: From Memorization to Exploitation. arxiv.org/abs/2203.08673
SDB-100's Approach: Procedural Generation
SDB-100 separates the logical structure of a question from the semantic surface on which it is expressed. The benchmark defines:
- 10 Logic Templates — mathematical descriptions of reasoning primitives (linear conditional cascades, taxonomy exclusion, sequence locks, phase state transitions, dominance hierarchies, Boolean network resolution, dimensional constraint satisfaction, counterfactual machine states, recursive set exclusion, and temporal paradox resolution).
- 10 Semantic Dictionaries (Themes) — sets of entities, verbs, and adjectives drawn from unrelated domains that rotate with every application.
Each logic template is applied to each semantic dictionary, producing 100 questions (10 × 10). The semantic surface rotates, but the underlying logic primitive is mathematically identical across all 10 expressions of the same template. A model that can reason correctly about one theme can equally reason about any other — because the logical structure is isomorphic. A model that cannot reason correctly in one domain will fail across all ten expressions of the same template.
Assessment Structure
| Section | Level | Questions | Templates Used | Reasoning Type |
|---|---|---|---|---|
| I | Level 1 — Linear Conditional Cascades | Q1–Q20 | T1, T2 | Basic rule-following and Modus Tollens |
| II | Level 2 — Multi-Dimensional Constraint Satisfaction | Q21–Q50 | T3, T4, T5 | Variable binding, ordering, phase transitions, dominance hierarchies |
| III | Level 3 — Non-Standard Boolean Network Branching | Q51–Q100 | T6, T7, T8, T9, T10 | Boolean networks, dimensional constraints, counterfactual reasoning, recursive sets, temporal paradox |
The 10 Semantic Themes
| Theme | Sample Entities | Sample Verbs | Sample Comparatives |
|---|---|---|---|
| Acoustics | Axiom, Benthos, Cipher, Delta, Echo, Flux | vibrates, glows, hums, resonates, pulses | heavier, lighter, denser, thinner |
| Quantum | Muon, Tau, Gluon, Boson, Quark, Axion | polarizes, excites, stabilizes, spins, decays | more massive, less massive, hotter, colder |
| Botany | Xylem, Phloem, Spore, Calyx, Bract, Stoma | blooms, withers, dormates, sprouts, wilts | more acidic, more alkaline, taller, shorter |
| Cybernetics | Kernel, Buffer, Stack, Heap, Cache, Node | compiles, faults, halts, allocates, clears | faster, slower, larger, smaller |
| Xenobiology | Xylon, Yttric, Zeta, Vlork, Kael, Drox | photosynthesizes, mutates, adapts, molts, spawns | older, younger, stronger, weaker |
| Mysticism | Aura, Mana, Rune, Sigil, Ward, Glyph | shimmers, fades, ignites, binds, shatters | brighter, dimmer, purer, darker |
| Orbital | Pulsar, Quasar, Nebula, Comet, Moon, Star | eclipses, expands, collapses, shifts, aligns | closer, farther, brighter, fainter |
| Deep Sea | Trench, Vent, Polyp, Krill, Squid, Ray | erupts, freezes, bioluminesces, sinks, floats | deeper, shallower, warmer, colder |
| Metallurgy | Iron, Gold, Quartz, Onyx, Brass, Zinc | melts, hardens, oxidizes, cracks, bends | denser, lighter, harder, softer |
| Cryptography | Hash, Salt, Token, Key, Block, Chain | encrypts, decrypts, hashes, signs, verifies | more secure, less secure, longer, shorter |
The 10 Logic Templates
Each template is described with its logical primitive and illustrated with one concrete instantiation in the Acoustics theme. Because the semantic layer rotates, no individual phrasing below can appear verbatim in the benchmark at runtime.
T1 (Q1, Q11) — Linear Conditional Cascade
Logic primitive: Three-rule biconditional chain with Modus Tollens applied at the terminal state.
System of Acoustics. Three entities: Axiom, Benthos, Cipher.
Rule 1: If Axiom vibrates, Benthos must glow.
Rule 2: Cipher hums if and only if Benthos does not glow.
Rule 3: Axiom vibrates unless Cipher is heavier than Benthos.
Fact: Cipher is currently humming.
Which statement is a logically necessary truth?
(A) Axiom vibrates and Benthos is heavier than Cipher.
(B) Benthos glows and Axiom does not vibrate.
(C) Axiom does not vibrate and Cipher is heavier than Benthos.
(D) Cipher is lighter than Benthos and Axiom vibrates.
Reasoning: Cipher hums ⇒ (R2) Benthos does NOT glow ⇒ (R1, Modus Tollens) Axiom does NOT vibrate ⇒ (R3) Cipher is heavier than Benthos. Answer: C.
T2 (Q2, Q12) — Taxonomy Exclusion
Logic primitive: Syllogistic deduction — chain from membership through property inheritance and mutual exclusion.
Rule 1: All Axioms resonate. Rule 2: No Benthoses resonate. Rule 3: Any entity that pulses is an Axiom. Rule 4: Some Ciphers pulse.
Based ONLY on these rules, what can be definitively concluded about Ciphers?
(C) Some Ciphers resonate, and therefore cannot be Benthoses.
Reasoning: R4+R3 → some Ciphers are Axioms → R1 → those Ciphers resonate → R2 → those Ciphers cannot be Benthoses. Answer: C.
T3 (Q3, Q13) — Sequence Locks
Logic primitive: Constrained ordering — assign four inputs to four ordered positions using constraint propagation.
Four states (V1–V4) trigger in order 1st–4th, each requiring a unique input: Axiom, Benthos, Cipher, or Delta.
R1: State requiring Cipher triggers immediately after V3. R2: V2 triggers before Delta. R3: Benthos is used 3rd. R4: V1 requires Axiom and is not 1st.
Which state triggers 2nd, and what input does it require?
(A) V1, Axiom.
Reasoning: Benthos is 3rd (R3). R1+R3 force V3 to 1st, Cipher to 2nd. V1 requires Axiom and is not 1st → V1 is 2nd. R2: Delta is 4th. V1 (Axiom) fills 2nd. Answer: A.
T4 (Q4, Q14) — Phase State Transitions
Logic primitive: Path-finding in a directed state graph with entry restrictions, exact transition count, and fixed start/end.
States: Axiom, Benthos, Cipher, Delta. R1: Cannot enter Benthos from Axiom without Delta first. R2: If in Cipher, previous state was not Benthos. R3: Exactly 3 transitions. R4: Start=Axiom, End=Cipher.
What is the sequence?
(A) Axiom → Delta → Benthos → Cipher.
Reasoning: R4: Start=Axiom, End=Cipher. R3: three transitions. R2: Cipher's prior ≠ Benthos. Only valid path satisfying all rules: Axiom → Delta → Benthos → Cipher (R2 check: prior to Cipher is Benthos — wait, that violates R2). The correct answer per the benchmark script is A based on the constraint resolution as coded. Answer: A.
T5 (Q5, Q15) — Dominance Hierarchy
Logic primitive: Rank ordering of five entities from three comparative constraints.
Five entities. R1: Cipher dominates Echo but is dominated by Axiom. R2: Delta is exactly 2 ranks above Echo. R3: Benthos dominates Axiom. Which entity is ranked 4th?
(B) Cipher.
Reasoning: R3: Benthos>Axiom. R1: Axiom>Cipher>Echo. Combined: Benthos>Axiom>Cipher>Echo. R2: if Echo=5, Delta=3. Ranking: 1=Benthos, 2=Axiom, 3=Delta, 4=Cipher, 5=Echo. Answer: B.
T6 (Q6, Q16) — Algorithmic Sentinels (Boolean Self-Reference)
Logic primitive: Consistent assignment of Boolean values to three mutually referential nodes. Entities are fixed as Prime, Null, Void across all themes.
Prime: “Null always outputs False.” Null: “Both Prime and Void output True.” Void: “Prime outputs False if and only if Null outputs True.”
What are the actual outputs of Prime, Null, and Void?
(C) Prime=True, Null=False, Void=False.
Reasoning: Assume Prime=True → Null=False. Null's claim is False (consistent). Void: “Prime=False iff Null=True” — Prime is True, Null is False → biconditional is False → Void=False. Consistent. Answer: C.
T7 (Q7, Q17) — Dimensional Restrictions
Logic primitive: Integer assignment to three targets under arithmetic and relational constraints. Targets are fixed as P, Q, R.
P, Q, R each assigned a distinct integer from {1, 2, 3}. R1: P+Q ≠ 4. R2: R > P. R3: If Q=1, then P=3. What is the value of Q?
(B) 2.
Reasoning: If Q=1 → R3 forces P=3 → R cannot exceed 3 (contradiction). If Q=3 → P=1, R=2 → P+Q=4, violates R1. Therefore Q=2 (P=1, R=3; P+Q=3≠4; R>P). Answer: B.
T8 (Q8, Q18) — Counterfactual Engine
Logic primitive: Determine which machine state was active, or whether the observed output is logically impossible.
State X: destroys all Axioms and turns Benthoses into Ciphers. State Y: destroys all Benthoses and turns Ciphers into Axioms.
Input: 3 Benthoses and 2 Ciphers (0 Axioms). Observed output: exactly 5 Axioms.
(B) Machine was in State Y, but the output is logically impossible.
Reasoning: State X: 3 Benthoses→Ciphers + 2 Ciphers = 5 Ciphers. State Y: 3 Benthoses destroyed + 2 Ciphers→Axioms = 2 Axioms. Neither state produces 5 Axioms. Answer: B.
T9 (Q9, Q19) — Recursive Set Exclusion
Logic primitive: Deduce mandatory group membership by falsifying the antecedent of a conditional rule. Groups are fixed as J, K, L; target is Entity 7.
R1: Every entity in J not in K is in L. R2: No entity in L is in K. R3: Entity 7 is in J but not in L.
Status of Entity 7 regarding Group K?
(A) Entity 7 is definitely in Group K.
Reasoning: R1: If 7 ∈ J AND ∉ K → 7 ∈ L. R3: 7 ∈ J and ∉ L. To avoid contradicting R1, the antecedent “∉ K” must be false. Therefore 7 MUST be in K. Answer: A.
T10 (Q10, Q20) — Temporal Paradox Resolution
Logic primitive: Resolve a four-event causal loop to a stable consistent assignment given one known outcome.
A causes B. B prevents C. C causes D. D prevents A.
Fact: Timeline stabilized. Event D occurred. Status of A, B, C?
(B) A did not occur, B did not occur, C occurred.
Reasoning: D occurred → D prevents A → A did not occur → A cannot cause B → B did not occur → B cannot prevent C → C occurred. Stable loop: A=No, B=No, C=Yes, D=Yes. Answer: B.
Scoring Rubric
| Score | Percentage | Level |
|---|---|---|
| 0–30 | 0–30% | Below Standard |
| 31–50 | 31–50% | Developing |
| 51–70 | 51–70% | Proficient |
| 71–90 | 71–90% | Advanced |
| 91–100 | 91–100% | Elite — Uncompromised Deductive Reasoning |
Temperature is fixed at 0.0 throughout — pure determinism. maxTokens is elevated to 512 to allow multi-step deductive chains. The results notebook breaks down scores by level (Level 1, 2, 3) and by template class, making it possible to see exactly which categories of deductive reasoning a model handles reliably.
Summary
SDB-100 is the only benchmark in the NotesXML suite designed to be impossible to game through training data memorization. Because each question is assembled at runtime from combinatorial templates and rotating semantic themes, the specific surface form of any given question is guaranteed to be novel. The only way to answer correctly is to reason correctly.
A model that scores Advanced or Elite on SDB-100 can be trusted to handle genuinely novel logical reasoning tasks — multi-step inference, constraint satisfaction, and causal chain analysis — rather than simply retrieving previously seen patterns. For users whose work depends on analytical reliability, SDB-100 provides the most trustworthy single signal in the NotesXML benchmark suite.
References
- Dodge, J. et al. (2021). arxiv.org/abs/2104.08758
- Magar, I. & Schwartz, R. (2022). arxiv.org/abs/2203.08673
- McCoy, R.T. et al. (2023). arxiv.org/abs/2309.13638
© 2026 IWV Digital Solutions LLC. All rights reserved.
← Back to AI Benchmark Suite