Talk rules,computesimilarity.
A pre-registered instrument that measures where a language model sits between similarity-based and rule-based generalization — and whether it applies the Bayesian size principle. Five model families, two concept domains, one surprising map.
Rules or similarity?
The literature answers both ways — on different tasks. A benchmark can't settle it. An instrument can.
When a language model generalizes a concept from a handful of in-context examples, what kind of learner is it imitating? One camp says similarity: graded feature overlap, like exemplar models from the psychology of categorization. The other says rules: entertain a hypothesis space, select a consistent rule and — in its Bayesian form — apply the size principle, trusting the narrowest rule more as consistent examples pile up.
The two accounts make opposite predictions about evidence. A similarity learner stays flat as demonstrations accumulate; a size-principle learner contracts toward the narrowest rule. Most work fits one account on one task — which can't tell “the model is a similarity learner” apart from “this task doesn't distinguish the accounts.”
So I built a coordinate system instead of a score: demonstrations that deliberately underdetermine the concept, five exact reference learners computed on every trial, and a single coordinate — λ — that reads off which learner the model behaves like. A flat λ where a rule learner predicts contraction isn't a failed experiment. It's a measurement, because the reference says exactly how large the missing effect should have been.
One coordinate: λ.
The consistent hypotheses form a lattice interval, from the meet up to the join. A probe that matches j of k attributes locates the model on it.
Positives carry all four relevant attributes, negatives none — so every single attribute, every partial conjunction and the full conjunction stay consistent. The concept is underdetermined by design. Labels are counterbalanced nonce words.
One graded law, many families.
Uninstructed, every family and every scale lands on the same graded ladder — blind-predicted before the runs, and confirmed twice.
- /01Graded, as predicted
λ ≈ j/k was written down before the runs at k = 4 and k = 5 — observed within ~2 seed-clustered SE at k = 4, and 6/6 cells graded at k = 5.
- /02No version-space contraction
λ stays flat from n = 4 to n = 32 at every scale, while the exact-Bayes reference contracts ≈ 0.11 → 0.03.
- /03No rule-shift with scale
32B is as graded as 0.5B. Base and instruct models land on the same ladder.
- /04Exemplar residue is family-specific
Qwen shows GCM-like sensitivity to irrelevant-attribute overlap (z = 6.96); Llama does not (z = 2.82).
Because the profile is algebraically the GCM on this design, it's framed as the models' default generalization policy — not a new law of cognition.
Operation, statistic, policy.
A reasoning model's own thinking toggle is a same-weights causal switch. So what does RL-trained deliberation actually change?
A one-pass version-space instruction inside RL-trained deliberation collapses λ̄ toward the Bayes reference — with calibration intact (sanity 0.97). Intersecting hypotheses is a capacity, not a default.
Spell the size principle out in the instruction and contraction appears: pooled slope z = −2.81 over 68 paired seeds. Real but shallow — exact Bayes contracts 0.19 → 0.000 over the same range.
Thinking on vs. off — identical weights — is null at 8B, 14B and 32B (+0.050 / +0.039 / −0.036). No scale spontaneously manages a hypothesis space.
Reasoning traces contain rule talk, yet the stated conjunctions are never applied as meets. Reasoning training buys faithful execution of an instructed inference policy — it doesn't change the inductive bias.
Same weights, two evidence laws.
Ported to Tenenbaum's number game — exact Bayes over 32 rules and 5,050 intervals — the same Qwen3-8B turns rule-like and contracts toward the Bayesian reference.
Number concepts come with a compact, nameable rule vocabulary — “even,” “multiple of 8,” “square.” Boolean conjunctions like red ∧ striped ∧ square don't. Domain-gating looks less like a switch between two inference engines and more like a reflection of which concepts are nameable in the model's prior.
One default, five identities.
The pre-registered universality bar — at least 3 of 4 families rule-like and contracting — was not met. That failure is the finding.
| Family | Δfit · S1 / S2 | Slope λ(32)−λ(1) · S1 / S2 | Replicated profile |
|---|---|---|---|
/01Qwen3-8B | +0.561 / +0.552 | −0.461 / −0.396 z −9.6 / −8.8 | Bayes rule induction Off-rule acceptance dies while in-rule acceptance rises — the full signature. |
/02Gemma-2-9B | +0.345 / +0.394 | −0.500 / −0.450 z −9.4 / −7.6 | Rule + contraction Both headline verdicts, plus a global-tightening component. |
/03Llama-3.1-8B | +0.300 / +0.347 | −0.167 / −0.208 ns ×2 | Rules, frozen evidence Selects rules, but no resolvable evidence dynamics. |
/04OLMo-2-13B | +0.034 / +0.035 | −0.750 / −0.583 z −7.4 / −5.5 | Tightening without rules Everything collapses toward the demonstrations. |
/05Mistral-7B-v0.3 | −0.089 / −0.026 | +0.140 / +0.183 z +2.6 / +3.0 | Similarity + expansion λ grows with n — the number-game analog of the Boolean default. |
The same five families that are homogeneous on Boolean concepts (31/31 cells, one graded default) are heterogeneous on numbers (five distinct, individually replicated profiles). The similarity default is family-universal; hypothesis-space induction is family-idiosyncratic. Flagged post-hoc — this contrast wasn't a pre-registered hypothesis.
Every kill, on the record.
Each hypothesis was frozen in a git commit before its run. Kills, exclusions and deviations carry the same weight as the positives.
Structure or familiarity?
The number game bundles two properties. The next domain is built to pull them apart.
Scaffold rules or nearest neighbours?
Integers have both a clean, decidable hypothesis lattice and a rule vocabulary saturated in pretraining text — so the number game can't say which one switches rule-like induction on. The substructure lattice over molecules has the first and not the second: containment is decided by subgraph matching, but nothing drills “accept iff it contains a primary sulfonamide” the way arithmetic drills “multiples of 8.”
The two readings predict opposite outcomes there. The design is frozen and pre-registered, both reference identities were recovered from planted oracles end-to-end, and the runs execute locally on Apple Silicon through MLX — no rented GPU. Molecules are opaque strings: the results license claims about model induction, not chemistry.
$ pip install -e .
$ lambda-lattice selftest
$ lambda-lattice score numbers \
data/echo_numgame_q8.json
$ lambda-lattice figuresThe self-test plants a Bayes oracle and a proximity oracle and checks the scorer recovers both. Every number on this page traces to a committed raw JSON.
Instruments,
not benchmarks.
λ-lattice is open source under MIT, with every raw result, pre-registration and figure in the repository. If you work on in-context learning, interpretability or cognitive models of induction, I'd like to compare notes.