Field notes / Experiment guide

Does the prefix signal survive unseen rule families?

The eight-letter prefix predicts held-out kappa. Is that a transferable letter-level rule, or do rules sharing an opening simply behave alike?

New to these ants? Start here.

An ant moves on a grid of coloured squares. Each colour tells it to turn left (L) or right (R). It advances that square to the next colour, then steps forward. A rule such as RLR is the list of turns, starting at colour zero. On a blank grid the same rule always makes the same trajectory.

Territory means every cell visited so far. Its boundary consists of visited cells with an unvisited neighbour. A highway is a repeating pattern of motion that carries the ant away. A tether means repeated returns to a small home region.

Growing like the cube root of time means taking eight times as many steps to double a linear size. The rotor walker is a comparison model that follows and rotates arrows stored on cells. A seed makes a random starting pattern reproducible; the ant itself can still move deterministically.

We distinguish c, the chance of discovering a cell per boundary step, from kappa (κ), time at the boundary normalized by perimeter divided by area. Explaining their values is harder than using them to account for growth.

The idea under test

Registered: the lookup works on seen families but collapses on unseen ones (confirmed), while a pairwise-interaction model keeps its signal across families and beats the lookup there (both failed).

What we learned

With whole 6-letter prefix families held out, every model falls below the training mean: the lookup goes from 0.59 to -0.16 and the pairwise model from 0.30 to -0.69.

What this leaves open

Only 43 uneven families and about five held out per fold, so held-out family means differ from the training mean; this does not show that no structure exists within a family.

Read the evidence and reproduction recipe ↗

Inside the experiment

Methods, registered predictions, exact commands and saved analysis. These are the working research documents; expand them for the detail behind the introduction.

Methods and original experiment record

Does the prefix signal survive holding out whole prefix families? (2026-10-20)

Question

../2026-10-20-kappa-prefix-additive/ could not separate higher-order interaction structure from memorisation of prefix families shared by train and test. If the signal is structure, a model should predict rules from families it has never seen; if it is memorisation, only the lookup should work on seen families and it should collapse on unseen ones.

Design (registered before running, 2026-10-20 07:10 +13:00, git 7339dc9)

Data: the 2185 stationary blob census rules (../2026-10-13-probe-growth-census/results/probe_summary.csv, target ln kappa). (The 162 fresh rules share their 6-letter prefix families with the census, so they cannot test unseen families.)
Family = first 6 letters. Grouped CV: 8 folds in which each fold holds out whole families (families assigned to folds by a seeded shuffle, seed 20261020); random CV: 8 folds of random rules (same seed), where test rules' families are in training. Out-of-fold strict R^2 against the training-fold mean, pooled over folds.
Models: P8 (8-letter prefix lookup with fallback to shorter keys, >= 3 rules); A2 (ridge, penalty 1.0, 7 position indicators + 21 pairwise products, as in the sibling experiment); A (additive ridge).
Bootstrap over rules for 95% intervals (2000 resamples).

Predictions (written before the run)

D1: P8 has random-CV R^2 >= 0.45 and grouped-CV R^2 < 0.30 (memorisation of families).
D2: A2 grouped-CV R^2 is within 0.10 of its random-CV R^2 (its signal, if any, is not family memorisation).
D3: A2 grouped-CV R^2 exceeds P8 grouped-CV R^2.

Failures are reported as failures; choices made after seeing results are labelled post hoc.

Results (`results/holdout.txt`; 2185 census rules, 43 families, 8-fold CV, pooled out-of-fold strict R^2)

model	random CV	family-held-out CV
A (additive)	+0.109 [+0.069, +0.150]	-0.162 [-0.200, -0.128]
A2 (pairwise)	+0.300 [+0.239, +0.357]	-0.690 [-0.857, -0.524]
P8 (lookup)	+0.591 [+0.468, +0.691]	-0.162 [-0.230, -0.107]

D1 HELD, D2 FAILED, D3 FAILED.

Reading. When whole 6-letter prefix families are held out, every model does worse than predicting the training mean (negative R^2); the lookup falls from 0.59 to -0.16 and the pairwise model from 0.30 to -0.69. So none of these models learns a letter-level rule that transfers to unseen families: the predictive signal is family membership (rules sharing an opening behave alike), which is what the paper's own caveat says (transfer within prefix families to longer rules, not to unseen families). This is the answer to the sibling experiment's question: the lookup's advantage over additive and pairwise models is memorisation of family means, not a structure the other models missed. Caveats: only 43 families, with uneven sizes, and each held-out fold removes about five whole families, so this is a hard extrapolation in which the held-out families' mean ln kappa can differ from the training mean by more than any letter-level model can recover (the negative R^2 mostly reflects family-mean shifts; the test does not show that letter-level structure is absent within a family or at lengths not tested); the pairwise ridge is untuned and overfits families (-0.69). Not tested: a model with a family-level random effect, or held-out families at coarser or finer prefix lengths (4 or 8 letters).

Reproduce

uv run experiments/2026-10-20-kappa-prefix-family-holdout/scripts/holdout.py -> results/holdout.txt, results/holdout.json.

Analysis notebook: notebook.ipynb

1 saved outputs · 1 code cells. Cells are not executed during publication. Download notebook

Does the prefix signal survive holding out whole prefix families?

Registered test (D1-D3 committed before the run; see README). On the 2185 stationary census rules (43 families = first 6 letters), 8-fold cross-validation either by random rules or by whole families. Models: additive ridge A, additive plus pairwise A2, 8-letter lookup P8. Score: pooled out-of-fold strict R^2 against the training-fold mean.

Assumptions. Same stationarity filter as the other experiments; ridge penalty fixed at 1.0.

Code

In [1]:

import json
from pathlib import Path

root = next(p for p in [Path.cwd(), *Path.cwd().parents] if (p / "AGENTS.md").exists())
d = json.loads((root / "experiments/2026-10-20-kappa-prefix-family-holdout/results/holdout.json").read_text())
print("input:", {k: v[:12] for k, v in d["input_sha256"].items()}, "rules", d["n"], "families", d["n_families"])
for m in ("A", "A2", "P8"):
    r, g = d["r2"][f"random/{m}"], d["r2"][f"grouped/{m}"]
    print(f"{m:3s} random CV {r:+.3f}   family-held-out CV {g:+.3f}")
print(d["verdicts"])
assert list(d["verdicts"].values()) == [True, False, False]  # D1 held, D2 and D3 failed, as in the README
assert all(d["r2"][f"grouped/{m}"] < 0 for m in ("A", "A2", "P8"))
assert abs(d["r2"]["random/P8"] - 0.591) < 2e-3

input: {'probe_summary.csv': 'c76c7927185b'} rules 2185 families 43
A   random CV +0.109   family-held-out CV -0.162
A2  random CV +0.300   family-held-out CV -0.690
P8  random CV +0.591   family-held-out CV -0.162
{'D1 P8 random >= 0.45 and grouped < 0.30': True, 'D2 A2 grouped within 0.10 of random': False, 'D3 A2 grouped > P8 grouped': False}

What this shows and does not show

Holding out whole 6-letter prefix families, every model falls below the training mean (lookup 0.59 to -0.16; pairwise 0.30 to -0.69). The predictive signal is family membership, not a letter-level rule that transfers to unseen families.
That answers the additive experiment's open question: the lookup's advantage over additive and pairwise models is memorisation of family means.
Not shown: that no structure exists within families. Only 43 uneven families, about five held out per fold; negative R^2 largely reflects held-out family means differing from the training mean. No family random-effect model or other family granularity was tried.

A family-level random effect plus within-family letter structure, and holding out families at 4 and 8 letters.

Read the dated lab log

Lab notebook: kappa prefix family holdout

Append-only. Newest entry last. Never edit an old entry; correct it with a new one (corrects: <timestamp>). Tags: intent, assumption, observation, interpretation, surprise, decision, dead-end, question, deviation. See experiments/README.md.

2026-10-20 07:10 (+13:00) · 7339dc9 · intent

Separate higher-order letter structure from prefix-family memorisation, which the additive experiment could not. Hold out whole 6-letter prefix families in cross-validation on the census. Predictions D1-D3 registered in README and committed (1787ecc) before scripts/holdout.py existed.

2026-10-20 07:30 (+13:00) · 1787ecc · observation

Ran holdout.py (seed 20261020, 8 folds, 43 families). Pooled out-of-fold strict R^2, random CV vs family-held-out CV: A +0.109 vs -0.162; A2 +0.300 vs -0.690; P8 +0.591 vs -0.162. D1 held, D2 and D3 failed.

2026-10-20 07:35 (+13:00) · 1787ecc · surprise

I expected the pairwise model to keep at least some signal across families (D2, D3). It does worst of all (-0.69), worse than the training mean. Even the lookup, which falls back to shorter prefixes for unseen families, is below the mean.

2026-10-20 07:40 (+13:00) · 1787ecc · interpretation

The prefix signal is family membership, not a transferable letter-level rule, consistent with the paper's caveat. Caveats: 43 uneven families, about five held out per fold, and held-out family means differ from the training mean; negative R^2 largely reflects that. The test does not show that no structure exists within families.

2026-10-20 07:40 (+13:00) · 1787ecc · question

Does a model with a family random effect plus within-family letter structure beat the family mean alone? And does holding out families at 4 or 8 letters change the picture (a graded test of how coarse a family is)? Not done.