Field notes / Experiment guide

Is the prefix signal additive over letters or does it need interactions?

The eight-letter lookup carries information spread over letters 2-7. Would a model that adds up one effect per letter position do as well?

New to these ants? Start here.

An ant moves on a grid of coloured squares. Each colour tells it to turn left (L) or right (R). It advances that square to the next colour, then steps forward. A rule such as RLR is the list of turns, starting at colour zero. On a blank grid the same rule always makes the same trajectory.

Territory means every cell visited so far. Its boundary consists of visited cells with an unvisited neighbour. A highway is a repeating pattern of motion that carries the ant away. A tether means repeated returns to a small home region.

Growing like the cube root of time means taking eight times as many steps to double a linear size. The rotor walker is a comparison model that follows and rotates arrows stored on cells. A seed makes a random starting pattern reproducible; the ant itself can still move deterministically.

We distinguish c, the chance of discovering a cell per boundary step, from kappa (κ), time at the boundary normalized by perimeter divided by area. Explaining their values is harder than using them to account for growth.

The idea under test

Registered before the run: an additive model scores below 0.40 held-out R^2, a model with pairwise interactions reaches at least 0.50, and the full lookup beats the additive model by at least 0.15.

What we learned

Two of three registered predictions held and one failed: an additive model scores 0.11 held-out R^2, adding all pairwise interactions gives 0.35 (predicted at least 0.50), and the full eight-letter lookup scores 0.62.

What this leaves open

The ridge penalty was not tuned and only binary indicators were used; the lookup also exploits prefix families shared between training and test, so interaction structure and family membership are not separated.

Read the evidence and reproduction recipe ↗

Inside the experiment

Methods, registered predictions, exact commands and saved analysis. These are the working research documents; expand them for the detail behind the introduction.

Methods and original experiment record

Additive or interacting? Letter-by-position models of ln kappa (2026-10-20)

Question

../2026-10-20-kappa-prefix-ablation/ found the 8-letter lookup's information is spread over letters 2-7, but a conjunctive key cannot separate main effects from interactions. Does an additive model over the same letters reach the lookup's R^2?

Design (registered before running, 2026-10-20 06:00 +13:00, git bfa8c0e)

Same split as the sibling experiments: 2185 stationary blob census rules (train), 162 fresh stationary rules (test), disjoint; strict held-out R^2 against the training mean.
Features: binary indicator (1 if the letter is L) for positions 2..8 (position 1 is constant); rules shorter than 8 letters get 0 for missing positions plus an indicator len < 8... (the fresh test rules are all >= 13 letters, so this only affects training).
A: ridge regression, 7 position indicators + the short-rule flag. A2: A plus all 21 pairwise products of the 7 indicators. Ridge penalty fixed at 1.0 on standardised features (not tuned). Reference: the 8-letter lookup P8 (>= 3 rules, fallback to shorter keys).
Bootstrap over test rules, 2000 resamples, seed 20261020.

Predictions (written before the run)

B1: the additive model A has test R^2 < 0.40.
B2: the pairwise model A2 has test R^2 >= 0.50.
B3: P8 exceeds A by at least 0.15.

Failures are reported as failures; anything chosen after seeing the results is labelled post hoc.

Results (`results/additive.txt`; 2185 train, 162 test, 0 in both)

model	strict held-out R^2	95% CI
A: additive, 7 position indicators	+0.109	[+0.003, +0.194]
A2: A + 21 pairwise products	+0.351	[+0.234, +0.447]
P8: 8-letter lookup	+0.624	[+0.473, +0.729]

P8 - A = +0.515 [+0.372, +0.634]. B1 HELD (A < 0.40), B2 FAILED (A2 = 0.35 < 0.50), B3 HELD (P8 - A >= 0.15).

Reading. An additive model over letter positions captures about a sixth of what the lookup does, and adding all pairwise interactions still leaves 0.27 of R^2 on the table. The signal therefore needs interactions of order three or higher among letters 2-8, consistent with it being a property of whole opening patterns (families of rules sharing a prefix) and not of individual letters. Caveats: ridge penalty fixed at 1.0 and not tuned (a tuned or nonlinear model such as a tree ensemble might reach more than A2 without being a lookup); binary L/R indicators only; the lookup's advantage is partly memorisation of prefix families that overlap between training and test, so "interaction structure" and "family membership" are not separated here.

Reproduce

uv run experiments/2026-10-20-kappa-prefix-additive/scripts/additive.py -> results/additive.txt, results/additive.json.

Analysis notebook: notebook.ipynb

1 saved outputs · 1 code cells. Cells are not executed during publication. Download notebook

Is the prefix signal additive over letters?

Registered test (B1-B3 committed before the run; see README). Same split as the sibling experiments: 2185 stationary census rules train, 162 fresh stationary rules test, disjoint. Ridge regression (penalty fixed at 1.0, not tuned) on binary 'letter is L' indicators for positions 2-8, with and without all pairwise products, against the 8-letter lookup P8.

Assumptions. Same stationarity filter and fresh-rule target as the W3c audit; binary indicators only.

Code

In [1]:

import json
from pathlib import Path

root = next(p for p in [Path.cwd(), *Path.cwd().parents] if (p / "AGENTS.md").exists())
d = json.loads((root / "experiments/2026-10-20-kappa-prefix-additive/results/additive.json").read_text())
print("inputs:", {k: v[:12] for k, v in d["input_sha256"].items()}, "train", d["n_train"], "test", d["n_test"], "overlap", d["overlap"])
for k, v in d["r2"].items():
    lo, hi = d["ci"][k]
    print(f"{k:3s} R^2 {v:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")
print(d["verdicts"])
assert d["overlap"] == 0
assert list(d["verdicts"].values()) == [True, False, True]  # B1 held, B2 failed, B3 held, as in the README
assert d["r2"]["A"] < 0.2 and 0.3 < d["r2"]["A2"] < 0.4 and abs(d["r2"]["P8"] - 0.624) < 2e-3

inputs: {'probe_summary.csv': 'c76c7927185b', 'prefix_audit_predictions.csv': '742f0fe1d776'} train 2185 test 162 overlap 0
A   R^2 +0.109  95% CI [+0.003, +0.194]
A2  R^2 +0.351  95% CI [+0.234, +0.447]
P8  R^2 +0.624  95% CI [+0.473, +0.729]
{'B1 additive A < 0.40': True, 'B2 pairwise A2 >= 0.50': False, 'B3 P8 - A >= 0.15': True}

What this shows and does not show

An additive model over letter positions scores 0.11; adding all pairwise interactions gives 0.35 (B2, which predicted at least 0.50, failed); the lookup scores 0.62. The signal needs interactions of order three or higher among letters 2-8.
Not shown: whether that is real interaction structure or memorisation of prefix families that overlap between training and test. The ridge penalty was not tuned and only binary indicators were used.

A split by prefix family (hold out whole 6-letter prefix families) to see whether any of the 0.62 survives without family overlap.

Read the dated lab log

Lab notebook: kappa prefix additive

Append-only. Newest entry last. Never edit an old entry; correct it with a new one (corrects: <timestamp>). Tags: intent, assumption, observation, interpretation, surprise, decision, dead-end, question, deviation. See experiments/README.md.

2026-10-20 06:00 (+13:00) · bfa8c0e · intent

Test whether the prefix signal for ln kappa is additive over letter positions or needs interactions. Predictions B1-B3 registered in README and committed (32c0f82) before scripts/additive.py existed.

2026-10-20 06:40 (+13:00) · 922a870 · observation

Ran additive.py (seed 20261020, ridge lambda 1.0, 2000 bootstraps). Held-out R^2 on 162 fresh stationary rules: additive A +0.109, additive+pairwise A2 +0.351, 8-letter lookup P8 +0.624. B1 held, B2 FAILED, B3 held.

2026-10-20 06:40 (+13:00) · 922a870 · deviation

The reader guide for this experiment was added before the run, saying 'registered but not yet run', because the blog renderer fails without a guide for each tracked experiment and main was about to receive this folder. Updated to the result after the run (see the guide row).

2026-10-20 06:45 (+13:00) · 922a870 · interpretation

Letter-by-position effects explain little; pairwise interactions explain about half of what the lookup does; the rest needs order-three-or-higher structure or family membership. I cannot separate these with this design, because the lookup also exploits prefix families shared between train and test. The B2 failure means I overestimated what pairs could recover.

2026-10-20 06:45 (+13:00) · 922a870 · question

Does a flexible model that cannot memorise families (trained on rules whose 6-letter prefix families are held out entirely) retain any of the 0.62? That is the family-versus-structure question and needs a split by prefix family; not done.