Field notes / Experiment guide

Faster growth measurements, checked byte for byte

Can the GPU produce the same full growth and shape measurements as the CPU probe for large batches of rules and seeds?

New to these ants? Start here.

An ant moves on a grid of coloured squares. Each colour tells it to turn left (L) or right (R). It advances that square to the next colour, then steps forward. A rule such as RLR is the list of turns, starting at colour zero. On a blank grid the same rule always makes the same trajectory.

Territory means every cell visited so far. Its boundary consists of visited cells with an unvisited neighbour. A highway is a repeating pattern of motion that carries the ant away. A tether means repeated returns to a small home region.

Growing like the cube root of time means taking eight times as many steps to double a linear size. The rotor walker is a comparison model that follows and rotates arrows stored on cells. A seed makes a random starting pattern reproducible; the ant itself can still move deterministically.

We distinguish c, the chance of discovering a cell per boundary step, from kappa (κ), time at the boundary normalized by perimeter divided by area. Explaining their values is harder than using them to account for growth.

The idea under test

The port should preserve every CSV byte while speeding up stepping-heavy independent trajectories.

What we learned

Equivalence checks pass. A fresh-build remeasurement supersedes the original stale-binary benchmark and gives a roughly four-to-sixfold gain on the recorded workload.

What this leaves open

The CPU baseline shifted substantially for an unexplained reason. The timings are few runs on one GPU, not a general performance guarantee.

Read the evidence and reproduction recipe ↗

Inside the experiment

Methods, registered predictions, exact commands and saved analysis. These are the working research documents; expand them for the detail behind the introduction.

Methods and original experiment record

GPU probe: `probe` on a CUDA device

Status: in progress (code and tests done 2026-10-16; timing waits for an idle machine). Goal: run probe MAX_LOG2 [GRID] batches (plain rules RULE and patched ants RULE@SEED) on the RTX 4000 Ada with CSV rows byte-identical to src/bin/probe.rs, so a batch like 2026-10-16-kappa-leadrun-fresh (2400 runs, probe 27 8192, about 3e11 steps, ~20 min on 6 CPU cores) stops occupying the CPU.

Question

Can the probe instrument (visited region, perimeter, fresh-neighbour counts, octave windows, gyration and ant moments) be ported to the tile-pool GPU runner with exactly the CPU's output, and how much faster is it than the 6-thread CPU probe on the real 2400-run batch?

Design (decided before any timing)

New files only: src/gpu/probe.cu (kernels), src/gpu/probe.rs (GpuProber, ProbeJob, ProbeConfig), src/bin/gpu_probe.rs, tests/gpu_probe.rs. ant.cu, GpuSweeper, fast, lean are untouched. The scheduler (lane refill between slices, results by job index, pool exhaustion -> requeue + halve the active-lane cap, solo failure -> GpuError) is copied from GpuSweeper::run rather than generalised, so the sweep's tests keep guarding it.
Cell = 16 bits: low byte colour, high byte unv + 1 for visited cells, 0 for never-visited. An all-zero tile is blank and unvisited, so the zeroed pool and the retire kernel's zeroing need no change, and a neighbour in an unallocated tile is unvisited without allocating. One load per step gives colour, visited flag and the boundary/fresh-neighbour counts (the current cell is always visited). 8 KiB tiles (vs 4 KiB in the sweep). A one-byte layout (5-bit colour + 3-bit unv + 1, n <= 32) is possible; not built unless VRAM binds.
Windows in the kernel: a lane's slice is cut into chunks that end at window boundaries (steps == 2^(k+1)); the hot loop has no window test. At a boundary the kernel writes one 104-byte record (area, perimeter, counters, region sums, ant sums, bounding box) to recs[lane][k] and resets the window counters. An escaping step ends the run without closing its window, so no partial window is ever emitted and escaped is set on all emitted rows.
Exact floats: the GPU accumulates ax, ay, axx as integers; the host converts them with as f64 and calls probe::finish_window, the same function the CPU probe now uses (same expression order). Exact while a window's axx < 2^53 (probe::window_sums_exact); GpuProber::run refuses jobs past that bound (2^27 steps on 8192 is fine).
Patch: PatchedAnt::initial_cell (hash of seed and relative position inside the 5x5 patch) is written when a tile is allocated, for the patch cells that fall in that tile. The patch straddles up to four tiles (at 8192 the start is a tile corner), so writing it with the first tile would be wrong.
Escape test |x| >= c-2 || |y| >= c-2 as one unsigned compare per axis; the bounding box is updated only on first visits (as the CPU does).
Refactor of the CPU side (src/probe.rs): WindowSums, finish_window, csv_rows, CSV_HEADER, window_sums_exact shared by probe, gpu_probe and the GPU library. Checked by comparing the old and new probe CSV on 18 mixed specs (all walker kinds), see NOTEBOOK.

Predictions (registered 2026-10-16 20:49 NZDT, before any timing)

P1 (correctness). gpu_probe 27 8192 on the 24 spot-check specs (every 100th line of 2026-10-16-kappa-leadrun-fresh/results/specs.txt, lines 1, 101, ..., 2301) is byte-identical to probe 27 8192; all proptests pass (windows equal, floats included). Also predicted: the full 2400-spec GPU CSV equals the existing kappa-leadrun CPU dump byte for byte. Falsified by any differing byte.
P2 (speed). On the full 2400-spec file at probe 27 8192, GPU wall time (whole process) is 2x to 6x faster than a clean probe on 6 CPU threads in the same session, central estimate 3.5x. That is below the sweep kernel's 4.9x: each ant gets no cheaper per step (an extra 16-bit load stream plus window sums on every step), 2400 jobs fill only 2400 of the >= 4096 lanes at which the sweep was still gaining throughput, and tails are not hidden by more jobs than lanes. Falsified if the speedup is >= 4.9x (then the probe is not costlier than the sweep) or < 2x.
P3 (memory). All 2400 ants fit in the default pool at once (oom_requeues == 0, pool about 19 GB; bounding-box tile estimate from the existing dump: 1.6e6 tiles = 13 GB at 8 KiB). Falsified by any requeue.
P4 (throughput vs lanes). Steps/s rises with lanes from 1024 to 2400 (every ant resident). Reported only if run.

Use it

source experiments/2026-10-16-gpu-ant-sweep/scripts/env.sh
cargo run --release --features cuda --bin gpu_probe -- 27 8192 --stats results/x.json < specs.txt > probe.csv

Same stdin lines and CSV as probe; only RULE and RULE@SEED lines (other walkers are rejected before running). Flags: --lanes N (default 4096), --slice N, --pool-gb G, --device K, --stats FILE. Library: turmite::gpu::{GpuProber, ProbeConfig, ProbeJob}; GpuProber::run(&jobs, progress) returns Vec<probe::ProbeResult> index-aligned with the jobs.

When to use it: hundreds or more RULE/RULE@SEED runs at probe 27 8192-sized budgets, on an otherwise idle machine, with --lanes about the job count (2400 lanes for the 2400-spec batch: all resident, 0 requeues, about 157 s vs about 912 s on 6 CPU threads; see "P2 re-measure"). For a handful of runs, other walkers, or anything timed on a binary you have not verified, use CPU probe. Before citing a timing: fresh CARGO_TARGET_DIR, build exit status checked, binary sha256 logged (scripts/bench_p2.sh). Workflow for a new batch: write the spec file, cmp a slice (scripts/spotcheck.sh) against probe, run the batch with --stats, and record command, commit and sha in the experiment's README. Guidance for all GPU tools: experiments/README.md and AGENTS.md.

Results

Recipe: bash scripts/bench.sh (commit 53ccccc, machine otherwise idle: no release/ processes, loadavg 1.0 before gpu1; the cpu phase's loadavg of 5.9 is its own 6 threads). Log: results/bench.log, stats results/bench_gpu{1,2}.json. Inputs: kappa-leadrun-fresh specs.txt, 2400 lines, sha256 1e9198bf...; RTX 4000 Ada, --lanes 2400.

Phase	Whole-process wall	Output vs existing CPU dump
gpu1	177.4 s (device path 161.0 s)	byte-identical
cpu (`probe 27 8192`, 6 threads)	681.6 s	byte-identical
gpu2	176.9 s	byte-identical

P1 held: 24-spec spot check (results/spotcheck.txt) and all 2400 specs are byte-identical; 5/5 gpu_probe proptests pass.
P2 held (registered 2x-6x, falsified at >= 4.9x or < 2x): speedup 681.6/177.4 = 3.84x (3.85x with gpu2), against a central estimate of 3.5x. n = 1 CPU run and 2 GPU runs on one spec set, so no between-run spread beyond gpu1 vs gpu2 (0.4 s); not a sweep over specs or rules. An earlier handover guess of ~7x came from a contaminated CPU time (1231 s) and is superseded.
P3 held: oom_requeues 0, pool 2,409,257 tiles. (The registered 1.6e6-tile bounding-box estimate was low by about 50%; the pool still sufficed.)
P4 not run. (Superseded 2026-10-18: P4 run, see below.)
Other tests after the src/probe.rs refactor: plain cargo test --release and cargo test --release --features cuda --test gpu --test onset all pass (exit 0; the plain run was logged tail-only, 5 result lines, all ok).

P4: throughput vs lanes (2026-10-18)

Recipe: EXPECT_SHA=91c072f7... bash scripts/bench_lanes.sh (HEAD 14b9a24, gpu_probe built fresh in ~/turmite-target-p4, sha256 91c072f715cf...; same 2400 specs as P2). Order 2400, 1024, 1536, 1536, 1024, 2400; no foreign process before or after any phase. Log results/bench_lanes.log, stats results/bench_lanes_<L>_r<k>.json. Every CSV byte-identical to the CPU dump, 0 requeues. Steps/s uses one numerator for all L, total_steps = 2.731e11 (budget of non-escaped runs; identical across L because outputs are).

Lanes	Whole-process wall, r1 / r2 (mean)	Device path `wall_secs`, r1 / r2 (mean)	Steps/s (device)	Per lane	Launches
1024	280.34 / 280.25 (280.30) s	278.14 / 278.12 (278.13) s	0.982e9	9.59e5	319
1536	219.30 / 219.41 (219.35) s	217.16 / 217.26 (217.21) s	1.257e9	8.19e5	256
2400	156.64 / 156.24 (156.44) s	154.35 / 154.16 (154.26) s	1.771e9	7.38e5	128

P4 held under the criterion fixed in NOTEBOOK before running (mean time strictly falls 1024 > 1536 > 2400, each gap larger than the larger repeat spread, same order on the device path). Gaps 60.9 s and 62.9 s against repeat spreads of at most 0.40 s. Batch throughput rises 1.28x (1024 to 1536) and 1.41x (1536 to 2400), 1.80x overall on the device path.
What it does and does not show: n = 2 per L, one spec set, one GPU. Only L = 2400 has every ant resident. Below that, ants queue (about 2.3 and 1.6 rounds of jobs), so the rise in batch steps/s is largely what queueing predicts and would fail only if per-lane throughput collapsed with occupancy. It did not collapse but did fall: per-lane steps/s is 23% lower at 2400 than at 1024 (memory pressure from more resident ants is the obvious candidate, not tested). P4 measures batch scheduling on this workload, not lane-level parallel efficiency at fixed occupancy.

Provenance caveat on P2 (found 2026-10-18). The P2 benchmark did not time a 53ccccc build. The binary it ran (sha256 4d3e395d...) was built 2026-10-16 20:46; the source on /mnt/f kept mtimes of 20:41-20:43 although its content changed later, so cargo never rebuilt it and bench.sh's | tail -1 hid that. P2's 3.84x and its byte-identity checks describe that build. The HEAD build is faster on the same specs (156 s vs 177 s at 2400 lanes) and also byte-identical, but P2 was not rerun against a fresh CPU baseline, so no HEAD speedup is claimed. Other artefacts built in ~/turmite-target since 10-16 may be stale the same way (see NOTEBOOK 2026-10-18).

Tests on the fresh build (2026-10-18): cargo test --release --features cuda --test gpu_probe with CARGO_TARGET_DIR=~/turmite-target-p4: 5/5 pass (17 s). The earlier proptest and post-refactor cargo test passes ran in ~/turmite-target and carry the same stale-build doubt as P2; this run supersedes the gpu_probe one. Plain cargo test --release and the gpu/onset tests were not rerun on a verified build.

P2 re-measure on a verified HEAD build (2026-10-19)

Recipe: bash scripts/bench_p2.sh (HEAD 6de71f2, clean tree for src/tests/Cargo.toml; fresh CARGO_TARGET_DIR=~/turmite-target-p2; build exit status checked, binaries verified newer than the sources). probe sha256 7d341949a912..., gpu_probe 91c072f715cf... (same as the P4 build). Same 2400 specs and dump as P2. Phases gpu1, cpu, gpu2 sequential; no foreign process before or after any phase (results/bench_p2.log, stats results/bench_p2_gpu{1,2}.json).

Phase	Whole-process wall	Output vs dump
gpu1 (`--lanes 2400`)	157.06 s (device path 154.79 s)	byte-identical
cpu (`probe 27 8192`, 6 threads)	912.49 s	byte-identical
gpu2	156.94 s (device path 154.72 s)	byte-identical

P2 is falsified at HEAD on its upper bound. Speedup 912.49/157.06 = 5.81x (5.81x with gpu2), at or above the registered 4.9x line (registered range 2x-6x, central 3.5x; "falsified if >= 4.9x"). The earlier "P2 held, 3.84x" was a stale-binary result (see the caveat above) and is superseded, not edited. By the registered criterion the verdict is that the probe on the GPU is not costlier per step than the sweep kernel was; the lower-bound and in-range part of P2 held only for the stale build.
The ratio moved mostly because of the CPU side. GPU went 177 -> 157 s (HEAD faster than the stale build), but the CPU probe took 912 s now vs 681.6 s then. The old CPU binary (sha f1a309cf) was from the suspect target dir, so it is unknown whether the old or the new probe differs in code (the src/probe.rs refactor) or whether the machine state differs (this CPU run had loadavg ~6 throughout, as expected for 6 threads). Cause of the 34% CPU slowdown is unexplained; I did not bisect it. Anything that depends on the CPU baseline (the speedup) inherits that uncertainty; the GPU time is stable across all four HEAD runs (154.2-154.8 s device path at 2400 lanes).
n = 1 CPU run, 2 GPU runs, one spec set, one GPU. No between-run CPU spread.

Tests on a verified build (2026-10-19), CARGO_TARGET_DIR=~/turmite-target-p2, HEAD 6de71f2: cargo test --release all test binaries ok (no failures); cargo test --release --features cuda --test gpu --test onset --test gpu_probe: gpu 7/7, gpu_probe 5/5, onset 1/1. This supersedes the earlier suspect-dir passes.

Script fixes (2026-10-19): scripts/bench.sh and scripts/spotcheck.sh no longer pipe the build through tail; they abort on build failure and when a binary is older than the sources. bench.sh still defaults to ~/turmite-target, so use CARGO_TARGET_DIR=<fresh dir> there. bench.log, bench_gpu*.json (the stale P2) are kept unchanged.

Analysis notebook: notebook.ipynb

5 saved outputs · 4 code cells. Cells are not executed during publication. Download notebook

GPU probe: `probe` on a CUDA device

Question and why it matters

Can the growth/shape probe (width, area, edge, gyration windows per ant) run on the GPU for thousands of (rule, seed) runs, with output byte-identical to the CPU probe, and how much faster is it? Links: README.md, NOTEBOOK.md.

Assumptions and definitions

CPU probe is the oracle; the GPU must equal it byte for byte (tested, held).
Speedup = CPU whole-process wall / GPU whole-process wall, same 2400 specs, probe 27 8192, 6 CPU threads vs one RTX 4000 Ada at 2400 lanes.
Assumed: machine idle during timing (checked by process list / load average in bench.log).

Provenance

Prints the git commit, date, seed and a SHA-256 for every input file.

Code

In [1]:

import hashlib, subprocess, datetime, json, re
from pathlib import Path
SEED = 0  # no randomness in this notebook: it only reads committed results
ROOT = next(p for p in [Path.cwd(), *Path.cwd().parents] if (p / "AGENTS.md").exists())
EXP = ROOT / "experiments" / "2026-10-16-gpu-probe"
R = EXP / "results"
inputs = [R/"bench.log", R/"bench_gpu1.json", R/"bench_gpu2.json", R/"spotcheck.txt", R/"spotcheck_specs.txt", R/"bench_lanes.log"] + [R/f"bench_lanes_{L}_r{k}.json" for L in (1024, 1536, 2400) for k in (1, 2)]
missing = [p for p in inputs if not p.exists()]
assert not missing, missing
print("notebook run", datetime.date.today(), "seed", SEED)
print("HEAD", subprocess.run(["git","-C",str(ROOT),"rev-parse","--short","HEAD"],capture_output=True,text=True).stdout.strip())
for p in inputs: print(hashlib.sha256(p.read_bytes()).hexdigest()[:16], p.relative_to(ROOT))

notebook run 2026-10-19 seed 0
HEAD 0169fee
1ff7b06d99ef84cc experiments/2026-10-16-gpu-probe/results/bench.log
cd692d366597c82c experiments/2026-10-16-gpu-probe/results/bench_gpu1.json
42713bbb43909ad6 experiments/2026-10-16-gpu-probe/results/bench_gpu2.json
254196fcde6ce8fc experiments/2026-10-16-gpu-probe/results/spotcheck.txt
088ea3b969c2352e experiments/2026-10-16-gpu-probe/results/spotcheck_specs.txt
1b50cfee957766c3 experiments/2026-10-16-gpu-probe/results/bench_lanes.log
16898b7f1aacd600 experiments/2026-10-16-gpu-probe/results/bench_lanes_1024_r1.json
0c48c44010c5316a experiments/2026-10-16-gpu-probe/results/bench_lanes_1024_r2.json
5bcefaeac7e6291f experiments/2026-10-16-gpu-probe/results/bench_lanes_1536_r1.json
3e72a933e617bc1e experiments/2026-10-16-gpu-probe/results/bench_lanes_1536_r2.json
a3969565a4154fdd experiments/2026-10-16-gpu-probe/results/bench_lanes_2400_r1.json
1a1ac5d3160c00ee experiments/2026-10-16-gpu-probe/results/bench_lanes_2400_r2.json

Data and method

bash experiments/2026-10-16-gpu-probe/scripts/bench.sh (WSL, repo root) times, sequentially, gpu_probe 27 8192 --lanes 2400 (gpu1), probe 27 8192 (cpu), and gpu2 on the 2400 kappa-leadrun-fresh specs. Heavy work is in the Rust binaries; this notebook parses bench.log.

Code

In [2]:

log = (R/"bench.log").read_text()
wall = {m.group(1): float(m.group(2)) for m in re.finditer(r"(gpu1|cpu|gpu2) wall ([\d.]+) s", log)}
ident = re.findall(r"cmp vs dump: (\w+)", log)
print(wall, ident)
print(log.splitlines()[0])
for k in ("gpu1","gpu2"):
    s = json.loads((R/f"bench_{k}.json").read_text()); print(k, s)
print("speedup gpu1: %.2fx  gpu2: %.2fx" % (wall["cpu"]/wall["gpu1"], wall["cpu"]/wall["gpu2"]))

{'gpu1': 177.355451168, 'cpu': 681.648633111, 'gpu2': 176.949274154} ['identical', 'identical', 'identical']
git 53ccccc dirty=60 2026-10-17T17:38:27+13:00
gpu1 {'jobs': 2400, 'wall_secs': 160.9595, 'device': 'NVIDIA RTX 4000 Ada Generation', 'lanes': 2400, 'grid': 8192, 'slice': 1048576, 'max_log2': 27, 'pool_tiles': 2409257, 'launches': 128, 'oom_requeues': 0}

gpu2 {'jobs': 2400, 'wall_secs': 161.2787, 'device': 'NVIDIA RTX 4000 Ada Generation', 'lanes': 2400, 'grid': 8192, 'slice': 1048576, 'max_log2': 27, 'pool_tiles': 2409257, 'launches': 128, 'oom_requeues': 0}
speedup gpu1: 3.84x  gpu2: 3.85x

Superseded 2026-10-19 (see "P2 re-measure" below): this 3.84x is from a stale binary; at HEAD P2 is falsified from above (5.81x, CPU baseline shift unexplained).

Results

GPU 177.4 s / 176.9 s vs CPU 681.6 s: 3.84x, all outputs byte-identical to the existing CPU dump (oom_requeues 0, pool 2.41e6 tiles).

Shows: the GPU port is exact on this workload and about 3.8x faster, inside the pre-registered 2x-6x (P2 held; P1, P3 held). Does not show: a general speed claim; n = 1 CPU run, one spec set, one GPU; lane scaling (P4) was not run.

Caveats and controls

The first GPU figure (177 s, 20:58 on 10-16) and the old CPU time (1231 s) were contaminated by other jobs; the clean CPU time is 681.6 s. A handover estimate of ~7x used the contaminated number and is superseded.
Registered pool estimate (1.6e6 tiles) was about 50% low; no requeues.
24-spec spot check: spotcheck.txt (byte-identical).

P4: throughput vs lanes (added 2026-10-18)

scripts/bench_lanes.sh runs gpu_probe 27 8192 --lanes L on the same 2400 specs for L = 1024, 1536, 2400, two repeats each, interleaved, with a freshly built HEAD binary (sha256 91c072f7...). Registered prediction: steps/s rises with lanes from 1024 to 2400. Criterion fixed before running (NOTEBOOK.md): mean time strictly falls 1024 > 1536 > 2400, each gap larger than the larger repeat spread, same order on the device path. Steps/s uses one numerator (total_steps, identical across L because outputs are).

Code

In [3]:

import statistics as st
llog = (R/"bench_lanes.log").read_text()
print(llog.splitlines()[1])
assert len(re.findall(r"cmp vs dump: identical", llog)) == 6, "not all phases byte-identical"
assert all(l.strip() == "foreign procs:" for l in llog.splitlines() if "foreign procs" in l), "contaminated phase"
wallp = {(int(m.group(1)), int(m.group(2))): float(m.group(3)) for m in re.finditer(r"lanes_(\d+)_r(\d) exit 0 wall ([\d.]+) s", llog)}
rows = {}
for L in (1024, 1536, 2400):
    s = [json.loads((R/f"bench_lanes_{L}_r{k}.json").read_text()) for k in (1, 2)]
    assert len({x["total_steps"] for x in s}) == 1 and s[0]["oom_requeues"] == 0
    w = [wallp[(L, k)] for k in (1, 2)]; d = [x["wall_secs"] for x in s]
    rows[L] = dict(wall=st.mean(w), wspread=max(w)-min(w), dev=st.mean(d), dspread=max(d)-min(d), steps=s[0]["total_steps"], launches=s[0]["launches"])
print("lanes  wall(mean, spread)   device(mean, spread)   steps/s(dev)  per-lane  launches")
for L, r in rows.items():
    sps = r["steps"]/r["dev"]
    print(f"{L:5d}  {r['wall']:7.2f} {r['wspread']:5.2f}       {r['dev']:7.2f} {r['dspread']:5.2f}        {sps:.3e}   {sps/L:.3e}  {r['launches']}")
Ls = (1024, 1536, 2400)
def holds(key, spr):
    big = max(rows[L][spr] for L in Ls)
    return all(rows[a][key] - rows[b][key] > big for a, b in zip(Ls, Ls[1:]))
print("P4 held:", holds("wall", "wspread") and holds("dev", "dspread"))

gpu_probe sha256 91c072f715cf5da6b5e2aaf380551d188d8c7d1058d60a2968cfb9c14b3653d3 mtime 2026-10-18 19:57:05.460035117 +1300
lanes  wall(mean, spread)   device(mean, spread)   steps/s(dev)  per-lane  launches
 1024   280.30  0.10        278.13  0.02        9.820e+08   9.590e+05  319
 1536   219.35  0.12        217.21  0.10        1.257e+09   8.186e+05  256
 2400   156.44  0.40        154.26  0.19        1.771e+09   7.378e+05  128
P4 held: True

Shows: P4 held. Batch steps/s rises 1.28x (1024 to 1536) and 1.41x (1536 to 2400), and the repeats agree to within 0.4 s. All six outputs are byte-identical to the CPU dump, the first full-scale identity check of the HEAD binary.

Does not show: lane-level parallel efficiency. Below 2400 lanes, ants queue (about 2.3 and 1.6 job-rounds), so most of the rise is just queueing. Per-lane steps/s actually falls 23% from 1024 to 2400 lanes; the cause is untested. Single spec set, n = 2 per L, one GPU.

Provenance caveat: the P2 timing above used a stale binary built 2026-10-16 20:46, not 53ccccc. Cargo did not rebuild because the source mtimes on /mnt/f predate later edits (README, NOTEBOOK 2026-10-18). Do not divide P4 times into P2's.

P2 re-measure on a verified HEAD build (added 2026-10-19)

scripts/bench_p2.sh: fresh target dir, build exit status checked, binaries verified newer than sources, sha256 logged; gpu1, cpu, gpu2 sequential on the same 2400 specs. Prediction (registered 2026-10-16, not edited): 2x-6x, falsified at >= 4.9x or < 2x.

Code

In [4]:

plog = (R/"bench_p2.log").read_text()
pw = {m.group(1): float(m.group(2)) for m in re.finditer(r"(gpu1|cpu|gpu2) exit 0 wall ([\d.]+) s", plog)}
assert len(re.findall(r"cmp vs dump: identical", plog)) == 3 and "cmp cpu vs gpu1: identical" in plog
print(plog.splitlines()[0]); print(plog.splitlines()[2]); print(plog.splitlines()[3])
print(pw)
print(f"speedup gpu1: {pw['cpu']/pw['gpu1']:.2f}x  gpu2: {pw['cpu']/pw['gpu2']:.2f}x; registered falsification line 4.9x")
print(json.loads((R/"bench_p2_gpu1.json").read_text())["wall_secs"], json.loads((R/"bench_p2_gpu2.json").read_text())["wall_secs"])

git 6de71f2 dirty(src,tests,Cargo.toml; filemode ignored)=0 2026-10-19T08:32:09+13:00
probe sha256 7d341949a912bc14a7b1c83f72a4fab14d41e6868e3717b7a6bfc19ecb190a29 mtime 2026-10-19 08:31:55.946175713 +1300
gpu_probe sha256 91c072f715cf5da6b5e2aaf380551d188d8c7d1058d60a2968cfb9c14b3653d3 mtime 2026-10-19 08:31:56.050175429 +1300
{'gpu1': 157.056047201, 'cpu': 912.48550584, 'gpu2': 156.938488521}
speedup gpu1: 5.81x  gpu2: 5.81x; registered falsification line 4.9x
154.7854 154.7216

Shows: all outputs byte-identical; 912.5 s / 157.0 s = 5.81x, so P2 is falsified at HEAD from above (>= 4.9x). Does not show: why. GPU time moved little (177 -> 157 s); the CPU baseline rose from 681.6 s to 912.5 s (+34%) for an unexplained reason (the old CPU binary came from the suspect target dir; the src/probe.rs refactor and machine state are candidates, not bisected). n = 1 CPU run, so quote the speedup as roughly 4-6x, not 5.8x. Tests on this build: cargo test --release ok; --features cuda --test gpu --test onset --test gpu_probe 7/7, 1/1, 5/5.

What this changes, next steps, open questions

Use gpu_probe for batches of hundreds+ plain-rule runs, with lanes >= number of jobs when they fit (P4). P2 was rerun on a verified build (5.81x, falsified from above). Next: find why the HEAD CPU probe is 34% slower than the old one; find why per-lane throughput drops with occupancy; optional one-byte packing for n <= 32 if VRAM binds. Build hygiene: on /mnt/f, do not trust cargo's freshness check for timed binaries; log and check the sha.

Read the dated lab log

Lab notebook: GPU probe

Append-only. Newest entry last. Never edit an old entry; correct it with a new one (corrects: <timestamp>). Tags: intent, assumption, observation, interpretation, surprise, decision, dead-end, question, deviation. See experiments/README.md.

2026-10-16 20:47 (+13) · 5b97961 (dirty tree) · intent

Task: port probe (plain rules and RULE@SEED) to the GPU with byte-identical CSV, so probe 27 8192 batches stop using the CPU. User message mid-session: "can you use cuda to avoid the cpu contention?" - taken as: the point of the port, and do the work on the GPU where possible, no CPU-heavy runs while their job is live. Plan: refactor the CPU window/CSV code into shared functions, new kernel + host module + binary + tests, register predictions, then time.

2026-10-16 20:48 (+13) · 5b97961 · observation

The user's own probe 27 8192 batch (3336 mutant specs, pid 2027659, ~590% CPU, started 20:36) is live. Per the task I do not benchmark while it runs and I do not touch it. My builds and tests run under nice, CPU oracles with RAYON_NUM_THREADS=2. Footprint estimate from the existing 2400-run dump (kappa-leadrun-fresh/dumps/probe.csv, last row per run): sum of bbox tiles (w/64+2)(h/64+2) = 1.61e6 (mean 670, max 10100 per run); 365 of 2400 runs escaped the 8192 board. At 8 KiB per tile that is about 13 GB, so it fits in 20 GB VRAM and a 16-bit cell is affordable. Also: the earlier kappa-leadrun probe wall time (19:46:58 -> 20:07:29 = 1231 s) overlapped the GPU-sweep benchmarks (see 2026-10-16-gpu-ant-sweep/NOTEBOOK.md, deviation), so it is not a clean CPU baseline; the CPU baseline is re-measured here.

2026-10-16 20:4x (+13) · 5b97961 · decision

Cell layout: one u16 per cell (colour | unv+1), 8 KiB tiles, instead of two byte planes (the task suggested measuring both). Reason: a single load per step gives everything. NOT measured against two planes; recorded as untested. One-byte packing (n <= 32) is the fallback if VRAM binds. Scheduler duplicated from GpuSweeper::run rather than generalised, so tests/gpu.rs keeps guarding an untouched sweep.

2026-10-16 20:4x (+13) · 5b97961 · decision

Patch init at tile allocation for the patch cells inside that tile (not "when the first tile is allocated" as the task suggested): at size 8192, c = 4096 is a multiple of 64, so the 5x5 patch spans four tiles. Tests include grids with c = 63, 64, 65 for this.

2026-10-16 20:4x (+13) · 5b97961 · observation

CPU refactor check: old binary saved as ~/probe_old before editing; new probe vs probe_old on 18 mixed specs (plain, @, rotor:, rw:, sbg:, brotor:, nrotor:, pw:, cookie:, rcookie:, errw:), at 17 1024 and 15 8192: cmp identical both times (149 and 125 lines). First gpu_probe vs probe_old on the ant specs (17 1024: 59 lines, 15 8192: 50 lines; 16 lanes, odd slice 100003): byte-identical on the first try.

2026-10-16 20:4x (+13) · 5b97961 · observation

cargo test --release --features cuda --test gpu_probe: 5/5 pass in 109 s (48 proptest cases x 7 rigs: one-tile 64, ragged 100, c=63/64/65 grids, tight pool of 16 tiles, 64 lanes at min_log2 8; scheduling independence by reversing job order; starved pool with oom_requeues > 0 asserted; mid-window escape; error paths).

2026-10-16 20:55 (+13) · 5b97961 · observation

Full-scale spot check (scripts/spotcheck.sh, every 100th spec of kappa-leadrun-fresh, 24 specs, 27 8192): GPU CSV byte-identical to the new CPU probe (439 rows, sha256 84fa1fa8..., 2 of 24 runs escaped the board). CPU side ran 1 thread niced (291 s) because the user's batch was live; GPU 54 s for 24 ants on 32 lanes. Output: results/spotcheck.txt.

2026-10-16 20:58 (+13) · 5b97961 · observation

All 2400 specs on the GPU (gpu_probe 27 8192 --lanes 2400), compared with the existing kappa-leadrun CPU dump (2026-10-16-kappa-leadrun-fresh/dumps/probe.csv): cmp IDENTICAL; oom_requeues 0; wall 177 s (device path 161 s). This run is NOT a benchmark: the user's 3336-spec probe was still running on 6 cores until 20:53:47 (their log), and the CPU baseline needed for a ratio had not been measured. Kept in scratch/ only. Prediction P1 (second half) and P3 hold.

2026-10-16 20:59 (+13) · 5b97961 · deviation

The user's batch finished at 20:53:47, but at ~20:59 a different session started building my tree into ~/turmite-target-probeport and running gpu_probe 27 8192 --lanes 2400 (pid 2051469, not mine). I did not touch it. A timing run now would overlap it on the GPU (and its CPU part on 1 core), so I wait until no release/ or cargo process has been seen for 3 minutes (~/quiet_649.sh) and then run scripts/bench.sh (gpu1, cpu, gpu2, never overlapping, with before/after process checks).

2026-10-16 21:02 (+13) · 5b97961 · decision

User said "pause" and then "prepare to wrap up for the day.. write a handover doc". Stopped before the clean benchmark; killed only my own wait helper. Wrote HANDOVER.md. State: code and correctness done (tests 5/5, 24-spec and 2400-spec byte-identity); timing (P2), the rest of the test suite after the src/probe.rs refactor, notebook.ipynb and the commit remain. The one 177 s GPU figure is contaminated (user's CPU batch live) and must not be cited as a benchmark. If the clean ratio lands near 7x, P2 (2-6x) is falsified on the fast side.

2026-10-17 17:56 (+13) · 53ccccc · observation

Clean benchmark (scripts/bench.sh, idle machine, phases sequential): gpu1 177.4 s, cpu 681.6 s, gpu2 176.9 s, all three CSVs byte-identical to the kappa-leadrun CPU dump. Speedup 3.84x. P2 (2x-6x, central 3.5x) HELD; P3 held (0 requeues); P1 held; P4 untested.

2026-10-17 17:57 (+13) · 53ccccc · interpretation

The handover's "about 7x, P2 falsified" was wrong: it divided by a CPU time (1231 s) contaminated by the GPU-sweep benchmarks. The clean CPU time is 681.6 s. The GPU figure was the same with and without the user's batch running (177 s both times), so the GPU path was never the contaminated side. Pool use (2.41e6 tiles) exceeded my 1.6e6 bounding-box estimate by about 50% without requeues. Single spec set, so 3.84x is a measurement for this workload, not a general speed claim.

2026-10-17 17:58 (+13) · 53ccccc · observation

cargo test --release (plain) and --features cuda --test gpu --test onset pass after the src/probe.rs refactor (plain log was tail-only).

2026-10-18 19:37 (+13) · 14b9a24 · intent

Run P4 as registered in the README ("Steps/s rises with lanes from 1024 to 2400 (every ant resident)"), text unchanged. Recipe: scripts/bench_lanes.sh, gpu_probe 27 8192 --lanes L on the same 2400 kappa-leadrun-fresh specs as P2, L in {1024, 1536, 2400}, two repeats each in interleaved order 2400,1024,1536,1536,1024,2400. Every CSV is cmp'd against the CPU dump. Operational criterion, fixed now: P4 holds if the mean whole-process wall time strictly decreases 1024 > 1536 > 2400 with each gap larger than the larger within-L repeat spread, and the same order holds for device-path time (wall_secs in --stats). Steps/s uses one numerator for all L (outputs are identical, so total_steps, the budget of non-escaped runs, is the same), so the ratio of steps/s is the inverse ratio of times. Caveat registered now: at L < 2400 ants queue and lanes are refilled between slices, so lower L also means more slices and more host turnarounds, and the 2400 case has an all-ants-resident tail with idle lanes that the queued cases partly avoid. The test therefore compares batch throughput, not per-lane throughput at fixed occupancy.

2026-10-18 19:37 (+13) · 14b9a24 · deviation

Machine not idle: another session's run_mutants.sh (worktree .claude/worktrees/paper-figure-proposals, CPU probe 27 8192 on 3336 mutant specs, 6 threads, started 19:34:34) is running. Not mine; not touched. bench_lanes.sh waits until no release/ or cargo process has been seen for 3 minutes before its first phase. The gpu_probe binary is rebuilt from HEAD (writes total_steps, which the P2 build did not), so its sha256 differs from bench.log's; same kernel source as 53ccccc (no src changes since).

2026-10-18 19:45 (+13) · 14b9a24 · surprise

The P2 benchmark (results/bench.log, header "git 53ccccc") did not run a binary built from 53ccccc. The binary at ~/turmite-target/release/gpu_probe before this session's rebuild had sha256 4d3e395d... (the value in bench.log) and mtime 2026-10-16 20:46:17, and its output format ("2400 jobs in 160.96s", --stats without total_steps/steps_per_sec) does not match 53ccccc's src/bin/gpu_probe.rs, which prints and writes both. So bench.sh's cargo build ... | tail -1 either failed or did not rebuild, and the pipe hid it. Consequence for P2: the timed binary is the 2026-10-16 20:46 build, the one that produced the 24-spec and 2400-spec identity checks that evening; its kernel and scheduler may differ from 53ccccc in ways I cannot reconstruct. The outputs were byte-identical to the CPU dump, so P1's conclusion stands for that binary; the 3.84x is a measurement of that binary, not provably of 53ccccc. Copy kept: scratch/p2_binary/gpu_probe (gitignored). bench_lanes.sh builds after the quiet wait, checks the build's exit status and logs mtime.

2026-10-18 19:45 (+13) · 14b9a24 · correction

In the 19:37 deviation entry, "same kernel source as 53ccccc (no src changes since)" is true of HEAD vs 53ccccc but says nothing about the P2 binary, which (previous entry) was not built from 53ccccc. P4 is therefore self-contained: its own 2400-lane phases are the baseline, and P4 numbers should not be divided into or by P2's.

2026-10-18 19:40 (+13) · 14b9a24 · correction

The two entries stamped "19:45" (surprise, correction) were written at about 19:38; I mis-stamped them. Content unchanged. bench_lanes.sh launched 19:38 (pid 165521), waiting for the other session's mutant batch (800/3336 at 19:39, expected done ~19:56).

2026-10-18 20:00 (+13) · 14b9a24 · observation

Cause of the stale P2 binary found. src/bin/gpu_probe.rs, src/gpu/probe.rs, src/gpu/probe.cu on /mnt/f carry mtimes 2026-10-16 20:41-20:43, older than the binary (20:46), although their content changed later (current source prints "steps in" and "setup", the binary does not). Cargo's freshness check therefore kept the old binary: bench_lanes.sh's build step "finished in 0.08s" and the first P4 phase started (19:55:19) with sha256 4d3e395d..., the P2 binary. I stopped that run (my own pid, killed after ~1 min), kept its log as scratch/bench_lanes_aborted_stale.log, deleted its partial CSV. Wider hazard: anything built in ~/turmite-target since 10-16 may be stale the same way, including the README's "5/5 gpu_probe proptests" and post-refactor cargo test claims.

2026-10-18 20:01 (+13) · 14b9a24 · decision

Build into a fresh CARGO_TARGET_DIR=~/turmite-target-p4 (did compile turmite, 8.5 s incremental of deps; sha256 91c072f7...; contains "steps in"). bench_lanes.sh now exports that dir and aborts unless the binary's sha equals EXPECT_SHA. The first phase's cmp against the CPU dump is the first full-scale identity check of the HEAD binary; if it differs, P4 timing is secondary and I stop. Source mtimes not touched. After the phases, if time allows, cargo test --release --features cuda --test gpu_probe in the fresh dir.

2026-10-18 20:27 (+13) · 14b9a24 · observation

bench_lanes.sh finished 20:25:24, all six phases clean (no foreign process before/after any phase; load ~1). Binary 91c072f7... (fresh build). Every CSV byte-identical to the CPU dump: first full-scale identity check of the HEAD binary, passed 6/6. Whole-process wall r1/r2: L=2400 156.64/156.24 s, L=1536 219.30/219.41 s, L=1024 280.34/280.25 s. Device path (wall_secs): 154.35/154.16, 217.16/217.26, 278.14/278.12 s. total_steps 273133076480 in every run (as expected, outputs identical). Launches 128/256/319; 0 requeues; pool_tiles 2409451/2416681/2420965 (2400/1536/1024).

2026-10-18 20:28 (+13) · 14b9a24 · interpretation

P4 HELD on the pre-fixed criterion: mean times strictly fall with L, gaps ~61-63 s vs repeat spread <= 0.40 s, same order on the device path. Device steps/s 0.982e9 / 1.257e9 / 1.771e9. This is mostly queueing: with refill, 2400 jobs at L=1024 is ~2.3 job-rounds, at 1536 ~1.6, at 2400 one. Per-lane steps/s falls with occupancy (9.59e5, 8.19e5, 7.38e5 per lane; -23% from 1024 to 2400), so the GPU is not idle-lane-bound at 1024 but loses per-lane speed as resident ants grow; cause untested (memory traffic is the guess). The test measures batch throughput on one spec set with n = 2 per L, not per-lane efficiency at fixed occupancy. HEAD at L=2400 (156 s) is faster than the stale P2 build (177 s); not compared further since the binaries differ and no new CPU baseline was taken.

2026-10-18 20:40 (+13) · 14b9a24 · observation

cargo test --release --features cuda --test gpu_probe on the fresh build (~/turmite-target-p4): 5/5 pass. Plain cargo test --release and --test gpu --test onset were last run in the possibly stale ~/turmite-target and are not yet rerun on a verified build. Open: paired P2 re-measure (rebuild probe at HEAD, re-time CPU, ~11 min).

2026-10-19 08:25 (+13) · 6de71f2 · intent

Finish HANDOVER items 1-4. Machine idle (no release/ or cargo processes; GPU 0-20% from the other adapter). New scripts/bench_p2.sh builds probe and gpu_probe in a fresh ~/turmite-target-p2, checks exit status and that binaries are newer than the sources, logs sha256, then times gpu1, cpu, gpu2. No tests while it runs.

2026-10-19 08:53 (+13) · 6de71f2 · observation

bench_p2.sh exit 0, no foreign process before/after any phase. Binaries: probe 7d341949..., gpu_probe 91c072f7.... Whole-process wall: gpu1 157.06 s, cpu 912.49 s, gpu2 156.94 s; device path 154.79/154.72 s. All three CSVs cmp-identical to the kappa-leadrun dump; cpu vs gpu1 identical. Speedup 5.81x.

2026-10-19 08:55 (+13) · 6de71f2 · interpretation

By the criterion registered 2026-10-16 (falsified if >= 4.9x), P2 is falsified at HEAD, from above. The stale-build 3.84x ("held") is superseded. The GPU time barely moved (177 -> 157 s); the CPU baseline went 681.6 -> 912.5 s (+34%) and that drives the ratio. Why the CPU probe is slower is unexplained (old binary f1a309cf from the suspect dir; possible code change in the src/probe.rs refactor, or machine state). Not bisected. Prediction text not edited.

2026-10-19 09:10 (+13) · 6de71f2 · observation

Verified-build tests (~/turmite-target-p2): cargo test --release all ok; --features cuda --test gpu --test onset --test gpu_probe: 7/7, 1/1, 5/5. bench.sh and spotcheck.sh fixed (build exit status, stale-binary check). The earlier stale-dir test passes are superseded.

2026-10-19 09:15 (+13) · 6de71f2 · question

Open: why is the HEAD CPU probe 34% slower than the old one? A cheap check is to time probe 27 8192 on a 100-spec subset with the old commit (before the src/probe.rs refactor) built in a fresh dir. Not done.

The idea under test

What we learned

What this leaves open

Inside the experiment

GPU probe: probe on a CUDA device

Question

Design (decided before any timing)

Predictions (registered 2026-10-16 20:49 NZDT, before any timing)

Use it

Results

P4: throughput vs lanes (2026-10-18)

P2 re-measure on a verified HEAD build (2026-10-19)

GPU probe: probe on a CUDA device

Question and why it matters

Assumptions and definitions

Provenance

Data and method

Results

Caveats and controls

P4: throughput vs lanes (added 2026-10-18)

P2 re-measure on a verified HEAD build (added 2026-10-19)

What this changes, next steps, open questions

Lab notebook: GPU probe

2026-10-16 20:47 (+13) · 5b97961 (dirty tree) · intent

2026-10-16 20:48 (+13) · 5b97961 · observation

2026-10-16 20:4x (+13) · 5b97961 · decision

2026-10-16 20:4x (+13) · 5b97961 · decision

2026-10-16 20:4x (+13) · 5b97961 · observation

2026-10-16 20:4x (+13) · 5b97961 · observation

2026-10-16 20:55 (+13) · 5b97961 · observation

2026-10-16 20:58 (+13) · 5b97961 · observation

2026-10-16 20:59 (+13) · 5b97961 · deviation

2026-10-16 21:02 (+13) · 5b97961 · decision

2026-10-17 17:56 (+13) · 53ccccc · observation

2026-10-17 17:57 (+13) · 53ccccc · interpretation

2026-10-17 17:58 (+13) · 53ccccc · observation

2026-10-18 19:37 (+13) · 14b9a24 · intent

2026-10-18 19:37 (+13) · 14b9a24 · deviation

2026-10-18 19:45 (+13) · 14b9a24 · surprise

2026-10-18 19:45 (+13) · 14b9a24 · correction

2026-10-18 19:40 (+13) · 14b9a24 · correction

2026-10-18 20:00 (+13) · 14b9a24 · observation

2026-10-18 20:01 (+13) · 14b9a24 · decision

2026-10-18 20:27 (+13) · 14b9a24 · observation

2026-10-18 20:28 (+13) · 14b9a24 · interpretation

2026-10-18 20:40 (+13) · 14b9a24 · observation

2026-10-19 08:25 (+13) · 6de71f2 · intent

2026-10-19 08:53 (+13) · 6de71f2 · observation

2026-10-19 08:55 (+13) · 6de71f2 · interpretation

2026-10-19 09:10 (+13) · 6de71f2 · observation

2026-10-19 09:15 (+13) · 6de71f2 · question

GPU probe: `probe` on a CUDA device

GPU probe: `probe` on a CUDA device