Metabolism: rule program, song-driven life, tribe VMs; M2.7 design (strains & allegiances, living wells)

metabolism/EVOLUTION.md

@@ -0,0 +1,233 @@
+# evolution — from a petri dish to a world
+
+Course correction after watching the demo, agreed with the code:
+
+**What works.** The song→analysis→events→wells→laws front-end genuinely connects
+to the sim (coupling/residual and fingerprint divergence are real). The
+determinism discipline is intact and now proven bit-for-bit in the browser.
+The demo shell and the tools share one `Run`.
+
+**What failed.** The *world behind the Run*. All songs grow the same three
+castes on the same 128² dish with the same opening and the same laws-as-multipliers.
+Statistically different, structurally identical. Two skins, not two habitats.
+And the camera watches it at the microscopic scale, so nothing larger ever has
+room to appear.
+
+**The correction** is one idea: the world must be a *macro* ecology — population
+density, not individual dots — governed by *structural rules* the song compiles
+to, watched by a camera that *travels* instead of cutting.
+
+---
+
+## Milestones
+
+### M0 — agreement ✔
+Representation: **population fields**. Rules: **a scoped rule program** (compiled
+per song, deterministic). Plan: this document. `DESIGN.md` updated to the new
+end-goal.
+
+### M1 — the macro world: density, substeps, geography ✔
+The entity sim is replaced by a population-field sim.
+
+- **World**: `FIELD=128` integrated density grid (rendered as `RENDER_FIELD=256`
+  speck texture); layers: food, three caste populations, three tinted trail
+  fields, wells, a static seeded jitter pair.
+- **Time**: `SUBSTEPS=5` sim steps per rendered frame (the old world: 1), so
+  local interactions propagate into coherent macro patterns.
+- **Physics**: forces are integrated on a coarse 64² grid (2×2 block sums —
+  long-range coupling at a fraction of the cost), then advected on the fine
+  grid with a square-root-free L1 axis split. Reaction: eat → logistic growth
+  with a total-density gate → deposit; starvation on empty ground. Events still
+  splat food and bloom the voice that spoke; rings land on strong hits.
+- **Home basins**: each caste anchors to its own wells (the field-world
+  "builder orbits its well"), so colonies persist as continents instead of
+  melting into a uniform haze.
+- **Look**: trails compose the geography; density renders as a glow plus
+  deterministic speck swarms — "billions of little guys" is texture, at zero
+  entity cost.
+- **Opening**: the dish wakes into the song's own first bars.
+
+*Outcome:* at 60 s the population is still geographically structured (8-block
+CV 1.5–1.9 vs ~0.1 before), songs and seeds diverge at a glance in-browser,
+openings differ per song, realtime holds at 640×360 (960×540 on reasonable
+hardware), and the test suite is green including a prepared-artifact
+bit-identity check. *M2 then replaced the multiplier laws with the rule
+program; the camera's travel grammar is M3.*
+
+### M2 — the rule program: structure, not knobs ✔
+The laws-as-multipliers are replaced by a **scoped rule program**
+(`src/song/rules.js`) — declarative per-song rules:
+
+- **castes** are rule entries: speed / seek / dispersion / cap / growth / eat /
+  deposit, plus *growth rules* — mutualism (a caste breeds fastest among the
+  others, so mixed crowds beat monochrome blobs) and a social floor (sparks
+  only bloom in a luminous crowd);
+- **interaction rules** per caste pair (the signed band-correlation graph);
+- per-section **operator regimes** — a breakdown drives disperse≈0 and
+  deposit high (the world settles and inscribes); a drop drives home≈0 and
+  turbulence+spawn high (the flood). Structural, scoped by the track's own
+  character (noisy → turbulence up, tonal → deposit up);
+- the **event→response** mapping: which voice feeds which caste, and how much
+  of every hit splashes to the others;
+- **memory** (trail/field half-lives, diffusion) and a **global carrying
+  capacity** so one song cannot flood while another starves;
+- compiled once per song, deterministically — the program is part of the song,
+  rebuilt bit-identically by the prepared-artifact path, and v3's rule search
+  now searches *programs*, not floats.
+
+*Outcome:* 23 tests green, including: program compilation is deterministic;
+different songs get structurally different programs; regimes are structurally
+distinct by kind (a growth=0 edit visibly halts the world — the program is
+executed, not decoration); genesis pre-simulates generations so frame 0 is
+already an ecosystem, with the amount tuned per song; the opening is seeded
+where the song's own hits landed. Mixing is a program rule: castes co-mingle
+in one shared geography (drone 0.93, dusk 0.78, runner 0.48 block-mix; was
+~0.02–0.15), the lens holds the whole world (zoom ~0.9–1.6), and divergence
+between songs grew to 16–40× where seeds stay ~0.01–0.05.
+
+**M2.5 — song-driven life, more generations, higher altitude ✔**
+- *The song spawns:* colonies are no longer placed at a pre-chosen set of
+  spots. Every hit lands food *and* births life there (the voice first and
+  loudest, splashes to the others); the drop regime visibly floods the ground
+  with new colonies where the track hits.
+- *Genesis:* the world pre-simulates several generations before frame 0 (the
+  amount comes from the song — intro length, onset density, pulse), so the
+  opening is an old, settled ecosystem, never a blank plate, and the film's
+  history starts clean at frame 0.
+- *Altitude:* the lens sits higher (zoom ~0.6–1.5), holding the whole map —
+  different cultures and the roads and fronts between them — in frame.
+
+### M2.6 — the tribes are programs ✔
+The M2 rule program steered *how hard* behaviours ran. Tribes go deeper: each
+tribe is a small **deterministic behaviour program** (`src/song/tribe.js`) — a
+tiny VM with
+
+- **sensing** (food, own/other densities, trail, law, hunger, rim-edge, time),
+- **memory** (four persistent registers, smoothed by the track's pulse),
+- **actions** (move, excrete, eat, hit, spawn),
+- **basic branching** (if hunger > threshold → seek food, else hold-and-ink),
+
+compiled **by the song**: the band-correlation matrix signs who a tribe
+approaches vs flees, flatness sets how much it branches on gradients, onset
+density sets its spawning, the seed jitters thresholds. The world no longer
+executes a hand-tuned force formula on the coarse decision grid — it runs what
+each tribe's program decides, every substep. Surprising behaviour is *possible*
+precisely because the programs branch: a starving tribe behaves differently
+from a fed one, and the threshold is musical.
+
+*Outcome:* the three societies are now balanced and mixed (drone 0.86 /
+runner 0.87 / dusk 0.95 block-mix; populations 40–76 k, castes near-equal), the
+geography holds (population CV 2.2–2.8). New tests pin the VM: programs compile
+deterministically and differ by song (restless/social/hoarding/breeding
+archetypes selected from the stats); branching changes behaviour; registers
+persist as memory; a no-op program visibly changes the film. Realtime holds at
+640×360 (tribe runs add ~4 ms/frame at 960×540 in node).
+
+*Honest limit:* the compiler's vocabulary is still a small archetype family, so
+the *structural* divergence between songs is modest (cross ≈ 0.16–0.18 vs
+within-seed ≈ 0.02–0.04, ~4–7×) even though the films look different. Broadening
+the instruction grammar (more branch shapes, richer memory, more action
+archetypes) is the natural next step for this layer.
+
+### M2.7 — one species, many minds: strains & allegiances (design agreed; not yet built)
+
+A further correction, agreed before building anything on M2.6: **drop tribes
+entirely.** Not three castes, not k tribes of fixed membership. The picture is
+*one population of specialised microorganisms building a bigger organism* —
+and then several such macro-organisms sharing a map, sometimes fighting,
+sometimes cooperating. Membership is a property of each organism, and it can
+change over time.
+
+**Wells keep the correction from the previous draft** — small, weak, many
+(8–16), born where a pitch class enters and holds, drifting with its harmony,
+dying when it leaves, all on a precompiled deterministic timeline — because
+wells are where bodies collide: everyone needs them, so a well is the natural
+front, its birth a land-rush, its death a retreat.
+
+**The population is one density field with two live properties:**
+
+- **strain mix (personality).** The song decides how many minds it needs —
+  *m* strains from the number of independent voices the mix resolves into,
+  bounded by arrangement depth, jittered by the seed — and compiles a
+  **LittleVM program per strain** from the grammar: senses (food, density,
+  trail, law, hunger, rim, well proximity and liveliness, body presence),
+  registers as memory, real branching, actions (move, graze, excrete, hit,
+  spawn, follow, patrol). Density is partitioned by strain; each strain
+  advects by its own program. And the mix **evolves by local selection**:
+  strains that eat and grow where they stand propagate, proportions shift
+  district by district — specialisation is visible and develops over the film.
+- **allegiance (the body).** A composition field over bodies (the song opens
+  2–4; a cap of ~8 keeps colour legible), advected frozen-in with the
+  population flux. Same body = **cooperation**: shared trails, reinforcement,
+  complementary strains building one structure. Different bodies = **contest**
+  where presences overlap: hit-actions spend density to push the other body's
+  ground and convert it, scaled by the song's harmony — consonant, locked
+  voices breed truce and symbiosis; dissonant, decorrelated voices breed
+  skirmish and siege. Allegiance changes by rule, never by fiat:
+  - **conversion** — a won front converts the locals it overruns;
+  - **defection** — a body whose presence collapses (starved, outflanked)
+    releases its members to the neighbouring body;
+  - **fission** — a coherent splinter cut off from its body, hit by a strong
+    onset, founds a new body with a new colour;
+  - **merger** — two bodies whose members fully intermix under a cooperative
+    regime become one.
+
+**The song sets the diplomacy.** Consonance and the section regimes map to
+war and peace: drops are expansion wars, breakdowns are truces and mutual
+building, an outro is a settling of borders. The body-to-body relations come
+from the music, not a constant.
+
+*Exit: two songs with the same pulse but different arrangement depth compile
+different strain counts and different diplomacies; at least one allegiance
+change (conversion, defection, fission or merger) is legible on screen per
+film; strain proportions measurably drift across the film (local selection
+actually runs); a well's birth, drift and death are each legible; the world's
+typed-array layers, trail tints and palette generalise from 3 fixed castes to
+the strain/body fields; the prepared-artifact path still rebuilds
+bit-identical timelines, programs and opening state.*
+
+### M3 — the lens: travel, don't cut
+Replace the cut-to-a-point grammar with a **move grammar**: *track* (follow a
+moving front), *reveal* (pan along a coastline), *establish* (hold wide and
+slow). Cuts are reserved for musical punctuation — strong onsets in drop/build.
+Because the world is macro, a pan teaches the map instead of disorienting.
+
+*Exit: a 120 s render where a single front can be followed across the world
+without losing bearings; cuts read as beats, not jumps.*
+
+### M4 — the film
+Mux frames + PCM to video. No longer the bottleneck: the preview pipeline and
+the lens exist; once M1–M3 make the world worth exporting, muxing is plumbing.
+
+---
+
+## The prepare step (pre-calculation)
+
+Viewing should be fast because the *expensive* work happens ahead of time, once:
+
+1. `tools/prepare.js` runs the WAV → analysis → compile pipeline in node and
+   writes a **prepared song** (`out/prepared/<song>-<seed>.json`): the compiled
+   rule program and every analysis structure the world, camera, and metrics
+   need — but not the raw PCM.
+2. The browser fetches the prepared song instead of analysing the WAV. The WAV
+   is still fetched, but only to feed the WebAudio clock (decoded straight to a
+   buffer; analysis skipped).
+3. `src/song/serialize.js` is the single serialize/deserialize for both sides —
+   the browser's `deserializeSong` must rebuild a song that is *bit-identical*
+   to one built by analysing the WAV, so prepare can never drift from the live
+   path. That equality is a test.
+
+This is exactly the DESIGN promise inverted and kept: the pipeline still defines
+the world; prepare only pays its cost once, off the viewing path.
+
+## Invariants, restated for the field world
+
+- **Determinism**: same song + same seed + same frame ⇒ same pixel. No
+  `Math.random`, no wall clock, no per-substep RNG. The static jitter noise
+  field is generated once from the seeded RNG at construction.
+- **Causality**: events still place food and rings where the track put them;
+  a section boundary still changes the rules at the boundary only.
+- **Persistence**: population fields carry their history; trails accumulate in
+  the same three tinted layers the renderer already composes.
+- **Consequence**: everything is a pure update rule over the previous substep;
+  nothing is re-rolled.