music-video-gen/metabolism/DESIGN.md

# metabolism — design

> **Review note:** the redesign of the two goal systems — the wells and the
> LittleVM castes — is presented as a standalone proposal in
> [`DESIGN-REVISION.md`](DESIGN-REVISION.md) (deltas, new design, open
> questions). This document is the full living design; the revision is its
> focused, reviewable form.

A music video generator whose output is **grown**, not mixed. The track is not
a parameter set the renderer modulates; it is the *substrate* a simulation
grows on. Given the same song and the same seed, the same world — bit for bit,
frame for frame, preview and export identical.

```
      song (PCM)
         │  analyse            — whole track, up front, fixed-rate
         ▼
   feature table ──► score         events (what spoke, when, how hard)
   (per frame)   ──► field         food, where the track put it
                  ──► graph        relations between voices
                  ──► laws         per-section regime
         │  seed
         ▼
   species genome ─────────────── rules the agents live by
         ▼
   world ◄── agents on the field, laws switch at boundaries
         ▼
   camera ◄── cuts to where the world just changed
         ▼
   framebuffer ──► PNG (tools) / canvas (demo) / video (later)
```

---

## The idea

Most music visualisers **modulate** one fixed stage with the song: parameters
go up and down, and every song ends up presenting the same toy in different
clothes. That is cheap to watch and cheap to build, and it is exactly what
we are not doing.

We want each song to produce a world that looks and behaves like *that song*
and no other — a world with its own places, its own creatures, its own
disasters. The mechanism is **metabolism over modulation**: the song supplies
the resource field, the event impulses and the per-section laws; the seed
picks the species, and the species is a *rule set*, not a parameter vector.

Two songs therefore differ in the way two habitats differ, not in the way two
skins differ. A noisy fast track grows a swarm with jittered motion and thin,
ephemeral trails. A tonal slow track grows weavers with long memories and
heavy ground markings. Same renderer, same simulation — different songs, so
different laws, so different worlds.

### Why rules and not parameters

A parameter changes *how strongly* a behaviour applies. A rule changes *what
behaviour exists*. Parameters are a knob; rules are a diet. The strongest
variation-per-cost you can buy is changing which rules a creature obeys, not
tuning how much of a rule it obeys:

- small budget → different rules → wildly different outcomes
- large population → multiply across hundreds of interacting agents for free

The genre of a track steers the *family* of rules; the seed steers the exact
world, the way a seed steers an ecology's exact history while the biome is
determined by the land.

---

## Three invariants

Every design decision below exists to keep these alive. If an edit weakens
one of them, the edit is wrong.

1. **Persistence** — things last. Agents live for seconds; trails last tens
   of seconds. The audience can *follow* something across the film rather than
   decoding a new random field every frame.
2. **Causality** — every visible event has an audible cause and a visible
   one. A hit lands as a ring on the ground where the track put it. A bloom
   has a cause; a migration has one. Nothing happens for no reason.
3. **Consequence** — the state at t is made of the state at t−1. The field is
   depleted by eating; trails accumulate from walking. The world records its
   own history instead of being re-rolled each frame.

A fourth, transport-level invariant: **determinism**. Same song, same seed,
same frame → same pixel, in node and in the browser, at any wall-clock time.

---

## The analysis: four structures from one pass

A single FFT pass over the whole track (fixed 44.1 kHz decode, 2048-bin FFT,
30 fps) produces a per-frame feature table: band energy (low/mid/high),
spectral flux per band, centroid (log-scaled), flatness, a dominant pitch,
energy, and chronicle of onsets. From that table derive four structures:

### Score — events
Onsets picked per band with an adaptive threshold, merged so one transient
fires once (not three times for three bands). Each event carries a voice
(low/mid/high), a strength, and a time.

**Where an event lands is musical.** The voice picks the ring (kick near the
centre, body in the middle, hats at the rim), the dominant pitch class picks
the angle (12 semitones → 12 sectors), strength picks how much food and how
wide. The track's rhythm is literally folded into the geometry of the food.

### Field — food
Events splat resource onto a 128×128 grid. On top of that, **wells** — small,
fertile places with a life of their own. There are many of them (8–16) and
each is small and weak: a tight kernel, a modest recharge rate, no single well
rich enough to carry a world — the same total fertility budget the old
permanent springs had, spread thin. Where wells come from and when they go is
the track's own harmony:

- **birth** — a well sprouts where a pitch class enters the music and *holds*
  (sustained chroma energy, seeded by the events that already landed there);
- **drift** — a well wanders slowly with its harmony, pulled along the
  smoothed pitch/centroid path, and the creatures anchored to it follow;
- **death** — when a pitch class leaves the song, its well starves: the
  recharge fades, the kernel decays, the colony must abandon it.

The chord does not just draw the map — it runs the map's *history*. Wells are
the story's spine: founding, settlement, exodus, migration, collapse. Each has
an audible cause and a visible one, and each lasts long enough to be followed.
The whole lifecycle is compiled at analysis time into a deterministic **well
timeline** (birth, path, death per well); the sim only plays it back. Wells
are also where bodies meet: everyone needs them, so a well is the natural
front between macro-organisms — and its death is a retreat, its birth a
land-rush.

### Graph — relations
The band-envelope correlation matrix now works at two scales. It decides the
**division of labour** — which strain motifs complement which, so a bass line
locked with the kick pairs diggers with haulers — and it feeds the
**diplomacy**: the harmony the track sustains maps to how bodies treat each
other, truce and trade versus skirmish and siege. Society is a property of
the mix, at both scales.

### Flora, gardens, walls — the workable world
Three made things with feedback loops, each demanding its own instructions:

- **flora** — wild food in ISLANDS: a dozen scattered patches that are food,
  regrow on their own, and bloom when the world starves (the pioneers seed it
  back), receding under the grazing of a well-fed crowd. Never enough to
  sustain a large world; always somewhere worth walking to.
- **gardens** — stone tilled by masons into cultivated beds that yield food
  for a while, then set back into rock. Tended beds (the farmers' work)
  yield up to threefold. Agriculture slowly consumes the world's stone.
- **walls** — quarried burden built into barriers: they defend the
  settlement, enemies strike and sap them, and unmaintained walls weather
  away. Nothing on this map is finished.

All work is AIMED — the VM's work actions land where the agent faces, not
beneath it — and the work-senses look the same way.

### Rock — the inert resource
The map also carries **rock**: height per cell, impassable above a threshold.
The world is born with a deterministic scatter of it; the song is its weather
— strong low/mid hits drop stone (crushing what they land on), strong high
hits shatter it, drops rain it, breakdowns weather it away. Food does not
recharge through rock, so capped ground is barren until opened. Builders
**quarry** it (rock → carried burden, and the grinding yields soil), **carry**
it, and **build** with it — walls rise where the colony works. Rock is the
one thing on the map that is neither food nor life: it shapes where the
geography can go at all.

### Law — regime per section
Section boundaries are found where the 4 Hz feature cloud changes its mind
(cheap self-similarity, not an FFT of the FFT), classified into
intro / build / drop / breakdown / sustain / outro. Each section becomes a
*law*: a multiplier set applied to the standing rules at the boundary. A drop
speeds everything up and opens the floodgates for spawning; a breakdown slows
the world and doubles deposit. A section boundary is the only place the rules
change — which is what makes the boundary audible in the picture.

---

## The species

The genome maps the summary statistics to *kinds of behaviour*:

| song stats | breeding rule | consequence |
|---|---|---|
| flatness ↑ | jitter ↑ | noisy track = jittery creatures |
| flatness ↓ | trail half-life ↑, deposit ↑ | tonal track = heavy ground markings |
| beat strength | alignment and pace | a pulse makes them march |
| band correlations | division of labour | complementary voices breed complementary strains |
| onset density | separation, split cost | dense track = thronged dish but crowded apart |
| spectral tilt | hue | bright mixes warm, dark mixes cold |

There are no castes and no tribes. There is **one population** — a single
density field — and every organism in it carries two properties:

**Personality — the strain.** The song decides how many kinds of minds its
world needs (*m*, read from how many independent voices the mix resolves
into, bounded by arrangement depth, jittered by the seed) and compiles a
**LittleVM program for each strain**: grazer, builder, sentinel, scout,
breeder — each with its own senses (food, densities, trail, law, hunger, rim,
well proximity and liveliness, body presence), registers as memory, real
branching, and actions (move, graze, excrete, hit, spawn, follow, patrol).
The population is a *mix* of strains the way a biofilm is: specialists
working the same ground. And the mix is itself alive — strains that eat and
grow where they are propagate locally; proportions shift district by
district. The film shows specialisation *evolving* inside one species.

**Allegiance — the body.** Every organism also belongs to a macro-organism —
its **body** — a polity of mixed strains that builds, defends and expands as
one. Allegiance is a *field*: the population partitioned by body, advected
frozen-in with the organisms as they move. **Allegiance does not change
sides.** A body is founded once and persists; war kills, it does not convert;
there is no defection and no merger — territories stay legible because their
colours never homogenise. What keeps more than one body alive is
**colonisation**: a settlement on ground far from every body core, blessed by
a strong onset, may plant a NEW flag (a fresh colour) — rare, and it grows
the cast rather than reshuffling it.

Bodies are few — the song opens 2–4, capped at ~8 so colour stays legible —
and they are the drama. The film is the story of macro-organisms **fighting
and holding ground**: skirmish lines at fronts where the loser bleeds, truces
in consonant sections, symbiosis where one body's waste is another's food.
The song sets the terms of diplomacy: consonant, harmonically locked material
breeds peace; dissonant, decorrelated material breeds rivalry; drops are
expansion wars; breakdowns are truces and mutual building.

A species name like "fast tonal weaver" falls out of the genome, so the report
can talk about the world and not the track.

## The world

A fixed-step, garbage-free simulation over typed arrays: every RNG stream
seeded, every update a pure function of the previous substep. The course
correction (**EVOLUTION.md**) replaces the entity ecology with a
**population-field** ecology — the same front-end (events place food, wells
born from and living by the chord, laws at section boundaries) feeding
*density* instead of discrete agents, stepped several times per rendered frame so macro patterns
form. Nothing here is realtime: no player, no clock, no allocation — typed
arrays, a step function, and a history we can regress against the song.

## The lens

### Camera
An observer with a motive, not a rig on rails:

- **shots cut when the world changes**, never on a timer that ignores the
  world. Cuts also fire on strong onsets during drop/build, so the edit hits
  like the track hits;
- the **subject is recent activity** (decayed 32×32 cell grid), weighted away
  from the previous subject — novelty without drift;
- **shot sizes alternate** in a deliberate rhythm — a wide establishing, a
  medium, a close push — and zoom follows coverage: a broad field gets a wide
  lens, a crowded knot gets a close-up;
- while held, the camera drifts and pushes in — a shot is an approach, not a
  still.

### Renderer
A tiny software framebuffer (additive blend, discs, rings, lines, bilinear
world blit) so the same pixels render in node (PNG, filmstrips) and in the
browser (ImageData). Layers: near-black key-tinted background → trail+field
composed at field resolution with a tight blur (line) and a faint wide blur
(glow) → event rings → agents as bodies with velocity streaks → vignette →
deterministic grain.

The field is drawn faintly *under* the trails, so the audience can see *why*
the creatures are where they are.

## Metrics: system, not pixels

Per-frame descriptor distance was rejected long ago (it cannot tell a dungeon
from a car and rewards noise). Instead we score the whole episode against its
own song:

- **coupling** — how much the ecology answers the track: cue response (does
  movement edge up just after onsets?) plus long-form correlation (smoothed
  movement vs smoothed energy);
- **residual** — 1 − R² of the sim's activity (motion *and* the population
  curve) regressed on all song features: the internal dynamics the song does
  not explain;
- **fitness = coupling × residual** — the best world keeps time *and* has a
  life of its own. A pure visualiser scores ~0 (no residual). A pure random
  walk scores ~0 (no coupling). The middle is the point.

A separate **fingerprint** (occupancy, clustering, trail fill, class mix)
lets us measure whether different songs grow different worlds, independent of
aesthetic judgement — and whether different seeds of one song stay in the same
biome.

## Determinism & preview === export

- one decode rate, one analysis pass, one feature table;
- no `Math.random`, no wall clock, no `requestAnimationFrame` in the sim;
- the world steps in typed arrays; ordering is canonical;
- the camera and renderer are pure functions of `(song, seed, frame)`;
- tools and browser share the same `Run`.

What will be previewed is exactly what will be exported.

---

## Build order & status

*Course-corrected in **EVOLUTION.md** — read it before touching the world or the
rules. The front-end (analysis → events → field → laws → palette) stands; the
world behind the `Run` is being rebuilt.*

- **v0 — substrate** ✔ one song, one species, alive on the dish.
  *Built:* analysis → structures → world → renderer → filmstrips, metrics,
  15 tests green.
- **v1 — variation** ✔ species sampled per seed; three reference songs
  (drone / runner / dusk) grow measurably distinct worlds (fingerprint
  divergence ≫ seed divergence; fitness separates them).
- **v2 — story** laws switch at section boundaries (done in the world);
  event-reactive camera cutting (done in camera.js); filmstrip + report tools
  wired.
- **M1 — macro world** (EVOLUTION.md) replacing the entity sim with a
  population-field sim: `FIELD=128` density grid (rendered at 256² texture),
  `SUBSTEPS` per frame, coarse-grid forces, home basins, speck-swarm rendering
  ("billions" as texture), song-specific openings. The invariants, determinism
  and the `Run` interface survive.
- **M2 — rule program** ✔ the genome/laws become a compiled, declarative
  per-song rule program: castes as rule entries (mutualism, social floors),
  interaction rules, per-section operator regimes (structural section
  changes), the event→response mapping, and a shared carrying capacity.
- **M2.7 — one species, many minds** ✔ built: tribes/castes dissolved into a
  single population field with two live properties — a **strain mix** of
  song-compiled LittleVM personalities evolving by local selection, and an
  **allegiance field** whose macro-organisms ("bodies") hold territory, wage
  war (which kills, never converts) and colonise virgin ground — on top of
  the living well timeline. The world's economy is food-limited: colonies
  bloom on wells, starve off bare ground, and must keep moving to keep
  eating.
- **M2.8 — rock** ✔ built: the inert resource — scattered at birth, dropped
  and shattered by the song, impassable, quarried by builders into carried
  burden and soil, walled where the colony works. See EVOLUTION.md.
- **M2.9 — flora, gardens, walls, farmers, aimed work** ✔ built: the
  interactable layer — see EVOLUTION.md.
- **M3 — lens** the camera travels (track / reveal / establish) and reserves
  cuts for musical punctuation.
- **M4 — film** mux frames + PCM to video.

## Current state, honestly

The world is alive, the songs diverge on the metrics, and the population is
now one species of many minds: strains compiled by the song, bodies that hold
territory and war over living wells. It is not yet a *film*: the reference
tracks are synthetic test signals, the lens still prefers the biggest knot to
a traveling shot (M3), and muxing is waiting (M4). The allegiance layer is
deliberately rigid — bodies never switch sides — so territories read; the
drama currently comes from war, colonisation and the wells' own lives.