music-video-gen/metabolism/DESIGN.md

metabolism — design

A music video generator whose output is grown, not mixed. The track is not a parameter set the renderer modulates; it is the substrate a simulation grows on. Given the same song and the same seed, the same world — bit for bit, frame for frame, preview and export identical.

      song (PCM)
         │  analyse            — whole track, up front, fixed-rate
         ▼
   feature table ──► score         events (what spoke, when, how hard)
   (per frame)   ──► field         food, where the track put it
                  ──► graph        relations between voices
                  ──► laws         per-section regime
         │  seed
         ▼
   species genome ─────────────── rules the agents live by
         ▼
   world ◄── agents on the field, laws switch at boundaries
         ▼
   camera ◄── cuts to where the world just changed
         ▼
   framebuffer ──► PNG (tools) / canvas (demo) / video (later)

---

The idea

Most music visualisers modulate one fixed stage with the song: parameters go up and down, and every song ends up presenting the same toy in different clothes. That is cheap to watch and cheap to build, and it is exactly what we are not doing.

We want each song to produce a world that looks and behaves like that song and no other — a world with its own places, its own creatures, its own disasters. The mechanism is metabolism over modulation: the song supplies the resource field, the event impulses and the per-section laws; the seed picks the species, and the species is a rule set, not a parameter vector.

Two songs therefore differ in the way two habitats differ, not in the way two skins differ. A noisy fast track grows a swarm with jittered motion and thin, ephemeral trails. A tonal slow track grows weavers with long memories and heavy ground markings. Same renderer, same simulation — different songs, so different laws, so different worlds.

Why rules and not parameters

A parameter changes how strongly a behaviour applies. A rule changes what behaviour exists. Parameters are a knob; rules are a diet. The strongest variation-per-cost you can buy is changing which rules a creature obeys, not tuning how much of a rule it obeys:

• small budget → different rules → wildly different outcomes
• large population → multiply across hundreds of interacting agents for free

The genre of a track steers the family of rules; the seed steers the exact world, the way a seed steers an ecology's exact history while the biome is determined by the land.

---

Three invariants

Every design decision below exists to keep these alive. If an edit weakens one of them, the edit is wrong.

1. Persistence — things last. Agents live for seconds; trails last tens of seconds. The audience can follow something across the film rather than decoding a new random field every frame.
2. Causality — every visible event has an audible cause and a visible one. A hit lands as a ring on the ground where the track put it. A bloom has a cause; a migration has one. Nothing happens for no reason.
3. Consequence — the state at t is made of the state at t−1. The field is depleted by eating; trails accumulate from walking. The world records its own history instead of being re-rolled each frame.

A fourth, transport-level invariant: determinism. Same song, same seed, same frame → same pixel, in node and in the browser, at any wall-clock time.

---

The analysis: four structures from one pass

A single FFT pass over the whole track (fixed 44.1 kHz decode, 2048-bin FFT, 30 fps) produces a per-frame feature table: band energy (low/mid/high), spectral flux per band, centroid (log-scaled), flatness, a dominant pitch, energy, and chronicle of onsets. From that table derive four structures:

Score — events

Onsets picked per band with an adaptive threshold, merged so one transient fires once (not three times for three bands). Each event carries a voice (low/mid/high), a strength, and a time.

Where an event lands is musical. The voice picks the ring (kick near the centre, body in the middle, hats at the rim), the dominant pitch class picks the angle (12 semitones → 12 sectors), strength picks how much food and how wide. The track's rhythm is literally folded into the geometry of the food.

Field — food

Events splat resource onto a 128×128 grid. On top of that, wells — small, fertile places with a life of their own. There are many of them (8–16) and each is small and weak: a tight kernel, a modest recharge rate, no single well rich enough to carry a world — the same total fertility budget the old permanent springs had, spread thin. Where wells come from and when they go is the track's own harmony:

• birth — a well sprouts where a pitch class enters the music and holds (sustained chroma energy, seeded by the events that already landed there);
• drift — a well wanders slowly with its harmony, pulled along the smoothed pitch/centroid path, and the creatures anchored to it follow;
• death — when a pitch class leaves the song, its well starves: the recharge fades, the kernel decays, the colony must abandon it.

The chord does not just draw the map — it runs the map's history. Wells are the story's spine: founding, settlement, exodus, migration, collapse. Each has an audible cause and a visible one, and each lasts long enough to be followed. The whole lifecycle is compiled at analysis time into a deterministic well timeline (birth, path, death per well); the sim only plays it back. Wells are also where bodies meet: everyone needs them, so a well is the natural front between macro-organisms — and its death is a retreat, its birth a land-rush.

Graph — relations

The band-envelope correlation matrix now works at two scales. It decides the division of labour — which strain motifs complement which, so a bass line locked with the kick pairs diggers with haulers — and it feeds the diplomacy: the harmony the track sustains maps to how bodies treat each other, truce and trade versus skirmish and siege. Society is a property of the mix, at both scales.

Law — regime per section

Section boundaries are found where the 4 Hz feature cloud changes its mind (cheap self-similarity, not an FFT of the FFT), classified into intro / build / drop / breakdown / sustain / outro. Each section becomes a law: a multiplier set applied to the standing rules at the boundary. A drop speeds everything up and opens the floodgates for spawning; a breakdown slows the world and doubles deposit. A section boundary is the only place the rules change — which is what makes the boundary audible in the picture.

---

The species

The genome maps the summary statistics to kinds of behaviour:

song stats	breeding rule	consequence
flatness ↑	jitter ↑	noisy track = jittery creatures
flatness ↓	trail half-life ↑, deposit ↑	tonal track = heavy ground markings
beat strength	alignment and pace	a pulse makes them march
band correlations	division of labour	complementary voices breed complementary strains
onset density	separation, split cost	dense track = thronged dish but crowded apart
spectral tilt	hue	bright mixes warm, dark mixes cold

There are no castes and no tribes. There is one population — a single density field — and every organism in it carries two properties:

Personality — the strain. The song decides how many kinds of minds its world needs (m, read from how many independent voices the mix resolves into, bounded by arrangement depth, jittered by the seed) and compiles a LittleVM program for each strain: grazer, builder, sentinel, scout, breeder — each with its own senses (food, densities, trail, law, hunger, rim, well proximity and liveliness, body presence), registers as memory, real branching, and actions (move, graze, excrete, hit, spawn, follow, patrol). The population is a mix of strains the way a biofilm is: specialists working the same ground. And the mix is itself alive — strains that eat and grow where they are propagate locally; proportions shift district by district. The film shows specialisation evolving inside one species.

Allegiance — the body. Every organism also belongs to a macro-organism — its body — a polity of mixed strains that builds, defends and expands as one. Allegiance is a field: the population partitioned by body, advected frozen-in with the organisms as they move. Allegiance does not change sides. A body is founded once and persists; war kills, it does not convert; there is no defection and no merger — territories stay legible because their colours never homogenise. What keeps more than one body alive is colonisation: a settlement on ground far from every body core, blessed by a strong onset, may plant a NEW flag (a fresh colour) — rare, and it grows the cast rather than reshuffling it.

Bodies are few — the song opens 2–4, capped at ~8 so colour stays legible — and they are the drama. The film is the story of macro-organisms fighting and holding ground: skirmish lines at fronts where the loser bleeds, truces in consonant sections, symbiosis where one body's waste is another's food. The song sets the terms of diplomacy: consonant, harmonically locked material breeds peace; dissonant, decorrelated material breeds rivalry; drops are expansion wars; breakdowns are truces and mutual building.

A species name like "fast tonal weaver" falls out of the genome, so the report can talk about the world and not the track.

The world

A fixed-step, garbage-free simulation over typed arrays: every RNG stream seeded, every update a pure function of the previous substep. The course correction (EVOLUTION.md) replaces the entity ecology with a population-field ecology — the same front-end (events place food, wells born from and living by the chord, laws at section boundaries) feeding density instead of discrete agents, stepped several times per rendered frame so macro patterns form. Nothing here is realtime: no player, no clock, no allocation — typed arrays, a step function, and a history we can regress against the song.

The lens

Camera

An observer with a motive, not a rig on rails:

• shots cut when the world changes, never on a timer that ignores the world. Cuts also fire on strong onsets during drop/build, so the edit hits like the track hits;
• the subject is recent activity (decayed 32×32 cell grid), weighted away from the previous subject — novelty without drift;
• shot sizes alternate in a deliberate rhythm — a wide establishing, a medium, a close push — and zoom follows coverage: a broad field gets a wide lens, a crowded knot gets a close-up;
• while held, the camera drifts and pushes in — a shot is an approach, not a still.

Renderer

A tiny software framebuffer (additive blend, discs, rings, lines, bilinear world blit) so the same pixels render in node (PNG, filmstrips) and in the browser (ImageData). Layers: near-black key-tinted background → trail+field composed at field resolution with a tight blur (line) and a faint wide blur (glow) → event rings → agents as bodies with velocity streaks → vignette → deterministic grain.

The field is drawn faintly under the trails, so the audience can see why the creatures are where they are.

Metrics: system, not pixels

Per-frame descriptor distance was rejected long ago (it cannot tell a dungeon from a car and rewards noise). Instead we score the whole episode against its own song:

• coupling — how much the ecology answers the track: cue response (does movement edge up just after onsets?) plus long-form correlation (smoothed movement vs smoothed energy);
• residual — 1 − R² of the sim's activity (motion and the population curve) regressed on all song features: the internal dynamics the song does not explain;
• fitness = coupling × residual — the best world keeps time and has a life of its own. A pure visualiser scores ~0 (no residual). A pure random walk scores ~0 (no coupling). The middle is the point.

A separate fingerprint (occupancy, clustering, trail fill, class mix) lets us measure whether different songs grow different worlds, independent of aesthetic judgement — and whether different seeds of one song stay in the same biome.

Determinism & preview === export

• one decode rate, one analysis pass, one feature table;
• no Math.random, no wall clock, no requestAnimationFrame in the sim;
• the world steps in typed arrays; ordering is canonical;
• the camera and renderer are pure functions of (song, seed, frame);
• tools and browser share the same Run.

What will be previewed is exactly what will be exported.

---

Build order & status

Course-corrected in EVOLUTION.md — read it before touching the world or the rules. The front-end (analysis → events → field → laws → palette) stands; the world behind the Run is being rebuilt.

• v0 — substrate ✔ one song, one species, alive on the dish. Built: analysis → structures → world → renderer → filmstrips, metrics, 15 tests green.
• v1 — variation ✔ species sampled per seed; three reference songs (drone / runner / dusk) grow measurably distinct worlds (fingerprint divergence ≫ seed divergence; fitness separates them).
• v2 — story laws switch at section boundaries (done in the world); event-reactive camera cutting (done in camera.js); filmstrip + report tools wired.
• M1 — macro world (EVOLUTION.md) replacing the entity sim with a population-field sim: FIELD=128 density grid (rendered at 256² texture), SUBSTEPS per frame, coarse-grid forces, home basins, speck-swarm rendering ("billions" as texture), song-specific openings. The invariants, determinism and the Run interface survive.
• M2 — rule program ✔ the genome/laws become a compiled, declarative per-song rule program: castes as rule entries (mutualism, social floors), interaction rules, per-section operator regimes (structural section changes), the event→response mapping, and a shared carrying capacity.
• M2.7 — one species, many minds ✔ built: tribes/castes dissolved into a single population field with two live properties — a strain mix of song-compiled LittleVM personalities evolving by local selection, and an allegiance field whose macro-organisms ("bodies") hold territory, wage war (which kills, never converts) and colonise virgin ground — on top of the living well timeline. The world's economy is food-limited: colonies bloom on wells, starve off bare ground, and must keep moving to keep eating.
• M3 — lens the camera travels (track / reveal / establish) and reserves cuts for musical punctuation.
• M4 — film mux frames + PCM to video.

Current state, honestly

The world is alive, the songs diverge on the metrics, and the population is now one species of many minds: strains compiled by the song, bodies that hold territory and war over living wells. It is not yet a film: the reference tracks are synthetic test signals, the lens still prefers the biggest knot to a traveling shot (M3), and muxing is waiting (M4). The allegiance layer is deliberately rigid — bodies never switch sides — so territories read; the drama currently comes from war, colonisation and the wells' own lives.