dejvino/music-video-gen
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Metabolism: song as substrate, ecology grown from the track

Browse Source 
A new top-level section building the metabolism concept: the track
supplies the food field, event impulses and per-section laws; the seed
picks the species (a rule set, not a parameter vector). Same song + seed
=> bit-identical frames; different songs => different worlds.

- analysis: single FFT pass -> per-frame feature table; score (onsets
  folded into world geometry via voice ring + pitch-class sector), wells
  (food placed by the harmony), graph (band correlations -> interaction
  matrix), laws (per-section regimes)
- sim: typed-array world, 3 castes (builder/orbit, forager/commute,
  spark/orbit-wide) with causality rings, persistence trails,
  consequence field; fully deterministic
- lens: software framebuffer renderer (PNG in node, ImageData in
  browser) + event-reactive camera (cuts on activity change, alternating
  shot sizes, drift-and-push)
- metrics: coupling x residual fitness (not per-frame descriptors) plus
  a grown-structure fingerprint; report + strip tools; 15 tests green
- DESIGN.md captures the concept, invariants and build order

3 reference test signals (drone/runner/dusk) grow measurably distinct
worlds while seeds of one song stay in the same biome.
This commit is contained in:
Dejvino
2026-10-24 08:23:33 +02:00
parent 6a2af2e45f
commit 13b0939221
26 changed files with 4469 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
metabolism/DESIGN.md
+250
View File
@@ -0,0 +1,250 @@
# metabolism — design
A music video generator whose output is **grown**, not mixed. The track is not
a parameter set the renderer modulates; it is the *substrate* a simulation
grows on. Given the same song and the same seed, the same world — bit for bit,
frame for frame, preview and export identical.
```
song (PCM)
│ analyse — whole track, up front, fixed-rate
▼
feature table ──► score events (what spoke, when, how hard)
(per frame) ──► field food, where the track put it
──► graph relations between voices
──► laws per-section regime
│ seed
▼
species genome ─────────────── rules the agents live by
▼
world ◄── agents on the field, laws switch at boundaries
▼
camera ◄── cuts to where the world just changed
▼
framebuffer ──► PNG (tools) / canvas (demo) / video (later)
```
---
## The idea
Most music visualisers **modulate** one fixed stage with the song: parameters
go up and down, and every song ends up presenting the same toy in different
clothes. That is cheap to watch and cheap to build, and it is exactly what
we are not doing.
We want each song to produce a world that looks and behaves like *that song*
and no other — a world with its own places, its own creatures, its own
disasters. The mechanism is **metabolism over modulation**: the song supplies
the resource field, the event impulses and the per-section laws; the seed
picks the species, and the species is a *rule set*, not a parameter vector.
Two songs therefore differ in the way two habitats differ, not in the way two
skins differ. A noisy fast track grows a swarm with jittered motion and thin,
ephemeral trails. A tonal slow track grows weavers with long memories and
heavy ground markings. Same renderer, same simulation — different songs, so
different laws, so different worlds.
### Why rules and not parameters
A parameter changes *how strongly* a behaviour applies. A rule changes *what
behaviour exists*. Parameters are a knob; rules are a diet. The strongest
variation-per-cost you can buy is changing which rules a creature obeys, not
tuning how much of a rule it obeys:
- small budget → different rules → wildly different outcomes
- large population → multiply across hundreds of interacting agents for free
The genre of a track steers the *family* of rules; the seed steers the exact
world, the way a seed steers an ecology's exact history while the biome is
determined by the land.
---
## Three invariants
Every design decision below exists to keep these alive. If an edit weakens
one of them, the edit is wrong.
1. **Persistence** — things last. Agents live for seconds; trails last tens
of seconds. The audience can *follow* something across the film rather than
decoding a new random field every frame.
2. **Causality** — every visible event has an audible cause and a visible
one. A hit lands as a ring on the ground where the track put it. A bloom
has a cause; a migration has one. Nothing happens for no reason.
3. **Consequence** — the state at t is made of the state at t−1. The field is
depleted by eating; trails accumulate from walking. The world records its
own history instead of being re-rolled each frame.
A fourth, transport-level invariant: **determinism**. Same song, same seed,
same frame → same pixel, in node and in the browser, at any wall-clock time.
---
## The analysis: four structures from one pass
A single FFT pass over the whole track (fixed 44.1 kHz decode, 2048-bin FFT,
30 fps) produces a per-frame feature table: band energy (low/mid/high),
spectral flux per band, centroid (log-scaled), flatness, a dominant pitch,
energy, and chronicle of onsets. From that table derive four structures:
### Score — events
Onsets picked per band with an adaptive threshold, merged so one transient
fires once (not three times for three bands). Each event carries a voice
(low/mid/high), a strength, and a time.
**Where an event lands is musical.** The voice picks the ring (kick near the
centre, body in the middle, hats at the rim), the dominant pitch class picks
the angle (12 semitones → 12 sectors), strength picks how much food and how
wide. The track's rhythm is literally folded into the geometry of the food.
### Field — food
Events splat resource onto a 128×128 grid. On top of that, **wells** — slow,
permanent fertile places — are placed by the track's own harmony: one well per
strong pitch class, on two rings, with richness proportional to the chroma
and the band balance. The chord is the map. Wells recharge slowly to a
ceiling, so a place has a carrying capacity, not an infinite tap.
### Graph — relations
The 3×3 correlation matrix of the three band envelopes becomes the species'
*interaction matrix*: bands that move together attract; a band that lives its
own life repels the others. For a bass line that locks with the kick, the
builders court the foragers; for a high band that does its own thing, the
sparks are ostracised. Society is a property of the mix.
### Law — regime per section
Section boundaries are found where the 4 Hz feature cloud changes its mind
(cheap self-similarity, not an FFT of the FFT), classified into
intro / build / drop / breakdown / sustain / outro. Each section becomes a
*law*: a multiplier set applied to the standing rules at the boundary. A drop
speeds everything up and opens the floodgates for spawning; a breakdown slows
the world and doubles deposit. A section boundary is the only place the rules
change — which is what makes the boundary audible in the picture.
---
## The species
The genome maps the summary statistics to *kinds of behaviour*:
| song stats | breeding rule | consequence |
|---|---|---|
| flatness ↑ | jitter ↑ | noisy track = jittery creatures |
| flatness ↓ | trail half-life ↑, deposit ↑ | tonal track = heavy ground markings |
| beat strength | alignment and pace | a pulse makes them march |
| band correlations | interaction matrix | correlated bands attract |
| onset density | separation, split cost | dense track = thronged dish but crowded apart |
| spectral tilt | hue | bright mixes warm, dark mixes cold |
There are three castes, derived from the three voices:
- **builder** — slow, heavy, deposits a lot, keeps near its well. It is the
ink-dropper of the world: it paints the structure the synth earned.
- **forager** — commutes between wells and draws the roads. Direction lives
in its streaks.
- **spark** — small, fast, orbits wide, deposits almost nothing. The counter-
melody: it makes the film read as inhabited rather than decorated.
A species name like "fast tonal weaver" falls out of the genome, so the report
can talk about the world and not the track.
## The world
A fixed-step simulation (~2600 typed, garbage-free agents; spatial hash for
neighbourhood; every RNG stream seeded). Per frame:
1. inject this frame's events — food is placed, and strong events also
*spawn* life and leave a **ring** on the ground (the clearest causality in
the film);
2. decay field and trails; recharge wells (rate set by the section's law);
3. each agent: smell the food gradient, obey its caste's relation to its well,
feel its neighbours (separate / align / cohere / inter-caste attraction),
wander with jitter, hit soft walls, eat, move, lay ink, split on
accumulation, die on starvation;
4. compact.
There is no player, no realtime budget, no entity objects — only typed arrays,
a step function, and a history we can regress against the song.
## The lens
### Camera
An observer with a motive, not a rig on rails:
- **shots cut when the world changes**, never on a timer that ignores the
world. Cuts also fire on strong onsets during drop/build, so the edit hits
like the track hits;
- the **subject is recent activity** (decayed 32×32 cell grid), weighted away
from the previous subject — novelty without drift;
- **shot sizes alternate** in a deliberate rhythm — a wide establishing, a
medium, a close push — and zoom follows coverage: a broad field gets a wide
lens, a crowded knot gets a close-up;
- while held, the camera drifts and pushes in — a shot is an approach, not a
still.
### Renderer
A tiny software framebuffer (additive blend, discs, rings, lines, bilinear
world blit) so the same pixels render in node (PNG, filmstrips) and in the
browser (ImageData). Layers: near-black key-tinted background → trail+field
composed at field resolution with a tight blur (line) and a faint wide blur
(glow) → event rings → agents as bodies with velocity streaks → vignette →
deterministic grain.
The field is drawn faintly *under* the trails, so the audience can see *why*
the creatures are where they are.
## Metrics: system, not pixels
Per-frame descriptor distance was rejected long ago (it cannot tell a dungeon
from a car and rewards noise). Instead we score the whole episode against its
own song:
- **coupling** — how much the ecology answers the track: cue response (does
movement edge up just after onsets?) plus long-form correlation (smoothed
movement vs smoothed energy);
- **residual** — 1 − R² of the sim's activity (motion *and* the population
curve) regressed on all song features: the internal dynamics the song does
not explain;
- **fitness = coupling × residual** — the best world keeps time *and* has a
life of its own. A pure visualiser scores ~0 (no residual). A pure random
walk scores ~0 (no coupling). The middle is the point.
A separate **fingerprint** (occupancy, clustering, trail fill, class mix)
lets us measure whether different songs grow different worlds, independent of
aesthetic judgement — and whether different seeds of one song stay in the same
biome.
## Determinism & preview === export
- one decode rate, one analysis pass, one feature table;
- no `Math.random`, no wall clock, no `requestAnimationFrame` in the sim;
- the world steps in typed arrays; ordering is canonical;
- the camera and renderer are pure functions of `(song, seed, frame)`;
- tools and browser share the same `Run`.
What will be previewed is exactly what will be exported.
---
## Build order & status
- **v0 — substrate** ✔ one song, one species, alive on the dish.
*Built:* analysis → structures → world → renderer → filmstrips, metrics,
15 tests green.
- **v1 — variation** ✔ species sampled per seed; three reference songs
(drone / runner / dusk) grow measurably distinct worlds (fingerprint
divergence ≫ seed divergence; fitness separates them).
- **v2 — story** laws switch at section boundaries (done in the world);
event-reactive camera cutting (done in camera.js); filmstrip + report tools
wired.
- **v3 — search** rule-space search with the coupling × residual fitness —
only after the substrate exists, and only as a sampling tool, never as a
user-facing knob.
## Current state, honestly
The world is alive and the songs diverge on the metrics, but it is not yet a
*film*: the camera and the soundtrack are not muxed, the renderer is a proof
at 480p, and the three reference tracks are synthetic test signals rather than
music. The next milestone is three videos — one per song — that a person can
sit through and tell apart without the report.