music-video-gen/flow-state/README.md

flow-state

Ambient/EDM music video generator. Drop in a track, get a full-length, non-story, music-reactive video. No sourced footage — every frame is generated, and the whole look is derived from the audio.

npm install
npm run dev          # http://localhost:5180

Drop an audio file onto the page (mp3, flac, wav, ogg). Analysis takes a second or two, then the video is ready to preview and export.

How it works

The track is decoded and analysed before the first frame renders, into a table with one row per video frame: band energies, onset flux, spectral centroid and flatness, a phase-locked beat grid, section boundaries, and lookahead fields.

Nothing reads a live AnalyserNode. Realtime preview maps audio.currentTime to a frame index; export counts frames. Both read the same rows, so what you preview is what you export — the exporter has no render path of its own.

Analysing the whole track up front also buys the thing a causal analyser cannot do: a build can anticipate its drop and arrive at the transition already at full tension, instead of reacting once the drop has landed.

Working with it

space	play / pause
← →	previous / next section boundary
L	loop the current section
D	debug HUD
O	toggle the corner title plate
, .	step one frame

test render exports 20 seconds around the playhead at full export quality. Use it before committing to a full render.

reroll re-seeds the whole track; reroll section changes only the section under the playhead; lock protects a section from further rerolls. Every parameter the generator chose is exposed under the scene tab and can be edited live.

The metronome button (look tab) mixes an audible click onto the detected beat grid. If the clicks don't sit on the beat, tempo detection is wrong and everything downstream inherits it — check this first when a track looks off.

Checks

npm test             # audio pipeline against synthetic ground truth
npm run lint:scenes  # determinism grep + scene schema/shader agreement

http://localhost:5180/checks.html runs the GPU gates for every phase. Add ?slow=1 for the full suite, ?phase=5 for one phase.

Seed variety test

Every other phase asks whether one video is correct. Phase 12 asks whether two videos are different — the one failure the rest of the suite cannot see, since a generator that ignores its seed passes determinism and liveness perfectly.

http://localhost:5180/checks.html?variety=1

Frames are reduced to a structural descriptor that is deliberately blind to brightness, contrast, hue and rotation, and sensitive to feature scale, orientation structure, composition, texture statistics and motion. Colour is measured but never counted — its job is to expose the case where two seeds differ only in palette. The score sits between two references the same instrument produced: the floor is how far one video travels from itself across its own sections, the ceiling is the same pipeline with every layer recast at random. Two checks gate the instrument itself before any number from it is trusted.

The report also sweeps all visualizations in the library, every pair, and names the structural twins — different scenes that are the same look in different colours, which the per-scene distinct gate cannot catch because it compares raw pixels.

Adding a scene

Quick walkthrough: HOWTO-visualizers.md.

A scene is a shader plus a params block. Everything else — uniform binding, UI controls, seeded per-track sampling, arc automation — is derived from the schema.

export const myScene = {
    name: 'My Scene',
    family: 'organic',          // flow organic minimal structural geometric glitch
    kind: 'fragment',
    params: {
        density: { type: 'float', range: [0, 1], default: 0.5, uniform: 'u_density', bias: 'density' },
        speed:   { type: 'float', range: [0.1, 2], default: 0.5, uniform: 'u_speed', rate: true },
        palette: { type: 'palette', count: 4 },
    },
    reactive: {
        density: { feature: 'bandLow', amount: 0.3 },
    },
    shader: `
vec4 scene(vec2 uv, vec2 p) {
    float t = u_time * u_speed + u_seed;
    return vec4(palRamp(fbm(p * 4.0 + t, 4)), 1.0);
}
`,
};

Register it in src/scenes/registry.js, then run npm run lint:scenes and the Phase 7 checks. Three rules the linter enforces, each of which has already caused a real bug here:

• Anything multiplying u_time must be rate: true. Phase is elapsed × rate, so modulating a rate jumps the phase by elapsed × delta — a minute in, a small wobble throws the image several whole units between frames. It measured as strobing at twice the accessibility limit.
• Don't reuse a contract uniform name (u_width, u_time, u_seed, …). It's a GLSL redefinition error, and the only symptom is a black frame.
• Use pal() / palRamp(), not hardcoded colours, or the look generator can't recolour the scene.

Scenes that composite over a background rather than being one declare role: 'accent'.

Layout

src/
  audio/      decode, STFT analysis, tempo, segmentation, FeatureTrack, metronome
  engine/     Timeline, Renderer, Layer, Compositor, passes, seeded rng, flash safety
  look/       palette (OKLCH), LookGenerator, ArcDriver
  params/     declarative schema, validation, serialisation
  scenes/     the library — shader/ and layers3d/
  export/     WebCodecs exporter
  ui/         preview surface
  checks/     phase gates, run from checks.html

PLAN.md has the full design and the reasoning behind each gate.

Forked from party-stage by copying what was useful, then fully detached — there are no imports across the directory boundary in either direction.