music-video-gen/flow-state

flow-state

Ambient/EDM music video generator. Drop in a track, get a full-length, non-story, music-reactive video. No sourced footage — every frame is generated, and the whole look is derived from the audio.

npm install
npm run dev          # http://localhost:5180

Drop an audio file onto the page (mp3, flac, wav, ogg). Analysis takes a second or two, then the video is ready to preview and export.

How it works

The track is decoded and analysed before the first frame renders, into a table with one row per video frame: band energies, onset flux, spectral centroid and flatness, a phase-locked beat grid, section boundaries, and lookahead fields.

Nothing reads a live AnalyserNode. Realtime preview maps audio.currentTime to a frame index; export counts frames. Both read the same rows, so what you preview is what you export — the exporter has no render path of its own.

Analysing the whole track up front also buys the thing a causal analyser cannot do: a build can anticipate its drop and arrive at the transition already at full tension, instead of reacting once the drop has landed.

The story

Everything above happens at a moment. On top of it the track gets a story: a plot chosen per song, four moments found in the audio (arrival, turn, climax, resolution), and a position for every section — which of the four drops this is, not just that it is a drop.

That position moves what the video does. The kind's anchor scene is reserved for the climax rather than opening every section of its kind; the dials ratchet with each recurrence; the cutting rate follows tension and the resolution holds; the song's cast arrives over the video instead of being fully stated in the first shot; and under a return plot the outro re-casts what the intro opened on.

The story never overrides the song. Its moments are read off the section energies, and its effect on parameter bias is bounded so it can decide which drop this is and never whether a breakdown is one. D shows the current plot, act and journey; the timeline marks the moments. EPIC-4.md has the design.

Working with it

space	play / pause
← →	previous / next section boundary
L	loop the current section
D	debug HUD
O	toggle the corner title plate
, .	step one frame

test render exports 20 seconds around the playhead at full export quality. Use it before committing to a full render.

reroll re-seeds the whole track; reroll section changes only the section under the playhead; lock protects a section from further rerolls. Every parameter the generator chose is exposed under the scene tab and can be edited live.

The metronome button (look tab) mixes an audible click onto the detected beat grid. If the clicks don't sit on the beat, tempo detection is wrong and everything downstream inherits it — check this first when a track looks off.

Checks

npm test             # audio pipeline against synthetic ground truth
npm run lint:scenes  # determinism grep + scene schema/shader agreement

http://localhost:5180/filmstrip.html renders every song in the bank as one frame every 30 seconds, so a video that is busy and going nowhere is visible as a row of interchangeable stills. http://localhost:5180/checks.html runs the GPU gates for every phase. Add ?slow=1 for the full suite, ?phase=5 for one phase.

Seed variety test

Every other phase asks whether one video is correct. Phase 12 asks whether two videos are different — the one failure the rest of the suite cannot see, since a generator that ignores its seed passes determinism and liveness perfectly.

http://localhost:5180/checks.html?variety=1

Frames are reduced to a structural descriptor that is deliberately blind to brightness, contrast, hue and rotation, and sensitive to feature scale, orientation structure, composition, texture statistics and motion. Colour is measured but never counted — its job is to expose the case where two seeds differ only in palette. The score sits between two references the same instrument produced: the floor is how far one video travels from itself across its own sections, the ceiling is the same pipeline with every layer recast at random. Two checks gate the instrument itself before any number from it is trusted.

The report also prints direction: the rank correlation between how far apart two probes are in time and how far apart they look. drift cannot tell a video that travels from one that merely keeps changing, and a video with a story raises it on purpose — so read the two together. A floor that rises with direction is an arc; a floor that rises without one is a shuffle.

The report also sweeps all visualizations in the library, every pair, and names the structural twins — different scenes that are the same look in different colours, which the per-scene distinct gate cannot catch because it compares raw pixels.

Adding a scene

Quick walkthrough: HOWTO-visualizers.md.

A scene is a shader plus a params block. Everything else — uniform binding, UI controls, seeded per-track sampling, arc automation — is derived from the schema.

export const myScene = {
    name: 'My Scene',
    family: 'organic',          // flow organic minimal structural geometric glitch
    kind: 'fragment',
    params: {
        density: { type: 'float', range: [0, 1], default: 0.5, uniform: 'u_density', bias: 'density' },
        speed:   { type: 'float', range: [0.1, 2], default: 0.5, uniform: 'u_speed', rate: true },
        palette: { type: 'palette', count: 4 },
    },
    reactive: {
        density: { feature: 'bandLow', amount: 0.3 },
    },
    shader: `
vec4 scene(vec2 uv, vec2 p) {
    float t = u_time * u_speed + u_seed;
    return vec4(palRamp(fbm(p * 4.0 + t, 4)), 1.0);
}
`,
};

Register it in src/scenes/registry.js, then run npm run lint:scenes and the Phase 7 checks. Three rules the linter enforces, each of which has already caused a real bug here:

• Anything multiplying u_time must be rate: true. Phase is elapsed × rate, so modulating a rate jumps the phase by elapsed × delta — a minute in, a small wobble throws the image several whole units between frames. It measured as strobing at twice the accessibility limit.
• Don't reuse a contract uniform name (u_width, u_time, u_seed, …). It's a GLSL redefinition error, and the only symptom is a black frame.
• Use pal() / palRamp(), not hardcoded colours, or the look generator can't recolour the scene.

Scenes that composite over a background rather than being one declare role: 'accent'.

Layout

src/
  audio/      decode, STFT analysis, tempo, segmentation, FeatureTrack, metronome
  engine/     Timeline, Renderer, Layer, Compositor, passes, seeded rng, flash safety
  look/       palette (OKLCH), LookGenerator, ArcDriver, Story
  params/     declarative schema, validation, serialisation
  scenes/     the library — shader/ and layers3d/
  export/     WebCodecs exporter
  ui/         preview surface
  checks/     phase gates, run from checks.html

PLAN.md has the full design and the reasoning behind each gate.

Forked from party-stage by copying what was useful, then fully detached — there are no imports across the directory boundary in either direction.