125 lines
4.6 KiB
Markdown
125 lines
4.6 KiB
Markdown
# flow-state
|
||
|
||
Ambient/EDM music video generator. Drop in a track, get a full-length, non-story,
|
||
music-reactive video. No sourced footage — every frame is generated, and the whole
|
||
look is derived from the audio.
|
||
|
||
```bash
|
||
npm install
|
||
npm run dev # http://localhost:5180
|
||
```
|
||
|
||
Drop an audio file onto the page (mp3, flac, wav, ogg). Analysis takes a second or
|
||
two, then the video is ready to preview and export.
|
||
|
||
## How it works
|
||
|
||
The track is decoded and analysed **before the first frame renders**, into a table
|
||
with one row per video frame: band energies, onset flux, spectral centroid and
|
||
flatness, a phase-locked beat grid, section boundaries, and lookahead fields.
|
||
|
||
Nothing reads a live `AnalyserNode`. Realtime preview maps `audio.currentTime` to a
|
||
frame index; export counts frames. Both read the same rows, so **what you preview is
|
||
what you export** — the exporter has no render path of its own.
|
||
|
||
Analysing the whole track up front also buys the thing a causal analyser cannot do:
|
||
a build can *anticipate* its drop and arrive at the transition already at full
|
||
tension, instead of reacting once the drop has landed.
|
||
|
||
## Working with it
|
||
|
||
| | |
|
||
|---|---|
|
||
| `space` | play / pause |
|
||
| `←` `→` | previous / next section boundary |
|
||
| `L` | loop the current section |
|
||
| `D` | debug HUD |
|
||
| `O` | toggle the corner title plate |
|
||
| `,` `.` | step one frame |
|
||
|
||
**test render** exports 20 seconds around the playhead at full export quality. Use
|
||
it before committing to a full render.
|
||
|
||
**reroll** re-seeds the whole track; **reroll section** changes only the section
|
||
under the playhead; **lock** protects a section from further rerolls. Every
|
||
parameter the generator chose is exposed under the *scene* tab and can be edited
|
||
live.
|
||
|
||
The **click track** button (look tab) mixes an audible click onto the detected beat
|
||
grid. If the clicks don't sit on the beat, tempo detection is wrong and everything
|
||
downstream inherits it — check this first when a track looks off.
|
||
|
||
## Checks
|
||
|
||
```bash
|
||
npm test # audio pipeline against synthetic ground truth
|
||
npm run lint:scenes # determinism grep + scene schema/shader agreement
|
||
```
|
||
|
||
`http://localhost:5180/checks.html` runs the GPU gates for every phase. Add
|
||
`?slow=1` for the full suite, `?phase=5` for one phase.
|
||
|
||
## Adding a scene
|
||
|
||
Quick walkthrough: `HOWTO-visualizers.md`.
|
||
|
||
A scene is a shader plus a params block. Everything else — uniform binding, UI
|
||
controls, seeded per-track sampling, arc automation — is derived from the schema.
|
||
|
||
```js
|
||
export const myScene = {
|
||
name: 'My Scene',
|
||
family: 'organic', // flow organic minimal structural geometric glitch
|
||
kind: 'fragment',
|
||
params: {
|
||
density: { type: 'float', range: [0, 1], default: 0.5, uniform: 'u_density', bias: 'density' },
|
||
speed: { type: 'float', range: [0.1, 2], default: 0.5, uniform: 'u_speed', rate: true },
|
||
palette: { type: 'palette', count: 4 },
|
||
},
|
||
reactive: {
|
||
density: { feature: 'bandLow', amount: 0.3 },
|
||
},
|
||
shader: `
|
||
vec4 scene(vec2 uv, vec2 p) {
|
||
float t = u_time * u_speed + u_seed;
|
||
return vec4(palRamp(fbm(p * 4.0 + t, 4)), 1.0);
|
||
}
|
||
`,
|
||
};
|
||
```
|
||
|
||
Register it in `src/scenes/registry.js`, then run `npm run lint:scenes` and the
|
||
Phase 7 checks. Three rules the linter enforces, each of which has already caused a
|
||
real bug here:
|
||
|
||
- **Anything multiplying `u_time` must be `rate: true`.** Phase is `elapsed × rate`,
|
||
so modulating a rate jumps the phase by `elapsed × delta` — a minute in, a small
|
||
wobble throws the image several whole units between frames. It measured as
|
||
strobing at twice the accessibility limit.
|
||
- **Don't reuse a contract uniform name** (`u_width`, `u_time`, `u_seed`, …). It's a
|
||
GLSL redefinition error, and the only symptom is a black frame.
|
||
- **Use `pal()` / `palRamp()`**, not hardcoded colours, or the look generator can't
|
||
recolour the scene.
|
||
|
||
Scenes that composite over a background rather than being one declare
|
||
`role: 'accent'`.
|
||
|
||
## Layout
|
||
|
||
```
|
||
src/
|
||
audio/ decode, STFT analysis, tempo, segmentation, FeatureTrack, click track
|
||
engine/ Timeline, Renderer, Layer, Compositor, passes, seeded rng, flash safety
|
||
look/ palette (OKLCH), LookGenerator, ArcDriver
|
||
params/ declarative schema, validation, serialisation
|
||
scenes/ the library — shader/ and layers3d/
|
||
export/ WebCodecs exporter
|
||
ui/ preview surface
|
||
checks/ phase gates, run from checks.html
|
||
```
|
||
|
||
`PLAN.md` has the full design and the reasoning behind each gate.
|
||
|
||
Forked from `party-stage` by copying what was useful, then fully detached — there
|
||
are no imports across the directory boundary in either direction.
|