music-video-gen/flow-state/DESIGN.md

flow-state — design decisions

The decisions this project has taken, and why. New work must obey them; changing one means updating this file and the gate that enforces it.

Doc map: PLAN.md is the full design, EPIC-2/3/4/5-3D.md are the epic designs, HOWTO-visualizers.md / HOWTO-variety.md are the scene-author references, MIGRATION.md is the shader→stage migration recipe.

1. Determinism spine

• The audio is fully analysed before the first frame renders, into a frame-indexed FeatureTrack table. Nothing reads a live AnalyserNode. Preview maps currentTime → frameIndex; export counts frames. Same rows, same image.
• No wall-clock (performance.now, Date) and no Math.random in engine/scenes/look/audio/params/actors/venues. Seeded Rng only; motion is analytic f(t, seed) — seek === playback, preview === export.
• dt is constant (1/fps); dropped preview frames hitch, never diverge.
• Feedback/ping-pong targets are explicitly cleared; shader programs are primed (Engine.prime) before frame-exact rendering — unlinked programs rendered the first ~10 frames wrong on the heaviest scene.
• Scene math lives in normalized aspect-corrected coords (uv, p); only u_pixelScale carries pixels, so 720p preview matches 4K export.
• Guarantee is per-machine bit-identical; cross-machine is perceptual. Same-machine allows max channel delta ≤ 1 (GPU load varies one LSB under heavy shaders); real bugs score in the tens.
• Seeks need warm-up (log(0.001)/log(decay) frames) whenever feedback is on. Section boundaries are NOT exact for free — the compositor's feedback buffer is global and carries across them. Clearing it per boundary would flash every transition; warm-up is the better trade.
• Previews link to their exact app frame (src/ui/preview.js: song + seed
  • frame + venue/viz pins + the probe's warm-up length, both sides primed). Exactness takes four agreements: the bank probes analyse the bank PCM (peak-normalised 16-bit mono — audio/songbank.js bankPcm), file decode is fixed at 44.1 kHz with bit-exact WAV parsing (never the context rate), pins run before any layer builds (rebuilding mid-context renders differently), and materials built mid-update compile synchronously (needsCompile) so no warm-up frame draws against an unlinked program.

2. Lookahead

Offline analysis is what buys anticipation: buildSlope ramps through the bars before a higher-energy section, so visuals arrive at the drop already at tension. A causal analyser can never do this. The outgoing layer of a crossfade keeps its pre-boundary slope — feeding it the new section's features collapses its ramp in one frame (a visible pop at the edit).

3. Scenes: shader + schema, nothing else

• A scene is a fragment shader (or model build/update) plus a params block. Uniform binding, UI sliders, seeded sampling, arc drift and phase gates are all derived from the schema. No per-scene wiring.
• Six families (flow geometric organic structural minimal glitch) chosen by section kind — a breakdown never lands on a strobing glitch scene.
• Personality traits (shape camera space style) are a contract: each track is built on 1–2 traits and a scene honouring fewer is never cast. The library intentionally shrinks per track. Declared traits need machine-checked evidence (lint grep + render-delta gate); a dead camera (sigCamera assigned, result ignored) shipped once and is now linted.
• rate: true is mandatory for anything multiplying u_time. Phase is elapsed × rate; modulating a rate jumps phase by elapsed × Δrate — measured as 6 flashes/s, twice the WCAG 2.3.1 ceiling.
• Whole-frame luminance never pulses on the beat (3-flashes/s ceiling). Pulse something local; quantise glitches to the bar grid.
• A scene reading prev() must paint a base image first (black first frames, seek-fragile otherwise). prev() readers can never be grounds.
• No contract-uniform collisions, no hardcoded saturated colours — the look generator recolours through pal()/palRamp() only.
• Loops need constant bounds with an early break on the deciding param (// lint: fixed-cost opt-out for genuinely fixed work).

4. Continuity: nothing disappears mid-shot

Actors and objects change only at camera cuts:

• Discrete params (int/bool, plus floats marked discrete: true that drive loop bounds / instance counts) hold constant for the whole cue. No drift, no lookahead ramp, no reactive modulation — those would pop members in/out mid-shot. They still vary per section/variant sample, so drops can be denser than intros; the change lands on a cut.
• A discrete param MAY declare slowAxis: the driver steps its travel on the section-held journey instead of sliding it.
• Reactive on a discrete param and (unstated) drift of one are lint errors.
• Venue furniture never hard-toggles on interpolated state: chamber junk and stage pylons fade (ride-crew pattern); ride roadside density is arrangement-only. Re-hangs happen only on story jumps, always under a camera cut (venueJumpAt); the venue layer itself is persistent and never dissolved (old-room/new-room ghosting read as a glitch).

5. Time structure: cues, cuts, camera

• The arc driver works in cues (one per shot): a shot cut and a section change take the same code path, differing only in transition length.
• Default transition is a slow dissolve (2 bars calm, 1 bar loud, capped at 40% of the shot). Hard cuts only above the energy threshold — on calm material a cut reads as a glitch. Even hard cuts are 2-frame ramps (a 1-frame swap of bright scenes is a flash).
• Framing lives at the shot level (size constant per shot, gaze live); the outgoing shot keeps travelling as it fades — freezing it stops the old image dead before it disappears.
• Sharp transitions cut colour instantly with the image and always reframe (a bigger reframe than a dissolve earns).
• Story variables are staged, not ramped: flat inside a section, blended ~1 bar after boundaries. Moments (arrival/turn/climax/resolution) are found in the audio; the story never overrides the song (bounded bias terms). All scenes share one axis sign so the video travels one way.

6. Colour and variety

• Palettes in OKLCH; hue from spectral tilt (log treble/body ratio — centroid and plain body-fraction both collapsed across masters); both ways round the wheel; directors choose 1–4 palettes, blended in OKLCH across dissolves.
• Per-track temperament (intensity/pace/detail/extremity) moves every param dial so one scene cast in two videos isn't the same footage twice.
• Overlays on 5–55% of stacks, never normal blend, never above 0.6 opacity — occasionally layered, never permanently cluttered.
• A scene varies when the SONG changes what's on screen: draw castMain/castChorus (form in 3D), size via stageScale(), place via stageNode(). No effect-knobs (glow/grain/chroma belong to post).
• Grain is a per-track treatment (mode/cell/rate/mask/chroma, ~45% of tracks get none); scene texture: 0..2 scales only its share.
• Absolute animation speed follows the song: motion bias is tempo-dominated and every rate: true param scales by per-track rateScale — slow songs got fast scenes before this.
• Venues have a per-track character the same way: the dungeon derives sealed/murky/big (dungeonCharacter: tonal darkness, doubling-compensated tempo, stillness, weight) from the measured summary, calibrated against the bank because arrangement intent doesn't survive measurement (ambient doubles tempo, hiss reads bright). Dark slow songs seal fully indoor; blazing masters always crack open.
• The dungeon also gets a per-track visual theme (dungeonTheme: crypt/overgrown/ember/flooded/windswept), scored off the same summary traits — one album must not render the same dungeon twice. Pace terms gate on tempoConfidence (doubled detectors must not read fast), jitter is small enough that decided races stay decided across seeds.

7. Grounds and composition

• surface is measured (metadata.json), never declared — nine scenes claimed canvas while painting under a third of the frame.
• A ground fills the frame on average and never vanishes under any identity (worst case counts, not the mean); hollow grounds are flattened via groundPersonalityFrom.
• Model layers share one Compositor perspective/depth rig; shader depth stays faked. Venue rooms are furnished with the track's monolith body so the room reads as this track's place.
• With a room present the viz plays ON its screens (window/wall/radio), never composited over it; model viz additionally stand in the shared world for shared-rig rooms. A directCamera room (ride) owns only its own rig — the shared rig keeps the subject grammar, or the viz is filmed through a room camera meant for another world and vanishes.

8. 3D on the same terms

• ActorSpec is data, not scene graph (serialisable, hashable); geometry via actorToGeometry from the same Identity.form the shaders march.
• Model motion is analytic, instanced, seeded; kit deformation reuses the song's silhouette params; castSolid impostors remain the fallback.
• New actors are a file plus a registry entry, gated like scenes.

9. Export and review

• WebCodecs primary, frame-dump+ffmpeg fallback; title card, edge fades, progress/cancel; A/V sync within one frame; survives a real upload test.
• Preview is the primary surface: transport, section jump/loop, live param edit, timeline strip, re-roll + locks, draft/full quality, 20s segment test-renders. Never commit to a full render blind.
• Every phase gate is measured; the battery (6 tracks ambient→hard EDM) is watched end to end — monotony is invisible to automation.

10. History

• Forked from party-stage, then detached: zero cross-imports.
• Corrections kept: boundary seeks need warm-up (not exact for free); layout is blind on full-frame fields; aggregate ratios (e.g. separation) swing run-to-run — steer by direct render comparisons.