dejvino/music-video-gen
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Block Mosh, Neon City and Flora visualizers

Browse Source 
Three new scene shaders to widen the library, now that the main families are
settled. Each is a shader plus a params block, registered in the single
source of truth (scenes/registry.js), so casting, UI, arc and checks pick
them up automatically.

- Block Mosh (glitch): a datamosh. Blocks pull the feedback buffer along
  per-block strokes on a bar-quantized grid so the corruption steps like an
  edit rather than crawling, and spills on onsets. Fills the most iconic
  gap in the glitch family.
- Neon City (structural): a receding skyline with instanced lit windows,
  distinct from the rolling ridgelines and the road grid it sits alongside.
- Flora (organic): an abstract plant of swaying stalks, teardrop petals and
  a bloom corona, all stamped in the track's signature shape.

Also adds HOWTO-visualizers.md, a quick reference for building the next one.
This commit is contained in:
Dejvino 2026-08-05 20:31:05 +02:00
parent ca5a68eb84
commit 7d59ef8e5f
6 changed files with 470 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

146
flow-state/HOWTO-visualizers.md Normal file
View File

@ -0,0 +1,146 @@
# How to build a visualizer (scene)
A scene is **a fragment shader plus a params block** — nothing else. Uniform
binding, the generated UI sliders, seeded per-track sampling, arc drift and the
phase-gate checks are all derived from the schema. There is no per-scene wiring.
Reference: `src/scenes/shader/kaleido-tunnel.js` (simple), `block-mosh.js`
(feedback/glitch), `scan-tear.js` (beat-quantised), `metaballs.js` (loops).
Read these before writing anything.
## The shape of a scene
```js
export const myScene = {
name: 'My Scene',
family: 'glitch', // flow | organic | minimal | structural | geometric | glitch
kind: 'fragment',
traits: ['camera', 'style'], // which personality traits it honours (see below)
params: {
density: { type: 'float', range: [0, 1], default: 0.5, uniform: 'u_density', bias: 'density' },
speed: { type: 'float', range: [0.1, 2], default: 0.5, uniform: 'u_speed', rate: true },
palette: { type: 'palette', count: 4 },
},
reactive: {
density: { feature: 'bandLow', amount: 0.3, response: 'smooth' },
},
shader: `
vec4 scene(vec2 uv, vec2 p) {
float t = u_time * u_speed + u_seed;
return vec4(palRamp(fbm(p * 4.0 + t, 4)), 1.0);
}
`,
};
```
Register it in `src/scenes/registry.js` (import + push into `MODULES`). Done.
## The shader contract
You write one function: `vec4 scene(vec2 uv, vec2 p)`.
- `uv` is 0..1 across the frame; `p` is centred, aspect-corrected, ~-1..1 on the
short axis. **Work in these, never pixels** — scale pixel-sized things by
`u_pixelScale` so a 720p preview matches a 4K export.
- The preamble is injected for you. Never declare `main()`, `u_resolution`,
`vUv`, or any contract uniform yourself.
**Uniforms available** (see `engine/shader-contract.js`):
- Frame: `u_time`, `u_frame`, `u_progress`, `u_seed`, `u_resolution`, `u_aspect`,
`u_pixelScale`, `u_opacity`.
- Audio, filled per frame from the FeatureTrack: `u_loudness`, `u_rms`,
`u_bandSub/low/mid/high/air`, `u_flux`, `u_centroid`, `u_flatness`, `u_width`,
`u_beat`, `u_beatPhase`, `u_barPhase`, `u_phrasePhase`, `u_sectionProgress`,
`u_sectionEnergy`, `u_buildSlope`.
- Personality (`u_sigSides`, `u_sigDrift`, `u_sigLine`, …) — see below.
- Feedback: `u_prev` sampler, read with `prev(vec2 uv)` (returns the previous
frame's colour; black if none). This is what makes trails/smear/datamosh work
for free.
**Helpers** (already in the preamble):
- Palette: `pal(int)`, `palRamp(float)` — always use these, or the look can't
recolour the scene.
- Noise: `hash11/12/22`, `vnoise`, `fbm`, `curl`, `rot`, `kaleido`.
- Personality: `sigShape`/`sigForm`, `sigCamera`, `sigFolded`, `sigEdge`,
`sigGrain`, `sigHorizonY`, `sigAir`.
- `sat(x)` = `clamp(x, 0, 1)`.
## Param schema fields
- `type`: `float` | `int` | `bool` | `vec2` | `palette`.
- `range` `[min, max]` required for numerics; `default` required unless you want
range[0] (prefer explicit defaults).
- `uniform`: the GLSL name. Must be `u_*` and must not collide with the contract
(see "traps").
- `bias`: which track-character axis nudges sampling — `energy`, `density`,
`motion`. This is how a loud drop gets denser scenes without the scene knowing
about audio.
- `rate: true`: **mandatory if the shader multiplies `u_time` by this param.**
- `reactive`: `{ feature, amount, response }`. `response`: `linear` (default) |
`spike` | `smooth` | `inverse`.
## Personality traits (`traits`)
The look generator builds each track on a signature of 1-2 traits, and a scene
that doesn't honour **all** of them is never cast in that track — so the library
intentionally shrinks per track. Only declare what you genuinely use:
| trait | what to call | lint evidence |
|---|---|---|
| `shape` | `sigShape` / `sigForm` | a `sigShape`/`sigForm(` call |
| `camera` | `sigCamera(p)` | a `sigCamera(` call |
| `space` | `sigHorizonY` / `sigAir` | those calls or `u_sigHorizon/Depth/Wash` |
| `style` | `sigEdge` / `sigGrain` / `sigFolded`, or `u_sigLine/Soft/Texture/Fold` | the calls / those uniforms |
The lint greps your shader and fails a declared trait with no evidence. Declaring
`[]` (none) is valid.
## Traps that have actually bitten here
1. **Anything multiplying `u_time` must be `rate: true`.** Phase is
`elapsed × rate`; modulating a rate jumps the phase by `elapsed × Δrate` — a
minute in a small wobble throws the image several units between frames, which
measured as strobing at 2× the accessibility limit. So don't react a rate
param, and add a *bounded* term instead: `u_time * u_speed + u_bandLow * 2.0`
is fine; `u_time * (u_speed + u_bandLow)` is not.
2. **Never reuse a contract uniform name** (`u_width`, `u_time`, `u_seed`, …).
It's a GLSL redefinition error; the only symptom is a black frame.
3. **Don't modulate whole-frame luminance on the beat.** A per-kick min→max→min
cycle is exactly what the WCAG 2.3.1 / Harding 3-flashes-per-second ceiling
bans. Pulse a *small local* term (per-block tint) instead, and use `smooth`
responses on loud things. If it's glitchy, **quantise it** — `floor(u_barPhase
* n) + floor(t * k) * n` makes corruption step on the grid instead of
crawling, which both reads better and stays below the flash rate (see
`scan-tear.js`, `block-mosh.js`).
4. **Determinism is absolute.** No `Math.random()`, `performance.now()`,
`Date.now()`, `new Date()` — use `hash*`/`fbm` for variation, and let time flow
through `u_time`/`u_frame` only. The lint greps for these.
5. **Set a base image.** A scene that only reads `prev()` is black for the first
frames and fragile under seek. Generate your own field underneath the effect.
6. Don't hardcode saturated `vec3(r,g,b)` literals when you declared a palette —
the lint flags more than two.
## Families
Chosen by section kind in the arc driver — a breakdown never lands on a strobing
glitch scene. Current counts (aim: 4-6 each): flow 2, organic 3, minimal 3,
structural 2, geometric 3, **glitch 3**. The thin families are the best place to
contribute next.
## Verify
```bash
npm run lint:scenes # schema/shader agreement both ways + determinism grep
npm test # audio pipeline (unaffected, but cheap)
```
Then the GPU gates in the browser at `http://localhost:5180/checks.html`:
Phase 2 (param range sweep — no NaN/black/white), Phase 5 (flash-rate sweep),
Phase 7 (per-scene acceptance: distinctness, liveness, animation, determinism,
4K budget). These iterate the registry, so a new scene is covered automatically
once registered.

2
flow-state/README.md
View File

@ -60,6 +60,8 @@ npm run lint:scenes # determinism grep + scene schema/shader agreement
## Adding a scene
Quick walkthrough: `HOWTO-visualizers.md`.
A scene is a shader plus a params block. Everything else — uniform binding, UI
controls, seeded per-track sampling, arc automation — is derived from the schema.

6
flow-state/src/scenes/registry.js
View File

@ -16,6 +16,9 @@ import { kaleidoTunnel } from './shader/kaleido-tunnel.js';
import { moireGrid } from './shader/moire-grid.js';
import { ridgeTerrain } from './shader/ridge-terrain.js';
import { scanTear } from './shader/scan-tear.js';
import { blockMosh } from './shader/block-mosh.js';
import { neonCity } from './shader/neon-city.js';
import { flora } from './shader/flora.js';
/**
* The scene library. Families exist so the arc driver can choose by section
@ -51,6 +54,9 @@ const MODULES = [
moireGrid,
ridgeTerrain,
scanTear,
blockMosh,
neonCity,
flora,
];
const errors = [];

74
flow-state/src/scenes/shader/block-mosh.js Normal file
View File

@ -0,0 +1,74 @@
// Glitch family: a datamosh. The frame is cut into a coarse grid of blocks and
// each block is pulled along a per-block stroke drawn from the PREVIOUS frame —
// so the corruption accrues and slithers across frames instead of tearing once.
//
// Everything steps on one quantised grid: the block field re-rolls on bar lines
// rather than crawling, which is what makes this read as an edit rather than as
// a smooth warp. On an onset the spill widens, which is the "hit".
export const blockMosh = {
name: 'Block Mosh',
family: 'glitch',
kind: 'fragment',
// Personality: see look/Personality.js.
traits: ['camera', 'style'],
params: {
block: { type: 'float', range: [4, 48], default: 22, uniform: 'u_blocks', bias: 'density' },
smear: { type: 'float', range: [0, 0.5], default: 0.18, uniform: 'u_smear', bias: 'energy' },
quantize: { type: 'float', range: [1, 8], default: 4, uniform: 'u_quantize' },
bleed: { type: 'float', range: [0, 1], default: 0.45, uniform: 'u_bleed' },
speed: { type: 'float', range: [0.1, 1.4], default: 0.5, uniform: 'u_speed', bias: 'motion', rate: true },
burst: { type: 'float', range: [0, 0.4], default: 0.12, uniform: 'u_burst' },
palette: { type: 'palette', count: 4 },
},
reactive: {
smear: { feature: 'bandHigh', amount: 0.3, response: 'smooth' },
bleed: { feature: 'flux', amount: 0.2, response: 'spike' },
burst: { feature: 'beat', amount: 0.25, response: 'smooth' },
},
shader: `
vec4 scene(vec2 uv, vec2 p) {
float t = u_time * u_speed + u_seed;
float beat = u_beat;
p = sigCamera(p);
// The thing being moshed: a sheared banded field, recoloured per track, so
// the corruption has real content to drag around even standing alone.
float shear = uv.y * (2.5 + u_sigLine * 3.0) + t * 0.45;
float field = fbm(vec2(uv.x * 1.2 + sin(shear) * 0.3, uv.y * 2.0 + t * 0.35), 3);
vec3 base = palRamp(field * 1.15 + uv.x * 0.2 + t * 0.03);
base *= 0.35 + 0.55 * sat(field * 1.5);
// One quantised grid for every jump below, so the mosh steps like an edit
// rather than crawling like a smooth warp.
float q = max(u_quantize, 1.0);
float st = floor(u_barPhase * q * 6.0) + floor(t * 4.0) * (q * 6.0);
vec2 cell = floor(uv * u_blocks);
// Each block gets one fixed stroke per step: a direction and a reach.
vec2 stroke = (hash22(cell + st * 31.0) - 0.5) * 2.0;
float reach = hash12(cell + st * 17.0);
vec2 dst = clamp(uv + stroke * u_smear * (0.3 + reach), 0.0, 1.0);
// The mosh pulls the PREVIOUS frame along the stroke and layers it over the
// fresh field. The previous frame is itself moshed, so smear accrues.
vec3 mosh = prev(dst);
float m = clamp(u_bleed * (0.5 + reach), 0.0, 1.0);
vec3 col = mix(base, mosh, m);
// Weak block tint, pulse-pinned to the beat but kept small — the frame as a
// whole never swings in luminance, so it stays clear of the WCAG 2.3.1
// flash ceiling even on a hard four-on-the-floor.
float blk = hash12(cell + st * 7.0);
col += pal(2) * u_burst * (0.5 + 0.5 * blk) * (0.5 + 0.5 * beat);
col += sigGrain(uv);
return vec4(col, 1.0);
}
`,
};
export default blockMosh;

119
flow-state/src/scenes/shader/flora.js Normal file
View File

@ -0,0 +1,119 @@
// Organic family: an abstract plant — a plume of swaying stalks with teardrop
// leaves and a glowing bloom at each crown.
//
// Abstract on purpose: no soil, no foreground, just the silhouette of growth.
// Every petal and the crown are stamped in the track's signature form, so a
// hexagonal track grows hexagonal flora. The stalk bends on bass and the bloom
// pulses on the beat — the plant dances to the track rather than the camera
// shaking.
export const flora = {
name: 'Flora',
family: 'organic',
kind: 'fragment',
// Personality: see look/Personality.js.
traits: ['shape', 'camera', 'style'],
params: {
count: { type: 'float', range: [1, 5], default: 3, uniform: 'u_count', bias: 'density' },
height: { type: 'float', range: [0.6, 2.0], default: 1.1, uniform: 'u_height', bias: 'energy' },
stem: { type: 'float', range: [0.008, 0.05], default: 0.02, uniform: 'u_stem' },
sway: { type: 'float', range: [0.05, 0.5], default: 0.2, uniform: 'u_sway', bias: 'motion' },
swayRate:{ type: 'float', range: [0.1, 1.2], default: 0.6, uniform: 'u_swayRate', rate: true },
leaf: { type: 'float', range: [0.12, 0.5], default: 0.26, uniform: 'u_leaf' },
bud: { type: 'float', range: [0, 1], default: 0.55, uniform: 'u_bud', bias: 'energy' },
palette: { type: 'palette', count: 4 },
},
reactive: {
sway: { feature: 'bandLow', amount: 0.25, response: 'smooth' },
bud: { feature: 'beat', amount: 0.3, response: 'smooth' },
},
shader: `
// Horizontal lean of a spine at parametric height h (0 base .. 1 tip), seeded
// per plant and swaying more toward the crown like a real stem. Returns a value
// scaled by u_sway later, so the reactive bend applies to every part at once.
float swayAt(float h, float phase, float t) {
return sin(t * (0.35 + fract(phase * 1.7)) + phase)
+ h * sin(t * (0.35 + fract(phase * 0.6)) + phase * 2.1);
}
// A petal: an elongated blossom drawn in the track's signature form, growing
// forward along dir from a node and clipped to a blade shape.
float petal(vec2 p, vec2 node, vec2 dir, float len, float width) {
vec2 off = p - node;
float along = dot(off, dir);
vec2 perp = vec2(-dir.y, dir.x);
float across = dot(off, perp);
float taper = 0.6 + 0.4 * sat(along / max(len, 1e-3));
vec2 q = vec2(along / max(len, 1e-3), across / max(width * taper, 1e-3));
float form = sigForm(q * 0.5, vec2(0.0), 0.5);
float clip = smoothstep(0.0, -0.1, along) * smoothstep(len + 0.12, len * 0.72, along);
return form * clip;
}
vec4 scene(vec2 uv, vec2 p) {
float t = u_time * u_swayRate + u_seed;
float beat = u_beat;
p = sigCamera(p);
// Soft vertical wash behind the plants.
vec3 col = mix(pal(0) * 0.03, pal(1) * 0.06, sat((p.y + 1.0) * 0.5));
for (int i = 0; i < 5; i++) {
if (i >= int(u_count)) break;
float fi = float(i);
float s = u_seed + fi * 43.7;
float baseX = (hash11(s * 1.7) - 0.5) * 1.25;
float height = u_height * (0.55 + 0.45 * hash11(s * 3.1));
float baseY = -0.92;
// Stem, sampled so the bend follows the sway curve rather than a chord.
float stemMask = 0.0;
for (int j = 0; j < 12; j++) {
float hj = (float(j) + 0.5) / 12.0;
float cx = baseX + swayAt(hj, s, t) * u_sway;
float cy = baseY + hj * height;
float d = length(p - vec2(cx, cy));
float taper = max(u_stem * (1.0 - hj * 0.8) + 0.002, 0.003);
stemMask = max(stemMask, smoothstep(taper + 0.004, taper - 0.004, d));
}
col = mix(col, mix(pal(1), pal(3), 0.3), stemMask);
// Stalk tip, reused for the petal rows and the bloom.
float tipX = baseX + swayAt(1.0, s, t) * u_sway;
float tipY = baseY + height;
// Petals up the upper half, alternating sides.
for (int l = 0; l < 6; l++) {
float fll = float(l);
float node = 0.45 + floor(fll * 0.5) * 0.12;
float side = fract(fll * 0.5) < 0.5 ? -1.0 : 1.0;
float nx = baseX + swayAt(node, s, t) * u_sway;
float ny = baseY + node * height;
vec2 dir = normalize(vec2(side, 0.55));
float len = u_leaf * 2.0;
float m = petal(p, vec2(nx, ny), dir, len, u_leaf);
vec3 leafCol = mix(pal(2), pal(3), fract(s * 0.4));
col = mix(col, leafCol, m);
col += pal(3) * m * (0.2 + 0.3 * beat);
}
// Bloom: a small corona of signature forms around the crown.
float bloom = 0.0;
for (int k = 0; k < 6; k++) {
float a = 6.2831853 * (float(k) + 0.5) / 6.0 + s;
vec2 off = vec2(cos(a), sin(a)) * u_leaf * (0.30 + 0.2 * beat);
bloom += sigForm(p - vec2(tipX, tipY), off, u_leaf * 0.5);
}
col = mix(col, pal(2), sat(bloom) * u_bud * (0.6 + 0.4 * beat));
}
col += sigGrain(uv);
return vec4(col, 1.0);
}
`,
};
export default flora;

123
flow-state/src/scenes/shader/neon-city.js Normal file
View File

@ -0,0 +1,123 @@
// Structural family: a receding skyline of towers with instanced lit windows.
//
// Distinct from Ridge Terrain (rolling fbm ridgelines) and Synthwave Run (a
// ground grid): this is architecture, read as a front-on skyline. Heights and
// footprints are per-tower noise, drawn far-to-near so a nearer tower correctly
// occludes the one behind while a taller far tower still rises above a short
// one in front. Windows are lit per facade cell, so the city has interior life
// rather than reading as a lit cardboard cut-out.
export const neonCity = {
name: 'Neon City',
family: 'structural',
kind: 'fragment',
// Personality: see look/Personality.js.
traits: ['space', 'camera', 'style'],
params: {
cells: { type: 'float', range: [0.5, 4], default: 2.0, uniform: 'u_cells', bias: 'density' },
layers: { type: 'int', range: [2, 8], default: 6, uniform: 'u_layers', bias: 'density' },
height: { type: 'float', range: [0.2, 2.5], default: 1.0, uniform: 'u_height', bias: 'energy' },
speed: { type: 'float', range: [0.02, 0.4], default: 0.08, uniform: 'u_speed', bias: 'motion', rate: true },
horizon: { type: 'float', range: [-0.3, 0.3], default: 0.0, uniform: 'u_horizon' },
haze: { type: 'float', range: [0, 1], default: 0.45, uniform: 'u_haze' },
reflect: { type: 'float', range: [0, 1], default: 0.5, uniform: 'u_reflect' },
pulse: { type: 'float', range: [0, 1], default: 0.5, uniform: 'u_pulse' },
glow: { type: 'float', range: [0, 1.5], default: 0.4, uniform: 'u_glow', bias: 'energy' },
palette: { type: 'palette', count: 4 },
},
reactive: {
height: { feature: 'bandLow', amount: 0.18, response: 'smooth' },
pulse: { feature: 'beat', amount: 0.2, response: 'smooth' },
},
shader: `
float towerEdge(vec2 grid, float mullions) {
vec2 g = fract(grid);
float steel = min(1.0 - smoothstep(0.0, 0.28 * mullions, g.x),
1.0 - smoothstep(0.0, 0.28 * mullions, g.y));
return steel;
}
vec4 scene(vec2 uv, vec2 p) {
float t = u_time * u_speed + u_seed;
float beat = u_beat;
p = sigCamera(p);
// Shares the track's ground line with every other scene that has one.
float horizon = clamp(u_horizon + sigHorizonY() * 0.3, -0.9, 0.9);
// Sky, darkening away from the horizon glow.
vec3 sky = mix(pal(3) * 0.06, pal(0) * 0.35,
smoothstep(-0.1, 0.7, p.y - horizon));
sky *= 0.4 + 0.6 * exp(-abs(p.y - horizon) * 3.0);
vec3 col = sky;
for (int i = 0; i < 8; i++) {
if (i >= u_layers) break;
float fi = float(i);
float depth = float(i) / max(float(u_layers - 1), 1.0); // 0 far .. 1 near
float parallax = mix(0.25, 1.0, depth);
// Horizontal column coordinate for this depth ring. Receding layers
// get more cells per screen and slide slower, giving parallax.
float xsc = mix(1.2, 3.4, depth) * u_cells;
float xc = p.x * xsc + t * 0.18 * parallax + fi * 31.7;
float cx = floor(xc);
float xf = fract(xc);
float h1 = hash11(cx * 13.1 + fi * 7.3 + u_seed);
float h2 = hash11(cx * 61.7 + fi * 5.9 + u_seed * 0.7);
// Footprint half width, and the tower's top above its base.
float footprint = (0.55 + h1 * 0.35) * 0.5; // in cell fracs
float inTower = smoothstep(footprint + 0.04, footprint - 0.04,
abs(xf - 0.5));
float baseY = horizon - (0.05 + depth * 0.9);
float topY = baseY + u_height * (0.35 + h1 * 1.6) * mix(0.45, 1.0, depth);
float body = step(p.y, topY) * step(baseY, p.y);
float tower = inTower * body;
if (tower < 0.01) continue;
// Facade, tinted by how far into the screen the ring sits.
float gy = clamp((p.y - baseY) / max(topY - baseY, 1e-4), 0.0, 1.0);
vec3 facade = mix(pal(1), pal(3), depth);
// Instanced window grid — each facade cell decides itself lit or dark,
// with the pulse raising the lit population and the beat underlining it.
vec2 g = vec2(xf * 5.0, gy * 9.0);
vec2 cellg = floor(g);
float lit = step(0.42, hash12(cellg + fi * 3.1));
float steel = 1.0 - min(smoothstep(0.0, 0.35, fract(g.x)),
smoothstep(0.0, 0.35, fract(g.y)));
vec3 windowCol = mix(pal(2), pal(3), 0.3);
vec3 fut = mix(col, windowCol,
lit * steel * (0.45 + 0.55 * u_pulse + beat * 0.25));
// Ground-couple: darker, breath where the tower meets the street.
fut *= 0.7 + 0.3 * smoothstep(0.0, 0.5, gy);
// Haze pushes the far rings into the sky, the way distance actually does.
col = mix(col, fut, tower * (1.0 - u_haze * (1.0 - depth)));
}
// Street: a slate floor with a molten reflection of the nearest towers.
if (p.y < horizon) {
float street = smoothstep(0.0, -0.6, p.y - horizon);
col = mix(col, pal(3) * 0.08, street);
float refl = exp(-abs(p.y - horizon) * 4.0) * u_reflect;
col += pal(2) * refl * (0.08 + 0.2 * beat) * street;
}
// Horizon haze bloom, the city's glow pooling where towers meet the sky.
col += pal(2) * exp(-abs(p.y - horizon) * 9.0) * u_glow;
col = sigAir(col, p, smoothstep(0.0, 1.4, length(p)));
col += sigGrain(uv);
return vec4(col, 1.0);
}
`,
};
export default neonCity;