dejvino/music-video-gen
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Make grain a treatment, tempo a governor, and params commit harder

Browse Source 
Grain was in every video. It was added twice unconditionally — every scene
called sigGrain, and the grade added its own on top — so the only thing that
varied between two tracks was how much of it there was. That makes grain the
renderer's fingerprint rather than a decision about one video.

It is now described rather than dialled (look/grain.js): a mode (off /
constant / swell / sections / transient), a cell size in pixels, a refresh
rate in frames, a mask (uniform, shadows, highlights, edges, bands) and a
chroma amount. Roughly 45% of tracks get none at all. The non-constant modes
carry a per-frame envelope computed in Show._postAt from frame and features
only, so preview and export still agree. Scene-side grain is gated the same
way, and a module can decline it outright with `texture: 0` — crisp line work
should stay crisp. The post tab grew a real grain block so any of it can be
forced per track.

Slow songs got fast scenes. `motion` bias was mostly section energy with
tempo as a small correction, so a 70bpm track's drop asked for nearly as much
speed as a 150bpm one. Motion is now tempo-dominated, and every `rate: true`
param is additionally scaled by a per-track rateScale — measured, 84bpm now
samples its rate params at 0.276 of range against 148bpm's 0.571.

Parameter sampling also commits harder: extremity starts at 0.45 rather than
0.25 and shapes the draw more aggressively. This was first pushed to 0.82 and
backed off to 0.72, because the gates caught the overshoot — seeds began
collapsing onto the same range ends and a sparse scene sampled at its low end
rendered effectively black.

Fallout worth recording: turning the default grade grain off exposed two
scenes that were never really animating. Dust Chamber and Eclipse Field
passed the Phase 7 movement gate only because per-pixel noise was moving
underneath them; both now breathe on their own fixed clock, and Dust Chamber
needed a brightness floor as well. Four scenes (Classic Wave, Silk Ribbon,
Kaleido Tunnel, Slow Orb) express the style trait ONLY through grain and so
cannot opt out yet; they hold a reduced share at 0.35 pending real edge and
softness response.

Phase 3's look-space check now measures its closest pair relative to image
brightness, the same correction Phase 10 already documents for sparse scenes:
the absolute number was being propped up by grain rather than by look-space
width. Five new Phase 10 checks cover grain distribution, treatment variety,
envelope range, the texture opt-out, and tempo. 93/93 pass.

Two transport bugs, both stale state surfacing in the UI:

- Loading a track replaces audio.src, which stops playback silently, so
  state.playing stayed true and the play button stayed on pause — the first
  click after a track change only flipped the flag back. Added stopPlayback().
- The controls row wrapped mid-song because two readouts that change length
  while playing were sized by their content: the clock crossing ten minutes
  and the section label whenever a scene name is long or a crossfade appears.
  The clock is now fixed-width with tabular figures and the section label is
  the row's only flexible item, laying out at zero width and ellipsising into
  whatever space is left. The two spacers competed with it for that space and
  are gone; its own text-align does their job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Dejvino
2026-08-06 07:04:27 +02:00
co-authored by Claude Opus 5
parent 44e1826fcb
commit 14d1204e82
34 changed files with 756 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
flow-state/src/look/LookGenerator.js
+30 -7
View File
@@ -10,6 +10,7 @@ import { scenes, scenesInFamily, FAMILIES } from '../scenes/registry.js';
import { sampleValues, defaultValues } from '../params/schema.js';
import { planShots } from './shots.js';
import { generatePersonality, sceneHonours, describePersonality } from './Personality.js';
import { deriveGrain, describeGrain, applyGrainToPost } from './grain.js';
/**
* Which families suit which section kind, in preference order.
@@ -43,13 +44,26 @@ function biasFor(section, summary) {
const measured = Math.min(1, section.energy / Math.max(1e-6, summary.meanLoudness * 1.6));
const energy = kindEnergy * 0.6 + measured * 0.4;
// 60bpm → 0, 180bpm → 1. Tempo, not energy, is what a viewer reads as
// "this is moving too fast for the song": a slow track can have a huge drop
// and still want scenes that drift. Motion used to be mostly energy with
// tempo as a small correction, which is why a 70bpm ballad got a drop
// biased to 0.9 motion and scenes that skittered over it.
const tempo = clamp01((summary.bpm - 60) / 120);
return {
energy,
density: Math.min(1, energy * 0.7 + section.flux * 1.2),
motion: Math.min(1, 0.25 + energy * 0.5 + (summary.bpm - 90) / 180),
motion: clamp01(0.12 + tempo * 0.55 + energy * 0.28),
// Applied on top of every `rate: true` param, so absolute animation
// speed scales with the song rather than only its sampled position in
// a range. Bounded well short of a stop or a blur. See params/schema.js.
rateScale: 0.45 + tempo * 0.95,
};
}
const clamp01 = (x) => Math.max(0, Math.min(1, x));
/**
* Scenes eligible for a section kind, weighted by how well the family fits.
*
@@ -146,10 +160,14 @@ function assignRostersByKind(sections, rng, signature = []) {
/**
* Post-processing and feedback derived from track character.
* Ambient material gets more feedback and bloom and less grain; dense club
* material gets tighter, punchier settings.
* Ambient material gets more feedback and bloom; dense club material gets
* tighter, punchier settings.
*
* Grain is NOT decided here — see look/grain.js. `post.grain` carries only the
* amount for the current frame, which the Show multiplies by the grain
* envelope, so a track can be clean, permanently dirty, or anything between.
*/
function derivePost(summary, rng) {
function derivePost(summary, rng, grain) {
const bright = summary.meanCentroid;
const noisy = Math.min(1, summary.meanFlatness * 3);
const dynamic = Math.min(1, summary.dynamicRange);
@@ -160,7 +178,7 @@ function derivePost(summary, rng) {
bloomThreshold: 0.45 + bright * 0.25,
bloomKnee: 0.25,
chroma: 0.05 + noisy * 0.25 + rng.range(0, 0.08),
grain: 0.012 + noisy * 0.03,
...applyGrainToPost(grain, {}),
vignette: 0.25 + (1 - bright) * 0.25,
contrast: 1.0 + dynamic * 0.15,
saturation: 1.0 + (1 - noisy) * 0.25,
@@ -272,7 +290,10 @@ export function generateLook(track, { seed = null, samples = null, overrides = n
const rosterByKind = assignRostersByKind(
track.sections, rng.fork('scenes'), personality.signature);
const { post, feedback } = derivePost(summary, rng.fork('post'));
// The grain treatment: usually none, and when present described rather than
// dialled. See look/grain.js.
const grain = deriveGrain(summary, rng.fork('grain'));
const { post, feedback } = derivePost(summary, rng.fork('post'), grain);
// Scenes that declare role 'accent' composite over a background rather than
// being one — most of their frame is empty by design. They are never chosen
@@ -326,6 +347,7 @@ export function generateLook(track, { seed = null, samples = null, overrides = n
palette,
personality,
paletteScheme: paletteSource.lastScheme,
grain,
post,
feedback,
sections,
@@ -403,7 +425,8 @@ function applyOverrides(look, overrides) {
export function describeLook(look) {
const kinds = look.sections.map((s) => `${s.kind}:${s.layers[0].module.name}`);
return `seed ${look.seed.toString(16)} · ${look.paletteScheme} · ` +
`${describePersonality(look.personality)} · ${[...new Set(kinds)].join(', ')}`;
`${describePersonality(look.personality)} · ${describeGrain(look.grain)} · ` +
`${[...new Set(kinds)].join(', ')}`;
}
export { defaultValues };
flow-state/src/look/Personality.js
+17 -9
View File
@@ -105,11 +105,13 @@ export function generatePersonality(summary, rng, countEligible = null) {
const style = {
lineWeight: 0.4 + bright * 0.4 + rng.range(-0.15, 0.25),
softness: 0.25 + (1 - bright) * 0.4 + rng.range(-0.1, 0.2),
// Surface grain is a texture trait, not a default. The old floor of
// ~0.12 put visible film grain on even a perfectly tonal track, and
// since every scene adds sigGrain AND the grade adds its own, that read
// as "grainy by default". Only tracks that want grit carry any.
texture: noisy * 0.4 + rng.range(0, 0.1),
// Surface grain is a texture trait, not a default, and most tracks have
// none at all. Every scene adds sigGrain and the grade can add its own,
// so anything short of a hard gate here reads as "grainy by default" —
// which is exactly what it read as when this was a floor of ~0.12 and
// then again when it was a small unconditional amount. A scene can also
// opt out entirely with `texture: 0` in its module.
texture: rng.bool(0.25 + noisy * 0.4) ? noisy * 0.35 + rng.range(0.02, 0.12) : 0,
// Fold counts stay low and are usually off. Symmetry is the fastest way
// to make a library look like one series and also the fastest way to
// make every track look like a screensaver.
@@ -128,8 +130,11 @@ export function generatePersonality(summary, rng, countEligible = null) {
// a quiet track can be intricate and a loud one can be blunt.
detail: rng.range(-0.6, 0.6),
// How far toward the ends of a range this track is willing to sample.
// The single most effective knob against "every video looks average".
extremity: rng.range(0.25, 0.95),
// The single most effective knob against "every video looks average",
// so the floor sits well above timid: a track at 0.25 sampled almost
// uniformly and produced the library's average look, and enough tracks
// did that to make the average look like the house style.
extremity: rng.range(0.45, 1.0),
};
const signature = pickSignature(rng, countEligible);
@@ -172,9 +177,12 @@ export function sceneHonours(module, signature) {
* the library regression checks build layers directly and would otherwise all
* shift at once.
*/
export function signatureUniforms(personality) {
export function signatureUniforms(personality, module = null) {
if (!personality) return NEUTRAL_UNIFORMS;
const { shape, camera, space, style } = personality;
// A scene may scale — or refuse — the track's surface grain. A clean vector
// look has no business being speckled just because the track is gritty.
const textureAffinity = module && module.texture !== undefined ? module.texture : 1;
return {
u_sigSides: shape.sides,
u_sigRound: shape.roundness,
@@ -195,7 +203,7 @@ export function signatureUniforms(personality) {
u_sigLine: style.lineWeight,
u_sigSoft: style.softness,
u_sigTexture: style.texture,
u_sigTexture: style.texture * textureAffinity,
u_sigFold: style.symmetry,
};
}
flow-state/src/look/grain.js
+193
View File
@@ -0,0 +1,193 @@
// Grain as a deliberate treatment rather than a permanent surface.
//
// Grain used to be two unconditional additions — every scene added sigGrain and
// the grade added its own on top — with only the amount varying. The result was
// that every track in the library was grainy, which made grain read as the
// renderer's fingerprint instead of as a choice about one video.
//
// So grain is now DESCRIBED, not dialled:
//
// mode — when it is present at all. A third of tracks get none, and of the
// rest most only carry it some of the time.
// scale — the size of a noise cell in pixels. 1 is film-fine; 5 is a coarse
// dither that reads as a different medium entirely.
// rate — how many frames a noise field survives. 1 boils; 5 is sticky
// static that sits on the image like dirt on a lens.
// mask — where it lands. Only in the shadows, only in the highlights, only
// toward the edges, or in horizontal bands.
// chroma — mono speckle or colour speckle.
//
// Two tracks that both "have grain" should still not look like each other.
/** Mask ids, mirrored in COMPOSITE_FRAG. */
export const GRAIN_MASKS = { uniform: 0, shadows: 1, highlights: 2, edges: 3, bands: 4 };
export const GRAIN_MODES = ['off', 'constant', 'swell', 'sections', 'transient'];
/** Section kinds a 'sections' grain can be pinned to. */
const GATEABLE_KINDS = ['intro', 'build', 'drop', 'sustain', 'breakdown', 'outro'];
export const NO_GRAIN = {
mode: 'off',
amount: 0,
scale: 1,
rate: 1,
mask: 'uniform',
chroma: 0,
kinds: [],
period: 24,
duty: 0.4,
};
/**
* The track's grain treatment.
*
* `noisy` (spectral flatness) tilts the odds but never forces the issue: a
* clean tonal track can still be the one that gets heavy dirt, because that is
* a legitimate art-direction choice and predictable mapping is what made the
* library uniform in the first place.
*/
export function deriveGrain(summary, rng) {
const noisy = Math.min(1, (summary.meanFlatness ?? 0.2) * 3);
// 'off' is the single most likely outcome, and deliberately so — a library
// where two videos in five have no grain at all is what makes the ones that
// do read as a decision.
const mode = rng.pickWeighted(
['off', 'constant', 'swell', 'sections', 'transient'],
[6 - noisy * 3, 1.5 + noisy * 2.5, 2.5, 2.5, 2],
);
if (mode === 'off') return { ...NO_GRAIN };
// Constant grain has to live with the image for the whole video, so it is
// held well below what an intermittent treatment can get away with.
const ceiling = mode === 'constant' ? 0.05 : 0.11;
const amount = rng.range(0.015, ceiling);
const scale = rng.pickWeighted([1, 1.5, 2, 3, 5], [4, 3, 3, 2, 1]);
// Coarse cells that also boil every frame read as a broken video signal, so
// the bigger the cell the more likely it is to hold still for a few frames.
const rate = rng.pickWeighted([1, 2, 3, 5], [5, 2 + scale, 1 + scale, scale]);
const mask = rng.pickWeighted(
['uniform', 'shadows', 'highlights', 'edges', 'bands'],
[3, 3, 1.5, 2, 1],
);
const kinds = mode === 'sections' ? pickKinds(rng) : [];
return {
mode,
amount,
scale,
rate,
mask,
// Colour speckle is the loudest of these choices and stays rare.
chroma: rng.bool(0.25) ? rng.range(0.3, 1) : 0,
kinds,
// Swell period in seconds. Long enough that it reads as the image
// breathing rather than as a flicker.
period: rng.range(12, 40),
duty: rng.range(0.25, 0.6),
};
}
/** One to three section kinds this grain belongs to. */
function pickKinds(rng) {
const count = rng.pickWeighted([1, 2, 3], [3, 3, 1]);
const pool = [...GATEABLE_KINDS];
const out = [];
for (let i = 0; i < count && pool.length; i++) {
const pick = rng.pick(pool);
out.push(pick);
pool.splice(pool.indexOf(pick), 1);
}
return out;
}
/**
* The 0..1 envelope on the grain amount for one frame.
*
* Deterministic in frame and features only — no state, no random source — so a
* preview frame and the exported frame agree, which is the same rule the noise
* itself follows.
*
* @param {object} spec from deriveGrain
* @param {object} ctx
* @param {number} ctx.time seconds into the track
* @param {string} ctx.sectionKind kind of the section this frame is in
* @param {object} ctx.features FeatureTrack row for this frame
*/
export function grainEnvelope(spec, { time = 0, sectionKind = '', features = null } = {}) {
if (!spec || spec.mode === 'off' || spec.amount <= 0) return 0;
switch (spec.mode) {
case 'constant':
return 1;
case 'swell': {
// Raised cosine over `period`, on for `duty` of it. The image drifts
// into grain and back out with nothing in the audio triggering it,
// which is what makes it feel like film rather than like a reaction.
const phase = (time % spec.period) / spec.period;
if (phase > spec.duty) return 0;
return 0.5 - 0.5 * Math.cos((phase / spec.duty) * Math.PI * 2);
}
case 'sections': {
if (!spec.kinds.includes(sectionKind)) return 0;
// Ease across the section edges so grain arrives with the section
// rather than snapping on at the cut.
const p = features ? features.sectionProgress : 0.5;
return smoothstep(0, 0.08, p) * smoothstep(0, 0.08, 1 - p);
}
case 'transient': {
if (!features) return 0;
// Rides flux, so grain answers hits and edits. Floored slightly
// above zero on loud material so it does not strobe on and off.
const hit = Math.min(1, (features.flux ?? 0) * 2.5);
const bed = Math.min(0.35, (features.sectionEnergy ?? 0) * 0.35);
return Math.max(bed, hit);
}
default:
return 1;
}
}
function smoothstep(a, b, x) {
const t = Math.max(0, Math.min(1, (x - a) / (b - a)));
return t * t * (3 - 2 * t);
}
/**
* Push a grain spec's static fields into a post object.
*
* The amount is the exception: for anything other than 'constant' it is owned
* by the per-frame envelope (see Show._postAt), so it is set to zero here and
* the envelope writes it every frame.
*/
export function applyGrainToPost(spec, post) {
post.grain = spec.mode === 'constant' ? spec.amount : 0;
post.grainScale = spec.scale;
post.grainRate = spec.rate;
post.grainMask = GRAIN_MASKS[spec.mask] ?? 0;
post.grainChroma = spec.chroma;
return post;
}
/** One line for the look panel and check output. */
export function describeGrain(spec) {
if (!spec || spec.mode === 'off') return 'grain: none';
const bits = [
`grain: ${spec.mode}`,
`${spec.amount.toFixed(3)}`,
`${spec.scale}px`,
spec.rate > 1 ? `every ${spec.rate}f` : 'per frame',
spec.mask,
];
if (spec.chroma > 0) bits.push('colour');
if (spec.kinds.length) bits.push(`on ${spec.kinds.join('+')}`);
return bits.join(' · ');
}