dsh-video-storyboard

dsh-video-storyboard

A scene-aware storyboard for text-only agents. Feed a video in, get one addressable grid out: numbered cells, a timestamp per cell, and the transcript line that falls inside it — so the model can look, cite a cell, and ask for a close-up instead of being told a summary.

The effect, in numbers. A 5-minute clip becomes one image — 9–16 frames for 384 tokens, where a frame-per-image tool spends 3,072–6,144 for the same coverage. A 90-second PV scans in 9 s and returns 12 cells whose shot boundaries match 41 detected cuts. Second call on the same file: 3.6 s (cached). No model weights, no image library, no font config.

Community project. Not affiliated with DeepSeek.

中文说明：README.zh.md — 创新点、实现原理、实测基线、路线图。

Requires: Node ≥ 20, ffmpeg/ffprobe on PATH. No other runtime dependencies.

A 27-second synthetic clip (9 shots, loud/quiet/loud audio) turned into one image: 9 numbered cells, one timestamp range each, and a footer strip showing loudness, silence and cut points on the same time axis. The clip is generated by ffmpeg — no third-party footage is used anywhere in this repo.

What the algorithm actually does

Eight decisions, each made because the obvious alternative failed on real footage:

#	Decision	Why not the obvious thing	Code
1	Scene-change detection on HSV at 64×36, three-channel mean of |ΔH|+|ΔS|+|ΔV|	Luma-only difference misses cuts that swap hue at equal brightness (2 of 9 cuts in our fixture; ffmpeg scdet finds 4/9, we find 9/9)	src/scan.js
2	Threshold chosen by rate, not by a score multiple — target ≈1 cut per 1.5 s, floor at score 12	3×p95 scored 38 on B-roll (fine) but 130 on fast-cut anime → zero cuts, on exactly the content that needs the most cells	src/scan.js
3	Cell placement = equal increments of the local cut-rate integral, snapped to real cuts (≤30 % of an ideal cell)	Uniform splitting gives a 6 s burst of 20 shots the same space as a 20 s static shot; snapping without a limit silently distorts the weights	src/density.js
4	Grid size from motion p90, not from duration	16 cells over a static single take = 16 identical frames; 4 cells over a PV = nothing	src/plan.js
5	Minimum gap capped by the material's own granularity (min(minGap, medianShot×0.9))	A fixed 1.2 s merged six 1 s shots into three, and then could not reach the requested cell count at all	src/segments.js
6	Sharpest frame wins — 3 candidates per segment, highest Laplacian variance	thumbnail=n= picks by histogram; blur does not change a histogram, so it happily returns a Gaussian-smeared credit card	src/sheet.js
7	Audio envelope on the same axis, inside the same image	A separate audio image costs another 384 tokens and loses the alignment, which is the entire point	src/audio.js, src/sheet.js
8	ASR chunked to ≤45 s at silence midpoints, non-speech segments filtered and reported	One 299 s request shifted every timestamp 13 s early; a music-only PV produced a fake 29 s line that landed in 5 cells	src/asr.js, src/pipeline.js

The two-stage read, and what a long film actually costs

A 2 h 35 m film is not 24 images up front (9,216 tokens). The cheap path is:

stage	cost	what you learn
video_probe	0	length, audio/subtitle tracks, how to read it
one overview (16 cells over the whole file)	384	structure — which 7.5-minute stretch is worth opening
open the 2–3 suspicious stretches	3 × 384	16 cells of detail each
total	1,536	same price as four images, but the first 384 already tells you whether to continue

The saving is not in shrinking any single image — it is in letting the model stop early. That is why video_probe returns a plan rather than a picture, and why textOnly paging exists.

Prior art, accuracy ceilings we do not try to beat, and the things we deliberately left out: docs/REFERENCES.md.

Long films: the text index has a budget

One image is 384 tokens, but the text index is unbounded — a 2 h 35 min film produced 64 cells and a 75 KB index, which exceeded the host's tool-result limit. The excess was folded into a file on disk and never read, so 44 minutes of dialogue were silently missing. Three things prevent that now:

• Per-cell cap (180 chars by default). An overview needs "what is this stretch about", not the full line.
• Cell-aligned truncation with an explicit continuation hint: [text stopped at cell 13 of 64 — call again with cellRange: [14, 64] and textOnly: true to read the rest]
• cellRange + textOnly — page through the index without re-rendering images or re-decoding frames.

video_storyboard({ input, cells: 16, sheets: 4 })                     // 4 images + first page of text
video_storyboard({ input, cells: 16, sheets: 4, cellRange: [14, 42], textOnly: true })
video_storyboard({ input, cells: 16, sheets: 4, cellRange: [43, 64], textOnly: true })

textOnly returns no image, so paging costs nothing in tokens; with the scan and ASR results cached it takes ~0.8 s per page after the first call.

CLI batch mode for a whole film, where 24 separate calls is not a workflow:

node bin/storyboard.mjs film.mp4 --batch 300 --out film.png
# → film-w01.png … film-w24.png  +  film-timeline.md (window | cells | cuts | chars | image)

A tail window under 30 s is merged into the previous one instead of spending 384 tokens on a 2-second window that holds a single cell. Window boundaries are snapped to real cuts (one full scan up front, --no-snap to disable), so a window never ends in the middle of a sentence.

Quick start

git clone https://github.com/SAOS-3624/dsh-video-storyboard.git
cd dsh-video-storyboard

node --test                                          # 154 tests (needs ffmpeg)
node bin/storyboard.mjs your-video.mp4 --cells 9 --text   # see exactly what the model would get
node bin/storyboard.mjs film.mp4 --batch 300              # long film: one png per 5-min window + a timeline.md
node scripts/install-dev.mjs                         # mount into DSH, then restart DSH

Installing into DSH. For a normal profile:

dsh plugin --profile web add dsh-video-storyboard

The desktop profile is managed exclusively by the Electron app, so dsh plugin refuses it — use node scripts/install-dev.mjs instead (it copies the package into the profile and registers the bundle). Either way, restart DSH afterwards: the plugin tree is composed at process start.

How sampling actually works (and what it is not)

Worth stating plainly, because it is easy to misread:

• Tokens are billed per image, not per frame. Everything here exists to put many frames into one image. 9–16 frames = 384 tokens.
• All cells are the same size. This tool does not shrink resolution in fast-cut sections. What it varies is how many seconds each cell covers — dense regions get short cells, calm regions get long ones (density(t) = clamp(localCutRate / meanCutRate, 1, 2.5)). Why not variable resolution: at 350×194 a halved cell is mush, and the token cost is flat anyway.
• The timeline is fully covered; individual shots are not guaranteed. 300 shots in 5 minutes across 16 cells cannot all be seen. That is what video_zoom is for.
• Continuous motion is not a cut. Any frame-difference scorer (this one, scdet, TransNetV2) gives high scores to a fast pan or a zoom. That is why peaks are suppressed within 0.35 s and why short segments get merged instead of each taking a cell.

Status (v0.1)

Piece	State
src/plan.js — layout / coverage / grouping / guardrails (pure functions)	✅ implemented
src/scan.js — HSV-weighted shot boundary detection	✅ implemented — 9/9 cuts, 0 false positives on the fixture, 47× realtime
src/segments.js — cuts → segments → fit to N cells	✅ implemented
src/probe.js — ffprobe + rotation / end-edge handling	✅ implemented
src/sheet.js — extract + cover-crop + burn index + tile, zero image library	✅ implemented
adaptive threshold + min-gap + density-driven grid sizing	✅ validated on 6 real clips
src/pipeline.js — orchestration shared by CLI and tool	✅ implemented
src/index.js — registers video_storyboard + video_zoom	✅ implemented, host contract verified
src/audio.js — loudness (EBU R128) + silence, no model	✅ implemented
src/asr.js — subtitle track + OpenAI-compatible ASR + cell alignment	✅ implemented (client verified against a local fake server)
src/cache.js — scan-result cache keyed by file+window+threshold	✅ implemented (2nd call on a 5-min clip: 13.4 s → 3.6 s)

npm test                                                  # 154 passing (unit + ffmpeg e2e + host contract)
node bin/scan-video.mjs video.mp4 9                       # 调阈值：切点 / 分数 / 段长 / top peaks
node bin/storyboard.mjs video.mp4 --cells 9               # 端到端出一张图 + manifest
node bin/storyboard.mjs film.mp4 --from 600 --to 780      # 只看某一段（不扫全片）
node bin/storyboard.mjs film.mp4 --plan 300               # 先出分段计划（不渲染）

Against ffmpeg -vf scdet on the same clip: scdet's default threshold finds 4 of 9 hard cuts; this scanner finds 9 of 9 with a 2× score margin to the strongest non-cut frame. Reason: scdet scores mean absolute frame difference (luma-ish), so a cut that swaps hue at equal brightness is invisible to it. Measured on the two such cuts in the fixture: brightness delta 0.6–0.8 %, hue delta 24/255 and 151/255.

Real-world check (6 clips of 1080p footage: B-roll, anime, PVs, a single take)

clip	len	p90	auto th	cuts	gap	grid
B-roll (50 mm lens)	299 s	7.9	16	141	2.1 s	16 ×1
anime feature	1090 s	15.0	43	408	2.7 s	16 ×2
special ED	80 s	11.6	21	42	1.9 s	4 ×1
PV (official)	92 s	17.0	54	44	2.1 s	9 ×1
PV (concept)	90 s	26.1	58	41	2.2 s	9 ×1
party singing (single take)	251 s	2.2	12	1	—	4 ×1

The threshold is chosen by rate, not by a score multiple. My first attempt was 3 × p95, which looked fine on the B-roll clip (38 → 88 cuts) and then died on fast-cut anime: the concept PV's p95 is 43, so ×3 = 130 → zero cuts, on the exact content that needs the most cells. Targeting ≈1 cut per 1.5 s (auto-tightened to 0.5 s on short clips, floored at score 12 so static footage stays a single shot) works across all six.

Grid size follows content density (p90), not duration alone: ≥15 → 12 s/cell, 5–15 → 20 s/cell, −18 && charsPerSec < 1` marks a music-driven clip, which keeps the cell floor at 9 instead of letting the "dialogue carries the info" rule shrink it. See docs/BASELINE.md.

Term dictionary. ASR w