dsh-edit-apart

EditApart — AI taste-driven editor (agent preset)

EditApart is an automation-faithful editor for the EditTogether community. It takes raw footage, builds a shot inventory, proposes an EDIT DECISION SCHEMA against a declared editorial rubric, renders it deterministically with ffmpeg/scenedetect (and ImageMagick for photos), critiques the render against the rubric, and revises until it converges.

It does not operate a video-editing GUI. It reasons about editorial judgment. The plugin registers both video and photo tools plus the taste-model tools (15 total) under one identity; the engines are bin/edit_apart_core.py (video), bin/photo_core.py (photo) and bin/taste_model.py (the per-creator model).

The core design

• Schema is the interlingua. The same EDIT DECISION SCHEMA is the thing you train on, the thing you render from, and the thing you critique. No gap between "AI intent" and "the render."
• General grammar is shared; taste is per-creator. One shared style-brain (film-grammar parameters) + a small per-creator identity latent z_u in a per-creator GGUF. Only the identity is per-creator — the artifact holds z_u and a digest of the trunk it belongs to, nothing else.
• Reward is dense + group-relative (GRPO). The critic scores a render AND yields per-element keep/drop/why deltas; r = overall + λ·Σ(deltas). Group relative advantage A_k = (r_k − mean(r))/std(r), then a PPO-clip step.
• Muon on weights, AdamW on the latent. Muon orthogonalizes momentum for 2D weight matrices; never the identity latent (Newton-Schulz degenerates on 1D). AdamW handles z_u and all 1D/bias/embed params.
• Self-play interleaved with history. propose_schema emits a group of candidate schemas from a deterministic grid; the group is logged, one candidate is selected, and the render/vision/creator outcome is logged back against it. Training consumes those groups overnight.
• Three model portions, and only one is the session model. General (the session model: reads footage, proposes, critiques) · FilmGPT style-brain (shared film grammar) · creator taste (z_u). A video-in/out model like GLM-5.3 (Z.ai / zai, OpenAI-compatible) replaces the general portion only — it reads the raw clip via {"type":"video_url","video_url":{"url":…}} instead of pre-extracted frames; the FilmGPT and taste portions are untouched.
• Model-authored video is opt-in architecture, not a preset claim. ffmpeg renders the final cut (render_schema). Fully AI-generated clips come from a separate store plugin (or the harness video_generate tool, e.g. grok-imagine-video), installed opt-in. When present, you may generate a clip and insert it into a regular edit as a generated segment (source + AI clip are separate inputs the renderer concatenates). The preset itself never claims to generate video.
• Photo editing is image-only — one fewer modality. The same taste-driven loop edits a single image via the edit-apart plugin (bin/photo_core.py), which needs only image input (no video modality, no video-in/out registration), so it works with the existing image-capable general model. The ImageMagick renderer is resolved at runtime (DSH_IMAGEMAGICK/PATH) and is not a hard dependency — a web-profile in-browser renderer may supply it.

The pipeline

The editor's tools (mounted as a preset plugin at plugins/edit-apart.mjs) are the thin facade; bin/edit_apart_core.py (video) and bin/photo_core.py (photo) are the deterministic engines they shell out to.

source ─► inventory ─► shot inventory
               │
     features ─► per-shot luminance/motion/audio
               │
  propose_schema + rubric ─► GROUP of candidate EDIT DECISION SCHEMAS
               │                    │         (logged to .editapart/taste_samples.jsonl)
               │                    ▼
               │        taste identity scores the group ─► selected schema
               │
    render_schema ─► deterministic ffmpeg render
               │
    critic_edit ─► score + per-element keep/drop/why deltas
               │                    │         (logged as that candidate's reward;
               │                    │          chosen_by=creator|agent = revealed preference)
    revise_schema ─► revised schema ─► re-render (bounded loop)
               │
   train_identity ─► shared style-brain GGUF + per-creator z_u GGUF

Status of the taste model. The identity learning is implemented and wired into the loop, not a scaffold. propose_schema builds and logs a group of candidates, selects one with the creator identity when it exists, critic_edit logs the reward for the candidate that was actually rendered, and train_identity fits the shared style-brain plus that creator's latent z_u and writes real GGUF artifacts. The measured claims and their limits are in Taste model below — including what it does not do yet (trained selector, not a fine-tuned language model; video only).

Photo editing (image-only)

A parallel single-image loop, mounted via the edit-apart plugin: photo_inspect → photo_propose (rubric → photo EDIT DECISION SCHEMA) → photo_render (ImageMagick) → photo_critic (objective + subjective) → photo_revise. It needs one fewer modality than video (image, not video), and its renderer is not a hard dependency (resolved at runtime; a web-profile in-browser ImageMagick/ffmpeg may supply it).

The taste model is wired in here too, with its own photo/v1 layout, its own group log and its own style-brain (style_brain_photo_v1.gguf), so a creator ends up with one identity per modality. Three things make this modality harder than video, and all three are measured rather than assumed:

• No temporal axis to lean on, so the layout synthesises spatial structure: crop geometry plus statistics of the region the crop keeps, read from a 4×4 region grid that photo_inspect computes from the same downscale as the global stats (the global numbers are unchanged).
• Candidate scoring costs a render. The photo reward is defined on the rendered image, so a K-candidate group is rendered and scored K times (video's objective critic is pure arithmetic). group=1 with no dataset stays the legacy, render-free proposal.
• A quantised reward ties the group. Exposure was a threshold, so a 6-candidate group scored 6 × −0.05, spread 0.0 — exactly zero group-relative signal. The reward is now graded (full credit inside the tolerance, linear falloff outside), which took that group to spread 0.47 and left 11/18 training groups informative.

Measured on a synthetic 1600×1200 image, two creators with opposite framing taste (tightest vs loosest crop) sharing one frozen trunk: 10/10 held-out briefs from the training family are picked differently, 10/10 in the predicted direction (mean crop area 0.155 vs 0.389). On one unseen crop geometry it transfers too — 10/10 over ten distinct briefs — but with a smaller margin: mean crop area 0.30 vs 0.45, a gap of 0.15 against 0.233 in-distribution. That is the geometry/content confound of docs/paper-findings.md (Finding 4c) narrowing the effect, not erasing it. (An earlier "0/10, no transfer" reading came from the objective bug described under Corrections; it was an artifact and is withdrawn.)

A four-arm follow-up (tests/experiment_photo_transfer.py: {1 image, 3 images} × {fixed, varied crop family}, shared probe ladder, matched steps) found the repair is partial and along the content axis, not the geometry axis: only the 3-image/narrow-family arm transferred to an unseen image and geometry (3/3, ratio 0.400), while both varied-geometry arms gave 0/3. A no-taste control identity showed the ordering metric's noise floor is ±0.36 — larger than most effects measured — so treat these results as suggestive, not established. docs/paper-findings.md Findings 4c/4d has the tables, the power analysis and the rules (train on the family you will use; include an out-of-family check and a no-taste control).

Taste model (wired, measured)

bin/taste_model.py is the per-creator model; bin/train_identity.py is its CLI. The loop in bin/edit_apart_core.py drives it.

Architecture. A shared trunk over 16 named edit features (pacing, shot selection, tempo, energy, order) feeds a hidden layer that the creator latent modulates three ways:

u = tanh-free ranking score =
      (a ⊙ (1 + tanh(W_s z + b_s))) @ W2 + b2     # latent gates the hidden activations
    + (z @ W_z + b_z)                              # latent output offset
    + x · (z @ W_l)                                # per-creator linear readout over the features

z_u is the only per-creator tensor, so a creator artifact is ~1 KB while the trunk is shared. The map is nonlinear in z (tanh gates, multiplicative terms) and interpolatable (the latent is continuous; a midpoint latent scores between its endpoints — asserted in the test suite).

Objective. For each clip the loop logs a group of candidates with the critic's dense reward. GRPO turns that into group-relative advantages A_k = (r_k − mean r)/std r, and the update is a PPO-clip surrogate on the group-softmax selection policy, plus an auxiliary pairwise preference loss over the pick that was actually rendered and a small entropy bonus — but only when the pick came from an external judge (chosen_by=creator|agent|human). A choice the system itself made (taste: the identity's selection was rendered; critic: nobody picked, the objective critic only scored) trains neither the preference term nor the reward bonus. It still learns through the group-relative advantage over the critic's rewards, which measures the edit rather than claiming anything about the creator. The loop therefore records who picked, not just what was picked; taste_status reports how many groups were excluded for this reason. (If the group record and the later reward record disagree, the reward record — the engine's own flow — wins.) Utilities are standardised within the candidate group before the softmax: this keeps the logits responsive so the latent can always reorder candidates — with an unbounded-then-squashed output the utilities saturate under the margin objective and the latent loses all authority over the ordering (measured: 12/12 neutral briefs identical, versus 12/12 different after the fix).

Optimizer split. Muon (quintic Newton-Schulz, coefficients and nesterov form from the reference Muon implementation) on 2D weights; AdamW on z_u and every 1D/bias parameter. Muon's lr is in spectral-norm units, so on these small matrices it must be ~10× below an Adam lr — at the higher lr the model stops fitting (measured: training accuracy stalls at ~0.5 versus ~0.85).

Artifacts. Real GGUF v3 files (F32 tensors + metadata): a shared style_brain_video_v1.gguf and a tiny per-creator .gguf carrying z_u, the feature layout, and the digest of the trunk it was trained against. A digest mismatch is refused rather than silently scoring with the wrong trunk.

How the loop feeds it

# 1. propose a GROUP (logged automatically); the best candidate by the identity
#    is selected, or by the objective critic reward when no identity exists yet
propose_schema inventory= rubric= group=4

# 2. render the selected candidate, then log that group's outcome
critic_edit schema= inventory= rubric= group_id= \
            candidate= chosen_by=creator

# 3. inspect and train
taste_status
train_identity creator=erkin            # writes .editapart/identity.gguf
train_identity creator=erkin freeze_style=true   # later: only z_u moves

# the photo loop is the same protocol on its own log/identity
photo_propose src= inspect= rubric= group=4
photo_critic schema= rubric= result= group_id= \
             candidate= chosen_by=agent
taste_status modality=photo
train_identity modality=photo creator=erkin

chosen_by=creator|agent matters: the pick is logged as a revealed preference, becomes that group's top reward, and gives the trainer a preference term. A rubric-derived reward alone cannot identify per-user taste: with no user-dependent term in the objective, two creators with opposite tastes differ on only 4/12 neutral briefs (p ≈ 0.21 against the unrelated-picks null), versus 8/12 (p ≈ 0.0006) once the revealed pick is in the objective (margins +3.6 s vs +7.0…+9.4 s). The override additionally fits the creator's own corpus far better than the auxiliary loss (0.83/1.00 vs 0.22/0.26), so both mechanisms are kept; tests/experiment_revealed_preference.py reproduces the whole table.

Multi-input effects (the v2 layout)

The v1 feature layout describes the resulting cut sequence — shot counts and durations, cuts per minute, pace band, motion, coverage, order, luminance, audio RMS. A transition, a grade, ducking and an overlay are decisions about how inputs combine (a transition joins two clips, ducking relates speech to music, an overlay composites layers), and none of them appears in that list. So with v1 the identity could only choose which selection to use; it could not set an effect's parameters at all. That is a real limitation, not a detail: an NLE-shaped editor is mostly effects.

video/v2 adds ten effect-parameter features (transition fraction, duration, alignment and asymmetry; grade deviation; LUT; reframe; duck depth; segment gain; overlay fraction) under a new spec id — never an edit of v1, because the layout list is part of an artifact's digest, so a v1 identity keeps loading and keeps refusing a v2 trunk instead of being silently reinterpreted with shifted columns. A log may hold both; the majority layout trains and the rest are counted (groups_other_spec), since mixing vector layouts would either crash the first forward pass or mis-align columns.

Two halves are needed, and both exist: a group that spans effects, so a preference can be learned — propose_schema(… effect_grid=true) pairs the selection grid with effect knobs (the default grid varies selection only, so every candidate in a legacy group agrees on every effect feature and nothing could ever teach an effect preference) — and an identity that can choose them: propose_schema(… refine=N) runs a deterministic coordinate ascent over the effect axes (transition type/duration/alignment, ducking, grade strength) scored by the identity, and the tuned schema joins the group as one more candidate. Two properties are deliberate:

• It needs an effect-aware identity. A v1 trunk has 16 inputs and cannot score a 26-dim vector, so the request is declined with a reason rather than silently ignored — and the feature spec a group is logged under is explicit (--feature-spec), because a group logged under v1 can never teach an effect.
• No identity means no effect taste. refine_effects with no scorer returns the schema untouched: with no identity there is no taste to apply, and maximising the objective critic there would quietly turn a rubric rule into a taste claim.

Measured (tests/test_taste_model.py::TestEffectTaste): groups are logged under v2 with a reward that rises only with the transition duration — a hidden preference expressed solely through an effect parameter — an identity is trained on it (train_rank_acc ≥ 0.8), and the refiner then chooses a dissolve and pushes the duration past 0.8 s from an all-cuts seed. The control run with no identity changes nothing. TestEffectFeatures pins the blindness of v1, the superset relation and the mixed-spec rule.

They are now rendered, and the rest are refused. cmd_render executes transitions — xfade between runs, acrossfade for the audio, and a