Multimedia Production Mastery

Multimedia Production Mastery

Automation recipes, mathematical formulations, and Python/FFmpeg templates for edits that are driven by a signal or a curve: audio beats, easing functions, and time-remap profiles. The unit of work here is a computed timeline, not a manual cut.

When to Use

Use this skill when one or more of the following are true:

• Beat synchronization — cutting or transitioning clips to align with the tempo, beats, or onsets of an audio track (librosa analysis feeding MoviePy/FFmpeg cuts).
• Speed ramping — smooth, non-linear acceleration/deceleration profiles (fast → slow-mo → fast) via time remapping.
• Zoom & pan curves — Ken Burns motion using sinusoidal or exponential ease-in/ease-out instead of robotic linear interpolation.
• Camera shake — procedural handheld or bass-impact shake generated from noise or onset strength.
• Styled dynamic subtitles — .ass files with word-by-word highlight colors, custom fonts, borders, and margins.
• Pipeline orchestration — wiring librosa + MoviePy 2.x + FFmpeg around pre-rendered clips (including AI-generated ones).

When NOT to use — routing table

Prerequisites

Python environment

All bundled scripts require the following packages in the active Python environment:

Verify before running any script:

python -c "import librosa, moviepy, numpy, scipy; print(librosa.__version__, moviepy.__version__)"

Expected output should show moviepy version starting with 2..

FFmpeg

FFmpeg must be on PATH. Verify:

ffmpeg -version

Bundled scripts location

All scripts live in scripts/ relative to this skill directory:

Bundled references

Load references/effects-landscape.md when the user asks about available AI-enhanced or classical video effects, their parameters, tooling, or underlying math — it is a catalog you can search for effect names and implementation details.

Procedure

Step 1 — Identify the driving signal

Determine what drives the edit:

Step 2 — Run or compose the matching recipe

Recipe A: Beat-synced cut from a folder of clips

python scripts/beat_sync.py --audio music.mp3 --dir .\raw_clips --output synced.mp4

Export detected beats for inspection or reuse:

python scripts/beat_sync.py --audio music.mp3 --dir .\raw_clips --output synced.mp4 --export-beats beats.json

For sparse or syncopated tracks where beat tracking misses, use onset detection:

python scripts/beat_sync.py --audio ambient.mp3 --dir .\raw_clips --output synced.mp4 --onset

Recipe B: Three-phase speed ramp (fast → slow-mo → fast)

python scripts/speed_ramp.py --video in.mp4 --output ramped.mp4 --mid-speed 0.15

Full parameter control:

python scripts/speed_ramp.py `
  --video in.mp4 `
  --output ramped.mp4 `
  --start-speed 2.0 `
  --mid-speed 0.2 `
  --end-speed 2.0 `
  --ramp-start 0.2 `
  --ramp-end 0.8

Recipe C: Word-highlight styled subtitles

Input JSON format (word-level timings, highlight flags the accent color):

[
  { "start": 0.5, "end": 1.2, "text": "Welcome", "highlight": false },
  { "start": 1.8, "end": 3.0, "text": "Future!", "highlight": true }
]

Generate the .ass file:

python scripts/subtitles_styled.py `
  --input words.json `
  --output subs.ass `
  --highlight-color "#00FFFF"

Step 3 — Core formulas (reference when composing custom code)

Non-linear motion curves

HARD RULE: Never linearly interpolate zoom or pan — it reads as robotic. Use sinusoidal ease-in-out:

z(t) = z_start + (z_end - z_start) * (1 - cos(pi * t / d)) / 2

where t is current time, d the ease duration. To keep pixel (x_f, y_f) centered on a W x H frame while zooming by z(t):

x(t) = x_f - W / (2 * z(t))
y(t) = y_f - H / (2 * z(t))

Time remapping (speed ramps)

Define S(t) = source time at output time t; the derivative S'(t) is the instantaneous speed multiplier. Build S(t) by integrating a speed profile.

• MoviePy 2.x: clip.time_transform(time_func) — the 1.x fl_time is gone. HARD RULE: time_transform() leaves the clip's duration undefined. Call .with_duration(target) immediately after, or output is truncated.
• FFmpeg: dynamic speed requires splitting into segments and applying setpts (video) + atempo (audio) per segment. HARD RULE: atempo only accepts [0.5, 2.0]; chain filters for anything outside (e.g. atempo=2.0,atempo=2.0 for 4x, atempo=2.0,atempo=1.5 for 3x).

ASS color formats

Two different hex conventions inside one file:

HARD RULE: Alpha is inverted relative to RGBA: 00 = fully opaque, FF = fully transparent. Invisible text usually means FF (transparent) where 00 (opaque) was intended.

Step 4 — FFmpeg recipes

Recipe 1: Jitter-free FFmpeg zoompan (Ken Burns)

zoompan evaluates positions in integer pixels, causing visible jitter at low zoom speeds. Pre-upscale to 4K first so the truncation error becomes sub-pixel:

ffmpeg -i in.jpg -vf "scale=3840:-2,zoompan=z='1+0.5*(1-cos(PI*on/149))/2':x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':d=150:fps=30:s=1920x1080" -c:v libx264 -pix_fmt yuv420p out.mp4

Recipe 2: The even-dimensions rule

HARD RULE: Most encoders (yuv420p especially) fail on odd width/height. After any crop or scale, force even dimensions:

ffmpeg -i input.mp4 -vf "scale='trunc(iw/2)*2:trunc(ih/2)*2'" output.mp4

Recipe 3: MoviePy 2.x non-linear speed ramp (custom code)

Integrate a speed profile to get the time-remap function, then apply it:

import numpy as np
from scipy.integrate import cumulative_trapezoid
from moviepy import VideoFileClip

src = VideoFileClip("in.mp4")
fps = src.fps
out_dur = 6.0
t_out = np.linspace(0, out_dur, int(out_dur * fps) + 1)

# Speed profile: fast -> slow -> fast
speed = 1.0 + 0.5 * (np.cos(2 * np.pi * t_out / out_dur) * 0.5 + 0.5)

# Integrate speed to get source timestamps for each output frame
s_src = np.clip(cumulative_trapezoid(speed, t_out, initial=0.0), 0, src.duration - 1e-3)
time_func = lambda t: float(np.interp(t, t_out, s_src))

ramped = src.time_transform(time_func).with_duration(out_dur)
ramped.write_videofile("out.mp4", fps=fps, audio=False)

src.close()
ramped.close()

Recipe 4: Proxy workflow for heavy footage

Validate cuts and beat alignment on a cheap proxy, render once on the original:

1. Generate proxy:

   ffmpeg -i input.mp4 -vf scale=-2:720 -c:v libx264 -crf 20 -c:a copy proxy.mp4
   

2. Iterate timing/beat scripts against proxy.mp4.
3. Swap the source path back to the 4K original for the final render only.

Step 5 — Onset fallback for difficult tracks

Standard beat tracking misses beats in sparse or heavily syncopated material. Fall back to onsets in custom code:

onset_frames = librosa.onset.onset_detect(y=y, sr=sr, backtrack=True)
times = librosa.frames_to_time(onset_frames, sr=sr)

(beat_sync.py --onset does this automatically.)

Pitfalls

1. MoviePy duration reset — time_transform() leaves the clip's duration undefined. Call .with_duration(target) immediately after, or output is truncated.
2. ASS inverted alpha — invisible text usually means FF (transparent) where 00 (opaque) was intended. Alpha is inverted relative to RGBA.
3. atempo range — multipliers outside [0.5, 2.0] silently fail unless chained (atempo=2.0,atempo=1.5 for 3x, atempo=2.0,atempo=2.0 for 4x).
4. librosa tempo type — beat_track() returns a NumPy array for tempo in librosa 0.11+; cast with float() before formatting.
5. Beat tracking on ambient/vocal tracks — standard beat tracking misses beats in sparse or heavily syncopated material. Use --onset flag or librosa.onset.onset_detect with backtrack=True.
6. Windows file locks — unclosed VideoFileClip/AudioFileClip handles keep files locked and crash subsequent runs. HARD RULE: Always .close() every clip before the script exits, including in error paths.
7. Odd output dimensions — yuv420p and most encoders fail on odd width/height. Always force even dimensions after any crop or scale operation.
8. MoviePy 1.x vs 2.x API — only use 2.x methods: time_transform / with_duration / subclipped / with_audio. The 1.x fl_time / set_duration / subclip methods do not exist in 2.x and will raise AttributeError.
9. zoompan integer jitter — zoompan evaluates positions in integer pixels. Without pre-upscaling to 4K, low zoom speeds produce visible jitter. Always pre-scale to a high resolution before zoompan.

Verification

Run through this checklist before considering the task complete:

• Environment check: librosa, moviepy (2.x), numpy, and scipy import successfully:

  python -c "import librosa, moviepy, numpy, scipy; print(librosa.__version__, moviepy.__version__)"
  

  Output must show moviepy version starting with 2..

• MoviePy 2.x API only: Only these methods are used: time_transform / with_duration / subclipped / with_audio. No 1.x fl_time / set_duration / subclip appears anywhere in the code.

• Even dimensions: Every scale/crop expression forces even output dimensions. Verify with:

  ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=p=0 output.mp4
  

  Both width and height must be even numbers.

• ASS color formats: Inline ASS tags use 6-digit hex with trailing & (\c&HBBGGRR&); style lines use 8-digit &HAABBGGRR; alpha semantics are inverted (00 = opaque, FF = transparent).

• Clip handles closed: All VideoFileClip/AudioFileClip handles are .close()d before the script exits. On Windows, verify no file locks remain by attempting to delete or rename the input file after the script finishes.

• atempo within range: Any atempo filter values are within [0.5, 2.0] or properly chained. Verify by inspecting the FFmpeg filtergraph string.

• Output plays: Quick sanity check:

  ffprobe -v error -show_entries format=duration -of csv=p=0 output.mp4
  

  Duration should match the expected target duration (not the source duration for ramped clips).

Related skills

• ffmpeg-video-editing — plain cut/trim/concat with no timing math
• ffmpeg-video-filters — constant-rate scale, crop, watermark, blur, uniform speed
• video-assembly-pipeline — stitching finished clips into a deliverable
• ffmpeg-audio-processing — audio extraction, normalisation, mixing
• capcut-video-automation — CapCut Desktop draft manipulation
• ffmpeg-format-conversion — container/format conversion with no timeline changes