Media Upscale

Media Upscale

Context: $ARGUMENTS

Quick start

• Photo 2x/4x → upscaled PNG: Real-ESRGAN → Step 2 recipe A.
• Anime / cel-shaded art: Real-CUGAN → Step 2 recipe B.
• Restore a face in a blurry photo: GFPGAN → Step 2 recipe E.
• Upscale a whole video: Real-ESRGAN frame pipeline → Step 3.
• Batch a folder of images: scripts/upscale.py batch → Step 4.

When to use

• User says "upscale", "super-res", "enhance", "4K-ify", "make it sharper", "AI upscale".
• Source is lower resolution than the target medium (SD → HD, HD → 4K, 480p anime → 1080p).
• ffmpeg's dnn_processing (SRCNN/ESPCN/EDSR) or a plain scale=...:flags=lanczos looks soft / blurry on real content.
• Face-only restoration of a scanned photo or low-res portrait.
• Do NOT use this skill for mere resizing without quality gain — use ffmpeg-video-filter (scale) instead.
• Do NOT use for temporal frame interpolation — that is media-interpolate.

Step 1 — Pick a model

Decision rules:

1. Real photos / video frames → Real-ESRGAN (realesr-animevideov3 for cartoons, realesrgan-x4plus for photos, realesrgan-x4plus-anime for anime stills).
2. Anime or illustration → Real-CUGAN (sharper cel edges than Real-ESRGAN), fall back to waifu2x-ncnn-vulkan for very small sources.
3. Faces are the focal point (portrait, ID card, old family photo) → GFPGAN (then blend with a Real-ESRGAN-upscaled background).
4. Need academic SOTA quality, don't mind Python/CUDA → HAT or SwinIR.
5. Multi-model pipeline (segment → upscale → denoise → sharpen chain) → chaiNNer.

Do NOT use: CodeFormer (S-Lab License, research-only — not commercial-safe), Topaz Video AI (commercial, paid). This skill covers open-source + commercial-safe only.

Step 2 — Install the chosen binary / package

scripts/upscale.py install <model> prints the exact package-manager command for your platform. It does not auto-install. Examples:

# Real-ESRGAN (ncnn-vulkan prebuilt — fastest install, zero Python)
# macOS:
brew install realesrgan-ncnn-vulkan          # if tap available, else download release
# or grab a release zip from: https://github.com/xinntao/Real-ESRGAN/releases

# Real-CUGAN (ncnn-vulkan)
# Download: https://github.com/nihui/realcugan-ncnn-vulkan/releases

# waifu2x-ncnn-vulkan
brew install waifu2x                           # macOS; tap: nihui/waifu2x-ncnn-vulkan

# GFPGAN (Python)
uv pip install gfpgan basicsr realesrgan facexlib

# SwinIR / HAT (Python + CUDA)
uv pip install torch opencv-python numpy basicsr
git clone https://github.com/JingyunLiang/SwinIR      # weights via model zoo
git clone https://github.com/XPixelGroup/HAT         # weights via releases page

# chaiNNer
# Download from: https://chainner.app/

Recipe A — Real-ESRGAN 4x on a photo

realesrgan-ncnn-vulkan \
  -i photo.jpg -o photo_4x.png \
  -n realesrgan-x4plus -s 4 -f png
# Available -n models: realesrgan-x4plus, realesrnet-x4plus, realesr-animevideov3,
#                      realesrgan-x4plus-anime
# -s scale: 2, 3, 4
# -t tile: 0 auto, or e.g. 256 to limit VRAM. Set -g <GPU_ID> for multi-GPU.
# -f format: png | jpg | webp

Through the driver script:

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py image \
  --model realesr --scale 4 --in photo.jpg --out photo_4x.png

Recipe B — Real-CUGAN on anime

realcugan-ncnn-vulkan \
  -i anime.png -o anime_4x.png \
  -n 0 -s 4 -f png
# -n 0 default, -n 3 for heavy denoise, -n -1 conservative; -s 2|3|4.

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py image \
  --model realcugan --scale 4 --in anime.png --out anime_4x.png

Recipe C — SwinIR (Python)

python SwinIR/main_test_swinir.py \
  --task real_sr --scale 4 --large_model \
  --model_path SwinIR/model_zoo/swinir/003_realSR_BSRGAN_DFO_s64w8_SwinIR-L_x4_GAN.pth \
  --folder_lq /path/to/input_dir --tile 400

Recipe D — HAT (Python)

python HAT/hat/test.py -opt HAT/options/test/HAT_SRx4.yml

Recipe E — GFPGAN face restoration + Real-ESRGAN background

python GFPGAN/inference_gfpgan.py \
  -i inputs/ -o results/ \
  -v 1.4 -s 2 --bg_upsampler realesrgan
# -v model version: 1.2, 1.3, 1.4; -s output scale factor.
# --bg_upsampler realesrgan re-uses Real-ESRGAN for non-face pixels.

Driver:

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py face-restore \
  --model gfpgan --in portrait.jpg --out portrait_restored.png --scale 2

Step 3 — Upscale a video (frame pipeline)

AI SR models are frame-based. Video workflow:

1. Extract frames with ffmpeg (lossless PNGs).
2. Upscale each frame with Real-ESRGAN / Real-CUGAN.
3. Remux the upscaled frame sequence with the original audio.

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py video \
  --model realesr --scale 4 --in clip.mp4 --out clip_4x.mp4 \
  --model-name realesr-animevideov3

What the script does:

# (1) frames out
ffmpeg -i clip.mp4 -qscale:v 1 -qmin 1 -qmax 1 -vsync 0 frames/%08d.png
# (2) upscale frames
realesrgan-ncnn-vulkan -i frames -o frames_4x -n realesr-animevideov3 -s 4 -f png
# (3) remux with original audio, pick an FPS matching the source
ffmpeg -framerate <src_fps> -i frames_4x/%08d.png -i clip.mp4 \
       -map 0:v -map 1:a? -c:v libx264 -preset slow -crf 17 -pix_fmt yuv420p \
       -c:a copy -movflags +faststart clip_4x.mp4

VRAM control: add -t 256 (tile size) to the Real-ESRGAN step if GPU VRAM is tight; 0 = auto.

Native video model choice: realesr-animevideov3 is tuned for video and runs faster than the still-image models on frame sequences.

Step 4 — Batch a folder

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py batch \
  --model realesr --scale 4 --in-dir photos/ --out-dir photos_4x/

Real-ESRGAN ncnn-vulkan natively accepts -i <dir> -o <dir> and processes every file in parallel. The wrapper adds logging, dry-run, and skips already-processed outputs.

Gotchas

• ffmpeg built-in SR is outdated. dnn_processing=model=espcn.pb / EDSR / SRCNN are 2014–2017 research models. On real content they are not competitive with Real-ESRGAN (2021+). Recommend ffmpeg built-ins only as a tiny fallback.
• -n model name, not path. realesrgan-ncnn-vulkan -n realesrgan-x4plus expects the model file to live under the binary's models/ directory (shipped in the release zip). If you get failed to load model, pass -m <dir> to point at your models.
• Scale factor must match the chosen model. realesrgan-x4plus is 4x-only. Don't pass -s 2 to a 4x model and expect 2x output — you'll get a downsample of 4x output.
• yuv420p even dimensions. After a 3x upscale of an odd-width video, libx264 will complain. Add -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" on the remux.
• VRAM. Real-ESRGAN on a 4K frame wants ~6GB at 4x. Use -t 256 (tile size) to process 256×256 tiles and stitch — slower but fits in 2GB.
• Face restoration bakes in features that weren't in the original. GFPGAN will hallucinate plausible detail; don't use it on ID photos you need legally faithful. For cinema/commercial work, only use it when the client has signed off on "AI reconstruction."
• CodeFormer is NOT in this skill. CodeFormer is S-Lab License research-only. Do not recommend it even though many tutorials online still do.
• Topaz Video AI is NOT in this skill. It is commercial paid software; outside the open-source + commercial-safe scope.
• RMBG v1.4 is NOT in this skill. Different project (background removal) — see media-matte. Also: v1.4 was NC; only v2.0 (Apache 2.0) is commercial-safe.
• Upscayl is AGPL-3.0. If you ship code derived from Upscayl's source in a commercial closed product you must release source. Running the bundled app to produce outputs is fine.
• chaiNNer is GPL-3.0. Same caveat as Upscayl: running it to produce outputs is fine, linking its code into a closed product is not.
• Video frame remux needs the source FPS exactly. Pull it from ffprobe first — don't assume 23.976 / 25 / 29.97.
• 8-bit → 10-bit matters. Upscaling an 8-bit source doesn't add dynamic range. If the source is 10-bit HDR, keep the frame dumps in a 16-bit PNG pipeline or use EXR to preserve the bit depth.

Examples

Example 1: 4x an old family photo

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py image \
  --model realesr --scale 4 --in grandma_1970.jpg --out grandma_4x.png

Example 2: Upscale a 480p anime clip to 1080p

# 480 * 2.25 ≈ 1080 → use Real-CUGAN 2x, then pad to 1080p if needed.
uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py video \
  --model realcugan --scale 2 --in ep01_480p.mp4 --out ep01_960p.mp4
ffmpeg -i ep01_960p.mp4 -vf "scale=-2:1080:flags=lanczos" -c:v libx264 -crf 18 ep01_1080p.mp4

Example 3: Restore a face in a scan, keep background

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py face-restore \
  --model gfpgan --in scan.jpg --out scan_restored.png --scale 2

Example 4: Batch 4x a folder of 1000 thumbnails

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py batch \
  --model realesr --scale 4 --in-dir thumbs/ --out-dir thumbs_4x/

Example 5: Check which upscale binaries are on PATH

uv run ${CLAUDE_SKILL_DIR}/scripts/upscale.py check

Troubleshooting

Error: realesrgan-ncnn-vulkan: command not found

Cause: binary not installed. Solution: Download from https://github.com/xinntao/Real-ESRGAN/releases and unzip. Or brew install realesrgan-ncnn-vulkan. Or run uv run .../upscale.py install realesr.

Error: failed to load model: realesrgan-x4plus

Cause: model .param / .bin files not next to the binary (expected at <binary>/models/). Solution: The release zip bundles them — always use the full zip. Or pass -m /path/to/models.

Output is blocky / has tile seams

Cause: -t tile size too small relative to content. Solution: Increase tile (-t 512 or -t 0 for auto). Or run without tiling on a GPU with more VRAM.

HAT / SwinIR inference: CUDA out of memory

Cause: tile size too large. Solution: Reduce --tile 400 → --tile 200. HAT's test.py reads tile from the YML.

Upscaled video stutters / is desynced

Cause: reassembled at wrong FPS, or VFR source. Solution: ffprobe -select_streams v -show_entries stream=r_frame_rate,avg_frame_rate clip.mp4 — pass that exact rational to -framerate. For VFR sources, first normalize with ffmpeg -vsync cfr.

GFPGAN "RuntimeError: CUDA error" on CPU machine

Cause: GFPGAN needs a GPU by default. Solution: python inference_gfpgan.py --device cpu ... (slow but works).

Reference docs

• Per-model URLs, input res ranges, GPU/CPU support, download commands → references/models.md.
• Full license table for every model bundled → references/LICENSES.md.