Character replacement & motion-transfer video benchmark: pose lock, scale & effective $/s (v1)
Suite motion-swap-v1 (v1, frozen 2026-11-16) · Pass@1 · Grader: scripts/benchmark-grade.py
Standardized single-draw (n=1) 5-second character-replacement benchmark comparing Kling 3.0 Motion Control, Kling 2.6 Motion Control, ByteDance DreamActor V2, Wan 2.2 Animate 14B, Higgsfield Genjutsu Lite and each::labs Lockstep on 2D limb-vector correlation, compact mascot scale preservation and background SSIM stability.
1. Leaderboard
| # | Model / workflow | Cases passed | Pass rate | Effective $/s | Nominal $/s | Median time | 2D limb-angle correlation | Subject scale preservation | Background SSIM stability | Delivered duration ratio | Credits |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Kling Motion Control 2.6-motion-720p Passed 3/3 character-swap tests (99.3% mean 2D pose correlation, 89.9% compact mascot scale accuracy, 184s median render) at $0.070/s effective usable cost. | 3/3 | 100% (3/3) | $0.070/s | $0.070/s | 184s | 99.3% | 87.9% | 87.4% | 100.0% | Self-funded |
| 2 | Kling 3.0 Motion Control 3.0-motion-720p Passed 3/3 character-swap tests (99.0% mean 2D pose correlation, 89.7% compact mascot scale accuracy, 269s median render) at $0.125/s effective usable cost. | 3/3 | 100% (3/3) | $0.125/s | $0.126/s | 269s | 99.0% | 86.9% | 87.4% | 100.0% | Self-funded |
| 3 | Wan 2.2 Animate 14B Replace (Open-Weight) 2.2-14b-replace-480p-open Passed 2/3 character-swap tests (90.3% mean 2D pose correlation, 39.1% compact mascot scale accuracy, 65s median render) at $0.059/s effective usable cost. | 2/3 | 67% (2/3) | $0.059/s 1.5x retry cost | $0.040/s | 65s | 90.3% | 70.7% | 97.8% | 100.0% | Self-funded |
| 4 | ByteDance DreamActor V2 v2-motion-transfer Passed 2/3 character-swap tests (86.0% mean 2D pose correlation, 83.7% compact mascot scale accuracy, 143s median render) at $0.062/s effective usable cost. | 2/3 | 67% (2/3) | $0.062/s 1.2x retry cost | $0.050/s | 143s | 86.0% | 79.7% | 87.8% | 100.0% | Self-funded |
| 5 | Higgsfield Genjutsu Lite Motion Transfer lite-480p Passed 2/3 character-swap tests (97.9% mean 2D pose correlation, 79.0% compact mascot scale accuracy, 230s median render) at $0.063/s effective usable cost. | 2/3 | 67% (2/3) | $0.063/s 1.3x retry cost | $0.050/s | 230s | 97.9% | 84.1% | 82.0% | 100.0% | Self-funded |
| 6 | each::labs Lockstep lockstep-480p Passed 2/3 character-swap tests (98.5% mean 2D pose correlation, 77.9% compact mascot scale accuracy, 187s median render) at $0.070/s effective usable cost. | 2/3 | 67% (2/3) | $0.070/s 1.5x retry cost | $0.047/s | 187s | 98.5% | 82.6% | 84.2% | 100.0% | Self-funded |
Kling Motion Control (2.6-motion-720p) — per-case scores & raw clip evidence (tested 2026-11-16)
human-dance-lock-5s: 1/1 usable · 5.03s @ 720p · billed $0.350 · pose_angle_corr=99.9%, scale_ratio_accuracy=92.8%, bg_ssim=92.9%, duration_ratio=100.0%small-mascot-scale-5s: 1/1 usable · 5.03s @ 720p · billed $0.350 · pose_angle_corr=98.6%, scale_ratio_accuracy=89.9%, bg_ssim=86.5%, duration_ratio=100.0%foreground-occlusion-5s: 1/1 usable · 5.03s @ 720p · billed $0.350 · pose_angle_corr=99.6%, scale_ratio_accuracy=81.0%, bg_ssim=83.0%, duration_ratio=100.0%
Kling 3.0 Motion Control (3.0-motion-720p) — per-case scores & raw clip evidence (tested 2026-11-16)
human-dance-lock-5s: 1/1 usable · 5.03s @ 720p · billed $0.630 · pose_angle_corr=99.9%, scale_ratio_accuracy=90.8%, bg_ssim=92.9%, duration_ratio=100.0%small-mascot-scale-5s: 1/1 usable · 5.03s @ 720p · billed $0.630 · pose_angle_corr=97.5%, scale_ratio_accuracy=89.7%, bg_ssim=86.4%, duration_ratio=100.0%foreground-occlusion-5s: 1/1 usable · 5.03s @ 720p · billed $0.630 · pose_angle_corr=99.6%, scale_ratio_accuracy=80.1%, bg_ssim=82.9%, duration_ratio=100.0%
Wan 2.2 Animate 14B Replace (Open-Weight) (2.2-14b-replace-480p-open) — per-case scores & raw clip evidence (tested 2026-11-16)
human-dance-lock-5s: 1/1 usable · 5.08s @ 480p · billed $0.200 · pose_angle_corr=99.9%, scale_ratio_accuracy=94.7%, bg_ssim=98.9%, duration_ratio=100.0%small-mascot-scale-5s: 0/1 usable · 5.08s @ 480p · billed $0.200 · pose_angle_corr=73.1%, scale_ratio_accuracy=39.1%, bg_ssim=98.9%, duration_ratio=100.0% · failure: 2D limb-angle correlation 73.1% < 78% gate; Subject scale distorted (39.1% < 75% gate; stretched toward driving skeleton)foreground-occlusion-5s: 1/1 usable · 5.08s @ 480p · billed $0.200 · pose_angle_corr=97.9%, scale_ratio_accuracy=78.1%, bg_ssim=95.7%, duration_ratio=100.0%
ByteDance DreamActor V2 (v2-motion-transfer) — per-case scores & raw clip evidence (tested 2026-11-16)
human-dance-lock-5s: 1/1 usable · 6.08s @ 720p · billed $0.250 · pose_angle_corr=82.3%, scale_ratio_accuracy=89.8%, bg_ssim=93.3%, duration_ratio=100.0%small-mascot-scale-5s: 1/1 usable · 6.08s @ 720p · billed $0.250 · pose_angle_corr=80.0%, scale_ratio_accuracy=83.7%, bg_ssim=86.4%, duration_ratio=100.0%foreground-occlusion-5s: 0/1 usable · 6.08s @ 720p · billed $0.250 · pose_angle_corr=95.7%, scale_ratio_accuracy=65.5%, bg_ssim=83.7%, duration_ratio=100.0% · failure: Subject scale distorted (65.5% < 75% gate; stretched toward driving skeleton)
Higgsfield Genjutsu Lite Motion Transfer (lite-480p) — per-case scores & raw clip evidence (tested 2026-11-16)
human-dance-lock-5s: 1/1 usable · 6s @ 480p · billed $0.250 · pose_angle_corr=99.5%, scale_ratio_accuracy=97.2%, bg_ssim=89.0%, duration_ratio=100.0%small-mascot-scale-5s: 1/1 usable · 6s @ 480p · billed $0.250 · pose_angle_corr=96.0%, scale_ratio_accuracy=79.0%, bg_ssim=92.2%, duration_ratio=100.0%foreground-occlusion-5s: 0/1 usable · 6s @ 480p · billed $0.250 · pose_angle_corr=98.3%, scale_ratio_accuracy=76.1%, bg_ssim=64.9%, duration_ratio=100.0% · failure: Background drift/redraw: SSIM 64.9% < 80% gate
each::labs Lockstep (lockstep-480p) — per-case scores & raw clip evidence (tested 2026-11-16)
human-dance-lock-5s: 1/1 usable · 5.08s @ 480p · billed $0.237 · pose_angle_corr=99.6%, scale_ratio_accuracy=94.6%, bg_ssim=91.3%, duration_ratio=100.0%small-mascot-scale-5s: 1/1 usable · 5.08s @ 480p · billed $0.237 · pose_angle_corr=96.6%, scale_ratio_accuracy=77.9%, bg_ssim=92.6%, duration_ratio=100.0%foreground-occlusion-5s: 0/1 usable · 5.08s @ 480p · billed $0.237 · pose_angle_corr=99.3%, scale_ratio_accuracy=75.3%, bg_ssim=68.8%, duration_ratio=100.0% · failure: Background drift/redraw: SSIM 68.8% < 80% gate
2. Script pass gates & metrics
- 2D limb-angle correlation
pose_angle_corrUnit: ratio · Higher is better · Required gate: gte 0.78
- Subject scale preservation
scale_ratio_accuracyUnit: ratio · Higher is better · Required gate: gte 0.75
- Background SSIM stability
bg_ssimUnit: ratio · Higher is better · Required gate: gte 0.8
- Delivered duration ratio
duration_ratioUnit: ratio · Higher is better · Required gate: gte 0.92
3. Frozen test cases (3)
Prompt: “Replace the dancer in the video with the character in the black FOURTHFEED crewneck sweatshirt from the reference image, keeping the exact arm crosses, footwork, camera framing and studio background.”
Pass rule: pose_angle_corr >= 0.78, scale_ratio_accuracy >= 0.75, bg_ssim >= 0.80, duration_ratio >= 0.92
- Frozen image: ff-char-dancer-v2.png ↗
sha256:0433a98e0ef3…— Proprietary full-body street dancer in FOURTHFEED pullover crewneck with both hands visible - Frozen video: ff-driving-dance-5s-v1.mp4 ↗
sha256:a8014d1696be…— 5s 480p 24fps proprietary full-body dance driving clip
Prompt: “Replace the performer in the video with the compact 3D fawn pug mascot in the black FOURTHFEED crewneck sweatshirt from the reference image while preserving its compact ~0.65x pug proportions and keeping the studio background intact.”
Pass rule: pose_angle_corr >= 0.78, scale_ratio_accuracy >= 0.75, bg_ssim >= 0.80, duration_ratio >= 0.92
- Frozen image: ff-char-mascot-v2.png ↗
sha256:6dac5089a725…— Proprietary compact 3D pug mascot in FOURTHFEED crewneck; tests whether pipelines stretch small mascots to adult human height - Frozen video: ff-driving-gesture-5s-v1.mp4 ↗
sha256:027b3edddb9c…— 5s 480p 24fps proprietary full-body gesture driving clip
Prompt: “Replace the subject in the video with the courier in the amber FOURTHFEED pullover anorak from the reference image, maintaining clean face and chest identity when the hand passes across the face and during the 90-degree turn.”
Pass rule: pose_angle_corr >= 0.78, scale_ratio_accuracy >= 0.75, bg_ssim >= 0.80, duration_ratio >= 0.92
- Frozen image: ff-char-courier-v2.png ↗
sha256:2308d4cd327f…— Proprietary full-body courier in FOURTHFEED amber pullover anorak - Frozen video: ff-driving-occlusion-5s-v1.mp4 ↗
sha256:eecf9d36bd8b…— 5s 480p 24fps proprietary hand-across-face + head-turn driving clip
This suite as data: .json · .md · /api/v1/benchmarks/motion-swap-v1