Benchmarks

Character replacement & motion-transfer video benchmark: pose lock, scale & effective $/s (v1)

Suite motion-swap-v1 (v1, frozen 2026-11-16) · Pass@1 · Grader: scripts/benchmark-grade.py

Standardized single-draw (n=1) 5-second character-replacement benchmark comparing Kling 3.0 Motion Control, Kling 2.6 Motion Control, ByteDance DreamActor V2, Wan 2.2 Animate 14B, Higgsfield Genjutsu Lite and each::labs Lockstep on 2D limb-vector correlation, compact mascot scale preservation and background SSIM stability.

1. Leaderboard

Sort by:
#Model / workflowCases passedPass rateEffective $/sNominal $/sMedian time2D limb-angle correlationSubject scale preservationBackground SSIM stabilityDelivered duration ratioCredits
1
Kling Motion Control 2.6-motion-720p

Passed 3/3 character-swap tests (99.3% mean 2D pose correlation, 89.9% compact mascot scale accuracy, 184s median render) at $0.070/s effective usable cost.

3/3100% (3/3)$0.070/s$0.070/s184s99.3%87.9%87.4%100.0%Self-funded
2
Kling 3.0 Motion Control 3.0-motion-720p

Passed 3/3 character-swap tests (99.0% mean 2D pose correlation, 89.7% compact mascot scale accuracy, 269s median render) at $0.125/s effective usable cost.

3/3100% (3/3)$0.125/s$0.126/s269s99.0%86.9%87.4%100.0%Self-funded
3
Wan 2.2 Animate 14B Replace (Open-Weight) 2.2-14b-replace-480p-open

Passed 2/3 character-swap tests (90.3% mean 2D pose correlation, 39.1% compact mascot scale accuracy, 65s median render) at $0.059/s effective usable cost.

2/367% (2/3)$0.059/s
1.5x retry cost
$0.040/s65s90.3%70.7%97.8%100.0%Self-funded
4
ByteDance DreamActor V2 v2-motion-transfer

Passed 2/3 character-swap tests (86.0% mean 2D pose correlation, 83.7% compact mascot scale accuracy, 143s median render) at $0.062/s effective usable cost.

2/367% (2/3)$0.062/s
1.2x retry cost
$0.050/s143s86.0%79.7%87.8%100.0%Self-funded
5
Higgsfield Genjutsu Lite Motion Transfer lite-480p

Passed 2/3 character-swap tests (97.9% mean 2D pose correlation, 79.0% compact mascot scale accuracy, 230s median render) at $0.063/s effective usable cost.

2/367% (2/3)$0.063/s
1.3x retry cost
$0.050/s230s97.9%84.1%82.0%100.0%Self-funded
6
each::labs Lockstep lockstep-480p

Passed 2/3 character-swap tests (98.5% mean 2D pose correlation, 77.9% compact mascot scale accuracy, 187s median render) at $0.070/s effective usable cost.

2/367% (2/3)$0.070/s
1.5x retry cost
$0.047/s187s98.5%82.6%84.2%100.0%Self-funded
Kling Motion Control (2.6-motion-720p) — per-case scores & raw clip evidence (tested 2026-11-16)
  • human-dance-lock-5s: 1/1 usable · 5.03s @ 720p · billed $0.350 · pose_angle_corr=99.9%, scale_ratio_accuracy=92.8%, bg_ssim=92.9%, duration_ratio=100.0%
  • small-mascot-scale-5s: 1/1 usable · 5.03s @ 720p · billed $0.350 · pose_angle_corr=98.6%, scale_ratio_accuracy=89.9%, bg_ssim=86.5%, duration_ratio=100.0%
  • foreground-occlusion-5s: 1/1 usable · 5.03s @ 720p · billed $0.350 · pose_angle_corr=99.6%, scale_ratio_accuracy=81.0%, bg_ssim=83.0%, duration_ratio=100.0%
Kling 3.0 Motion Control (3.0-motion-720p) — per-case scores & raw clip evidence (tested 2026-11-16)
  • human-dance-lock-5s: 1/1 usable · 5.03s @ 720p · billed $0.630 · pose_angle_corr=99.9%, scale_ratio_accuracy=90.8%, bg_ssim=92.9%, duration_ratio=100.0%
  • small-mascot-scale-5s: 1/1 usable · 5.03s @ 720p · billed $0.630 · pose_angle_corr=97.5%, scale_ratio_accuracy=89.7%, bg_ssim=86.4%, duration_ratio=100.0%
  • foreground-occlusion-5s: 1/1 usable · 5.03s @ 720p · billed $0.630 · pose_angle_corr=99.6%, scale_ratio_accuracy=80.1%, bg_ssim=82.9%, duration_ratio=100.0%
Wan 2.2 Animate 14B Replace (Open-Weight) (2.2-14b-replace-480p-open) — per-case scores & raw clip evidence (tested 2026-11-16)
  • human-dance-lock-5s: 1/1 usable · 5.08s @ 480p · billed $0.200 · pose_angle_corr=99.9%, scale_ratio_accuracy=94.7%, bg_ssim=98.9%, duration_ratio=100.0%
  • small-mascot-scale-5s: 0/1 usable · 5.08s @ 480p · billed $0.200 · pose_angle_corr=73.1%, scale_ratio_accuracy=39.1%, bg_ssim=98.9%, duration_ratio=100.0% · failure: 2D limb-angle correlation 73.1% < 78% gate; Subject scale distorted (39.1% < 75% gate; stretched toward driving skeleton)
  • foreground-occlusion-5s: 1/1 usable · 5.08s @ 480p · billed $0.200 · pose_angle_corr=97.9%, scale_ratio_accuracy=78.1%, bg_ssim=95.7%, duration_ratio=100.0%
ByteDance DreamActor V2 (v2-motion-transfer) — per-case scores & raw clip evidence (tested 2026-11-16)
  • human-dance-lock-5s: 1/1 usable · 6.08s @ 720p · billed $0.250 · pose_angle_corr=82.3%, scale_ratio_accuracy=89.8%, bg_ssim=93.3%, duration_ratio=100.0%
  • small-mascot-scale-5s: 1/1 usable · 6.08s @ 720p · billed $0.250 · pose_angle_corr=80.0%, scale_ratio_accuracy=83.7%, bg_ssim=86.4%, duration_ratio=100.0%
  • foreground-occlusion-5s: 0/1 usable · 6.08s @ 720p · billed $0.250 · pose_angle_corr=95.7%, scale_ratio_accuracy=65.5%, bg_ssim=83.7%, duration_ratio=100.0% · failure: Subject scale distorted (65.5% < 75% gate; stretched toward driving skeleton)
Higgsfield Genjutsu Lite Motion Transfer (lite-480p) — per-case scores & raw clip evidence (tested 2026-11-16)
  • human-dance-lock-5s: 1/1 usable · 6s @ 480p · billed $0.250 · pose_angle_corr=99.5%, scale_ratio_accuracy=97.2%, bg_ssim=89.0%, duration_ratio=100.0%
  • small-mascot-scale-5s: 1/1 usable · 6s @ 480p · billed $0.250 · pose_angle_corr=96.0%, scale_ratio_accuracy=79.0%, bg_ssim=92.2%, duration_ratio=100.0%
  • foreground-occlusion-5s: 0/1 usable · 6s @ 480p · billed $0.250 · pose_angle_corr=98.3%, scale_ratio_accuracy=76.1%, bg_ssim=64.9%, duration_ratio=100.0% · failure: Background drift/redraw: SSIM 64.9% < 80% gate
each::labs Lockstep (lockstep-480p) — per-case scores & raw clip evidence (tested 2026-11-16)
  • human-dance-lock-5s: 1/1 usable · 5.08s @ 480p · billed $0.237 · pose_angle_corr=99.6%, scale_ratio_accuracy=94.6%, bg_ssim=91.3%, duration_ratio=100.0%
  • small-mascot-scale-5s: 1/1 usable · 5.08s @ 480p · billed $0.237 · pose_angle_corr=96.6%, scale_ratio_accuracy=77.9%, bg_ssim=92.6%, duration_ratio=100.0%
  • foreground-occlusion-5s: 0/1 usable · 5.08s @ 480p · billed $0.237 · pose_angle_corr=99.3%, scale_ratio_accuracy=75.3%, bg_ssim=68.8%, duration_ratio=100.0% · failure: Background drift/redraw: SSIM 68.8% < 80% gate

2. Script pass gates & metrics

  • 2D limb-angle correlationpose_angle_corr

    Unit: ratio · Higher is better · Required gate: gte 0.78

  • Subject scale preservationscale_ratio_accuracy

    Unit: ratio · Higher is better · Required gate: gte 0.75

  • Background SSIM stabilitybg_ssim

    Unit: ratio · Higher is better · Required gate: gte 0.8

  • Delivered duration ratioduration_ratio

    Unit: ratio · Higher is better · Required gate: gte 0.92

3. Frozen test cases (3)

Full-body fast arm-cross dance transfer (street dancer)human-dance-lock-5s · 5s @ 480p

Prompt: “Replace the dancer in the video with the character in the black FOURTHFEED crewneck sweatshirt from the reference image, keeping the exact arm crosses, footwork, camera framing and studio background.”

Pass rule: pose_angle_corr >= 0.78, scale_ratio_accuracy >= 0.75, bg_ssim >= 0.80, duration_ratio >= 0.92

  • Frozen image: ff-char-dancer-v2.png ↗ sha256:0433a98e0ef3… — Proprietary full-body street dancer in FOURTHFEED pullover crewneck with both hands visible
  • Frozen video: ff-driving-dance-5s-v1.mp4 ↗ sha256:a8014d1696be… — 5s 480p 24fps proprietary full-body dance driving clip
Cross-scale compact 3D pug mascot swap without adult-height stretchingsmall-mascot-scale-5s · 5s @ 480p

Prompt: “Replace the performer in the video with the compact 3D fawn pug mascot in the black FOURTHFEED crewneck sweatshirt from the reference image while preserving its compact ~0.65x pug proportions and keeping the studio background intact.”

Pass rule: pose_angle_corr >= 0.78, scale_ratio_accuracy >= 0.75, bg_ssim >= 0.80, duration_ratio >= 0.92

  • Frozen image: ff-char-mascot-v2.png ↗ sha256:6dac5089a725… — Proprietary compact 3D pug mascot in FOURTHFEED crewneck; tests whether pipelines stretch small mascots to adult human height
  • Frozen video: ff-driving-gesture-5s-v1.mp4 ↗ sha256:027b3edddb9c… — 5s 480p 24fps proprietary full-body gesture driving clip
Hand-across-face occlusion & rapid head-turn recovery (techwear courier)foreground-occlusion-5s · 5s @ 480p

Prompt: “Replace the subject in the video with the courier in the amber FOURTHFEED pullover anorak from the reference image, maintaining clean face and chest identity when the hand passes across the face and during the 90-degree turn.”

Pass rule: pose_angle_corr >= 0.78, scale_ratio_accuracy >= 0.75, bg_ssim >= 0.80, duration_ratio >= 0.92

This suite as data: .json · .md · /api/v1/benchmarks/motion-swap-v1