Standardized AI video benchmarks
Every model runs the exact same frozen Cloudflare R2 inputs (5s, 480p, Pass@3 max draws) and is graded 100% by deterministic scripts — zero subjective eye-balling. Failed draws stay in the bill so Effective $/s reflects what a usable second actually costs in production.
OCR character accuracy, optical-flow foreground motion, 2D limb vector correlation and background SSIM are enforced at build time.
Total USD billed across up to 3 draws divided by usable output seconds. 0/3 usable draws = DNF.
Products built by our team never enter ranked tables. Maker-supplied API keys run the identical frozen suite and carry a disclosure tag.
Benchmark suites
- Character replacement & motion-transfer video benchmark: pose lock, scale & effective $/s (v1)motion-swap-v1 · 3 frozen cases · 6 runs
Standardized single-draw (n=1) 5-second character-replacement benchmark comparing Kling 3.0 Motion Control, Kling 2.6 Motion Control, ByteDance DreamActor V2, Wan 2.2 Animate 14B, Higgsfield Genjutsu Lite and each::labs Lockstep on 2D limb-vector correlation, compact mascot scale preservation and background SSIM stability.
Current Effective $/s leader: Kling Motion Control ($0.070/s, 100% pass rate)
- Image-to-video benchmark: classic community stress tests, OCR legibility & effective $/s (v1)product-i2v-v1 · 3 frozen cases · 10 runs
Standardized single-draw (n=1) 5-second image-to-video benchmark across three classic AI-video stress tests (curved metallic soda can splash orbit, rainy neon corner diner push-in, and desert highway vintage rally car lateral tracking). Graded 100% by deterministic scripts (Apple Vision OCR with language autocomplete disabled + dense Farnebäck optical flow + Laplacian sharpness retention), pricing every model uniformly against public fal.ai retail per-second rates.
Current Effective $/s leader: Vidu Q4 ($0.047/s, 67% pass rate)
Want your video model or workflow blind-tested on a frozen suite? Send test API keys or credits to support@fourthfeed.com. All runs — wins and failures alike — are published with raw output clips.
This page as data: .json · .md · /api/v1/benchmarks