标签：#videogeneration

On the AI video ad platform I work on, every scene goes through the same painful loop: write a prompt, send it to an AI video model provider, wait two minutes, open the result, squint at the frame, and decide what went wrong. Camera too wide. Product missing from the hero shot. Color palette drifted warm when the brand brief says cool neutrals. Avatar looks like a different person than scene three. That loop was manual, slow, and expensive. Each regeneration burns GPU credits. Operators were becoming prompt engineers by accident — and still missing subtle failures until stitch time, when fixing scene four means re-rendering everything downstream. The insight behind vision-in-the-loop prompt authoring is simple: the model that wrote the prompt can also look at its own output and rewrite the prompt with surgical fixes. Not a full replan — a per-scene correction grounded in the actual generated frame, not the operator's memory of what they hoped would appear. The manual loop we were trying to kill Before this work shipped, the swipe iteration flow looked like this: Plan — Claude generates a scene-by-scene script with visual prompts Generate — each scene renders independently through an AI video model provider Review — operator opens the portal, compares frames to the reference ad Rewrite — operator edits prompts in a text field, often guessing at what the model misread Regenerate — repeat until acceptable or budget exhausted Steps three and four are where throughput dies. An experienced operator can spot "product not visible" in three seconds, but translating that into prompt language — "medium close-up, product centered in lower third, shallow depth of field" — takes another minute per scene. Multiply by twelve scenes and three swipe iterations, and a single ad creative consumes an hour of human attention that should be spent on brand strategy, not frame inspection. The generated frame is ground truth. The original prompt is a hypothesis. Vision-in-the-loop closes the

TL;DR In generative video pipelines, running cheap low-step sketches to pick parameters sounds like free optimization. But when prompts go out-of-distribution, surrogate scorers return noise, turning a \$0.002 check into a bad decision that triggers a \$15 compounding failure. Here's why skipping the cheap step is sometimes the cheapest option. Three numbers run Scenematic's generation loop. A think-frame costs \$0.002. A full render costs \$0.50. A bad scene that slips through and gets built on costs about \$15.50, because the scene chain compounds it before anyone looks. The constant in lib/generation-loop.ts carries the arithmetic in a comment: 15.502, // CALIBRATION_TARGET: 0.002 + 0.50 + 15.00 . Most of the pipeline exists to keep spend at the cheap end of that ladder. One module decides when the cheap step should be skipped entirely. A hundred-contract baseline then put numbers on how often that decision was wrong. 1. The rehearsal lib/think-frames.ts generates quick, low-inference-step sketches before committing to a full-quality keyframe. The file header credits DeepGen's think tokens as the inspiration. Each sketch tries a different preservation focus, character, environment, mood, composition, or atmosphere, with its own image-to-image strength and seed. The reward mixer scores the batch and the winner's parameters go to the full render. The economics only work if those scores mean something. That assumption fails quietly, and it fails hardest on the prompts where a rehearsal looks most useful. 2. Where the scores stop meaning anything Scoring a sketch of A detective leans forward across a metal table, interrogating a nervous suspect under fluorescent lights works fine. The scorer has seen a thousand shots like it. Scoring A sentient equation writes itself across a blackboard that extends infinitely in all dimensions does not fail loudly. It returns a number, and the number is noise. Both prompts are verbatim from the baseline harness. lib/ood-detector.ts

Intro Day 20! I lined up 10 AIs that turn a single photo into a few seconds of video. Half ran locally on my DGX Spark, half in the cloud 🐱 What I used: DGX Spark (LTX-2.3 / Wan 2.2) / 8 cloud models via fal.ai / ComfyUI / ffmpeg The setup Item Value Input One identical photo (my cat on a desk) Length 6 seconds Settings Identical The only variable The prompt Easy prompt The cat looks at the camera and meows once. It opens its mouth, meows, then closes it. Its tail flicks and its ears twitch. Hard prompt The cat stands upright on its hind legs in a kitchen, wearing a small apron, holding a knife in its front paws and chopping vegetables on a cutting board. Steam rises from a pot behind it. Please, just watch it Some of the cats came out with very long legs. Anyway. First half is the easy prompt, second half the hard one. On the easy prompt, local and cloud were a fair match . On the hard one... cloud, I think...! Three rankings below. Ranking 1: Time Time per 6-second clip on the hard prompt. Rank Model Where Time 🥇 LTX-2.3 Cloud 41s 🥈 Wan 2.7 Cloud 92s 🥉 Happy Horse 1.1 Cloud 97s 4 Veo 3.1 Cloud 128s 5 Kling 3 Pro Cloud 205s 6 Seedance 2.0 Cloud 210s 7 LTX-2.3 Local 315s 8 Wan 2.2 Local 651s 9 daVinci-MagiHuman Cloud 710s 10 HunyuanVideo 1.5 Cloud 796s A 19x spread. Look at 1st and 7th. Same model, LTX-2.3 , nearly the same resolution. The only difference is where it ran — 7.6x . Local setup DGX Spark (GB10, 128GB unified memory, ~273GB/s). ComfyUI headless, workflows over its API. LTX-2.3 is distilled fp8 at 8 steps. At 1088×1920 peak memory hit 77.8GB, about 60% of 128GB. That was the ceiling. Dropping to 512×768 finishes in 70s, but with one-fifth the pixels. Wan 2.2 is I2V-A14B fp8, 20 steps, 480×640. Higher resolution does not finish in reasonable time. Ranking 2: Cost Rank Model Per 6 seconds 🥇 Local Electricity only 🥈 LTX-2.3 (cloud) $0.36 🥉 Wan 2.7 $0.90 4 Kling 3 Pro $1.01 5 Happy Horse 1.1 $1.08 6 Veo 3.1 $2.40 7 Seedance 2.0 $4.09 — HunyuanVideo / MagiHum

The Problem Nobody Talks About Everyone's posting AI-generated videos — characters speaking with lip-sync, manga panels coming alive, virtual idols dancing. The pitch: "just describe what you want." I tried. For months. Here's what I got: Character's face morphed by frame 2 "Slowly looks up" became "violent head shake" Voice-over sounded like Google Translate Same prompt, 3 runs, 3 completely different results No idea what to include or how long the prompt should be Tutorials were either too vague ("be detailed") or too technical (parameter tuning from line 1). The real issue: video prompts are structurally different from text/image prompts. You need to simultaneously control visuals, motion, audio, camera, and consistency constraints — in the right order, at the right length. What I Found A Skill in the Model Studio official repo called happyhorse-prompt-studio . It doesn't teach you theory — it asks you questions and assembles the prompt for you . 4-phase flow: 1. Inspiration Menu Shows you 4 "flavors" of what HappyHorse can do: Flavor What it does A · Voiced Manga Drama Characters talk to each other, with voice + lip-sync B · Character Voice PV Single character self-introduction, 8-10 sec C · Manga Panel Motion Static manga panel starts breathing D · Virtual Idol MV Idol performance with choreography 2. Discovery Asks you conversationally: character appearance, scene, emotion, dialogue, voice type, art style, camera. 3. Prompt Assembly Assembles using the HappyHorse Formula : Scene + Subject + Motion + Audio + Quality Key techniques: @「Image n」 syntax locks character identity across shots Dialogue ≤15 characters (split shots if longer) Japanese prompts work best (HappyHorse is JP-optimized) Always end with キャラの顔・髪・衣装が変わらない (face/hair/outfit stays unchanged) 4. Quality Check Auto-reviews: completeness, compliance, cost estimate, optimization tips. Before vs. After Dimension Writing myself With Prompt Studio Attempts needed 10-20 before one usable 2-3 to satisfacti

Video AI systems consistently fail to track what happens when the camera looks away: when a scene pans away from an object in motion and returns, current models re-render the object in its original position rather than showing the logical result of off-screen change. Scaling to more parameters makes this failure worse, not better, according to WRBench , a new benchmark that tests what researchers call "world model reliability." The benchmark presents AI video systems with scenes where something happens off-screen — the camera pans away while an object is in motion, or while a light changes, or while an open door should stay open — then pans back to see what the system believes should have happened. A system that genuinely models the world would track what occurred during the off-screen interval. Current systems mostly don't. Key facts What: A new benchmark tests whether video AI systems can track what happens to parts of a scene the camera isn't currently showing. Across 23 models, the answer is mostly no — and making the models larger made the problem worse, not better. When: 2026-06-19 Primary source: read the source (arXiv 2606.20545) The benchmark covers twenty-three different video generation models and nearly ten thousand video clips across six categories of off-screen change, each designed to test a different aspect of world continuity: objects in motion, light sources changing, object states such as open or closed doors, and several others. This gives a comprehensive picture rather than a single narrow test. The most striking finding is the scaling result. The researchers tested one of the more capable video generation systems at two different sizes: a smaller version and one with more than ten times as many parameters. More parameters didn't help. Scaling made the off-screen tracking problem measurably worse. The larger model produced more realistic-looking frames, but it was less accurate about what should have happened to the parts of the scene it wasn't