MotionGraphicsBench Verifies Motion Graphics Generation Programmatically
Summary:Figma AI has released MotionGraphicsBench, a 100-instance benchmark for evaluating whether large language models can animate static SVGs from natural-language motion-edit prompts. The benchmark isolates motion generation from asset creation by supplying all required visual elements. Unlike reference-video matching or VLM-as-judge evaluation, each instance uses a deterministic, human-readable verification program that accepts multiple valid implementations and checks individual prompt requiremen
Core release: programmatic verification instead of visual resemblance
Figma AI has released MotionGraphicsBench, a 100-instance benchmark for testing whether large language models can animate static SVGs from natural-language motion-edit prompts. The setup resembles a designer's workflow: all required visual assets are supplied, isolating motion generation from graphic creation.
The benchmark's central contribution is its deterministic verification layer. Each instance includes a human-readable program that checks spatial, appearance, and temporal constraints while accepting multiple valid implementations. A model can use a different trajectory, easing pattern, or animation structure and still pass if the requested behavior is preserved. This avoids brittle pixel-level reference matching and the subjectivity of VLM judging.
How the verifier works
Every instance combines a static SVG, a motion-edit prompt, and a verification program. The program decomposes the request into testable constraints: whether an element reaches a specified coordinate, whether a required pose appears, whether rotation and timing are correct, and whether multiple animations remain synchronized.
The programs were constructed with a multi-agent pipeline using human-labeled animations and synthesized test cases. They provide three practical advantages:
- Reproducibility: identical inputs produce identical requirement-level judgments, making them suitable for regression testing.
- Implementation flexibility: trajectories need not match a single reference animation.
- Actionable errors: the report identifies whether pose, trajectory, or timing constraints failed rather than returning only an aggregate score.
On a balanced set of 180 animations, the verification programs achieve 93.3% raw agreement with human judgments, compared with 83.3% for the GPT-6 Astra VLM baseline. They also outperform that baseline in both approval and rejection recall, indicating stronger discrimination on invalid outputs. However, 93.3% still leaves a 6.7% disagreement rate. The abstract does not provide confidence intervals, category-level error analysis, or correlations among judge outputs, so this should not be interpreted as a definitive audit-accuracy estimate for production use.
Model results: more attempts help, but do not eliminate structural failures
The study evaluates 47 configurations spanning 32 LLMs from 12 providers. Pass@1 ranges from 6.0% to 67.7%, a more than tenfold spread. This suggests that one-shot generation cannot yet provide a consistently reliable motion-design agent.
The task requires models to generate executable animation code, reason over a timeline, preserve geometric constraints, and coordinate multiple animated objects. Each of these capabilities can fail independently, making a simple visual score insufficient.
Fifteen models are also evaluated at their highest available reasoning effort with 32 attempts per instance. Every model improves with additional attempts, but pass@32 still ranges from 38.0% to 95.0%. Two implications follow:
- Sampling and deeper reasoning increase the probability of finding a valid implementation, but do not remove systematic errors.
- Evaluations must report pass@1 alongside pass@k. A best-of-32 result can hide substantial latency, inference cost, and variability in production.
Those costs remain unspecified. Every candidate may require SVG parsing, rendering, geometric evaluation, and program execution. If a production agent averages dozens of attempts per request, end-to-end latency and cost per accepted animation can overwhelm the apparent quality gain. Token consumption, render time, verification latency, and cost per passing instance should therefore be added to future benchmark reports.
Common failure modes
The verification programs reveal three recurring weaknesses:
- Pose mismatch: an element moves but fails to reach a requested pose or coordinate;
- Trajectory noncompliance: direction, path, or easing behavior differs from the instruction;
- Poor event coordination: individual animations may be acceptable, while their ordering, delays, and synchronization are wrong.
The last category is particularly important. Code-generation models often satisfy local constraints independently while failing to maintain dependencies across a shared timeline. Professional motion design depends precisely on such cross-object and cross-time coordination, which a single-frame VLM review may overlook.
Deployment recommendations
MotionGraphicsBench is best used as a pre-release acceptance suite rather than a complete production-agent framework. It supports three immediate uses:
- Regression testing: detect broken constraints after changing prompts, model versions, or code-generation pipelines.
- Capability profiling: separate spatial movement, appearance, keyframing, and synchronization results rather than relying on one aggregate score.
- Model routing: use lightweight generation for simple motion requests and reserve stronger models or iterative agents for trajectories and multi-event coordination.
Teams should supplement the public set with proprietary assets and business rules, preserve a human-labeled holdout, and report pass@1, best-of-k pass rates, average generation count, P95 latency, and cost per accepted output. Pinning model, renderer, and verifier versions is also essential for meaningful longitudinal comparisons.
Evaluation boundaries
The benchmark contains 100 expert-curated instances. That is sufficient to validate the evaluation methodology but too small to represent every animation workflow. Its Lottie-derived, SVG-based format is closely aligned with programmatic motion editing, not video generation, 3D animation, or unrestricted creative design. That isolation improves diagnostic clarity but limits extrapolation to open-ended design work.
Programmatic verification also evaluates only requirements encoded in the verifier. It may miss occlusion behavior, visual rhythm, brand rules, or unconventional but valid motion. A robust production pipeline should combine deterministic assertions, human sampling, and VLM review rather than treating them as interchangeable judges.
The work's main contribution is to move motion-generation evaluation from vague visual similarity toward executable and diagnosable constraint checking. It exposes the remaining weakness of current models in precise motion control and provides a more reliable interface for regression testing design agents.
— END —Source: Hugging Face Daily Papers (2026-11-07)
Tags: #MotionGraphicsBench #LLM Evaluation #Programmatic Verification #SVG Animation #Multi-Agent Systems
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments