WorldFoundry Opens a Unified Infrastructure for Reproducible World-Model Evaluation

Summary:WorldFoundry is an open-source infrastructure for reproducible inference and evaluation across video, 3D/4D, embodied-action, and interactive world-model systems. It consolidates model and benchmark discovery, runtime and asset preparation, execution orchestration, artifact management, and visualization without flattening modality-specific input, output, and evaluation semantics. Its main contribution is evaluation governance rather than a new world-model architecture. Explicit manifests, durab

---

An Infrastructure for World Models, Not Another World Model

WorldFoundry places video generation, 3D/4D environments, embodied actions, and interactive world systems within a shared discovery, execution, and evaluation workflow. Its focus is not a new generative architecture. Instead, it addresses a practical research problem: under what conditions did a model run, what evidence did it produce, and can a third party reproduce the result?

The framework applies a stricter definition to generative world models. A valid system must maintain history-conditioned persistent state, update that state in response to actions or interventions, and preserve temporal, causal, and dynamical consistency across observed and counterfactual trajectories. Visual plausibility is therefore only an entry requirement rather than evidence of world intelligence.

How the Unified Pipeline Handles Heterogeneous Systems

WorldFoundry separates the workflow into several composable stages:

1. Discovery catalogs models, assets, benchmarks, and metrics.
2. Preparation builds runtime environments and prepares input assets through explicit manifests.
3. Execution orchestrates inference, simulation, and interaction tasks.
4. Artifact management persists outputs, trajectories, configurations, logs, and scores.
5. Visualization supports inspection of videos, 3D/4D scenes, and action feedback.
6. Evaluation links scorecards to the artifacts that support them.

The central implementation choice is the use of modality-specific adapters. These preserve each system's native inputs, execution semantics, output representations, and evaluation protocols. WorldFoundry standardizes contracts and orchestration rather than forcing video generators, 3D generators, and robot policies into a single tensor interface or metric.

That approach lowers adapter costs but introduces its own risks. If an adapter changes observation timing, action semantics, or scoring behavior, nominally unified cross-modality comparisons may still be invalid.

Evidence Chains Matter More Than Aggregate Scores

The infrastructure explicitly separates execution readiness from evaluative claims. Manifests record dependencies, model versions, datasets, entry points, and output types. Artifact contracts determine which files, trajectories, logs, and configurations must survive a run. Evidence-aware scorecards then connect reported results to the underlying artifacts.

For world models, reproducibility depends on parameters that are often omitted from leaderboards: random seeds, initial states, action sequences, observation timestamps, physics parameters, solver settings, model versions, and numerical precision. Two runs can receive the same perceptual-quality score while remaining scientifically incomparable.

The catalog currently contains 270 model entries, 74 benchmarks, and 62 registered metrics, with readiness and execution evidence recorded separately. Registration count should not be treated as capability coverage. Many entries may only be discoverable or pending execution, while cross-modality tasks may differ in difficulty, sampling budget, and data splits. Credible comparisons therefore require disclosed execution coverage, failure rates, and metric versions.

Engineering Trade-offs

A unified workflow improves reproducibility and artifact reuse, but it also introduces concrete costs:

• Dependency isolation: Video diffusion, 3D asset pipelines, and robot simulators often require incompatible CUDA, driver, Python, and system-library versions. A single image or node risks conflicts and scheduler fragmentation. Backend-specific environments connected through standard task contracts are likely more practical.
• Storage growth: Long videos, point clouds, 4D volumes, and per-step trajectories can consume large amounts of object storage. Durable artifacts need content addressing, deduplication, tiered storage, and lifecycle policies.
• Scheduling latency: Interactive world-model tasks combine model inference, environment stepping, and policy feedback. They should not share the same scheduling assumptions as offline batch generation.
• Metadata scalability: Model discovery, version indexing, and score aggregation eventually require transactional metadata and idempotent task execution to avoid repeated initialization and inconsistent records.
• Fairness: Equal assets do not imply equal compute budgets. Parameter count, sampling steps, resolution, action frequency, and simulator step size must be reported alongside quality.

Moving Beyond Visual Plausibility

The framework covers perceptual grounding, persistent state, controllable transformation, dynamics-aware prediction, and closed-loop interaction. These dimensions can be operationalized through separate evaluation axes:

• Persistence: Does the system preserve internal state across interruptions, occlusion, and long-horizon rollouts?
• Controllability: Do interventions produce semantically consistent changes under both observed and unobserved conditions?
• Physical consistency: Do collisions, velocities, contacts, and constraints obey the intended dynamics?
• Temporal consistency: Are object identity, occlusion relations, and scene topology maintained over time?
• Closed-loop utility: Can the generated environment support planning or control over multi-step interactions?

No single metric covers all five properties. Geometric or CLIP-based similarity can measure visual proximity but cannot establish dynamic interactivity. Video-quality benchmarks cannot replace state-retention or counterfactual tests. Retaining native evaluation protocols is sensible, but reports should also disclose metric definitions, random seeds, sample sizes, confidence intervals, and failure cases.

Practical Adoption Path

Teams should begin with one modality and establish a minimal closed loop: pin model and dataset versions, define task manifests and artifact directories, execute in a clean environment, and ensure every score maps to a concrete video, trajectory, configuration, and log. Only after that workflow is stable should additional modalities be added to test whether the contracts remain reusable.

Evaluation plans should preregister primary metrics and compute budgets to reduce cherry-picking. Interactive systems should report end-to-end latency, throughput, peak memory, environment-stepping cost, and policy success rate. Generative systems should add temporal consistency, action controllability, and counterfactual response tests rather than relying on visual quality alone.

Production deployments also need sandboxing and resource quotas. Model adapters, third-party assets, and interactive backends should receive only the file-system, network, and device permissions they require. Artifact signing, dependency locking, and immutable version identifiers are essential if reproducibility is to survive beyond the original maintainers' environment.

Value and Limits

WorldFoundry's main contribution is to move world-model research from isolated demonstrations toward auditable experimental infrastructure. It does not require every model to share an architecture; instead, it connects models, benchmarks, runtimes, and evidence through durable contracts, making persistent and controllable dynamics measurable across heterogeneous systems.

Infrastructure alone does not establish scientific validity. Catalog size is not model capability, and standardized adapters do not automatically guarantee comparable metrics. Strong evidence would require independent reruns under shared assets, budgets, and protocols, including failure cases, resource costs, and complete artifacts. Only then can the gap between visual synthesis and world intelligence become a measurable engineering problem rather than a conceptual one.

---

Source: Hugging Face Daily Papers (2026-11-07)

— END —

Tags: #World Models Follow #Generative AI Follow #Reproducible Evaluation Follow #Multimodal Infrastructure Follow #Evidence-Aware Scorecards Follow