BIBIMBAP Benchmarks LLM Editing of IFC Models, Best Score Reaches Just 50.2%
Summary:BIBIMBAP evaluates language-driven Building Information Modeling as an executable Text-to-BIM benchmark. Its 100 curated tasks cover create, read, update, and delete operations across spatial, geometric, topological, numeric, and conceptual changes. Every task includes a natural-language instruction, an IFC model, structured expected outputs, and automated test scripts. The strongest baseline reaches only 50.2%, with constraint preservation and spatial reasoning emerging as recurring failure mod
BIBIMBAP Turns IFC Editing into an Executable Evaluation
Large language models can retrieve information from Building Information Modeling systems through natural language, but retrieval is not equivalent to correct model editing. BIBIMBAP introduces a dedicated benchmark for modifying Industry Foundation Classes (IFC). Given a natural-language instruction and an existing IFC model, a model must perform create, read, update, or delete operations and return a model that preserves valid engineering semantics and constraints.
The benchmark contains 100 curated tasks spanning spatial, geometric, topological, numeric, and conceptual changes. Each task provides a natural-language prompt, an input IFC file, expected structured outputs, and test scripts for automated evaluation. This is more rigorous than subjective comparison because the evaluator can directly inspect entities, properties, geometry, and relationships.
The Text-to-BIM Technical Chain
IFC is not a flat table or a simple JSON document. Building elements are defined through object relationships, spatial containment, geometric representations, and property sets. A model may correctly create a wall, door, window, or floor and still break the file by producing incorrect coordinates, missing parent-child relationships, unsynchronized properties, or inconsistent geometry.
A typical Text-to-BIM pipeline has four stages:
- Parse the target object, operation, and constraints from the instruction.
- Locate the relevant entities and their upstream and downstream dependencies in the IFC graph.
- Generate the IFC change while preserving geometric, spatial, and property consistency.
- Validate the result with rules and automated test scripts.
The third stage is the main difficulty. Moving a window is not merely changing coordinate values; it may also affect openings, wall relationships, spatial boundaries, and property representations. Deleting a wall requires deciding whether floors, connections, or spatial divisions still reference it. The model must track dependencies rather than emit a locally plausible code fragment.
The Best Baseline Reaches Only 50.2%
The strongest reported baseline scores 50.2%, indicating that current LLMs do not reliably master IFC editing. Failures concentrate on constraint preservation and spatial reasoning. A model may generate syntactically valid IFC while omitting related entities that must remain synchronized, or correctly perform an explicitly described action while misunderstanding spatial relationships.
These failures are particularly risky in production. BIM files support clash detection, quantity calculation, facilities management, and regulatory review. An apparently correct local modification can introduce hidden engineering errors. In incremental modeling workflows, a bad edit may be saved and become part of subsequent model history, allowing errors to accumulate.
More importantly, some outputs are correct despite flawed reasoning. A model may accidentally match the expected result, skip relevant validation, or apply an implementation that does not follow the instruction's semantics. Scoring only final test results would overstate genuine BIM understanding and obscure failure attribution.
Interpreting the Results
Automated testing improves scalability, but validity depends on test coverage. IFC permits multiple legitimate representations, and the same semantic target can be serialized through different object structures. A test overly bound to one serialization may reject equivalent answers. A test that checks only a few attributes may, however, allow geometric or topological violations through.
A more rigorous evaluation should distinguish at least three layers: whether the file can be parsed, whether the edit matches the requested semantics, and whether existing constraints remain valid. Reasoning traces or tool-call sequences should also be recorded to identify accidental successes. Reporting pass rates by operation type is more informative than a single aggregate score.
The 100-task suite is useful for exposing capability gaps, but it remains too small to represent complex parametric modeling, template propagation, and multidisciplinary coordination in production projects.
Deployment Recommendations
Teams integrating BIM agents should prefer constrained edit plans or tool calls over asking a model to emit an entire IFC file. The execution layer should parse the current model, generate a transactional change plan, and validate it with geometry, relationship, and property checkers. Any failed operation should roll back to the pre-edit state.
Practically, teams can build a small internal regression set covering their common element types, property templates, and modeling conventions. Model selection should consider constraint-preservation rates on updates, spatial-task accuracy, recovery behavior, and token and tool-call cost rather than aggregate success alone.
For long-lived BIM projects, auditability is equally important. Every edit should preserve the original instruction, affected entities, dependency information, validation results, and model version so that errors can be traced to a specific reasoning or execution stage. BIBIMBAP shows that LLMs remain far from reliable manipulation of complex BIM models, while providing a concrete foundation for measuring that gap.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-10)
Tags: #LLM Evaluation #Text-to-BIM #IFC #Building Information Modeling #Spatial Reasoning
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments