BIBIMBAP: A Benchmark for Instructional BIM-Based Automated Programming
Paper · https://arxiv.org/abs/2611.09933
arXiv:2611.09933v1 Announce Type: new Abstract: Large Language Models (LLMs) are emerging as natural-language interfaces for Building Information Modeling (BIM), successfully achieving information retrieval, but no benchmark exists to evaluate their ability to edit Industry Foundation Classes (IFC). We introduce BIBIMBAP, a Text-to-BIM benchmark with 100 curated tasks covering Create-Read-Update-Delete (CRUD) operations across spatial, geometric, topological, numeric, and conceptual categories. Each task includes a natural-language prompt, an IFC model, expected structured outputs, and test scripts for automated evaluation. Baseline results show limited performance (best: 50.2%) and frequent failures in constraint preservation and spatial reasoning, with some correct outputs achieved despite flawed reasoning.
First seen by Oriel on .
AI summary
BIBIMBAP introduces a 100-task benchmark for evaluating whether LLMs can edit IFC models from natural-language instructions, with baseline performance reaching 50.2%.
- Tasks cover Create-Read-Update-Delete operations across spatial, geometric, topological, numeric, and conceptual categories.
- Each task provides a natural-language prompt, an IFC model, expected structured outputs, and automated test scripts.
- Baseline systems often fail to preserve constraints and reason about spatial relationships.
- Some outputs are correct despite flawed reasoning, highlighting a limitation of output-only evaluation.
#bim #ifc #llm benchmarks #automated programming
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.09933v1 | https://arxiv.org/abs/2611.09933 Read discussion |