Oriel — AI & technology worth noticing, every day.

BIBIMBAP: A Benchmark for Instructional BIM-Based Automated Programming

Paper · https://arxiv.org/abs/2611.09933

arXiv:2611.09933v1 Announce Type: new Abstract: Large Language Models (LLMs) are emerging as natural-language interfaces for Building Information Modeling (BIM), successfully achieving information retrieval, but no benchmark exists to evaluate their ability to edit Industry Foundation Classes (IFC). We introduce BIBIMBAP, a Text-to-BIM benchmark with 100 curated tasks covering Create-Read-Update-Delete (CRUD) operations across spatial, geometric, topological, numeric, and conceptual categories. Each task includes a natural-language prompt, an IFC model, expected structured outputs, and test scripts for automated evaluation. Baseline results show limited performance (best: 50.2%) and frequent failures in constraint preservation and spatial reasoning, with some correct outputs achieved despite flawed reasoning.

First seen by Oriel on .

AI summary

BIBIMBAP introduces a 100-task benchmark for evaluating whether LLMs can edit IFC models from natural-language instructions, with baseline performance reaching 50.2%.

  • Tasks cover Create-Read-Update-Delete operations across spatial, geometric, topological, numeric, and conceptual categories.
  • Each task provides a natural-language prompt, an IFC model, expected structured outputs, and automated test scripts.
  • Baseline systems often fail to preserve constraints and reason about spatial relationships.
  • Some outputs are correct despite flawed reasoning, highlighting a limitation of output-only evaluation.

#bim #ifc #llm benchmarks #automated programming

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.09933v1https://arxiv.org/abs/2611.09933 Read discussion