An Interpreter Layer for AEC: Cross-Modal Grounding & Ontology-Driven Retrieval

VideoFull Paper Walkthrough

ProblemThe physical–digital traceability gap

AEC projects standardise on OpenBIM/IFC, yet day-to-day site coordination still runs on manual interpretive labour. Subcontractors log unstructured on-site evidence — photos, chat messages, annotated floorplan patches — into forms; construction managers re-key it into the system. The front-line worker sees a local condition but not the BIM GUID; the coordinator sees the model but not the site context; the developer in the office receives it late and degraded, and plans against it anyway.

Bridging that gap by hand leaks in two ways: semantic drift as detail is re-described and re-typed, and context decay as notes go stale and people roll off the project. It is worst in high-rise work, where dozens of windows and walls are visually identical — a photo of "a window" could be any of them, so even a faithful description never pins down which one.

In a BIM-native workflow every issue should resolve to a GUID linked back to the model, so downstream teams — issue tracking, compliance, planning — inherit it cleanly. Ungrounded records break that chain: not query-able, not traceable, and the decisions built on them arrive late and drift from site reality.

Before any system can reason about what changed or what comes next, it must answer the question that unlocks the rest — where, exactly, is this? That is spatial grounding: a hallucination-resistant link from site evidence to the exact model element. This work introduces an interpreter middleware to close that gap — streamlining data across many stakeholders and formats by making on-site evidence traceable.

ApproachNeuro-symbolic Architecture Interpreter

Key result: the representation, not a bigger model, separates identical elements

ContributionsWhat this work delivers

ArchitectureThe neuro-symbolic interpreter

Architecture in five stages: multimodal input → neural extraction (VLM + LoRA) → Constraints JSON → symbolic retrieval (priority Cypher cascade over a Neo4j IFC graph) → calibrated soft-rerank (optional Graph-RAG/LLM reorder) → ranked GUIDs.

The neural side translates evidence into a typed spatial address; the symbolic side names only GUIDs that exist in the model. Design rationale: reliability over flexibility (deterministic templates instead of Text-to-Cypher), and an IFC-native graph built from the schema itself rather than noisy document-level extraction.

1. Multimodal input — a site photo, a short field note, and a marked floorplan crop.
2. Neural perception (VLM) — a fine-tuned Qwen2.5-VL + LoRA emits a typed Constraints JSON {storey, ifc_class, spatial_relations[{predicate, object_type, direction}]}, each field a {value, confidence, source} record. It describes; it never names a GUID.
3. Symbolic retrieval — the constraints compile into a deterministic Cypher cascade over the enriched Neo4j IFC graph; a recall-safe p0∪p1 union returns a candidate pool with the target retained ~100% of the time. The relational predicates build the pool, not the within-pool ranking.
4. Deterministic slot discriminator + soft rerank — an OpenCV specialist reads the opening position-slot (i of M) from the plan; candidates are reordered by a soft score (storey + class + confidence-weighted slot match) that never removes a candidate.
5. Calibrated answer / defer — a temperature-scaled confidence commits the top GUID when it clears the threshold, and otherwise defers the candidate pool to a human reviewer.

A fuller engineered stack — a ResNet size-band filter and a Gemini Graph-RAG LLM reranker — was evaluated and not adopted: on the topology-filtered pool the LLM reorder is redundant and lowers Top-1 (6.7% → 1.7%) while breaking the 100% recall guarantee. The deployed ranking is deterministic end-to-end apart from the VLM perception front end.

Key ModulesHow it works under the hood

Engineering decisions, measured against alternatives

Each contract has an experiment behind it: what to let the VLM do, what to push into deterministic geometry, when to trust an extracted address, and how much graph context is recoverable from a flat image.

Engineering question	Decision	Comparison / evidence
Should the VLM directly name the BIM element?	No. The VLM emits typed JSON; deterministic queries name only existing IFC GUIDs.	Fine-tuned VLM reaches 100% GT-in-pool but only 6.7% Top-1 — sibling selection needs explicit spatial structure.
Can generic retrieval solve the task?	No. Retrieval is ontology-constrained, then re-ranked by a spatial address over the IFC graph.	Lexical/dense baselines stay at 1.7% Top-1; the address ceiling reaches 78.5% Top-1 / 98.1% Top-10 (n=60).
Should noisy address fields hard-filter candidates?	No. The address is a soft prior with calibrated answer/defer routing, preserving recall.	Realized end-to-end reaches 58.9% Top-1 (67.6% with oracle coarse fields); selective prediction lifts the answered subset to 73.4%.
How deep should graph context go?	Compile one-hop context into the element record; do not chase deep relation chains from an image.	Realizable discrimination saturates at one hop: the confusable set shrinks from 13 to about 8, with little further realized gain.
Learned perception vs deterministic specialists?	VLM for coarse semantics; OpenCV/ResNet specialists for count, ordinal slot, and size.	The VLM reaches 100% on storey/class but 0% on slot/size; the realized slot specialist reaches 58.9% Top-1 end-to-end (n=35 fillers).

InnovationReading space from a flat image

The paper proved the architecture sound — with perfect perception the symbolic engine keeps the correct element 100% of the time and compresses the shortlist toward one, so the bottleneck is perception, not graph logic. The core advance deepens the deterministic specialists into a principled spatial-interpretation layer: instead of ad-hoc cues, the system recovers a spatial address — a relational key read from plain 2D images, computable from the BIM model and recoverable from the evidence. Three questions, answered with measurements:

One image, one GUID — the address narrowing the pool at each stage

Median candidate pool on the AP held-out targets (n=60). Each stage adds one slice of the spatial address; the same real case is traced below.

Stage	What it does	Median pool
VLM typed output	The fine-tuned VLM emits {storey, ifc_class, spatial_relations[]} — each field a {value, confidence, source} record, no GUID	1233
Coarse attribute-only retrieval	storey + IFC class filter the graph to same-type elements	46
Topology-aware retrieval	FILLS / NEXT_TO / CONNECTS_TO edges narrow to same-host-wall siblings	~9
Spatial address (KG traversal)	the type-conditional address — position-slot (i of M) for an opening — reorders the siblings to the target	1

A calibration gate (AUROC 0.80) then routes ANSWER (commit the GUID) vs DEFER (return the pool), and the deterministic slot soft-rerank produces the final ranked candidates — traced on one real case below.

ResultsArchitecture Oracle Experiment

The Oracle test measures the symbolic engine under perfect extraction — no VLM hallucination noise — to isolate how far the graph-query strategy can go. With the full spatial address it drives the median pool from 1233 candidates down to 1: the exact position-slot is a decisive discriminator once supplied, which is exactly why perception, not graph logic, is the bottleneck.

Downstream retrieval — the black box plateaus; the address does not

AP held-out benchmark (n=60), p0∪p1 planner.

Model / diagnostic	GT-in-pool	Top-1	Top-10	MRR@10
Zero-shot Gemini	91.7%	1.7%	18.3%	0.056
Fine-tuned VLM (best end-to-end)	100%	6.7%	30.0%	0.110
+ realized position-slot specialist (end-to-end)	100%	58.9%	67.1%	—
+ realized slot, oracle coarse (upper bound)	100%	67.6%	80.9%	—
+ type-conditional spatial-address ceiling	100%	78.5%	98.1%	0.854

The ceiling row is a diagnostic upper bound, not a deployed end-to-end system. The realized row is the actual spatial-address path currently built — a deterministic position-slot specialist for the 35 window/door filler cases; realized wall/other extractors are future work. The realized 58.9% is the aggregate over those 35 fillers; on the realistic cluttered-floor subset it is 21.2% end-to-end (39.1% with oracle coarse fields) — the aggregate is flattered by 18 sparser re-rendered upper-floor plans, so we report the split.

The calibration is real — and pays off as deferral

Worked cases — answer vs defer

Two real held-out cases through the same gate: when the detector is correct and confident it commits; when it is wrong but unsure it abstains and returns candidates — the behaviour that makes a triage tool trustworthy.

Workflow ImpactFrom manual search to verification

Triage effort proxy — pool review becomes verification

Median candidate pool on the held-out targets; expected inspections ≈ ½·pool under a 15 s/inspection assumption — read as scale, not measured labour. The position-slot row is the oracle spatial-address ceiling (Top-1 78.5%).