An Interpreter Layer for AEC: Cross-Modal Grounding & Ontology-Driven Retrieval

The ProblemThe physical–digital traceability gap

AEC projects standardise on OpenBIM/IFC, yet day-to-day site coordination still runs on manual interpretive labour: people translate unstructured evidence — photos, chat messages, floorplan patches — into model-linked, schema-compliant records. The front-line worker sees a local condition but not the BIM GUID; the coordinator sees the model but not the site context.

The final step is choosing one element from a retrieved shortlist that already contains the answer (median 76 candidates, ground truth present 100% of the time). So the difficulty is not retrieval but discriminating among visually identical siblings — what separates two windows is not in their pixels, it is in the building's relational structure. A black-box matcher plateaus at 6.7% right-first with the answer already in the pool. This work makes that relational signal explicit as a spatial address that can be read from evidence and executed against BIM.

AbstractOverview

ContributionsWhat this work delivers

ArchitectureThe neuro-symbolic interpreter

Five stages: multimodal input → neural extraction (VLM + LoRA) → Constraints JSON → symbolic retrieval (priority Cypher cascade over a Neo4j IFC graph) → Graph-RAG reranking → ranked GUIDs. The neural side translates evidence into a typed spatial address; the symbolic side names only GUIDs that exist in the model. Design rationale: reliability over flexibility (deterministic templates instead of Text-to-Cypher), and an IFC-native graph built from the schema itself rather than noisy document-level extraction.

Key ModulesHow it works under the hood

IFC Parse & Enrich Engine

Converts raw IFC into a query-ready Neo4j graph, enriching topology beyond native containment/fill with retrieval-oriented edges: NEXT_TO, CONNECTS_TO, ADJACENT_TO, ON_STOREY.

IfcOpenShell · Neo4j

Symbolic Planner + Graph-RAG

Nine priority strategies; the recall-preserving p0∪p1 union planner combines attribute filtering with spatial-topology ranking. Templates are deterministic — same input, same query, same result.

Cypher · priority cascade

Synthetic Data Pipeline

Skeleton-first generation fixes ground truth before adding stochastic visual evidence: IFC mining → Blender renders with hard negatives → Gemini text → LLM-as-judge filter → 990 cases, no real labels.

Blender · Gemini · LLM-as-judge

Deterministic Visual Specialists

OpenCV handles counting and ordinal slot position; ResNet classifies element size bands — the sub-tasks VLMs do unreliably. These inject discriminative cues into the reranker.

OpenCV · ResNet

Fine-tuned VLM Extractor

Qwen2.5-VL + LoRA (G-series) emits a typed contract {storey, ifc_class, spatial_relations[]}. Fine-tuning saturates coarse fields (storey/class → 100%) and learns relation typing (direction 0% → 82%), but discriminating slot/size stay at 0% — the evidence for delegating them to specialists.

Qwen2.5-VL · LoRA · Constraints JSON

Calibrated Routing & Control

Every field carries a {value, confidence, source} record. An orchestration layer routes on calibrated confidence (AUROC 0.80) — applying constraints softly and deferring to a human instead of guessing. Studied as static → learned router → LLM-agent ablation on accuracy, latency, and repeatability.

temperature scaling · answer/defer gate

Engineering decisions, measured against alternatives

Each contract has an experiment behind it: what to let the VLM do, what to push into deterministic geometry, when to trust an extracted address, and how much graph context is recoverable from a flat image.

InnovationReading space from a flat image

The thesis proved the architecture sound — with perfect perception the symbolic engine keeps the correct element 100% of the time and compresses the shortlist toward one, so the bottleneck is perception, not graph logic. The core advance deepens the deterministic specialists into a principled spatial-interpretation layer: instead of ad-hoc cues, the system recovers a spatial address — a relational key read from plain 2D images, computable from the BIM model and recoverable from the evidence. Three questions, answered with measurements:

ResultsThe address makes the architecture realizable

Oracle fingerprint ladder

Median live pool · AP held-out benchmark (n=60)

Fingerprint level	Median pool
L0 — no filter	1233
L1 — storey + class	46
L3 — + direction + subtype	9
L4 — + exact position slot*	1

*L4 extractable for the addressable filler subset. L3 fingerprints provide the dominant compression.

Downstream retrieval — the black box plateaus; the address does not

AP held-out benchmark (n=60), p0∪p1 planner.

The ceiling row is a diagnostic upper bound, not a deployed end-to-end system. The realized row is the actual spatial-address path currently built — a deterministic position-slot specialist for the 35 window/door filler cases; realized wall/other extractors are future work.

The calibration is real — and pays off as deferral

Worked cases — answer vs defer

Two real held-out cases through the same gate: when the detector is correct and confident it commits; when it is wrong but unsure it abstains and returns candidates — the behaviour that makes a triage tool trustworthy.

Workflow ImpactFrom manual search to verification

Triage effort — search becomes verification

Arm	Median inspections	≈ time / element	Success@1
Manual scan, no ranking	38.0	~570 s	1.8%
Coarse storey + class	23.0	~345 s	4.9%
+ spatial-address ceiling	0.5	~8 s	78.5%

A ranking-derived effort proxy (15 s/inspection assumption), not a user study.

DemoOne case, end-to-end

A browser demo takes an uploaded construction photo plus a natural-language description and highlights the matched IFC elements in a 3D viewer — no AEC software required. The interactive version lets you pick a representative held-out case and inspect both the knowledge-graph spatial address that drives retrieval and the grounded element highlighted in 3D.