LMBuild: Evaluating LLM Agents for Generating Buildable and Functional Structures
Jiateng Liu, Rushi Wang, Cheng Qian, Xuejun Zhang, Sun Li, Jiayu Liu, Yifan Shen, Xu Cao, Jiarui Yao, Bingxuan Li, Ruhi Sarikaya, Heng Ji
LMBuild evaluates LLM agents on generating 3D structures that are physically buildable and functional, not just visually plausible.
How can we systematically evaluate whether LLM agents can generate 3D structures that are not just visually plausible, but physically buildable and functional?
Current 3D generation benchmarks prioritize visual aesthetics and geometric consistency, leaving a gap in whether generated objects can actually be fabricated or perform their intended mechanical functions. LMBuild provides an interactive environment where agents use tools to retrieve or create parts, define joints, and specify assembly sequences for real-world structures. Evaluations reveal that while frontier models excel at structural soundness, they struggle significantly with functional affordance and physical operability, often failing to generate complete kinematic chains or operable assemblies.
Paper Primer
The benchmark hinges on a hierarchical evaluation framework that moves beyond geometry to assess four dimensions: structural soundness, functional affordance, design quality, and physical realization. Agents interact with a standardized harness to "speak structures into existence," using tools to retrieve catalog parts or create novel components, which are then validated through simulation-based diagnostics and rule-based reports.
Frontier models have largely solved structural soundness and visual alignment, but functional operability remains a major bottleneck.
Even the strongest models achieve only mid-30s scores on simulation-based operability, while functional part completeness and kinematic correctness remain significantly lower than design quality metrics. Frontier closed-source models achieve near-perfect structural soundness, yet functional affordance scores for joints and kinematics rarely exceed 50/100.
Part creation is a critical capability that distinguishes frontier models from weaker ones.
Weaker models degrade sharply when forced to create parts, whereas frontier models effectively use creation to tailor geometry, often outperforming their retrieval-only performance. A 32B-parameter model saw its performance drop from 61.7 to 34.1 when moving from retrieval-only to creation-only settings.
Why does this benchmark require an interactive environment rather than static generation?
Generating buildable structures requires long-horizon reasoning, part-level decomposition, and iterative refinement based on physical constraints, which cannot be captured by single-shot geometric synthesis.
Does providing more detailed prompts help models overcome these functional gaps?
Yes; enriching prompts with explicit functional requirements and attributes significantly improves kinematic correctness and physical operability, suggesting that models struggle more with inferring functional intent than with executing it once specified.
Introduction
We expose the gap between visually plausible 3D outputs and their real‑world buildability and functionality.
LLM‑based agents can now generate intricate 3D structures, but visual plausibility does not guarantee that the resulting objects can actually be assembled or used. Existing evaluations largely ignore structural soundness, functional affordances, and physical realizability, leaving a critical gap in assessing real‑world applicability.
Agents must go beyond producing attractive geometry and instead generate objects that can be physically assembled and function as intended.
To fill this gap we introduce LMBuild, a benchmark that provides (1) an interactive environment with construction tools, (2) a curated dataset of 2,549 reference objects enriched with real‑world part and joint information, and (3) a four‑level evaluation hierarchy covering structural soundness, functional affordance, design quality, and physical realization.
The core challenge is bridging the divide between visually plausible designs and physically buildable, functional structures.
The LMBuild Framework
We detail the LMBuild framework, its interaction protocol, and the experimental pipeline.
The LMBuild framework provides a unified environment for testing LLM agents that must construct physically realizable 3‑D objects. It combines an interactive harness, a set of manipulation tools, and a hierarchical metric suite that judges buildability, operability, and design quality. Below we introduce the framework, its baseline protocol, and the step‑by‑step experimental pipeline.
The framework is a sandbox where an LLM agent can assemble parts, assign joints and materials, and output a complete assembly sequence that could be built in the real world.
How does LMBuild differ from prior simulation environments for 3‑D generation?
Previous simulators typically require the agent to output full geometry and explicit joint parameters, which forces the model to solve both design and physics simultaneously. LMBuild separates the design task (part selection, joint choice, material assignment) from low‑level physics by validating feasibility after each step, allowing agents to focus on high‑level construction decisions.
The protocol defines a single‑round interaction where the agent receives only a target label and an image, then autonomously builds the object by calling toolbox functions.
Why does the baseline allow both retrieval and creation instead of fixing one mode?
Allowing both lets the evaluation measure how much the agent relies on existing inventories versus its capacity to invent novel components. The constrained retrieval‑only and creation‑only ablations then reveal the separate value of each capability.
Initialize the agent harness with the user’s textual instruction and reference image.
Agent queries the toolbox to list available parts and inspects the current scene.
Using tool calls, the agent selects or creates parts, places them, and adjusts poses.
Agent decides joint configurations and material assignments for each component.
Agent proposes an assembly sequence that respects kinematic constraints.
The environment renders six views, runs a rule‑based diagnostic (collision, stability, joint issues, missing functions), and returns the report.
If multi‑round revision is enabled, the agent receives the report and repeats steps 2‑6 to improve the design.
**Figure 1.** LMBuild Framework (Using real example produced by GPT-6 Astra). Given a text prompt and conditional image, the builder LLM agent uses tools to inspect and select existing parts, create new parts, and place and adjust components to construct the wheelchair. It then specifies joints, materials, and an assembly sequence to support kinematics and physical-world realization.
To populate LMBuild, we repurpose several public 3‑D datasets—LEGO toy collections, static structures, mechanical assemblies, and daily‑object CAD models—augmenting them with Wikipedia‑derived part and affordance annotations. Table 1 summarizes the source sizes and the split used for evaluation (LMBuild‑Core, 200 objects).
Dataset and Benchmark Configuration
We detail how the LMBuild dataset is assembled, split, and measured.
To create a benchmark for 3‑D generation we start from a large collection of raw models, filter them for structural soundness, and then split the remaining set into a compact “core” subset and a larger “full” subset. For each subset we report the number of models and two derived metrics: average parts per model (a proxy for buildability) and average affordance (how easily the parts can be assembled).
The table provides a breakdown of the LMBuild dataset, categorized by geometry type and data source. It includes columns for "Original Size," and for both "LMBuild-Core" and "LMBuild-Full" configurations, it lists "Size" and "Avg. Parts / Aff." metrics.
Hierarchical Evaluation Metrics
We describe the hierarchical evaluation pipeline, its metrics, and the experimental variations.
The evaluation framework quantifies buildability and functionality through a hierarchy of four complementary dimensions, while varying feedback rounds and prompt detail to probe agent capabilities.
Structural Soundness measures whether a generated geometry can exist in the real world without collapsing or intersecting parts.
Assembly Affordance evaluates whether the generated object contains the parts and joint relationships required for its intended function.
Select an object from the core benchmark (200 items).
Prompt the LLM agent to generate a 3‑D structure; optionally include fine‑grained descriptions.
Run the hierarchical metric suite (S, A, D, R) on the generated model, applying the clip function to each sub‑metric.
If $K>1$, feed the metric feedback to the agent and repeat steps 2–3 for the specified number of rounds.
Aggregate the final scores (average of the four dimensions) to obtain the overall evaluation result.
Apply clip: $S_{\text{clipped}}=78$, $A_{\text{clipped}}=65$, $D_{\text{clipped}}=70$, $R_{\text{clipped}}=60$.
Feedback is returned to the agent; the second generation yields raw values $S=0.85$, $A=0.78$, $D=0.73$, $R=0.68$.
Clip again: $S_{\text{clipped}}=85$, $A_{\text{clipped}}=78$, $D_{\text{clipped}}=73$, $R_{\text{clipped}}=68$.
Final aggregated score = (85 + 78 + 73 + 68) / 4 = 76.
Iterating with feedback improves both structural soundness and affordance, showing that the metric suite can guide the agent toward more buildable designs.
**Figure 2.** **Left:** Generation examples produced by the LMBuild framework. Colors are used only to visually distinguish decomposed components. **Right:** The generated structures are evaluated using our comprehensive set of evaluation metrics. We briefly illustrate the reference source and general evaluation procedure for each metric; full evaluation details are provided in Appendix A.4.
By systematically varying $K$ and prompt granularity, we can isolate whether failures stem from ambiguous specifications or from the agent’s inability to translate detailed requirements into buildable structures.
Experimental Results
We evaluate 30 systems, exposing gaps between proprietary and open‑weight models.
The paper’s premise is that LLM agents often produce visually plausible 3D structures that nevertheless cannot be built or operated; LMBuild supplies a standardized environment and hierarchical metrics to expose these failures.
Closed‑source LLMs dominate overall performance, achieving the highest scores on most hierarchical metrics.
Table 2 shows the top closed‑source model reaches 99.6 on $S.1$ while the best open‑source model peaks at 95.9 on $S.2$.
Weaker models lose performance sharply when forced to create parts, whereas stronger models maintain or improve their scores.
Qwen3‑VL‑32B scores 61.7 under retrieval‑only but drops to 34.1 when limited to creation‑only.
Explicit functional specifications in prompts yield large gains, while additional revision rounds add only modest improvements.
Prompting improves functional‑part completeness by +10.3 points and operability by +12.0 points (p < 0.001); three revision rounds increase overall scores by only ~2%.
**Table 2.** Evaluation results across 30 systems under Baseline Interaction Protocol (§ 2.1) on core set, assessed using our hierarchical evaluation metrics (§ 2.3). We use $^\dagger$ (P) to denote systems integrated with Particulate to obtain joint information (Appendix A.2). Background colors encode performance on a common 0–100 scale, transitioning from blue (lower performance) through gray to orange (higher performance). Systems unable to produce the required output are assigned as N/A.
**Figure 3.** Model Performance Change across evaluation metrics under three part-access settings with 25 sampled LMBuild objects. Allowing part creation can hurt weaker models, even when catalog parts remain available, suggesting that the added design freedom increases decision complexity. In contrast, frontier models benefit substantially from creating structure-specific parts, revealing a stronger ability to exploit flexible construction mechanisms rather than relying only on retrieval.
**Figure 4.** Average performance change across models from (1) additional revision rounds, (2) explicit attribute requirements, and (3) explicit functional requirements. Revision yields small but consistent gains, while explicit prompts substantially improve functional-part completeness, kinematics, and especially operability. **Bars:** 95% bootstrap confidence intervals over models; **Stars:** Wilcoxon signed-rank test over episodes (*p < 0.05, **p < 0.01, ***p < 0.001)
Proprietary models consistently outperform open‑weight counterparts, but the gap narrows on targeted functional metrics when prompts are enriched.
Related Work
We situate our work among prior 3D generation research and outline ethical and reproducibility considerations.
Recent language models can generate 3D content by outputting procedural scripts, scene blueprints, or CAD operation sequences, linking natural‑language understanding to programmatic geometry creation. Visual feedback loops enable training and iterative refinement of the generated geometry. However, these approaches mainly target visual correctness and editability, not whether the resulting structures are physically buildable or functionally operable.
Work on part‑aware generation treats objects as collections of components, predicting joints, articulation, and physical attributes such as scale and material. Complementary lines of research address stable brick construction, assembly programs, component placement, and robotic execution. Together they highlight the need to evaluate part structure, physical behavior, and functional realization of assembled objects.
Existing 3D evaluation metrics focus on visual quality, text alignment, and fine‑grained geometric consistency, sometimes extending to multimodal appearance judgments. For assembled structures, prior work adds stability, component selection, placement, and construction success checks. LMBuild extends this landscape by jointly assessing structural soundness, affordance, design quality, and physical realization using geometric checks, component assessments, simulation‑based motion tests, and VLM judgments.
We introduced LMBuild, a benchmark that measures LLM‑based agents across structural soundness, functional affordance, design quality, and physical realization. Experiments show strong performance on visual and structural criteria but notable gaps in functional part completeness, kinematic correctness, and operability in simulation. These findings underscore the challenge of turning object specifications into coherent, buildable, and functional assemblies.
Large language models were employed as writing assistants to improve manuscript clarity and as coding agents for evaluation‑environment construction and code maintenance. All research decisions, experimental designs, implementations, analyses, and reported results were reviewed and verified by the authors. AI tools did not influence the scientific conclusions.
Our open evaluation framework aims to promote transparent, reproducible assessment of LLM agents for real‑world structure generation, emphasizing buildability and functional validity under real‑world constraints. While the benchmark facilitates rigorous model comparison, it captures only a subset of requirements for safety‑critical engineering, and deployment must involve domain‑specific validation and human oversight.
We provide representative data samples, evaluation materials, and the full codebase, including scripts, model trajectories, and environment specifications, in the supplementary materials. This open release enables independent verification of our results and supports future extensions of LMBuild for more powerful agents.
Design Quality Metrics
We define calibrated Design Quality metrics and Physical Realization checks for 3D outputs.
Design Quality quantifies how well a generated 3D artifact satisfies buildability and functional criteria, using both perceptual VLM judgments and programmatic signals.
Why combine perceptual VLM scores with programmatic signals instead of using only one?
Perceptual scores capture holistic visual plausibility, while programmatic signals provide concrete evidence of structural defects; together they reduce blind spots that each alone would miss.
D.1 measures whether a design is broken into meaningful, independently realizable parts rather than a monolithic mesh.
How does the granularity term differ from simply counting parts?
It normalizes the count mismatch on a logarithmic scale, so a factor‑2 difference yields the same penalty regardless of absolute size.
D.2 evaluates the visual appeal of a generated structure without reference to any target design.
Why is there no computed signal for aesthetics?
Visual appeal is subjective and hard to capture with deterministic geometry checks; the VLM judges provide a learned proxy aligned to human perception.
D.3 checks whether the generated artifact preserves the overall organization dictated by the task condition.
What would happen if we used equal weighting for D.3 instead of 3:1?
Because the perceptual assessment already captures most alignment cues, over‑emphasizing the computed terms would dilute the metric’s sensitivity to visual mismatches.
R.1 verifies that the declared part insertion order can be executed without collisions and that each intermediate sub‑assembly remains stable.
Why is the intermediate‑state score $S$ weighted by one‑half in the final aggregation?
Because insertability is a hard feasibility constraint, while stability is a softer quality measure; the weighting reflects their relative importance.
R.2 checks that each part’s assigned material class matches the set of acceptable materials for its semantic role.
What does a 0.5 score indicate for a part’s material compatibility?
It means the assigned material belongs to the same broader family as an acceptable class, offering partial but not full compliance.
The table presents metrics $D.1$, $D.2$, and $D.3$ with corresponding ranges for $\ell_J arrow u_J$, $\ell_C arrow u_C$, and the ratio $\alpha : \beta$.
Operability and Realization
Operability R quantifies whether generated artifacts can perform their intended tasks.
Operability measures whether a generated artifact can actually perform the task it was designed for, by checking that all required parts, connections, motions, and human‑use geometry are present and functional.
Why does Operability use a weakest‑link aggregation instead of averaging the sub‑scores?
Because a capability is only as strong as its most limiting requirement; averaging could mask a critical missing joint or unreachable control, whereas the minimum directly penalizes any such failure.
**Table 5.** Alignment between calibrated Design (D) scores and human ratings on 7 held-out objects (6 systems, 42 designs per dimension; 4 annotators). Cells report system mean scores with ranks in parentheses. We report system-level Spearman $\rho$ and MAE, item-level Spearman $\rho$, and human–human $\rho$ as a reference for annotator agreement.
Limitations and Failure Analysis
Interpreting metric consistency and outlining the benchmark’s current constraints.
LLM agents often produce visually plausible 3D structures that nevertheless cannot be built or operated; LMBuild supplies a standardized environment and a hierarchical metric suite to expose these physical and functional shortcomings.
**Table 2.** Performance of various models across different subsets (Full core, Sample A, Sample B) evaluated on four metrics: Soundness (S), Affordance (A), Design (D), and Realization (R).
**Table 8.** Failure modes of the six closed-source APIs on the 200-object core set (% of tasks; thresholds in parentheses). All models submit a design on every task.
**Table.** Failure mode analysis across different models.
Brick‑based generators (e.g., BrickGPT, LegoACE) achieve top Soundness scores because their lattice representation enforces connectivity and prevents interpenetration by construction.
Part‑based mesh generators (Particulate, Cube3D+CubePart) inherit geometric compatibility from their generation pipelines, yet they may still lack mechanically meaningful connections.
Systems that output smaller assemblies (e.g., Qwen3.5‑27B) naturally satisfy local Soundness and Realization criteria, because fewer parts reduce the chance of faulty interfaces.
Mesh generators excel at aesthetic quality, producing smooth, watertight surfaces, but this visual appeal does not guarantee that the generated object matches the target design or exhibits functional behavior.
PhysX‑Anything’s explicit material prediction improves material plausibility scores, yet the system still fails to deliver complete functional assemblies.
Three recurring factors explain why some models outperform on isolated metrics: (1) the output representation enforces certain properties, (2) reduced assembly complexity eases metric satisfaction, and (3) visual‑quality optimization boosts appearance‑related scores.
Evaluation scope is deliberately limited to reproducible geometric, semantic, kinematic, and simulation tests; it does not certify manufacturability, safety, or performance under real‑world tolerances.
Physical models used in LMBuild are simplified (e.g., tipping analysis, rigid‑body simulation with fixed density and friction), which may affect scores for designs that rely on deformable or material‑specific behavior.
The benchmark’s core split of 50 objects is coverage‑oriented, not a population‑level sample; broader evaluation on the full 2,399‑object split is needed for comprehensive assessment.
Design Quality combines deterministic metrics with Vision‑Language Model judges calibrated against human annotators; while system‑level rankings align well, item‑level ambiguity remains.
Comparability across systems is constrained by differing interfaces, single‑run evaluations, and varying tool budgets; future work should incorporate repeated generations and standardized native tool‑calling.
Common failure modes in current LLM‑based 3D agents are (1) missing or incorrectly placed functional components, (2) inadequate joint specifications, and (3) incomplete or non‑executable assembly sequences.
Agent Harness Details
Describes the LMBuild harness variants and interaction protocols for different agent types.
The LMBuild harness provides a common interactive assembly environment that heterogeneous model families can access through a lightweight, model‑specific interface. Closed‑source agents issue shell commands, while open‑source models use an OpenAI‑compatible chat interface with text‑serialized tool calls. Domain‑specific generators that lack iterative tool support are evaluated using their native pipelines.
Three environment specifications are defined: Tier A permits only catalog part retrieval and placement; Tier B (the primary baseline) adds the ability to create new components when the catalog lacks a suitable part; Tier C removes the catalog entirely, requiring agents to create every component before placement.
The environment enforces a hard budget of 160 tool calls per episode, including at most 6 calls to check, 4 calls to render, and 80 placed parts. Part‑creation calls are limited to 30 in Tier B and 60 in Tier C.
Episodes may span multiple revision rounds; calling submit ends the current round and returns a structured diagnostic (condition, isometric, front, and side‑view images). The agent revises the same assembly and submits again until the final submit yields `episode_done`=true.
The shared system prompt instructs agents to assemble a physical object from 3D parts, requiring the result to be buildable, stable, and functional. It defines a world frame (metres, +Z up, +X right, front facing –Y) and specifies that parts are positioned by their bounding‑box centre with rotations applied in X‑Y‑Z order.
Agents must declare each part’s role, choose a material from a predefined list, specify joints (type and limits), and provide an assembly sequence that mirrors a human build order. Connections are considered valid when surfaces touch within about 4 mm or when a joint’s origin lies within 2 cm of both parts.
Closed‑source agents (Claude Haiku 4.5, Sonnet 5, Opus 5, Fable 5.1) interact via a shell wrapper that invokes the LMBuild session with JSON‑encoded tool arguments; image‑producing tools return PNG paths that Claude can read.
Open‑source LLMs and Vision‑Language Models (VLM) are served through vLLM using an OpenAI‑compatible chat endpoint; tool calls are expressed as JSON blocks, and parsing tolerates limited fallback formats while enforcing strict JSON syntax.
Domain‑specific structure generators are excluded from the LMBuild harness when their native pipelines cannot perform iterative tool calls; they receive the benchmark condition supported by their original inference pipeline and operate in a single‑generation mode.
Environment Implementation
Implementation specifics of the LMBuild environment, tools, and interaction limits.
LMBuild is a stateful, pure‑Python environment where the entire episode state—placed instances, poses, part programs, joints, assembly sequence, counters, and a full call trace—is stored as a single JSON‑serialisable dictionary. This design lets an episode be driven in‑process or from a shell, checkpointed after every round, and replayed exactly.