SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation
Yanlun Tu, Huacan Wang, Ziyue Zhou, Jie Zhou, Ningyan Zhu, Ge Chen, Wangyi Chen, Tengfei Zhou, Yifan Zhou, Dasheng Yang, Xiaofeng Mou, Hui Zhang, Yi Xu
SemaPLC uses a verification-gated agent harness to ground PLC code generation in project context and runtime behavior.
How can we reliably integrate LLM-generated PLC code into existing industrial projects using verification-gated agent loops?
Industrial control logic is rarely an isolated unit, yet current generation systems struggle to integrate code into existing projects or ensure it behaves correctly on a live runtime. SemaPLC anchors generation in the target project's context and refuses to terminate until external checks—specification audits, compilation, and live runtime validation—confirm the logic works. On a 65-task project-context benchmark, this verification-gated approach achieves a 52.2% mean dynamic behavior score, significantly outperforming fixed-pipeline baselines that drop to single digits.
Paper Primer
SemaPLC operates as an event-driven agent harness that treats verification results as binding constraints rather than optional feedback. The core mechanism is a verification gate: the agent must repair any failed check until the tool logs provide evidence of success, with any code modification voiding all prior verdicts to ensure the final output is identical to the version that passed.
Runtime validation is the most discriminative metric for PLC code reliability.
While static behavior scores for different methods cluster within 4 points, dynamic behavior scores diverge sharply, ranging from 22.4 to 31.4 for baselines versus 52.2 for SemaPLC. A 20.8-point lead over the strongest baseline on the project-context track.
The harness acts as a model-agnostic reliability layer.
Across seven backbone models, SemaPLC consistently attains the highest strict verified pass rate, narrowing the cross-model performance spread from 37.6 points (bare) to 14.6 points. An average improvement of 17.3 percentage points over the bare backbone.
Why is runtime validation necessary if static and formal checks are already performed?
Formal verification often fails on stateful timing constructs like timers (TON) that span scan cycles, and static analysis cannot detect logic errors in output selection under competing conditions. Runtime validation directly observes temporal and state behavior, making these failures observable and repairable.
Does this harness require more computational resources than existing methods?
On the project-context track, SemaPLC requires significantly more model interactions (34.1 requests per task vs 6.9 for the strongest baseline) because it iterates until external checks are satisfied, whereas fixed pipelines are limited by a preset iteration budget.
Introduction and Motivation
Why generating PLC code in isolation fails to meet real‑world integration and safety needs.
Industrial PLC programs are typically written as collections of program organization units (POUs) that must interoperate within a larger project and obey strict safety constraints. Existing LLM‑based generators produce POUs in isolation, leaving a critical gap: the generated code often cannot be dropped into an existing project nor guaranteed to behave correctly at runtime.
The harness treats code generation as a collaborative loop where an LLM proposes edits, external verification tools certify each edit, and the loop only stops when all checks agree the program is ready for deployment.
How does the SemaPLC harness differ from earlier “LLM‑for‑PLC” pipelines that also use compilation feedback?
Earlier pipelines stop when the LLM judges its own output adequate, even if the compiler later reports an error. The SemaPLC harness never trusts the model’s self‑assessment; it requires an external compiler or runtime verdict to be logged as a pass before termination, turning verification into the gating condition rather than an optional post‑hoc check.
Empirically, the harness raises the mean verified pass rate to 72.6 % on 117 independent‑POU tasks and achieves the highest dynamic‑behavior score (52.2) on the 65 project‑context tasks, a stark contrast to baselines that cluster between 22.4 and 31.4 when evaluated on live PLC runtimes.
The gap between isolated POU generation and project‑integrated PLC code is closed by enforcing verification‑gated completion.
Prior Work and Problem Scope
We formalize two generation tracks and define the verification‑gated pipeline that underlies SemaPLC.
Two distinct generation tracks are considered: the function track, which produces a single POU from a requirement, and the project‑context track, which augments an existing PLC project with new logic. Both tracks are evaluated against verification criteria that gate acceptance.
The function track maps a requirement Rf and an interface If to a single POU Lf, while the project‑context track augments an existing project P with new logic Lp, producing an integrated program P′.
LLM4PLC uses a large language model to generate Structured Text (ST) code, feeding compiler and SMV feedback into a user‑guided iterative loop.
AutoPLC extends LLM4PLC with vendor‑aware ST generation, retrieving case examples, recommending APIs, and debugging directly inside the vendor IDE.
Agents4PLC introduces a five‑agent closed loop that validates generated code with PLCverif and demonstrates the approach on a 117‑task benchmark.
Verification‑gated generation loop
Across both tracks, the same notation (R, X, L) is used, and success is gated by the verification criteria defined above. The function track relies on a single VerifiedPass check, while the project‑context track requires compilation, static analysis, and dynamic trace similarity to all meet their respective thresholds.
The SemaPLC Agent Harness
The harness iteratively refines PLC code via a verification‑gated loop that guarantees correctness before acceptance.
Generating PLC code directly from a natural‑language requirement often produces programs that fail syntax, compilation, or runtime checks, forcing costly manual rewrites.
The agent repeatedly generates code, runs a suite of verification checks, and only accepts the program when every check logs a passing verdict.
Grounding produces $\Gamma$: a table of existing variables and I/O tags.
Generate $L$: a ladder program with a start rung, stop rung, and motor coil.
Run spec check – fails because the stop rung lacks a de‑energize condition; diagnostic $e_{\text{spec}}$ points to the missing rung.
Repair $L$ by adding a “stop” contact; reset $\mathcal{V}$.
Run compile check – passes; run runtime check – fails due to a timer overflow; diagnostic $e_{\text{runtime}}$ identifies the timer.
Repair $L$ by adjusting the timer preset; budget $B$ decrements to 2.
All three checks now pass; $\mathcal{V}$ records verified verdicts and the loop exits.
The loop guarantees that the final program is exactly the version that earned every logged pass, eliminating hidden divergences between generation and verification.
Ground the task context $X$ to obtain $\Gamma$.
Generate candidate code $L$ conditioned on $R$ and $\Gamma$.
For each check $c\in K$, run $c$ on $L$ and collect verdict $v$ and diagnostic $e_c$.
If all $v$ satisfy the completion criteria, return $(L,\mathcal{V})$.
Otherwise, select failing checks with retries $<r$ and invoke REPAIR using their diagnostics.
If REPAIR changes $L$, clear $\mathcal{V}$; decrement budget $B$ and repeat.
Verification‑gated generation (Algorithm 1)
**Figure 1.** Overview of the SEMAPLC agent harness: a model-agnostic agent core, grounded in the task or project context, acts through a shared PLC MCP tool layer, and a verification gate decides completion.
**Algorithm 1** Verification-gated generation in SEMAPEC **Require:** requirement $R$; context $X$ (POU interface or project); required checks $K$ with completion criteria; per-check retry limit $r$; interaction budget $B$ **Ensure:** implementation $L$ with logged verification results $\mathcal{V}$, or failure 1: $\Gamma \leftarrow \text{GROUND}(X)$; $L \leftarrow \text{GENERATE}(R, \Gamma)$; $\mathcal{V} \leftarrow \emptyset$ 2: **while** budget $B$ remains **do** 3: $\quad$ **for all** $c \in K$ without a valid verdict in $\mathcal{V}$ **do** 4: $\quad \quad (v, e_c) \leftarrow \text{RUNCHECK}(c, L)$ {spec audit, compilation, live runtime} 5: $\quad \quad \mathcal{V}[c] \leftarrow v$ if log entry $e_c$ confirms $v$, else *unchecked* {earned claims} 6: $\quad$ **end for** 7: $\quad$ **if** $\mathcal{V}$ satisfies the completion criteria **then** return $(L, \mathcal{V})$ {accept} 8: $\quad F \leftarrow \{c \in K \mid \mathcal{V}[c] \text{ failed, retries}(c) < r\}$ 9: $\quad$ **if** $F = \emptyset$ **then** return failure with $\mathcal{V}$ {no repairable check} 10: $\quad L' \leftarrow \text{REPAIR}(L, \{e_c \mid c \in F\})$ 11: $\quad$ **if** $L' \neq L$ **then** $\mathcal{V} \leftarrow \emptyset$ {edit invalidation} 12: $\quad L \leftarrow L'$ 13: **end while** 14: **return** failure with $\mathcal{V}$ {budget exhausted}
How does verification‑gated generation differ from a naïve “generate‑then‑check” pipeline?
In the naïve pipeline the generated code is checked once; failures are reported but the code is not automatically repaired. Verification‑gated generation closes the loop: each failure triggers a targeted REPAIR step, clears prior verdicts, and re‑runs all checks, guaranteeing that the final accepted program has passed every check on the exact bytes that were logged.
Experimental Setup
Defines evaluation tracks, tasks, models, baselines, and metrics used to assess the harness.
We evaluate the harness along four research questions (RQ1–RQ4) that probe function‑level correctness, project‑grounded reliability, multi‑layer verification, and interaction cost relative to strong baselines.
Function‑track experiments use 117 independent‑POU tasks from the Agents4PLC benchmark; 43 of these required PLC‑engineer repair of defective verification properties.
Project‑context track experiments draw 65 tasks from Spec2Control, each providing a plant narrative, a function‑block interface catalog, and an empty entry harness for generated logic to compile and deploy within the full IEC 61131‑3 ST project.
Both tracks run seven backbone models (MiniMax‑M2.7, MiniMax‑M3, Qwen3.5‑Plus, DeepSeek‑V4‑Flash, DeepSeek‑V4‑Pro, GLM‑5.2, GPT‑5.5) and three published baselines (LLM4PLC, AutoPLC, Agents4PLC) using identical model endpoints.
The bare configuration removes the verification‑gated harness, allowing us to isolate the harness’s contribution to overall performance.
Metrics for the function track require candidates to compile with RuSTy and pass PLCverif model‑checking of requirement‑derived properties.
For the project‑context track we report static behavior S (percentage of oracle assertions satisfied) and dynamic behavior D (mean agreement of core output ports across up to six runtime scenarios), using the full 65‑task denominator and assigning a zero score to any failure to compile, deploy, or produce a trace.
Function-Level Reliability
SEMAPLC’s full harness delivers the highest strict verified pass rate across all models.
SEMAPLC attains the highest strict verified pass rate on all seven models, with an overall mean of 72.6 %.
Table 1 shows SEMAPLC’s overall mean 72.6 % versus the strongest baseline Agents4PLC at 63.9 % and a worst‑case of 67.5 % that exceeds every baseline’s mean.
**Table 1.** Function track: strict verified pass rate (%, denominator 117; inconclusive and empty generations count as failure). All methods are graded by the same held-out judge. bare is SEMAPLC with the harness stripped from the same backbone (no skills and tools); full is the complete harness. DS = DeepSeek.
Project-Level Integration Results
Project-level evaluation shows SemaPLC outperforms baselines in compilation success and dynamic reliability.
SemaPLC builds 89.4% of programs on average, surpassing baselines that achieve only 58.7%–81.5%.
Table 2 reports integrated compilation success rates across the project track.
Across the seven benchmark models, SemaPLC attains a mean dynamic‑behavior score of 52.2, well above the best baseline of 31.4, and never drops below 30, whereas fixed pipelines fall to single‑digit scores on their worst cases.
On the strongest GPT‑5.5 model, the dynamic advantage narrows to 1.8 points (65.4 vs 63.6) and SemaPLC trails baselines on static behavior (84.1 vs up to 88.8), showing that its gains stem from runtime verification rather than raw model capacity.
Figure 2 illustrates a coking‑refinery control task where low‑flow (< 50 kg/hr) requires SlaveSP = 500 and a transmitter fault requires SlaveSP = 2500; LLM4PLC and AutoPLC fail to compile, Agents4PLC compiles but overwrites the fault condition, while SemaPLC generates an intermediate candidate that compiles, detects the mismatch via runtime verification, repairs the logic, and re‑verifies the correct behavior.
**Figure 2.** Project-track case study (coking-refinery plant, task “Section 8”). LLM4PLC and AutoPLC fail compilation; Agents4PLC compiles but mishandles the fault because its low-flow assignment overwrites the fault default (the 500 result is inferred from source logic, not a runtime trace). SemaPLC’s intermediate candidate also compiles with incorrect behavior, but its runtime-verification step exposes the 500-versus-2500 mismatch, triggers a cause-specific repair, and re-verification confirms both abnormal cases. Identifiers are as each method generated them (Agents4PLC names the setpoint `Air_Flow_SP`).
Verification Layer Ablation
How each verification layer changes static, dynamic, and cost metrics.
The central premise—wrapping LLMs in a verification‑gated harness—has already shown strong static scores. This section asks whether each verification component actually contributes, by removing them one‑by‑one and measuring the drop.
Static verification checks properties that can be proven without running the code (e.g., compilation success), while dynamic verification observes the program’s behavior at runtime (e.g., correct output values).
Each added layer raises the dynamic score monotonically while static scores move only modestly. The cost (tokens and model requests) climbs sharply, especially for the full runtime layer, which is the most expensive step.
Adding the full runtime verification layer boosts the dynamic reliability score by +10.4 points.
Dynamic score rises from 43.7 (with compilation only) to 54.1 with runtime checks (Table 3).
Supplementary Data and Audits
Appendices detail audits, tool suites, and data access for the PLC verification framework.
The function‑track oracle audit proceeded in three blind rounds: an initial flagging pass, a full re‑derivation pass using brute‑force truth tables and state‑machine simulation, and a final adjudication pass.
The project‑track oracle audit examined 130 function‑block implementations across 65 control tasks, generating assertions automatically for 63 tasks and hand‑written ones for the remaining two.
The PLC tool suite lives on a single tool server exposing a Model Context Protocol (MCP) stdio interface and a command‑line interface; the latter includes a declarative verify runner that builds, drives, asserts, and cleans up in one plan.
Evaluation isolation separates what a method sees during generation from what scores the result; the agent observes requirements, compiler diagnostics, and its own audit feedback, while scoring uses only the hidden reference and golden trace.
Questions & answers
What is SemaPLC and what is its main contribution?
SemaPLC is an event-driven agent harness for generating IEC 61131-3 Structured Text (ST) PLC code that treats verification results as binding constraints rather than optional feedback. Its main contribution is a verification-gated completion mechanism: the agent must repair any failed check—specification audit, compilation, or live runtime validation—until all checks pass on the exact code bytes that were logged, refusing to terminate until external evidence of correctness is obtained.
What problem does SemaPLC address?
SemaPLC addresses the gap between isolated PLC program-organization-unit (POU) generation and project-integrated PLC code: existing LLM-based generators produce POUs in isolation that often cannot be dropped into an existing project or guaranteed to behave correctly at runtime. It also addresses the failure mode of earlier pipelines that stop when the LLM judges its own output adequate, even if the compiler later reports an error.
Why is runtime validation necessary if static and formal checks are already performed?
Formal verification often fails on stateful timing constructs like timers (TON) that span scan cycles, and static analysis cannot detect logic errors in output selection under competing conditions. Runtime validation directly observes temporal and state behavior, making these failures observable and repairable.
How does SemaPLC's verification-gated approach differ from a naïve generate-then-check pipeline?
In a naïve pipeline, generated code is checked once and failures are reported but not automatically repaired. SemaPLC's verification-gated generation closes the loop: each failure triggers a targeted REPAIR step, clears all prior verdicts, and re-runs all checks, guaranteeing that the final accepted program has passed every check on the exact bytes that were logged.
How does SemaPLC differ from earlier LLM-for-PLC pipelines that also use compilation feedback?
Earlier pipelines stop when the LLM judges its own output adequate, even if the compiler later reports an error. SemaPLC never trusts the model's self-assessment; it requires an external compiler or runtime verdict to be logged as a pass before termination, turning verification into the gating condition rather than an optional post-hoc check.
What benchmarks and datasets were used to evaluate SemaPLC?
Two tracks were used: the function track used 117 independent-POU tasks from the Agents4PLC benchmark (43 of which required PLC-engineer repair of defective verification properties), and the project-context track used 65 tasks from the Spec2Control benchmark, each providing a plant narrative, a function-block interface catalog, and an empty entry harness for generated logic to compile and deploy within a full IEC 61131-3 ST project.
What backbone models and baselines were evaluated?
Seven backbone models were tested: MiniMax-M2.7, MiniMax-M3, Qwen3.5-Plus, DeepSeek-V4-Flash, DeepSeek-V4-Pro, GLM-5.2, and GPT-5.5. Three published baselines were compared: LLM4PLC, AutoPLC, and Agents4PLC, all using identical model endpoints.
What metrics were used to evaluate the two tracks?
The function track required candidates to compile with RuSTy and pass PLCverif model-checking of requirement-derived properties. The project-context track reported static behavior S (percentage of oracle assertions satisfied) and dynamic behavior D (mean agreement of core output ports across up to six runtime scenarios), using the full 65-task denominator and assigning zero to any failure to compile, deploy, or produce a trace.
What are the key results on the project-context track?
Across seven benchmark models, SemaPLC attains a mean dynamic-behavior score of 52.2, well above the best baseline score of 31.4, and never drops below 30, whereas fixed pipelines fall to single-digit scores on their worst cases. On the strongest model (GPT-5.5), the dynamic advantage narrows to 1.8 points (65.4 vs. 63.6) and SemaPLC trails baselines on static behavior (84.1 vs. up to 88.8).
What are the key results on the function track?
The harness raises the mean verified pass rate to 72.6% on the 117 independent-POU function-track tasks. The paper does not report a specific numeric comparison to baselines for this track beyond noting this figure.
Does the verification-gated harness require more computational resources than existing methods?
Yes. On the project-context track, SemaPLC requires significantly more model interactions—34.1 requests per task versus 6.9 for the strongest baseline—because it iterates until external checks are satisfied, whereas fixed pipelines are limited by a preset iteration budget.
What does the verification layer ablation show?
Removing verification layers one by one shows that each added layer raises the dynamic score monotonically, while static scores move only modestly. The cost in tokens and model requests climbs sharply, especially for the full runtime layer, which is the most expensive step.
What are the limitations of SemaPLC?
SemaPLC requires significantly more model interactions per task (34.1 vs. 6.9 for the strongest baseline), making it computationally expensive. On the strongest model (GPT-5.5), its dynamic advantage over baselines narrows to only 1.8 points and it trails on static behavior, suggesting its gains stem from runtime verification rather than raw model capacity rather than being universally dominant.
How does SemaPLC handle project-context integration specifically?
SemaPLC anchors generation in the target project's context by providing the agent with a plant narrative, a function-block interface catalog, and an empty entry harness; generated logic must compile and deploy within the full IEC 61131-3 ST project. The project-context track requires passing compilation, static analysis, and dynamic trace similarity checks, all of which must meet their respective thresholds before the agent terminates.
What tooling infrastructure does SemaPLC rely on?
The PLC tool suite lives on a single tool server exposing a Model Context Protocol (MCP) stdio interface and a command-line interface; the latter includes a declarative verify runner that builds, drives, asserts, and cleans up in one plan. The function track uses RuSTy for compilation and PLCverif for model-checking.
How was the oracle audit conducted to ensure benchmark quality?
The function-track oracle audit proceeded in three blind rounds: an initial flagging pass, a full re-derivation pass using brute-force truth tables and state-machine simulation, and a final adjudication pass; 43 tasks required PLC-engineer repair of defective verification properties. The project-track oracle audit examined 130 function-block implementations across 65 control tasks, generating assertions automatically for 63 tasks and hand-writing them for the remaining two.
How is evaluation isolation maintained to prevent data leakage?
Evaluation isolation separates what a method sees during generation from what scores the result: the agent observes requirements, compiler diagnostics, and its own audit feedback, while scoring uses only the hidden reference and golden trace.
Who authored SemaPLC and where was it published?
The paper does not state the author names or the publication venue in the provided text.
Key terms
- PLC (Programmable Logic Controller)
- An industrial digital computer used to automate electromechanical processes such as control of machinery on factory assembly lines or chemical plants.
- POU (Program Organization Unit)
- A modular building block of IEC 61131-3 PLC programs, such as a function block or program, that encapsulates a piece of control logic.
- IEC 61131-3
- The international standard that defines programming languages for PLCs, including Structured Text (ST), Ladder Diagram, and Function Block Diagram.
- Structured Text (ST)
- A high-level, Pascal-like programming language defined in IEC 61131-3 used to write PLC control logic.
- Verification gate
- A mechanism in SemaPLC that prevents the agent from terminating until all external checks—compilation, static analysis, and runtime validation—have been logged as passing on the exact code bytes submitted.
- Dynamic behavior score (D)
- A metric for the project-context track measuring the mean agreement of core output ports between generated PLC code and a golden reference trace across up to six runtime scenarios, with zero assigned for any failure to compile, deploy, or produce a trace.
- Static behavior score (S)
- A metric for the project-context track measuring the percentage of oracle assertions satisfied by the generated PLC code without requiring live execution.
- Agents4PLC
- A published benchmark and baseline system for LLM-based PLC code generation that provides 117 independent-POU tasks used in SemaPLC's function-track evaluation.
- Spec2Control
- A benchmark providing 65 project-context PLC tasks, each with a plant narrative, function-block interface catalog, and an empty entry harness, used for SemaPLC's project-context track evaluation.
- RuSTy
- A compiler used in the function-track evaluation to check whether generated IEC 61131-3 Structured Text PLC code compiles successfully.
- PLCverif
- A model-checking tool used in the function-track evaluation to formally verify that generated PLC code satisfies requirement-derived properties.
- TON (Timer On-Delay)
- A standard IEC 61131-3 function block that implements a time-delay timer whose state spans multiple PLC scan cycles, making it difficult to verify with static or formal methods.
- Model Context Protocol (MCP)
- A protocol used in SemaPLC's tool server to expose PLC tooling via a stdio interface, allowing the agent to invoke compilation, verification, and runtime checks programmatically.
- Scan cycle
- The repeating execution loop of a PLC in which all inputs are read, the control program is executed, and all outputs are written, typically occurring many times per second.
- LLM4PLC
- A published baseline system for LLM-based PLC code generation used as a comparison point in SemaPLC's project-context track experiments.
- AutoPLC
- A published baseline system for LLM-based PLC code generation used as a comparison point in SemaPLC's project-context track experiments.
- Bare configuration
- An ablation variant of SemaPLC that removes the verification-gated harness, used to isolate the harness's contribution to overall performance.
- Golden trace
- A reference runtime execution trace produced by a correct PLC implementation, used as the ground truth for scoring dynamic behavior of generated code.