Compile by Training: Turning Natural-Language Specifications into Local Neural Functions

Yuntian Deng, Pengyu Nie, Stuart Shieber

Compile by training turns natural-language specifications into reusable, compact neural functions via synthetic supervision.

How can we replace expensive, high-latency remote model calls with lightweight, local neural functions generated automatically from natural-language specifications?

Recurring text-processing tasks are often too fuzzy for rigid code but too frequent to justify the latency and cost of calling a large remote model for every input. The authors propose "compile by training": a system that uses large teacher models to synthesize task-specific examples, which then train a lightweight adapter for a compact, local interpreter. This process produces a versioned, reusable neural program that runs locally without further teacher calls, achieving 83.6% semantic accuracy on tasks where previous fast-compilation methods failed.

Paper Primer

The system treats neural function creation as a software build step. It initializes a lightweight adapter using an amortized compiler, then refines it through gradient descent on synthetic data generated by teacher models, effectively trading a minute of compile-time compute for higher runtime accuracy.

Compile by training significantly outperforms one-shot amortized weight prediction on difficult specifications.

Evaluation on FuzzyBench-Hard, a subset where the baseline fast compiler produced zero exact matches. Mean LLM Exact Match (LEM) score increased from 0.224 to 0.836.

Why is this approach preferred over simply calling a large model at runtime?

Calling a large model for every input incurs repeated network latency, provider costs, and external service dependencies. Compiled functions run locally, allowing them to be cached, versioned, and composed like standard software artifacts.

How does the system handle the latency of generating synthetic training data?

The compiler overlaps teacher synthesis with model training, starting the optimization process as soon as the first batches of examples are available rather than waiting for the entire dataset to be generated.

The Problem of Remote Model Dependency

We expose the gap of costly remote models and propose compile‑by‑training to build reusable neural functions.

Many recurring text tasks are easy to describe yet too fuzzy for hand‑written rules, while invoking a large remote model for each input repeatedly incurs latency, cost, and external‑service dependence.

Hand‑coded rules cannot capture the inherent ambiguity of many text functions, and repeatedly querying a massive remote model is prohibitively expensive for high‑frequency use.

The key trade‑off is achieving high model capability through a one‑time compilation cost, thereby reducing ongoing operational latency and expense.

System Overview

The system compiles a natural‑language spec into a reusable local neural function.

To avoid repeatedly invoking large remote models, the system turns a natural‑language specification into a compact, locally runnable neural function.

A Local Neural Function is a tiny program that contains a frozen language‑model interpreter together with a lightweight adapter and prompt that together implement the behavior described by the original specification.

Compile receives $s$ and queries the teacher model to generate synthetic (email, subject) pairs.

Training fits a tiny LoRA adapter and a prompt so that the frozen interpreter maps the email text to the correct subject.

The resulting artifact $ps$ (a .paw file) is saved locally.

Run loads $ps$, concatenates the prompt with $x$, and feeds the combined text to the frozen model.

The model outputs “Meeting tomorrow”, which is returned as $\hat{y}$.

This toy example shows that after compilation the heavy teacher model is only used once; subsequent calls are handled entirely by the lightweight local program.

How does this differ from fine‑tuning the whole language model on the same task?

Fine‑tuning updates all parameters of the large model, requiring expensive GPU time and storage for each new function. In contrast, Compile by Training keeps the base model frozen and learns only a tiny adapter plus a prompt, so the resulting $ps$ is orders of magnitude smaller and can be deployed locally without re‑training.

**Figure 1.** One specification, one compile job, one local function. The running email-triage example is entered in the public playground, compiled asynchronously while the user continues browsing, and then invoked through the local SDK. The teacher and GPU are compile-time dependencies; repeated function calls use the packaged program.

The Compile by Training Mechanism

Turn a natural‑language specification into a compact neural program via synthetic examples.

Remote large models incur latency and cost for recurring tasks; the method sidesteps this by compiling a specification into a tiny, locally runnable program.

The trick is to treat the specification as a “training recipe”: synthesize a tiny dataset that embodies the desired behavior, then train only a lightweight adapter to reproduce it.

Initialize a rank‑2 LoRA adapter with random weights.

Feed each input pair to the frozen interpreter, compute the loss $L(\theta_s)$, and take a gradient step.

After three gradient updates the adapter predicts 3, 9, and 7 exactly for the three inputs.

Package the final adapter weights and the prompt “add two numbers” into a program $p_s$.

The adapter learns only the arithmetic mapping; the heavy language model never changes, so the program remains tiny and fast.

Instead of hand‑labeling data, the system asks a powerful teacher model to generate examples that obey the user’s specification.

Why not simply use the teacher’s outputs as the final program?

Teacher outputs are tied to the teacher model’s weights and cannot be executed without that model. Training a LoRA adapter transfers the behavior to a frozen, lightweight interpreter, enabling fast, self‑contained execution.

A LoRA adapter injects a low‑rank update into a frozen language model, altering its behavior with only a few extra parameters.

What makes LoRA adapters so lightweight compared to full fine‑tuning?

LoRA restricts learning to two small matrices $A$ and $B$ of rank $r$, adding only $2 r d$ parameters (with $d$ the model dimension). Full fine‑tuning would modify all $d^2$ weights, increasing storage and compute dramatically.

Send the specification to the teacher APIs; receive a validated synthetic dataset $D_s$.

Initialize the LoRA adapter with parameters $\theta^{(0)}$ and scaffold $r_s$.

Optimize $\theta_s$ by minimizing $L(\theta_s)$ on $D_s$ (e.g., 100‑step cosine schedule).

After training, bundle $\theta_s$, $r_s$, the original specification, and interpreter metadata into program $p_s$.

Deploy $p_s$; at run time the interpreter loads the adapter and scaffold once, then processes new inputs directly.

Using the compiled program via PAW.

**Figure 2.** **Compile by training.** At compile time, teacher models turn a specification into validated supervision, which specializes a LoRA adapter for a compact frozen interpreter. The compiler packages the adapter and run-time scaffold into a reusable program, which handles new inputs at run time without teacher calls.

Interactive Compilation

Enabling minute‑scale interactive compilation by overlapping synthesis, coordinating workers, and persisting jobs.

Interactive compilation must shrink the end‑to‑end latency and let users keep working while a program is being built.

Instead of waiting for every teacher‑generated example before training, the system starts training as soon as the first batch is ready—like a kitchen that begins cooking the first dish while the pantry is still being stocked.

Step 1: Teacher requests for E₁ and E₂ are sent; they return after 0.8 s.

Step 2: As soon as E₁ and E₂ arrive, the trainer forms Batch 1 and performs one update step.

Step 3: While Batch 1 is training, the teacher continues producing E₃ and E₄ (arriving at 1.5 s).

Step 4: When the trainer finishes Batch 1, it immediately builds Batch 2 from E₃ and E₄ without waiting for E₅–E₈.

Step 5: This pattern repeats until all 8 examples are consumed, keeping the GPU busy throughout.

Overlapping removes the idle gap that would exist if training waited for the full set of 8 examples before starting.

How does this streaming approach differ from simply pre‑fetching all teacher examples before training?

Pre‑fetching still requires the full set of examples to be available before the first training step, leaving the GPU idle while the slowest teacher response is awaited. Streaming launches training as soon as the first batch is ready and only pauses when it catches up, so the GPU remains active for the majority of the compile.

**Figure 3.** Deployed finetune service. The API submits a job to a shared queue, which assigns it to an available GPU. Workers reuse cached teacher outputs, call teachers for missing examples, execute the compiler in Figure 2, and report progress. Completed artifacts are stored centrally and returned through the same job record. Solid arrows are requests or artifacts; dashed arrows are status and cache traffic.

The user sees only a persistent job indicator; they can navigate away, start new tasks, and later retrieve the compiled program from the same job record.

Evaluation and Benchmarking

Compile‑by‑training boosts LEM by 0.612 on FuzzyBench‑Hard.

The paper replaces slow remote models with a small local neural adapter trained on synthetic examples, then evaluates the resulting system.

A curated slice of the FuzzyBench suite where the fast PAW compiler fails to produce exact matches, exposing the toughest correctness cases.

Why is FuzzyBench‑Hard harder than the regular FuzzyBench benchmark?

Because the fast PAW compiler produces zero exact matches on these specifications, so any correct prediction must rely on deeper semantic understanding rather than surface‑level token alignment.

A compiler that caches teacher‑generated outputs and reuses them across multiple compilations, thereby spreading the synthesis cost over many calls.

How does the Amortized Compiler differ from the fast PAW compiler?

Both aim for low latency, but the Amortized Compiler relies on cached teacher outputs while the PAW fast compiler still invokes the teacher for each compile, leading to higher latency but no reuse benefits.

Compile‑by‑training improves mean LEM on FuzzyBench‑Hard by 0.612 absolute.

Mean LEM rises from 0.224 (fast amortized compiler) to 0.836 (compile‑by‑training).

**Table 1.** Mean LEM across controlled teacher-mixture and data-scaling sweeps.

**Figure 4.** Compile-time optimization buys a large correctness gain. Mean LEM on FuzzyBench-Hard for PAW’s fast amortized compiler and our compile-by-training compiler.

Deployment and Applications

We showcase deployed compiled functions and their real‑world uses.

Paw‑helper illustrates how many compiled functions can be orchestrated inside a live website assistant, combining fuzzy decisions with deterministic code.

**Figure 5.** A website helper as a program tree. (a) A page-aware question is routed through finetuned classifiers, answerers, selectors, and validators alongside deterministic links and retrieval. The highlighted CS486 path combines the course answer with a Piazza search and merge. (b) The live course helper returns the result.

Avatar Director turns natural‑language commands into a small action DSL that the browser validates and animates, letting users script 3‑D motions without writing code.

**Figure 6.** From language to executable behavior. A finetuned PAW program maps a natural-language command to an action program; deterministic browser code parses and validates the DSL before rendering the resulting motion. In the live web demo, DSL interpretation and 3D rendering run in the browser.

The PAW framework also powers a bidirectional English↔Claudish translator, preserving meaning while adopting or stripping the distinctive Claude‑style rhetoric.

**Figure 7.** **English to Claudish.** The live interface applies the English-to-Claudish PAW program to a plain-English sentence. Switching direction invokes the separately finetuned Claudish-to-English program.

The PAW translation service handled over 100 k requests in two weeks, proving its scalability.

Between August 22 and September 2 2026, the live demo completed 100,747 successful translation requests.

Conclusion

We add a compile‑by‑training option that trades a minute of compilation for higher accuracy.

Compile by Training turns a user‑provided function description into a neural program by first generating synthetic examples with a large language model and then fine‑tuning a LoRA adapter for a compact interpreter. This adds a new point to PAW’s speed–accuracy trade‑off: the fast compiler produces a LoRA adapter in a single forward pass (seconds), whereas Compile by Training spends about a minute to achieve higher accuracy. We deployed the resulting programs in the public PAW demo, enabling a multi‑site website helper, a language‑controlled 3D avatar, and a bidirectional English–Claudish translator.

Acknowledgments

We acknowledge funding sources and cite related works that informed this study.

Yuntian Deng acknowledges support from NSERC (RGPIN-2024-05178) and a Google Research Award for Machine Learning Research and Education with TPUs. Pengyu Nie acknowledges support from NSERC (RGPIN-2024-04909). The authors also thank the following works for inspiration: Prompt2Model (Viswanathan et al., 2023), Self‑instruct (Wang et al., 2023), Qwen3 technical report (Yang et al., 2025), Adaptive budget allocation for parameter‑efficient fine‑tuning (Zhang et al., 2023), Program‑as‑weights (Zhang et al., 2026), and LoRA land (Zhao et al., 2024).

Experimental Details and Validation

Key configuration details and evaluation metrics for reproducibility.

This appendix provides the concrete settings and evaluation details needed to reproduce the experiments. We generate {n} diverse (input, output) example pairs following the specification, then list the configuration fields—interpreter, teachers, data, optimization, adapter, and execution—used for the public finetuned standard setup (Table 2). The LLM Exact Match (LEM) metric measures binary, specification‑grounded semantic correctness, with the GPT‑5.5 grader tolerating formatting variations while rejecting value, order, structure, or logic errors.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers