Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng
Program-as-Weights (PAW) compiles fuzzy natural-language tasks into compact, locally executable neural adapters.
How can we turn natural-language specifications into portable, lightweight neural "programs" that execute locally without needing a full LLM?
Developers often outsource "fuzzy" tasks—like parsing messy logs or ranking search results—to large language model APIs because they resist rigid, rule-based programming. This reliance creates fragile, costly, and non-reproducible software dependencies. PAW solves this by reframing the foundation model as a tool builder: a 4B-parameter compiler translates a natural-language specification into a small, reusable neural adapter (a LoRA). A frozen, lightweight interpreter then executes this adapter locally as a standard function. This approach allows a 0.6B-parameter model to outperform a 32B-parameter model on fuzzy tasks while using 50x less memory and running entirely offline.
Paper Primer
PAW programs are hybrid artifacts consisting of a discrete "pseudo-program" (a paraphrased task description with examples) and a continuous LoRA adapter. The compiler acts as a neural translator, mapping the user's intent into these two components: the discrete part shields the interpreter from ambiguity, while the continuous part provides the fine-grained behavioral control needed for execution.
The system functions like a traditional compiler-runtime stack: the compiler is the heavy-lifting "build" step, and the interpreter is the "runtime" that executes the compiled binary. The LoRA mapper is the core mechanism: it pools hidden states from the compiler, passes them through a shallow MLP, and projects them into mixing coefficients for a set of shared LoRA bases.
PAW achieves superior performance on fuzzy tasks compared to direct prompting of much larger models.
On the FuzzyBench dataset, a 0.6B-parameter interpreter running PAW programs achieves 73.78% exact match accuracy. This outperforms direct prompting of a 32B-parameter model (68.70%) while reducing inference memory usage by approximately 50x.
The compiler-generated adapter is robust to noisy, real-world specifications.
When tested against heavy-typo specifications, the PAW pipeline degrades significantly less than a system that feeds raw specifications directly to the interpreter. The pseudo-program acts as a denoiser, narrowing the performance gap between clean and noisy inputs to just 4.5 percentage points.
Why use a hybrid (discrete + continuous) program instead of just a continuous adapter?
The discrete pseudo-program provides a clear, self-contained instruction set that protects the small interpreter from typos and ambiguity in the original user specification, while the continuous LoRA adapter supplies the precise behavioral control that text alone cannot capture.
What is the scope of this system — can it handle tasks beyond text processing?
Yes; by swapping the text-only compiler for a vision-language model, the same frozen 0.6B interpreter can execute PAW programs for image-conditioned tasks like diagram understanding and structured data extraction.
The Program-as-Weights Paradigm
We expose the gap: fuzzy tasks defy rule‑based code and current LLM APIs are costly.
Many real‑world functions—such as alerting on important log lines, repairing malformed JSON, or ranking search results by intent—cannot be expressed cleanly with deterministic rules. Developers therefore outsource these “fuzzy” tasks to large language‑model APIs, paying high inference costs, sacrificing reproducibility, and losing offline locality.
A fuzzy function is a behavior that humans can describe with natural language, examples, or constraints, but that resists precise symbolic coding because the specification is inherently ambiguous or noisy.
**Figure 1.** Overview of the Program-as-Weights paradigm. Top: compile once in the cloud. A natural-language description of a fuzzy function (here, “classify if this is urgent”) is fed to a neural compiler, which produces a neural program. Bottom: run locally. A small frozen neural interpreter loads the compiled program and runs the user’s input (“Need your signature by EOD!”) to produce the output (“urgent”). The compiled program is a single file that can be cached, version-controlled, and called offline like any other library function.
The key insight is shifting from per‑input prompting of massive LLMs to compiling reusable, portable weight files that run locally.
The PAW Architecture
How a lightweight LoRA adapter is generated from text and run on a frozen model.
The PAW pipeline must generate a tiny, task‑specific adapter without retraining the massive language model, otherwise deployment costs explode.
PAW treats a natural‑language specification as a program that is compiled into a lightweight set of neural weights, which a frozen interpreter can execute directly.
How does PAW differ from simply prompting a large model with a task description?
Prompting leaves the model weights unchanged; PAW injects a dedicated LoRA adapter that directly modifies the attention and feed‑forward matrices, giving the interpreter a permanent, low‑overhead bias toward the task.
The compiler is a separate language model that reads the specification and produces the hidden‑state tensor that later becomes the adapter.
Why not use the same model for both compiling and interpreting?
Separating them lets the interpreter remain frozen (no risk of drift) while the compiler can be updated or swapped without redeploying the large model, preserving device‑side stability.
Prompt the pseudo compiler $C_p$ with the specification $s$ to obtain the discrete pseudo‑program $p_{\text{discrete}}$.
Feed $[s\;|\;p_{\text{discrete}}\;|\;\text{EOS}\;|\;\tau_{1:T}]$ to the LoRA compiler $C_L$ and collect hidden states $H$ from $L$ depth‑aligned layers.
Mean‑pool $H$ across layers and prefix positions to get $\bar{h}$.
Pass $\bar{h}$ through the LoRA mapper to produce mixing coefficients $\alpha^{A},\alpha^{B}$ and assemble LoRA matrices $A^{\text{ex}}_{l,m}, B^{\text{ex}}_{l,m}$ for every interpreter layer $l$ and module type $m$.
Attach the resulting LoRA adapter to the frozen interpreter, prepend $p_{\text{discrete}}$ to user input $x$, and run autoregressive generation to obtain $\hat{y}$.
The mapper collapses the compiler’s high‑dimensional hidden states into a small set of basis‑weighted LoRA matrices that can be hot‑swapped into the interpreter.
How does this LoRA mapper differ from standard LoRA fine‑tuning?
Standard LoRA learns $A$ and $B$ directly from gradient descent on downstream data. Here the mapper predicts $A$ and $B$ from the compiler’s hidden states, enabling zero‑gradient, on‑the‑fly generation of adapters.
The MLP $\phi$ maps $\bar{h}$ to mixing coefficients $\alpha^{A}=[0.8,\,0.2]$, $\alpha^{B}=[0.3,\,0.7]$.
Compute $A^{\text{ex}} = 0.8\,A^{(m)}_{1} + 0.2\,A^{(m)}_{2} = \begin{bmatrix}0.8 & 0.2\\0.2 & 0.8\end{bmatrix}$.
Compute $B^{\text{ex}} = 0.3\,B^{(m)}_{1} + 0.7\,B^{(m)}_{2}$ (analogous calculation).
Attach $A^{\text{ex}}$ and $B^{\text{ex}}$ to the corresponding attention module of the interpreter.
The mapper turns a tiny 2‑dimensional summary into full LoRA matrices, showing how a single vector can control many parameters.
The interpreter is a frozen language model that treats the generated LoRA adapter as part of its own parameters and runs the user’s input through this augmented network.
Why is the interpreter kept frozen instead of fine‑tuned per task?
Freezing guarantees that the base language capabilities (grammar, world knowledge) remain stable, while the lightweight LoRA adapter provides task‑specific steering at negligible memory and compute cost.
**Figure 2.** Text-to-LoRA instantiation of PAW (Section 3.2). *Left.* The trained LoRA compiler reads the function specification, the pseudo-program produced by an off-the-shelf prompted pseudo compiler $C_p$ (not depicted), and a fixed sequence of learned prefix tokens; it emits prefix-position hidden states $H$. *Middle.* The LoRA mapper mean-pools $H$, passes it through an MLP, and projects into mixing coefficients that compose LoRA matrices ($A^{\text{ex}}, B^{\text{ex}}$) over shared learnable bases (eq. (3)). *Right.* The frozen interpreter ingests $p_{\text{discrete}}$ prepended to the user input $x$, with the LoRA hot-attached, and generates the output autoregressively. The same pipeline holds for the prefix-tuning precursor (Section 3.3, with architecture in Figure 18); only the mapping from compiler hidden states to PEFT module changes (LoRA $arrow$ KV-cache mapper).
Training and FuzzyBench-10M
Training uses a frozen interpreter and a PEFT compiler that learns LoRA adapters from fuzzy specifications.
Training a full language model for each fuzzy function is prohibitively expensive; the method instead trains only a tiny PEFT compiler while keeping the interpreter frozen.
Instead of fine‑tuning the entire model, we learn a small LoRA adapter that translates a fuzzy specification into weight updates for a frozen interpreter.
The interpreter receives the adapter $a=1.0$ and computes the token probability $P_{\phi}(y\mid a, x)=0.8$.
The negative log‑likelihood for this token is $-\log 0.8 \approx 0.22$.
Since the batch contains only this example, the loss $L(\theta)$ equals 0.22.
Back‑propagation scales the gradient by the derivative of the loss w.r.t. $a$, updating the mapper weight $w$ while leaving $\phi$ untouched.
The adapter’s magnitude directly controls the interpreter’s confidence; a larger $a$ would increase the probability of the correct token, reducing the loss.
**Table 1.** Two PEFT instantiations. Both methods outperform the prompting
Public benchmarks for “compile a fuzzy function” do not exist, so we built FuzzyBench‑10M, a 10‑million‑example corpus covering realistic developer tasks.
A massive collection of (specification, input, output) triples that span seven high‑level task families, providing the training signal for PAW compilers.
**Figure 3.** **FuzzyBench-10M task-family distribution.** 29 incremental thematic versions are mapped to 7 high-level families. “Core text processing & NLP” is the largest family because the v1 base layer (2.5M examples; 277 base categories) covers parsing, classification, NER, coreference, and sentiment; the remaining 7.5M examples spread across the other six families.
To probe robustness we create eight noise‑perturbed test variants (typos, grammar errors, ambiguity, formatting drift, combined noise, terse phrasing, casual phrasing, paraphrase) at three intensity levels each.
**Figure 18.** **Prefix-tuning precursor architecture (Section 3.3).** (a) *Compile*. The user describes a fuzzy function (e.g., “extract the final answer”); the trained prefix compiler reads the description plus a handful of representative I/O examples and produces a per-example KV prefix — the “neural binary” that constitutes the compiled program. (b) *Interpret*. A small frozen interpreter loads the compiled KV prefix into its attention cache at every layer and processes user inputs locally as a callable function, in the manner of standard prefix-tuning [Li and Liang, 2021]. This is the prefix-tuning instantiation of the same compiler–interpreter abstraction depicted in Figure 2; only the mapping from compiler hidden states to per-example PEFT module differs (KV cache here, LoRA in Section 3.2). *Note*: our prefix-tuning precursor’s exact training-time input format follows the same $[s \mid p_{\text{discrete}} \mid \text{EOS} \mid \tau_{1:T}]$ structure as the LoRA instantiation.
Performance and Baselines
PAW outperforms baselines, scales across interpreters, and cuts inference memory dramatically.
PAW with a 0.6 B interpreter achieves 73.78% exact‑match accuracy on FuzzyBench, surpassing 32 B direct prompting (68.70%) while using roughly 50× less inference memory.
Exact‑match measured on the verified test set; inference memory ≈1.2 GB (bf16) versus ≈60 GB for the 32 B model.
Direct prompting feeds the task description straight into a large language model, relying on the model’s internal knowledge without any intermediate adapter.
**Table 3.** Performance on six image-conditioned tasks: three CoSyn-400K diagram-understanding tasks (Chemical, Circuit, Music), the structured-output Im2LaTeX-100K and Im2SMILES-20K tasks, and the open-ended visual question answering TextVQA task.
Ablations and Robustness
We isolate each component to see how its removal impacts accuracy and robustness.
This section systematically removes or alters each design choice to quantify its contribution to overall performance and noise tolerance.
**Table 4.** Architectural variants of the LoRA mapper. “More expressive” design choices that we expected to help all underperformed the simple default.
More expressive LoRA‑mapper designs reduce accuracy by up to $6.3$ percentage points compared to the simple default.
Per‑position aggregation scores $0.5598$, a drop of $0.0625$ from the default $0.6223$ (Table 4).
**Table.** Comparison of interpreter input formats on accuracy under clean and heavy typo conditions.
PAW exceeds full fine‑tuning by $15.4$ percentage points.
PAW achieves $0.7378$ versus $0.5840$ for full fine‑tuning (Table 5).
PAW exceeds the strongest fixed LoRA (rank $64$) by $21.7$ percentage points.
PAW $0.7378$ versus fixed LoRA $r\!=\!64$ at $0.5210$ (Table 5).
**Table.** Quantization findings on the 0.6B Qwen3 interpreter, validated at a 4096-example test subset: a 4-bit base (`Q4_K_M`, ~484 MB) plus a `Q4_0` LoRA adapter (~23 MB per program) loses only 1.3 points relative to bf16, and a `Q6_K` base plus `Q4_0` adapter is statistically indistinguishable from bf16.
PAW’s accuracy drops only $1.9$ points under the worst‑case combined noise.
Clean accuracy $0.6692$ versus $0.6499$ in the “all‑noise” column (Table 6).
The authors hypothesize that this robustness stems from the discrete pseudo‑program that denoises specifications before they reach the interpreter.
**Table 7.** The pseudo-program protects the interpreter from noisy specifications. On heavy-typo specifications, feeding raw spec is 4.5 points worse than feeding the pseudo-program to the interpreter.
Pseudo‑program improves heavy‑typo accuracy by $4.5$ points.
Heavy‑typo accuracy $0.6108$ with pseudo‑program versus $0.5662$ with raw spec (Table 7).
Training a variant that bypasses the pseudo‑program confirms the hypothesis: the gap widens to $4.5$ points on noisy specs, while the clean‑input advantage remains modest.
Practical Local Execution
PAW runs offline with tiny size and negligible accuracy loss.
PAW ships a tiny developer interface: a single file can be downloaded, cached, and invoked through a Python or JavaScript API, eliminating any external calls after the first download.
Quantizing the 0.6B Qwen3 interpreter to a `Q6_K` base with a `Q4_0` LoRA adapter yields no measurable accuracy loss compared to bf16.
Table 8 shows 0.6575 accuracy for `Q6_K` + `Q4_0` versus 0.6580 for bf16, a difference of 0.0005.
**Figure 4.** **Developer interface.** *Left:* the compiler translates a natural-language specification into a neural program. *Right:* the interpreter loads this program and exposes it as a local function. **Listing 1: Compile a fuzzy function** ```python import programasweights as paw # (1) Describe the fuzzy function spec = """Classify if this email requires immediate attention. """.strip() # (2) Compile to a neural program program = paw.compile(spec, \ slug="email-triage") ``` **Listing 2: Run the compiled program locally** ```python import programasweights as paw # (3) Load the compiled program fn = paw.function("email-triage") # (4) Call it like a function print(fn("Thesis defense moved " "to 3pm; need your " "signature today.")) # -> "immediate" ```
Beyond the core quantization results, we evaluated PAW on five real‑world scenarios, each confirming that the LLM is only needed at compile time.
Event‑driven log monitoring replaces a naive wait‑loop, intent‑based site navigation provides instant natural‑language search, semantic search reranking adds fuzzy matching without per‑request LLM calls, a 10‑function pipeline reaches 93 % on TOOLCALL‑15, and the multilingual Alien‑Taboo game runs locally on a tiny server.
Related Work
We situate PAW among hypernetworks, PEFT adapters, synthetic data, distillation, and neural‑program trends.
Hypernetworks were introduced to synthesize a target network’s weights from a compact embedding, later adapted for continual learning and multi‑task NLP. The text‑conditioned subfamily directly maps natural‑language task descriptions to PEFT parameters in a single forward pass (e.g., Hyper, HINT, HyperTuning, Text‑to‑LoRA, Generative Adapter, HyperSteer, Gist, MEND).
Works most closely aligned with PAW include SHINE, HypeLoRA, Doc‑to‑LoRA, and Latent Context Compilation, all of which emit LoRA adapters from language inputs in one forward pass. LoRAHub further demonstrates shared‑basis adapters across tasks, a design parallel to PAW’s shared‑basis mapper.
PAW distinguishes itself by (a) emitting a hybrid discrete pseudo‑program plus continuous PEFT program, (b) training on programmer‑style fuzzy‑function specifications from FuzzyBench‑10M, and (c) targeting a developer‑facing API where the compiled program is a versioned, distributable artifact.
PEFT adapters such as adapters, prefix‑tuning, prompt tuning, LoRA, AdaLoRA, DoRA, and QLoRA are well‑studied building blocks. Unlike these methods, PAW’s compiler produces the PEFT module on‑the‑fly from a textual specification rather than learning it via gradient descent for each task.
T‑Few argues that PEFT can outperform in‑context learning at lower deployment cost; PAW shares the low‑cost motivation but differs by generating PEFT from a description instead of learning it per task.
FuzzyBench‑10M follows the LLM‑generated instruction paradigm of Self‑Instruct and Unnatural Instructions, yet it is task‑class‑specific (29 thematic versions) and includes a verified test split where two strong LLMs must agree on outputs.
Model‑distillation works such as ALCHEmist, Binder, and Distilling Step‑by‑Step also aim to replace LLM inference with smaller models, but they either distill logic into textual programs or fine‑tune per task, whereas PAW compiles directly into neural weights.
The broader neural‑program literature has long explored embedding executable logic in networks; recent trends push for locally executed small models as API replacements. PAW’s novelty lies in using a compiler‑generated weight program to configure a fixed interpreter, sidestepping per‑task fine‑tuning.
Practical on‑device deployment benefits from post‑training quantization (GPTQ, AWQ, QLoRA) and lightweight runtimes (llama.cpp, wllama); PAW directly adopts these techniques to enable efficient local execution of compiled programs.
Appendices
Appendix details the web UI, prompts, dataset versions, and component analyses for PAW.
We provide a hosted web interface that accepts a fuzzy specification, compiles it on a GPU‑backed server, lets the user test the result interactively, and finally exports the compiled program as either a serialized weight file or a program identifier for the Python API. The compilation step runs remotely so users need no local GPU; the downloaded program runs entirely offline on the frozen interpreter.
**Figure 5.** **Step 1: Compile a program from natural language.** The user specifies a fuzzy function in natural language. Image inputs are also supported.
**Figure 6.** **Step 2: Interactively test the compiled program.** Users can provide test inputs and inspect the corresponding outputs, enabling rapid validation and refinement before download.
**Figure 7.** Step 3: Execute the program locally via Python. Once compiled, the program can be loaded and invoked through a simple Python API; subsequent execution requires no internet access.
FuzzyBench construction uses two families of prompts. Half of the specifications are generated without exemplar I/O pairs, and the other half include examples; this mixture yields a broader variety of tasks. Additional prompts generate input–output examples conditioned on a given specification.
Compiler and interpreter prompts come in two styles. The “minimal” style wraps a specification in a simple [SPEC]…[`END_SPEC`] [`PSEUDO_PROGRAM`] block, while the “examples” style adds task‑description‑plus‑examples pseudo‑programs. The minimal style is used by the trained PAW compiler at inference time; the examples style is used by the off‑the‑shelf reference compiler during training rollouts.
**Figure 12.** User prompt for generating input/output examples given a specification.
**Figure 13.** Compiler prompt, minimal style. Used by the trained PAW compiler at inference.
**Figure 14.** Compiler prompt, examples style. Used by the off-the-shelf reference compiler (Qwen3-4B-Instruct-2507) to generate the rollouts used during training.
Appendix D collects the materials supporting the multimodal experiments in Table 3, including the compiler and interpreter prompts and a component‑decomposition ablation of the prefix‑tuning precursor on six image tasks.
Section D.1 isolates the contribution of the discrete pseudo‑program and the continuous KV‑cache. “Discrete pseudo only” emits a pseudo‑program without any continuous PEFT; “Continuous KV‑cache only” injects a per‑example prefix‑tuning cache but feeds the raw spec to the interpreter; the full “PAW prefix‑tuning” combines both.
**Table 9.** **Image-task component decomposition (prefix-tuning precursor).** EM/accuracy on the six image tasks. Adding the discrete pseudo-program helps on classification-style tasks (Circuit, Chemical, TextVQA) but hurts on long-form structured generation (Im2LaTeX, Im2SMILES), where “Continuous KV-cache only” is the strongest variant.
The cross‑task pattern is consistent: for tasks that output short phrases (Circuit, Chemical, Music, TextVQA) the discrete pseudo‑program adds 5–40 EM points; for long‑form markup tasks (Im2SMILES, Im2LaTeX) removing the pseudo‑program recovers 6–8 EM points.
Figure 18 (not shown here) visualizes the prefix‑tuning precursor architecture: the compiler emits a compact KV prefix that the frozen interpreter loads as a callable function.
FuzzyBench‑10M was built incrementally over 29 thematic versions. Each version adds 2 000 new validation and 2 000 new test specifications, expanding the dataset from 2.5 M to 10 M examples.
**Table 10.** FuzzyBench-10M version timeline. Each version is incremental on top of the previous one, with 2,000 new validation and 2,000 new test specifications added per version.
Training configuration mirrors the two‑stage compile/interpret rhythm. The compile stage uses the PAW compiler (LoRA or prefix‑tuning) to produce per‑example adapters; the interpret stage loads these adapters into the frozen Qwen3 0.6B or Qwen3.5 0.8B interpreter.
**Figure 17.** Interpreter prompt for image-conditioned specifications.
Overall, the appendix supplies all auxiliary materials—prompts, UI screenshots, tables, and architectural diagrams—required to reproduce the PAW system and its multimodal extensions.