AutoIndex: Learning Representation Programs for Retrieval
Sam O'Nuallain, Nithya Rajkumar, Ramya Narayanasamy, Hanna Jiang, Shreyas Chaudhari, Andrew Drozdov
AutoIndex optimizes document representation programs via iterative, agentic failure analysis to improve retrieval quality.
How can we automatically optimize document preprocessing (chunking, filtering, metadata) to improve retrieval performance without retraining the retriever itself?
Retrieval systems often treat document representation—how text is chunked, normalized, and structured—as a static preprocessing step, leaving significant performance gains on the table. AutoIndex treats representation as an optimization target, using an iterative loop where an analysis agent diagnoses retrieval failures and a code agent synthesizes executable programs to transform documents before indexing. On the Complex Retrieval Unified Multi-task Benchmark (CRUMB), this approach improves average Recall@100 by +8.4% and nDCG@10 by +8.3% over a fixed BM25 baseline without requiring retriever fine-tuning.
Paper Primer
The core move is black-box code synthesis: the system treats the document-to-index mapping as a program $\theta$ that is iteratively refined. The analysis agent identifies specific failure modes—such as vocabulary mismatch or noise from LaTeX markup—and the code agent proposes targeted transformations, like field reweighting or surgical cleaning, to resolve them.
Learned representation programs consistently outperform static, domain-agnostic chunking.
AutoIndex achieved average gains of +8.4% in Recall@100 and +8.3% in nDCG@10 across 8 heterogeneous retrieval tasks on the CRUMB benchmark. Largest gains reached +30.5% in Recall@100 and +43.6% in nDCG@10.
The system relies on a "search history" to stabilize trajectories, preventing the code agent from repeating destructive edits or over-fitting to idiosyncratic validation queries. This history acts as a soft prior, ensuring that new candidate programs build upon successful transformations rather than oscillating between competing strategies.
Why is this approach better than simply fine-tuning the retriever?
Fine-tuning the retriever requires updating weights or embeddings, which is computationally expensive and often requires specialized training data. AutoIndex optimizes the input representation instead, allowing it to improve retrieval performance while keeping the underlying retriever and indexing backend entirely fixed.
Does this method require an LLM call at query time?
No. The learned representation program is applied once at indexing time. Once the index is built, the retrieval process uses standard, efficient lookups with no further LLM involvement.
Document representation should be treated as an explicit, learnable optimization target rather than a fixed infrastructure choice. Practitioners can use this framework to adapt existing retrieval pipelines to specific corpora without the overhead of model training.
Introduction and Motivation
AutoIndex learns executable programs to optimize document representation for retrieval.
Retrieval‑augmented generation (RAG) systems depend on a retriever that searches a pre‑indexed corpus, yet the way documents are turned into indexable units is usually a static, hand‑crafted preprocessing step. This static representation limits how well the retriever can match queries because it cannot adapt to the downstream task’s nuances. AutoIndex addresses this gap by searching over executable representation programs that transform raw documents into retrieval‑aware units, keeping the retriever itself fixed.
The CRUMB Benchmark aggregates eight heterogeneous retrieval tasks to measure how well a system generalizes across domains.
BM25 is a classic probabilistic ranking function that scores documents based on term frequency, inverse document frequency, and document length.
The key shift is to treat document representation itself as the optimization target rather than continually tuning the retriever.
Prior Approaches to Retrieval Tuning
We position AutoIndex among related work on dynamic chunking, index enrichment, and program optimization.
Retrieval pipelines often rely on handcrafted preprocessing—chunk size, overlap, title inclusion, and metadata templates—to boost downstream LLM performance, but exhaustively exploring these settings is costly because each variant requires rebuilding the index. AutoIndex sidesteps this bottleneck by searching a broader space of executable programs that can chunk, enrich, or reorganize documents, guided by failure analysis and metric‑driven signals rather than manual grid searches. This contrasts with prior dynamic‑chunking approaches that either encode whole documents before pooling or invoke an LLM per document.
Index‑enrichment techniques improve retrieval by augmenting documents prior to indexing, e.g., Doc2Query expands texts with predicted queries but can introduce hallucinated content that inflates the index. Recent LLM‑based methods add summaries, canonicalizations, extracted facts, rationales, or learned rewrites, while approaches like Document Optimization and RL‑Index fine‑tune models to rewrite documents using retrieval feedback. AutoIndex differs by keeping the retriever fixed and searching over executable representation programs instead of learning a single transformation model.
AutoIndex also relates to LLM‑driven program‑optimization research, where models propose executable artifacts that are refined via feedback loops. Iterative reasoning frameworks such as ReAct, RLMs, and Parallel Thinking demonstrate that structured tool use and candidate trajectory evaluation can improve robustness, and RAG optimization systems search over pipeline components like retrievers and rerankers. AutoIndex applies verifiable program search to the pre‑indexing representation program, targeting a different object than typical RAG pipelines.
The AutoIndex Framework
AutoIndex iteratively refines executable document programs to boost fixed retriever performance.
Static indexing leaves retrieval quality hostage to the original document format, so even a strong BM25 retriever can miss relevant evidence. AutoIndex tackles this by letting the system rewrite documents into a form the retriever can actually exploit.
A script that reads a raw document and emits the chunks the retriever will actually see.
Running $\theta$ on A produces one chunk: “is a Pythagorean identity.” (the LaTeX token is dropped).
Running $\theta$ on B produces one chunk unchanged: “The price of a laptop is \$999.”
The index now contains two chunks, each still linked to its source document.
A query “Pythagorean theorem” retrieves the chunk from A because BM25 matches “Pythagorean”.
A query “laptop price” retrieves the chunk from B unchanged.
By stripping LaTeX, the program eliminates noisy tokens that would otherwise dominate BM25 scores, while preserving the semantic core needed for retrieval.
Is a “representation program” just a preprocessing pipeline?
No. A preprocessing pipeline is usually a fixed sequence of heuristics, whereas a representation program is an executable script that can contain conditionals, loops, and arbitrary logic, and it is synthesized automatically by AutoIndex.
A tool‑using policy that looks at where the current index fails and writes a short natural‑language diagnosis.
How does the Analysis Agent differ from a standard error‑analysis report?
Standard reports are usually human‑written after the fact and may be vague. The Analysis Agent automatically extracts concrete retrieval failures using the same BM25 scorer the system will later use, and it outputs a structured natural‑language summary that the Code Agent can directly consume.
A code‑generating policy that takes the diagnostic summary and past attempts, then writes new representation scripts.
Why not just ask a generic LLM to write a new program from scratch?
Unconstrained generation often produces syntactically invalid or semantically irrelevant code. By conditioning on the concrete summary s(t) and the concrete history H(t), the Code Agent focuses its search on fixes that directly address observed failures.
Run the current Representation Program $\theta$(t) on the whole corpus to build index C_$\theta$(t).
Use the fixed BM25 retriever R to score validation queries and compute the objective J($\theta$(t)).
Invoke the Analysis Agent to produce a diagnostic summary s(t) from the failures.
Feed s(t) and the accumulated search history H(t) to the Code Agent, which emits N candidate programs.
Evaluate each candidate under J; keep those with $\Delta$`J_i` ≥ $\tau$ in A(t).
If A(t) is non‑empty, select the best candidate (or a merged program) as the new incumbent $\theta$(t+1); otherwise keep $\theta$(t).
Append the evaluated candidates and their outcomes to H(t+1) and repeat for the next iteration.
Iterative AutoIndex optimization.
**Figure 1.** AutoIndex optimization loop. Given a source corpus, validation queries, and a fixed retriever, AutoIndex builds an index from the current representation program, uses an Analysis Agent to diagnose retrieval failures, and uses a Code Agent to synthesize candidate updates. Candidates are evaluated by building new indexes and measuring validation retrieval quality. Updates that improve the objective become the next incumbent.
**Figure 3.** Worked AutoIndex example on a LaTeX-heavy StackExchange article. The Analysis Agent flags repeated inline LaTeX; the Code Agent emits a threshold-gated `strip_latex` step (chunk: 1,847 → 1,102 tokens); top-3 retrieval flips from off-topic math references to relevant pricing articles, improving both Recall@100 and nDCG@10.
Main Results
AutoIndex boosts recall and nDCG across CRUMB tasks, with especially large gains over BM25 baselines.
We evaluate AutoIndex on the CRUMB benchmark, measuring Recall@100 and nDCG@10 across eight retrieval tasks. All experiments use the qwen3‑coder model with search history enabled and five iterations of the learned representation program.
AutoIndex improves Recall@100 across all CRUMB tasks, achieving an average gain of +10.4% over the BM25 full‑document baseline.
Table 1 shows improvements ranging from +2.1% to +361.4% across categories.
**Table 1.** Recall@100 for AutoIndex (qwen3-coder, search history enabled, 5 iterations) on held-out CRUMB test splits, against both the BM25 full-document baseline and CRUMB's passage-corpus baseline. AutoIndex values are mean ± std over 3 seeds. $\Delta$ is relative to the baseline named in each row. Per-split $\Delta$ is computed from unrounded scores. Bold entries in the $\Delta$ vs. Full-Doc row indicate a relative gain of at least 10%. CRUMB does not provide a passage corpus for CodeRetrieval, PaperRetrieval, or TheoremRetrieval, so those columns and the passage-relative AVG are marked "—". There is a counterpart using Claude Sonnet 4.6 in Table 4.
**Table 2.** nDCG@10 for AutoIndex (qwen3-coder, search history enabled, 5 iterations) on held-out CRUMB test splits, against both the BM25 full-document baseline and CRUMB's passage-corpus baseline. Same parameters as Table 1. Bold entries in the $\Delta$ vs. Full-Doc row indicate a relative gain of at least 10%.
AutoIndex consistently improves recall across diverse CRUMB tasks.
Ablations and Dynamics
We dissect how iteration, history, and analysis each drive AutoIndex’s gains.
Full AutoIndex (5‑iteration pipeline) improves Recall@100 on every CRUMB split.
Table 3 shows positive $ΔRecall@100$ for all eight splits, the strongest gain being +30.5 % on SetOpEntity.
Limiting AutoIndex to a single iteration yields gains on only three splits.
Table 3 reports positive $ΔRecall@100$ for ClinicalTrial, CodeRetrieval, and LegalQA; the other five splits regress or stay flat.
Removing search history leaves AutoIndex beneficial on five splits but produces large per‑split swings.
Table 3 shows +5.9 % on CodeRetrieval and +4.7 % on StackExchange, while three splits see modest drops.
Omitting the Analysis Agent shrinks gains, keeping six splits positive with markedly lower $ΔRecall@100$.
Table 3 indicates all six positive splits gain less than 10 %, compared to up to +30.5 % with the full pipeline.
**Figure 2.** Iteration dynamics for three representative CRUMB splits under qwen3-coder. Curves show $\Delta$nDCG@10 relative to the BM25 baseline. Filled markers indicate adopted candidates and open markers indicate rejected candidates. AutoIndex exhibits different search patterns across splits, including early gains, gradual accumulation, and near-threshold rejected improvements.
Extended Results and Statistics
Claude Sonnet 4.6 AutoIndex gains on CRUMB are quantified across key metrics.
AutoIndex with Claude Sonnet 4.6 raises nDCG@10 by +17.2 % relative to the BM25 full‑document baseline on the held‑out CRUMB test splits.
Table 4 reports BM25 nDCG@10 = 52.5 ± 0.04 and AutoIndex nDCG@10 = 5.2 ± 0.5, yielding a $\Delta$ of +17.2 %.
AutoIndex improves Recall@100 on 7 of 8 CRUMB splits, with the strongest lifts observed for TheoremRetrieval, SetOpEntity, and LegalQA. The remaining split shows a negligible change, confirming that the representation program benefits most heterogeneous tasks.
**Figure 4.** $\Delta$Recall@100 relative to the BM25 full-document baseline across CRUMB splits for both Code Agent backbones. Bars show relative improvement; error bars show standard deviation. AVG is the unweighted macro-average and is shown without error bars.
Computational Cost and Latency
Appendix A3 details computational costs and two case studies illustrating AutoIndex’s adaptive representation programs.
AutoIndex learns executable representation programs that turn raw documents into indexable units, sidestepping any need to tune the retriever itself.
Table 5 shows that the Sonnet‑4.6 backbone consumes more tokens and incurs higher LLM‑only wall‑clock time than the Qwen3‑coder backbone.
**Table 5.** Token usage and LLM latency statistics per AutoIndex run for each backbone, averaged over runs. Analysis, code, and completion tokens are the mean per-run token totals attributed to the Analysis Agent, Code Agent, and generated completions respectively. Tokens/call and LLM s/call report per-request averages, with the combined figure pooling analysis and code calls; LLM-only wall/run excludes time spent on preprocessing execution, evaluation, and indexing.
In the TipOfTheTongue study the Code Agent kept a single full‑document chunk but repeated the plot section three times and the cast section twice, boosting term‑frequency for narrative content while preserving surrounding context.
The StackExchange study discovered that heavy LaTeX markup polluted BM25 scores; the Code Agent therefore introduced a narrow stripper that activates only when a document exceeds a LaTeX‑density threshold, improving Recall@100 without harming prose‑only pages.
Design Decisions and Heuristics
Design choices for optimization target, validation split, BM25, dual agents, and analysis tools.
We adopt Recall@100 as the primary optimization target because downstream RAG can recover from imperfect rankings but cannot recover from missing evidence. Recall@10 was considered but left too little headroom for candidates to clear the acceptance threshold on the hardest splits.
Earlier experiments used very small validation pools, which weakened the optimization signal; on splits like PaperRetrieval most queries already had their gold document in the top 100 under a reasonable baseline. A 1:2 validation/evaluation split supplies enough validation queries to surface actionable failure patterns while preserving a representative held‑out set.
We use BM25 as the primary retriever because it is strong, transparent, and its sensitivity to segmentation, normalization, and term reweighting makes it an ideal testbed for learned representation programs. The implementation uses bm25s v0.2.14 with k1 = 1.5, b = 0.75, lucene method, and MaxP aggregation from chunks to source documents, following the CRUMB evaluation protocol. Text is lowercased, tokenized with regex‑based word tokenization, English stopword removal, and no stemming.
We split the system into two agents because a single coding agent given only aggregate Recall@100/nDCG@10 deltas produced low‑signal, generic hypotheses. Separating roles keeps the code agent’s context free of the long document and query excerpts that the analysis agent must consume, which is crucial for weaker backbone models.
The analysis agent is equipped with three read‑only tools: `bm25_retrieve`(query, `top_k`) to fetch top chunks from the current index, `read_file`(`file_path`, `max_chars`) to inspect the original documents behind those chunks, and `grep_search`(pattern, `file_path`, `max_results`) to locate specific terms without overloading its context window. Together they let the agent retrieve, inspect, and reason about preprocessing factors that cause failures or successes.
We restrict the analysis agent to a curated, balanced slice of validation queries. This dramatically reduces search time and prevents the agent from fixating on idiosyncratic queries that do not generalize.
Prompt design for both agents was iteratively refined by inspecting early outputs and correcting recurring failure modes: lack of hypothesis diversity, over‑anchoring on suggested ideas, over‑fitting to domain context, and inefficient tool use that caused context overflow. The final prompts are documented in Appendix A.4.
CRITICAL: BM25 Chunking Tradeoffs Before recommending any chunking or filtering strategy, understand these tradeoffs: 1. **Over-chunking is dangerous.** Splitting every document into many small chunks (e. g. by section or paragraph) balloons the corpus from N chunks to 5-20x N chunks. This has several negative effects: - Short boilerplate-heavy chunks score artificially high due to BM25 length normalization, creating more false positives - IDF values shift because terms now appear across more chunks - The retrieval candidate pool covers fewer unique documents (1000 retrieved chunks might only span 100 docs instead of 1000) 2. **Filtering removes signal too.** Aggressively removing text you think is "noise" can destroy matches where those terms were actually helping. Recommend filtering only when you have concrete evidence that specific content hurts more than it helps. 4. **Think about the full corpus, not just failure cases.** A change that fixes 5 queries but breaks 10 is a net loss. Every recommendation should consider the queries currently succeeding --- preserve them --- alongside the queries currently failing. The code agent is free to **add new chunks, remove chunks, modify chunks, refactor existing helpers, or rewrite the preprocessing from scratch** if it has a principled reason to do so. You do not need to constrain yourself to "additive-only" recommendations --- but flag the regression risk for any destructive change so the code agent can weigh it. CRITICAL: Validation vs. Held-Out Evaluation **The queries you are analyzing are a validation set used to guide hypothesis selection.** Your recommendations will ultimately be judged on a separate, larger held-out evaluation set that you never see during the loop. **Concrete sizes for this run:** {{`VAL_QUERY_COUNT`}} validation queries, {{`EVAL_QUERY_COUNT`}} held-out evaluation queries. A pattern that affects only 1 validation query is a {{`VAL_ONE_QUERY_PCT`}} swing on val --- almost certainly noise that will not generalize. Be especially skeptical when the validation set is small (under ~50 queries): the per-query granularity is large enough that a hypothesis can look like a clean win on val while being random noise on eval. - A fix that perfectly addresses 3-4 specific validation queries but doesn't generalize will hurt overall eval performance - The smaller the number of validation queries a pattern affects, the more skeptical you should be that it generalizes - Treat the validation failures and successes as **samples from a broader distribution**, not as the complete picture of what's broken **Focus on root causes that would affect many queries across the full corpus, not symptoms specific to the validation set you are given.** CRITICAL: Generalize, Don't Overfit. Be Open to New Frames. Your goal is to find **broad patterns that apply across many queries**, not to craft fixes for individual failure cases. - **Explore a wide range of failures AND successes**, not just the first few. Look across different query styles, document lengths, and topic areas. - **Investigate successes too, not only failures.** A success tells you what currently works --- what signal is the index already exploiting? Any change you recommend should preserve that signal, not destroy it. Comparing "what makes a success a
For larger corpora we propose a two‑stage evaluation: first approximate candidate programs on sampled subsets, then validate promising ones on the full corpus. Budget‑aware strategies are left as future work.
Code Agent Prompting
Guidelines and interface for the Code Agent that generates BM25 preprocessing scripts.
This appendix spells out the exact prompt given to the Code Agent, including the required Python interface, metric objectives, and non‑negotiable constraints on document identifiers.
Section A.4 introduces the full prompt that drives the Code Agent.
Skeleton of the required Preprocessor class for the Code Agent.
Two non‑negotiable rules are that chunk.`doc_id` must be copied verbatim from the source Document and that the opaque hash values of `doc_id` cannot be parsed or used as retrieval signals.
The primary goal is to increase Recall@100 without sacrificing nDCG@10; any hypothesis that improves Recall by 0.01 R@100 while dropping nDCG by 0.03 is considered a net loss.
Chunking should stay modest—typically one to four chunks per document—because excessive fragmentation inflates the index size without proportional benefit.
The analysis workflow proceeds in five stages: (1) select diverse failure and success cases, (2) retrieve top‑5 BM25 results per query, (3) compare gold versus top‑ranked documents, (4) extract term‑level patterns distinguishing failures from successes, and (5) synthesize generalizable document properties that can be addressed without harming existing strengths.
Analysis Agent Prompting
Guidelines for the analysis agent to diagnose BM25 failures and propose corpus‑wide preprocessing changes.
You are an expert information‑retrieval analyst whose job is to explain why BM25 succeeds on some queries and fails on others, then suggest corpus‑wide preprocessing changes.
Analysis‑agent workflow (high‑level pseudocode).
Generated Code Example 1
Appendix A6 provides the generated preprocessing script used for the CRUMB case study.
Listing 1 shows the automatically generated preprocessing script used in the CRUMB case study. It prepares raw StackExchange posts for downstream indexing.
Generated Code Example 2
Python implementation of the section‑weighted repetition preprocessor.
Section‑weighted repetition preprocessor (AutoIndex representation program)