It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

Runpeng Dai, Kaili Huang, Changsung Kang, Ciya Liao

CoGR co-evolves query and item keyword generators using reinforcement learning to optimize direct lexical matching.

How can we improve keyword-based retrieval by co-optimizing query and item generators using reinforcement learning?

Retrieval systems often struggle to align query intent with item descriptions, forcing a trade-off between semantic understanding and the efficiency of keyword-based inverted indexes. CoGR trains separate Large Language Models (LLMs) to generate compact keyword sets for both queries and items, which are then matched directly through an inverted index. The framework uses a two-stage pipeline: supervised fine-tuning establishes an initial alignment, followed by alternating reinforcement learning where each generator optimizes against the other's frozen index. This co-evolution significantly improves retrieval quality, outperforming strong sparse, dense, and generative baselines on both internal and public search benchmarks.

Paper Primer

The core mechanism is an alternating reinforcement learning loop that treats the retrieval index as a fixed environment for each side. By rewarding the query generator with retrieval F1 and the item generator with a counterfactual marginal reward—measuring the F1 change caused by its specific keywords—the two keyword spaces progressively align to maximize matching accuracy.

CoGR achieves superior retrieval F1 compared to state-of-the-art sparse, dense, and generative baselines.

Performance on the internal APP Marketplace dataset and the public WANDS benchmark. 10.9% and 36.1% improvement in F1 over the strongest baseline, respectively.

The co-evolving design is critical: ablations show that optimizing only one side (query-side only) or failing to use the marginal reward signal leads to significant performance degradation, confirming that joint adaptation is necessary for effective keyword alignment.

Why use keyword-based matching instead of the dense vector similarity common in modern retrieval?

Keyword-based matching preserves compatibility with existing inverted index infrastructure, allowing the system to leverage the efficiency of lexical search while using LLMs to generate more semantically precise representations than traditional BM25.

What is the scope of this framework — does it require specific item types?

The framework is agnostic to the item type, as it relies on textual titles and descriptions; it has been validated on both application marketplace data and product search datasets, where it handles many-to-many relevance mappings effectively.

Researchers can now treat retrieval as a co-evolving generation task, moving away from static index construction toward systems that dynamically adapt both query and item representations through retrieval-metric feedback.

Introduction to CoGR

We expose the retrieval misalignment problem and introduce CoGR to co‑evolve query‑ and item‑side keyword generators.

Retrieval is the first stage of modern search and advertising pipelines; it selects a candidate set from a massive item universe for downstream ranking and auction. Errors at this stage are largely irreversible—missed items cannot be recovered later, while irrelevant candidates overload subsequent models. Consequently, a retriever must balance broad coverage (recall) against precise candidate selection (precision).

Classical lexical methods such as BM25 rely on explicit terms and inverted indexes, making keyword‑based retrieval especially prevalent in sponsored search. Dense retrieval overcomes exact lexical overlap by embedding queries and items into a shared continuous space and matching via vector similarity. More recently, generative retrieval replaces similarity‑based matching with autoregressive generation of semantic identifiers, yet it faces challenges in identifier design, decoding scalability, and generalization.

Recent work leverages large language models (LLMs) for retrieval by prompting them for query expansion, data synthesis, or keyword generation, and by training them with feedback from downstream retrievers. However, these approaches typically train a generator on only one side—most often the query side—and still depend on a separate downstream retriever for matching, leaving a natural question: can we train LLMs to jointly construct retrieval representations for both queries and items?

We propose Co‑evolving Generative Retrieval (CoGR), a framework that trains separate LLMs to generate compact keyword sets for queries and items, matching them directly through an inverted index. This design preserves compatibility with existing keyword‑based infrastructure while eliminating the need for a downstream matcher. CoGR aligns the two keyword spaces via a two‑stage pipeline: supervised fine‑tuning (SFT) establishes an initial aligned space, then alternating reinforcement learning (GRPO) co‑evolves the query‑ and item‑side generators, each optimizing the same retrieval F1 objective.

Retrieval that represents both queries and items as sets of discrete keywords, then matches them by exact term overlap using an inverted index.

**Figure 1.** The overall CoGR training pipeline and performance. CoGR consists a SFT stage and a co-evolving RL stage looping between Item and Query LLMs. CoGR achieves strong performance gains on both internal APP marketplace and public item search benchmarks.

The misalignment problem in generative retrieval—where query and item representations drift apart—prevents effective direct matching.

SFT Initialization

We align query and item keyword generators via supervised fine‑tuning, then co‑evolve them with reinforcement learning.

Retrieval requires a shared lexical bridge between a user query $q$ and a large item universe $I$, but naïve keyword generators often produce disjoint vocabularies that cripple recall.

CoGR first aligns the query‑side and item‑side generators with a supervised fine‑tuning step, then repeatedly refines them in an alternating reinforcement‑learning loop so that the keywords they emit become mutually supportive.

How does CoGR differ from a standard keyword‑based retrieval pipeline?

Standard pipelines fix a single static keyword extractor (often hand‑crafted) and never adapt it to the retrieval task. CoGR, by contrast, learns two generators that are jointly optimized: the SFT stage aligns them, and the RL stage continuously reshapes them so that query keywords and item keywords reinforce each other’s relevance.

SFT seeds both generators with keywords that already overlap for known relevant pairs, giving the later RL stage a solid recall foundation.

Pool the item keywords: $\mathcal{B}_q = \{\text{apple},\text{fruit},\text{red},\text{banana},\text{fruit},\text{yellow}\}$.

Count frequencies: “fruit” appears twice, all others once.

Select the top‑$N=2$ keywords: $S_q = \{\text{fruit},\text{apple}\}$ (ties broken arbitrarily).

Fine‑tune $G_{\text{SFT}}^{q}$ on the pair $(q, S_q)$ and $G_{\text{SFT}}^{i}$ on each $(i, S_i)$.

The SFT step guarantees that every query keyword set contains at least one word (“fruit”) that is guaranteed to appear in all its relevant items, establishing a baseline overlap.

**Algorithm 1** SFT initialization of both sides. **Require:** Query and item sets $\mathcal{Q}, \mathcal{I}$, base LLM $G_0$, budgets $M, N$ 1: **for all** items $i \in \mathcal{I}$ **do** 2: $\quad S_i \leftarrow M$ keywords sampled from $G_0(i)$ 3: **end for** 4: **for all** queries $q \in \mathcal{Q}$ **do** 5: $\quad \mathcal{B}_q \leftarrow \uplus_{i \in \text{rel}(q)} S_i$ $\quad \triangleright$ multiset 6: $\quad S_q \leftarrow$ top-$N$ most frequent keywords of $\mathcal{B}_q$ 7: **end for** 8: $G_{\text{SFT}}^q \leftarrow \text{SFT}(G_0; \{(q, S_q)\}_{q \in \mathcal{Q}})$ 9: $G_{\text{SFT}}^i \leftarrow \text{SFT}(G_0; \{(i, S_i)\}_{i \in \mathcal{I}})$

Stage 1: Run SFT initialization (Algorithm 1) to obtain $G_{\text{SFT}}^{q}$ and $G_{\text{SFT}}^{i}$.

Stage 2: Enter the alternating RL loop—update $G^{q}$ while keeping $G^{i}$ fixed, then update $G^{i}$ while keeping $G^{q}$ fixed.

After each RL iteration, recompute keyword sets $S_q$ and $S_i$ and evaluate retrieval performance.

Terminate when retrieval metrics stop improving.

The Alternating RL Loop

Alternating reinforcement learning co‑evolves query and item generators for better keyword retrieval.

After SFT the generators still produce keywords that are not tuned for the retrieval task; jointly training them would make each chase a moving target. The paper therefore adopts an alternating reinforcement‑learning loop that freezes the opposite side’s index while updating the other side.

We train the query‑side generator while the item index stays fixed, then flip: train the item‑side generator while the query index stays fixed. By alternating, each side sees a stable retrieval environment, avoiding the instability of simultaneous updates.

Round 1: Freeze the item index (built from initial item generator). Train the query generator; it emits $S_{q_1}=\{k_1,k_2\}$ and $S_{q_2}=\{k_3\}$.

Compute F1 for each query against the frozen item index; obtain rewards $R_q(S_{q_1})=0.6$, $R_q(S_{q_2})=0.8$.

Update the query generator with these rewards, then rebuild the query index from the updated generator.

Round 2: Freeze the newly built query index. Train the item generator; it emits $S_{i_1}=\{k_4\}$, $S_{i_2}=\{k_5,k_6\}$ (the latter exceeds $K_{\text{max}}$ and gets reward $-1$).

Form a counterfactual item index by swapping $S_{i_1}$ only; recompute F1 for each query and take the difference as the item reward.

Update the item generator and rebuild the item index; the loop repeats.

The alternating schedule isolates each generator’s effect on retrieval, making the credit assignment tractable.

Given a frozen item index, the query generator is rewarded for producing keyword sets that retrieve many relevant items while avoiding irrelevant ones.

How does this differ from a standard policy‑gradient language model that maximizes likelihood of reference keywords?

Standard likelihood training rewards every generated token equally, ignoring retrieval impact. Here the reward is the retrieval‑oriented F1 score (subject to a budget), so the generator learns to pick keywords that improve the downstream matching performance, not just to mimic a reference set.

With the query index frozen, the item generator is rewarded for keyword changes that increase the overall query‑to‑item F1, measured by comparing a counterfactual index (where only that item’s keywords change) to the reference index.

Why not simply train the item generator to maximize the same F1 reward used for the query side?

Optimizing directly on the global F1 would entangle the contributions of all items, making credit assignment ambiguous. By measuring the marginal effect of a single item’s keywords, the reward isolates the item’s impact, yielding a clearer learning signal.

Algorithm 2: Co‑evolving RL loop.

**Figure 2.** Co-evolving reinforcement learning in CoGR. Training alternates between query-side and item-side optimization, with the opposite-side index kept frozen during each stage. Left: Query-side RL. The item LLM from the previous round constructs a frozen item index. For each query, the query LLM samples keyword sets, retrieves items through keyword matching, and is optimized with GRPO using the resulting retrieval $F_1$ as reward. Right: Item-side RL. The updated query LLM constructs a frozen query index. For each sampled keyword set $S_i$, we construct a counterfactual item index by replacing only the reference keywords of item $i$ while leaving all other items unchanged. The item LLM is optimized with GRPO using the resulting marginal contribution to query-side retrieval quality, as defined in Equation 2.3.

Main Performance Results

Experimental results showing CoGR’s retrieval performance across datasets.

CoGR co‑evolves query‑side and item‑side generators via an alternating RL loop to improve keyword‑based retrieval.

CoGR achieves the highest overall $F_1$ score on both validation datasets.

Table 2 shows $F_1$ = 0.682 on WANDS and $F_1$ = 0.396 on Internal, surpassing every baseline.

**Table 2.** Retrieval performance on the validation datasets. Columns suffixed with @100 (MRR@100, NDCG@100, P@100, R@100, $F_1$@100) are computed on the top-100 retrieved items, while P, R, and $F_1$ are the macro-averaged precision, recall, and $F_1$ over the full retrieved set. * denotes that the Item-side parameters are frozen. The best value in each column within a dataset is shown in bold. Metrics at cutoffs 10 and 1000 are reported in Appendix C.

**Table 7.** Held-out query-side retrieval metrics at cutoffs 10 and 1000, using the same method grouping as Table 2. * denotes that the Item-side parameters are frozen. The best value in each column within a dataset is shown in bold.

Dynamics and Ablations

We evaluate how each CoGR component and training choice impacts retrieval performance.

Co‑evolving dynamics alternate reinforcement‑learning updates between the query‑side and item‑side generators, gradually aligning their keyword spaces.

Alternating RL raises validation F₁ from roughly 0.16 to 0.40 after five co‑evolving rounds.

Figure 3 shows the stepwise increase, with the largest jump in the first round.

**Figure 3** Evaluation $F_1$ over cumulative training steps on the Internal dataset, starting from the SFT-initialized generators. Training alternates between query-side and item-side RL phases and labels V1–V5 denote the successive co-evolving rounds on each side. Evaluation $F_1$ increases from approximately 0.16 before co-evolving RL to 0.40 after five rounds of alternation.

We next ablate three training‑design choices: the marginal item‑side reward, separate generators, and the SFT initialization.

All three ablated variants fall short of the full CoGR model, confirming that each design choice contributes positively to overall performance.

Keyword evolution analysis reveals that co‑evolving RL drives the vocabulary toward longer, more specific phrases while pruning generic unigrams.

**Figure 4.** Keyword vocabulary dynamics under co-evolving RL on the Internal dataset. (a) Keywords added (blue) and dropped (red) between the post-SFT vocabulary and that after five RL rounds; bubble area is proportional to the number of entities associated with each n-gram. (b) Distribution of distinct n-grams by length across training rounds. (a) and (b) are computed over the union of query- and item-side keywords. (c) Number of distinct indexed n-grams on the item (app) and query sides across rounds, shown on a log scale.

Finally, we assess how the amount of textual context supplied to each generator affects retrieval quality.

Removing item descriptions lowers F₁ from 0.3963 to 0.3759.

Table 4 reports the “– Description” variant.

Adding search‑result snippets raises F₁ from 0.3963 to 0.4379.

Table 4 reports the “+ Search Results” variant.

Related Retrieval Methods

Survey of sparse, dense, and generative retrieval paradigms relevant to CoGR.

BM25 scores documents by matching query terms against term frequencies, rewarding exact lexical overlap.

Dense retrieval encodes queries and documents into fixed‑dimensional vectors and finds nearest neighbors via inner‑product similarity.

Beyond classic sparse and dense paradigms, generative retrieval replaces nearest‑neighbor lookup with autoregressive generation of document identifiers.

Learned term‑weighting model that expands queries with additional lexical tokens while preserving the inverted‑index structure.

Deep Contextualized Term weighting that learns term importance directly from relevance signals.

Extends DeepCT by incorporating impact scores that capture term importance across the corpus.

Unified Contextualized Inverted List that learns a single dense representation per term for efficient retrieval.

Sparse Lexical AnD Dense Embedding that learns sparse token weights while preserving a dense semantic component.

Dense Passage Retrieval that trains bi‑encoders on question‑answer pairs to produce high‑quality passage embeddings.

Approximate Nearest Neighbor Negative Contrastive Learning that mines hard negatives via ANN during training.

Enhances dense retrieval with improved negative sampling, denoising, and multi‑stage training.

Generative Text Retrieval that scales pretrained encoders for dense retrieval.

Direct Sequence Indexing that assigns semantic identifiers to documents and trains a seq2seq model to generate them.

Neural Contextual Identifier improves DSI by augmenting queries and using prefix‑aware decoding.

Jointly learns document tokenization and retrieval, optimizing identifiers for relevance.

Adaptive Semantic Identifier learns document indexes from retrieval supervision.

Memory‑Efficient Vector Index that constructs compact identifiers for generative retrieval.

Retrieval‑Oriented Prefix‑Optimized Representation builds identifiers from quantized vectors.

Uses document titles as lexical identifiers within a pretrained vocabulary space.

Constrains generation to document substrings, enabling controlled lexical identifier creation.

Combines multiple lexical views to produce richer document identifiers.

Learns lexical identifiers from retrieval supervision, moving beyond fixed strings.

Novel identifier generation that integrates semantic and lexical cues.

Abstractive Content‑driven Identifier that generates keyphrases or summaries as document IDs.

Additional Retrieval Metrics

Retrieval performance at cutoffs 10 and 1000 highlights CoGR 4B’s leading MRR@1000.

CoGR 4B attains the top MRR@1000, surpassing the next best method by a small margin.

Table 7 shows CoGR 4B reaching 0.9090 while the runner‑up records 0.9067.

Across both cutoffs, CoGR variants consistently rank above dense and sparse baselines, demonstrating the benefit of co‑evolving generators even when item‑side parameters remain frozen.

Example Generated Keywords

Appendix D shows how CoGR’s alternating RL reshapes generated keywords and lists the base models for all retrieval baselines.

Table 8 illustrates how the keywords produced by the query‑side and item‑side generators evolve over the alternating reinforcement‑learning rounds and how they improve when extra context (search hints or item descriptions) is supplied.

CoGR repeatedly updates a query‑side generator and an item‑side generator so that the keywords each produces become better matches for the other, yielding mutually optimized retrieval cues.

Initial generation (w/o Search) yields: “album, anniversary, birthday, birthdays, digital, photo, search, sharing, tracker, …”.

Providing the search hint leads to a refined list (w/ Search): “genealogy, family history, genealogy app, family calendar, ancestry, family tree”.

External hints act as a soft constraint that pushes the generator from a broad, topic‑agnostic set toward a focused, intent‑aligned set of keywords.

Table 9 lists the pretrained models that serve as the backbone for each retrieval baseline, ensuring that performance differences stem from the retrieval strategy rather than from differing model capacities.

**Table 9.** Base models used for each baseline.

**Table.** (a) Query side. Round-5 keywords are also shown for a prompt that additionally contains online search hints for the query.

Experimental Setup

We detail the datasets, baselines, and training configuration used to evaluate CoGR.

Phase 2 of CoGR implements an iterative co‑evolution between the query‑side generator $G^{q}$ and the item‑side generator $G^{a}$, starting from the post‑SFT models. Each iteration trains one generator against an index built by the latest version of the opposite generator, gradually aligning their keyword spaces.

**Figure 5** Prompts used to fine-tune the query-side ($G^q$) and item-side ($G^i$) generators.

We evaluate on two industrial search corpora. The internal APP marketplace dataset contains 13,500 training queries and 1,500 evaluation queries over 39,600 applications, with roughly 1,000 relevant items per query. The public WANDS dataset provides 430 training and 50 evaluation queries over 42,994 products, with about 200 relevant items per query.

Table 1 lists these statistics alongside the size of each item universe, highlighting the many‑to‑many relevance pattern that distinguishes our setting from traditional sparse‑annotation benchmarks.

For comparison we adopt three families of baselines. Sparse retrieval includes BM25 and SPLADE‑v2; dense retrieval covers DPR, ANCE, and Qwen3‑Embedding‑4B (both zero‑shot and ANCE‑finetuned); generative retrieval comprises DSI, DSI‑QG, RIPOR, and DeepRetrieval, the latter sharing a reinforcement‑learning formulation with our query‑side component.

Both generators use the Qwen3‑4B‑Instruct backbone (CoGR‑4B) or Qwen3‑1.7B (CoGR‑1.7B). In Phase 1 (SFT) we set the query‑side top‑$N$ to 15 and the item‑side keyword budget $M$ to 10. During Phase 2 we cap the generated keyword set at $K_{\max}=30$, alternating optimization for five rounds (10 GRPO epochs for $G^{q}$ and 5 epochs for $G^{a}$ per round).

Conclusion

We recap CoGR’s gains and outline future research directions.

We introduced CoGR, a co‑evolving generative retrieval framework that trains LLMs to directly construct keyword‑based retrieval representations on both query and item sides. By combining an aligned SFT initialization with alternating GRPO optimization against a frozen opposite‑side index, CoGR enables the two keyword spaces to progressively co‑adapt under a shared retrieval F1 objective, yielding consistent gains over strong sparse, dense, and generative baselines on the internal APP Marketplace and WANDS datasets.

More broadly, our work demonstrates the feasibility of a co‑evolving keyword‑matching framework in which query‑ and item‑side representations are jointly adapted through retrieval feedback. This framework leaves several directions for future work: first, the reward can be extended beyond relevance metrics such as F1 to downstream business objectives like irrelevant ads percentage and revenue gain; second, our current ranking stage uses BM25 over the generated keywords, so designing stronger retrieval ranking could further improve overall retrieval quality.

Training Details

Implementation specifics for prompts, hyperparameters, and efficient reward computation.

The section begins by specifying the prompts used for the two generators. The query‑side generator (Gq) is instructed to expand a user query into a comma‑separated list of keywords without explanation. The item‑side generator (Gi) extracts keywords from an item’s title and description under the same “list‑only” constraint.

**Table 5.** Decoding hyperparameters used for RL rollouts, initialization, evaluation, and index construction.

During RL training the authors replace the default sampling scheme with unconstrained stochastic sampling to promote exploration. They also adopt a fully online RL pipeline, generating fresh rollouts from the current policy at every update to avoid off‑policy bias.

**Table 6.** GRPO hyperparameters for item-side and query-side training. Hyperparameter names follow the original verl configuration fields.

Computing the item‑side reward naively would require rebuilding the entire index for each rollout. Instead, the implementation caches three per‑query counts: $n_{\text{ret}}(q)$ (retrieved items), $n_{\text{tp}}(q)$ (true positives), and $n_{\text{rel}}(q)$ (relevant items), enabling a constant‑time reference F1 score $F_{\text{ref}}^{1}(q)=\frac{2\,n_{\text{tp}}(q)}{n_{\text{ret}}(q)+n_{\text{rel}}(q)}$.

For each item $i$ the set of queries that retrieve it under the reference keywords is cached as $Q_{\text{ref}}^{i}$. After sampling a new keyword set $S_i$, a single lookup against the frozen query‑side inverted index yields $Q_{\text{cand}}^{i}$, and the symmetric difference $Q_{\Delta}^{i}=Q_{\text{cand}}^{i}\triangle Q_{\text{ref}}^{i}$ identifies the affected queries. Their candidate F1 scores are updated directly using $F_{\text{cand}}^{1}(q)$, and the item‑side reward $R_i(S_i)=\sum_{q\in Q_{\Delta}^{i}}\bigl(F_{\text{cand}}^{1}(q)-F_{\text{ref}}^{1}(q)\bigr)$ is computed only over this small set, avoiding a full index rebuild.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers