Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-Agent RL

Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li

Co-RL uses cross-agent supervision to enable label-free reinforcement learning without the bias-amplification of self-rewarding.

How can language models improve their reasoning performance using reinforcement learning without relying on ground-truth labels or external verifiers?

Reinforcement learning for reasoning typically requires ground-truth labels, which are expensive and scarce. While self-rewarding methods attempt to learn from a model's own outputs, they often reinforce existing biases and lead to training collapse because the model supervises itself.

Paper Primer

The core mechanism hinges on decorrelated supervision: by using a peer's majority-vote answer as the reward target, the system forces agents to align with an external, independently updated perspective. This cross-agent signal acts as a corrective filter, preventing the model from simply reinforcing its own incorrect predictions.

Co-RL expands the basin of correct convergence compared to self-rewarding.

Theoretical analysis shows that while self-rewarding is self-confirming (reinforcing the currently favored answer regardless of correctness), Co-RL's cross-agent supervision allows one agent's correct prediction to correct another's mistake. Outperforms prior label-free self-rewarding methods by 0.8–2.0% and multi-agent RL methods like CoMAS by 4.0% on average.

Cohort diversity is essential for effective cross-agent learning.

Using heterogeneous model families, sizes, and rephrased prompts reduces correlated errors, ensuring that peer supervision provides a distinct, informative signal rather than echoing the same mistakes. Combining different model families and data decoupling (Different family+) achieves the strongest label-free performance.

Why does this approach require multiple agents instead of just one?

A single agent supervising itself creates a self-reinforcing loop that amplifies its own biases and errors. By using a peer agent, Co-RL provides an independent, decorrelated signal that can correct errors the agent would otherwise reinforce.

Does this method require more compute than standard self-rewarding?

Co-RL is designed to be lightweight and symmetric; it requires no external LLM judge or learned reward model. While it trains multiple agents, the agents are decoupled and interact only at the reward stage, allowing them to improve simultaneously within a single training run.

Researchers can now achieve supervised-level reasoning performance in label-scarce domains by leveraging cross-agent supervision, effectively turning the "error-overlap" problem into a source of corrective signal.

Introduction: The Label-Free Challenge

We expose the reliance on ground‑truth rewards as the core obstacle to label‑free reasoning.

Reinforcement learning for reasoning still depends on verifiable, ground‑truth rewards, which are expensive to collect and become scarce as tasks outpace human evaluation. Self‑rewarding approaches try to sidestep this by letting a model judge its own outputs, but the feedback loop reinforces existing biases, reduces response diversity, and can lead to training collapse.

The fundamental obstacle is obtaining a learning signal that is sufficiently independent from a model’s own predictions. If an external reference is unavailable, can independently trained agents supply corrective feedback that a single model cannot generate on its own?

Reasoning without any external reward labels, relying solely on internally generated signals that are not self‑reinforcing.

Co‑RL operationalizes this insight: a cohort of decoupled models samples completions, aggregates them by majority vote, and each model receives rewards from the peer’s aggregated answer. By training all agents simultaneously, the framework supplies each policy with supervision that originates from a partner’s independently updated weights, preventing the self‑reinforcing bias of single‑model reward schemes.

**Figure 1.** Comparison of Co-RL with prior label-free RL methods. Both TTRL and Co-rewarding derive rewards from self-generated agreement, and CoMAS scores multi-turn interactions with one of its own agents acting as judge. Co-RL instead derives rewards directly from peer votes. Beyond two agents, the votes pass along a directed ring (N=3 shown).

The bottleneck of ground‑truth supervision in RL can be bypassed by peer‑based rewards that avoid self‑reinforcing collapse.

Prior Approaches to Self-Reward

Survey of label‑free RL approaches and their limitations.

Methods that generate reward signals from a model’s own behavior without external labels.

Using a second, conditionally independent view to supervise a learner, originally proposed for semi‑supervised classification.

Approaches that involve multiple agents interacting either at inference time or during training to improve reasoning performance.

Reinforcement‑learning‑based reward design for vision‑language models, extending the RLVR paradigm beyond text‑only settings.

Reinforcement Learning for Reasoning

We introduce Group Relative Policy Optimization, a group‑wise advantage method for RL reasoning.

Reinforcement learning treats a language model $\pi_{\theta}$ as a policy that generates a response $y$ token‑by‑token given a prompt $x$. The learning objective $J(\theta)$ maximizes the expected scalar reward $r(x,y)$ over the data distribution.

GRPO evaluates each sampled response against its peers, turning raw rewards into a group‑relative advantage that drives policy updates.

Group mean $= (1.0+0.5+0.2)/3 = 0.5667$.

Group std $= \sqrt{\frac{(1.0-0.5667)^2+(0.5-0.5667)^2+(0.2-0.5667)^2}{3}} \approx 0.366$.

Advantages $\hat{A}_{1}= (1.0-0.5667)/0.366 \approx 1.18$, $\hat{A}_{2}= (0.5-0.5667)/0.366 \approx -0.18$, $\hat{A}_{3}= (0.2-0.5667)/0.366 \approx -1.00$.

Assume token‑level ratios $\rho_{k,t}=1.2$ for all $k,t$ and clipping $\delta=0.2$. The clipped ratio is $\operatorname{clip}(1.2,0.8,1.2)=1.2$, so the min term equals $1.2\hat{A}_{k}$ for each $k$.

Aggregating over tokens (say each response has $|y_{k}|=5$ tokens) yields a surrogate contribution of $\frac{1}{3}\sum_{k} \frac{1}{5} 5\cdot 1.2\hat{A}_{k}=1.2\cdot\frac{1}{3}\sum_{k}\hat{A}_{k}=1.2\cdot(1.18-0.18-1.00)/3 \approx 0.0$ (the advantages cancel).

Group‑relative normalization can produce zero‑mean advantages, which stabilizes updates but also means the net gradient may vanish unless the KL term or other regularizers provide a driving signal.

How does GRPO differ from standard PPO, which also uses clipped ratios?

Standard PPO clips a single scalar advantage computed against a learned baseline, whereas GRPO first normalizes raw rewards across a *group* of sampled responses, producing a per‑response advantage $\hat{A}_{k}$ that reflects relative performance within that batch. This group‑wise step replaces the external baseline and makes the method robust to the absence of ground‑truth rewards.

When external verifiers are unavailable, prior work constructs rewards from the model’s own outputs (e.g., TTRL’s majority‑vote reward, Intuitor’s confidence, RENT’s entropy). Although these self‑generated signals can drive policy updates, they risk reinforcing the policy’s existing biases, reducing diversity, and eventually causing training collapse.

The Co-RL Framework

Co‑RL lets multiple agents teach each other using peer‑generated pseudo‑rewards.

Without ground‑truth rewards, a model needs an independent signal; otherwise it merely reinforces its own mistakes.

Think of two students grading each other’s essays: each writes an answer, then the other’s majority grade becomes the reward.

How does Co‑RL differ from standard multi‑agent RL where agents share a common reward?

Standard multi‑agent RL gives every agent the same environment reward, which can be noisy or sparse. Co‑RL instead constructs a *peer‑based* reward that is specific to each agent’s peer, ensuring that the signal reflects agreement with an *independent* perspective rather than a shared external signal.

It is a binary signal: 1 if an agent’s extracted answer equals the majority answer of its designated peer, 0 otherwise.

Agent 2’s majority vote is B (appears twice), so ĥ$a_{-1}$(x) = B.

Agent 1 receives rewards r = {0, 1, 0} because only the second answer (B) matches the peer’s majority.

Agent 1’s reward vector is normalized (mean = 1/3, std ≈ 0.47) before entering GRPO.

Agent 1’s peer (Agent 2) computes its own majority vote = B, so ĥ$a_{-2}$(x) = B.

Agent 2’s rewards r = {1, 1, 0} because two answers already equal the peer’s majority.

The binary reward isolates agreement from the absolute correctness of any single answer, allowing agents to improve by learning to align with each other’s diverse perspectives.

Sample a batch B of unlabeled prompts from D.

For each prompt x in B, each agent n generates K responses $y_k^{n}$ and extracts answers $a_k^{n}$=g($y_k^{n}$).

For each agent n, compute the peer’s majority‑vote pseudo‑label ĥ$a_{-n}$(x) using the cyclic peer (n‑1).

Assign binary rewards $r_k^{n}$=1 iff $a_k^{n}$=ĥ$a_{-n}$(x).

Normalize rewards within each agent’s rollout group to obtain group‑relative advantages.

Update each policy $\pi_{θ_n}$ independently with one GRPO step using the normalized advantages.

**Figure 3.** Overview of Co-RL with two agents. Each agent samples $K$ responses to the same unlabeled question and generates a pseudo label with majority vote. Each rollout is then rewarded by agreement with the cohort's pseudo label, and updates its own policy, with no sharing parameters and no gradient exchange except the cross-reward process.

Diversity among the agents is the second pillar of Co‑RL; if agents make the same mistakes, the peer reward becomes noisy.

Diverse models bring complementary error patterns, so one agent’s correct answer can rescue the other’s mistake.

Agent A solves the original prompt and outputs answer 2.

Agent B solves the rewritten prompt and also outputs answer 2.

Both agents’ majority‑vote pseudo‑label is 2, so each receives reward 1.

If only one agent had been used, a mis‑parsing could have yielded answer 3, leading to a self‑reinforcing error.

Even a superficial rewrite can break a shared lexical bias, allowing the agents to correct each other’s systematic misinterpretations.

Learning Dynamics of Co-RL

We analyze how cross‑agent rewards reshape learning dynamics compared with self‑rewarding.

Self‑rewarding agents train on pseudo‑labels they generate themselves, so a systematic mistake is reinforced rather than corrected. CO‑RL replaces this self‑generated signal with peer‑based rewards, allowing one agent’s correct prediction to fix another’s error.

Diversity among agents creates corrective feedback: each agent can learn from the others’ successes, turning disagreements into a learning signal.

Compute the update magnitude for each agent: $q_K(p)=\eta\,p(1-p) = 0.5\times0.9\times0.1=0.045$ for $A$, $0.5\times0.2\times0.8=0.08$ for $B$.

Agent $A$’s majority vote is correct, so $\phi_K(p_B)=+1$; agent $B$ receives a correct peer label.

Update $B$: $p_B \leftarrow p_B + q_K(p_B)\times\phi_K(p_A)=0.2+0.08\times(+1)=0.28$.

After one iteration $p_A$ stays high and $p_B$ moves toward the correct side; repeated updates drive $p_B$ above $0.5$.

Even a weak agent can be pulled to correctness when paired with a strong partner, illustrating the power of diversity.

How does diversity‑based learning differ from the self‑rewarding approach?

Self‑rewarding uses the agent’s own majority vote, which reinforces the current belief; diversity‑based learning uses a peer’s vote, so an agent can be nudged toward the correct answer even when its own belief is below $½$.

Proposition 2 shows that self‑rewarding is self‑confirming: the update sign matches the current majority, so if $p<\tfrac12$ the correct answer’s probability keeps decreasing, and if $p>\tfrac12$ it keeps increasing. Consequently the method cannot pull a wrong belief toward correctness.

Theorem 1 proves that with cross‑agent supervision the basin of attraction expands: any pair of agents whose combined confidence exceeds one ($p_A+p_B>1$) converges to the correct consensus, while pairs below one converge to the wrong consensus. The line $p_A+p_B=1$ forms a separatrix between these outcomes.

**Figure 2.** (a) Agreement between the two models, (b) pseudo-label accuracy, and (c) evaluation performance; (d) Error overlap before RL for two pairs each from a different family, the same family, and the same model under a different seed.

Empirical Performance and Benchmarks

Co‑RL delivers consistent gains over self‑rewarding baselines across models and tasks.

We first evaluate CO‑RL against single‑agent self‑rewarding baselines on two 3B language models, then extend the comparison to multi‑agent settings, vision‑language models, and controlled budget experiments.

CO‑RL outperforms all self‑rewarding baselines, improving average performance by 8.0 % on Qwen2.5‑3B.

Table 1 shows the Same‑family variant achieving this gain over the best baseline (TTRL).

In the CoMAS multi‑agent setting (Table 2), CO‑RL surpasses the prior state‑of‑the‑art method by 4.0 % average while using only half as many agents and requiring no external judging mechanism.

**Figure.** Training dynamics across different models (Qwen-3B, Llama-3B, Qwen-7B, Llama-8B) comparing RENT, TTRL, Intuitor, GT-Reward, and Co-RL methods. The figure is divided into three panels: (a) Validation accuracy, (b) Standard deviation of reward, and (c) Mean completion length.

**Figure 5.** Training dynamics for Qwen2.5-VL-7B trained with InternVL3.5-8B on open-r1. (a) Evaluation accuracy, (b) mean completion length, and (c) the accuracy of the exchanged pseudo-labels together with the agreement between the two agents. Co-RL keeps improving and holds its completion length, while TTRL peaks and then degrades in both.

Co‑RL consistently outperforms single‑agent self‑rewarding baselines.

Data Decoupling and Rephrasing

Ablation experiments quantify the impact of each Co‑RL component.

We evaluate how each Co‑RL design choice contributes to performance by removing or altering it and measuring the resulting drop.

Rephrasing training questions doubles the question length while preserving answers.

In a sample of 300 aligned pairs, every rewrite kept the original answer.

CO‑RL (Different family+) raises Qwen2.5‑7B average accuracy from 49.0 % to 53.6 %.

Table 8 shows a 4.6 % absolute gain over the base model.

CO‑RL (Different family+) raises Llama‑3.1‑8B‑Instruct average accuracy from 44.7 % to 47.7 %.

Table 8 reports a 3.0 % absolute improvement.

CO‑RL (Different family+) outperforms the strongest self‑rewarding baseline by 0.8 % on Qwen2.5‑7B.

Self‑rewarding baselines achieve 52.8 % average; CO‑RL reaches 53.6 %.

CO‑RL (Different family+) outperforms the strongest self‑rewarding baseline by 1.1 % on Llama‑3.1‑8B‑Instruct.

Self‑rewarding baselines achieve 46.6 % average; CO‑RL reaches 47.7 %.

CO‑RL (Different family+) surpasses GT‑Reward on Llama‑3.1‑8B‑Instruct (47.7 % vs 47.1 %).

Both methods use no ground‑truth labels; CO‑RL yields a higher average.

CO‑RL improves Qwen2.5‑VL‑7B vision‑language average by 7.2 %.

Table 9 shows the variant reaching 57.2 % versus the base 50.0 %.

CO‑RL improves InternVL3.5‑8B vision‑language average by 6.3 %.

Table 9 reports 56.3 % versus the base 50.0 %.

CO‑RL improves Gemma‑3‑12B vision‑language average by 5.8 %.

Table 9 shows 55.8 % versus the base 50.0 %.

CO‑RL surpasses GT‑Reward on Gemma‑3‑12B (47.56 % vs 45.17 %).

Both lack ground‑truth supervision; CO‑RL attains a higher score.

**Example 1** - **Original:** How many vertical asymptotes does the graph of $y = \frac{2}{x^2+x-6}$ have? - **Rephrased:** The function $f(t) = \frac{2}{t^2+t-6}$ describes the temperature of a chemical reaction over time $t$. How many vertical asymptotes appear on the graph of this function? - **Answer:** 2 **Example 2** - **Original:** In triangle $ABC$, $AB = AC = 14$ and $BC = 26$. What is the length of the shortest angle bisector in $ABC$? Express your answer in simplest radical form. - **Rephrased:** A triangular park has two equal sides of 14 meters and a third side of 26 meters. The city plans a path from each corner that bisects its angle, and will build only the shortest one. How long is that path? Express your answer in simplest radical form. - **Answer:** $3\sqrt{3}$

**Table 8.** Results at 7B and 8B on the seven-benchmark suite (%). Base and GT-Reward serve as references and are excluded from the ranking.

Extended Benchmark Results

Appendix D lists the full performance tables that support the paper’s main claims.

This appendix expands the quantitative evidence presented in the main text, showing per‑benchmark scores for each model variant and highlighting the consistent advantage of the Co‑RL approach.

**Table.** Performance comparison of TTRL and Co-RL across different models and benchmarks.

**Table 11.** Matched-budget comparison between TTRL and Co-RL on multimodal reasoning benchmarks. Both settings train the same two base models, Qwen2.5-VL-3B and InternVL3.5-2B. The ensemble rows pool four rollouts from each of the two models for majority voting (maj@8, $T = 0.6$, top-$p$ 0.95). Rows are grouped by training set. MMR1 runs use the corrected multiple-choice grader and open-r1 runs the legacy grader, so the two blocks are not compared against each other.

Conclusion and Summary

We present the full pre‑training error‑decoupling measurements for model pairs, detailing diversity metrics before any RL.

We report the complete pre‑training error‑decoupling analysis for all model pairs considered in the paper. All numbers are computed on base checkpoints before any reinforcement‑learning fine‑tuning, using 500 zero‑shot MATH problems (levels 3–5) with rule‑based answer extraction. For each pair we count the four cells of Table 5 and derive the diversity measures defined by Kuncheva and Whitaker (2003).

**Table 5.** The four outcomes for a pair of models A and B. Every problem falls into exactly one cell, and all four diversity measures are counts over these cells.

**Table 6.** Error decoupling before RL, by what the two models differ in, sorted by $\kappa$ within each block

**Table 7.** One model held fixed, partner varied. Capability is identical to the seed-only row along each ladder, so the source of the partner is the only variable.

The three separation levels are cleanly distinct—no pair falls between them—so crossing families consistently lowers $\kappa$ and raises c compared to within‑family or seed‑only variations. Averaging within each group, different families improve $\kappa$ from 0.53 to 0.38 and increase c from 23.2 % to 30.8 %, confirming that diversity of pretrained errors is the primary driver of Co‑RL’s gains.

Proof of Proposition 1

Derives the reward‑induced GRPO dynamics underlying Proposition 1.

We now work through the algebra that yields the GRPO update when rewards are generated by peer agreement, first for a fixed pseudo‑label and then for the two concrete instantiations used in the paper.

Infinitesimal Update Limits

Derives continuous dynamics for Co‑RL and analyzes the symmetric two‑agent system.

We first pass to a continuous‑time limit, replacing the discrete update of the pseudo‑label probability $p_{n}$ with its time derivative.

Specializing to two symmetric agents $A$ and $B$, the majority‑vote signal from the partner’s rollouts is captured by $\phi_{K}(p)$.

Proposition 2 shows that a majority‑vote self‑rewarding scheme pushes the probability toward the nearest extreme (0 or 1) depending on whether the initial $p(0)$ lies below or above $1/2$.

The conserved quantity partitions the state space into three basins: if $p_{A}(0)+p_{B}(0)>1$ the trajectory converges to $(1,1)$, if $p_{A}(0)+p_{B}(0)<1$ it converges to $(0,0)$, and if $p_{A}(0)+p_{B}(0)=1$ it converges to the saddle point $(\tfrac12,\tfrac12)$.

Consequently, the basin of correct convergence for self‑rewarding RL is $\{p_{A}>1/2,\;p_{B}>1/2\}$, whereas Co‑RL expands this basin to $\{p_{A}+p_{B}>1\}$, establishing a strictly larger set of initial conditions that lead to unanimous correct answers.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers