Mosaic: Multi-Session Agentic Reinforcement Learning
Yilong Xu, Minrui Xu, Xinyu Zhou, Xiang Long, Haochen Tan, Zhijiang Guo
Mosaic optimizes multi-session agentic RL by hierarchically calibrating advantages based on session usage and marginal utility.
How can reinforcement learning agents effectively learn from multi-session trajectories—where tasks are broken across multiple interactions—rather than being restricted to single-session, continuous contexts?
Agentic reinforcement learning typically assumes a single, continuous interaction session, but modern techniques like context compaction and subagent forking force trajectories to span multiple sessions, creating a mismatch between training and inference. Mosaic addresses this by replacing coarse trajectory-level broadcasting with a hierarchical optimization target: it modulates trajectory advantages based on session counts and shapes individual session advantages using marginal information gain. On the BrowseComp benchmark, this approach improves accuracy by 10.1% over broadcasting while reducing the average number of sessions per trajectory.
Paper Primer
The core move is a critic-free hierarchical optimization that treats the trajectory and the session as distinct, coupled optimization targets. Mosaic calibrates the global trajectory advantage using a session-aware modulation factor, then refines local session behavior by injecting marginal information gain as a shaping signal.
Mosaic improves task accuracy while increasing session efficiency.
Experiments on BrowseComp using Qwen3.5-4B show Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios. 10.1% relative accuracy improvement with a reduction in average sessions from 3.17 to 2.35.
The method hinges on two mechanisms: session-aware trajectory advantage estimation, which uses centered ranks of session counts to penalize excessive session usage in successful trajectories, and session-level advantage shaping, which uses type-conditioned normalization to stabilize marginal gain signals across different session types.
Why is the standard "broadcasting" approach insufficient for multi-session agents?
Broadcasting replicates a single trajectory-level reward across all sessions, providing only coarse supervision that fails to distinguish the contribution of individual sessions or guide how computation should be allocated across a multi-session trajectory.
Does Mosaic fundamentally change the agent's behavior, or just optimize it?
Mosaic does not induce a new behavioral mechanism; instead, it strengthens existing patterns of session utility and allocation, encouraging the agent to suppress low-utility computation while continuing execution when it yields meaningful progress.
Effective multi-session agentic RL requires moving beyond outcome-only rewards to optimize the utility of individual sessions, treating the session count as a signal for exploration efficiency rather than a target for maximization.
Introduction and Motivation
We expose the limitation of single-session RL and motivate multi-session agentic learning.
Current agentic RL pipelines treat each rollout as a single, continuously accumulated context. When techniques such as context compaction or subagent forking split that rollout into multiple logical sessions, the single‑session assumption no longer holds, creating a training‑inference mismatch.
Agentic RL must handle trajectories that can be broken into several independent sessions rather than a single uninterrupted stream.
Compressing the execution history replaces older tokens with a compact representation, freeing up context slots for later actions.
Forking spawns a sub‑agent that pursues a sub‑goal, creating a parallel branch of interaction within the same overall task.
To address this mismatch we define a new setting—Multi‑Session Agentic Reinforcement Learning—that admits an arbitrary number of sessions per trajectory. A naïve baseline copies the trajectory‑level advantage to every token (broadcasting), but it offers no session‑specific guidance. Mosaic solves this by inserting session‑usage signals into advantage estimation and shaping each session’s advantage according to its marginal contribution.
**Figure 1 |** The comparison between multi-session and single-session trajectories.
The key shift is moving from a single‑session RL view to a multi‑session framework that can explicitly allocate credit across sessions.
Preliminaries and Baseline Methods
We define the multi-session agentic framework and the broadcasting mechanism for propagating trajectory-level rewards.
Agentic RL aims to maximize expected task outcomes by training a policy $\pi_{\theta}$ to generate trajectories through reasoning and tool use. In this setting, the agent interacts with an environment over a sequence of turns, receiving a scalar reward $R(\tau)$ only upon completing the task.
GRPO estimates policy advantages by comparing multiple rollouts of the same task against each other, eliminating the need for a separate learned critic model.
While standard GRPO assumes a single continuous training segment, agentic workflows often span multiple sessions—such as when an agent performs context compaction to manage limited windows. This creates a training-inference mismatch that limits optimization potential.
Broadcasting enables multi-session training by treating a trajectory as a set of sessions and propagating the single trajectory-level advantage to every token across all sessions.
The Mosaic Pipeline
We introduce session‑aware advantage estimation to guide multi‑session RL toward efficient trajectories.
Standard GRPO treats advantage solely as a function of relative rewards, which ignores how many interaction sessions a trajectory consumes. In multi‑session tasks this blind spot lets the policy waste steps without improving success.
It augments the raw outcome advantage with a session‑based signal that rewards using just enough sessions to succeed and penalizes unnecessary extra steps.
Success subgroup $G_1$ (size 2): ranks of $N$ are 1 for $2$ and 2 for $4$, so $f(N)=\{0,1\}$ and $u = \{-1,1\}$.
Failure subgroup $G_0$ (size 2): ranks of $N$ are 1 for $3$ and 2 for $5$, yielding the same $u = \{-1,1\}$.
With $\rho=0.4$, compute $\alpha_{\text{session}} = -0.4\,u$, giving $\alpha_{\text{session}} = \{0.4,-0.4\}$ for each trajectory.
Apply Eq. (10): a successful trajectory with $N=2$ gets $\hat{A}_{\text{sat}} = (1+0.4)\hat{A}_{\text{traj}}$, while the successful one with $N=4$ gets $(1-0.4)\hat{A}_{\text{traj}}$, reflecting the penalty for extra sessions.
The example shows how the same $\rho$ value simultaneously rewards efficient successes and penalizes wasteful ones, while still preserving the sign dictated by the outcome.
When all trajectories in a group share the same outcome, the session signal would be ineffective; we therefore fall back to a pure sign‑only advantage or, if no successes exist, use a fixed penalty $-\beta_1 u_i$ to keep learning signal alive.
**Figure 2 |** Overview of our proposed MOSAIC pipeline for multi-session agentic RL.
How does the session‑aware advantage differ from vanilla GRPO’s raw advantage?
Vanilla GRPO scales advantage only by relative rewards within a prompt group. The session‑aware version multiplies that baseline by $(1+\alpha_{\text{session}})$, where $\alpha_{\text{session}}$ encodes whether the trajectory used too many or too few sessions relative to its peers, thus shaping the magnitude without flipping the sign.
Experimental Setup and Benchmarks
Key performance numbers on BrowseComp illustrate Mosaic’s advantage.
MOSAIC achieves the highest accuracy among all methods on BrowseComp multi‑session tasks.
Table 1 shows MOSAIC reaches 32.46 % accuracy in the Multi‑Session/Subagent setting, surpassing the next best baseline.
Figure 3 visualizes the large accuracy jump when moving from single‑session to multi‑session inference, while Figure 4 shows that MOSAIC reaches higher rewards faster and stabilizes session growth earlier.
**Table.** Comparison of different methods across Single-Session, Multi-Session / Compact, and Multi-Session / Subagent inference settings, categorized by training on Single-Session or Multi-Session data.
On BrowseComp, MOSAIC delivers the strongest performance metrics across multi‑session configurations.
Ablation Study and Performance Analysis
Ablation and mechanistic studies quantify each Mosaic component’s impact.
Removing either the trajectory‑level session‑aware advantage estimation or the session‑level advantage shaping reduces Mosaic’s performance relative to the Broadcasting baseline. The compact scenario shows a clear drop, while the subagent scenario exhibits only a modest decline.
**Figure 3.** (a). Percentage of the 972 tasks failed by the base model under single-session inference that are solved by different methods under multi-session inference with compact. (b). Comparison of the computation time for marginal information gain. (c)(d). Ablation results on BrowseComp.
Mechanistic analysis shows that session‑aware trajectory‑level advantage estimation orders advantages consistently: successful trajectories exhibit decreasing advantage with more sessions, while failures show the opposite trend. The fallback mechanism further injects session‑dependent variation even when outcomes are identical.
**Figure 5.** The two left panels show the relationship between trajectory-level advantage and session count. The two right panels show the distributions of marginal information gain for normal and compact sessions under global normalization and type-conditioned normalization.
Global normalization down‑weights compact sessions because their raw marginal gains have lower means, creating a systematic bias. Type‑conditioned normalization rescales each session type separately, eliminating this bias and delivering balanced shaping signals.
Table 2 demonstrates that Mosaic does not increase the total number of sessions but improves session quality and allocation. Broadcasting raises all three metrics, whereas Mosaic achieves higher accuracy and positive‑IG without inflating session count, indicating more efficient use of existing sessions.
Theoretical Analysis and Training Curves
We examine the ordering constraints on session‑aware advantage and the gating mechanisms that bound its influence.
Session‑aware advantage augments the trajectory‑level advantage with a signal that reflects how many sessions a trajectory used. The signal is intended to rank trajectories within the same outcome subgroup while preserving the dominant effect of the binary outcome reward.
Constraint C1 requires that among successful trajectories, fewer sessions receive a higher advantage. Because $u_i<u_j$ when $N_i<N_j$, the multiplicative factor $1-\rho u_i$ is larger for the shorter trajectory, yielding a larger $\hat A^{\text{sat}}$.
Constraint C2 mirrors C1 for unsuccessful trajectories: a larger session count yields a less negative advantage.
Constraint C3 states that the outcome reward dominates the ordering: any successful trajectory always receives a higher advantage than any unsuccessful one, regardless of session count.
When all trajectories share the same outcome (homogeneous‑reward groups), the method falls back to a pure session‑signal term. For all‑success groups the fallback is $-\beta_1 u_i$; for all‑failure groups it is $\beta_0\min(u_i,0)$, preserving the weak orderings of C1 and C2 while guaranteeing that no unsuccessful trajectory becomes positive.
We now examine how the session signal is fused with the outcome advantage. Two families of fusion are considered: an additive correction and a multiplicative modulation.
Multiplicative modulation has two limitations. First, the absolute correction scales with $|\hat A^{\text{traj}}_i|$, so imbalanced outcome groups experience asymmetric session effects. Second, when $\hat A^{\text{traj}}_i=0$ (all trajectories share the same outcome), the multiplicative term provides no signal at all.
To retain a session signal in homogeneous‑outcome groups we adopt a hybrid rule: use bounded multiplicative modulation when outcomes differ, and fall back to an additive session term when the outcome advantage vanishes (Eq. 14 in §3.1).
We now describe how marginal information gain (IG) is computed for each non‑terminal training session.
After normalizing $\Delta r_s$ across a prompt group we obtain the session‑level shaping signal $r_s$, which is added to the trajectory‑level advantage with coefficient $\lambda_r$.
**Figure 6.** Training curves involving baselines trained under single-session setting and multi-session setting of context compaction scenario.
**Figure 7.** Training curves involving baselines trained under multi-session setting of subagent forking scenario.
Discussion and Future Directions
We outline open challenges for reproducible multi‑session RL, state transfer, asynchrony, and agent harnesses.
Reproducible research requires open training datasets and execution environments. Multi‑session trajectories naturally arise in long‑horizon tasks, but public data are scarce, limiting scale and reproducibility.
Cross‑session state transfer is a central learning objective. Context compaction must preserve enough working state for continued execution, subagent forking must supply information to complete a subtask, and merging must integrate subtask results into a state usable by the main agent. Under a limited context budget, the key question is which information should be carried forward to support subsequent decisions.
CompactionRL already jointly optimizes task execution and context summarization (Li et al., 2026).
Asynchronous training amplifies variation in trajectory lengths and completion times, increases policy staleness, and complicates within‑group comparisons and sample reuse. Prioritizing completed rollouts can bias selection toward shorter‑session trajectories, shaping learned execution strategies. Our experiments use a colocated setup with alternating rollout and training phases to isolate the effects of the optimization signals.
A remaining question is how multi‑session optimization can remain reliable under asynchronous sampling and policy staleness. SAO explores asynchronous optimization with one rollout per prompt, using a learned value model (Hou et al., 2026).
Further investigation is needed into how rollout groups are formed, how stale samples are handled, and how collected trajectories can be reused effectively.
General multi‑session RL can accommodate workflows where execution no longer follows a single, continuously accumulated context prefix. Context compaction and subagent forking exemplify such discontinuities, as seen in modern agent harnesses like Claude Code. Systematically examining mechanisms that disrupt context continuity—such as dropping reasoning tokens or tool responses—will improve multi‑session RL under these conditions.
When discontinuity occurs, the choice between introducing a new session boundary and applying lossy concatenation with the preceding session can significantly affect optimization effectiveness.
Implementation Details and Hyperparameters
Implementation specifics, hyperparameters, and evaluation metrics for multi‑session rollouts.
The magnitude comparison relies on the $\lambda$₍rrs₎ correction from Eq. 31; any remaining opposing correction cannot flip the sign of the trajectory‑level advantage, only reduce its magnitude to zero.
The two gates address complementary failure modes: the first blocks a positive shaping signal from reinforcing an all‑failure session, while the second preserves the direction of any non‑zero base advantage, allowing finer local adjustments.
**Table 3 | Hyperparameters and implementation settings.**
This table lists various hyperparameters categorized by their function, including Rollout, Training schedule, GRPO, KL regularization, Exploration regularization, Optimizer, Compact only, Subagent only, LLM judge, IGPO only, and Distributed training. Each entry specifies the hyperparameter name and its corresponding value or setting.