SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for LLMs

Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao

Supervised fine-tuning suffers from task interference, while reinforcement learning enables stable multi-task coexistence via orthogonal gradient updates.

Why does Supervised Fine-Tuning (SFT) cause task interference in multi-task LLM training, and how can Reinforcement Learning (RL) be structured to allow harmonious multi-task coexistence?

Multi-task training for large language models often leads to catastrophic forgetting, where learning a new task like coding degrades existing performance in math or logic. The authors identify that Supervised Fine-Tuning (SFT) updates are dense and overlapping, whereas Reinforcement Learning (RL) updates are sparse and approximately orthogonal. This geometric difference allows RL to decouple task optimization, enabling a "Parallel-RL" paradigm where independent task updates are merged to achieve superior performance without retraining.

Paper Primer

SFT forces models to fit fixed expert trajectories, causing dense, high-magnitude parameter updates that conflict across tasks. In contrast, RL uses advantage-based optimization that acts as a filter, confining updates to minimal, sparse changes that naturally avoid interfering with other task subspaces.

RL enables stable multi-task growth where SFT collapses.

In multi-stage training, SFT performance declines by 23.1% relative to the base model, while RL achieves a 24.9% gain. 48% performance gap between SFT and RL in multi-stage settings.

Parallel-RL effectively decouples task-specific capabilities.

Merging independent task updates via Parallel-RL retains 102.8% of single-task RL performance with only 5% additional adaptation time.

Why does RL avoid the task conflicts that plague SFT?

SFT interference is norm-limited, scaling with the absolute gradient magnitude of expert trajectories. RL interference is variance-limited, bounded by the diversity of intra-group rollouts, which filters out common-mode conflicts.

What is the core mechanism behind Parallel-RL?

Parallel-RL leverages the empirical observation that RL updates for different tasks are approximately orthogonal. This allows independent task updates to be summed or merged without significant mutual interference.

The Multi-Task Interference Problem

Why supervised fine‑tuning collapses across tasks while reinforcement learning scales.

Supervised Fine‑Tuning (SFT) and Reinforcement Learning (RL) behave oppositely when many tasks are trained sequentially. SFT’s dense updates clash across tasks, causing performance collapse; RL’s sparse, small‑magnitude updates remain nearly orthogonal, allowing tasks to coexist.

SFT adjusts all model parameters together, so each new task overwrites the improvements made for previous tasks, leading to interference.

**Figure 1.** SFT and RL exhibit significant differences during multi-stage training. SFT shows task conflicts, which impairs performance, while RL can steadily improve performance across tasks.

SFT’s dense updates prevent it from scaling across many tasks because they cause severe interference.

Empirical Evidence of Task Conflict

Preliminary evidence shows RL preserves performance where SFT collapses.

RL updates maintain or improve accuracy across tasks, while SFT updates cause large drops under Multi‑Stage training.

Table 1 shows Multi‑Stage RL gains +14.4 % versus a –23.1 % drop for Multi‑Stage SFT; overall RL average gain across strategies is +24.9 % compared to SFT.

SFT applies dense, large‑magnitude gradient steps that interfere across tasks; RL makes sparse, small‑magnitude updates that stay orthogonal.

How does RL achieve sparsity when the same model parameters are updated?

RL’s KL‑constrained objective limits each update’s L₂ norm, and the reward‑driven gradient points toward task‑specific directions. Because the KL term penalizes large deviations from the current policy, the optimizer takes only tiny steps that happen to lie in subspaces unused by other tasks, yielding near‑orthogonal updates.

**Figure 2.** Analysis of Parameter Update Dynamics across SFT and RL. The heatmaps display the pairwise cosine similarity (off-diagonal) and the $L_2$ norm magnitude (diagonal) of parameter update vectors for different tasks. Left: SFT update has a larger absolute similarity value ($\approx 1.0$). Right: RL updates show near-zero similarity, indicating orthogonal and task-specific parameter adjustments.

Mechanisms of Update Dynamics

RL updates are tiny, sparse, and orthogonal across tasks, unlike SFT.

RL updates are orthogonal across tasks, with average cosine similarity ≈ 10⁻⁵, whereas SFT updates align (≈ 0.3).

Measured pairwise cosine similarity between the parameter‑update vectors $ΔW$ for each task.

**Figure 3.** Distribution of the score function $S$ sampled during training by RL and SFT on different tasks (Math and Science). We use t-SNE to reduce the dimensionality for visualization.

Why RL Updates Coexist

We prove that RL updates are sparse and orthogonal, enabling safe additive merging across tasks.

RL‑based fine‑tuning changes only a tiny slice of a massive model, and those changes line up in nearly independent directions. This property lets us add task‑specific updates without the destructive interference seen in dense SFT gradients.

RL updates touch just a few parameters, like editing only the relevant lines in a gigantic codebase instead of rewriting the whole file.

Initial gradient vector (all zeros): $(0,0,0,0,0,0,0,0)$.

Advantage $A$ is non‑zero for actions affecting parameters 3 and 7, producing raw gradients $(0,0,0.4,0,0,0,-0.3,0)$.

Apply learning rate $ \eta = 0.1$: $\Delta W_{\text{RL}} = (0,0,0.04,0,0,0,-0.03,0)$.

Non‑zero fraction = $2/8 = 25\%$, far smaller than a dense SFT update that would modify all 8 entries.

Resulting norm $\|\Delta W_{\text{RL}}\| \approx 0.05$, an order of magnitude lower than a typical SFT norm $\approx 0.5$.

Sparsity arises because the advantage acts as a gate, allowing only reward‑relevant parameters to move.

How does this differ from the dense gradients produced by supervised fine‑tuning?

In SFT the loss weights every token equally, so $\nabla_{\theta}\log \pi_{\theta}(y|x)$ contributes to all parameters; there is no scalar gate like $A(x,y)$ to zero out most entries, resulting in a dense update.

When two tasks push the same parameter in opposite directions, their gradients cancel—like two people pulling a rope from opposite ends.

Why can’t we achieve the same orthogonality by simply zero‑masking SFT gradients?

Zero‑masking would discard useful learning signals; RL’s sparsity is a by‑product of the advantage weighting, preserving informative directions while naturally limiting overlap.

**Figure 4.** Analysis of RL training on interfering tasks. Top: Distribution of the score function $S$ sampled during RL training on math, code and game. Bottom: Corresponding performance comparison on Math (AIME25) and Code (LiveCodeBench) tasks. Training game models in parallel with math or code models can lead to significant performance losses.

The Parallel-RL Paradigm

Parallel‑RL merges orthogonal task‑specific RL updates into a single model.

Because RL updates are sparse, low‑magnitude, and empirically orthogonal across tasks, we can add them directly instead of training a single model sequentially.

GRPO computes a task‑wise gradient by averaging rollout‑wise score functions weighted by zero‑mean, standardized advantages, which cancels the dense mean direction.

How does GRPO differ from a vanilla policy‑gradient estimator?

Standard policy gradient averages raw rewards times the score function, so the mean direction $\bar{S}_i$ remains and gradients from different tasks overlap heavily. GRPO first standardizes the rewards, enforcing a zero‑sum advantage; this subtracts the mean direction and leaves only the residual score components, which are statistically independent across tasks and therefore orthogonal.

Since each task’s RL update lives in its own near‑orthogonal subspace, we can run all tasks in parallel, collect their updates $\Delta W_i$, and merge them with a simple additive rule.

Compute the dot product $\langle\Delta W_{1},\Delta W_{2}\rangle = 0.02\!\times\!(-0.01) + (-0.01)\!\times\!0.03 + 0.00\!\times\!(-0.02) = -0.0005$, essentially zero.

Apply linear averaging: $M_{\text{avg}} = (\Delta W_{1}+\Delta W_{2})/2 = (0.005,\; 0.01,\; -0.01)$.

Add to the base model: $W_{\text{final}} = W_{\text{base}} + M_{\text{avg}}$.

Because the two deltas are orthogonal, the merged update retains the full magnitude $\sqrt{0.005^{2}+0.01^{2}+0.01^{2}} \approx 0.015$, which is the root‑sum‑of‑squares of the originals.

When orthogonal, simple averaging preserves the total “energy” of the updates, so no task’s signal is lost.

Why is a plain average sufficient instead of a more sophisticated merging scheme?

Orthogonality guarantees that the inner product between any two task deltas is near zero, so the combined norm is the quadrature sum of the individual norms. Averaging therefore retains the full magnitude of each direction without destructive cancellation; more complex schemes only offer marginal gains while adding computational overhead.

**Table 3.** Quantitative comparison of score function ($S$) indicators between SFT and RL. RL exhibits significantly smaller magnitude and inter-task similarity, alongside a bounded residual $\delta$.

**Figure 5.** The Trade-off between Single-Task Exploration and Multi-Task Coexistence. When rollout temperature $\tau$ grows, we calculates the performance of Math Model, intra-group gradient variance (measured by $-\log V$), and Naive Parallel-RL accuracy on MATH500. Increasing $\tau$ improves single-task performance but also increases gradient variance, leading to higher gradient interference. As a result, the non-monotonic trend of Parallel-RL performance reveals a sweet spot that balances exploration benefits and interference costs.

Main Experimental Results

Parallel‑RL delivers strong multi‑task gains while keeping interference low.

The earlier premise noted that dense SFT updates cause gradient interference, whereas sparse RL updates can be merged additively without loss.

Adapted Parallel‑RL reaches 102.8 % of the single‑task RL benchmark while delivering a 9.4 % absolute gain over the base model.

Reported in the main results table (Table 4) where the adapted variant outperforms all baselines across the four tasks.

**Figure 6.** Distribution of the score function $S$ sampled during training by RL and SFT on different tasks (Math, Code and Logic). We use t-SNE to reduce the dimensionality of the high-dimensional score function vectors for visualization.

**Table 5.** Performance (%) ablation study of Naive Parallel-RL. $\Delta_{Target}$ and $\Delta_{Others}$ highlight the change in accuracy on the corresponding task and other tasks after removing a certain module.

Parallel‑RL achieves superior multi‑task performance without the interference that plagues SFT.

Ablation and Decoupling Analysis

Removing each task‑specific update reveals how Parallel‑RL isolates task performance.

We evaluate how removing each task‑specific update $\\Delta W_i$ during the merging process impacts performance.

Removing the task‑specific update $\\Delta W_i$ causes a 7.1 % drop on its target task.

Table 5 reports an average $\Delta$Target of 7.1 % when each $\\Delta W_i$ is omitted.

The same ablation yields a 0.6 % average improvement on the remaining tasks.

Table 5 shows an average $\Delta$Others of +0.6 % across all non‑target tasks.

These results demonstrate that Parallel‑RL isolates task representations, allowing other tasks to remain stable—or even benefit—when one task’s update is removed.

**Figure 7.** Pass@K Analysis of Naive Parallel RL with full parameter GRPO

Related Literature

We survey post‑training paradigms, SFT vs RL comparisons, multi‑task learning, and model merging.

Post‑training paradigms for large language models fall into two families: supervised fine‑tuning (SFT) and reinforcement learning (RL). Recent SFT work has moved beyond plain instruction following to incorporate Chain‑of‑Thought (CoT) prompting and specialized reasoning fine‑tuning, dramatically improving performance on mathematical and logical tasks. Parallelly, RL research has shifted toward outcome‑driven reasoning optimization, using verifiable rewards to boost capabilities in domains such as math and code.

Comparative analyses consistently highlight a functional split: SFT tends to memorize training data, whereas RL promotes generalization. Chu et al. (2025) phrase this as “SFT memorizes, RL generalizes”, while Matsutani et al. (2025) describe RL as “squeezing” policies toward high‑reward regions and SFT as “expanding” the sampling space. Additional studies report that RL can recover generalization lost after aggressive SFT, that RL updates affect only a small subnetwork, and that hybrid pipelines combine on‑policy RL with off‑policy SFT to leverage both strengths.

In multi‑task settings, SFT typically mixes diverse datasets, but severe task conflicts arise and evolve during training, prompting adaptive data selection, curriculum design, and conflict‑aware reweighting. RL‑based multi‑task research instead explores multi‑stage training, curriculum learning, mixed‑data RL that mitigates gradient imbalance, and scaling strategies for heterogeneous reasoning data. Empirical work shows that while some tasks benefit mutually, others cause negative transfer, and RL consistently exhibits far less catastrophic forgetting than SFT.

Model merging has emerged as a cost‑effective way to combine expert models. Early approaches used simple weight averaging or Task Arithmetic on task vectors, but naïve vector addition often leads to severe performance drops due to task conflicts. Subsequent methods introduce subspace pruning and sparsification—TIES‑Merging prunes based on parameter magnitude and dominant sign, while DARE randomly drops parameters and rescales the remainder. More recent techniques compute precise merging coefficients using Fisher Information or closed‑form solutions for linear layers.

Task Selection and Coexistence

Parallel‑RL’s benefits hinge on task compatibility and a trade‑off between exploration and inter‑task interference.

The central premise—that RL updates are sparse enough to merge without loss—still depends on two practical factors: whether tasks can be trained together and how much exploration each task induces.

We deem two tasks “co‑existent” when their score‑function distributions $S$ occupy distinct regions in a t‑SNE embedding; visual separation suggests the underlying neural circuits do not overlap.

As a counter‑example we merged a “Game” task (SynLogic logical‑reasoning data) with the Math and Code models. The merged models suffered noticeable drops in both directions.

The theoretical bound $|IRL(i, j)| \le V_i \cdot V_j$ (Section 4.3) tells us that intra‑group variance $V$ controls the magnitude of gradient interference across tasks.

Higher sampling temperature $\tau$ raises the entropy of the policy $\pi_\theta$, which widens the rollout distribution and thus inflates $V$. While this boosts single‑task exploration, it also tightens the interference bound.

We varied $\tau$ from 0.4 to 0.8 for the Math model (LoRA on a data subset). Accuracy rose with $\tau$, but the negative log‑variance $-\log V$ fell, indicating larger $V$. The merged Parallel‑RL accuracy first improved then degraded, producing a clear exploration–interference trade‑off.

Extended Experimental Results

Appendix E presents expanded results confirming Parallel‑RL’s superior accuracy and task decoupling.

We extend the empirical suite introduced in Section 5.2, reporting full tables for the two training regimes (LoRA‑GRPO and full‑parameter PPO) and additional analyses that probe robustness and modularity.

Adapted Parallel‑RL outperforms the Single‑Task RL baseline, reaching $103\%$ of its average accuracy.

Table 4 reports an average of $103\%$ for Adapted Parallel‑RL versus $100\%$ for the corresponding Single‑Task RL models.

For the LoRA regime, even the naïve sum‑merge (Naive Parallel‑RL) retains $98\%$ of single‑task accuracy, while the adapted version adds a $5.3\%$ gain over the Base Model. In the full‑parameter PPO regime, Naive Parallel‑RL improves $5.3\%$ and the adapted variant $9.1\%$, confirming that sparse RL updates mitigate gradient interference.

**Table 4.** The accuracy (%) on different tasks. Superscripts denote accuracy change vs. corresponding single-task baseline. $\Delta$ Base shows the average performance improvement over the Base Model. Retention (%) represents the percentage of performance retained compared to the Single-Task Training Model before merging for Parallel Training method. Bold represents the best performance.

**Table 6.** The accuracy (%) of several multi task training methods. Superscripts denote the performance gap relative to the corresponding Single-Task Baseline. $\Delta$ Base shows the average performance improvement over the Base Model. Retention (%) represents the percentage of performance retained compared to the Single-Task Baseline.

**Table 7.** Comprehensive ablation study across different model settings. Superscripts denote the performance gap ($\Delta$) relative to the full Parallel-RL model. Red indicates drop, Blue indicates improvement.

The ablation results (Table 7) show that each task’s RL‑derived update vector primarily influences its own metric, with minimal cross‑task side effects. This modularity suggests future Parallel‑RL systems could compose capabilities on demand.

Pass@k analysis (Section E.4) reveals that while Parallel‑RL may lag at Pass@1, its curves converge with Single‑Task RL by $k=16$, indicating that the merged policy retains the same reasoning ceiling as the individual task‑specific policies.

Qualitative Case Study

Naive Parallel‑RL matches task‑specific models on two illustrative tasks.

Naive Parallel‑RL attains correct answers on both the math and logic case‑study tasks, matching the task‑specific models.

On the math task it outputs $n=6$, and on the logic task it identifies both speakers as knaves—identical to the Math and Logic models.

The case study juxtaposes responses from the base model, task‑specific models, and the Naive Parallel‑RL model on two representative tasks—one mathematical and one logical. The base model fails on both, while the task‑specific models succeed. The Naive Parallel‑RL model reproduces the correct answers of the specialized models.

Experimental Configuration

Provides full dataset, training, and evaluation configurations for reproducibility.

We study four representative reasoning tasks—Math, Science, Code, and Logic—each with dedicated training and evaluation data.

For Supervised Fine‑Tuning (SFT) we use subsets of OpenR1‑Math‑220k, AM‑DeepSeek‑Distilled‑40M, AM‑Thinking‑v1‑Distilled, and the knights‑and‑knaves dataset, augmenting missing chain‑of‑thoughts via DeepSeek‑R1 API and rule‑based verification from Logic‑RL.

For Reinforcement Learning (RL) we draw from DeepScaleR‑Preview‑Dataset (Math), DeepCoder‑Preview‑Dataset (Code), the science portion of AM‑Thinking‑v1‑Distilled, and a knights‑and‑knaves subset for Logic.

Evaluation spans six benchmarks: MATH500 and AIME2025 for Math (reported as avg@16), selected MMLU subjects and GPQA‑Diamond for Science, LiveCodeBench for Code, and knights‑and‑knaves for Logic.

Training builds on DeepSeek‑R1‑Distill‑Qwen‑1.5B and 7B, run on a machine with eight NVIDIA A100 (80 GB) GPUs.

We employ LoRA adapters with rank $r = 64$ and scaling factor $\alpha = 32$ to enable efficient parameter analysis, and follow a fixed task order (Math → Science → Code → Logic) for multi‑stage training.

TIES merging is used with the default configuration and a sensitivity parameter of $0.8$.

SFT fine‑tuning is performed via the LLaMA‑Factory framework on teacher CoT trajectories, using a learning rate of $1 \times 10^{-5}$ and a sequence cutoff of $8\text{K}$ tokens.

RL training uses the VeRL framework with the GRPO algorithm, a learning rate of $3 \times 10^{-6}$, sampling temperature $0.6$, top‑p $0.95$, $G = 16$ rollouts per prompt, a maximum output length of $8\text{K}$ tokens, and the KL divergence penalty is disabled for this analysis.

We thank the developers of LLaMA‑Factory and VeRL for their extensible, open‑source frameworks that streamlined our implementation.

Evaluation primarily uses the lighteval toolkit with sampling temperature $0.6$ and top‑p $0.95$; logical reasoning is assessed with the dedicated Logic‑RL evaluation suite.

All evaluation runs are executed on the same eight NVIDIA A100 (80 GB) GPUs to ensure a consistent computational environment.

Proof of Lemma 4.3

We prove that GRPO gradient inner products depend only on intra‑group rollout differences.

Lemma 4.3 characterizes the inner product of task‑wise training gradients under GRPO. The proof demonstrates that the mean score direction cancels, leaving only the covariance of the per‑rollout deviations.

Expand the inner product of the gradients as defined in Definition 4.2.

Substitute the decomposition $S_{i,k}= \bar S_i + \delta S_{i,k}$ and apply the zero‑sum property of the standardized advantages.

Insert the simplified expressions back into the inner product and factor the $1/G^2$ term.

Taking the expectation over $x\sim D_i$ and $x'\sim D_j$ completes the proof.

Proof of Theorem 4.5

We prove Theorem 4.5’s SFT gradient‑interference upper bound using Jensen and Cauchy‑Schwarz.

Define the SFT interference term as the expected inner product of expert score functions over two independent task data distributions.

Apply Jensen’s inequality to move the absolute value inside the expectation.

Bound the absolute inner product with Cauchy–Schwarz.

Factor the expectation into a product because the samples are independent.

Square both sides to obtain the final upper bound used in Theorem 4.5.

Generalization Analysis Part I

Extends the variance‑limited interference bound from GRPO to other advantage‑normalized RL algorithms, notably PPO.

We now show that the variance‑limited interference argument, proved for GRPO, extends to any advantage‑normalized on‑policy RL algorithm.

Substituting this hierarchy back into the SFT expression and invoking the bounds $M_i$, $M_j$ from Assumption 4.4 yields the SFT interference bound (Eq. UB‑SFT).

Thus the GRPO bound (Eq. UB‑RL) holds, confirming the variance‑limited interference claim for this algorithm.

We now turn to a widely used on‑policy method, Proximal Policy Optimization (PPO), and derive the same bound under its batch‑level advantage normalization.

While the algebraic bound matches the GRPO case, the interpretation of $\delta S$ differs: in GRPO it captures intra‑prompt policy divergence, whereas in PPO it captures variation across prompts within a batch. Consequently PPO tends to be more perturbative, but because batch samples usually share a task domain the bound remains governed by the same variance terms.

Generalization Analysis Part II

Analyzes why SFT gradients stay aligned while RL gradients become orthogonal under weak coupling.

For RL updates (e.g., GRPO), the gradient is weighted by normalized advantages $\hat A_{i,k}$ that satisfy $\sum_{k=1}^{G}\hat A_{i,k}=0$ and $\sum_{k=1}^{G}\hat A_{i,k}^{2}=G$.

Equations (9) and (10) reveal a directional contrast between SFT and RL. In SFT, averaging preserves the shared dominant direction $\mu$, which yields a strictly positive lower bound on cross‑task cosine similarity. In RL, the zero‑sum advantage weighting removes $\mu$ algebraically, so the cosine similarity is governed only by the residual coupling $\gamma_{i,j}$.

Consequently, under weak residual coupling, RL gradients from different tasks become nearly orthogonal, offering an additional explanation for the observed task coexistence. This does not imply that all RL task pairs are non‑interfering; the factor $\gamma_{i,j}$ may be sizable in some cases, and assessing such situations is discussed in Appendix D.1.

Prompt Templates

Provides the exact prompt templates used for training and evaluation across tasks.

This appendix lists the exact system and user prompts employed for each of the four benchmark families during both training and evaluation.

Training prompt for Math and Science tasks

Training prompt for Logic task (Knights & Knaves)

Training prompt for Code task (LiveCodeBench)

Evaluation prompt for AIME benchmark

Evaluation prompt for MATH500 benchmark

Evaluation prompt for GPQA benchmark

Evaluation prompt for MMLU benchmark

Evaluation prompts follow the Lighteval framework

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers