Knowledge–Geometry Decoupling: Refreshable Pretrained Transfer for Streaming Recommendation

Zixuan Wang, Yuhong Chen, Yuxuan Zhu, Guidong Lei, Zhiluohan Guo, Yu Zhao, Kun Wang, Bangyang Hong, Kangle Wu, Yabo Ni, Anxiang Zeng, Cong Fu, Hui Li

KGD decouples pretrained behavioral knowledge from task-specific geometry to enable stable, refreshable transfer in streaming recommenders.

How can we decouple pretrained knowledge from task-specific geometry in streaming recommenders to prevent catastrophic forgetting and gradient interference during continuous updates?

Industrial recommenders rely on pretraining, but continuous data drift forces frequent model refreshes that overwrite or invalidate previously learned task knowledge. Knowledge–Geometry Decoupling (KGD) solves this by separating the pretrained encoder from the task learner, using Behavioral Multi-Token Prediction (BMTP) to learn clean, transition-based knowledge and Anchored Calibration Residuals (ACR) to write task-specific geometry orthogonally. This architecture allows the encoder to refresh daily without interfering with downstream task adaptation, outperforming standard transfer baselines by 4–12% on public benchmarks and lifting key business metrics in production.

Paper Primer

Standard pretraining treats every adjacent item in a user sequence as a dependency, which injects noise when sequences span multiple interest sessions. BMTP filters these transitions, supervising the encoder only on collaboratively or semantically related items to build a stable, denoised behavioral geometry.

KGD is a two-layer system: the encoder owns the behavioral knowledge, while the task learner owns the discriminative geometry. The task learner reads the encoder's states via read-only cross-attention and writes its own geometry as an orthogonal residual (ACR) anchored to the pretrained embedding, ensuring task gradients never corrupt the base model.

KGD sustains performance gains over long-term streaming data where traditional transfer methods fail.

A 90-day production study on Shopee Homepage Search shows KGD maintains its advantage while baselines show no gains, resulting in a 1.75% lift in GMV per user and 1.53% in advertising revenue. 4–12% improvement over strong pretrain-transfer baselines across eight public benchmarks.

Why does the paper argue that standard fine-tuning is insufficient for streaming recommendation?

Fine-tuning forces a single parameter set to satisfy two conflicting objectives: the pretrained behavioral objective and the task-specific discriminative objective. This conflict causes task gradients to overwrite pretrained knowledge, effectively destroying the model's learned structure during every refresh cycle.

How does KGD avoid the latency overhead typically associated with complex multi-module architectures?

KGD maintains the same per-request latency as single-backbone baselines because the task learner's cross-attention mechanism operates on the encoder's existing hidden states without requiring additional sequential inference passes.

For streaming systems, KGD demonstrates that decoupling knowledge acquisition from task adaptation is the necessary primitive for maintaining model performance under continuous distribution drift.

Abstract

We introduce Knowledge–Geometry Decoupling and Behavioral Multi‑Token Prediction to enable continual, non‑destructive adaptation for streaming recommendation.

Industrial recommenders increasingly follow a pretrain‑then‑transfer pipeline, but behavioral distribution drift creates two core challenges: deciding what behavioral knowledge to capture from user sequences, and determining how to transfer that knowledge while the pretrained model is continuously refreshed.

We answer these questions with Knowledge–Geometry Decoupling (KGD). The framework introduces Behavioral Multi‑Token Prediction (BMTP), a pretraining objective that filters supervision to only collaboratively or semantically related future items, producing cleaner, more transferable behavioral knowledge. KGD then assigns pretrained knowledge to a refreshable encoder and task‑specific geometry to a read‑only cross‑attention layer, while an Anchored Calibration Residual (ACR) writes geometry orthogonal to the pretrained embedding.

Empirically, KGD yields 4–12 % gains on eight public benchmarks and maintains its edge over a 90‑day production stream where baselines stagnate. Deployed at Shopee, a live A/B test shows a 1.75 % lift in GMV per user and a 1.53 % increase in advertising revenue. The implementation is open‑sourced for reproducibility.

The Challenge of Streaming Recommendation

We expose why standard fine‑tuning falters under continual drift and frame the problem we address.

Pretraining a large model on abundant data and then transferring it to a downstream task has become the default recipe in language and vision. In recommendation systems, however, the underlying behavioral distribution shifts continuously, so the pretrained encoder must be refreshed throughout the service lifecycle.

In a streaming recommender, both the item catalog and user interests evolve, breaking the assumption that adjacent tokens belong to a single coherent intent.

These observations raise two fundamental questions for streaming recommendation: (Q1) what knowledge should be learned, and (Q2) how that knowledge can be transferred while the representation is continuously refreshed.

**Figure 1.** Two challenges motivating KGD: (a) Unrelated sessions (gear vs. pants) with weak similarity lead to noisy next-token prediction. (b) Pretraining and task gradients conflict on shared parameters. Details are provided in Appendix B.

Standard fine‑tuning fails in streaming production because it forces a single parameter set to simultaneously learn stale pretrained knowledge and constantly shifting task geometry.

System Constraints and Preliminaries

We outline the streaming recommendation setup and introduce Knowledge–Geometry Decoupling to separate knowledge from geometry.

In large‑scale recommender systems, user histories often contain unrelated item subsequences (e.g., “bicycle gear” followed by “pants”), and naïvely treating all adjacent items as dependent injects noisy signals into the model. Existing denoising attention or LLM‑based relabeling either only down‑weight irrelevant items or require prohibitive per‑sequence inference.

KGD splits the model into a frozen knowledge layer that preserves pretrained behavior and a trainable geometry layer that reshapes representations for each downstream task.

How does KGD differ from standard adapter‑based fine‑tuning?

Standard adapters keep the backbone partially trainable, so gradients from the downstream task still modify pretrained weights and can overwrite knowledge. KGD freezes the entire pretrained backbone, routing all task‑specific updates through a separate adapter, which guarantees that knowledge parameters never change during streaming refreshes.

We formalize the streaming recommendation problem as follows: let $U$ be the set of users and $I$ the item catalog. Each user $u\in U$ has an ordered behavior sequence $S_u=(i_1,\dots,i_T)$ with $i_t\in I$. A transformer encoder $f_{\theta}$ maps a prefix $(i_1,\dots,i_t)$ to a contextual vector $h_t\in\mathbb{R}^d$, and each item $i$ has a static embedding $e_i\in\mathbb{R}^d$ stored in a table $E$.

Downstream tasks (e.g., click‑through‑rate prediction) train on curated supervision that defines a discriminative geometry different from the pretraining one, and the system must be refreshed repeatedly as new behavior streams arrive. Our method ensures that each refresh updates only the geometry adapters while the frozen knowledge layer remains stable, satisfying the continual‑learning requirement of streaming recommendation.

Behavioral Multi-Token Prediction

Pretraining the encoder on filtered collaborative and semantic transitions builds a clean geometry for downstream tasks.

In recommendation streams, raw adjacency mixes unrelated sessions, so a naïve next‑token loss would embed noisy co‑occurrences into the item space. The section’s trick is to filter out those spurious transitions and keep only signals that reflect lasting collaborative or semantic relations.

BMTP treats the encoder like a librarian who only catalogs items that either appear together frequently (collaborative) or share a clear semantic theme, ignoring random co‑occurrences that arise from session switches.

Collaborative axis: starting at $t=1$, the next item $i_2$ has similarity 0.2 < 0.7, so it is skipped; $i_3$ has similarity 0.9 ≥ 0.7, thus $j_{\text{col}}=3$ and $S_{\text{col}}(1)=\{i_3\}$.

Semantic axis: at $t=1$, $i_2$ reaches similarity 0.8 ≥ 0.6, so $j_{\text{sem}}=2$ and $S_{\text{sem}}(1)=\{i_2\}$.

The encoder receives two targets for position 1: $i_3$ (collaborative) and $i_2$ (semantic), and is trained to increase $\langle h_1, e_{i_3}\rangle$ and $\langle h_1, e_{i_2}\rangle$ while competing against all catalog items.

Filtering discards the noisy $i_1\!\to\!i_2$ transition that would have been a false signal from a session boundary, leaving only the durable collaborative and semantic signals.

Observation 1 shows that when the downstream task merely rescales the logits with a temperature $\tau$, the optimal embeddings differ from the pretrained ones only by a global factor $\alpha = 1/\tau$, leaving the geometry intact.

How does BMTP differ from the standard next‑token prediction used in language modeling?

Standard next‑token prediction treats every immediate successor as a target, which in recommendation mixes unrelated session boundaries. BMTP first filters successors by collaborative or semantic similarity thresholds, so the encoder only learns from transitions that reflect stable relationships, not from noisy adjacency.

**Figure 2.** Overview of KGD. BMTP pretrains the encoder by supervising collaborative and semantic transitions filtered by $\tau_{col}$ and $\tau_{sem}$. The task learner reads encoder states via read-only cross-attention and writes its own geometry via ACR.

Decoupled Read-Write Ownership

Decoupled ownership isolates pretrained knowledge from task adaptation, enabling continual refresh without interference.

When a single embedding set must serve both the pretrained transition model and a downstream discriminative task, their gradients quickly become orthogonal, so fine‑tuning erases the pretrained geometry while freezing blocks any useful adaptation.

Think of the pretrained encoder as a library catalog that never changes, while each downstream task writes its own sticky notes on a separate pad – the notes can be read but never rewrite the catalog.

Compute $s_k = 1 + \operatorname{ReLU}(0.3) = 1.3$.

Pick low‑rank factors $Z^{(k)}=[0.5,\, -0.2]$ (shape $1\times2$) and $B^{(k)}=\begin{bmatrix}0.2&0.1\\-0.1&0.3\\0.0&0.2\\0.1&-0.1\end{bmatrix}$ (shape $4\times2$).

Form the residual $\Delta e^{(k)}_{\perp}=Z^{(k)} B^{(k)T}= [0.5, -0.2]\begin{bmatrix}0.2&-0.1&0.0&0.1\\0.1&0.3&0.2&-0.1\end{bmatrix}= [0.12,\; -0.11,\; -0.04,\; 0.07]$.

Project $\Delta e^{(k)}_{\perp}$ onto the orthogonal complement of $e_{\text{pre}}$ (subtract its component along $e_{\text{pre}}$), yielding $\Delta e^{(k)}_{\perp,\text{ortho}}\approx[0.10,\;-0.12,\;-0.05,\;0.07]$.

Combine: $i = 1.3\cdot e_{\text{pre}} + \Delta e^{(k)}_{\perp,\text{ortho}} \approx [0.78,\,0.26,\,0.13,\,0.13] + [0.10,\!-0.12,\!-0.05,\,0.07] = [0.88,\,0.14,\,0.08,\,0.20]$.

The residual adds discriminative directions without rotating the pretrained vector; the final embedding keeps the original orientation (dominant component still aligns with $e_{\text{pre}}$) while gaining task‑specific nuance.

How does the Anchored Calibration Residual differ from simply adding a bias vector to the embedding?

A plain bias can point anywhere, potentially overwriting the pretrained direction. ACR first scales the pretrained vector (preserving its orientation) and then adds a low‑rank component that is explicitly forced orthogonal to $e_{\text{pre}}$, so the original knowledge remains intact and readable.

**Figure 5.** Full gradient-cosine results across 8 public benchmarks. We measure the cosine similarity between gradients from the pretraining loss and the task loss on the same parameter group under joint optimization.

Implementation and Training

Describes how KGD is instantiated, trained daily, and served with Adapter‑based Cross‑Attention.

When a single model must host both pretrained knowledge and rapidly evolving task signals, gradient interference degrades performance, especially in streaming recommendation where data arrives continuously.

ACR inserts a lightweight adapter that reshapes the raw item embeddings before they re‑enter the frozen encoder, allowing the downstream learner to read the encoder’s representations without altering its parameters.

Linear projection: $E' = E\,W_{adv}^\top = \begin{bmatrix}2&2\\3&1\end{bmatrix}$.

ReLU (no negative values): $E'' = \max(0, E') = \begin{bmatrix}2&2\\3&1\end{bmatrix}$.

Feed $E''$ into the frozen encoder; suppose the encoder maps them to $\tilde{H} = \begin{bmatrix}0.5&0.8\\0.3&0.6\end{bmatrix}$.

The learner’s cross‑attention query $\rho$ (a 2‑dim vector) attends to $\tilde{H}$, yielding $H' = \text{softmax}(\rho \tilde{H}^\top)\tilde{H}$.

Even a tiny adapter can steer the encoder’s hidden states toward the task’s geometry while leaving the encoder’s internal weights untouched.

The encoder is a standard self‑attention transformer trained with BMTP, mapping raw embeddings $E$ to hidden states $H = \text{Transenc}(E)$. The task learner first applies ACR to $E$, re‑encodes the adapted embeddings to obtain $\tilde{H}$, and finally performs read‑only cross‑attention $H' = \text{Transtask}(\rho,\tilde{H},\tilde{H})$ before the task head computes the loss.

Task‑specific queries $\rho$ are instantiated per downstream need: for retrieval $\rho$ is a user‑profile vector; for ranking $\rho$ aggregates candidate features, allowing each candidate to read a shared user‑history encoding independently.

Daily refresh loop – encoder is refreshed once, learner is updated thereafter.

Serving incurs modest overhead: the learner adds roughly twice the dense parameter count and 20 % more sparse item embeddings, yet per‑request latency stays on par with a single‑backbone baseline because the query token still traverses only the shared encoder when the learner is bypassed.

**Table.** Performance comparison of different pretraining and transfer strategies across various domains (Arts, Beauty, CDs, Phones, Office, Software, Toys, Games) using N@50 and R@50 metrics. The table evaluates strategies including Scratch, TE&FT, TA&FT, TE&FE, TA&FE, TA&FD, TA&FA, and KGD, comparing NTP, MTP, and BMTP methods.

Results Overview

KGD delivers up to 33.7% AUC improvement over the strongest published baseline under a test‑before‑train evaluation.

KGD achieves up to 33.7% higher AUC than the best published baseline under the test‑before‑train protocol.

Measured on the 90‑day trajectory, the last‑day AUC of KGD exceeds the top baseline by 33.70%.

Public Benchmark Performance

KGD + BMTP dominates public and industrial benchmarks across click and order metrics.

KGD (S3) with BMTP attains the highest click AUC, click GAUC, order AUC, and order GAUC on the industrial benchmark.

Table 2 shows values 0.7867 (click AUC), 0.7826 (click GAUC), 0.9015 (order AUC), 0.8477 (order GAUC), surpassing every other listed strategy.

TE&FT freezes the pretrained encoder while fine‑tuning a lightweight adapter on streaming data, enabling rapid adaptation without overwriting core knowledge.

**Table.** Performance comparison of different strategies and pretraining objectives across click and order metrics.

Industrial Streaming Performance

KGD delivers the highest 28‑day click AUC on industrial data, confirming the ownership advantage.

KGD (S3) achieves the highest 28‑day click AUC of 0.7867, surpassing the next best baseline by +0.001.

Table 2 reports 0.7867 for KGD versus 0.7859 for the strongest competitor (TA&FE).

Ownership decoupling, not the refresh schedule alone, drives the gain: when the same daily‑refresh schedule (S3) is applied to a shared‑parameter baseline (TA&FT), performance actually drops, confirming that refreshing entangled parameters repeatedly overwrites geometry.

**Figure 3.** 90-day trajectories of click and order AUC on the industrial dataset. For readability, curves are smoothed using a Savitzky–Golay filter with a ten-day window, and the unsmoothed curves are given in the Appendix E, Fig. 10.

Scaling experiments (Table 8) demonstrate that, because knowledge and geometry are owned by separate modules, increasing encoder capacity improves the refreshed behavioral knowledge, while expanding the task‑learner boosts geometric expressiveness—benefits that cannot be realized when a single backbone must serve both objectives.

Ablation results (Table 3) confirm each KGD component’s contribution: removing semantic BMTP, collaborative BMTP, or the Adapter‑based Cross‑Attention (ACR) each lowers click AUC by 0.001–0.002, and the full KGD configuration consistently outperforms all stripped variants.

Ablation and Sensitivity Analysis

We quantify each KGD component’s impact and validate gains with a large‑scale A/B test.

Table 3 isolates the contribution of each KGD component by removing it from the full model. We report the resulting AUC changes on head‑user and tail‑user subsets.

The collaborative filter mainly protects head‑user performance, while the semantic filter is crucial for tail users. On the Amazon dataset the two effects disappear because aggressive sampling skews the user distribution.

Removing the Adapter‑based Cross‑Attention forces the task to rely on a frozen pretrained encoder, which degrades performance across all metrics.

Sharing a backbone between encoder and task head, even when fine‑tuned, yields lower accuracy than the full KGD separation, confirming that knowledge and geometry must reside in distinct parameter sets.

Matching KGD’s total parameter count with a shared backbone still underperforms, and on sparse public datasets it overfits, indicating that the gain stems from how capacity is allocated rather than its size.

**Figure 8:** Sensitivity to the collaborative denoising threshold $\tau_{col}$. The best performance is typically obtained around $\tau_{col} \in [0.4, 0.6]$, indicating that graph-based collaborative filtering should remove noisy transitions while preserving sufficient co-occurrence structure.

**Figure 9.** Sensitivity to the semantic denoising threshold $\tau_{\text{sem}}$. Performance generally peaks at high semantic thresholds, around $\tau_{\text{sem}} \in [0.8, 0.9]$, suggesting that highly content-similar items provide reliable supervision for pretraining.

We deployed KGD at the ranking stage of Shopee’s homepage search, splitting traffic by user‑ID hash into 10 % control and 10 % treatment buckets (≈10 M users each) while keeping a 10 % A/A bucket for validation. The control runs the production OneRank model; the treatment runs KGD after 1.5 months of offline pretraining and daily updates.

A one‑week reversal after rollout caused GMV to drop by ‑1.21 % and revenue by ‑1.50 %, confirming the forward gain is not a transient artifact. Training time increased from one to two hours per day, serving latency remained at 120 ms, and memory usage doubled because the read‑only learner adds a second backbone.

Empirical Evidence of Drift

Empirical evidence of drift, noisy adjacency, and gradient conflict motivates decoupling.

The central premise—that pretrained knowledge and task‑specific geometry compete for shared parameters—implies that a frozen representation will become stale in a streaming recommendation setting.

Figure 4 quantifies daily drift on the production stream by computing the Jensen–Shannon divergence ($\text{JSD}$) of item‑ID and user‑ID interaction distributions against a fixed reference day. The divergence rises monotonically over 13 days, showing that neither the item pool nor the audience stabilizes, so a single snapshot cannot remain representative.

**Figure 4.** Jensen–Shannon divergence (JSD) of daily item-ID and user-ID interaction distributions, measured both between adjacent days and relative to the initial day over the industrial stream.

Adjacency does not guarantee dependency: user sessions often concatenate unrelated interest bursts, so treating every adjacent pair as a training signal injects noise.

Gradient‑conflict evidence (Fig 5) shows that cosine similarity between $\nabla_\theta L_{\text{pre}}$ and $\nabla_\theta L_{\text{task}}$ is often weak or negative across embedding and Transformer layers, confirming that shared parameters cause interference.

Loss dynamics (Fig 6) reveal bidirectional interference: during the pretraining‑only phase the task loss stagnates or rises, while after enabling task gradients the pretraining loss climbs, showing that each objective perturbs the other.

**Figure 6.** Loss dynamics under continual pretrain-task optimization. The dashed line separates the pretraining-only stage from the joint update stage. On the left, only the pretraining loss is back-propagated, while the task loss is monitored. On the right, the task loss is allowed to update the shared parameters.

Embedding‑geometry discrepancy (Fig 7) shows that denoised pretrained embeddings form tight, category‑aware clusters, while embeddings learned from scratch on the task are diffuse and intermix categories, evidencing divergent geometric structures.

**Figure 7.** t-SNE visualization of item embedding spaces learned by pretraining and task training. Each point denotes an item, and colors indicate the category label.

Extended Experimental Results

Extended Amazon benchmark results confirm KGD’s advantage across NDCG@20 and Recall@20.

Table 9 expands the Amazon evaluation to NDCG@20 and Recall@20, showing that Knowledge–Geometry Decoupling (KGD) remains superior to all baselines and that the Behavioral Multi‑Token Prediction (BMTP) pre‑training further lifts performance.

**Table 9.** Additional results on Amazon with NDCG@20 and Recall@20.

**Figure.** Performance comparison of different methods on Click AUC (top row) and Order AUC (bottom row) over time, from week 4 to week 13.

Related Work and Conclusion

We recap KGD’s gains and place it among recent recommender pretraining approaches.

We identified two core obstacles for streaming recommendation: the next‑token objective discards useful signal as noise, and pretrained knowledge competes with task‑specific geometry on shared parameters. Knowledge–Geometry Decoupling (KGD) resolves this by separating a refreshable knowledge layer (via BMTP) from a read‑only cross‑attention path and a write‑only geometry module (Anchored Calibration Residual). Empirically, KGD outperforms the strongest baselines by 4–12 % across eight public benchmarks and lifts GMV per user by 1.75 % in a live A/B test on Shopee.

Autoregressive pretraining of user‑behavior sequences, inspired by large language models, has become a dominant paradigm in recommender systems. Recent work frames this as Generative Pretraining, training on massive behavior logs via Next‑Token and Multi‑Token Prediction to capture reusable behavioral knowledge that can be transferred downstream.

Downstream adaptation of such pretrained models follows two main routes: full fine‑tuning, which updates all parameters for the target task, and frozen or partially frozen transfer, which keeps the pretrained encoder fixed and trains only task‑specific modules. Notable examples include PeterRec’s lightweight patches, GPSD’s frozen embeddings to avoid the “one‑epoch” phenomenon, and SORT’s industrial‑scale deployment of frozen‑transfer pipelines.

Implementation Details

Implementation specifics for datasets, metrics, backbones, and KGD settings.

Public benchmark data follow the 5‑core Amazon setting; Table 4 lists eight datasets with sample counts, click and order rates, and sparsity.

**Table 4.** Statistics of eight public benchmarks.

Industrial data are summarized in Table 5, featuring roughly 13 B samples, billion‑scale items, and a 28‑day span.

**Table 5.** Statistics of the industrial dataset.

For the long‑term decay experiment we extend training by 62 days, yielding a 90‑day dataset of about 42 B samples.

Evaluation on public benchmarks uses NDCG to reward higher‑ranked relevant items and Recall to measure hit rate within the top K candidates.

Industrial offline evaluation reports AUC for point‑wise ranking and GAUC averaged over users; online A/B tests add GMV per user, CTR, CVR, and a human‑checked irrelevant‑item rate.

We build KGD on two backbones: ManCAR for public benchmarks and OneRank for industrial deployment, preserving reproducibility and leveraging the strongest online baseline.

The BMTP pretraining encoder draws collaborative signals from LightGCN item‑graph embeddings ($\tau_{c}$ol = 0.5) and semantic signals from Qwen3‑Embedding2 ($\tau_{s}$em = 0.8) using a simple concatenated text prompt.

The task learner writes geometry via the Anchored Calibration Residual, setting low‑rank dimension r to one quarter of the embedding size and adding an orthogonality regularizer to protect pretrained knowledge.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers