Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

Maximo Eduardo Rulli, Thomas Vaitses Fontanari, Simone Petruzzi, Federico Alvetreti, Giorgio Strano, Donato Crisostomi, Giorgos Nikolaou, Tommaso Mencattini, Andrea Santilli, Emanuele Rodolà, Simone Scardapane, Alessio Devoto

Diffusion Language Models internally track denoising progress via a steerable, low-dimensional latent signal.

Do Diffusion Language Models (DLMs) implicitly track their denoising progress, and can we use this latent signal to steer their generation?

Diffusion Language Models (DLMs) generate text by iteratively denoising masked tokens, yet they lack explicit timestep conditioning, leaving it unclear if they internally track their own progress. The authors identify a latent signal related to the denoising step ($\tau$) within the residual stream, which they extract using MLP probes and characterize as a coherent, low-dimensional subspace. Steering the model along this subspace predictably modulates generation confidence and entropy, confirming the signal is functionally relevant to the model's internal dynamics.

Paper Primer

The authors treat the fraction of unmasked tokens as an empirical proxy for denoising progress. By averaging hidden states across sequences at each denoising step, they isolate mean activation vectors that capture a robust, global representation of this progress across the model's depth.

The denoising-progress signal is causally active and steerable.

Intervening on the identified latent subspace shifts model confidence and entropy in directions consistent with the target denoising step, while random perturbations of equal norm produce negligible effects. Steering produces predictable changes in model entropy and confidence, with KL divergence scaling proportionally to the distance between the current and target denoising steps.

The model exhibits a "depth-correction" mechanism: when the signal is perturbed in early layers, the model progressively recomputes and recovers the original denoising-progress representation in deeper layers.

Why does this signal matter if the model is already generating text?

The signal is not merely descriptive; it is functionally entangled with the model's token distribution. Steering it allows for systematic control over generation behavior, such as modulating confidence or triggering premature sequence termination.

Is this representation consistent across the entire model?

Yes, the signal exhibits a shared, low-dimensional parabolic geometry across layers. While individual layers may implement it differently, the representation is consistently organized within a subspace that captures over 90% of the variance.

The Latent Clock in DLMs

We expose how diffusion language models internally track denoising progress without an explicit timestep.

Diffusion Language Models (DLMs) generate text by repeatedly denoising masked tokens, yet they receive no explicit timestep input. Consequently, the models must infer how far along the denoising trajectory they are solely from internal signals.

$\tau$ is the model’s internal estimate of how many denoising steps remain, inferred from the residual activations even though no clock is provided.

How does $\tau$ differ from an explicit timestep conditioning that some diffusion models use?

Explicit conditioning supplies the exact step value as an input token, whereas $\tau$ is inferred post‑hoc from the model’s own hidden state. The former tells the model “where you are”; the latter lets the model “figure out where it is” from its internal dynamics.

**Figure 1:** 3D projection of latent denoising step modelling for LLaDA. LLaDA represents its $\tau$ subspace as a low-dimensional manifold-like curve, progressively modelling the denoising progress from all [MASK] (red) to no remaining [MASK] (purple).

DLMs lack explicit timestep conditioning, yet must track progress to succeed.

Diffusion Language Models

Background explains DLM masking, training loss, inference unmasking, and introduces LLaDA and Dream.

Diffusion Language Models (DLMs) are trained to reconstruct a clean token sequence from a partially masked version, where each token is independently replaced by a special [MASK] with probability $s$.

LLaDA is a lightweight decoder that extracts a latent representation of the denoising trajectory without running the full diffusion process.

Dream is a steering module that nudges the DLM’s hidden states so that the decoded sequence follows a desired denoising‑progress trajectory.

During inference the model runs $T$ denoising steps; at each step $t$ it predicts distributions for the currently masked tokens and then unmaskes a subset according to a policy that determines how many tokens are revealed.

We address three questions: first we probe the residual stream for the latent signal $\tau_t$, then we causally steer it with Dream, and finally we evaluate how well steering improves downstream generation.

Probing the Latent Signal

We probe each layer’s residual stream with MLPs to recover the denoising‑progress signal $\tau$.

To answer RQ1 we ask whether the latent denoising‑progress signal $\tau$ can be read directly from a model’s residual stream. The approach is to attach a lightweight MLP probe to each layer and train it to predict the current $\tau$ value.

Each token’s hidden vector in the residual stream already contains a compressed hint of the global denoising step, so a tiny classifier can extract $\tau$ from a single vector.

Feed $h$ into the layer‑1 MLP $\phi_1$, which consists of a linear layer $W\in\mathbb{R}^{1\times4}$ and a sigmoid activation.

Assume $W = (0.8,\; -0.5,\; 0.3,\; 0.2)$. Compute the pre‑sigmoid score $s = W h = 0.8\cdot0.2 -0.5\cdot(-0.1) + 0.3\cdot0.05 + 0.2\cdot0.3 = 0.16 + 0.05 + 0.015 + 0.06 = 0.285$.

Apply sigmoid: $\phi_1(h) = \sigma(s) = 1/(1+e^{-0.285}) \approx 0.57$.

Compute squared error $(\tau - \phi_1(h))^2 = (0.42 - 0.57)^2 \approx 0.0225$.

The probe can produce a reasonable $\tau$ estimate from a single 4‑dimensional vector, illustrating that the residual stream already carries the global denoising signal.

How does this residual‑stream probing differ from the linear probes commonly used to extract semantic features in vision models?

Standard linear probes are trained on a batch of activations and often rely on a shared classifier across layers. Here each probe is a dedicated MLP per layer, trained on a single token’s vector without any batch‑level context, so the signal it extracts must be present locally rather than emerging from population statistics.

**Figure 2.** MLP probes recover the $\tau$ signal. The $R^2$ coefficient (higher is better) degrades as we probe deeper into the model, to the point where the MSE almost reduces to half the variance of $\tau$. Moreover, both [MASK] and non-[MASK] tokens seem to carry this information, attaining similarly high coefficients.

MLP probes achieve $R^2 > 0.5$ across every layer, confirming that $\tau$ is recoverable from individual token representations.

Across LLaDA and Dream, the heatmap in Figure 2 shows $R^2$ values ranging from 0.59 to 0.95, with all layers exceeding the 0.5 threshold.

Steering the Denoising Process

Evaluates how to extract and steer the denoising‑time signal $\tau$ using mean activation vectors.

Having shown that the residual stream carries a latent denoising‑time signal, we now ask whether that signal can be isolated and used to steer generation.

By averaging hidden states across many examples at the same denoising step, instance‑specific noise disappears and the systematic $\tau$ direction remains.

Sample 1: (1.0, 0.5), Sample 2: (0.9, 0.6), Sample 3: (1.1, 0.4), Sample 4: (1.0, 0.5).

Average across the four samples: $\mu_{2,1}$ = ((1.0+0.9+1.1+1.0)/4, (0.5+0.6+0.4+0.5)/4) = (1.0, 0.5).

Repeating for all bins yields a sequence of mean vectors that moves linearly from (0.8, 0.3) at t = 0 to (1.2, 0.7) at t = 3, illustrating a monotonic drift aligned with $\tau$.

The monotonic shift shows that the $\tau$ direction is encoded as a low‑dimensional linear trend across the mean vectors, which can be targeted for intervention.

Figure 3 visualises this trend: each colored line (one per layer) plots $\phi$ₗ($\mu$₍t,l₎) against the normalized timestep t/100, and the lines hug the diagonal, confirming a near‑perfect linear relationship.

**Figure 3.** $\phi_l(\mu_{t,l})$ and $\mu_{t,l}$ are highly correlated. For each of the found $\mu_{t,l}$ we compute $\phi_l(\mu_{t,l})$ and plot it against the corresponding $\tau = t/100$. We observe high correlations for both LLaDA (left) and Dream (right).

We replace the current mean vector at step t with the mean vector of a target step \hat{t}, thereby nudging the hidden state along the $\tau$ direction while leaving orthogonal information untouched.

How does Mean Vector Steering differ from simply adding a bias term to the hidden state?

Adding a bias would alter every component of the hidden state indiscriminately, destroying information orthogonal to $\tau$. Mean Vector Steering first removes the existing $\tau$ component ($\mu_{t,l}$) and then injects only the target $\tau$ component ($\mu_{\hat{t}$,l}), leaving the orthogonal subspace untouched.

Figure 4 compares the deterministic steering (blue) with the random‑direction baseline (red) across three downstream metrics: entropy change $\Delta$\bar{S}_t, confidence change $\Delta$\bar{c}_t, and KL‑divergence \overline{KL}_t.

**Figure 4.** Steering on layer 29 downstream effects. We steer the activations of layer 29 using Equation (4) targeting different $\hat{t}$ values (blue), and compare the resulting entropy, confidence, and KL-divergence variations against norm-matched Equation (5) perturbations (red). Steering along the found $\tau$ directions produces systematic and interpretable effects.

Mean activation vectors expose a robust $\tau$ direction, and swapping those means provides a clean, controllable way to steer diffusion language models.

Downstream Steering Effects

Steering $\tau$ consistently shifts confidence, entropy, and token distribution, confirming causal control.

The paper’s central premise is that a latent signal $\tau$ tracks denoising progress; here we test whether explicitly steering that signal changes downstream behavior. We compare clean token distributions $p_{j,t,n}$ with their steered counterparts $\tilde{p}_{j,t,n}$ across confidence, entropy, and KL divergence.

$\tau$ steering induces roughly twice the KL divergence of random perturbations.

Random perturbations of matched norm produce KL divergence about ½ of that caused by $\tau$‑based steering across all tested steps.

**Figure 5.** Average probe drift across layers and steps after steering in LLaDA. The model progressively compensates for the perturbation, reducing the discrepancy between clean and steered representations of $\tau$.

Steering shallow layers (e.g., layer 6) yields effects indistinguishable from the random baseline, suggesting early perturbations are attenuated by later computation. Probe‑drift measurements confirm this: the average deviation between clean and steered $\tau$ predictions shrinks toward zero in deeper layers. This null result aligns with the PCA finding that most variance resides in a low‑dimensional subspace.

Steering $\tau$ consistently shifts model confidence and entropy, confirming causal control over the denoising process.

Geometric Characterization of τ

We reveal that the steering signal lives in a few principal directions, enabling low‑dimensional control.

Having shown that the latent clock $\tau$ reliably steers model outputs, we now ask how the underlying representation is organised inside the network.

The mean‑vector representations across layers lie near a thin manifold that can be captured by just a handful of principal components.

How does this “low‑dimensional manifold” differ from a standard PCA analysis of hidden states?

Standard PCA merely describes variance; here the principal components are shown to *carry* the steering signal. Restricting interventions to the top‑k components preserves the causal effect on $\tau$, whereas perturbations orthogonal to that subspace have no systematic impact.

**Figure 6.** PCA distribution and sampled projection on the top-2 principal components of the mean vectors. Left: PCA distribution of the mean vectors for LLaDA; notably, across layers, most of the variance of the identified set of mean vectors can be explained by fewer than three dimensions. Right: sampled projections for Dream. We use the mean vectors obtained from layer 25 of Dream. We observe a parabola-like geometry.

**Figure 7.** Low-dimensional subspace steering in LLaDA. Using Equation (7), we steer the model within the two-dimensional subspace ($k = 2$) spanned by the top principal components of the layer-29 mean vectors. Steering within the subspace closely resembles the unrestricted one, while the orthogonal perturbation produces incoherent effects.

**Figure 8.** The shared 2-dimensional geometry of $\hat{t}$ in LLaDA. We compute an average trajectory by taking the standardised 2D PC projections of the mean vectors across the model. The parabolic geometry is shared across layers. PC1 and PC2 are standardised to unit variance.

Cross-Layer Representation

We reveal a shared 2‑D trajectory of mean vectors and examine its consistency across layers.

Many layers display a similar parabolic shape in their mean‑vector components. To capture this, we posit a model‑level 2‑D trajectory $f(t)\!:\![0,100]\!\to\!\mathbb{R}^2$ that traces the top two principal components across the entire network. By aggregating standardized 2‑D projections from every layer and averaging them per index $t$, we obtain a smooth, non‑parametric curve.

Figure 8 (not shown here) confirms that the empirical points cling to the proposed curve, with error bars staying within roughly 0.1, indicating the trajectory faithfully captures the model‑wide geometry. This implies that the principal components of the mean‑vector distributions share the same parabolic form, differing only by scale and offset.

Having established a shared geometric pattern, we ask whether this representation is encoded consistently throughout depth or emerges separately in each layer.

To answer this, we compute the average pairwise cosine similarity of centred vectors for each index $t$ across all layers. Figure 9 shows that in LLaDA most layers align strongly, except layer 32 which is nearly orthogonal; Dream, by contrast, exhibits blocks of aligned layers separated by largely uncorrelated regions.

**Figure 9:** Average cosine similarity of same-indexed vectors across layers in LLaDA. We observe that most layers maintain a highly correlated representation of the same index $t$, and that this relationship degrades with distance. Layer 32 instead maintains a representation that is largely independent of the other layers.

Next we probe the internal computations of a single layer using the same probing technique as Equation (3). Measuring cosine similarity of same‑dimensional representations inside the model shows that operations before and after the MLP remain highly correlated, while the MLP’s internal representations of $\tau$ are almost orthogonal, except for the post‑up‑projection matrix and the post‑activation, which stay correlated.

**Figure 10.** Average cosine similarity across layer components in LLaDA. Where $h' := h_l + \text{Self-Attention}(\cdot)$, and $\hat{h} := \text{RMSNorm}(h')$. We clearly see how across most of the internal representations there is high-correlation or almost complete orthogonality. We note how self-attention and the outputs of the out matrix seem to be anticorrelated.

Figure 11 zooms in on the self‑attention versus post‑$W_{\text{out}}$ vectors, exposing a clear transition around $t=50$: indices below 50 are positively aligned, those above 50 are negatively aligned, and the boundary region is near orthogonal.

**Figure 11.** Average cosine similarity of self-attention and post-$W_{out}$ vectors. Layer-29 vectors from LLaDA reveal a clear geometric organization over $t$: representations for distant indices become nearly anti-aligned, and a transition region emerges around $t = 50$, where vectors are approximately perpendicular.

Prior work (Zhou et al., 2026) reports that concepts are steerable only at specific denoising stages, whereas our findings show the $\tau$‑related signal persists and is recomputed throughout diffusion. Rossi et al. (2026) focus on length‑awareness via downstream performance, contrasting with our characterization of an internal representation tied directly to denoising progress.

Additional Probing Results

Additional probing and steering experiments reveal layer‑wise signal accessibility and robustness across lengths.

MLP probes on Dream achieve high R² throughout most of the model, but their accuracy declines noticeably in the deepest layers.

Linear probes follow $\tau$ closely in early and middle layers, yet their performance drops more sharply than MLP probes and eventually collapses to the baseline of predicting the mean $\tau$ in the final layers.

Across the broad mid‑to‑deep range, MLP probes retain markedly higher R², indicating that the $\tau$ signal is still present but becomes less linearly accessible.

The choice of training tokens influences probe performance: for LLaDA, restricting training to non‑[MASK] tokens widens the gap relative to using [MASK] or all tokens, whereas Dream shows little sensitivity to the token subset.

Mean‑vector steering on Dream at layer 25 reproduces the trend seen in LLaDA: steering toward later denoising steps lowers entropy and raises confidence.

Steering experiments across a range of intervention layers and target bins $\hat{t}$ show consistent downstream effects for both LLaDA and Dream.

Varying the generation length to $L=64$ and $L=128$ (with matching denoising steps) yields the same steering trajectory, confirming that the observed phenomena are not artifacts of a specific length.

On downstream benchmarks, LLaDA’s performance remains within a few points of its base score across all $\hat{t}$ values, while Dream’s performance drops sharply on GSM8K when $\hat{t}=100$.

Steering Depth Analysis

Limits of linear probing and depth‑wise steering in DLMs.

Diffusion Language Models embed a latent denoising‑progress signal $\tau$ in their residual stream; this signal can be decoded and causally steered.

Linear probes lose predictive power deeper in the network, eventually collapsing to the mean‑$\tau$ baseline for both LLaDA (top) and Dream (bottom). In contrast, MLP probes retain most of the signal until the final layers.

**Figure 13.** $R^2$ coefficients of MLPs in Dream. The $R^2$ coefficient slightly degrades as we probe deeper into the model, which remains consistent with the observations of Figure 2.

**Figure 14.** **Mean-steering downstream effects on layer 25 of Dream.** We steered the activations in layer 25 using the mean-ratio vectors (blue) targeting different $\hat{t}$ values, and measured the variation in entropy, confidence and the KL divergence. We compared it against random perturbations (red). Steering with the mean vectors has an effect that is consistent with what would be expected.

```python def block(d_in, d_out): return nn.Sequential( nn.Linear(d_in, d_out), nn.LayerNorm(d_out), nn.GELU(), ) class MLPProbe(nn.Module): def __init__(self, dim, bounded=True): super().__init__() w = min(dim, 1024) self.inp = block(dim, w) self.middle_in = block(w, w) self.expansion = nn.Sequential( block(w, 2 * w), block(2 * w, w), ) self.middle_out = block(w, w) self.out = nn.Linear(w, 1) self.bounded = bounded def forward(self, x): h = self.middle_in(self.inp(x)) e = self.expansion(h) h = self.middle_out(e) + h # residual out = self.out(h) if self.bounded: return out.sigmoid() return out ```

Table 1 quantifies the impact of mean‑steering on downstream tasks. For LLaDA‑1.5 the scores remain stable across $\hat{t}$ values, while Dream‑7B suffers a sharp drop on GSM8K and HumanEval as $\hat{t}$ approaches 100.

Qualitative inspection of Dream’s failed GSM8K generations reveals that steering toward $\hat{t}=100$ causes premature emission of end‑of‑sequence (EOS) tokens. Because the model is trained to keep emitting EOS once it appears, an early EOS collapses the output into a degenerate, EOS‑filled sequence.

This behaviour suggests that the $\tau$ signal is not a content‑agnostic progress counter; it is entangled with the token distribution, amplifying the probability of completion tokens such as EOS as denoising advances.

Low‑dimensional steering (Section D) shows that a 1‑D projection of the mean vectors already reproduces the downstream impact of full‑vector steering. Expanding the subspace to 2‑D yields almost identical effects, indicating that the 2‑D parabola captures the majority of $\tau$‑relevant geometry.

Dimensions orthogonal to $\tau$ (denoted $\Delta t arrow \hat{t}_{l,\perp}$) exhibit negligible influence on model behaviour, confirming that most of the steering power resides in the identified low‑dimensional subspace.

Depth‑correction analysis (Section E) plots probe drift $\bar{\delta}_l$ for each denoising step. Early steps produce large absolute drift that propagates up to layer 31, whereas later steps’ drift is quickly damped within a few layers above the injection site.

Moreover, the residual drift decreases monotonically as the steered step becomes more compatible with $\hat{t}=100$, i.e., as $t$ approaches the end of the denoising trajectory.

Dream Model Analysis

Dream’s mean activations reveal structured low‑dimensional geometry and heterogeneous layer organization.

We extend the analysis of Dream’s mean activations, mirroring the LLaDA results presented earlier. The focus is on uncovering hidden geometric and alignment patterns that characterize the latent denoising signal $τ$.

**Figure a.** The shared 2-dimensional geometry of $\hat{t}$ in Dream. We compute an average trajectory by taking the standardised 2D PC projections of the mean vectors across the model. The parabolic geometry is shared across layers, similarly to LLaDA. PC1 and PC2 are standardised to unit variance.

**Figure b.** 3D visualisation of the PCA projection of the mean vectors for layer 25 of Dream. As in LLaDA, we again observe a low-dimensional ordered structure. Notably, two points, corresponding to $t = 1$ and $t = 100$, lie far from the rest of the geometry, suggesting a special representation.

Dream’s $τ$ representations align heterogeneously across layers, partitioning the network into three computational stages.

Cosine‑similarity analysis shows high similarity within blocks (layers 1–6, 7–20, 21–27) and weaker similarity between them.

Within each Dream layer, the mean‑vector components are more orthogonal than in LLaDA, yielding three high‑similarity blocks that mirror the computational stages.

Cosine similarity across pre‑MLP components remains high, while MLP components split into a gate path and an up‑projection + Hadamard product path; self‑attention and MLP vectors are anticorrelated.

Mean‑vector steering at various layers induces downstream drifts in entropy, confidence, and KL divergence that depend on both the intervention layer and the target denoising bin $\hat{t}$.

Interventions at layers 1, 25, 50, 75, 100 produce measurable changes in the three metrics across denoising steps, with larger effects for deeper interventions and extreme $\hat{t}$ values.

Steering using only the top‑1, top‑2, top‑4, or top‑10 principal components of the mean‑vector space yields effects comparable to steering with the full mean vector.

Experiments show negligible differences in downstream metric changes when restricting steering to these low‑dimensional subspaces.

**Figure 15.** Linear probe performance. The linear probes degrade markedly as we probe deeper into the model—more so than the MLP probes shown in Figure 2 and Figure 13—and fall to the mean-$\tau$ baseline in the final layers for both LLaDA (top) and Dream (bottom). This indicates that, at the token level, $\tau$ information becomes increasingly hard to decode linearly with depth, while an MLP still recovers much of it until the last few layers.

**Figure 16.** Probe drift across layers and denoising steps when steering at layer 6. Each column shows the absolute drift $\bar{\delta}_l$ between clean and steered probe predictions at a given denoising step $t$, resolved across probe layers. We can observe how only for $\hat{t} = 100$ the perturbation at some steps penetrates all across the model and achieves a distortion at the last layers.

Conclusion

We wrap up findings, note limits, relate work, and address ethics.

We characterise a denoising‑progress‑related signal in the residual streams of DLMs. It is probe‑decodable across layers and token types, causally steerable via mean‑vector directions, repeatedly corrected across depth, and lives in a structured low‑dimensional subspace.

Steering this representation yields predictable changes in entropy, confidence, and $KL$ divergence, demonstrating that the recovered directions are functionally relevant rather than merely descriptive.

A natural next question is how this signal is computed. Our results suggest DLMs track information closely tied to the fraction of unmasked tokens, but the exact circuit remains unknown.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers