Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features that Drive Retrieval

Ilia Semenkov, Daria Kleeva, Ivan Dakhtin, Zarina Maksudova, Alex Ossadtchi

A physically constrained MEG decoder that maps speech retrieval performance to specific cortical sources and stimulus features.

How can we decode perceived speech from non-invasive MEG recordings while simultaneously identifying the specific cortical sources and stimulus features that drive the decoding?

Deep learning models can retrieve speech from brain activity, but their internal weights are typically "black boxes" that do not correspond to known cortical locations or neural rhythms. The authors replace standard unconstrained layers with a front-end parameterized by spherical harmonics and trainable temporal filters, forcing the network to learn representations that align with the physics of MEG sensors and the physiology of neural sources. This architecture achieves high retrieval accuracy while allowing researchers to map learned weights directly to cortical maps and identify which specific speech features—such as intensity or phonemes—drive the model's decisions.

Paper Primer

The model uses a factorized spatial-temporal front-end where each branch is matched to a specific neural source. It functions like a radio tuner: the spatial layer acts as a directional antenna pointing at a specific cortical patch, while the temporal filter acts as a band-pass filter isolating the rhythmic activity of that patch.

The model achieves high retrieval accuracy with a significantly more compact architecture than previous black-box decoders.

On the MEG-MASC dataset, the model reaches 39.75% Top-1 accuracy across 1005 candidates using 20× fewer parameters than the baseline.

The decoder relies on specific, identifiable stimulus features to perform retrieval.

Paired MEG occlusion analysis shows that 15 of 19 tested features contribute to retrieval, with silence, sound intensity, vowels, and acoustic onsets having the largest effects.

Why is it necessary to constrain the architecture rather than just using post-hoc explanation methods?

Post-hoc methods often explain latent features that do not map to the physics of the forward model (how cortical currents reach sensors). By building constraints into the architecture, the learned weights are inherently interpretable as cortical sources and temporal dynamics.

Does this physical grounding come at the cost of retrieval performance?

No; the authors report that their constrained model outperforms larger, less-constrained configurations on the same dataset, suggesting that physiological priors act as a useful inductive bias rather than a performance bottleneck.

Introduction and Motivation

Introducing a physics‑constrained, branch‑based decoder that maps MEG signals to cortical sources and speech features.

Standard MEG decoding models treat the brain as a black box, so the relationship between recorded magnetic fields and specific speech features remains hidden. This paper proposes a branch‑based interpretable architecture that explicitly maps MEG signals to cortical sources and to acoustic representations, thereby exposing the neural basis of speech perception.

Pre‑trained audio representations that capture phonetic and prosodic structure; the decoder learns to align MEG activity with these embeddings.

We modify the decoder in four ways. First, we replace the 2D Fourier spatial‑attention layer with a real spherical‑harmonics parameterization, matching the roughly spherical geometry of the MEG helmet. Second, we compress the subject‑specific latent space to $K = 25$ branches, a physiologically plausible bottleneck. Third, we attach trainable depthwise temporal filters so each branch captures a source’s dynamics. Fourth, we explicitly remove ocular and cardiac artifacts before training to prevent non‑cortical, stimulus‑locked shortcuts.

On the cleaned MEG‑MASC dataset the model reaches $39.75\%\pm0.34$ Top‑1 accuracy among 1005 candidates while using roughly twenty times fewer trainable parameters than the prior decoder. Its learned weights can be projected into source space, revealing generators that align with the canonical speech‑perception network and showing left‑hemisphere branches enriched for high‑frequency rhythmic components.

The shift from black‑box decoding to an interpretable source‑feature mapping makes MEG‑based speech decoders useful scientific instruments, not just performance benchmarks.

Biophysical Framework

Foundations for linking MEG sensor data to cortical sources and factorized neural models.

The MEG signal recorded at the sensor array is a linear superposition of contributions from many cortical dipoles plus sensor noise.

A cortical patch of synchronously active pyramidal neurons behaves like a single equivalent current dipole whose location, orientation, and time‑varying amplitude generate the magnetic field measured by the MEG sensors.

Classical MEG analysis extracts source activity by applying a spatial filter $\mathbf{w}_k^{\top}$ to the sensor time series.

Three common strategies to obtain source estimates are: (1) exhaustive search over an inverse‑operator grid (high interpretability, low sensitivity); (2) data‑driven optimal spatial filtering (efficient but limited to simple output functions); (3) end‑to‑end deep learning (powerful but opaque).

Factorized spatial‑temporal neural architectures treat each branch as a pair of a spatial filter and a temporal convolution, enabling a compact yet expressive decoder.

The number of effective branches $K$ is inferred by adding branches until decoding performance stops improving; the resulting $K$‑dimensional subspace $\hat{S}$ captures the task‑informative signal.

Subspace $\hat{S}$ can be compared to forward‑model source spaces using RAP‑MUSIC or its successors, providing a principled way to validate that the learned branches align with physiologically plausible cortical sources.

The LISA Architecture

Describes the interpretable branch‑based front‑end and its training objective.

Standard MEG decoding treats the sensor array as an opaque black box, making it impossible to link learned representations to concrete cortical sources or frequency bands. The core trick is to replace that opacity with a branch‑based front‑end that forces each latent dimension to correspond to a physically interpretable source‑frequency pair.

Each virtual channel is built by projecting the 3‑D sensor layout onto a set of real spherical‑harmonic basis functions, yielding a fixed spatial weighting that captures a specific cortical topography.

Compute each $Y_{\ell}^{q}$ for the four sensors (e.g., $Y_{0}^{0}=0.5$, $Y_{1}^{-1}= \sin\theta\sin\phi$, etc.).

Form $c_{j m}$ by summing $\gamma_{q,\ell} Y_{\ell}^{q}$ across $\ell,q$ for each sensor.

Apply softmax across the four sensors to obtain $c_j$ (e.g., $c_j = [0.40, 0.25, 0.20, 0.15]$).

Given a sensor snapshot $x(t)=[1.0, -0.5, 0.2, 0.0]$, compute the virtual channel value $SA_j(x(t)) = 0.40 \cdot 1.0 + 0.25 \cdot (-0.5) + 0.20 \cdot 0.2 + 0.15 \cdot 0.0 = 0.28$.

Even with a tiny harmonic set the spatial filter can emphasize a subset of sensors, illustrating how the branch learns a physiologically plausible topography.

How does this spherical‑harmonic attention differ from the 2‑D Fourier spatial attention used in earlier work?

Fourier attention assumes a flat sensor grid and represents spatial patterns with sinusoids defined on a plane; spherical harmonics respect the true curvature of the MEG helmet, allowing each virtual channel to capture a genuine cortical dipole pattern rather than an artifact of planar projection.

The front‑end stacks three operations—spherical‑harmonic spatial filtering, a shared 1 × 1 unmixing convolution, and a subject‑specific projection—so that each of the $K$ branches ends up representing a single cortical source in a narrow frequency band.

Compute raw branch signals: $a_{s,1}=0.6 \cdot 1.0+0.2 \cdot 0.5+0.1 \cdot (-0.2)+0.1 \cdot 0.0=0.71$, $a_{s,2}=0.1 \cdot 1.0+0.3 \cdot 0.5+0.4 \cdot (-0.2)+0.2 \cdot 0.0=0.05$.

Apply zero‑padding and convolve each $a_{s,k}$ with its kernel $h_k$ (here a single‑step illustration): $r_{s,1}=0.2 \cdot 0.71+0.5 \cdot 0.71+0.3 \cdot 0.71=0.71$, $r_{s,2}=-0.1 \cdot 0.05+0 \cdot 0.05+0.1 \cdot 0.05=0$.

The resulting branch outputs are $[0.71,\;0]$, clearly showing that branch 1 carries most of the signal while branch 2 is near‑silent for this input.

The example shows how a spatial filter can route activity preferentially to one branch, and how the depthwise temporal kernel can further amplify or suppress that activity.

Why not simply concatenate the spherical‑harmonic filters with a standard fully‑connected layer instead of using the three‑step $C\!\to\!W_u\!\to\!W_s$ pipeline?

The three‑step pipeline isolates three distinct inductive biases: (1) geometry‑aware spherical bases, (2) a shared affine mixing that reduces redundancy across participants, and (3) a subject‑specific selection that forces each branch to specialize. A monolithic fully‑connected layer would entangle these factors, making it harder to interpret which part of the weight matrix corresponds to a cortical source.

The decoder stacks dilated 1‑D convolutions with residual connections, progressively expanding the receptive field while preserving the $K$‑branch structure, before projecting to the 768‑dimensional wav2vec embedding.

Forward pass through the non‑linear decoder (B=2, K=25)

Why use dilated convolutions with residuals instead of a simple linear projection from the $K$ branches to the wav2vec space?

Dilated convolutions enlarge the temporal context without adding many parameters, letting the decoder capture long‑range dynamics (e.g., syllable‑level patterns) that a single linear map cannot represent. Residual connections keep gradients stable across the two blocks, which is crucial when training with limited MEG data.

**Figure 1.** Interpretable front-end made as a collection of branches, the $k$-th branch is highlighted in blue. During training each branch gets matched to a neural source with specific spatial and dynamical properties.

**Figure 2.** Our network's architecture for MEG-to-audio embedding alignment. A 3-second, 208-channel MEG segment is processed by an interpretable front end: spherical-harmonic 3D spatial attention maps the sensor signals to 270 geometry-constrained virtual channels; a 1 × 1 unmixing convolution applies a learned affine transformation in this channel space; the subject-specific layer then projects the representation to K interpretable branches selected by the subject ID. Each branch is passed through a depthwise temporal convolution with a 150 ms kernel, producing filtered branch-wise signals. These signals are processed by a convolutional module with B residual convolutional blocks, where we evaluate B ∈ {0, . . . , 5} and use B = 2 in the main architecture, followed by a convolutional head that outputs the MEG embedding aligned with the wav2vec audio embedding.

Training and Ablations

Ablation studies reveal which LISA components most affect retrieval performance.

We probe LISA’s design by systematically removing components and measuring the resulting drop in Retrieval Accuracy.

To see which neural information a decoder relies on, we swap out MEG segments that contain a target feature with matched segments that lack it, then observe how retrieval ranks change.

**Figure 12:** Architectural ablations of the network's front end. Bars show the change in Top-1 and Top-10 retrieval accuracy relative to the full model. The full spatial–temporal factorization performs best, with the largest degradation observed when subject-conditioned spatial mappings are removed.

**Figure 13.** Effect of temporal-filter support on retrieval accuracy. Points show changes in Top-1 and Top-10 accuracy relative to the 150 ms default. The one-sample condition contains no temporal context and acts only as branch-wise scaling. All models used the same training seed and validation-based checkpoint selection.

**Figure 10.** Feature-space compression of the target wav2vec representation. The feature dimension is reduced using either a fixed PCA projection or a trainable linear projection optimized with the retrieval loss. Learned feature reduction preserves retrieval accuracy over a wide range of dimensions, whereas PCA degrades substantially faster in the low-dimensional regime.

**Figure 11.** Temporal-resolution compression of the target wav2vec trajectory. Temporal PCA, trainable linear reduction, local pooling, and global pooling methods are compared. Unlike feature compression, temporal compression causes a clear performance loss, and global pooling collapses to near-chance retrieval.

How does this paired‑MEG occlusion differ from a standard feature‑ablation where the feature is simply zeroed out?

Instead of zeroing, we replace the entire MEG segment with a real donor segment that either contains the feature (matched control) or lacks it. This preserves the overall signal statistics, timing, and sensor‑space structure, isolating the informational contribution of the feature itself.

Decoding Accuracy

Key retrieval gains and spatial insights from the interpretable branch architecture.

Standard MEG decoding treats the brain as a black box; our branch‑based interpretable architecture explicitly links MEG signals to cortical sources and acoustic features.

Paired occlusion shows that silence contributes the strongest positive rank‑contrast ($\Delta$r = 75.62) among all tested speech‑related features.

$\Delta$r = 75.62, p < 10⁻⁴, positive in all 27 participants.

**Figure 3.** Retrieval accuracy as a function of the number of interpretable branches $K$ and the number of convolutional blocks in the decoder. For each configuration, Top-1 and Top-10 test accuracy are reported at the checkpoint with the lowest validation loss. Accuracy increases sharply from very small $K$ to approximately $K = 10-25$, then enters a broad plateau; larger $K$ does not produce systematic gains and can mildly degrade performance. Across decoder depths, the 0-block model is consistently weaker, while models with 2–5 convolutional blocks form a similar high-performing regime. The main 2-conv, $K = 25$ configuration lies on this compact high-accuracy plateau.

**Figure 4.** 3D spherical-harmonic attention learned by architectures with varying numbers of non-linear convolutional blocks (B) and branch counts (K = 5, 10, 25). $N_{eff} = (\sum_m p_m^2)^{-1}$ is the inverse-Simpson effective number of sensors, where $p_m = \bar{c}_m / \sum_{m'} \bar{c}_{m'}$; smaller values indicate that attention is concentrated on fewer sensors. For visualization maximal value was capped to 99-th percentile, but all $N_{eff}$ are computed with full attention weights without clipping.

**Figure 8.** Test retrieval accuracy as a function of paired MEG–audio segment duration for models with $K = 25$ branches and $B \in \{0, 2, 5\}$ convolutional blocks. Each point reports final-test accuracy from the checkpoint with the lowest validation loss. Audio embeddings were regenerated directly from the continuous sounds at every duration, and all conditions used the same 5 s-feasible anchor set and the same 991-candidate retrieval database. The 3 s points belong to this regenerated ablation and are distinct from the main 1005-candidate evaluation.

**Figure 9.** Top-$n$ test retrieval accuracy for the main two-block, $K = 25$ model across paired MEG–audio segment durations. Curves were calculated using the similarity rank of the correct audio segment among the same 991 candidates. The advantage of longer segments is present throughout the evaluated range $n = 1, \dots, 50$.

Beyond raw accuracy, the modified attention scores (average channel weight $\bar{c}_m$) become increasingly spatially structured as K and decoder depth grow, reflected in the decreasing effective sensor count `N_eff`.

Cross‑participant SVD of the 675 spatial filters uncovers bilateral temporal activity, frontal motor‑preparation patterns, and a left‑hemisphere dominant faster spectral component (~13 Hz) absent on the right, aligning with asymmetric sampling theories of auditory processing.

Feature Occlusion Analysis

Key quantitative findings on segment duration, feature compression, and model robustness.

Longer paired MEG–audio segments substantially improve retrieval, with Top‑1 accuracy rising from 14.37 % at 1.5 s to 62.20 % at 5 s.

Two‑block model Top‑1: 14.37 % (1.5 s) → 28.03 % (2.25 s) → 39.94 % (3 s) → 52.66 % (4 s) → 62.20 % (5 s); the trend holds for three decoder depths and for models with zero or five convolutional blocks.

Paired MEG occlusion measures how removing a specific stimulus feature from the MEG input changes retrieval rank, revealing whether that feature’s neural signature aids decoding.

**Figure 7.** Paired MEG occlusion effects for 19 stimulus features. For each participant, the plotted effect is the retrieval-rank difference between feature-absent replacement (“removal”) and matched feature-present replacement (“control”), after averaging donor realisations, eligible queries, and multiple sessions. Positive values indicate that preserving feature-associated MEG information retained a better rank. Grey points show participant effects and diamonds show group means. Green violins show the feature-wise sign-flip null distributions in rank-difference units; stars mark one-sided single-step max-T familywise-error-corrected p < 0.05. Every rank was computed against the complete, unchanged bank of 1005 candidates. Because the masks differed in duration and in their sets of eligible queries, effect magnitudes should not be read as a calibrated ranking of feature-encoding strength across features.

**Figure 14.** Robustness of paired MEG occlusion effects across six model initializations. Small grey points show the participant-balanced group mean rank difference for each trained model, and diamonds show the across-seed median. Numbers above each feature indicate how many models met the pre-specified positive-effect criterion: a positive mean contrast, one-sided single-step max-$T$ $p_{FWER} < 0.05$, and a valid feature-present control. Because the inferential test was directional, 0/6 for the random word-list contrast means that no model showed corrected evidence in the positive direction; it does not test the reproducibility of its negative effect. The division between panels is solely for visualization and has no inferential meaning.

Discussion and Cortical Sources

We interpret how the interpretable front end links MEG sources to speech retrieval.

We revisit the paper’s premise: standard MEG decoders treat the brain as a black box, whereas our Interpretable Branch architecture explicitly maps MEG signals to cortical sources and acoustic features.

The front end consists of multiple Interpretable Branches, each a spatial‑temporal filter tuned to a neuronal population with its own cortical location and second‑order dynamics; their outputs feed convolutional layers that generate wav2vec embeddings for the retrieval task.

Before training we removed ocular and cardiac components, because without this preprocessing the network could exploit peripheral signals—a classic shortcut‑learning failure—rather than genuine cortical activity.

Despite using roughly twenty times fewer trainable parameters than the decoder of Défossez et al., our main configuration (K = 25, two convolutional blocks) attains 39.75 % Top‑1 and 70.40 % Top‑10 retrieval, placing it in the same performance band as prior work while remaining compact and interpretable.

Ablation experiments reveal that increasing branch count to K = 270 (14.8× parameters) actually lowers Top‑1 by 3.60 % and Top‑10 by 3.14 %; swapping 3D spherical‑harmonic attention for a 2D version costs ~1 % accuracy, and removing branch‑wise temporal filters incurs a similar penalty.

Temporal support matters: a one‑tap (no context) condition is ~1 % below the 150 ms default, while extending the filter to 490 ms yields the best scores (+0.58 % Top‑1, +1.17 % Top‑10). Retrieval improves up to ~150 ms and then saturates.

From a conventional MEG viewpoint, accuracy rises with a few branches and plateaus around K ≈ 10–25; adding more channels degrades performance because task‑related activity lives in a low‑dimensional subspace of the sensor space.

We mapped the learned spatial patterns using RAP‑MUSIC dipole fitting on a shared template brain, locating dominant sources in bilateral auditory cortex and frontal lobe; the detailed maps (Figure 5) show clusters along the superior and middle temporal gyri and peri‑Sylvian cortex.

**Figure 5.** (a) First ten singular vectors calculated separately for all $L_2$-normed spatial filters and topographies. (b) Dominant dipoles derived from the cross-subject spatial patterns of Figure 5a using the RAP-MUSIC algorithm [45] (subspace correlation threshold 0.8). Dipoles cluster in bilateral auditory cortices, medial temporal lobe and frontal lobe on the fsaverage anatomy.

Temporal analyses reveal low‑frequency evoked activity dominates the clusters, with occasional alpha and lower‑beta contributions, matching prior reports of slow oscillations tracking speech envelopes.

Occlusion experiments demonstrate that the decoder leverages a broad set of speech cues—silence, loudness, acoustic onsets, phoneme classes, and high‑surprisal words—rather than a single privileged linguistic representation.

Silence and strong acoustic onsets boost Retrieval Accuracy, reflecting the model’s reliance on both sustained acoustic state and rapid changes that segment the speech stream.

At the phonetic level, vowels produce the strongest positive contrast; stops, fricatives, sibilants, nasals, liquids/glides, and schwa also contribute, though the feature masks differ in duration and coverage.

Linguistically, high‑surprisal words increase retrieval, while predictive entropy, lexical rarity, and random word‑list intervals show no positive effect; the latter even yields a negative contrast across all models.

Compressing the wav2vec feature axis from 768 to ~12 dimensions retains retrieval performance, whereas PCA degrades earlier, indicating that MEG aligns with a compact, task‑selected subspace rather than the full embedding geometry.

Global pooling of the within‑segment temporal trajectory collapses performance to near chance, underscoring the necessity of preserving fine‑grained temporal structure.

Removing the subject‑conditioned spatial mappings causes the largest accuracy drop; eliminating the attention layer costs 4–5 % points, confirming that both spatial personalization and attention are essential components.

Overall, Retrieval Accuracy should be viewed as a composite metric reflecting the integration of acoustic state, phonetic detail, and contextual surprisal within a compact subspace, not as a direct readout of linguistic content.

These findings explain why audio‑aligned objectives, which preserve pause structure and fine temporal variation, outperform text‑aligned targets on non‑invasive MEG data.

By exposing branch‑wise topographies and spectra, the Interpretable Branch front end bridges high‑performance black‑box decoding and classical neuroimaging, offering a practical tool for time‑resolved imaging on individual anatomy.

**Figure 6.** The 12 largest clusters for the main $K = 25$ model, computed using the Cable Spool Fort recordings from the first session. Each column shows the medoid of one cluster. Rows show, from top to bottom, the sensor-space spatial pattern, the temporal pattern computed using the zero-mean temporal kernel, its magnitude spectrum, and the corresponding MNE-Python [47] source-magnitude estimate on the fsaverage surface in left- and right-hemisphere lateral views.

Limitations include reliance on a single MEG corpus (27 participants) and a shared template forward model, which introduces spatial uncertainty; future work will test individual forward models and broader datasets.

Our front‑end assumes linear space‑time factorization, potentially missing inseparable cortical waves, and the paired substitution analysis cannot fully isolate causal contributions because feature masks overlap.

Subspace Correlation Scan

Supplementary algorithms, robustness checks, and model‑size comparisons.

This appendix collects the low‑level algorithm, additional validation, and full model‑size tables that support the main text.

The scan repeatedly projects out already‑selected cortical sources and measures how well each remaining source’s lead‑field aligns with the current data subspace, stopping when alignment falls below a threshold.

How does this scan differ from the classic RAP‑MUSIC source‑localisation algorithm?

RAP‑MUSIC also iteratively projects out identified sources, but it operates on sensor‑level covariance matrices and stops when the residual power falls below a noise floor. The Recursive Subspace Correlation Scan works in the MEG‑derived signal subspace, uses the cosine of principal angles as its score, and applies an explicit acceptance threshold $θ$ on alignment rather than on residual power.

**Algorithm 1** Mapping the signal subspace $S$ to cortical sources via a recursive subspace-correlation scan. 1: **Input:** data-subspace basis $G_D \in \mathbb{R}^{M \times R}$, forward-model matrix $G_M$, and acceptance threshold $\theta$ 2: **Initialise:** $k \leftarrow 0, A_0 \leftarrow [], P_0 \leftarrow I, G_{D,0} \leftarrow G_D, G_{M,0} \leftarrow G_M$ 3: **while** $k < R$ **do** 4: $\quad$ **for** $i = 1, \dots, Q$ **do** 5: $\quad \quad \rho_1^k(i) \leftarrow \max \text{subcorr}(S_{D,k}, S_{M_i,k})$ 6: $\quad$ **end for** 7: $\quad i_{k+1} \leftarrow \arg \max_i \rho_1^k(i)$ 8: $\quad$ **if** $\rho_1^k(i_{k+1}) \leq \theta$ **then** 9: $\quad \quad$ **break** 10: $\quad$ **end if** 11: $\quad k \leftarrow k + 1$ 12: $\quad A_k \leftarrow [A_{k-1}, G_M^{i_k}]$ 13: $\quad P_k \leftarrow I - A_k A_k^\dagger$ 14: $\quad G_{D,k} \leftarrow P_k G_{D,k-1}, G_{M,k} \leftarrow P_k G_{M,k-1}$ 15: **end while** 16: **Output:** selected source sites $i_1, \dots, i_k$ and their subspace correlations

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers