Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
Zhuofan Josh Ying, Guanglong Sun, Nolan Ho, Kieran Liu, Rui Yang, Andrew Chen, Nikolaus Kriegeskorte, Pony Ma, Hossein Adeli
BrainLoRA uses parameter-efficient fine-tuning to adapt 104B-parameter multimodal models for brain encoding.
Can we improve brain encoding accuracy by fine-tuning pretrained backbones with parameter-efficient methods (PEFT) instead of just training readouts on frozen features?
Brain encoding models typically rely on frozen, task-optimized backbones because training large models on scarce neural data is computationally prohibitive and prone to overfitting. The authors use Low-Rank Adaptation (LoRA) to fine-tune large multimodal backbones, while replacing redundant sliding-window feature extraction with a single-pass causal encoding that reduces compute by 350×. This approach sets a new state-of-the-art on the Algonauts 2025 benchmark, outperforming prior models with only 5% of the trainable parameters.
Paper Primer
The core mechanism hinges on treating the backbone as a fixed prior and injecting low-rank adapter layers that learn to specialize the representation for neural prediction. By exploiting the backbone's innate causal masking, the system processes entire stimulus clips in one forward pass, enabling efficient training of models up to 104B parameters.
BrainLoRA achieves superior brain encoding performance compared to frozen-backbone baselines.
Performance gains are consistent across nine multimodal backbones (4.7B–104B parameters) on the Algonauts 2025 challenge. Outperforms the previous state-of-the-art (MIRAGE) using only 5% of the trainable parameters.
The adapted models remain interpretable and reveal increased cross-modal integration.
Feature attribution recovers canonical sensory topography (e.g., visual cortex driven by video) while showing that adaptation increases audio input contribution and cross-modal mixing within the backbone. Audio contribution increases by 39% on average across brain parcels after adaptation.
Why is this approach more practical than full fine-tuning?
Full fine-tuning of large backbones is computationally expensive and often underperforms frozen baselines due to the scarcity of neural data. PEFT updates only a small set of parameters, retaining the backbone's strong pretrained priors while allowing for specialized adaptation.
What is the role of the multi-LoRA kernel?
It enables the joint training of many adapters on a shared backbone by fusing routed updates into a single batched gather-matrix multiplication, providing a 29.8× speedup over sequential training.
Researchers can now treat large multimodal backbones as flexible foundations for brain encoding, scaling up model size while scaling down the cost of adaptation through PEFT and efficient multi-adapter kernels.
The Case for PEFT in Brain Encoding
We expose the limitation of frozen backbones and propose PEFT to adapt them efficiently.
Brain encoding models predict neural responses to stimuli, but the dominant paradigm relies on frozen pretrained backbones with only a readout trained on limited neural data. This frozen‑backbone approach leaves a large expressive capacity untapped, because the backbone cannot specialize to the nuances of brain signals. Parameter‑efficient fine‑tuning (PEFT) offers a way to adapt these powerful backbones using a tiny fraction of trainable parameters, unlocking both efficiency and performance gains.
Predicting how the brain responds to external stimuli by mapping sensory inputs to neural activity patterns.
Adjust a large pretrained model by training only a small set of additional parameters, leaving the bulk of the model unchanged.
**Figure 1.** From frozen backbones to PEFT, and three directions to scale PEFT for brain encoding. (a) The dominant paradigm: a frozen pretrained backbone with readouts trained for brain encoding. The backbone prior is strong, but it is not specialized for brain encoding. (b) We advocate using PEFT to adapt strong pretrained backbones, retaining the powerful prior while allowing flexible neural adaptation. This suggests three complementary scaling directions: Scale up to stronger priors leveraging more capable backbones; Scale down the size of trainable adapters for brain encoding with limited data; and Scale out the number of adapters with different specializations.
The key shift is moving from frozen readouts to adapted backbones via PEFT, unlocking both efficiency and performance for brain encoding.
Background and Prior Paradigms
Survey of how brain‑encoding models incorporate neural data and formal definition of the model.
Brain‑encoding research varies mainly by where the neural signal meets the model pipeline. We group prior work into three families: frozen‑backbone readout, backbone fine‑tuning, and auxiliary‑signal approaches.
Pretrained task‑optimized backbones are kept static; only a linear readout is trained on neural recordings to map backbone features to brain responses.
Neural recordings are used to adapt the entire pretrained backbone, often via parameter‑efficient fine‑tuning (PEFT) such as LoRA, rather than only training a readout.
Neural data are incorporated through auxiliary objectives, self‑supervised pretraining on brain recordings, or end‑to‑end training from scratch, often without outperforming large frozen‑backbone models.
Only the final linear mapping from backbone features to neural responses is learned; the backbone itself stays exactly as pretrained.
These definitions make explicit the three axes along which brain‑encoding models can be scaled: larger pretrained backbones, smaller adaptation dimensionality, and multiple parallel adaptations combined at inference time.
Experimental Setup and Input Design
We detail the single-pass causal masking pipeline, LoRA integration, and readout strategies that enable efficient brain encoding.
We evaluate on the CNeuroMod/Algonauts 2025 challenge, which provides whole‑brain fMRI recordings (1,000 parcels) while subjects watch seven seasons of *Friends*. The backbone is a causally masked multimodal transformer, so we can replace the costly overlapping sliding‑window extraction with a single forward pass per stimulus window.
**Figure 2** One causally masked pass replaces a sliding window. The baseline re-encodes an overlapping window per fMRI timestep; a single pass over the window gives features at every timestep.
We prevent the model from seeing any future stimulus by zero‑masking entries beyond the current timestep, ensuring that each prediction depends only on past information.
How does causal masking differ from the overlapping sliding‑window method used previously?
Sliding‑window extraction recomputes the full transformer forward pass for each overlapping segment, incurring $O(\text{window\_size} \times \text{overlap})$ work. Causal masking computes the forward pass once and extracts all timesteps from the same hidden states, removing that redundancy.
Step 1: Multiply the attention scores matrix $S$ by $M$, zero‑masking all future positions.
Step 2: Apply softmax row‑wise; each row now sums to 1 over only past and current tokens.
Step 3: Multiply the masked attention weights by the value matrix $V$, producing hidden states $H$ that contain information up to the current timestep only.
Step 4: Read out $H$ at each row to obtain predictions for $t_1$ through $t_5$ in a single pass.
The mask guarantees strict causality without needing separate forward passes for each timestep, which is why the method achieves the massive compute reduction.
Encode the entire 149 s stimulus window (100 $TR$) in one forward pass through the causally masked backbone.
Apply the pre‑computed causal mask to the attention scores, ensuring each timestep only attends to its past.
Extract the hidden representation for every $TR$ directly from the masked output.
Feed the per‑timestep representations to the readout module (linear or cross‑attention) for brain‑parcel prediction.
Optionally prepend a 60 s context lead to the window to mitigate edge effects at the start of each segment.
PEFT Performance and Mechanisms
Adapter fine‑tuning outperforms full fine‑tuning and frozen backbones across all tested models.
LoRA + cross‑attention reaches 0.327 Pearson r on the in‑distribution (ID) test set, surpassing the prior MIRAGE ensemble.
Table 1 shows the BrainLoRA (Cross‑attn) single‑model entry at 0.327 ID, higher than the MIRAGE 15‑model ensemble (0.315 ID).
LoRA injects two small trainable matrices $A\in\mathbb{R}^{d\times r}$ and $B\in\mathbb{R}^{r\times d}$ into each layer, leaving the original weight $W$ frozen while learning a low‑rank correction $AB$.
How does LoRA differ from simply adding a new linear layer on top of the frozen backbone?
LoRA modifies every existing linear transformation by a low‑rank additive term, whereas a new top‑layer only changes the final mapping. The former lets the backbone’s internal representations adapt to the task while keeping the backbone weights unchanged.
After adapting the backbone with LoRA, the readout is recomputed analytically by solving a ridge‑regression problem on the frozen adapter outputs.
Why not simply fine‑tune the readout with gradient descent instead of using the closed‑form solution?
Gradient descent would require many passes over the data and could get stuck in local minima, while the closed‑form ridge solution is exact, fast, and provably yields the lowest possible training error for a linear readout given the fixed features.
Forward pass uses $W_{\text{eff}} = W + A B$; with $A$ and $B$ zero, $W_{\text{eff}} = W$.
During adaptation, $A$ learns a matrix [[0.1,0.0],[0.0,0.2],[0.0,0.0],[0.0,0.0]] and $B$ learns [[0.5,0.0,0.0,0.0],[0.0,0.5,0.0,0.0]].
The product $A B$ equals [[0.05,0.0,0.0,0.0],[0.0,0.10,0.0,0.0],[0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0]], adding only 8 trainable parameters (2 % of $W$).
After training, the effective weight $W_{\text{eff}}$ improves the downstream task loss by 3.5 % compared to using $W$ alone.
The adapter’s low‑rank structure lets it capture task‑specific directions while leaving the bulk of the pretrained knowledge untouched.
**Figure 3** Adaptation improves performance on nine tested backbones. All backbones share the same training configuration except for the feature extraction layer (App. I). LoRA improves the performance consistently.
**Table 1.** Main results on CNeuroMod/Algonauts 2025. Validation is on Friends season 6; ID and OOD are the challenge server's scores on held-out Friends season 7 and OOD movies. Two rows within 0.00077 are not separated by seed variability (App. D). Rows marked † are cited authors' own numbers quoted from the papers. Our method BRAINLORA achieves a new state of the art.
These results demonstrate that a modest LoRA adapter plus a closed‑form readout re‑fit can replace costly full fine‑tuning while delivering superior brain‑encoding performance.
Scaling and Robustness Analysis
Ablations reveal which components matter and guide three scaling directions for PEFT brain encoding.
We systematically removed or altered each design choice to quantify its impact on brain‑encoding performance, then interpreted the results to propose three complementary scaling directions for PEFT.
Replacing the sliding‑window input with a single causally masked forward pass reduces performance by $0.0038$ under the frozen readout head.
Table 2 (Input‑design ablation) reports a $0.0038$ drop when the full‑clip causal pass is used instead of sliding windows.
Restricting the causal pass to the 149 s training window incurs an additional $0.0046$ loss, which a 60 s context lead largely recovers.
Table 2 shows a $0.0046$ performance decrease when the causal pass is limited, and the authors observe recovery with a 60 s lead.
Omitting the closed‑form re‑fit for LoRA r16 + linear readout lowers validation $r$ from $0.3263$ to $0.3202$, a drop of $0.0061$.
Table 2 lists $0.3263$ (with re‑fit) versus $0.3202$ (without) for the frozen‑backbone configuration.
Switching from a linear readout to a gated‑MLP readout or to a cross‑attention readout yields no measurable improvement (differences $<0.001$).
Table 2 reports virtually identical scores for gated‑MLP and cross‑attention variants compared with the linear baseline.
Removing the audio stream reduces performance for both Qwen3.8‑27B ($0.3026$) and Gemma‑3‑27B ($0.3191$) backbones, confirming audio’s contribution.
Table 2 “No audio” row lists the two backbone scores.
Applying cross‑attention to the same pooled feature set yields no measurable gain, indicating that the observed benefits stem from access to a larger token set rather than the attention mechanism itself.
App. K reports a null result for cross‑attention on pooled features.
Increasing the LoRA rank or replacing the linear readout with a gated‑MLP does not improve performance after the closed‑form re‑fit, as the gains disappear in the final evaluation.
App. E shows that the apparent improvements vanish once the closed‑form re‑fit is applied.
The fused multi‑LoRA training kernel achieves a $29.8\times$ speedup over sequential training, while a single fused step costs only $1.08\times$ a single‑adapter step.
Figure 8 reports a $29.8\times$ overall speedup and a per‑step cost of $1.08\times$ compared with the baseline.
The full 261‑model ensemble reaches $0.3483$ validation accuracy, $0.3428$ on‑distribution accuracy, and $0.2483$ out‑of‑distribution accuracy, the highest reported on the public leaderboard.
Section 5.5 reports these three accuracy numbers for the full ensemble.
A carefully selected 15‑model subset attains $0.3465$ validation accuracy—within $0.0018$ of the full ensemble—while using only one‑seventeenth of the inference cost.
Section 5.5 states the subset’s accuracy and its cost reduction relative to the full pool.
Fitting the ensemble scaling law $m(n)=m_{\infty}-A\,n^{-b}$ yields an exponent $b\approx0.94$, indicating near‑linear improvement with ensemble size.
Section 5.5 reports $b\approx0.94$ with $R^{2}=0.998$ for the fit.
Limitations: models are trained on a single audiovisual‑narrative benchmark with four subjects, and neither joint training across datasets nor modality extensions were explored.
The “Limitations” paragraph enumerates these constraints.
Ensemble Scaling Analysis
Limits of ensemble scaling and alternative PEFT methods are examined.
Ensembling multiple adapted models yields diminishing returns: the mean performance follows a power‑law convergence toward an asymptotic limit, with most of the gap closed by the first few members.
Figure 5b plots $-\log_{10}(m_{\infty}-r)$ versus ensemble size; equal vertical steps correspond to equal fractions of the remaining gap. At $n=20$ the 5th–95th percentile range is 0.3360–0.3409 around a mean of 0.3388, and 16 % of subsets already saturate at the limit.
Table 10 enumerates the 261‑model pool used for the ensemble analysis. Each row lists a distinct configuration; members that differ only by random seed are grouped, with the seed count shown. The pool spans backbones (Qwen3‑Omni, Gemma 3 27B, Ming‑flash 104B, etc.), ranks, leads, and readout variants.
Across all non‑LoRA families, the gain over the frozen backbone is positive and statistically robust, but LoRA delivers the largest absolute improvement while using a comparable parameter budget.
Extended Related Work
Extended related work, design details, efficiency analysis, and hyperparameter choices.
The appendix expands the related‑work survey, grouping prior approaches into five families that differ in how they incorporate neural data into stimulus models.
The dominant paradigm freezes a pretrained backbone and fits a readout to neural responses, a design that underlies most Algonauts 2025 entries.
A second family fine‑tunes the backbone itself with brain data; PEFT methods such as LoRA achieve near‑full‑fine‑tune performance while updating only a tiny fraction of parameters.
A third family jointly optimises a primary task and an auxiliary neural‑alignment loss, but must guard against spurious gains by using shuffled‑target controls.
A fourth family pretrains models solely on neural recordings; recent work shows limited gains for downstream stimulus decoding.
A fifth family merges stimulus and brain tokens in a single model, typically adapting a large backbone with LoRA, yet they have not yet surpassed specialised encoders on voxelwise correlation.
Finally, batched multi‑adapter kernels enable routing many adapters in a single fused gather‑matmul, a solved inference problem that our training kernel extends to the backward pass.
Section B details the full input design; Table 3 reports the underlying numbers, showing that a single causal pass reduces extraction compute by two orders of magnitude while preserving accuracy.
The table presents a comparison of different stimulus design strategies and model configurations, evaluating performance metrics across various methods including MIRAGE head (frozen, LoRA, and re-fit) and Linear (LoRA and re-fit) approaches. It includes columns for Extract (GPU-h), performance values for different model heads, training time (s/step), and training cost (GPU-h).
With LoRA applied to the MIRAGE head, the single‑pass design surpasses the sliding‑window protocol, and when the backbone is adapted, the single‑pass matches or exceeds the sliding window.
Context‑lead experiments reveal that adding a 30 s lead improves performance by +0.00396, and a further 30 s adds +0.00142, after which accuracy plateaus, justifying the choice of a 60 s lead.
Within‑window accuracy drops at the start of the window by 0.0393 ± 0.0043 without lead, and the lead recovers only part of this deficit, indicating residual errors unrelated to missing context.
Section C evaluates efficiency: our fused multi‑adapter kernel is 29.8× faster at $N\!=\!32$ adapters than running $N$ separate jobs, and the fused step costs only 1.08× a single‑adapter step.
**Table 5.** **LoRA rank.** Shipped recipe otherwise unchanged (QWEN3-OMNI, linear readout, extraction layer 32, 60 s lead, LoRA lr $6 \times 10^{-4}$, 1,429 steps), Friends s06 subject-mean after the closed-form re-fit. Rank and readout learning rate were not varied independently; the matched comparison is the two rows at $3 \times 10^{-4}$. Diverged runs are counted in parentheses.
The table presents the performance metrics for different feature-extraction layers, categorized by whether the layer is "Frozen" or "Trained".
The table displays performance metrics across three modalities (audio, vision, text) for different prefix lengths (140s, 160s, 180s, 200s) and a baseline condition ("same span twice").
**Table 9.** Cosine between representations at shared timesteps, prefix pass against single pass, on *`friends_s01e01a`* with a 209 s span, feature extraction layer 32.
Section D describes the selection protocol: all design choices were fixed on the Friends season‑6 validation split, and leaderboard scores were only reported after submission.
Section E outlines hyperparameter selection; LoRA rank, feature‑extraction layer, and schedule length were tuned on the same validation split, with the final configuration chosen for robustness.
LoRA rank analysis (Table 5) confirms rank 16 as the optimal trade‑off, balancing performance gain against seed variance and avoiding divergence at higher ranks.
Feature‑extraction layer analysis (Table 6) shows a flat performance neighborhood around layers 30–34; layer 32 is selected as a stable compromise.
Schedule‑length analysis (Table 8) demonstrates that a single epoch suffices; extending to two epochs degrades cosine similarity, reinforcing the one‑epoch training choice.
Section F investigates local non‑causality; despite identical prefixes, cosine similarity deviates (Table 9) due to length‑dependent bf16 rounding, not future‑stimulus leakage.
Hyperparameter Configurations
Learning‑rate choices, backbone sweeps, and ablation outcomes are detailed.
Each PEFT family uses its own tuned learning rate: $1 \times 10^{-2}$ for IA³, $3 \times 10^{-4}$ for the bottleneck adapter, and $6 \times 10^{-4}$ for LoRA. The IA³ and bottleneck rates sit at interior optima of their respective grids, while LoRA’s rate would place IA³ near the bottom of its curve. A plausible cause is bf16 precision: multiplicative updates smaller than ≈0.4 % cannot be represented, forcing a higher IA³ rate, whereas additive updates have no such floor.
Adapter size does not translate to faster training steps. The IA³ model requires 13.1 s per step, whereas the 149× larger Houlsby adapter needs 11.4 s, because step time is dominated by the backbone’s forward and backward passes. The smaller adapter only reduces optimizer‑state storage, not the compute bottleneck.
The table presents various model configurations and their corresponding performance metrics on the s06 dataset.
The table presents the impact of different learning rate combinations for LoRA and Readout on the s06 (pre-re-fit) metric.
This table lists various model configurations and their corresponding member and seed counts.
The table compares various parameter-efficient fine-tuning methods across five columns: Method, Params, s06, $\Delta$ frozen, % of LoRA gain, and $\Delta$ / M params.
Changing the random seed yields negligible ensemble gain, unlike varying the readout or backbone. Using the 15 seeds of the main configuration, adapters trained from different seeds compute nearly identical functions, so their predictions add little independent signal for averaging.
Weight updates from two seeds are almost orthogonal: $\cos(\Delta W_i, \Delta W_j)=0.026$, while their relative distance is $1.396\approx2$. This stems from the factorization $\Delta W = BA$, where $A$’s row space changes little during training, making seed‑specific updates orthogonal regardless of the learned function.
Activations show increasing agreement across layers: a single module’s output change correlates at 0.151, rising to 0.355 after one layer, 0.595 after sixteen, and 0.860 at the feature‑extraction layer (still 45 % of the feature norm). By the final layer the agreement settles between 0.74 and 0.82, exceeding the prediction of an independent‑noise model.
Pairing an adapter with a readout trained on another seed changes prediction by only 0.0012 (raw readout) or 0.0024 (re‑fit readout), both within the seed spread. Under a shared readout, the prediction change is 0.418 of the prediction norm with correlation $r=0.934$ across seeds, indicating that seed‑specific deviations are small and largely uninformative for the fMRI target.
Several design choices did not improve performance: (i) Gated‑MLP readout adds +0.0007 over linear readout; (ii) cross‑attention over pooled features yields +0.00030, attributable to a larger token set rather than attention itself; (iii) expanding the token sequence to the full 19 099 tokens gives no gain; (iv) reducing training data proportionally lowers accuracy; (v) per‑subject adapters decrease the score by ~0.0019, matching a compute‑matched control.
The shift‑target control tests whether the adapter’s gain depends on the specific fMRI‑shaped supervision. The frozen‑backbone control in Table 2 shows that the +0.0356 gain does not arise from readout training alone, suggesting the adapter can profit from any stimulus‑aligned supervision, not only the original target.
Supplementary Data Tables
Additional quantitative analyses and implementation details supporting the main results.
This appendix compiles the detailed tables, calibration curves, and implementation specifics referenced throughout the paper.
Across all calibration budgets, LoRA consistently outperforms the frozen backbone, while anchored ridge dominates ridge‑from‑scratch after 20 min, and the low‑rank deviation adds only marginal improvements.
**Table 18.** Results. The pool leaves 32.5% [30.0, 35.3] of the cross-subject stimulus-locked variance unexplained (Tab. 18). A single model leaves 36.2%, so averaging 261 models recovers 3.7 points. The four subjects agree to within 1.4 points. In variance units the pool misses 0.043 of a unit-variance parcel, against an ISC of 0.133.
The table displays parameters and statistics for 8 blocks, including start time $\delta_b$, width $w_b$, time interval, and counts for full, clipped, and zero values.
**Table 13.** Shifted-target negative control. Friends s06 subject-mean after the closed-form re-fit on aligned training data, one run per configuration. Differences are paired per parcel with a functional-block bootstrap interval; the last column counts the parcels on which the first configuration scores higher.
The table presents performance metrics for different subject configurations (sub-01, sub-02, sub-03, sub-05) across various experimental conditions: "frozen", "3-subj., held out", "3-subj., in-sample", and "4-subject". It also includes comparative columns showing differences relative to "frozen", "in-sample", and "4-subj." configurations. Values in the "3-subj., held out" column are bolded for all subjects.
**Table.** Transferred readout. The adapter's gain is +11.1 points of NNDb's ceiling, compared with +10.7 on Algonauts validation (Tab. 15). Paired over the parcels both datasets admit, the difference is +0.55 points [+0.01, +1.06]. The adapter improves 989 of 1,000 parcels and all 20 participants. In raw r the gain looks smaller (+0.0232 against +0.0356) and in relative terms larger (23% against 12%). Both differences reflect where each dataset's frozen baseline sits relative to its ceiling, not the adapter.