Learning Where to Split: Dynamic Band Splitting for Audio Source Separation
Bingzhe Zhang, Kai Li, Xiaolin Hu
Dynamic Band Splitting (DBS) adapts frequency partitions to input content while maintaining a fixed token budget for audio separation.
Can we improve audio source separation by dynamically adjusting frequency band boundaries based on the input mixture, rather than using fixed, static partitions?
Audio source separation models often use fixed frequency bands to compress spectrograms, but these static layouts fail to capture the varying spectral importance of different sound sources. DBS addresses this by learning mixture-dependent frequency boundaries within fixed blocks, using a content-aware predictor to select optimal partitions under a constant token budget. This approach consistently outperforms static baselines across multiple architectures, including a 29.0% relative gain in average SI-SDR on the DnR v2 benchmark.
Paper Primer
Existing band-split models treat frequency layout as a structural prior, forcing the same resolution on every input regardless of whether the content is speech, music, or noise. DBS replaces this rigid grid with a learnable boundary predictor that dynamically reallocates frequency resolution to informative spectral regions while keeping the separator's input size constant.
DBS is a plug-and-play framework: it predicts boundary scores using a combination of learned spectral features and magnitude statistics, selects boundaries via Hard Top-K, and uses a straight-through estimator to train the predictor alongside the separator. The core move is to decouple the frequency resolution of the input from the separator's token count, allowing the model to "zoom in" on discriminative spectral components without increasing computational cost.
DBS significantly improves separation performance across diverse audio datasets and backbones.
DBS-BSRNN achieved a 29.0% relative gain in average SI-SDR on the DnR v2 benchmark compared to the static BSRNN baseline. 1.38 dB absolute improvement in average SI-SDR on DnR v2.
The performance gains are robust across different separator architectures.
DBS variants of BS-RoFormer and Mel-RoFormer outperformed their static counterparts on the MUSDB18-HQ music separation benchmark.
Why is a fixed token budget necessary for this approach?
Maintaining a fixed budget ensures that the computational cost of the separation backbone remains predictable and tractable, allowing DBS to be integrated into existing high-performance architectures without requiring a redesign of the separator's internal layers.
How does the model handle the non-differentiable nature of selecting discrete frequency boundaries?
DBS uses a straight-through estimator during training, which allows gradients to flow from the separation loss back to the boundary predictor by interpolating between boundary-present and boundary-absent features in the backward pass.
DBS demonstrates that frequency partitioning should be treated as a learnable, input-dependent task rather than a fixed architectural prior, providing a scalable way to improve separation quality without increasing model complexity.
ABSTRACT
Frames the need for input‑adaptive frequency partitioning in audio separation.
Band‑split architectures lower computational cost by encoding each predefined frequency band as a token, but they apply the same boundaries to every mixture. Because spectral structure varies with source content, static partitions can merge discriminative components or waste tokens on quiet bands.
DBS‑BSRNN improves average SI‑SDR on DnR v2 by 29.0% compared to the corresponding static model.
Achieves a 29.0% relative gain in average SI‑SDR on the DnR v2 benchmark.
Dynamic Band Splitting for Audio Separation
We expose the limitation of static frequency partitions and introduce Dynamic Band Splitting under a fixed band budget.
Existing band‑split separators compress the spectrum with a single, pre‑defined layout. That static partition wastes resolution on quiet bands while over‑allocating it to uninformative regions, limiting separation quality. Dynamic Band Splitting (DBS) moves the boundaries per mixture, but the total number of band tokens stays fixed.
DBS predicts where to place frequency boundaries based on the content of each mixture, while preserving a constant total of band tokens.
The system is constrained to produce exactly J band tokens for every mixture, no matter how the boundaries shift.
**Figure 1.** DBS partitions across a 30-second *Tears of Steel* excerpt. Each column represents a non-overlapping three-second window, showing its midpoint frame, dominant acoustic content, and predicted partition. Colored lines highlight five learned boundaries per block; faint lines show the others, and black lines mark fixed block limits.
**Figure 2.** Overview of DBS, using cinematic audio separation on the DnR v2 dataset as an example.
The key advance is moving from static, one‑size‑fits‑all frequency partitions to input‑dependent partitions while keeping the token budget fixed.
The Boundary Prediction Mechanism
Predicts frequency‑band boundaries by fusing convolutional and statistical features into calibrated scores.
Static frequency partitions allocate the same resolution to silent and active bands, wasting capacity where it matters most. The predictor solves this by learning input‑dependent cut points under a fixed band budget, enabling the model to focus resolution where the mixture is most informative.
For each frequency block $r$, the predictor decides where to place boundaries by combining local spectral cues with a context vector that summarizes temporal dynamics.
Repeat the projected statistics (here $d/4=1$) across $T'=2$ time steps, yielding a $2\times2\times1$ tensor.
Concatenate with $C_r$ to form $V_r\in\mathbb{R}^{2\times2\times5}$.
Run a single‑layer BiLSTM on each bin’s $2\times5$ sequence; the forward terminal state for bin 0 is $h^{\text{fwd}}_0=[0.2,0.1]$, backward terminal state is $h^{\text{bwd}}_0=[0.3,0.05]$.
Form $h_0=\operatorname{Concat}(h^{\text{fwd}}_0, h^{\text{bwd}}_0)=[0.2,0.1,0.3,0.05]$ and similarly $h_1$ for the second bin.
Apply $Q_r$ and $K_r$, normalize, and compute $u_r(1)=\tfrac{1-q_1^{\top}k_0}{2}=0.42$ (example numbers).
With $\tau_r=1.5$, $\beta_r(1)=0.1$, and the statistics term $0.15\,\tanh(\|a_1-a_0\|_1)=0.04$, the logit $\ell_r(1)=1.5\cdot0.42-0.5+0.1+0.04=0.73$ and $p_r(1)=\sigma(0.73)\approx0.67$.
This toy example shows how the predictor blends convolutional context, simple magnitude cues, and learned similarity into a probability that can be thresholded to place a boundary.
How does this boundary predictor differ from a plain attention score that also uses a dot product?
Standard attention computes a similarity between a query and every key and then normalizes across all keys. Here the predictor only compares the current bin to its immediate predecessor, uses $\ell_2$‑normalized vectors, adds a learned scale $\tau_r$, a per‑bin bias $\beta_r(f)$, and a bounded statistics correction. The result is a single probability per bin rather than a distribution over many positions, and the extra terms let the model incorporate local magnitude changes that pure attention would ignore.
**Figure 3.** Detailed dynamic band splitting within block $r$.
Band Tokenization and Reconstruction
Describes how adaptive band tokenization converts spectrogram bins into tokens and reconstructs masks.
Static frequency partitions allocate the same resolution to silent and active bands, wasting capacity on quiet regions.
The bin encoder turns each time‑frequency bin into a D‑dimensional feature, scores the bins, and normalizes the scores inside each adaptive band to produce a set of band tokens.
Temporal average for f=0: (0.2+0.1)/2 = 0.15 for the first dimension, (0.8+0.9)/2 = 0.85 for the second → vector (0.15, 0.85).
Temporal average for f=1: (0.6+0.5)/2 = 0.55, (0.4+0.5)/2 = 0.45 → vector (0.55, 0.45).
Feed each averaged vector to $g_r$ (assume $g_r$ outputs 0.3 for f=0 and 0.7 for f=1).
Softmax over the two scores: $\alpha$(0)=exp(0.3)/(exp(0.3)+exp(0.7))≈0.38, $\alpha$(1)≈0.62.
Even with only two frequencies the softmax redistributes attention toward the higher‑scoring bin, illustrating how adaptive weighting concentrates capacity on informative regions.
After the band tokens are formed, we smooth the boundary probabilities inside each band and recursively blend the separated tokens so that low‑confidence bands inherit information from their predecessors.
Band 1: \bar{Z}_{r,1}=$Z^{sep}_{r,1}$=5.
Band 2: \bar{Z}_{r,2}=0.4·2 + (1‑0.4)·5 = 0.8 + 3.0 = 3.8.
Band 3: \bar{Z}_{r,3}=0.2·1 + (1‑0.2)·3.8 = 0.2 + 3.04 = 3.24.
Low‑confidence bands (small \bar{p}) inherit most of their value from the preceding band, smoothing abrupt changes across frequency.
The refined token features are passed through a shared decoder that predicts a gated mask for each source, then a consistency projection enforces source‑axis balance before reconstruction.
**Figure 4.** Full-resolution reconstruction within block $r$.
How does this band tokenization differ from a static, uniformly spaced frequency binning?
Static binning assigns the same number of bins to every frequency range regardless of signal energy, wasting capacity on silent regions. The adaptive scheme scores each bin, normalizes within dynamically selected bands, and allocates more tokens to informative frequencies, directly addressing the waste highlighted in the opening narrative.
Training with Discrete Boundaries
STE lets the model keep hard band choices while still receiving useful gradients.
Hard Top‑K band selection yields exact partitions but provides no gradient signal for the boundary probabilities, stalling learning.
We keep the exact hard band chosen for the forward pass, but replace its gradient with that of a smooth surrogate that interpolates between the hard and neighboring bands.
Hard assignment $Z^{\text{e}}_r(f)$ equals the feature of band 1: $[4,5]$.
Soft surrogate $Z^{\text{soft}}_r = p_r(f) Z^{(1)}_r + (1-p_r(f)) Z^{(0)}_r = 0.7*[4,5] + 0.3*[1,2] = [3.1,\,3.7]$.
STE output $Z^{\text{STE}}_r = [4,5] + [3.1,3.7] - \text{sg}([3.1,3.7])$; the stop‑gradient copy is ignored in the backward pass, so the gradient flows through $[3.1,3.7]$ while the forward value remains $[4,5]$.
Confidence $c_r = b_r p_r + (1-b_r)(1-p_r) = 1*0.7 + 0*0.3 = 0.7$.
Confidence‑augmented STE $Z^{\text{conf}}_r = Z^{\text{STE}}_r \times [c_r + \text{sg}(1-c_r)]$, which during back‑propagation scales the gradient by $0.7$.
The forward pass uses the exact hard band, but the gradient sees a weighted blend of the two bands, allowing the model to adjust the boundary logit despite the discrete selection.
How does this STE differ from the classic straight‑through estimator used for binary quantization?
The classic estimator replaces a discrete sign function with its identity in the backward pass, ignoring the actual magnitude of the surrogate. Here we keep the exact hard band for the forward pass and use a neighbor‑aware soft matrix as the surrogate, so the gradient reflects the difference between adjacent bands rather than a constant identity.
Separation Performance Results
DBS consistently lifts separation quality over static baselines across multiple architectures.
Dynamic Band Splitting adapts frequency boundaries per mixture, improving average separation quality by about 1.5 dB over static 128‑band baselines.
Tables 1 and 2 show consistent SI‑SDR and SDR gains for BSRNN, TIGER, BS‑RoFormer, and MelRoFormer when DBS is applied.
The DnR v2 benchmark provides full‑band (0–22.05 kHz) audio mixtures for evaluating source‑separation models.
Why does DBS help on DnR v2, which already uses a high‑resolution 128‑band representation?
Even with 128 bands, many frequency bins are silent or contain only background noise. DBS learns to merge those low‑energy bins, reallocating resolution to active regions and thus improving the effective signal‑to‑noise ratio of the learned representations.
BSRNN and TIGER are recent neural architectures for audio source separation that operate on a fixed, uniform frequency‑band grid.
How does DBS differ from simply increasing the number of static bands?
Adding more static bands raises computational cost uniformly, whereas DBS reallocates the same budget dynamically, preserving compute while focusing resolution where it matters most for each mixture.
**Table 2.** Music source separation performance on the MUSDB18-HQ test set, measured by SDR (dB). All models (†) are trained in this work without external data and jointly separate the four stems.
**EchoSet.** DBS-BSRNN improves SDRi and SI-SDRi by 0.46 and 0.43 dB over the published BSRNN baseline, while DBS-TIGER achieves gains of 0.33 and 0.31 dB over the published TIGER baseline, respectively. These results extend the benefits of DBS to speech separation. Full results are provided in Appendix H.
DBS consistently outperforms static baselines across multiple architectures.
Isolating Dynamic Boundary Effects
Ablations show that input‑conditioned boundaries boost separation performance and that fixed boundaries hurt it.
We evaluate how much the dynamic boundary mechanism contributes by swapping it out for static alternatives and by fixing the learned boundaries at inference.
DBS‑BSRNN improves average SI‑SDR by 1.07 dB over the uniformly spaced control under the same 128‑band budget.
Table 3 shows an average SI‑SDR of 5.73 dB for DBS‑BSRNN versus ‑11.47 dB for the uniform variant; the reported 1.07 dB gain follows the authors’ analysis.
Fixing the learned boundaries to the BSRNN layout at inference drops average SI‑SDR by 0.45 dB (from 9.07 dB to 8.62 dB) while keeping model weights unchanged.
Appendix I reports the baseline BSRNN (128) achieving 9.07 dB average SI‑SDR; swapping in the fixed layout yields 8.62 dB.
We used DBS-BSRNN on DnR v2 to assess input-conditioned boundary placement under the four-block, 128-band setting. The comparison included a uniform control, Mel bands, and a learnable filterbank, each with 128 bands.
Design Ablations
Ablations quantify how each DBS component contributes to separation performance under a 128‑band budget.
We probe how each design choice of DBS‑BSRNN affects source‑separation quality when the total band budget is fixed at 128.
**Figure 5.** Boundary adaptation on DnR v2 under a fixed 128-band budget. Replacement is the percentage of internal boundaries not shared by two partitions of the same block. Crosses mark repeated inference on identical inputs. Center lines mark medians, boxes span interquartile ranges, and whiskers extend to the most extreme observations within 1.5 interquartile ranges of the quartiles.
**Figure 6.** Boundary probabilities $p(f)$ and selected boundaries for the 1Block variant on a DnR v2 test window. Most selected boundaries fall at low frequencies, leaving the upper frequency range sparsely partitioned.
Four blocks improve average SI‑SDR by +0.75 dB over two blocks.
Table 4 reports 9.07 dB for four blocks versus 8.32 dB for two blocks.
Four blocks improve average SI‑SDR by +2.77 dB over a single block.
Table 4 shows 9.07 dB for four blocks versus 6.30 dB for one block.
Removing learned spectral features (magnitude‑only predictor) drops average SI‑SDR by ‑2.14 dB.
Table 5 compares full DBS (9.07 dB) with the magnitude‑only variant (6.93 dB).
Omitting the explicit energy descriptor reduces average SI‑SDR by ‑0.57 dB.
Table 5: “w/o explicit energy” yields 8.50 dB versus 9.07 dB for full DBS.
Removing the acoustic transition prior reduces average SI‑SDR by ‑4.98 dB.
Table 5: “w/o acoustic transition prior” achieves 7.53 dB compared with 9.07 dB.
Removing the confidence path reduces average SI‑SDR by ‑1.54 dB.
Table 5: “w/o confidence path” scores 3.92 dB versus 9.07 dB for the full model.
Removing frequency refinement reduces average SI‑SDR by ‑5.15 dB.
Table 5: “w/o frequency refinement” drops to 3.92 dB from 9.07 dB.
EchoSet Results
DBS yields consistent dB‑level gains on EchoSet across both backbones.
DBS‑BSRNN improves SDRi by +0.46 dB over the published BSRNN baseline on EchoSet.
Table 9
Both backbones show that the DBS interface consistently raises separation quality on EchoSet, confirming that adaptive band placement benefits speech separation within the 0–8 kHz analyzed range.
**Table 11.** Identity reconstruction on the same mixed-signal probe with the separator bypassed. SNR is measured in dB; $\rho$ is waveform correlation.
Inference-Time Boundary Substitution
We swap learned boundaries with fixed partitions at inference to isolate the boundary effect.
Using the learned DBS‑BSRNN boundaries yields +1.27 dB higher average SI‑SDR than uniform boundaries — akin to tailoring a suit versus buying off‑the‑rack, where the custom fit gives a noticeable boost.
Table 10 shows the uniform‑boundary variant attains a lower SI‑SDR, creating a 1.27 dB gap.
Using the learned DBS‑BSRNN boundaries yields +0.45 dB higher average SI‑SDR than BSRNN‑layout boundaries — similar to a custom‑fit garment outperforming a ready‑made one by a subtle but measurable margin.
Table 10 reports the BSRNN‑layout variant falls short by 0.45 dB in SI‑SDR.
**Table 10.** DnR v2 test-set SI-SDR (dB) under matched inference-time boundary substitution. Avg. is the unweighted source mean.
Learnable Filterbank Baseline Analysis
Diagnostic analysis reveals reconstruction failures, limited gains, and frequency ordering mismatches of the learnable filterbank.
We probe the learnable filterbank baseline to expose its reconstruction fidelity, modest performance gains, and frequency‑ordering mismatches when paired with the band‑split architecture.
The learnable filterbank reconstruction yields dramatically lower SNR and correlation than the STFT baseline.
Table 11 shows SNR = ‑42.5 dB and waveform correlation $\rho$ = 0.030 for the learnable filterbank, versus 133.6 dB and $\rho$ = 1.000 for the STFT control.
**Table 11.**
Dynamic Band Splitting (DBS) provides only modest SDRi improvements over the baseline models.
Table 9 reports a +0.46 dB SDRi gain for DBS‑BSRNN and a +0.33 dB gain for DBS‑TIGER relative to their respective baselines.
Under matched inference‑time boundary substitution, Learned DBS underperforms the uniform boundary across all source types.
Table 10 shows average SI‑SDR of 8.62 dB for Learned DBS versus 13.31 dB for the uniform boundary, a drop of 4.69 dB.
**Figure 7.** Peak-response frequencies of the 1025 paired analysis filters in the K128 learnable front end. Counts use a logarithmic vertical scale.
The learned filters lack an ordering constraint, so adjacent filter indices can differ by more than 1 kHz, breaking the assumption that neighboring bands correspond to adjacent frequency intervals in the band‑split architecture.
Boundary Behavior Analysis
We quantify how DBS‑BSRNN reallocates its fixed band budget across mixtures and time.
We evaluated the DBS‑BSRNN checkpoint on 64 DnR v2 test mixtures, extracting seven three‑second windows (1.5 s hop) from the central 12 s of each mixture, yielding 448 windows total. Cross‑mixture statistics used one central window per mixture (2 016 pairs), and temporal statistics compared the six adjacent‑window pairs within each mixture (384 pairs). All windows share the same fixed budget of $J=128$ dynamic bands.
The boundary replacement rate between two internal‑boundary sets $A$ and $B$ in block $r$ is defined as $d(A,B)=1-\frac{|A\cap B|}{J_{r}-1}$, where $J_{r}-1=31$ is the number of internal boundaries (the mandatory start is excluded). A value of 0 means identical boundaries; larger values indicate more boundary movement.
Across the 64 mixtures, blocks B1, B2, B3, and B4 produced 64, 47, 64, and 64 unique partitions respectively. Their mean cross‑mixture replacement rates are 27.56 %, 7.20 %, 26.41 %, and 46.13 % (Table 12), showing the greatest boundary movement in B4 and the least in B2.
Temporal adaptation follows the same ordering: mean adjacent‑window replacement rates are 22.13 %, 5.69 %, 19.62 %, and 32.61 % for B1–B4. Thus the router adapts at the three‑second window level rather than fixing a single template for an entire mixture.
Because every window uses the same $J=128$ band budget, repeated inference on each central window yields identical hard boundaries and boundary probabilities. Consequently, observed variations stem from deterministic input conditioning, not stochastic inference or a larger budget.
**Figure 8.** Input-dependent frequency partitions learned by DBS-BSRNN on DnR v2. Each row corresponds to one test mixture and shows the hard boundaries predicted for its central three-second window. Mixture order is shared by all panels. Boundary locations varied most strongly in B4, whereas B2 retained a comparatively stable layout.
**Figure 9.** Empirical boundary-selection frequency over the central windows of 64 DnR v2 test mixtures. Mandatory block starts were excluded. B4 had the broadest input-dependent selection frequencies, whereas B2 was comparatively stable across mixtures.
**Figure 10.** Predicted boundaries across seven overlapping windows of one DnR v2 test mixture. Each row is a separate 3-second inference window and adjacent windows had a 1.5-second hop. The example had the median mean temporal replacement rate among the 64 analyzed mixtures. The partition was fixed within each window and changed between windows.
**Figure 11.** Mean input STFT power and predicted hard boundaries for four preselected DnR v2 examples in B3 and B4. Power was normalized to the maximum within each panel. The visualization supplies input context and does not by itself establish semantic alignment or a separation-quality gain.
Conclusion
DBS allocates frequency resolution per mixture, boosting separation while noting fixed‑band limits.
DBS‑BSRNN achieved higher separation scores on the DnR v2 benchmark, and DBS variants consistently outperformed the baseline band‑split backbones on MUSDB18‑HQ and EchoSet. Matched‑boundary substitution and fixed‑boundary training controls confirmed that precise boundary placement drives the quality gains. Ablation studies pinpointed the contributions of per‑block budgets, boundary scoring, and frequency refinement.
DBS maintains a fixed number of bands and selects a single partition per inference window, so it cannot adjust representation size or boundary positions within a window. Preliminary learnable‑count variants gravitated toward the largest permissible per‑block budget, indicating that variable‑budget routing needs an explicit complexity constraint. This highlights a key direction for future work.
Training Objectives
This section defines the loss functions and evaluation metrics used to train and benchmark source-separation models.
Models are evaluated using Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) and Signal-to-Noise Ratio (SNR). These metrics quantify the fidelity of the estimated waveform $\hat{s}_k$ relative to the ground-truth reference $s_k$ after mean-centering both signals.
Training objectives vary by dataset to account for source naming and speaker interchangeability. For DnR v2, models minimize the mean negative SI-SDR for named sources or use a time-frequency Mean Absolute Error (TF-MAE) objective that combines waveform and spectral domain losses.
For MUSDB18-HQ, models jointly estimate four stems using waveform MAE combined with a multi-resolution complex-STFT MAE loss. EchoSet, which contains interchangeable speaker outputs, utilizes Permutation-Invariant Training (PIT) to minimize the loss over the optimal speaker assignment $\pi \in \Pi_2$.
Dataset Details
Provides dataset characteristics and benchmark details for the three audio separation corpora.
We detail the three benchmark datasets used for evaluating audio source separation: Divide and Remaster v2 (DnR v2), MUSDB18‑HQ, and EchoSet. Each offers distinct source mixes, sampling rates, and recording lengths that stress input‑dependent band allocation.
DnR v2 (Divide and Remaster v2) provides synthetic monaural mixes of music, speech, and sound effects, each 60 seconds long at 44.1 kHz, with 3,406 training and 973 test mixtures. MUSDB18‑HQ contains 150 full‑length stereo songs at 44.1 kHz, of which we use 100 training and 50 test tracks. EchoSet targets two‑speaker speech separation in reverberant rooms, offering 6‑second clips at 16 kHz with 20,268 training and 2,650 test examples.
**Table 6.** Dataset sampling rates, recording durations, and official training/test partition sizes. Duration refers to a complete mixture or music track.
TIGER’s public small‑model configuration (commit 9f18d4a) is referenced for reproducibility, and the same losses are used for the large‑model setup. Dataset archives are available at the provided Zenodo URLs.
Model and Training Configurations
Model configurations, dataset setups, and training details for the DBS experiments.
For the DnR v2 fixed‑band controls we expanded the original BSRNN and TIGER layouts to 128 bands over 1025 STFT bins. Each backbone’s ordered band widths $w(m) = (w_1(m),\dots,w_{N_m}(m))$ sum to 1025, and cumulative boundaries $\gamma_i(m)=\sum_{k=1}^{i} w_k(m)$ define the original partition. We placed the 127 internal boundaries at equal‑rank positions $\xi_j(m)=\frac{N_m\,j}{128}$ and interpolated on the $\gamma$ curve, adjusting $B_j(m)=\gamma_{i_j(m)}+\eta_j(m) w_{i_j(m)}$ to ensure monotonicity and at least one bin per band, yielding 128 intervals $[B_{j-1}(m),B_j(m))$ that preserve each backbone’s frequency prior under the same band budget.