Spectral-Spatial Interpretation
Haotian Ma, Ruqi Yang, Philip Townsend
Spectral-Spatial Interpretation (SSI) assigns localized channel-importance scores to multispectral images using conditional Shapley values.
How can we generate unified spatial-spectral explanations for multispectral models that capture both where the model looks and which channels it relies on?
Existing attribution methods either collapse multispectral data into a single spatial heatmap or rank spectral channels globally, failing to capture how channel importance shifts across different regions of an image. SSI treats each grid cell and spectral channel as a joint player, fixing the surrounding image context and applying the Shapley value to the channels at each specific location. This approach outperforms dense, flat, and rank-1 baselines on joint faithfulness metrics while requiring significantly fewer evaluations than hierarchical alternatives.
Paper Primer
Multispectral models often rely on specific bands—like near-infrared for vegetation or short-wave infrared for burn scars—that are only decisive in certain spatial regions. Standard methods either aggregate these channels into a single heatmap, losing spectral nuance, or rank channels globally, assuming the same spectral profile applies to the entire image.
SSI is a model-agnostic attribution method that defines a conditional game at each spatial position: it fixes the observed surrounding context and uses the Shapley value to allocate importance among the channels at that specific location. This produces a full spectral-spatial attribution matrix where the channel-aggregated score at any position exactly recovers the contextual contrast of masking that entire region.
SSI achieves superior joint faithfulness compared to dense, flat, and rank-1 baselines.
Across CUB-200, EuroSAT, and HLS Burn Scar datasets, SSI consistently yields the lowest joint deletion AUC and highest insertion AUC. On HLS Burn Scar, SSI outperforms the MC-Owen estimator across all faithfulness metrics while using only ~6% of the evaluation budget.
Why does SSI use a conditional game rather than a global one?
Global games average channel relevance across the entire image, which hides localized evidence where a band might be decisive in one region but irrelevant in another. By conditioning on the observed surrounding context, SSI isolates the specific spectral contribution at each location.
What is the primary limitation of rank-1 attribution methods like RISE?
Rank-1 methods assume a factorized structure where a spatial heatmap is multiplied by a global channel vector. This fails when different regions have distinct spectral signatures, as the model's reliance on specific bands varies spatially.
SSI provides a principled way to interpret multispectral models by preserving the spatial-spectral hierarchy, enabling researchers to identify exactly which bands drive predictions in specific, physically meaningful regions.
The Problem of Fragmented Explanations
Identifies the gap in existing post‑hoc explanations for multispectral data and introduces SSI.
Spatial heatmaps assign a relevance score to each pixel, indicating where the model focuses. Global channel scores assign a single importance value to each spectral band, indicating overall contribution.
Existing post‑hoc explanations provide either spatial heatmaps or global channel scores, but not both jointly, which is insufficient for multispectral data where channel relevance is spatially localized and physically meaningful.
Many modern vision models process inputs with multiple physically meaningful channels, yet the same spectral band can be decisive in one region and irrelevant in another, breaking the rank‑1 assumption $A(g,c)=s(g)q(c)$ that a single channel profile applies everywhere.
Figure 1 illustrates how distinct physical features align with separate spectral channels, motivating the need for a joint spectral‑spatial attribution matrix rather than separate heatmaps or global rankings.
**Figure 1.** Spectral channels carry spatially distinct physical information. On a Painted Bunting (a), the red belly, green back, and blue head each align with a different RGB channel; on a Sentinel-2 scene (b), water, forest, and built-up areas align with B3, B8A, and B12. SSI is designed to recover this joint spectral-spatial structure.
The limitation of current methods in handling multispectral data is their inability to provide localized, channel‑specific evidence.
Related Work and SSI Overview
We situate SSI among prior attribution methods and define its core concepts.
Related work splits into spatial heatmaps, spectral channel scores, and joint spectral‑spatial modeling, each with distinct limitations.
SSI attributes a prediction to each position‑channel pair by fixing the surrounding image and distributing credit among the channels at that position.
How does SSI differ from standard spatial‑only attribution?
Standard spatial attribution aggregates over all spectral channels, so a highlighted region tells only that some wavelength matters there; SSI keeps the channel dimension, assigning separate scores to each wavelength at each pixel, thus revealing which specific bands drive the decision locally.
The joint attribution matrix $\Phi(g,c)$ records the attribution value for every spatial position $g$ and every spectral channel $c$, giving a full map of where each wavelength contributes.
**Figure 2.** Overview of SSI. For each spatial position $g$, SSI fixes the observed surrounding context and allocates the local contribution among channels, producing a full joint attribution matrix $\Phi(g, c)$ for spatial and spectral interpretation.
Standard spatial heatmaps collapse all spectral information, so they cannot tell which band is responsible for a highlighted region; SSI’s joint matrix separates contributions, enabling precise spectral‑spatial diagnostics.
The SSI Mechanism
Defines the Shapley‑value based attribution that powers SSI and shows how it is computed.
Standard post‑hoc explainers either paint a spatial heatmap or assign a single score per spectral channel, leaving the analyst without a clear link between a specific channel and the region it influences. SSI resolves this by computing a joint attribution matrix that attributes each channel to each pixel.
For a given pixel g, treat each spectral channel as a player in a cooperative game; the Shapley value tells how much each channel contributed to the model’s prediction for that pixel, given the observed surrounding context.
All subsets R of {1,2,3} that exclude channel 1 are {∅, {2}, {3}, {2,3}}.
Compute marginal contributions: $w_g$({1})−$w_g$(∅), $w_g$({1,2})−$w_g$({2}), $w_g$({1,3})−$w_g$({3}), $w_g$({1,2,3})−$w_g$({2,3}).
Apply Shapley weights: each term is multiplied by |R|!·(C−|R|−1)! / C! = 1/6, 1/3, 1/3, 1/6 respectively.
Sum the weighted contributions to obtain $\Phi$(g,1) = 0.45 (example numeric result).
Repeat the same process for channels 2 and 3, yielding $\Phi$(g,2)=0.30 and $\Phi$(g,3)=0.25.
The example shows how the Shapley formula distributes the total model output (which sums to 1.0) among the three channels, illustrating the efficiency property in a concrete, small‑scale setting.
How does SSI’s Shapley‑value attribution differ from the Owen value used in hierarchical attribution?
Both allocate value to channels within a pixel, but the Owen value first averages over all possible external spatial coalitions before distributing to channels, yielding a globally efficient attribution. SSI fixes the observed surrounding image, computes the conditional game $w_g$, and then applies the Shapley formula, so its attributions reflect the actual context rather than an average over unseen contexts.
To estimate the Shapley values efficiently, SSI samples permutations of the C channels at each pixel. In practice we draw K = min(2C, 10) random orderings; when 2C ≤ 10 we enumerate every subset exactly.
Permutation‑sampling estimator for $\Phi$(g,·) at a single pixel.
SSI reports the full joint attribution tensor $\Phi$, but downstream analyses often collapse it: summing over channels yields a spatial importance map $\Delta$(g), while summing over pixels yields a spectral importance vector.
Empirical Results and Evaluation
Experimental results demonstrate SSI’s superior joint faithfulness across datasets.
Recall that SSI computes a joint attribution matrix $\Phi$(g,c) linking each spatial position g to each spectral channel c, addressing the fragmentation of prior post‑hoc methods.
SSI attains the highest insertion AUC (0.7132) on CUB‑200, surpassing all baselines.
Table 1 reports insertion AUC 0.7132 ± 0.019 for SSI, whereas IG achieves 0.2895 ± 0.022 and RISE 0.6886 ± 0.017.
**Figure 3.** Spatial and spectral aggregation views. Panels (c) and (f) show spatial localization using mIoU at different coverage thresholds. Panels (b) and (e) report spectral Hit@K, while panels (a) and (d) show channel deletion AUC (lower is better).
SSI consistently outperforms baselines in joint faithfulness.
Theoretical Foundations
Appendix A details the theoretical foundations and proofs underlying SSI’s attribution method.
This appendix collects the formal proofs for the local Shapley characterization, the consistency of contextual aggregation, and the trade‑off with global efficiency. It also details the sampling estimator, its concentration bound, and connections to flat Shapley and Owen‑style hierarchical accounting.
For a given spatial position, the attribution of each spectral channel is computed only with respect to the other channels at that same position, ignoring any contribution from other positions.
How does contextual aggregation differ from the global efficiency property that many attribution methods aim for?
Contextual aggregation distributes value only among channels at a single position while keeping the rest of the image fixed. Global efficiency requires the sum of all attributions (over every position and channel) to equal the model’s total output. Because the external context is fixed, the two criteria can conflict: the per‑position sums may each equal the marginal gain of adding that position’s channels, which can add up to more than the overall model value.
For any fixed spatial position $g$, the local attribution map $\beta_g$ that satisfies local efficiency, within‑position symmetry, null‑player, and additivity is uniquely given by the Shapley value of the conditional game $w_g$.
Define the conditional game $w_g(R)=v(B_g\cup R)-v(B_g)$ over the channel set $S_g$.
The Shapley value is the unique solution to the four local axioms.
Substituting $p=(g,c)$ yields the channel‑wise attribution $\Phi(g,c)=\operatorname{Sh}_c(w_g)$.
For any position $g$ and channel $c$, the contextual aggregation attribution satisfies $\Phi(g,c)=v(P)-v(P\setminus S_g)=\Delta(g)$.
Local Shapley efficiency gives $\Phi(g,c)=w_g(S_g)-w_g(\emptyset)$.
Since $w_g(\emptyset)=0$ and $B_g=P\setminus S_g$, we have $\Phi(g,c)=v(P)-v(P\setminus S_g)$.
If an attribution rule satisfies contextual aggregation for every position, it cannot simultaneously satisfy global efficiency for all games.
Assume contextual aggregation holds: $\alpha(g,c)=\Delta(g)$ for all $g,c$.
Summing over all positions yields $\sum_{g,c}\alpha(g,c)=\sum_{g}\Delta(g)$.
Global efficiency would require $\sum_{g,c}\alpha(g,c)=v(P)$ for every game.
Counterexample: two positions $g,h$ each with one channel, with $v(\{a,b\})=1$ and all other subsets $0$.
Thus the two properties are mutually exclusive in general.
For any position $g$ and channel $c$, the permutation‑sample estimator $\Phi_K(g,c)$ is unbiased and satisfies a Hoeffding‑type concentration bound.
Define the marginal contribution $M_{g,c}(\pi)=w_g(\operatorname{Pred}_\pi(c)\cup\{(g,c)\})-w_g(\operatorname{Pred}_\pi(c))$.
The Shapley value equals the expectation of $M_{g,c}(\pi)$ over a uniform random permutation.
Therefore $\Phi_K(g,c)=\frac{1}{K}\sum_{k=1}^K M_{g,c}(\pi_k)$ is an unbiased estimator.
Each $M_{g,c}(\pi)$ lies in $[a_{g,c},b_{g,c}]$; applying Hoeffding’s inequality gives the tail bound.
Union‑bounding over all $|G|C$ players yields the statement in the proposition.
There exist value functions with identical quotient games over positions but different summed flat‑Shapley attributions for a given position.
Consider two positions $g,h$ with channel sets $S_g=\{a_1,a_2\}$ and $S_h=\{b\}$.
Define $v_1$ as the unanimity game $v_1(\{a_1,a_2,b\})=1$ and zero otherwise.
Flat Shapley gives $\phi_{v_1}(a_1)=\phi_{v_1}(a_2)=\phi_{v_1}(b)=\tfrac13$, so the sum over $S_g$ equals $\tfrac23$.
Define $v_2$ with $v_2(\{a_1,b\})=v_2(\{a_2,b\})=\tfrac12$ and zero otherwise.
Flat Shapley now yields $\phi_{v_2}(a_1)=\phi_{v_2}(a_2)=\tfrac16$, so the sum over $S_g$ equals $\tfrac13$.
Both games have identical quotient values $v_Q$ but different position‑level sums, proving the claim.
When an attribution over position‑channel players satisfies global efficiency, position‑level aggregability, quotient‑level Shapley consistency, within‑position symmetry, null‑player, and additivity, the unique solution is the Owen value $\psi(g,c)=\sum_{T\subseteq G\setminus\{g\}}\frac{|T|!\,(|G|-|T|-1)!}{|G|!}\operatorname{Sh}_c\bigl(w^{T}_g\bigr)$.
The player set $P$ is partitioned into a priori unions $\{S_g\}_{g\in G}$.
The quotient game $v_Q$ over spatial positions is defined by $v_Q(T)=v\bigl(\bigcup_{g'\in T}S_{g'}\bigr)$.
For any external coalition $T\subseteq G\setminus\{g\}$, the internal game $w^{T}_g$ is given by $w^{T}_g(R)=v\bigl(R\cup\bigcup_{g'\in T}S_{g'}\bigr)-v\bigl(\bigcup_{g'\in T}S_{g'}\bigr)$.
Owen’s characterization for games with a priori unions states that the unique value satisfying the listed axioms is the Owen value.
Substituting the partition $\{S_g\}$ yields the closed form in the statement.
Empirical evaluation (Section A.5) confirms that using $K=\min(2C,10)$ permutations yields estimation errors well below the inter‑method gaps reported in the main tables, validating the practical adequacy of the sampling bound.
Section A.8 sketches an optional region‑restricted variant of SSI that limits the external context to a user‑specified spatial window, trading off attribution fidelity for further computational savings.
Computational Efficiency and Scaling
Datasets, baselines, and computational‑cost details for SSI.
B.1 Datasets, Models, and Preprocessing. We fine‑tune a ResNet‑18 classifier on CUB‑200‑2011 (11,788 RGB images, $C=3$ channels) using the ground‑truth class logit as the scalar response. EuroSAT provides $13$ spectral bands; we train a ResNet‑18 from scratch on all $C=13$ channels. HLS Burn Scar supplies six bands; we train a lightweight UNet for burn‑scar segmentation and use the average burn‑class logit over the evaluation region as the scalar response. All optimizer settings, learning rates, epochs, and preprocessing choices are fixed per dataset before attribution evaluation.
B.2 Attribution Baseline Details. Integrated Gradients (IG) follows a straight‑line path from a masking baseline to the input with $300$ integration steps; dense attributions are summed within each grid cell and channel to obtain a score per position‑channel player $(g,c)$. DeepSHAP uses $1{,}000$ background samples and the same aggregation. KernelSHAP treats each $(g,c)$ as an interpretable feature, sampling $2|P|+2048$ coalitions (deterministic singleton/complement pairs plus $2048$ random coalitions). RISE rank‑1 control generates $M=8{,}000$ random binary masks, computes a spatial importance map $s_{\text{rise}}(g)$, and multiplies it by a global channel score $q(c)$ to form $A_{\text{rise}}(g,c)=s_{\text{rise}}(g)\,q(c)$. Expected Gradients, Guided IG, and FullGrad are also evaluated on the HLS Burn Scar task, with dense attributions summed per grid cell while retaining channel indices. SSI computes the full‑context joint attribution $\Phi(g,c)$ from Eq. (6) over the entire position‑channel player set; spatial or spectral summaries are obtained by summing $\Phi(g,c)$ over $c$ or $g$, respectively.
B.3 Computational Cost. The permutation count $K$ denotes the number of channel orderings sampled per cell’s conditional game $w_g$. The evaluation count $N_{\text{eval}}$ is the total number of model forward passes required for all attribution scores of one image. Reading Eq. (9) literally, each sampled permutation $\pi$ evaluates $w_g$ on $C\!+\!1$ cumulative subsets; $w_g(\emptyset)=0$ needs no model evaluation, and the endpoint $v(B_g)$ is shared across all $K$ permutations. Hence the per‑cell forward‑pass bound is $K(C\!+\!1)$ in the worst case (conservatively reported), while cached implementations reduce the count toward $K(C\!-\!1)+2$. For small $C$ (e.g., CUB‑200 with $C=3$) SSI enumerates all $2C$ subsets; for larger $C$ (EuroSAT $C=13$, HLS $C=6$) it uses $K=10$ permutations. Table 4 reports $N_{\text{eval}}$ under this worst‑case convention.
**Table.** Comparison of computational costs (number of model forward passes) for different attribution methods across three datasets with varying grid sizes and channel counts ($C$).
Computational‑cost trade‑offs are crucial for large‑scale multispectral data: SSI’s joint attribution matrix can be obtained with a fraction of the forward passes required by MC‑Owen, enabling practical deployment on high‑resolution satellite imagery.
Metric Definitions
Defines the evaluation metrics and protocols used to assess attribution quality.
Joint deletion starts from the original input and masks position‑channel players $(g,c)$ in decreasing order of signed attribution score, recording the target‑class logit after each step. Lower deletion AUC and higher insertion AUC indicate better attribution quality.
Rank‑1 projection reshapes the grid‑level attribution scores into a matrix $A\in\mathbb{R}^{RC\times N}$ and computes its best rank‑1 approximation $A=\sigma_1\,u_1\,v_1^{\top}$, where $\sigma_1$ is the largest singular value and $u_1$, $v_1$ are the corresponding singular vectors.
Spatial aggregation obtains a spatial map by summing over channels: $A_{\text{sp}}(g)=\sum_c A(g,c)$. The map is thresholded at several top‑coverage levels and compared to ground‑truth masks using mean Intersection over Union (mIoU).
Spectral aggregation produces channel scores by summing over spatial cells: $A_{\text{ch}}(c)=\sum_g A(g,c)$. Channels are then masked in decreasing $A_{\text{ch}}(c)$ order, and hit‑rate metrics compare the top‑ranked channels against known spectral priors.
Deletion AUC measures how fast the model’s confidence collapses as the most‑attributed position‑channel players are removed; a lower area under the curve means the attribution correctly identified the most influential features. Insertion AUC measures how quickly confidence rises when the most‑attributed players are added back; a higher area indicates the attribution captures truly useful evidence.
ROAD (Remove and Debias) evaluates faithfulness by removing features in two orders: most‑relevant first (MoRF, lower AUC is better) and least‑relevant first (LeRF, higher AUC is better). It mitigates evaluation artifacts caused by distribution shift when features are removed.
**Table 5.** ROAD evaluation on HLS Burn Scar. LeRF AUC (higher is better) and MoRF AUC (lower is better). Values are mean ± reported variability, as in the author rebuttal. Best scores per column are in bold.
Spectral band priors encode physically grounded expectations about which channels should be positively associated with each class. For HLS Burn Scar we use Red, SWIR1, and SWIR2; for EuroSAT vegetation classes we use red‑edge and NIR bands; industrial classes rely on SWIR bands; other classes receive no positive prior.
Grid‑resolution experiments (Tables 6–9) vary the spatial grid from $4\times4$ to $8\times8$ and report joint deletion AUC, rank‑1 deletion AUC, and GridSum insertion AUC. SSI consistently achieves the best scores across all resolutions, demonstrating robustness to grid size.
**Table 6.** Faithfulness test on HLS Burn Scar with 4x4 grid. Values are mean ± bootstrap std. Best in bold.
**Table.** Comparison of different methods across metrics including Ch.Del. ↓, Hit@1, Hit@2, Hit@3, Hit@4, and Hit@5.
**Table 8.** Faithfulness test on CUB-200 with 8x8 grid. Values are mean ± bootstrap std. Best in bold.
**Table 9.** Faithfulness test on EuroSAT with 4x4 grid. Values are mean ± bootstrap std. Best in bold.
Spatial localization curves (Tables 10–11) report mIoU at multiple top‑coverage thresholds. SSI consistently achieves the highest mIoU across all thresholds on both CUB‑200 and HLS Burn Scar, confirming precise spatial pinpointing.
**Table 10.** CUB-200 spatial localization. mIoU at top-coverage levels. Values are mean ± bootstrap std. Higher is better. Best scores are in bold.
**Table 11.** HLS Burn Scar spatial localization. mIoU at top-coverage levels. Values are mean $\pm$ bootstrap std. Higher is better. Best scores are in bold.
Full spectral hit curves (Tables 12–13) evaluate channel‑wise deletion AUC and Hit@K metrics. SSI attains the lowest channel deletion scores and the highest hit rates across all K values for both HLS Burn Scar and EuroSAT.
**Table 13.** EuroSAT spectral aggregation. Lower channel deletion scores and higher hit rates are better. Values are mean ± bootstrap std. Best scores are in bold.