Color Pass-Through via Camera-Display Coupling

Ruikang Li, Molin Li, Jiarui Wu, Zhe Wei, Pengpeng Liu, Tianfan Xue

Color Pass-Through couples camera and display into a single learned system to eliminate perceptual color shifts.

How can we accurately reproduce real-world colors on a smartphone screen despite the inherent color, brightness, and contrast shifts introduced by the camera-display pipeline?

Smartphone displays often fail to reproduce the true colors of a scene because cameras and displays are calibrated as independent, isolated stages, creating an information bottleneck that accumulates error. The authors treat the camera and display as a single, coupled system, using a lightweight neural projector to learn the end-to-end mapping and a one-step observer-specific coefficient to correct residual color casts. This approach achieves a +2.0 point gain in user studies and over 2× improvement on quantitative metrics compared to standard ICC-based calibration pipelines.

Paper Primer

Standard imaging pipelines decompose the capture-to-display process into two separate stages, mapping scene radiance to an intermediate color space and then to display signals. This separation ignores the high-dimensional nature of light, forcing the system to discard spectral information and resulting in washed-out or tinted images that do not match the original scene.

Color Pass-Through replaces this decoupled workflow with an end-to-end learned neural projector: it maps captured camera measurements directly to display signals, then applies a low-dimensional correction to account for the observer's specific spectral sensitivities. The core move is to treat the camera-display pair as a single autoencoder, where the projector acts as a learned pseudo-inverse of the combined hardware response.

Improved perceptual color reproduction across diverse illuminants.

User study ratings and quantitative metrics (MSE) compared against default ISP, multi-illuminant auto-white balance, and ColorChecker calibration. +2.0 points on a 5-point scale and >2× improvement on quantitative metrics.

Why does this approach require per-device calibration instead of a universal fix?

Because the camera-display projector is specific to the hardware's unique spectral sensitivities and non-linearities, and the residual color cast depends on the specific observer's spectral response, which varies between humans and digital sensors.

What is the "camera-null" subspace, and why does it matter?

It represents spectral components invisible to the camera but visible to the human eye; because the camera cannot "see" these components, it cannot correct them, necessitating a separate learned predictor to compensate for the resulting color mask.

Introduction to Color Pass-Through

We expose and address the persistent color mismatch between captured scenes and smartphone displays.

When a smartphone captures a real‑world scene and shows it on its screen, the image often looks noticeably different in hue, brightness, and contrast. This perceptual gap persists despite advances in sensor dynamic range and display gamut, because most pipelines split the high‑dimensional capture‑to‑display process into separately calibrated camera and display stages linked by low‑dimensional color transforms, creating bottlenecks and accumulating errors.

To close this gap we introduce **Color Pass‑Through**, an end‑to‑end learned framework that treats the camera and display as a single coupled system rather than two isolated components. By optimizing the full capture‑to‑display path jointly, we both reduce error accumulation and enable a lightweight one‑step calibration for each observer, yielding substantially more faithful color reproduction.

Instead of calibrating camera and display separately, we learn a direct mapping that takes a captured image and produces the exact colors a specific observer would see on the target screen.

**Fig. 1: Color Pass-Through.** Capturing a real scene under multiple unknown light sources and re-displaying it on a smartphone screen often introduces perceptual color inconsistencies (a). Learned multi-illuminant auto white balance reduces illumination-induced color casts (b), while per-scene color-checker calibration maps the image into a standard color space (c); but neither reproduces the same color. In contrast, our method better preserves the perceived scene colors for both digital and human observers (d).

The core gap is the persistent color and brightness mismatch between real‑world scenes and their screen‑displayed reproductions.

Background and Prior Methods

We situate our approach among ICC pipelines, color constancy, and display management literature.

Traditional pipelines separate camera calibration from display calibration, while recent work on color constancy and display management tackles each side in isolation.

Two independent stages map sensor data to a device‑independent reference (e.g., CIE XYZ) and then back to device‑dependent signals.

BPAM adjusts pixel intensities to preserve overall brightness while adapting local colors to the target observer.

CSRNet is a lightweight neural network that predicts a per‑pixel color‑correction matrix conditioned on the scene.

Methods that estimate scene illumination and apply a global 3 × 3 correction to the captured RGB image.

Techniques that map device‑independent color values to display‑specific radiance, often using parametric transfer functions.

**Fig. 2.** Comparison of Methods and Objectives for Color Reproduction.

The Color Pass-Through Framework

Describes how the mapping is split into a learned camera‑display projector and an observer‑specific correction.

When the observer differs from the phone’s camera, a naïve color‑pass‑through leaves a faint tint on the display – the core pain this section eliminates.

Think of the camera and the display as a single optical system; the projector learns how to undo their combined effect so that the camera sees exactly what the display shows.

Compute the combined operator $C D = \begin{bmatrix}0.78&0.22\\0.38&0.62\\0.45&0.55\end{bmatrix}$.

Take its pseudo‑inverse (via SVD) $ (C D)^{\dagger} \approx \begin{bmatrix}1.12&-0.32\\-0.28&1.28\end{bmatrix}$.

Apply $F_C$ to the captured camera color $C s = \begin{bmatrix}0.58\\0.44\\0.46\end{bmatrix}$: $F_C(C s) \approx (C D)^{\dagger} (C s) = \begin{bmatrix}0.60\\0.40\end{bmatrix}$, which recovers the original $s$.

Verify idempotence: applying $F_C$ again yields the same $s$.

This toy example shows that even a linear pseudo‑inverse can perfectly undo the camera‑display chain on the subspace spanned by $C$ and $D$, motivating the neural surrogate for the full nonlinear case.

Isn't $F_C$ just a standard linear color‑transform?

No — a plain linear matrix cannot model the display’s gamma curve, tone‑mapping, or spatial demosaicing artifacts. The learned $F_C(\cdot;\theta)$ is a *non‑linear* function (MLP with positional encoding) that adapts to those effects while still respecting the pseudo‑inverse relationship on the camera‑visible subspace.

**Fig. 3: Overall System of Our Color Pass-Through.** (a) Full color pass-through model corresponding to Eq. (11); (b) Objective function and evaluation criteria.

After undoing the camera‑display chain, a tiny per‑observer tweak removes the color cast that lives in the spectrum invisible to the camera.

Compute the hidden contribution $\hat a_i e = 0.12 \times [0.2,0.5,0.3] = [0.024, 0.060, 0.036]$.

Apply the observer‑specific scaling: $\phi \cdot (\hat a_i e) = [1.0,0.9,1.1] \odot [0.024,0.060,0.036] = [0.024, 0.054, 0.040]$ (element‑wise multiplication).

If the projected camera‑display color is $[0.45, 0.40, 0.38]$, subtract the correction to obtain the final display color $[0.426, 0.346, 0.340]$.

This tiny three‑parameter correction can shift the displayed hue enough to eliminate the perceptual “color mask” seen by non‑camera observers.

How does this differ from a conventional white‑balance adjustment?

White‑balance rescales the three RGB channels uniformly based on a measured illuminant. $\phi$ instead removes a *spectrally hidden* component that the camera cannot see at all, using a learned per‑pixel scalar $\hat a_i$ and a basis $e$ that lives outside the camera’s measurement space.

**Fig. 4:** Learning $\mathcal{F}_C$ from Recaptured Data. A pixel-wise model is trained to approximate $\mathcal{F}_C$ via (a) a re-capture protocol and (b) pixel-aligned input–target pairs.

Experimental Results

Quantitative and user‑study results demonstrate our method’s superior color accuracy.

Recall that the pipeline splits the overall mapping into a learned camera‑display projection $F_C$ and an observer‑specific calibration factor $\phi$, then applies a one‑time coefficient fit for a target observer. This section reports how well each component fits, how the full model compares to strong baselines, and how robust it is under limited data and illumination changes.

**Fig. 19:** Visualized Fitting Comparisons of $\mathcal{F}_C$. Compared with a CNN-based baseline (BPAM [57]) and an MLP-based baseline (CSRNet [39]), our proposed network yields superior quantitative results and lower errors in the corresponding error maps.

**Fig. 8: Separated-view Comparison.** The left three columns show images observed through a phone display, while the rightmost column directly observes the real scene. Our method preserves more consistent colors than other baselines.

**Fig. 9: Direct in-scene Comparison.** Each column shows a smartphone screen with the processed image hovering over the real scene. Our method reproduces closer colors.

**Fig. 10:** User study on brightness and color accuracy. Participants rated similarity to the reference on a 5-point Likert scale (1 for least similar, 5 for most similar).

**Fig. 12:** Learning efficiency of $\hat{\mathcal{F}}_C$.

**Fig. 13:** Robustness of calibrated $\varphi$.

Our end‑to‑end learned correction consistently outperforms commercial pipelines and calibration baselines in both objective metrics and human preference.

Ablation Studies and Design Choices

Ablation studies dissect each component of the Color Pass‑Through pipeline.

This section reports the ablation experiments that isolate the contribution of each stage in the Color Pass‑Through pipeline. We also present quantitative diagnostics that justify the design choices.

The pipeline is divided into three stages—pretraining, calibration, and inference—so that a generic mapping is learned once and then adapted to a specific observer.

Why separate pretraining from calibration instead of learning a single end‑to‑end model?

Pretraining captures the generic camera‑display relationship once, allowing the cheap 3D‑grid calibration to specialize the model for each observer without expensive retraining.

We first recover the camera spectral sensitivity $C$ and the dominant null‑space direction $e$, then train two networks: one for the camera‑display projector and one for the observer‑specific null‑color coefficients.

What would happen if the camera‑null predictor were omitted?

The model would fail to correct metameric‑black colors, producing residual hue errors especially when the target observer differs from the camera.

A single calibration coefficient $\phi$ is selected from a precomputed 3D‑grid, fixing the observer‑specific color bias for all subsequent images.

Why use a 3D‑grid search instead of per‑scene optimization for $\phi$?

The grid yields a stable, scene‑independent coefficient, avoiding costly per‑scene fitting and reducing the risk of overfitting to a single image.

**Fig. 18: Illustration of the Effectiveness of the AvgPool Layer for Learning $\mathcal{F}_C$.** Compared with the vanilla input (a), the input augmented with an AvgPool layer (b) enables the pixel-wise model to distinguish edge pixels introduced by demosaicing (c), thereby producing results with fewer colored edge artifacts (d).

**Fig. 20:** Comparison of ISP-Augmented Color Pass-through. In addition to the original input image (top left), we use the burst-fusion image as a condition (top right) to augment Color Pass-Through. We compare designs inspired by (a) NiLUT [25] and CSRNet [39], (b) SVDLUT [51] and SABLUT [50], (c) HDRNet [33] and DeepUPE [75], and (d) our design. Our method produces results with natural highlights (first row), vivid colors (second row), richer details (third row), and lower noise (fourth row).

**Fig. 21: Comparison of Curve-Augmented Color Pass-Through.** (a) Because displays have a more limited dynamic range than real scenes, the learned mapping may fail in out-of-range regions. (b) To mitigate this issue, we apply a tone-mapping-like curve to the input image, using identity mapping in low-intensity regions and compression in high-intensity regions. (c) Visual comparison with and without the proposed curve. The augmented version produces more consistent color variations in strongly illuminated regions (e.g., the bottom corners).

Practical Implementation and Extensions

We discuss practical model forms, low‑rank correction, and extensions beyond the core pipeline.

Smartphone pipelines distort colors; we split the mapping into a camera‑display projection and an observer‑specific calibration factor.

In practice the camera‑display projector $F_C$ is realized as a pixel‑wise nonlinear network: we first subtract a learned residual $\delta$ᵢ from the raw camera color $C_{si}$, then feed the corrected value into $F_C$, preserving the full‑model formulation even when $F_C$ is nonlinear.

A sufficient condition for $\delta_{i}$ = 0 is that every direction invisible to the camera is also invisible to the observer, i.e. null(C) ⊆ null(M). Under this inclusion the residual (I‑$P_M$)s lies in both null spaces, eliminating the need for a learned correction.

When M = T C with an invertible matrix $T$, the inclusion becomes an equality (null(C)=null(M)), which is the classic Luther–Ives condition up to a linear transform.

Increasing the number of camera channels (e.g., RGBW or multispectral CFA) shrinks null(C), making the inclusion easier to satisfy and often removing the correction term entirely.

Substituting $s_i$ = $r_i$ + $n_i$ into the definition of $\delta_{i}$ shows that the first term cancels, leaving $\delta_{i}$ = C (I‑$P_M$) $n_i$, i.e. the correction originates solely from the camera‑null component.

Because $n_i$ lives in a high‑dimensional spectral space, we approximate it with a low‑rank basis {$e_k$}: $n_i$ ≈ $\Sigma_{k=1}$^K $a_k^{(i)}$ $e_k$, with K ≪ L.

Empirically the first principal component dominates, so we set $K$=1. The correction simplifies to $\delta_{i}$ ≈ $\phi$ $a_i$, a rank‑1 model that the main paper adopts (Eq. 9).

This interpretation explains why a tiny set of learnable parameters (the scalar $a_i$ and the 3‑vector $\phi$) suffices to capture the otherwise high‑dimensional camera‑null effect.

**Fig. 6:** Effectiveness of the Two Learned Components for Color Pass-Through. (a) When using another camera of the same model as the digital observer: (i) the learned camera–display projection alone is sufficient to achieve pass-through, (ii) no observer calibration for M is required; (b) When observer is different from camera (e.g. a DSLR): (i) the camera–display projection alone is insufficient, (ii) adding the camera–null correction for a target observer M yields consistent color reproduction.

Additional experiments (Section C) evaluate the AvgPool augmentation for $F_C$ and compare baseline learners. The AvgPool on the green channel supplies a lightweight spatial cue that improves edge handling, reducing colored‑edge artifacts.

Baseline comparison shows that, despite BPAM’s superiority on generic color‑transfer tasks, our modest MLP‑plus‑AvgPool design outperforms both BPAM and CSRNet when fitting the pixel‑wise projector $F_C$, confirming that $F_C$ has minimal spatial dependence.

Section D explores two extensions. First, we integrate the model with an in‑phone ISP pipeline by conditioning on an HDR‑fused image while feeding the raw sensor data as input; the variant in Fig. 20(d) yields the best results.

Second, we augment the input with a tone‑mapping‑like curve to better preserve highlights, mitigating over‑exposure on displays with limited peak brightness.

**Fig. 15:** Estimating Results of Camera Sensitivity and Null-Space Components. (a) Capture setup and estimated camera sensitivities. (b) PCA variance ratios of the estimated camera null space over the full dataset (left), and variance ratios of the top-3 components for each sample (right), PC1 denotes the first principal component.

**Fig. 16: Observer-Specific 3D Palette Calibration.** (a) A one-time calibration is performed in an arbitrary scene by sampling $\varphi$ in a 3D-grid space. (b) The sampled results are showed on the display. (c) The observer selects the one with $\varphi$ that best matches the real scene, which is then fixed for all subsequent experiments.

Limitations: the learned model is tied to a specific camera–display pair and does not generalize across devices without additional raw‑to‑raw translation; also, current displays cannot reproduce the full dynamic range of real scenes, so extreme highlights still suffer.

Mathematical Derivation

Formal derivations of the linear and nonlinear camera‑display models.

Figure 17 visualizes the logical flow of the derivations that follow; keep it in mind as a roadmap.

**Fig. 17:** Schematic Overview of Detailed Mathematical Proof in Sec. B.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers