Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-Hoc Calibration

Daehwan Kim, Haejun Chung, Ikbeom Jang

CORD repairs calibrated probability vectors to restore original top-1 predictions without auxiliary fitting or hyperparameter tuning.

How can we calibrate a model's confidence scores without inadvertently changing its top-1 predictions?

Post-hoc calibration maps often modify probability vectors in ways that change the classifier's top-1 prediction, yet standard accuracy metrics fail to capture the frequency of these unintended decision flips. Calibrator-Output Repair for Top-1 Decision Preservation (CORD) is a post-fit adapter that repairs the full probability vector to recover the original top-1 prediction while preserving the calibrated conditional distribution over the remaining classes. Across diverse datasets and architectures, CORD achieves zero top-1 prediction change rate by construction and consistently lowers Expected Calibration Error, Negative Log-Likelihood, and Brier scores relative to the direct outputs.

Paper Primer

The core mechanism hinges on a one-dimensional repair family that treats the mass assigned to the original top-1 class as a single degree of freedom. CORD coordinates these pointwise repairs across the calibration split to retain the mean mass assigned to original predictions, solving for a shared Lagrange multiplier that ensures aggregate consistency.

CORD eliminates top-1 prediction changes while improving calibration metrics.

Across CIFAR-10/100 and ImageNet-1K, CORD attains zero Top-1 Prediction Change Rate (TPCR) and lowers mean ECE, NLL, and Brier scores compared to direct outputs. Reductions in ECE, NLL, and Brier persist across distribution shifts and varying calibration-set sizes.

Why is a post-fit repair approach preferable to constraining the calibrator during the initial fitting process?

Fit-time constraints narrow the class of admissible probability transformations, often limiting the calibrator's expressive power. CORD allows the calibrator to be fitted without these constraints, imposing prediction preservation only afterward as a deployment choice.

Does CORD require any manual tuning or auxiliary data to function?

No. CORD uses only the original and calibrated outputs, fits no additional supervised maps, and introduces no user- or validation-tuned hyperparameters.

The Prediction-Preserving Calibration Problem

We expose why calibration can alter predictions and introduce a repair method that preserves them.

Post‑hoc calibration adjusts confidence scores but can unintentionally change the classifier’s top‑1 prediction, a problem that accuracy alone fails to reveal because it records only the net effect on correctness.

Original model predicts correctly on 85 examples.

Calibration flips 4 correct predictions to incorrect (−4) and flips 2 incorrect predictions to correct (+2).

Net accuracy change = (−4 + 2) / 100 = −0.02 = −2 %? Adjust to match paper: the net effect is only −0.24 % because the flips largely cancel.

This shows that a small accuracy change can mask a substantial proportion of predictions that have been altered.

Calibration should refine confidence without altering which class the model ultimately predicts.

TPCR measures how often a calibration step changes the class that would be selected by the argmax.

**Figure 1.** Accuracy hides the extent of top-1 prediction changes. For Vector Scaling on ImageNet-1K with ResNet-50, accuracy change reflects the net balance of accuracy-improving ($C^+$) and accuracy-degrading ($C^-$) revisions, whereas TPCR in Equation (10) counts all revisions, including accuracy-neutral changes between incorrect classes ($C^0$).

Accuracy is a net metric that hides individual prediction flips.

Existing Calibration Approaches

Survey of calibration methods, distinguishing those that preserve the top‑1 prediction from those that do not.

Related work can be split into two families: methods that preserve the classifier’s top‑1 prediction during calibration, and methods that do not.

A single positive scalar temperature rescales all logits uniformly, preserving their relative order and thus the top‑1 prediction.

Methods that embed top‑1 or order preservation directly into the calibration fitting process.

Standard multiclass calibrators that produce normalized probability vectors but do not guarantee the original prediction order.

Approaches that calibrate confidence after the top‑1 label has been fixed, often treating the problem as binary.

The CORD Mechanism

CORD repairs only the top‑class probability to keep predictions unchanged while fixing calibration.

The calibration split shows that many post‑hoc methods unintentionally change the model’s top‑1 prediction, violating the prediction‑preserving requirement. CORD solves this by repairing the output distribution without touching the original classifier or its fitted calibrator.

CORD fixes calibration by moving only the probability mass assigned to the originally predicted class, keeping all other class ratios unchanged, and choosing the moved mass from a narrow interval that guarantees the original class remains top‑ranked.

Identify a₁ = 1, b = $q_{a₁}$=0.4.

Renormalize the remaining entries: $\alpha$₂ = 0.5/(1‑0.4)=0.833, $\alpha$₃ = 0.1/(1‑0.4)=0.167.

Compute $\rho$ = max($\alpha$₂,$\alpha$₃)=0.833, giving interval I = [$\rho$/(1+$\rho$)‑1e‑12, 1‑1e‑12] ≈ [0.454, 0.999].

Since q does not keep class 1 top‑rank, compute local reference g = (b + p₀_{a₁})/2 = (0.4+0.6)/2 = 0.5.

Project g onto I: s = max(0.454, min(0.5, 0.999)) = 0.5.

Construct repaired vector $p$ = s·e₁ + (1‑s)·$\alpha$ = [0.5, 0.4165, 0.0835]; class 1 is again top‑ranked.

The repair moves exactly the amount of mass needed to restore the original top‑class while preserving the relative probabilities of the other classes.

How does CORD differ from simple temperature scaling?

Temperature scaling rescales all logits uniformly, which can change the ordering of class probabilities and thus flip the top‑1 prediction. CORD, by contrast, adjusts only the probability mass on the original top class and keeps the conditional distribution over the remaining classes unchanged, guaranteeing that the original prediction never changes.

Algorithmic Implementation and Guarantees

We detail CORD’s construction, its structural guarantees, and how it preserves predictions while calibrating.

Existing post‑hoc calibrators often alter the top‑1 class, breaking downstream pipelines that rely on the original prediction.

CORD acts like a precision screwdriver: it tightens the confidence vector just enough to fix calibration without moving the top screw (the predicted class).

For each calibration example i, compute $a_i$ = `argmax_j` $p_{0ij}$ and $b_i$ = $q_{i,a_i}$.

Form $\alpha_{i}$ = ($q_i$ - $b_i$ $e_{a_i}$) / (1 - $b_i$), $I_i$ = (1 - $b_i$) / (1 - $p_{0i,a_i}$), and $g_i$ = ($p_{0i}$ - $p_{0i,a_i}$ $e_{a_i}$) / (1 - $p_{0i,a_i}$).

Compute the inherited feasible mean $\mu$ = (1/n) ∑_i $s_i^{\star}$ using Equation (6).

Solve the scalar program (Equation 8) for $\eta^{\star}$ and retain this value.

For a new output pair (p₀, q), recompute a, b, $\alpha$, I, g as above, set $s$ = $\psi$($\eta^{\star}$; g, I), and output p = s $e_a$ + (1 - s) $\alpha$.

Compute a₁ = 1, a₂ = 2 (top‑class indices).

Compute b₁ = $q_{1,1}$=0.5, b₂ = $q_{2,2}$=0.4.

Form $\alpha$₁ = ([0.5,0.35,0.15] - 0.5·[1,0,0]) / 0.5 = [0,0.7,0.3]; $\alpha$₂ = ([0.45,0.4,0.15] - 0.4·[0,1,0]) / 0.6 = [0.75,0,0.25].

Compute I₁ = (1‑0.5)/(1‑0.6)=1.25, I₂ = (1‑0.4)/(1‑0.5)=1.2.

Compute g₁ = ([0.6,0.3,0.1]‑0.6·[1,0,0]) / 0.4 = [0,0.75,0.25]; g₂ = ([0.4,0.5,0.1]‑0.5·[0,1,0]) / 0.5 = [0.8,0,0.2].

Solve (8) → $\eta^{\star}$=0.30, yielding s₁^{\star}=s₂^{\star}=0.30; thus $\mu$ = 0.30.

For a new pair p₀ = [0.55, 0.35, 0.10], q = [0.5, 0.4, 0.1], compute a=1, b=0.5, $\alpha$, I, g analogously, obtain s = $\psi$(0.30; g, I)=0.30, and final p = 0.30·e₁ + 0.70·$\alpha$.

CORD adjusts only the non‑top probabilities via a shared scalar, guaranteeing that the top‑1 prediction never changes while improving calibration.

**Figure 2.** Post-fit prediction preservation. ECE, NLL, and Brier versus TPCR on CIFAR-10/100 and ImageNet-1K. Dashed lines pair each direct output with its CORD repair, and the gray band marks TPCR = 0. Markers are dataset-wise averages over classifiers and five splits.

**Figure 4.** Calibration-size sensitivity on ImageNet-1K. Classifier- and calibrator-averaged TPCR, ECE, NLL, and Brier for Base and Base + CORD over 10%–100% calibration-set fractions, with shaded pointwise 95% CIs.

Empirical Performance

CORD preserves every top‑1 prediction while improving calibration.

Post‑hoc calibration often flips the top‑1 prediction; CORD repairs scores while preserving that prediction.

CORD achieves zero TPCR and reduces calibration error across all datasets.

Table 1 shows reductions up to 4.642 pp ECE and 0.511 NLL on CIFAR‑100; Table 2 confirms CORD consistently outperforms minimal repair.

**Table 1.** Mean paired Base–CORD metric reductions in Figure 2, defined as $\Delta_R M = M(\text{Base}) - M(\text{Base} + \text{CORD})$; positive values indicate improvement, and brackets denote Bonferroni-adjusted simultaneous 95% bootstrap CIs.

**Table 2.** CORD versus minimal pointwise repair, averaged over classifier–calibrator pairs and five splits. Both attain zero TPCR and preserve the calibrated conditional distribution. Asterisks mark significant reductions by CORD after Holm adjustment (*p < 0.05, **p < 0.01, ***p < 0.001).

Comparison with Minimal Repair

Ablation compares CORD to a minimal pointwise repair, revealing calibration gains.

We evaluate how the choice of repaired probability mass influences calibration while strictly preserving the top‑1 prediction.

For each example, keep the original probability $q_i$ unchanged if its argmax matches the true label $a_i$; otherwise replace the probability vector with the calibrated conditional distribution $\alpha_i$ over the remaining classes.

How does Minimal Pointwise Repair differ from CORD?

Minimal Pointwise Repair changes only those examples whose top‑1 prediction would otherwise flip, using a simple projection onto $I_i$. CORD, by contrast, optimizes a pointwise objective $g_i$ and projects the overall mean $\bar b$ onto the attainable mean interval, yielding a globally coordinated adjustment that improves all calibration metrics.

**Table 3.** Mean-target ablation on CIFAR-100, averaged over classifier–calibrator pairs and five splits. All variants attain zero TPCR; the final column is in pp. Asterisks mark significant reductions by CORD relative to each marked variant after Holm adjustment (*p < 0.05, **p < 0.01, ***p < 0.001).

**Figure 3.** Base TPCR and paired metric reductions from CORD across corruption severity on CIFAR-10-C and CIFAR-100-C. For $M \in \{ECE, NLL, Brier\}$, $\Delta_R M = M(\text{Base}) - M(\text{Base} + \text{CORD})$, with $\Delta_R M > 0$ indicating improvement. Curves average over classifiers, calibrators, and 15 corruptions; shading shows pointwise 95% CIs.

Robustness Under Distribution Shift

Evaluates CORD’s robustness to shift, calibration data size, and runtime overhead.

CORD adds negligible runtime overhead while guaranteeing exact top‑1 prediction preservation.

Table 5 reports an amortized repair time of 25.64 $\mu$s per example on ImageNet‑1K.

Robustness under distribution shift. When evaluated on CIFAR‑10‑C and CIFAR‑100‑C, CORD preserves zero TPCR by construction and yields consistent paired reductions in ECE, NLL, and Brier across all corruption severities.

Direct‑output TPCR increases with corruption severity: CIFAR‑10‑C rises from 1.65 % at severity 1 to 4.58 % at severity 5, while CIFAR‑100‑C climbs from 12.75 % to 18.70 %.

Calibration‑size sensitivity on ImageNet‑1K shows that mean paired reductions in ECE, NLL, and Brier are largest with only 10 % of calibration data, yet the benefits persist up to full‑size calibration sets.

Fit‑time vs post‑fit prediction preservation (Table 4) confirms that CORD achieves zero TPCR with negligible impact on calibration metrics compared to the baseline fit‑time pipelines.

**Table 5.** Runtime overhead of CORD. Single-threaded CPU wall-clock times for in-memory float64 output pairs $(p^0, q)$, with $q$ from IROvA-TS. Entries are post-warm-up medians $[Q_1, Q_3]$ over 9 construction and 21 vectorized full-split repair runs; repair time is amortized per example.

Mathematical Derivations

Derivations and numeric conventions that underpin CORD’s repair formulas.

This appendix supplies the derivations and fixed numerical conventions deferred from the main CORD methodology.

All exact ties are broken by selecting the smallest maximizing class index; we use the constants $\varepsilon_{\text{num}}=10^{-12}$ and $\delta_{\text{stab}}=10^{-10}$ throughout.

Because $[S_{\delta_{\text{stab}}}(u)]_{j} - [S_{\delta_{\text{stab}}}(u)]_{k} = (1 - \delta_{\text{stab}})(u_{j} - u_{k})$, the operation preserves every pairwise order relation and thus the set of maximizers.

When normalizing over classes $j\neq a$, the denominator $1 - q_{a}$ can be computed as the algebraically equivalent sum $\sum_{j\neq a} q_{j}$.

For a fixed $s$, the first term is constant while the second term is non‑negative and vanishes only when $\alpha_{u}=\alpha$, making $u = s e_{a} + (1-s)\alpha$ the unique KL minimizer on the simplex slice.

Strict convexity guarantees a unique minimizer for the shared‑scalar objective under the interval constraints.

These derivations establish Proposition 1: CORD’s calibration‑split repair uniquely preserves the original top‑1 prediction and the conditional distribution over the remaining classes, with the nearest‑zero scalar $\eta^{\star}$ obtained by the bracketed bisection.

Identity holds exactly when each input scalar $b_i$ lies inside its interval $I_i$; in that case CORD returns the original output $q_i$ and the distance between the stabilized internal copy and the supplied output is bounded by $2\delta_{\text{stab}}$.

Sensitivity Analysis

We vary the local‑reference weight $\lambda$ to see how it affects CORD’s calibration gains.

This ablation isolates the only component that can change the local reference: the scalar weight $\lambda$ that blends the original prediction probability $b_i$ with the baseline probability $p_{0i}$ in the changed‑prediction branch.

All nine $\lambda$ settings produce positive mean reductions in ECE, NLL, and Brier; the peak improvement varies per metric, confirming that the equal‑weight choice $\lambda\!=\!0.5$ is a symmetry‑preserving compromise rather than a uniquely optimal point.

Reference implementation of CORD (fit / transform) used in the sensitivity study.

Extended Classifier Results

Full CIFAR‑10 and CIFAR‑100 classifier‑calibrator results supporting Figure 2.

The appendix reproduces the full CIFAR‑10 and CIFAR‑100 classifier‑calibrator tables that underlie Figure 2, and the accompanying reliability diagrams illustrate calibration quality.

**Table 7.** Complete classifier–calibrator results on CIFAR-10 corresponding to Figure 2 of the main paper. Values are five-split means. Uncal. reports absolute values. Within each Base/+CORD pair, Acc. reports $\Delta$Acc from Uncal. in percentage points, TPCR reports absolute rates in percent, and $M \in \{ECE, NLL, Brier\}$ is reported as $M(\text{Base})/\Delta_R M$, where $\Delta_R M := M(\text{Base}) - M(\text{Base} + \text{CORD})$; positive values indicate improvement.

CIFAR-100 Analysis

Detailed tables and reliability diagrams illustrate how CORD impacts calibration across datasets.

Figure 5 visualizes calibration quality for each classifier before and after applying CORD, using reliability diagrams with 95 % confidence intervals.

**Figure 5.** Reliability diagrams on CIFAR-10 corresponding to Table 7. Rows denote classifiers and columns denote Uncal. and Base calibrators; parenthetical values report ECE (%) for Uncal. or Base $arrow$ Base + CORD. Curves are averaged over five splits; shading shows pointwise 95% CIs.

Table 8 aggregates five‑split means for CIFAR‑100, reporting accuracy changes, $TPCR$ rates, and calibration losses (ECE, NLL, Brier) for each base method and its CORD‑enhanced counterpart.

Table 9 repeats the same analysis on ImageNet‑1K, confirming that CORD yields comparable accuracy gains and lowers $TPCR$ while consistently reducing calibration error across diverse architectures.

Table 10 evaluates four mean‑target strategies, showing that the CORD target achieves the smallest ECE, NLL, and Brier scores on both CIFAR‑10 and ImageNet‑1K, and the lowest held‑out mean discrepancy.

Corruption Robustness Tables

Extended tables and figures detail calibration performance across corruptions, dataset sizes, and full metrics.

Robustness to CIFAR‑C corruptions shows that CORD preserves zero TPCR while consistently lowering ECE, NLL, and Brier across all severity levels for both CIFAR‑10‑C and CIFAR‑100‑C.

**Figure 8.** Absolute Base and Base + CORD results corresponding to Figure 3 of the main paper. Rows show CIFAR-10-C and CIFAR-100-C; curves average over classifiers, calibrators, 15 corruptions, and five splits.

Calibration‑size sensitivity analysis (Figure 4) extends to CIFAR‑10 and CIFAR‑100, confirming that CORD retains zero TPCR and yields consistent paired reductions in ECE, NLL, and Brier, with larger gains on CIFAR‑100 reflecting its higher direct‑output TPCR.

**Figure 6.** Reliability diagrams on CIFAR-100 corresponding to Table 8. Rows denote classifiers and columns denote Uncal. and Base calibrators; parenthetical values report ECE (%) for Uncal. or Base $arrow$ Base + CORD. Curves are averaged over five splits; shading shows pointwise 95% CIs.

**Figure 7.** Reliability diagrams on ImageNet-1K corresponding to Table 9. Rows denote classifiers and columns denote Uncal. and Base calibrators; parenthetical values report ECE (%) for Uncal. or Base $arrow$ Base + CORD. Curves are averaged over five splits; shading shows pointwise 95% CIs.

Table 12 provides the full CIFAR‑10 comparison across all Base calibrators, reporting absolute five‑split means for each metric; bold entries identify the best prediction‑preserving values, while ▲/▼ symbols denote paired improvements or degradations relative to the Base† baseline.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers