Noise2Noise researchers test L1 vs L2 in self-supervised denoising on Kodak24 and SIDD; Retraining on SIDD pairs yields 9.4–11.0 dB gains

Noise2Noise Revisited: Training Pair Distributions Dominate Loss Choice in Self-Supervised Denoising

Noise2Noise Revisited: Training Pair Distributions Dominate Loss Choice in Self-Supervised Denoising Dingyan Shang* Independent Researcher Frisco, USA dingyanshang@gmail.com *Corresponding author Zhenyu Xu Independent Researcher Fulshear, USA zhenyuxu0918@gmail.com Youting Wang Independent Researcher Mountain View, USA wang.yout@northeastern.edu Bonan Shen Independent Researcher Long Island City, USA shenbonan2@gmail.com Bowen Liu Independent Researcher South San Francisco, USA bliu0962@usc.edu Abstract—Noise2Noise (N2N) trains denoisers on pairs of inde- pendently corrupted observations, eliminating clean references. We stress-test two natural conjectures about why the L1 loss outperforms L2 here. First, the hypothesis that the L1 loss confers robustness via parameter sparsity confuses the loss with Lasso regularization: an explicit Lasso penalty produces the predicted sparsity yet fails to reproduce L1’s cross-noise behavior, while L1- and L2-trained weight distributions are indistinguishable. Second, the population optima of the two losses coincide exactly for symmetric signal posteriors and nearly so for concentrated ones. Measured differences are therefore dominated by optimiza- tion dynamics (bounded-influence gradients), which we probe with gradient statistics and contaminated-target training. On Kodak24 with five synthetic noise families, the L1 loss holds a statistically significant edge over L2, below 1 dB PSNR, holding across three seeds on 13 of the 14 noise columns. On real camera noise the loss is not the decisive variable in distribution: on official SIDD validation blocks, synthetic-Gaussian-trained N2N models gain only 0.8 to 3.7 dB over the noisy input regardless of loss, while retraining on SIDD’s own noisy pairs, never reading ground truth, gains 9.4 to 11.0 dB, far ahead of BM3D. All metrics are on raw network outputs, and the study makes no leaderboard claim. The training pair distribution, not the loss, carries the inductive bias. That design rule applies wherever clean references are unobtainable, from microscopy to industrial inspection sensors. Index Terms—image denoising, self-supervised learning, Noise2Noise, robust statistics, real-noise benchmarks, industrial inspection, nondestructive evaluation I. INTRODUCTION Supervised deep denoisers [1], [2] require registered clean/noisy pairs, costly or impossible in microscopy, in low- light photography, and in industrial nondestructive testing, where each unit under inspection is unique and a noise-free reference of it is physically unobtainable [3]. Noise2Noise (N2N) [4] showed that two independent noisy observations of the same signal suffice: under zero-mean noise, the network trained to map one observation to the other converges in expectation to the clean-target minimizer. Two practical questions remain. (i) How much of N2N’s observed cross-noise robustness is attributable to the loss function? (ii) The empirical edge of the L1 loss over L2 in restoration networks is documented [5] and attributed there to optimization behavior; is that attribution right, or does the credit belong to parameter sparsity? We answer both with a controlled protocol and make three contributions:

  1. A corrected theoretical framing. We distinguish the L1 loss (a residual penalty; maximum likelihood under a Laplace residual model) from L1 regularization (a weight penalty; Lasso [6]), and we delimit what the loss can change: under zero-mean training noise the MSE optimum is exactly the posterior mean, and the MAE optimum coincides with it when the signal posterior is symmetric, and approximately when it is concen- trated. The measurable loss-function effect reduces to finite-sample optimization dynamics (MAE’s bounded- influence gradients [5], [7]), supported by gradient statis- tics and contaminated-target training.
  2. A control experiment dissociating the L1 loss from L1 regularization. We train the same U-Net with MSE, MAE, and MSE+Lasso under identical schedules. Only the explicit Lasso penalty produces weight sparsity, and it does not inherit the MAE cross-noise profile, ruling out sparsity as the carrier of L1’s edge.
  3. A loss-controlled quantification and closure of the real-noise domain gap. On the official SIDD valida- tion blocks [8], synthetic-Gaussian-trained N2N recovers only +0.8 to +3.7 dB over the noisy input regardless of loss. Retraining the same five losses on SIDD-Medium noisy/noisy pairs (scene-disjoint 128/32 split, ground truth never read during training) lifts every variant to +9.4 to +11.0 dB, well ahead of BM3D [9]. The gap itself is established knowledge [10], [11]; new here is its same-architecture, same-budget measurement across five losses within the N2N paradigm, its closure with arXiv:2609.16788v1 [cs.LG] 15 Sep 2026

noisy pairs only, and a content-matched control that splits the gap into ≈5 dB of scene statistics and ≈3 dB of noise distribution. This ≈8 dB regime gap dwarfs every in-distribution loss effect we measure (≤0.8 dB between MAE and MSE on synthetic noise, ≤2.3 dB across the five real-pair losses); only out of distribution do loss effects grow large, governing robustness rather than accuracy (Section V-D). All evaluations report PSNR, SSIM, and LPIPS [12] on the raw network output; a post-hoc image-enhancement stage present in an earlier draft was removed as an evaluation confound (Section VI-B). II. RELATED WORK Denoising without clean targets. N2N [4] trains on in- dependently corrupted pairs. Blind-spot methods remove even the second observation: Noise2Void [13] and Noise2Self [14] predict masked pixels under a pixel-independence assump- tion; Laine et al. [15] add noise-model-aware posterior infer- ence to the blind-spot architecture; Noise2Same [16] relaxes strict J-invariance; Noise2Sim [17] mines self-similar patches; Neighbor2Neighbor [18], Recorrupted-to-Recorrupted [19], and Noisier2Noise [20] synthesize N2N-style pairs from single noisy images; Self2Self [21] trains on one image alone with dropout ensembling; Blind2Unblind [22] revisits blind-spot information loss. Real-noise self-supervision. On real sRGB camera noise, spatial correlation breaks the pixel-independence assumption of blind-spot training. That Gaussian-trained denoisers transfer poorly to real photographs is itself established: CBDNet [10] motivates realistic noise modeling with this failure, and Brooks et al. [11] trace it to the sRGB processing pipeline. AP-BSN [23] and CVF-SID [24] are current self-supervised references on SIDD (35.97 and 34.81 dB on the validation split; super- vised state of the art is 39–40 dB). More recent systems attack precisely the mismatch we quantify, aligning the effective training distribution with the test noise: Positive2Negative [25] replaces information-lossy masking and downsampling with renoising-consistency training, next-scale prediction [26] builds cross-scale targets that decorrelate real noise without destroying detail (37.1 dB), and Learning-to-Recorrupt [27] learns the recorruption map when the noise distribution is unknown. Our real-pair N2N results (33–35 dB, Section V-D) sit just below this self-supervised band, which is consistent with our deliberately compact backbone and budget. We do not compete with these systems; the real-noise setting is a controlled probe of whether the loss function or the training pair distribution carries the inductive bias. Our N2N-on-real- pairs configuration is simply [4] applied to SIDD’s two noisy shots per scene; its value is diagnostic. N2N trained on SIDD- Medium pairs already appears as a baseline row in the raw- domain tables of [18]; we run the configuration in sRGB, loss- controlled. Classical baselines. BM3D [9] remains the reference non- learning denoiser and is still being refined, e.g. by varying the transform in its collaborative-filtering stage [28]; we therefore treat its configuration as an experimental variable rather than a constant (Section VI-B). Loss functions for restoration. Zhao et al. [5] compare L2, L1, and SSIM-based losses for restoration networks and attribute L1’s edge over L2 to optimization behavior rather than statistical modeling, consistent with the framing we derive in Section III. Charbonnier [29] and Huber losses interpolate between the two regimes. III. THEORY: WHAT THE LOSS CAN AND CANNOT DO A. Likelihood View and Its Limit For a residual model r = x −fθ(x′), minimizing P i ρ(ri) is maximum likelihood under p(r) ∝e−ρ(r): MSE corresponds to a Gaussian residual (optimal predictor: conditional mean), MAE to a Laplace one (conditional median). In the N2N setting the training target is x2 = s + n2 with n2 symmetric zero-mean noise independent of x1 and of the signal s: true by construction in our synthetic arm, violated by signal-dependent real noise (Section V-D). The MSE optimum is then exactly the posterior mean, E[x2 | x1] = E[s | x1], with no further assumptions. The MAE optimum is the median of p(x2 | x1), i.e., the signal posterior p(s | x1) convolved with the noise density, and convolving a skewed posterior with a symmetric density does not symmetrize it. Hence, pointwise in the predicted value z given x1, arg min z E

(z −x2)2 x1

= arg min z E[ |z −x2| | x1] (1) holds exactly when p(s | x1) is symmetric about its mean, and approximately whenever the posterior is concentrated relative to the noise, whose symmetric density then dominates the convolution.1 Neither condition is guaranteed for natural images, so the population-level gap between the two losses is the mean–median distance of p(x2 | x1): bounded above by one standard deviation of that distribution, typically far smaller, but not zero. Systematic differences between trained networks therefore combine this small statistical component with the finite-sample optimization path: the MAE gradient is sign(r), so each residual’s influence on the update is bounded [7], while the MSE gradient grows linearly in r, so rare large residuals dominate updates. The sub-dB gaps we measure (Section V-B) are consistent with near-coincident optima. We treat cross-noise evaluation as an empirical question. B. The L1-Loss-versus-L1-Regularization Confusion Lasso regularization augments a base loss J0 with a weight penalty, J = J0+α P i |wi|, whose fixed points exhibit sparse w [6]. The loss J0 = ∥x2 −fθ(x1)∥1 penalizes pixel-domain residuals; its gradient flows to activations, not weights, and no mechanism connects it to weight sparsity. Section V-C tests the dissociation directly. 1Clamping training pairs to [0, 1] truncates the noise asymmetrically at sat- urated pixels, so the zero-mean-symmetry premise is itself only approximate at the range bounds.

C. Hypotheses (H1) MAE-class losses (MAE, Charbonnier, Huber) general- ize no worse than MSE to additive heavy-tailed test noise, with any advantage concentrated in perceptual metrics. (H2) Weight sparsity is neither necessary nor sufficient for the MAE-class cross-noise profile. (H3) When the test noise violates the addi- tive zero-mean training assumption structurally (multiplicative speckle at high variance; real signal-dependent sRGB camera noise), N2N fails irrespective of loss, and matching the training pair distribution to the test noise restores the guarantee. IV. METHOD A. Architecture A U-Net [30] (depth 4, base width 48, BN, ReLU) with a global residual connection: the network predicts a correction added to its input, DnCNN-style [1]; held fixed across all loss variants. B. Training Distributions Synthetic arm. 400 color images from the BSDS500 train+test splits (the standard CBSD400 set) [31] (disjoint from all evaluation sets). Each sample is a random 128×128 crop; x1, x2 receive independent additive Gaussian noise with σ1, σ2 ∼U(5, 50) (8-bit scale). The training family is de- liberately narrow so that cross-noise behavior is attributable to the loss. Real arm. SIDD-Medium