A paper presented at ICLR 2026, titled "On the Interpolation Effect of Score Smoothing in Diffusion Models", analyzes why diffusion models often produce novel samples that go beyond their training data. The authors argue that this apparent "creativity" follows from how neural network training tends to smooth the learned score function, a phenomenon they call score smoothing.
Denoising as the core learning task
Training a diffusion model starts by taking real data samples (for example, photos of cats) and progressively corrupting them with noise until they are unrecognizable. The model is trained to reverse this corruption step-by-step, a process known as denoising. If a model learned the denoising mapping perfectly based solely on the training examples, it would reproduce exact copies of those examples at generation time — behavior known as memorization.
In practice, diffusion models usually generalize and create new samples. The authors describe denoising with an analogy: random noise is like a cloud of gas particles in a room, and a learned score function (SF) acts as a force field that pulls each particle so that meaningful shapes emerge. If the SF were perfect, particles would be driven exactly to the training data points (memorization).
A 1-D illustrative example: where interpolation comes from
To illustrate the mechanism, the paper considers a one-dimensional world with only two training points, +1 and −1. A "perfect" score function would switch sign sharply halfway between them, so particles on the left move toward −1 and those on the right toward +1, producing memorization.
However, in practice the score is approximated by a neural network trained with regularization. Regularizers such as weight decay (WD) make it difficult for networks to fit functions with very sharp cliffs. As a result, the learned score becomes smoother in the middle region: the abrupt sign change is replaced by a gentler slope. This score smoothing slows particle flow in the central zone and leads particles to settle between the two training points, producing interpolated values.
The authors verified this effect experimentally by training two-layer ReLU networks to fit the 1-D score, optimizing parameters with the AdamW algorithm under varying amounts of weight decay. Stronger weight decay yielded a smoother learned score in the middle area and a larger interpolation region. They also note that explicit regularization is not the only source of smoothing: implicit regularization from gradient-based training algorithms can produce similar effects.
High-dimensional data and manifold recovery: why samples stay realistic
Real-world data like high-resolution images live in high-dimensional pixel spaces, most of which correspond to meaningless noise. Recognizable images occupy only a small subset, the data manifold. The model must infer the manifold from finite training samples and produce new points along it — a task called manifold recovery.
In multi-dimensional settings, score smoothing acts in a direction-dependent way. Along directions tangent to the data manifold, smoothing produces an effect analogous to the 1-D case, slowing tangential collapse toward individual training points and encouraging interpolation. Along directions normal to the manifold, however, the ideal score is already comparatively smooth (often nearly linear for a flat manifold), so further smoothing has little impact on pushing samples toward the manifold.
Consequently, score smoothing does not simply slow particle movement in all directions (which would trap particles in noisy space and yield blurry outputs). Instead, it reduces the tendency to collapse onto the exact training points along tangential directions while still allowing samples to reach the manifold along normal directions. That balance yields outputs that are both realistic (they land on the data manifold) and novel (they occupy spaces between original training points).
Conclusions and open questions
The paper concludes that the creativity of diffusion models can be understood as a predictable mathematical outcome of how neural networks learn: they rarely fit perfectly sharp score functions, and that imprecision creates bridges that interpolate between known data. In applications such as image generation or drug discovery, this implies diffusion models may propose new images or molecular configurations that blend features of training examples rather than simply recalling exact copies.
The authors emphasize this work as an initial step: it remains to be seen how the effect scales with more complex data distributions and neural architectures. Nonetheless, demonstrating a formal link between learning-induced smoothing and interpolation opens the door to intentionally designing models that maintain useful creativity while mitigating blind memorization. The code for the numerical experiments used to produce figures in the paper has been released by the authors.
Acknowledgements
The authors thank Sreenivas Gollapudi and Ravi Kumar for supporting the project and Mark Simborg and Kimberly Schwede for help preparing the blogpost.
