扩散模型DDPM数学推导过程完整版（下）

训练目标（损失函数）

我们定义

$$p_{\theta }(x_0)$$

表示模型在参数$\theta$下生成的数据点$x_0$的概率密度；$x_0$表示一张真实的图片。即$p_{\theta }(x_0)$描述了“样本$x_0$”出现的可能性有多大。

我们的训练目标就是
$$\underset{\theta }{\max }{p}_{\theta }(x_0)$$

我们一般会写成负对数的形式：

$$\underset{\theta }{\min }-\log (p_{\theta }(x_0))$$

但是有一个关键的问题，计算$x_0$的概率不能被有效的计算，它需要依赖于$x_0$之前的所有步长。

作为一种解决方案，我们引入这个目标的变分下界：

$$-\log (p_{\theta }(x_0))\le -\log (p_{\theta }(x_0))+D_{KL}(q(x_{1:T}|x_0)||p_{\theta }(x_{1:T}|x_0))$$

下面解释一下KL散度的含义：模型真实的前向扩散过程与模型学习到的反向去噪过程到底有多接近。

若这两个分布一模一样则$D_{KL}=0$，说明模型学习到了“完美的反向扩散的过程”。

我们可以将KL散度进行展开，有：

$$\begin{array}{rl}{D}_{KL}=& \mathbb{E}\left[\log \frac{q(x_{1:T}\mid x_0)}{p_{\theta }(x_{1:T}\mid x_0)}\right]\\ =& \mathbb{E}\left[\log \frac{q(x_{1:T}\mid x_0)}{\frac{p_{\theta }(x_0,x_{1:T})}{p_{\theta }(x_0)}}\right]\\ =& \mathbb{E}\left[\log \frac{q(x_{1:T}\mid x_0)}{\frac{p_{\theta }(x_{0:T})}{p_{\theta }(x_0)}}\right]\\ =& \mathbb{E}\left[\log (\frac{q(x_{1:T}|x_0)}{p_{\theta }(x_{0:T})})+\log (p_{\theta }(x_0))\right]\\ =& \mathbb{E}\left[\log \frac{q(x_{1:T}|x_0)}{p_{\theta }(x_{0:T})}\right]+\log (p_{\theta }(x_0))\end{array}$$

所以$-\log (p_{\theta }(x_0))\le -\log (p_{\theta }(x_0))+D_{KL}(q(x_{1:T}|x_0)||p_{\theta }(x_{1:T}|x_0))$，可以作如下变换：

$$\begin{array}{rl}-\log (p_{\theta }(x_0))& \le -\log (p_{\theta }(x_0))+D_{KL}(q(x_{1:T}|x_0)||p_{\theta }(x_{1:T}|x_0))\\ -\log (p_{\theta }(x_0))& \le -\log (p_{\theta }(x_0))+\mathbb{E}\left[\log \frac{q(x_{1:T}|x_0)}{p_{\theta }(x_{0:T})}\right]+\log (p_{\theta }(x_0))\\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[\log \frac{q(x_{1:T}|x_0)}{p_{\theta }(x_{0:T})}\right]\end{array}$$

• 向前过程：$x_0\to x_1\to \cdots \to x_T$：
  $$q(x_{1:T}\mid x_0)=q(x_1\mid x_0)q(x_2\mid x_1)q(x_3\mid x_2)\cdots q(x_T\mid x_{T-1})=\prod _{t=1}^{T}q(x_t\mid x_{t-1})$$

• 反向过程：$x_T\to x_{T-1}\to \cdots \to x_0$：
  $$p_{\theta }(x_{0:T})=p(x_T)p_{\theta }(x_{T-1}\mid x_T)p_{\theta }(x_{T-2}\mid x_{T-1})\cdots p_{\theta }(x_0\mid x_1)=p(x_T)\prod _{t=1}^T{p}_{\theta }(x_{t-1}\mid x_t)$$

至此可以进一步化简成：

$$\begin{array}{rl}-\log (p_{\theta }(x_0))& \le \mathbb{E}\left[\log \frac{q(x_{1:T}|x_0)}{p_{\theta }(x_{0:T})}\right]\\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\log (\frac{{\prod }_{t=1}^{T}q(x_t|x_{t-1})}{{\prod }_{t=1}^T{p}_{\theta }(x_{t-1}|x_t)})\right]\\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\sum _{t=1}^T\log (\frac{q(x_t|x_{t-1})}{p_{\theta }(x_{t-1}|x_t)})\right]\\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\sum _{t=2}^T\log (\frac{q(x_t|x_{t-1})}{p_{\theta }(x_{t-1}|x_t)})+\log (\frac{q(x_1|x_0)}{p_{\theta }(x_0|x_1)})\right]\end{array}$$

因为模型训练要拟合的是真实的“反向去噪分布”，所以我们要利用贝叶斯公式将$q(x_t\mid x_{t-1})$和$q(x_1\mid x_0)$转换成$q(x_{t-1}\mid x_t)$和$q(x_0\mid x_1)$的形式，且我们希望$q(x_t\mid x_{t-1})$引入我们的已知条件$x_0$，使其能够更有效的学习。

$$\begin{array}{rl}q(x_t|x_{t-1})& =\frac{q(x_{t-1}|x_t)q(x_t)}{q(x_{t-1})}\\ & =\frac{q(x_{t-1}\mid x_t,x_0)q(x_t\mid x_0)}{q(x_{t-1}\mid x_0)}\end{array}$$

所以我们可以进一步化简为：

$$\begin{array}{rl}& \\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\sum _{t=2}^T\log (\frac{q(x_t|x_{t-1})}{p_{\theta }(x_{t-1}|x_t)})+\log (\frac{q(x_1|x_0)}{p_{\theta }(x_0|x_1)})\right]\\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\sum _{t=2}^T\log (\frac{q(x_{t-1}|x_t,x_0)q(x_t|x_0)}{p_{\theta }(x_{t-1}|x_t)q(x_{t-1}|x_0)})+\log (\frac{q(x_1|x_0)}{p_{\theta }(x_0|x_1)})\right]\\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\sum _{t=2}^T\log (\frac{q(x_{t-1}|x_t,x_0)}{p_{\theta }(x_{t-1}|x_t)})+\sum _{t=2}^T\log (\frac{q(x_t|x_0)}{q(x_{t-1}|x_0)})+\log (\frac{q(x_1|x_0)}{p_{\theta }(x_0|x_1)})\right]\end{array}$$

现在我们拿出${\sum }_{t=2}^{T}log(\frac{q(x_t|x_0)}{q(x_{t-1}|x_0)})$这一项进行化简操作：

$$\begin{array}{rl}& \sum _{t=2}^T\log (\frac{q(x_t|x_0)}{q(x_{t-1}|x_0)})\\ & =\log (\prod _{t=2}^T\frac{q(x_t|x_0)}{q(x_{t-1}|x_0)})\\ & =\log (\frac{q(x_2|x_0)q(x_3|x_0)q(x_4|x_0)...q(x_T|x_0)}{q(x_1|x_0)q(x_2|x_0)q(x_3|x_0)...q(x_{T-1}|x_0)})\\ & =\log (\frac{q(x_T|x_0)}{q(x_1|x_0)})\end{array}$$

代换到原式中有：

$$\begin{array}{rl}& \\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\sum _{t=2}^T\log (\frac{q(x_{t-1}|x_t,x_0)}{p_{\theta }(x_{t-1}|x_t)})+\log (\frac{q(x_T|x_0)}{q(x_1|x_0)})+\log (\frac{q(x_1|x_0)}{p_{\theta }(x_0|x_1)})\right]\end{array}$$

然后再看最后两项：

$$\begin{array}{rl}& \log (\frac{q(x_T|x_0)}{q(x_1|x_0)})+\log (\frac{q(x_1|x_0)}{p_{\theta }(x_0|x_1)})\\ & =\log (q(x_T|x_0))-\log (q(x_1|x_0))+\log (q(x_1|x_0))-\log (p_{\theta }(x_0|x_1))\\ & =\log (q(x_T|x_0))-\log (p_{\theta }(x_0|x_1))\end{array}$$

代回到原式中有：

$$\begin{array}{rl}-\log (p_{\theta }(x_0))& \le \mathbb{E}\left[-\log (p(x_T))+\sum _{t=2}^T\log (\frac{q(x_{t-1}|x_t,x_0)}{p_{\theta }(x_{t-1}|x_t)})+\log (q(x_T|x_0))-\log (p_{\theta }(x_0|x_1))\right]\\ -\log (p_{\theta }(x_0))& \le \mathbb{E}\left[\log (\frac{q(x_T|x_0)}{p(x_T)})+\sum _{t=2}^T\log (\frac{q(x_{t-1}|x_t,x_0)}{p_{\theta }(x_{t-1}|x_t)})-\log (p_{\theta }(x_0|x_1))\right]\\ -\log (p_{\theta }(x_0))& \le D_{KL}(q(x_T|x_0)|||p(x_T))+\sum _{t=2}^T{D}_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))-\log (p_{\theta }(x_0|x_1)\end{array}$$

最终我们可以写成这样的形式：

$$L_{VLB}=D_{KL}(q(x_T|x_0)|||p(x_T))+\sum _{t=2}^T{D}_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))-\log (p_{\theta }(x_0|x_1)$$

其中第一项$D_{KL}(q(x_T|x_0)|||p(x_T))$与可学习参数$\theta$无关，可以不作为优化的目标。

优化目标由两部分组成：

• 最小化$D_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))$：最大化每一个去噪声操作和加噪声逆操作的相似度。
• 最小化$-\log (p_{\theta }(x_0|x_1)$：已知$x_1$时，让最后复原原图$x_0$的概率更高。

---

最小化：$D_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))$

若一维正态分布$P\sim \mathcal{N}({\mu }_1,{\sigma }_1)$，$Q\sim \mathcal{N}({\mu }_2,{\sigma }_2)$则$P,Q$之间的KL散度为：
$$D_{KL}(P||Q)=log\frac{{\sigma }_2}{{\sigma }_1}+\frac{{\sigma }_1^2+({\mu }_1-{\mu }_2{)}^2}{2{\sigma }_2^2}-\frac{1}{2}$$

对于$D_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))$，我们可知$q(x_{t-1}|x_t,x_0)\sim \mathcal{N}(x_{t-1};\tilde{\mu }\left(x_t,x_0\right),{\tilde{\beta }}_t\mathbf{I})$，$p_{\theta }\left(x_{t-1}\mid x_t\right)\sim \mathcal{N}(x_{t-1};{\mu }_{\theta }\left(x_t,t\right),{\Sigma }_{\theta }\left(x_t,t\right))$

其中${\Sigma }_{\theta }(x_t,t)={\tilde{\beta }}_t$。

回顾：因为${\tilde{\beta }}_t$是固定噪声日程，在DDPM的原始论文中是线性递增的。

$${\Sigma }_{\theta }(x_t,t)={\tilde{\beta }}_t\mathbf{I}=\frac{1-{\bar{\alpha }}_{t-1}}{1-{\bar{\alpha }}_t}\cdot {\beta }_t\mathbf{I}$$

所以

$$D_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))=log\frac{{\tilde{\beta }}_t}{{\tilde{\beta }}_t}+\frac{{\tilde{\beta }}_t^2+({\mu }_{\theta }(x_t,t)-{\tilde{\mu }}_t(x_t,t){)}^2}{2{\tilde{\beta }}_t^2}-\frac{1}{2}=\frac{1}{2{\tilde{\beta }}_t^2}||{\mu }_{\theta }(x_t,t)-{\tilde{\mu }}_t(x_t,t)|{|}^2$$

根据扩散模型DDPM数学推导过程完整版（上）有:

• 采样阶段的理论反向均值为：

$$\tilde{\mu }\left(x_t,x_0\right)=\frac{1}{\sqrt{{\alpha }_t}}\left(x_t-\frac{1-{\alpha }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_t\right)$$

• 采样阶段的模型反向均值为：
  $${\mu }_{\theta }(x_t,t)=\frac{1}{\sqrt{{\alpha }_t}}(x_t-\frac{1-{\alpha }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{\theta }(x_t,t))$$

故

$$D_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))=\frac{1}{2{\tilde{\beta }}_t^2}||{\mu }_{\theta }(x_t,t)-{\tilde{\mu }}_t(x_t,t)|{|}^2=\frac{(1-{\alpha }_t{)}^2}{2{\alpha }_t(1-{\bar{\alpha }}_t){\tilde{\beta }}_t^2}||{ϵ}_t-{ϵ}_{\theta }(x_t,t)|{|}^2$$

在DDPM论文指出，若将前面的系数全部丢掉，模型的效果更好，最终第一个优化目标$D_{KL}(q(x_{t-1}|x_t,x_0)||p_{\theta }(x_{t-1}|x_t))$可变为$||{ϵ}_t-{ϵ}_{\theta }(x_t,t)|{|}^2$。

---

最小化：$-\log (p_{\theta }(x_0|x_1)$

对于$-\log (p_{\theta }(x_0|x_1)$，我们已知$p_{\theta }(x_0|x_1)\sim \mathcal{N}(x_0;{\mu }_{\theta }(x_1,1),{\tilde{\beta }}_1\mathbf{I})$，这就意味着：

$$p_{\theta }(x_0\mid x_1)=\frac{1}{(2\pi {\tilde{\beta }}_1^2{)}^{1/2}}\exp \left[-\frac{\Vert x_0-{\mu }_{\theta }(x_1,1){\Vert }^2}{2{\tilde{\beta }}_1^2}\right]$$

故

$$-\log (p_{\theta }(x_0|x_1)=-log\frac{1}{\sqrt{2\pi }{\tilde{\beta }}_1^2}+\frac{||x_0-{\mu }_{\theta }(x_1,1)|{|}^2}{2{\tilde{\beta }}_1^2}$$

可以观察到第一项$-log\frac{1}{\sqrt{2\pi }{\tilde{\beta }}_1^2}$与$\theta$无关为常数，第二项$\frac{||x_0-{\mu }_{\theta }(x_1,1)|{|}^2}{2{\tilde{\beta }}_1^2}$与$\theta$有关。所以我们只需要优化第二项即可（${\alpha }_1={\bar{\alpha }}_1=1-{\beta }_1$）：

$$\begin{array}{rl}\frac{(x_0-{\mu }_{\theta }(x_1,1){)}^2}{2{\tilde{\beta }}_1^2}& =\frac{1}{2{\tilde{\beta }}_1^2}||x_0-\frac{1}{\sqrt{{\alpha }_1}}(x_1-\frac{1-{\alpha }_1}{\sqrt{1-{\bar{\alpha }}_1}}{ϵ}_{\theta }(x_1,1))|{|}^2\\ & =\frac{1}{2{\tilde{\beta }}_1^2}||x_0-\frac{1}{\sqrt{{\alpha }_1}}(\sqrt{{\bar{\alpha }}_1}{x}_0+\sqrt{1-{\bar{\alpha }}_1}{ϵ}_1-\frac{1-{\alpha }_1}{\sqrt{1-{\bar{\alpha }}_1}}{ϵ}_{\theta }(x_1,1))|{|}^2\\ & =\frac{1}{2{\tilde{\beta }}_1^2{\alpha }_1}||\sqrt{1-{\bar{\alpha }}_1}{ϵ}_1-\frac{1-{\alpha }_1}{\sqrt{1-{\bar{\alpha }}_1}}{ϵ}_{\theta }(x_1,1)|{|}^2\\ & =\frac{1-{\bar{\alpha }}_1}{2{\tilde{\beta }}_1^2{\alpha }_1}||{ϵ}_1-{ϵ}_{\theta }(x_1,1)|{|}^2\end{array}$$

最终可以将第二个优化目标$-\log (p_{\theta }(x_0|x_1)$变为$||{ϵ}_1-{ϵ}_{\theta }(x_1,1)|{|}^2$。

---

综合以上推导，我们得到两个等价的优化目标：

• 最小化$||{ϵ}_t-{ϵ}_{\theta }(x_t,t)|{|}^2$
• 最小化$||{ϵ}_1-{ϵ}_{\theta }(x_1,1)|{|}^2$

可以注意到，目标 (2) 实际上是目标 (1) 在$t=1$时的特例。
因此，我们可以将最终的优化目标统一为对所有时间步 $t$ 的形式：

$$\underset{\theta }{\min }||{ϵ}_t-{ϵ}_{\theta }(x_t,t)|{|}^2$$

由此，扩散模型的最终损失函数可写为：

$$\mathcal{L}(\theta )=||{ϵ}_t-{ϵ}_{\theta }(x_t,t)|{|}^2$$

即网络通过最小化预测噪声与真实噪声之间的均方误差（MSE）来学习近似真实的反向扩散过程。

---

参考文献

[1] 扩散模型(Diffusion Model)详解：直观理解、数学原理、PyTorch 实现
[2] diffusion model 原理讲解 公式推导
[3] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models