【推导】神经网络梯度推导

基于网易云课堂龙良曲老师的《深度学习与pytorch》中对神经网络梯度推导过程，从易到难循序渐进；

目前只收录了感知机部分，后续待补充

1. 感知机梯度推导

单一感知机

其中损失函数表示为

$$E=\frac12(O_0^1-t)^2$$

$sigmoid$ 函数求导回顾

$$\begin{aligned} g'(z)=\frac{e^{-z}}{(1+e^{-z})^2}=\frac{e^{-z}+1-1}{(1+e^{-z})^2}=\frac1{1+e^{-z}}-\frac1{(1+e^{-z})^2}=g(z)(1-g(z)) \end{aligned}$$

梯度求导过程

$$\begin{aligned}\frac{\partial E}{\partial w_{j0}}&=\left(O_{0}-t\right) \frac{\partial O_{0}}{\partial w_{j0}} \\ \frac{\partial E}{\partial w_{j0}}&=\left(O_{0}-t\right) \frac{\partial \sigma\left(x_{0}\right)}{\partial w_{j0}} \\ \frac{\partial E}{\partial w_{j0}}&=\left(O_{0}-t\right) \sigma\left(x_{0}\right)\left(1-\sigma\left(x_{0}\right)\right) \frac{\partial x_{0}^{1}}{\partial w_{j0}} \\ \frac{\partial E}{\partial w_{j0}}&=\left(O_{0}-t\right) O_{0}\left(1-O_{0}\right) \frac{\partial x_{0}^{1}}{\partial w_{j0}} \\ \frac{\partial E}{\partial w_{j0}}&=\left(O_{0}-t\right) O_{0}\left(1-O_{0}\right) x_{j}^{0}\end{aligned}$$

多输出感知机

其中损失函数表示为

$$E=\frac12(O_i^1-t_i)^2$$

梯度求导过程

$$\begin{aligned} \frac{\partial E}{\partial w_{j k}}&=\left(O_{k}-t_{k}\right) \frac{\partial O_{k}}{\partial w_{j k}} \\ \frac{\partial E}{\partial w_{j k}}&=\left(O_{\mathrm{k}}-t_{k}\right) \frac{\partial \sigma\left(x_{k}\right)}{\partial w_{j k}} \\ \frac{\partial E}{\partial w_{j k}}&=\left(O_{k}-t_{k}\right) \sigma\left(x_{k}\right)\left(1-\sigma\left(x_{k}\right)\right) \frac{\partial x_{k}^{1}}{\partial w_{j k}} \\ \frac{\partial E}{\partial w_{j k}}&=\left(O_{k}-t_{k}\right) O_{\mathrm{k}}\left(1-O_{k}\right) \frac{\partial x_{k}^{1}}{\partial w_{j k}} \\ \frac{\partial E}{\partial w_{j k}}&=\left(O_{k}-t_{k}\right) O_{\mathrm{k}}\left(1-O_{k}\right) x_{j}^{0} \end{aligned}$$

2. 多层感知机推导

引入链式法则

$$\frac{\partial E}{\partial w_{j k}^{1}}=\frac{\partial E}{\partial O_{k}^{1}} \frac{\partial O_{k}^{1}}{\partial x}=\frac{\partial E}{\partial O_{k}^{2}} \frac{\partial O_{k}^{2}}{\partial O_{k}^{1}} \frac{\partial O_{k}^{1}}{\partial x}$$

多层感知机结构

为了方便运算，设置

$$\delta_k^K=\left(O_{k}-t_{k}\right) O_{\mathrm{k}}\left(1-O_{k}\right)$$

以上数据都是可以直接求出

$$\begin{aligned} \frac{\partial E}{\partial W_{i j}} &=\frac{\partial}{\partial W_{i j}} \frac{1}{2} \sum_{k \in K}\left(O_{k}-t_{k}\right)^{2} \\ \frac{\partial E}{\partial W_{i j}} &=\sum_{k \in K}\left(O_{k}-t_{k}\right) \frac{\partial}{\partial W_{i j}} O_{k} \\ \frac{\partial E}{\partial W_{i j}} &=\sum_{k \in K}\left(O_{k}-t_{k}\right) \frac{\partial}{\partial W_{i j}} \sigma\left(x_{k}\right)\\ \frac{\partial E}{\partial W_{i j}}&=\sum_{k \in K}\left(O_{k}-t_{k}\right) \sigma\left(x_{k}\right)\left(1-\sigma\left(x_{k}\right)\right) \frac{\partial x_{k}}{\partial W_{i j}} \\ \frac{\partial E}{\partial W_{i j}}&=\sum_{k \in K}\left(O_{k}-t_{k}\right) O_{k}\left(1-O_{k}\right) \frac{\partial x_{k}}{\partial O_{j}} \cdot \frac{\partial O_{j}}{\partial W_{i j}} \\ \frac{\partial E}{\partial W_{i j}}&=\sum_{k \in K}\left(O_{k}-t_{k}\right) O_{k}\left(1-O_{k}\right) W_{j k} \frac{\partial O_{j}}{\partial W_{i j}}\\ \frac{\partial E}{\partial W_{i j}}&=O_{j}\left(1-O_{j}\right) \frac{\partial x_{j}}{\partial W_{i j}} \sum_{k \in K}\left(O_{k}-t_{k}\right) O_{k}\left(1-O_{k}\right) W_{j k}\\ \frac{\partial E}{\partial W_{i j}}&=O_{i} O_{j}\left(1-O_{j}\right) \sum_{k \in K} \delta_{k} W_{j k} \end{aligned}$$

同样，将

$$\delta_j^J=O_{j}\left(1-O_{j}\right) \sum_{k \in K} \delta_{k} W_{j k}$$

设置为从J层到以后的信息

总结

For an output layer node $k \in K$

$$\frac{\partial E}{\partial W_{j k}}=O_{j} \delta_{k}$$

where

$$\delta_{k}=O_{k}\left(1-O_{k}\right)\left(O_{k}-t_{k}\right)$$

For a hidden layer node $j \in J$

$$\frac{\partial E}{\partial W_{i j}}=O_{i} \delta_{j}$$

where

$$\delta_{j}=O_{j}\left(1-O_{j}\right) \sum_{k \in K} \delta_{k} W_{j k}$$