1 梯度下降

$$f(x+\Delta x,y+\Delta y)\simeq f(x,y)+\frac{\partial f(x,y)}{\partial x}\Delta x+\frac{\partial f(x,y)}{\partial y}\Delta y$$

$$\Delta z=f(x+\Delta x,y+\Delta y)-f(x,y)\simeq \frac{\partial f(x,y)}{\partial x}\Delta x+\frac{\partial f(x,y)}{\partial y}\Delta y$$

$$\Delta z\simeq \frac{\partial f(x,y)}{\partial x}\Delta x+\frac{\partial f(x,y)}{\partial y}\Delta y$$

$$\Delta z\simeq (\frac{\partial f(x,y)}{\partial x},\frac{\partial f(x,y)}{\partial y})(\Delta x,\Delta y)$$

$$\Delta z\simeq (\frac{\partial z}{\partial x},\frac{\partial z}{\partial y})\cdot (\Delta x,\Delta y)=\nabla z\cdot (\Delta x,\Delta y)$$

如果想要让z的下降速度最快就要保证两个向量方向完全相反，也就是要保证如下公式成立

$$(\Delta x,\Delta y)=-\eta \nabla z$$

2 NN误差反向传播

参数w和b的梯度表示

$$\frac{\partial C}{\partial w_{ji}^l}={\delta }_j^l{a}_i^{l-1},\frac{\partial C}{\partial b_j^l}={\delta }_j^l(l=2,\mathrm{3...})$$

δ的计算方法

输出层的误差反向传播计算方法，此处L代表输出层

$${\delta }_j^L=\frac{\partial C}{\partial a_j^L}{a}^{\prime }(z_j^L)$$

$$C=\frac{1}{2}\{(t_1-a_1^L{)}^2+(t_2-a_2^L{)}^2\}$$

$${\delta }_j^L=\frac{\partial C}{\partial a_j^L}{a}^{\prime }(z_j^L)=(a_j^L-t_j)a^{\prime }(z_j^L)$$

隐藏层的误差反向传播计算方法，层l和下一层l+1的递推关系，m为层l+1的神经单元个数，l为大于等于2的整数

$${\delta }_i^l=({\delta }_1^{l+1}{w}_{1i}^{l+1}+{\delta }_2^{l+1}{w}_{2i}^{l+1}+...+{\delta }_m^{l+1}{w}_{mi}^{l+1})a^{\prime }(z_i^l)$$

输出层的神经单元误差

$${\delta }_j^3=\frac{\partial C}{\partial z_j^3}=\frac{\partial C}{\partial a_j^3}\frac{\partial a_j^3}{\partial z_j^3}=\frac{\partial C}{\partial a_j^3}{a}^{\prime }(z_j^3)$$

隐藏层的神经单元误差

$${\delta }_i^2=({\delta }_1^3{w}_{1i}^3+{\delta }_2^3{w}_{2i}^3)a^{\prime }(z_i^2)(i=1,2,3)$$

3 CNN误差反向传播

输出层的梯度分量

$${\frac{\partial C}{\partial w^{On}}}_{k-ij}={\delta }_n^O{a}_{ij}^{Pk},{\frac{\partial C}{\partial b^O}}_n={\delta }_n^O$$

n为输出层神经单元的编号，k为池化层子层编号，ij为池化子层神经单元行列编号(i,j=1,2)

卷积层的梯度分量

$$\frac{\partial C}{\partial w_{ij}^{Fk}}={\delta }_{11}^{Fk}{x}_{ij}+{\delta }_{12}^{Fk}{x}_{ij+1}+...+{\delta }_{44}^{Fk}{x}_{i+3j+3}$$

k为过滤器的编号，ij为过滤器行列的编号（i,j=1,2,3）

$$\frac{\partial C}{\partial b^{Fk}}={\delta }_{11}^{Fk}+{\delta }_{12}^{Fk}+...+{\delta }_{44}^{Fk}$$

k为过滤器的编号

输出层δ的计算方法

$${\delta }_n^O=\frac{\partial C}{\partial z_n^O}=\frac{\partial C}{\partial a_n^O}\frac{\partial a_n^O}{\partial z_n^O}=\frac{\partial C}{\partial a_n^O}{a}^{\prime }(z_n^O)$$

n为输出层神经单元的编号

$$C=\frac{1}{2}\{(t_1-a_1^O{)}^2+(t_2-a_2^O{)}^2+(t_3-a_3^O{)}^2\}$$

$${\delta }_n^O=(a_n^O-t_n)a^{\prime }(z_n^O)$$

以上为代价函数示例及其导数，带入δ式可得

$$\frac{\partial C}{\partial a_n^O}=a_n^O-t_n（n=1,2,3）$$

求导数得

卷积层δ的计算方法

δ i j F k = { δ 1 O w k − i ′ j ′ O 1 + δ 2 O w k − i ′ j ′ O 2 + δ 3 O w k − i ′ j ′ O 3 } × ( 当 a i j F k 在区块中为最大值时为 1 否则为 0 ) × a ′ ( z i j F k ) \delta ^{Fk}_{ij}=\{\delta ^{O}_{1}w^{O1}_{k-i'j'}+\delta ^{O}_{2}w^{O2}_{k-i'j'}+\delta ^{O}_{3}w^{O3}_{k-i'j'}\}\times(当a^{Fk}_{ij}在区块中为最大值时为1否则为0)\times a'(z^{Fk}_{ij}) δijFk​={δ1O​wk−i′j′O1​+δ2O​wk−i′j′O2​+δ3O​wk−i′j′O3​}×(当aijFk​在区块中为最大值时为1否则为0)×a′(zijFk​)

k为卷积层子层的编号，ij为卷积层神经单元的编号，i’j’是卷积层i行j列神经单元连接池化层神经单元的位置