NNDL 作业11 LSTM

习题6-4 推导LSTM网络中参数的梯度，并分析其避免梯度消失的效果

先来推个实例：

看式子中间，上半部分并未有连乘项，而下半部分有 $C_t$ 到 $C_{t-1}$ 的连乘项，从这可以看出，LSTM能缓解梯度消失，梯度爆炸只是不易发生。

下面咱们来求一下： $\frac{\partial C_{t}}{\partial C_{t-1}}$

$\frac{\partial C_{t}}{\partial C^{t-1}}=\frac{\partial C_{t}}{\partial F_{t}}\frac{\partial F_{t}}{\partial H_{t-1}}\frac{\partial H_{t-1}}{\partial C_{t-1}}+\frac{\partial C_t}{\partial I_t}\frac{\partial I_t}{\partial H_{t-1}}\frac{\partial H_{t-1}}{\partial C_{t-1}}+\frac{\partial C_{t}}{\partial\tilde{C}_{t}}\:\frac{\partial\tilde{C}_{t}}{\partial H_{t-1}}\frac{\partial H_{t-1}}{\partial C_{t-1}}+\frac{\partial C_{t}}{\partial C_{t-1}}$

展开得：

$\begin{aligned}&\frac{\partial C_t}{\partial C^{t-1}}=C_{t-1}\sigma^{\prime}(\cdot)U_{f}*O_{t-1}tanh^{\prime}(C_{t-1})+\widetilde{C}_t\sigma^{\prime}U_{i}*O_{t-1}tanh^{\prime}(C_{t-1})+\\&I_ttanh^{\prime}(\cdot)U_{c}*O_{t-1}tanh^{\prime}(C_{t-1})+F_t\end{aligned}$

通过调节 $U_{f}$ 、 $U_{i}$ 、 $U_{h}$ 来使 $\frac{\partial C_{t}}{\partial C_{t-1}}$ 接近于1，从而防止梯度消失太快。

此问题我是看的视频学习的，参考链接：【【重温经典】大白话讲解LSTM长短期记忆网络如何缓解梯度消失，手把手公式推导反向传播】https://www.bilibili.com/video/BV1qM4y1M7Nv?p=5&vd_source=d58e25af805a85358e5bc9060257ecdd

习题6-3P 编程实现下图LSTM运行过程

同学提出，未发现 $h_{t-1}$ 输入。可以适当改动例题，增加该输入。

实现LSTM算子，可参考实验教材代码。

1. 使用Numpy实现LSTM算子

import numpy as np#定义激活函数
def sigmoid(x):return 1/(1+np.exp(-x))#权重
input_weight=np.array([1,0,0,0])
inputgate_weight=np.array([0,100,0,-10])
forgetgate_weight=np.array([0,100,0,10])
outputgate_weight=np.array([0,0,100,-10])#输入
input=np.array([[1,0,0,1],[3,1,0,1],[2,0,0,1],[4,1,0,1],[2,0,0,1],[1,0,1,1],[3,-1,0,1],[6,1,0,1],[1,0,1,1]])y=[]   #输出
c_t=0  #内部状态for x in input:g_t=np.matmul(input_weight,x) #候选状态i_t=np.round(sigmoid(np.matmul(inputgate_weight,x)))  #输入门after_inputgate=g_t*i_t       #候选状态经过输入门f_t=np.round(sigmoid(np.matmul(forgetgate_weight,x))) #遗忘门after_forgetgate=f_t*c_t      #内部状态经过遗忘门c_t=np.add(after_inputgate,after_forgetgate) #新的内部状态o_t=np.round(sigmoid(np.matmul(outputgate_weight,x))) #输出门after_outputgate=o_t*c_t     #新的内部状态经过输出门y.append(after_outputgate)   #输出print('输出：',y)

运行结果：

2. 使用nn.LSTMCell实现

import numpy as np
import torch
import torch.nn as nn#实例化
input_size=4
hidden_size=1
cell=nn.LSTMCell(input_size=input_size,hidden_size=hidden_size)
#修改模型参数 weight_ih.shape=(4*hidden_size, input_size),weight_hh.shape=(4*hidden_size, hidden_size),
#weight_ih、weight_hh分别为输入x、隐层h分别与输入门、遗忘门、候选、输出门的权重
cell.weight_ih.data=torch.tensor([[0,100,0,-10],[0,100,0,10],[1,0,0,0],[0,0,100,-10]],dtype=torch.float32)
cell.weight_hh.data=torch.zeros(4,1)
print('cell.weight_ih.shape:',cell.weight_ih.shape)
print('cell.weight_hh.shape',cell.weight_hh.shape)
#初始化h_0,c_0
h_t=torch.zeros(1,1)
c_t=torch.zeros(1,1)
#模型输入input_0.shape=(batch,seq_len,input_size)
input_0=torch.tensor([[[1,0,0,1],[3,1,0,1],[2,0,0,1],[4,1,0,1],[2,0,0,1],[1,0,1,1],[3,-1,0,1],[6,1,0,1],[1,0,1,1]]],dtype=torch.float32)
#交换前两维顺序，方便遍历input.shape=(seq_len,batch,input_size)
input=torch.transpose(input_0,1,0)
print('input.shape:',input.shape)
output=[]
#调用
for x in input:h_t,c_t=cell(x,(h_t,c_t))output.append(np.around(h_t.item(), decimals=3))#保留3位小数
print('output:',output)

运行结果：

3. 使用nn.LSTM实现

import numpy as np
import torch.nn# 设置参数
input_size = 4
hidden_size = 1
# 模型实例化
Lstm = torch.nn.LSTM(input_size=input_size, hidden_size=hidden_size, batch_first=True)
# 权重
Lstm.weight_ih_l0.data = torch.tensor([[0, 100, 0, -10], [0, 100, 0, 10], [1, 0, 0, 0], [0, 0, 100, -10]],dtype=torch.float32)
Lstm.weight_hh_l0.data = torch.zeros(4, 1)
# 初始化内部状态
h_t = torch.zeros(1, 1, 1)
c_t = torch.zeros(1, 1, 1)
# 输入的数据[batch_size,seq_len,input_size]
input = torch.tensor([[[1, 0, 0, 1], [3, 1, 0, 1], [2, 0, 0, 1], [4, 1, 0, 1], [2, 0, 0, 1], [1, 0, 1, 1],[3, -1, 0, 1], [6, 1, 0, 1], [1, 0, 1, 1]]], dtype=torch.float32)
y, (h_t, c_t) = Lstm(input, (h_t, c_t))
y = torch.round(y * 1000) / 1000
print(f"输出：{y}")

输出结果：