FisherExpectation_of_Hessian_of_Negative_log_likelihood_1">证明Fisher等于Expectation of Hessian of Negative log likelihood.

符号约定

• $f_{\theta }(\cdot )$: 概率密度

• $p(x|\theta )=p_{\theta }(x)=\prod _i^N{f}_{\theta }(x_i)$: 似然函数

• $s(\theta )={\nabla }_{\theta }{p}_{\theta }(x)$: score function，即似然函数的梯度。

• $I=E_{p_{\theta }(x)}[({\nabla }_{\theta }log{p}_{\theta }(x))({\nabla }_{\theta }log{p}_{\theta }(x){)}^T]$: Fisher矩阵。

• $I_{i,j}(\theta )=E_{p_{\theta }(x)}[(D_{i}log{p}_{\theta }(x))(D_{j}log{p}_{\theta }(x))]$: 为Fisher的第i行第j列元素。其中$D_i=\frac{\partial }{\partial {\theta }_i};D_{i,j}=\frac{\partial }{\partial {\theta }_i\partial {\theta }_j}$。

• $H_{i,j}=D_{i,j}log{P}_{\theta }(x)$: Hessian矩阵的第i行第j列元素。

证明

证明目标：
$$I_{i,j}(\theta )=-E_{p_{\theta }(x)}[H_{i,j}]$$
从$H_{i,j}$入手。
$$\begin{array}{rl}{H}_{i,j}& =D_{i,j}log{P}_{\theta }(x)\\ & =D_i(\frac{D_j{p}_{\theta }(x)}{p_{\theta }(x)})\\ & =\frac{(D_{i,j}{p}_{\theta }(x))\cdot p_{\theta }(x)-D_i{p}_{\theta }(x)D_j{p}_{\theta }(x)}{p_{\theta }^2(x)}\\ & =\frac{D_{i,j}{p}_{\theta }(x)}{p_{\theta }(x)}-\frac{D_i{p}_{\theta }(x)}{p_{\theta }(x)}\frac{D_j{p}_{\theta }(x)}{p_{\theta }(x)}\end{array}$$
故右式：

$$\begin{array}{rl}-E_{p_{\theta }(x)}(H_{i,j})& =-E_{p_{\theta }(x)}[\frac{D_{i,j}{p}_{\theta }(x)}{p_{\theta }(x)}]+E_{p_{\theta }(x)}[(\frac{D_i{p}_{\theta }(x)}{p_{\theta }(x)})\cdot (\frac{D_j{p}_{\theta }(x)}{p_{\theta }(x)})]\end{array}$$

其中:
$$\begin{array}{rlr}{E}_{p_{\theta }(x)}(\frac{D_{i,j}{p}_{\theta }(x)}{p_{\theta }(x)})& =\int \frac{D_{i,j}{p}_{\theta }(x)}{p_{\theta }(x)}\cdot p_{\theta (x)}\cdot dx& \\ & =D_{i,j}\int p_{\theta }(x)\cdot dx& (\text{积分求导换序})\\ & =D_{i,j}1& (\text{对常数求导，为0})\\ & =0& \end{array}$$

且根据复合函数求导可知：
$$\frac{D_i{p}_{\theta }(x)}{p_{\theta }(x)}=D_{i}log{p}_{\theta }(x)$$

故右式为:
$$\begin{array}{rl}& E_{p_{\theta }(x)}[(\frac{D_i{p}_{\theta }(x)}{p_{\theta }(x)})\cdot (\frac{D_j{p}_{\theta }(x)}{p_{\theta }(x)})]=E_{p_{\theta }(x)}[(D_{i}log{p}_{\theta }(x))(D_{j}log{p}_{\theta }(x))]\\ & =I_{i,j}(\theta )\end{array}$$
得证

实际应用中，计算$H$非常复杂，但是计算$I$并将其作为$H$的近似值是比较容易的，一些剪枝方法中就利用了这一点，如NAP [Network Automatic Pruning Start NAP and Take a Nap]（基于OBS，OBD）

参考链接：

https://zhuanlan.zhihu.com/p/546885304?utm_psn=1840735001693523969
https://zhuanlan.zhihu.com/p/546885304?utm_psn=1840431492376969216
https://jaketae.github.io/study/fisher/
https://mark.reid.name/blog/fisher-information-and-log-likelihood.html
https://bobondemon.github.io/2022/01/07/Score-Function-and-Fisher-Information-Matrix/