requires_grad，grad_fn，grad的含义及使用2

对requires_grad，grad_fn，grad的含义及使用进行补充说明，请先看此文章。
本文对grad的含义进行补充说明
grad为：当执行完了backward()之后，通过x.grad查看x的梯度值。比如z=f(x,y)，那么$x.grad=\frac{\partial z}{\partial x}{|}_{x=a}$，a为x的初始值。

分析下面代码：

>>> x = torch.ones(2, 2, requires_grad=True) # 2x2全为1的tensor
>>> y = x + 2
>>> z = y * y * 3
>>> out = z.mean()
>>> print(z, out)
tensor([[27., 27.],
        [27., 27.]], grad_fn=<MulBackward0>) tensor(27., grad_fn=<MeanBackward0>)
>>> out.backward()
>>> print(x.grad)
tensor([[4.5000, 4.5000],
        [4.5000, 4.5000]])

假设 x = [ x 1 x 2 x 3 x 4 ] = [ 1 1 1 1 ] x=$$\begin{array}{c}\left[\begin{array}{cc}{x}_1& x_2\\ x_3& x_4\end{array}\right]\end{array}$$= $$\begin{array}{c}\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right]\end{array}$$ x=[x1​x3​​x2​x4​​]​=[11​11​]​
则 y = [ y 1 y 2 y 3 y 4 ] = [ x 1 + 2 x 2 + 2 x 3 + 2 x 4 + 2 ] = [ 3 3 3 3 ] y=$$\begin{array}{c}\left[\begin{array}{cc}{y}_1& y_2\\ y_3& y_4\end{array}\right]\end{array}$$= $$\begin{array}{c}\left[\begin{array}{cc}{x}_1+2& x_2+2\\ x_3+2& x_4+2\end{array}\right]\end{array}$$= $$\begin{array}{c}\left[\begin{array}{cc}3& 3\\ 3& 3\end{array}\right]\end{array}$$ y=[y1​y3​​y2​y4​​]​=[x1​+2x3​+2​x2​+2x4​+2​]​=[33​33​]​
z = [ z 1 z 2 z 3 z 4 ] = [ 3 y 1 2 3 y 2 2 3 y 3 2 3 y 3 2 ] = [ 27 27 27 27 ] z=$$\begin{array}{c}\left[\begin{array}{cc}{z}_1& z_2\\ z_3& z_4\end{array}\right]\end{array}$$= $$\begin{array}{c}\left[\begin{array}{cc}3y_1^2& 3y_2^2\\ 3y_3^2& 3y_3^2\end{array}\right]\end{array}$$= $$\begin{array}{c}\left[\begin{array}{cc}27& 27\\ 27& 27\end{array}\right]\end{array}$$ z=[z1​z3​​z2​z4​​]​=[3y12​3y32​​3y22​3y32​​]​=[2727​2727​]​
$$out=\frac{z_1+z_2+z_3+z_4}{4}$$

则 x . g r a d = [ ∂ o u t ∂ x 1 ∣ x 1 = 1 ∂ o u t ∂ x 2 ∣ x 2 = 1 ∂ o u t ∂ x 3 ∣ x 3 = 1 ∂ o u t ∂ x 4 ∣ x 4 = 1 ] x.grad=$$\begin{array}{c}\left[\begin{array}{cc}\frac{\partial out}{\partial x_1}{|}_{x_1=1}& \frac{\partial out}{\partial x_2}{|}_{x_2=1}\\ \frac{\partial out}{\partial x_3}{|}_{x_3=1}& \frac{\partial out}{\partial x_4}{|}_{x_4=1}\end{array}\right]\end{array}$$ x.grad=[∂x1​∂out​∣x1​=1​∂x3​∂out​∣x3​=1​​∂x2​∂out​∣x2​=1​∂x4​∂out​∣x4​=1​​]​
我们以$\frac{\partial out}{\partial x_1}{|}_{x_1=1}$为例进行说明：

$\frac{\partial out}{\partial x_1}=\frac{\partial out}{\partial z_1}\frac{\partial z_1}{\partial y_1}\frac{\partial y_1}{\partial x_1}=\frac{1}{4}6y_1=4.5$

从上面我们可以看出来，x的梯度值（x.grad）就是out对x的偏导值。还可以看出，要求out对x的偏导，需要知道out对z的偏导、z对y的偏导、y对x的偏导。我觉得requires_grad设为True的作用就是保存这些偏导信息（也称为梯度信息），比如y的requires_grad设为True，则会保存“y对x的偏导”这个信息。
requires_grad()默认为False，但这不意味着我们需要一个一个地去分别给x、y、z、out设置requires_grad=True。因为pytorch中规定：只要某一个输入需要相关梯度值，则输出也需要保存相关梯度信息，这样就保证了这个输入的梯度回传，即我们只需要设置x的requires_grad=True，就可以保证y、z、out的requires_grad=True。

再看下面一个注意点：

>>> print(y.grad)
E:\softwares\anaconda\envs\pytorch\lib\site-packages\torch\_tensor.py:1013: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at  aten\src\ATen/core/TensorBody.h:417.)
  return self._grad
None

无法打印处y.grad，这是因为并不是每个requires_grad()设为True的值都会在backward的时候得到相应的grad。它还必须为leaf。
leaf：requires_grad为True的张量(Tensor),如果他们是由用户创建的,则它们是叶张量(leaf Tensor).这意味着它不是运算的结果,因此gra_fn为None。【参考：链接】