requires_grad，grad_fn，grad的含义及使用2
对requires_grad，grad_fn，grad的含义及使用进行补充说明，请先看此文章。
本文对grad的含义进行补充说明
grad为：当执行完了backward()之后，通过x.grad查看x的梯度值。比如z=f(x,y)，那么
x
.
g
r
a
d
=
∂
z
∂
x
∣
x
=
a
x.grad= \frac{\partial z}{\partial x}|_{x=a}
x.grad=∂x∂z∣x=a，a为x的初始值。
分析下面代码：
>>> x = torch.ones(2, 2, requires_grad=True) # 2x2全为1的tensor
>>> y = x + 2
>>> z = y * y * 3
>>> out = z.mean()
>>> print(z, out)
tensor([[27., 27.],
[27., 27.]], grad_fn=<MulBackward0>) tensor(27., grad_fn=<MeanBackward0>)
>>> out.backward()
>>> print(x.grad)
tensor([[4.5000, 4.5000],
[4.5000, 4.5000]])
假设
x
=
[
x
1
x
2
x
3
x
4
]
=
[
1
1
1
1
]
x=\begin{gathered} \begin{bmatrix} x_1 & x_2 \\ x_3 & x_4 \end{bmatrix} \end{gathered}= \begin{gathered} \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} \end{gathered}
x=[x1x3x2x4]=[1111]
则
y
=
[
y
1
y
2
y
3
y
4
]
=
[
x
1
+
2
x
2
+
2
x
3
+
2
x
4
+
2
]
=
[
3
3
3
3
]
y=\begin{gathered} \begin{bmatrix} y_1 & y_2 \\ y_3 & y_4 \end{bmatrix} \end{gathered}= \begin{gathered} \begin{bmatrix} x_1+2 & x_2+2 \\ x_3+2 & x_4+2 \end{bmatrix} \end{gathered}= \begin{gathered} \begin{bmatrix} 3 & 3 \\ 3 & 3 \end{bmatrix} \end{gathered}
y=[y1y3y2y4]=[x1+2x3+2x2+2x4+2]=[3333]
z
=
[
z
1
z
2
z
3
z
4
]
=
[
3
y
1
2
3
y
2
2
3
y
3
2
3
y
3
2
]
=
[
27
27
27
27
]
z=\begin{gathered} \begin{bmatrix} z_1 & z_2 \\ z_3 & z_4 \end{bmatrix} \end{gathered}= \begin{gathered} \begin{bmatrix} 3y_1^2 & 3y_2^2 \\ 3y_3^2 & 3y_3^2 \end{bmatrix} \end{gathered}= \begin{gathered} \begin{bmatrix} 27& 27 \\ 27 & 27 \end{bmatrix} \end{gathered}
z=[z1z3z2z4]=[3y123y323y223y32]=[27272727]
o
u
t
=
z
1
+
z
2
+
z
3
+
z
4
4
out = \frac{z_1+z_2+z_3+z_4}{4}
out=4z1+z2+z3+z4
则
x
.
g
r
a
d
=
[
∂
o
u
t
∂
x
1
∣
x
1
=
1
∂
o
u
t
∂
x
2
∣
x
2
=
1
∂
o
u
t
∂
x
3
∣
x
3
=
1
∂
o
u
t
∂
x
4
∣
x
4
=
1
]
x.grad=\begin{gathered} \begin{bmatrix} \frac{\partial out}{\partial x_1}|_{x_1=1} & \frac{\partial out}{\partial x_2}|_{x_2=1} \\ \frac{\partial out}{\partial x_3}|_{x_3=1} & \frac{\partial out}{\partial x_4}|_{x_4=1} \end{bmatrix} \end{gathered}
x.grad=[∂x1∂out∣x1=1∂x3∂out∣x3=1∂x2∂out∣x2=1∂x4∂out∣x4=1]
我们以
∂
o
u
t
∂
x
1
∣
x
1
=
1
\frac{\partial out}{\partial x_1}|_{x_1=1}
∂x1∂out∣x1=1为例进行说明：
∂ o u t ∂ x 1 = ∂ o u t ∂ z 1 ∂ z 1 ∂ y 1 ∂ y 1 ∂ x 1 = 1 4 6 y 1 = 4.5 \frac{\partial out}{\partial x_1}=\frac{\partial out}{\partial z_1}\frac{\partial z_1}{\partial y_1}\frac{\partial y_1}{\partial x_1}=\frac{1}{4}6y_1=4.5 ∂x1∂out=∂z1∂out∂y1∂z1∂x1∂y1=416y1=4.5
从上面我们可以看出来，x的梯度值（x.grad）就是out对x的偏导值。还可以看出，要求out对x的偏导，需要知道out对z的偏导、z对y的偏导、y对x的偏导。我觉得requires_grad设为True的作用就是保存这些偏导信息（也称为梯度信息），比如y的requires_grad设为True，则会保存“y对x的偏导”这个信息。
requires_grad()默认为False，但这不意味着我们需要一个一个地去分别给x、y、z、out设置requires_grad=True。因为pytorch中规定：只要某一个输入需要相关梯度值，则输出也需要保存相关梯度信息，这样就保证了这个输入的梯度回传，即我们只需要设置x的requires_grad=True，就可以保证y、z、out的requires_grad=True。
再看下面一个注意点：
>>> print(y.grad)
E:\softwares\anaconda\envs\pytorch\lib\site-packages\torch\_tensor.py:1013: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten\src\ATen/core/TensorBody.h:417.)
return self._grad
None
无法打印处y.grad，这是因为并不是每个requires_grad()设为True的值都会在backward的时候得到相应的grad。它还必须为leaf。
leaf：requires_grad为True的张量(Tensor),如果他们是由用户创建的,则它们是叶张量(leaf Tensor).这意味着它不是运算的结果,因此gra_fn为None。【参考：链接】
