backward方法中gradient参数的意义

首先，我们要清楚使用backward()的目的，是为了求出某个张量对于某些标量节点的梯度。

举个例子：

设 $\begin{array}{rcl} x & = & [x_{1}, x_{2}, x_{3}], z & = & {x_{1}}^{2} + {x_{2}}^{2} + {x_{3}}^{2} + 6 \end{array}$

那么 z.backward()表示的就是张量x对于标量z的梯度，即
$\begin{aligned} [\frac{\partial z}{\partial x_{1}}, \frac{\partial z}{\partial x_{2}}, \frac{\partial z}{\partial x_{3}}] \end{aligned}$
代码则是：

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x ** 2 + 2
z = torch.sum(y)
z.backward()
print(x.grad)

这个还是比较好理解的。

但是当如果我们想对一个向量使用backward()的时候，又该如何使用呢

首先，我们要清楚，backward()的作用一直都是求出某个张量对于某些标量节点的梯度。若我们使用的对象是向量，我们可以用一些“方法”，来让我们的向量变为标量

举个例子：

设 $\begin{array}{rcl} x & = & [x_{1}, x_{2}, x_{3}], y & = & [y_{1}, y_{2}, y_{3}] \end{array}$

$\begin{aligned} y_{1} & = x_{1} x_{2} x_{3} \\ y_{2} & = x_{1} + x_{2} + x_{3} \\ y_{3} & = x_{1} + x_{2} x_{3} \end{aligned}$
自然，我们是无法直接对$y$使用backward()的（因为$y$是一个向量而非标量）

在这里，我们用一个函数$a = g(y)$ （我们先不关心函数的具体实现），来把向量变为一个标量，然后，我们就可以对a使用backward(),表示的就是张量x对于标量a的梯度，即
$[\frac{\partial a}{\partial x_{1}}, \frac{\partial a}{\partial x_{2}}, \frac{\partial a}{\partial x_{3}}]$

从数学公式角度去理解：
$\begin{aligned} \frac{\partial a}{\partial x_{1}} & = \frac{\partial a}{\partial y_{1}} \frac{\partial y_{1}}{\partial x_{1}} + \frac{\partial a}{\partial y_{2}} \frac{\partial y_{2}}{\partial x_{1}} + \frac{\partial a}{\partial y_{3}} \frac{\partial y_{3}}{\partial x_{1}} \\ \frac{\partial a}{\partial x_{2}} & = \frac{\partial a}{\partial y_{1}} \frac{\partial y_{1}}{\partial x_{2}} + \frac{\partial a}{\partial y_{2}} \frac{\partial y_{2}}{\partial x_{2}} + \frac{\partial a}{\partial y_{3}} \frac{\partial y_{3}}{\partial x_{2}} \\ \frac{\partial a}{\partial x_{3}} & = \frac{\partial a}{\partial y_{1}} \frac{\partial y_{1}}{\partial x_{3}} + \frac{\partial a}{\partial y_{2}} \frac{\partial y_{2}}{\partial x_{3}} + \frac{\partial a}{\partial y_{3}} \frac{\partial y_{3}}{\partial x_{3}} \end{aligned}$
可以写成一下矩阵形式：

$\begin{aligned} [\frac{\partial a}{\partial x_{1}}, \frac{\partial a}{\partial x_{2}}, \frac{\partial a}{\partial x_{3}}] & = [\frac{\partial a}{\partial y_{1}}, \frac{\partial a}{\partial y_{2}}, \frac{\partial a}{\partial y_{3}}] [\begin{array}{llll} \frac{\partial y_{1}}{\partial x_{1}} & \frac{\partial y_{1}}{\partial x_{2}} & \frac{\partial y_{1}}{\partial x_{3}} \\ \frac{\partial y_{2}}{\partial x_{1}} & \frac{\partial y_{1}}{\partial x_{2}} & \frac{\partial y_{1}}{\partial x_{3}} \\ \frac{\partial y_{3}}{\partial x_{1}} & \frac{\partial y_{1}}{\partial x_{2}} & \frac{\partial y_{1}}{\partial x_{3}} \end{array}] \end{aligned}$
其中， $[\begin{array}{llll} \frac{\partial y_{1}}{\partial x_{1}} & \frac{\partial y_{1}}{\partial x_{2}} & \frac{\partial y_{1}}{\partial x_{3}} \\ \frac{\partial y_{2}}{\partial x_{1}} & \frac{\partial y_{1}}{\partial x_{2}} & \frac{\partial y_{1}}{\partial x_{3}} \\ \frac{\partial y_{3}}{\partial x_{1}} & \frac{\partial y_{1}}{\partial x_{2}} & \frac{\partial y_{1}}{\partial x_{3}} \end{array}]$ 是可以计算得到的，而由于a的函数未确定，所以 $\begin{aligned} [\frac{\partial a}{\partial y_{1}}, \frac{\partial a}{\partial y_{2}}, \frac{\partial a}{\partial y_{3}}] \end{aligned}$ 是无法计算的。

那么，我们该如何去计算这个梯度呢？

答案是，用gradient参数定义出这个 $[\frac{\partial a}{\partial y_{1}}, \frac{\partial a}{\partial y_{2}}, \frac{\partial a}{\partial y_{3}}]$ !

没错！gradient参数的数学意义其实就是这个 $[\frac{\partial a}{\partial y_{1}}, \frac{\partial a}{\partial y_{2}}, \frac{\partial a}{\partial y_{3}}]$ !

让我们回到代码当中

# coding utf-8
import torch

x1 = torch.tensor(1, requires_grad=True, dtype=torch.float)
x2 = torch.tensor(2, requires_grad=True, dtype=torch.float)
x3 = torch.tensor(3, requires_grad=True, dtype=torch.float)
x = torch.tensor([x1, x2, x3])
y = torch.randn(3)
y[0] = x1 * x2 * x3
y[1] = x1 + x2 + x3
y[2] = x1 + x2 * x3
y.backward(torch.tensor([0.1, 0.2, 0.3], dtype=torch.float))
print(x1.grad)
print(x2.grad)
print(x3.grad)

'''
运行结果：
tensor(1.1000)
tensor(1.4000)
tensor(1.)
'''

在这代码当中backward()里的参数torch.tensor([0.1, 0.2, 0.3], dtype=torch.float), 其实就是定义了 $[\frac{\partial a}{\partial y_{1}}, \frac{\partial a}{\partial y_{2}}, \frac{\partial a}{\partial y_{3}}]$ 为 $[0.1,0.2,0.3]$，那么上面的过程即为：

$\begin{aligned} [\frac{\partial a}{\partial x_{1}}, \frac{\partial a}{\partial x_{2}}, \frac{\partial a}{\partial x_{3}}] & = [0.1 ， 0.2 ， 0.3] [\begin{array}{llll} x_{2} x_{3} & x_{1} x_{3} & x_{1} x_{2} \\ 1 & 1 & 1 \\ 1 & x_{3} & x_{2} \end{array}] & = [0.1 ， 0.2 ， 0.3] [\begin{array}{llll} 6 & 3 & 2 \\ 1 & 1 & 1 \\ 1 & 3 & 2 \end{array}] & = [1.1, 1.4, 1.0] \end{aligned}$
这么一看是不是恍然大悟了

为了验证我们的合理性，我们这里可以试着给出a的实际定义，让 $\begin{aligned} [\frac{\partial a}{\partial y_{1}}, \frac{\partial a}{\partial y_{2}}, \frac{\partial a}{\partial y_{3}}] \end{aligned}$ 变得可以计算。

设 $\begin{aligned} a & = 0.1 y_{1} + 0.2 y_{2} + 0.3 y_{3} \end{aligned}$

这样刚好对应于我们上面代码中定义的gradient，而且a是一个标量，可以直接使用backward()了

让我们再重写代码，对a使用backward(),观察是否能得到同样的结果

# coding utf-8
import torch

x1 = torch.tensor(1, requires_grad=True, dtype=torch.float)
x2 = torch.tensor(2, requires_grad=True, dtype=torch.float)
x3 = torch.tensor(3, requires_grad=True, dtype=torch.float)
y = torch.randn(3)
y[0] = x1 * x2 * x3
y[1] = x1 + x2 + x3
y[2] = x1 + x2 * x3
A = 0.1 * y[0] + 0.2 * y[1] + 0.3 * y[2]
# x = torch.tensor([x1, x2, x3])
# y.backward(torch.tensor([0.1, 0.2, 0.3], dtype=torch.float))
A.backward()
print(x1.grad)
print(x2.grad)
print(x3.grad)

'''
运行结果：
tensor(1.1000)
tensor(1.4000)
tensor(1.)
'''

运行结果是一样的，证明我们对gradient的解释是合理的。

参考：https://www.cnblogs.com/zhouyang209117/p/11023160.html

posted @ 2023-04-14 00:02 Davy-Chen 阅读(859) 评论(4) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

Davy-Chen

backward方法中gradient参数的意义

公告