rancher server偶发504问题排查

问题

rancher/server 部署到gce莫名其妙地出现504，导致访问异常，看containers logs访问互联网都是失败的

#docker logs
[root@rancher 037c3949be815f9311700d771678a3d7c875627287a62fd3f029b6ee38857d93]# journalctl -u docker -f 
-- Logs begin at Tue 2020-10-13 12:24:20 UTC. --
Nov 13 10:38:05 rancher dockerd[12162]: time="2020-11-13T10:38:05.335827262Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.158533175Z" level=info msg="Loading containers: done."
Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.244847016Z" level=info msg="Docker daemon" commit=4484c46d9d graphdriver(s)=overlay2 version=19.03.13
Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.244930649Z" level=info msg="Daemon has completed initialization"
Nov 13 10:38:06 rancher systemd[1]: Started Docker Application Container Engine.
Nov 13 10:38:06 rancher dockerd[12162]: time="2020-11-13T10:38:06.297384575Z" level=info msg="API listen on /var/run/docker.sock"
Nov 13 10:39:18 rancher dockerd[12162]: time="2020-11-13T10:39:18.271599362Z" level=info msg="Container 632ebda68cfbb44909669307112bb6e5e158ecaeb5677667be18320e948bf290 failed to exit within 10 seconds of signal 15 - using the force"
Nov 13 10:39:18 rancher dockerd[12162]: time="2020-11-13T10:39:18.544146121Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 19 03:54:02 rancher dockerd[12162]: time="2020-11-19T03:54:02.985747420Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 19 03:54:28 rancher dockerd[12162]: time="2020-11-19T03:54:28.748323212Z" level=warning msg="IPv4 forwarding is disabled. Networking will not work."
^C
[root@rancher 037c3949be815f9311700d771678a3d7c875627287a62fd3f029b6ee38857d93]# ^C
[root@rancher 037c3949be815f9311700d771678a3d7c875627287a62fd3f029b6ee38857d93]# cat /proc/sys/net/ipv4/ip_forward
0

原因

google-gce容器os更新后默认关闭ip_forward

解决

cat > /etc/sysctl.d/docker.conf <<-'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sysctl  --system

开启网络转发