K8S实用整理(08)-kubelet启动参数修改方法(配置Enabling Unsafe Sysctls)

暂基于kubespary自动部署的1.9.0-coreos版本，kubelet服务相关配置文件：

文件1：/etc/systemd/system/kubelet.service 

/etc/systemd/system/kubelet.service

文件内容为：

[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Wants=docker.socket

[Service]
EnvironmentFile=-/etc/kubernetes/kubelet.env
ExecStartPre=-/bin/mkdir -p /var/lib/kubelet/volume-plugins
ExecStart=/usr/local/bin/kubelet \
                $KUBE_LOGTOSTDERR \
                $KUBE_LOG_LEVEL \
                $KUBELET_API_SERVER \
                $KUBELET_ADDRESS \
                $KUBELET_PORT \
                $KUBELET_HOSTNAME \
                $KUBE_ALLOW_PRIV \
                $KUBELET_ARGS \
                $DOCKER_SOCKET \
                $KUBELET_NETWORK_PLUGIN \
                $KUBELET_VOLUME_PLUGIN \
                $KUBELET_CLOUDPROVIDER
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

　文件2：/etc/kubernetes/kubelet.env

/etc/kubernetes/kubelet.env

文件内容：
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=2"
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=172.28.2.211 --node-ip=172.28.2.211"
# The port for the info server to serve on
# KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node1"






KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests \
--cadvisor-port=0 \
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
--node-status-update-frequency=10s \
--docker-disable-shared-pid=True \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--tls-cert-file=/etc/kubernetes/ssl/node-node1.pem \
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem \
--anonymous-auth=false \
--cgroup-driver=cgroupfs \
--cgroups-per-qos=True \
--fail-swap-on=True \
--enforce-node-allocatable=""  --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true  --feature-gates=Initializers=False,PersistentLocalVolumes=False  "
KUBELET_NETWORK_PLUGIN="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

KUBELET_VOLUME_PLUGIN="--volume-plugin-dir=/var/lib/kubelet/volume-plugins"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
KUBELET_CLOUDPROVIDER=""

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
~

 

修改需求：

Enabling Unsafe Sysctls

With the warning above in mind, the cluster admin can allow certain unsafe sysctls for very special situations like e.g. high-performance or real-time application tuning. Unsafe sysctls are enabled on a node-by-node basis with a flag of the kubelet, e.g.:

$ kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,net.ipv4.route.min_pmtu' ...
修改需求：
kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'
修改方法：修改环境变量文件/etc/kubernetes/kubelet.env，修改为
（添加了--experimental-allowed-unsafe-sysctls='kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'"）
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=2"
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=172.28.2.211 --node-ip=172.28.2.211"
# The port for the info server to serve on
# KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node1"






KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests \
--cadvisor-port=0 \
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
--node-status-update-frequency=10s \
--docker-disable-shared-pid=True \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--tls-cert-file=/etc/kubernetes/ssl/node-node1.pem \
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem \
--anonymous-auth=false \
--cgroup-driver=cgroupfs \
--cgroups-per-qos=True \
--fail-swap-on=True \
--enforce-node-allocatable=""  --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true  --feature-gates=Initializers=False,PersistentLocalVolumes=False  \
--experimental-allowed-unsafe-sysctls='kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'"
KUBELET_NETWORK_PLUGIN="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

KUBELET_VOLUME_PLUGIN="--volume-plugin-dir=/var/lib/kubelet/volume-plugins"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
KUBELET_CLOUDPROVIDER=""

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
~                                                                                



　重启kubelet服务，查看是否修改成功：


systemctl restart kubelet


systemctl status kubelet


ps aux | grep kubelet | grep kernel




root@node1:~# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since 四 2018-03-08 17:20:38 CST; 2min 51s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 14844 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 14851 (kubelet)
    Tasks: 17
   Memory: 50.4M
      CPU: 27.792s
   CGroup: /system.slice/kubelet.service
           └─14851 /usr/local/bin/kubelet --logtostderr=true --v=2 --address=172.28.2.211 --node-ip=172.28.2.211 -

3月 08 17:23:28 node1 kubelet[14851]: E0308 17:23:28.518287   14851 pod_workers.go:186] Error syncing pod 082ad73d
3月 08 17:23:28 node1 kubelet[14851]: W0308 17:23:28.518432   14851 container.go:393] Failed to create summary rea
3月 08 17:23:28 node1 kubelet[14851]: I0308 17:23:28.887368   14851 kubelet.go:1881] SyncLoop (PLEG): "centos1_def
3月 08 17:23:28 node1 kubelet[14851]: W0308 17:23:28.887505   14851 pod_container_deletor.go:77] Container "84c856
3月 08 17:23:29 node1 kubelet[14851]: I0308 17:23:29.188203   14851 kuberuntime_manager.go:403] No ready sandbox f
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624710   14851 remote_runtime.go:92] RunPodSandbox from runti
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624792   14851 kuberuntime_sandbox.go:54] CreatePodSandbox fo
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624814   14851 kuberuntime_manager.go:647] createPodSandbox f
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624923   14851 pod_workers.go:186] Error syncing pod 082ad73d
3月 08 17:23:29 node1 kubelet[14851]: W0308 17:23:29.625543   14851 container.go:393] Failed to create summary rea
root@node1:~# ps aux | grep kubelet | grep kernel
root     14851 12.5  0.7 696144 121368 ?       Ssl  17:20   0:24 /usr/local/bin/kubelet --logtostderr=true --v=2 --address=172.28.2.211 
--node-ip=172.28.2.211 --hostname-override=node1 --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests 
--cadvisor-port=0 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --node-status-update-frequency=10s 
--docker-disable-shared-pid=True --client-ca-file=/etc/kubernetes/ssl/ca.pem --tls-cert-file=/etc/kubernetes/ssl/node-node1.pem 
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem --anonymous-auth=false --cgroup-driver=cgroupfs --cgroups-per-qos=True 
--fail-swap-on=True --enforce-node-allocatable= --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf 
--kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true 
--feature-gates=Initializers=False,PersistentLocalVolumes=False 
--experimental-allowed-unsafe-sysctls=kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu 
--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin 
--volume-plugin-dir=/var/lib/kubelet/volume-plugins





如上修改成功。


创建POD，参考页面：K8S实用整理(10)-Kubernetes配置POD内核参数sysctl  http://www.cnblogs.com/DaweiJ/articles/8528687.html




Kubelet Configurations We Should Care About


下面是我梳理的，我认为必须关注的flag。


 





flagvalue





--address
0.0.0.0




--allow-privileged
false




--cadvisor-port int32
4194




--cgroup-driver string
cgroupfs




--cluster-dns stringSlice
10.0.0.10 //todo




--cluster-domain string
caas.vivo.com




--cni-bin-dir string
/opt/cni/bin




--cni-conf-dir string
/etc/cni/net.d




--docker-endpoint string
unix:///var/run/docker.sock




--eviction-hard string
memory.available<4Gi,<br/> nodefs.available<20Gi,<br/> imagefs.available<5Gi




--eviction-max-pod-grace-period int32
30




--eviction-minimum-reclaim string
memory.available=500Mi,<br/> nodefs.available=2Gi,,<br/> imagefs.available=2Gi




--eviction-pressure-transition-periodduration
5m0s




--eviction-soft string
memory.available<8Gi,<br/> nodefs.available<100Gi,<br/> imagefs.available<20Gi




--eviction-soft-grace-period string
memory.available=30s,<br/> nodefs.available=2m,<br/> imagefs.available=2m




--experimental-fail-swap-on
+




--experimental-kernel-memcg-notification
+




--feature-gates string
AllAlpha=false




--file-check-frequency duration
20s




--hairpin-mode string
promiscuous-bridge




--healthz-port int32
10248




--image-gc-high-threshold int32
60




--image-gc-low-threshold int32
40




--image-pull-progress-deadline duration
2m0s




--kube-api-qps int32
5




--kube-reserved mapStringString
cpu=200m,memory=16G




--kubeconfig string
/var/lib/kubelet/kubeconfig




--max-pods int32
50




--minimum-image-ttl-duration duration
1h




--network-plugin string
cni




--pod-infra-container-image string
vivo.registry.com/google_containers/pause-amd64:3.0




--pod-manifest-path string
/var/lib/kubelet/pod_manifest




--port int32
10250




--protect-kernel-defaults
+




--read-only-port int32
10255




--require-kubeconfig
+




--root-dir string
/var/lib/kubelet




--runtime-request-timeout duration
2m0s




--serialize-image-pulls
false




--sync-frequency duration
1m0s




--system-reserved mapStringString
cpu=100m,memory=32G




--volume-plugin-dir string
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/




--volume-stats-agg-period duration
1m0s






 


 


下面是我最终梳理的，认为需要真正显示设置的flag，如下：




/usr/bin/kubelet —address=0.0.0.0  --port=10250  --allow-privileged=false --cluster-dns=10.0.0.1  --cluster-domain=caas.vivo.com --max-pods=50  --network-plugin=cni  --require-kubeconfig  --pod-manifest-path=/etc/kubelet.d/ --pod-infra-container-image=vivo.registry.com/google_containers/pause-amd64:3.0  --eviction-hard=memory.available<4Gi,nodefs.available<20Gi,imagefs.available<5Gi  --eviction-max-pod-grace-period=30  --eviction-minimum-reclaim=memory.available=500Mi,nodefs.available=2Gi,imagefs.available=2Gi  --eviction-pressure-transition-period=5m0s  --eviction-soft=memory.available<8Gi,nodefs.available<100Gi,imagefs.available<20Gi  --eviction-soft-grace-period=memory.available=30s,nodefs.available=2m,imagefs.available=2m  --experimental-kernel-memcg-notification  --experimental-fail-swap-on  --system-reserved=cpu=100m,memory=8G  --kube-reserved=cpu=200m,memory=16G --hairpin-mode=promiscuous-bridge  --image-gc-high-threshold=60  --image-gc-low-threshold=40  --serialize-image-pulls=false --protect-kernel-defaults  --feature-gates=AllAlpha=false 





　　











            

            
posted @ 
2018-03-08 17:27 
Cslc-DaweiJ 
阅读(13098) 
评论(0) 
编辑 
收藏 
举报