验证k8s-DeamonSet方式批量部署效率特性

1.拉镜像 

docker pull ubuntu

2.重新制作镜像

更新镜像源

sudo cp -a /etc/apt/sources.list /etc/apt/sources.list.bak

sudo sed -i "s@http://.*archive.ubuntu.com@http://mirrors.tools.huawei.com@g" /etc/apt/sources.list
sudo sed -i "s@http://.*security.ubuntu.com@http://mirrors.tools.huawei.com@g" /etc/apt/sources.list

apt-get update

安装docker.io

apt-get install docker.io

安装启动systemctl的init

apt-get install init

重新提交新的docker镜像

docker commit -m "my ubuntu" 7cab2a374559 ubuntu:v1.01

3.执行特权容器

docker run -itd  --privileged 1a9ccf05fa81  /usr/sbin/init

4.导出需要的镜像

docker save -o kubeedge-pause.tar xxx

docker save -o busybox-131.tar xxx

 

docker load -i /mnt/kubeedge-pause.tar

docker load -i /mnt/busybox-131.tar

 

5.edgecore对应版本

edgecore.yaml修改 hostname

 

sed -i "s/myubuntu/$hostname/g" /etc/kubeedge/config/edgecore.yaml

echo $(pwd)

/mnt/edgecore >> edgecore.log 2>&1

 

6.edgecore报错 

kubeedge/pause   Error response from daemon: error creating aufs mount to /var/lib/docker/aufs/mnt/6762a84cf98a5f0abd413a801e782f4435c94c16d006a2332db599bd142d2bd3-init:

挂载daemon.json

{
"registry-mirrors": [
"https://dockerhub.azk8s.cn",
"https://docker.mirrors.ustc.edu.cn",
"http://hub-mirror.c.163.com" ,
"https://k8s.gcr.io",
"https://github-releases.githubusercontent.com",
"http://www.baidu.com"
],
"insecure-registries": [
"151.0.128.17:7443",
"rnd-dockerhub.huawei.com",
"docker-hub.tools.huawei.com",
"registry.me:5000",
"http://docker.mirrors.ustc.edu.cn",
"https://k8s.gcr.io",
"https://github-releases.githubusercontent.com",
"https://registry-1.docker.io"
],
"exec-opts": [
"native.cgroupdriver=cgroupfs"
],
"storage-driver": "vfs"
}

7.删除悬浮镜像

docker image prune

8.基于docker的ubuntu重做镜像

FROM ubuntu:v1.01

USER root

WORKDIR /mnt

COPY . .

RUN mkdir -p /etc/docker/ && \
    mv ./daemon.json /etc/docker/

RUN mv ./run.sh /usr/local/bin

RUN chmod 774 /usr/local/bin/run.sh

RUN echo "root ALL=(ALL:ALL) NOPASSWD:ALL" >> /etc/sudoers

RUN mkdir -p /etc/kubeedge/config/ &&\
    mv /mnt/edgecore.yaml /etc/kubeedge/config/

#ENTRYPOINT  ["/usr/sbin/init"]

9.内部执行后

root@controller108:~/gwx/edge/myubuntu# docker exec -it 9e9e5c36ebf8 /bin/bash
root@9e9e5c36ebf8:/mnt# docker ps
CONTAINER ID   IMAGE                COMMAND    CREATED      STATUS      PORTS     NAMES
ca4cb9d3c4e7   kubeedge/pause:3.1   "/pause"   2 days ago   Up 2 days             k8s_POD_edgemesh-agent-fgrsr_kubeedge_fd3b0cbe-ca65-4f1e-98d1-36d0eee1182b_0

 

 

raq:

1.Failed to allocate directory watch: Too many open files

vim /etc/sysctl.conf

fs.inotify.max_user_instances=512
fs.inotify.max_user_watches=262144

sysctl -p

sysctl fs.inotify

 

2. ws.go:32] 2023/04/06 01:02:23 http: Accept error: accept tcp [::]:10000: accept4: too many open files; retrying in 1s

修改点：

/etc/security/limits.conf

* soft nofile 65535
* hard nofile 65535

ulimit -n

cat /proc/$PID/limits

nofile 坑cloudcore并不会使用 /etc/security/limits.conf 中配置的 nofile 数值，而是会默认设置为 1024

应该是配置文件问题，目前没有发现nofile 默认为 1024，可能需要看源码哪里设置了1024

> head /etc/security/limits.conf
# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
/etc/security/limits.conf 中的配置对 systemd 启动的进程是无效的。
我们的 cloudcore 是通过 systemd 启动的，在没有设置的情况下，使用的是系统默认的 nofile 限制数据，
Soft Limit 为 1024，Hard Limit 为 4096。

systemd 中的 nofile，要通过 LimitNOFILE 参数设置，
例如编辑 /usr/lib/systemd/system/supervisord.service 将LimitNOFILE设置为 10000。

[Unit]
Description=Process Monitoring and Control Daemon
After=rc-local.service nss-user-lookup.target

[Service]
Type=forking
LimitNOFILE=10000
ExecStart=/usr/bin/supervisord -c /etc/supervisord.conf

[Install]
WantedBy=multi-user.target

 

保存文件后，需要调用 systemctl daemon-reload 刷新服务配置。

# 查看默认的 LimitNOFILE
> systemctl show -p DefaultLimitNOFILE
4096
# 查看某一个 Service 的 LimitNOFILE
> systemctl show supervisord -p LimitNOFILE
10000