21. nerdctl buildkitd containerd构建容器镜像
21.1 概述
容器技术除了docker之外,还有coreOS的rkt、google的gvisor、以及docker开源的containerd、redhat的podman、阿里的pouch等,为了保证容器生态的标准性和健康可持续发展、包括Linux基金会、Docker、微软、红帽、谷歌和IBM等公司在2015年6月共同成立了一个叫open cintainer(OCI)的组织,其目的就是指定开放的标准的容器规范,目前OCI一共发布了两个规范,分别是runtime spec和image format spec,有了这两个规范,不同的容器公司开发的容器只要兼容这两个规范,就可以保证容器的可移植性和相互可操作性
https://containerd.io
https://gvisor.dev
https://podman.io
https://github.com/alibaba/pouch
https://github.com/rkt/rkt
https://github.com/moby/buildkit
21.2 buildkitd组成部分
buildkitd(服务端)目前支持runc和containerd作为镜像构建环境,默认是runc,可以更换为containerd
buildctl(客户端)负责解析Dockerfile文件,并向服务端buildkitd发出构建请求
21.2 buildkitd部署
下载地址:Release v0.10.6 · moby/buildkit · GitHub
root@k8s-master1:/usr/local/src
root@k8s-master1:/usr/local/src
root@k8s-master1:/usr/local/bin/bin
root@k8s-master1:/usr/local/bin
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
[Socket]
ListenStream=%t/buildkit/buildkitd.sock
[Install]
WantedBy=sockets.target
root@k8s-master1:/usr/local/bin
[Unit]
Description=BuildKit
Requires=buildkit.socket
After=buildkit.socketDocumentation=https://github.com/moby/buildkit
[Service]
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true
[Install]
WantedBy=multi-user.target
systemctl enable buildkitd.service
systemctl start buildkitd.service
如果使用命令直接登录会报错
root@k8s-master1:~
Enter Username: quyi
Enter Password:
ERRO[0017] failed to call tryLoginWithRegHost error="failed to call rh.Client.Do: Get \"https://harbor.nbrhce.com/v2/\": x509: certificate relies on legacy Common Name field, use SANs instead" i=0
FATA[0017] failed to call rh.Client.Do: Get "https://harbor.nbrhce.com/v2/" : x509: certificate relies on legacy Common Name field, use SANs instead
root@k8s-master1:~
Enter Username: quyi
Enter Password:
WARN[0007] skipping verifying HTTPS certs for "harbor.nbrhce.com"
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/
Login Succeeded
vim /etc/profile
source <(nerdctl completion bash)
source /etc/profile
nerdctl login --insecure-registry harbor.nbrhce.com
nerdctl pull centos:7.9.2009
nerdctl tag centos:7.9.2009 harbor.nbrhce.com/baseimages/centos:7.9.2009
nerdctl push --insecure-registry centos:7.9.2009 harbor.nbrhce.com/baseimages/centos:7.9.2009
root@k8s-master1:~
root@deploy-harbor:/apps/certs
/apps/certs
root@deploy-harbor:/apps/certs
root@deploy-harbor:/apps/certs
ca.crt 100% 2061 1.6MB/s 00:00
harbor.nbrhce.com.cert 100% 2155 1.4MB/s 00:00
harbor.nbrhce.com.key 100% 3247 4.5MB/s 00:00
root@k8s-master1:/etc/containerd/certs.d/harbor.nbrhce.com
Enter Username: quyi
Enter Password:
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/
Login Succeeded
root@k8s-master1:~
root@k8s-master1:~
root@k8s-master1:~
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:48838bdd8842b5b6bf36ef486c5b670b267ed6369e2eccbf00d520824f753ee2)
index-sha256:48838bdd8842b5b6bf36ef486c5b670b267ed6369e2eccbf00d520824f753ee2: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:dead07b4d8ed7e29e98de0f4504d87e8880d4347859d839686a31da35a3b532f: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:eeb6ee3f44bd0b5103bb561b4c16bcb82328cfe5809ab675bb17ab3a16c517c9: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.8 s total: 3.5 Ki (2.0 KiB/s)
FROM ubuntu:22.04
MAINTAINER "nginx"
RUN apt update && apt install -y iproute2 ntpdate tcpdump telnet traceroute nfs-kernel-server nfs-common lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute gcc openssh-server lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute iotop unzip zip make
ADD nginx-1.22.0.tar.gz /usr/local/src/
RUN cd /usr/local/src/nginx-1.22.0 && ./configure --prefix=/apps/nginx && make && make install && ln -sv /apps/nginx/sbin/nginx /usr/bin
RUN groupadd -g 2088 nginx && useradd -g nginx -s /usr/sbin/nologin -u 2088 nginx && chown -R nginx.nginx /apps/nginx
ADD nginx.conf /apps/nginx/conf/
ADD frontend.tar.gz /apps/nginx/html/
EXPOSE 80 443
CMD ["nginx" ,"-g" ,"daemon off;" ]
效果
自行安装
root@k8s-master1:~
root@k8s-master1:~
root@k8s-master1:~
namespace = "k8s.io"
root@k8s-master1:/usr/local/src
root@k8s-master1:/usr/local/src
root@k8s-master1:/usr/local/src
21.3 业务容器构建规划
业务容器化优势
1.提高资源利用率,节约部署IT成本
2.提高部署效率,基于kubernetes实现微服务的快速部署与交付、容器的批量调度与秒级启动
3.实现横向扩容、灰度部署、回滚、链路追踪、服务治理等
4.可根据业务负载进行自动弹性伸缩
5.容器将环境和代码打包在镜像内,保证了测试与生产运行环境的一致性
6.紧跟云原生社区技术发展的步伐,不给公司遗留技术债,为后期技术升级务实基础
7.为个人储备前沿技术,提供个人level
21.4 实现nginx代理habor
root@deploy-harbor:/apps/harbor
Stopping harbor-jobservice ... done
Stopping harbor-core ... done
Stopping registry ... done
Stopping redis ... done
Stopping harbor-db ... done
Stopping harbor-portal ... done
Stopping registryctl ... done
Stopping harbor-log ... done
root@deploy-harbor:/apps/harbor
root@deploy-harbor:/apps/harbor
root@deploy-harbor:/apps/harbor
root@deploy-harbor:/apps/harbor
root@k8s-master1:~
root@k8s-master1:~
root@k8s-master1:~
root@k8s-master1:~
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module
root@k8s-master1:~
root@k8s-master1:~
root@k8s-master1:~
harbor.nbrhce.com.crt harbor.nbrhce.com.key
root@k8s-master1:~
client_max_body_size 2048m;
server {
listen 80;
listen 443 ssl;
server_name harbor.nbrhce.com;
ssl_certificate /apps/nginx/certs/harbor.nbrhce.com.crt;
ssl_certificate_key /apps/nginx/certs/harbor.nbrhce.com.key;
ssl_session_cache shared:sslcache:20m;
ssl_session_timeout 10m;
location / {
proxy_pass http://192.168.1.75;
}
}
root@k8s-master1:~
192.168.1.70 k8s-master1
192.168.1.71 k8s-etcd1
192.168.1.72 k8s-etcd2
192.168.1.73 k8s-etcd3
192.168.1.74 k8s-node1
192.168.1.75 k8s-deploy-harbor harbor.nbrhce.com
192.168.1.75 easzlab.io.local
root@k8s-master1:~
[registry."harbor.nbrhce.com" ]
http = true
insecure = true
root@k8s-master1:~
namespace = "k8s.io"
debug = false
debug_full = false
insecure_registry = true
root@k8s-master1:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212
问题记录 没有识别到自签证书 但是我重启buildkitd服务之后就好使了 不是我证书的原因
21.5 案例nginx toncat前后端分离加业务
nerdctl pull centos:7.9.2009
nerdctl tag centos:7.9.2009 harbor.nbrhce.com/baseimages/centos:7.9.2009
root@k8s-master1:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212
FROM harbor.nbrhce.com/baseimages/centos:7.9.2009
MAINTAINER auth= this is tomcat image
ADD jdk-8u212-linux-x64.tar.gz /usr/local/src/
RUN ln -sv /usr/local/src/jdk1.8.0_212 /usr/local/jdk
ADD profile /etc/profile
ENV JAVA_HOME /usr/local/jdk
ENV JRE_HOME $JAVA_HOME /jre
ENV CLASSPATH $JAVA_HOME /lib/:$JRE_HOME /lib/
ENV PATH $PATH :$JAVA_HOME /bin
root@k8s-master1:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43
FROM harbor.nbrhce.com/pub-image/jdk-base:v8.212
MAINTAINER auth= this is tomcat image
RUN mkdir /apps /data/tomcat/webapps /data/tomcat/logs -pv
ADD apache-tomcat-8.5.43.tar.gz /apps
RUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-8.5.43 /apps/tomcat && chown -R tomcat.tomcat /apps /data -R
root@k8s-master1:~/k8s-data/dockerfile/web/magedu/tomcat-app1
FROM harbor.nbrhce.com/pub-image/tomcat-base:v8.5.43
ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD server.xml /apps/tomcat/conf/server.xml
ADD app1.tar.gz /data/tomcat/webapps/myapp/
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
RUN useradd nginx && chown -R nginx.nginx /data/ && chown -R nginx.nginx /apps/
EXPOSE 8080 8443
CMD ["/apps/tomcat/bin/run_tomcat.sh" ]
root@k8s-master1:~/k8s-data/dockerfile/web/magedu/tomcat-app1
TAG=$1
nerdctl build -t harbor.nbrhce.com/demo/tomcat-app1:${TAG} .
nerdctl push harbor.nbrhce.com/demo/tomcat-app1:${TAG}
root@k8s-master1:~/k8s-data/dockerfile/web/magedu/tomcat-app1
Tomcat started.
root@k8s-node1:~/k8s-data/yaml/magedu/tomcat-app1
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: magedu-tomcat-app1-deployment-label
name: magedu-tomcat-app1-deployment
namespace: magedu
spec:
replicas: 1
selector:
matchLabels:
app: magedu-tomcat-app1-selector
template:
metadata:
labels:
app: magedu-tomcat-app1-selector
spec:
containers:
- name: magedu-tomcat-app1-container
image: harbor.nbrhce.com/demo/tomcat-app1:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
protocol: TCP
name: http
env :
- name: "password"
value: "123456"
- name: "age"
value: "18"
volumeMounts:
- name: magedu-images
mountPath: /usr/local/nginx/html/webapp/images
readOnly: false
- name: magedu-static
mountPath: /usr/local/nginx/html/webapp/static
readOnly: false
volumes:
- name: magedu-images
nfs:
server: 192.168.1.75
path: /data/k8sdata/images
- name: magedu-static
nfs:
server: 192.168.1.75
path: /data/k8sdata/static
---
kind: Service
apiVersion: v1
metadata:
labels:
app: magedu-tomcat-app1-service-label
name: magedu-tomcat-app1-service
namespace: magedu
spec:
type : NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
nodePort: 30092
selector:
app: magedu-tomcat-app1-selector
root@k8s-node1:~/k8s-data/yaml/magedu/tomcat-app1
root@k8s-master1:~/k8s-data/yaml/magedu/tomcat-app1
root@k8s-master1:~/k8s-data/yaml/magedu/tomcat-app1
root@k8s-master1:~/k8s-data/yaml/magedu/tomcat-app1
root@k8s-node1:~/k8s-data/dockerfile/web/pub-images/nginx-base
FROM harbor.nbrhce.com/baseimages/nbrhce-centos-base:7.9.2009
MAINTAINER zhangshijie@magedu.net
RUN yum install -y vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop
ADD nginx-1.22.0.tar.gz /usr/local/src/
RUN cd /usr/local/src/nginx-1.22.0 && ./configure && make && make install && ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx &&rm -rf /usr/local/src/nginx-1.22.0.tar.gz
root@k8s-node1:~/k8s-data/dockerfile/web/pub-images/nginx-base
nerdctl build -t harbor.nbrhce.com/pub-image/nginx-base:v1.22.0 .
nerdctl push harbor.nbrhce.com/pub-image/nginx-base:v1.22.0
root@k8s-node1:~/k8s-data/dockerfile/web/magedu/nginx
FROM harbor.nbrhce.com/pub-image/nginx-base:v1.22.0
ADD nginx.conf /usr/local/nginx/conf/nginx.conf
ADD app1.tar.gz /usr/local/nginx/html/webapp/
ADD index.html /usr/local/nginx/html/index.html
RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/images && useradd nginx
EXPOSE 80 443
CMD ["nginx" ]
user nginx nginx;
worker_processes auto;
daemon off;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream tomcat_webserver {
server magedu-tomcat-app1-service.magedu.svc.magedu.local:80;
}
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
location /webapp {
root html;
index index.html index.htm;
}
root@k8s-node1:~/k8s-data/yaml/magedu/nginx
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: magedu-nginx-deployment-label
name: magedu-nginx-deployment
namespace: magedu
spec:
replicas: 1
selector:
matchLabels:
app: magedu-nginx-selector
template:
metadata:
labels:
app: magedu-nginx-selector
spec:
containers:
- name: magedu-nginx-container
image: harbor.nbrhce.com/demo/nginx-web1:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
protocol: TCP
name: http
- containerPort: 443
protocol: TCP
name: https
env :
- name: "password"
value: "123456"
- name: "age"
value: "20"
volumeMounts:
- name: magedu-images
mountPath: /usr/local/nginx/html/webapp/images
readOnly: false
- name: magedu-static
mountPath: /usr/local/nginx/html/webapp/static
readOnly: false
volumes:
- name: magedu-images
nfs:
server: 192.168.1.75
path: /data/k8sdata/images
- name: magedu-static
nfs:
server: 192.168.1.75
path: /data/k8sdata/static
---
kind: Service
apiVersion: v1
metadata:
labels:
app: magedu-nginx-service-label
name: magedu-nginx-service
namespace: magedu
spec:
type : NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
nodePort: 30090
- name: https
port: 443
protocol: TCP
targetPort: 443
nodePort: 30091
selector:
app: magedu-nginx-selector