编译安装Kubernetes 1.29 高可用集群(9)--Harbor私有仓库部署

1. Harbor软件安装

安装docker-ce，再次不熬述

1.1 下载解压harbor

# wget https://github.com/goharbor/harbor/releases/download/v2.10.3/harbor-offline-installer-v2.10.3.tgz
# mkdir /opt/software
# tar -xvf harbor-offline-installer-v2.10.3.tgz -C /opt/software

1.2 安装docker-compose

＃ curl -L https://github.com/docker/compose/releases/download/2.28.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
＃ chmod +x /usr/local/bin/docker-compose
＃ docker-compose --version
Docker Compose version 2.28.1

1.3 生成ssl证书文件

# mkdir {harbor安装路径}/cert && cd {harbor安装路径}/cert

# openssl genrsa -out harbor-ca.key 4096
# openssl req -x509 -new -nodes -sha512 -days 36500 \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=harbor/OU=harbor-CNPC/CN=harbor.test.local" \
  -key harbor-ca.key \
  -out harbor-ca.crt

# openssl genrsa -out harbor.test.local.key 4096
# openssl req -sha512 -new \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=harbor/OU=harbor-CNPC/CN=harbor.test.local" \
  -key harbor.test.local.key \
  -out harbor.test.local.csr

# cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor.test.local   //自定义的域名
DNS.2=k8s-harbor      //hostname
EOF

# openssl x509 -req -sha512 -days 36500 \
  -extfile v3.ext \
  -CA harbor-ca.crt -CAkey harbor-ca.key -CAcreateserial \
  -in harbor.test.local.csr \
  -out harbor.test.local.crt

# openssl x509 -inform PEM -in harbor.test.local.crt -out harbor.test.local.cert

1.4 将ssl证书拷贝至打包docker主机

scp /opt/harbor/cert/harbor.test.local.crt root@192.168.83.83:/etc/docker/cert
scp /opt/harbor/cert/harbor.test.local.key root@192.168.83.83:/etc/docker/cert

1.5 修改harbor配置文件

# cp /opt/software/harbor/harbor.yml.tmpl /opt/software/harbor/harbor.yml
# vi /opt/software/harbor/harbor.yml
hostname: 192.168.83.213  //harbor主机IP

http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: {harbor http端口}
# https related config
https:
  # https port for harbor, default is 443
  port: {harbor https端口}
  # The path of cert and key files for nginx  
  certificate: /opt/harbor/cert/harbor.test.local.crt
  private_key: /opt/hatbor/cert/harbor.test.local.key

harbor_admin_password: {harbor登录密码}

database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: {harbor数据库登录密码}

data_volume: /opt/harbor/data

1.6 安装harbor

# cd /opt/software/harbor
# ./prepare
prepare base dir is set to /opt/software/harbor
Unable to find image 'goharbor/prepare:v2.10.3' locally
v2.10.3: Pulling from goharbor/prepare
# ./install.sh --with-trivy

[Step 0]: checking installation environment ...

Note: docker version: 26.1.3

Note: docker-compose version: 2.28.1

[Step 1]: loading Harbor images ...
注：--with-trivy选项是添trivy扫描器

1.7 查看生成的images和开发的端口

# docker images
REPOSITORY                      TAG       IMAGE ID       CREATED      SIZE
goharbor/harbor-exporter        v2.10.3   7aa694114453   9 days ago   107MB
goharbor/redis-photon           v2.10.3   359683ebd59c   9 days ago   165MB
goharbor/trivy-adapter-photon   v2.10.3   6d51017555d9   9 days ago   516MB
goharbor/harbor-registryctl     v2.10.3   96efb27ac641   9 days ago   152MB
goharbor/registry-photon        v2.10.3   f73ad548d874   9 days ago   84.7MB
goharbor/nginx-photon           v2.10.3   0ae16f53f286   9 days ago   154MB
goharbor/harbor-log             v2.10.3   9ee81cca16ce   9 days ago   163MB
goharbor/harbor-jobservice      v2.10.3   053df0842d37   9 days ago   143MB
goharbor/harbor-core            v2.10.3   ed909856be82   9 days ago   171MB
goharbor/harbor-portal          v2.10.3   c8c29cea2ef3   9 days ago   162MB
goharbor/harbor-db              v2.10.3   c623b46fd89d   9 days ago   269MB
goharbor/prepare                v2.10.3   292e199c8873   9 days ago   210MB

# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:1514          0.0.0.0:*               LISTEN      1601/docker-proxy         
tcp6       0      0 :::443                  :::*                    LISTEN      1733/docker-proxy   
tcp6       0      0 :::80                   :::*                    LISTEN      1745/docker-proxy 

1.8 访问登录harbor

https://{harbor主机ip}，默认用户名/密码:admin/Harbor12345

2. 打包docker服务器配置

2.1 添加私有仓库地址

# cat /etc/docker/daemon.json 
{
    "registry-mirrors": [
        "https://docker.1panel.live",
        "https://docker.anyhub.us.kg",
        "https://dockerpull.com"],
    "insecure-registries": ["https://harbor.test.local"]   //私有仓库地址
}

# systemctl restart docker
# docker info
 Insecure Registries:
  harbor.test.local

2.2 修改docker启动程序指定ssl登录证书

# vi /usr/lib/systemd/system/docker.service
将
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
改为
ExecStart=/usr/bin/dockerd \
--tlsverify \
--tlscacert=/etc/docker/cert/harbor.test.local.crt \
--tlscert=/etc/docker/cert/harbor.test.local.crt \
--tlskey=/etc/docker/cert/harbor.test.local.key \
-H fd:// --containerd=/run/containerd/containerd.sock

# systemctl daemon-reload
# systemctl restart docker

2.3 登录harbor

# docker login harbor.test.local -u admin -p Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

2.4 上传镜像至harbor

### harbor项目test
# docker tag alpine:3.20.1 harbor.test.local/test/alpine:3.20.1
# docker push harbor.test.local/test/alpine:3.20.1
The push refers to repository [harbor.test.local/test/alpine]
bab9b310e2b0: Pushed 
94e5f06ff8e3: Pushed 
3.20.1: digest: sha256:6880c9b6d1d09e1f9bce70217ec4a3c2435911bd2ba61fafc0b5da15c3204092 size: 740

3. 各k8s-node主机containerd配置修改 

# vi /etc/containerd/config.toml
###149还开始修改为
         [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.test.local".auth]
            username = "admin"
            password = "Harbor12345"
            
# cat > /etc/containerd/certs.d/harbor.test.local/hosts.toml << EOF
server = "https://harbor.test.local"
[host."https://harbor.test.local"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
EOF

# systemctl restart containerd