kubernetes的Harbor镜像私库在线部署（二）
一 Harbor简介
Harbor支持高级特性，如用户管理、访问控制、活动监视和实例之间的复制。
Project Harbor是一个开源的可信云本地注册项目，用于存储、标记和扫描上下文。
Harbor扩展了开源Docker分发版，增加了用户通常需要的功能，如安全、身份和管理。
Harbor是由VMWare在Docker Registry的基础之上进行了二次封装，加进去了很多额外程序，而且提供了一个非常漂亮的web界面。
1 Harbor的功能
harbor除了具有基础的拉取和推送镜像功能外，还有以下功能。
1.1 效率
搭建了组织内部的私有容器registry服务，可以减少访问公共registry服务的网络需求。
1.2 镜像复制
可在实例之间复制镜像。
1.3 审计
所有对registry服务进行的操作均会被记录，便于日后审计。
1.4 管理界面
有友好、易用的图形管理界面，还可以进行镜像的查询和搜索。
1.5 镜像远程复制
镜像可以在多个镜像私库之间复制、同步。
1.6 REST API
提供了REST API，用于更加方便地与外系统集成。
1.7 镜像扫描
准确地说，这是安全工具Clair的功能，通过集成Clair的功能可对存储在harbor上的镜像漏洞进行警示。
1.8 国际化
支持多种语言实时切换
1.9 访问控制
提供了基于角色的访问控制，用户和镜像私库之间通过项目进行关联，用户对项目的镜像可根据角色设定不同的访问权限，还可以集成企业目前拥有的用户管理系统，例如：AD和LDAP.
2 主机初始化
2.1 配置CentOS主机yum仓库
下载并安装阿里云yum源
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
或
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo2.2 配置kubernetes的yum仓库
cat << END > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
END2.3 配置docker yum源
添加阿里的 docker镜像源，使用其中的 containerd 软件
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@harbor yum.repos.d]# ll
total 12
drwxr-xr-x 2 root root 187 May 14 10:50 bak
-rw-r--r-- 1 root root 2523 Aug 4 2022 CentOS-Base.repo
-rw-r--r-- 1 root root 2081 May 14 03:00 docker-ce.repo
-rw-r--r-- 1 root root 276 May 14 10:51 kubernetes.repo运行以下命令生成缓存
yum clean all && yum makecache2.4 主机名配置
cat > /etc/hosts <<EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.6 k8s-master01
192.168.1.7 k8s-slave01
192.168.1.8 k8s-slave02
192.168.1.100 harbor.image.com
EOF3 安装docker
yum install -y docker-ce注意：在设置Docker配置文件时，需要将harbor访问地址，设置在Docker的Http访问白名单中，不然本服务器的Docker会拒绝访问私有Harbor服务(推送、拉取镜像)。
3.1 私有镜像地址
由于harbor采用的是htp服务，而docker仓库默认采用htps，如果对私服采用docker login、pull、push等命令操作非htps的docker regsitry的时就会报错，所以需要在docker的配置文件中将harbor私服地址配置到insecure-registry作为不安全的地址运行访问。
私有镜像地址的配置，如下所示：
mkdir -p /etc/docker
cat << EOF > /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://kn0t2bca.mirror.aliyuncs.com"],
"insecure-registries": ["192.168.1.100"]
}
EOF加载配置使修改的配置文件生效systemctl daemon-reload
3.2 重启docker
systemctl restart docker && systemctl enable docker
4 安装docker compose
4.1 介绍docker compose
docker-Compose（这里使用版本号：v2.26.0）是用于定义和运行多个Docker容器（程序）的工具。通过 Compose 我们可以使用 YML文件来配置应用程序需要的所有服务。然后使用一个命令，就可以从 YML 文件配置中创建并启动所有服务。
4.2 下载docker compose
cd /opt/
curl -SL https://github.com/docker/compose/releases/download/v2.26.0/docker-compose-linux-x86_64 -o docker-compose-linux-x864-v2.26.04.3 配置环境
cd /opt/
cp docker-compose-linux-x864-v2.26.0 /usr/local/bin/docker-compose4.4 权限配置
设置docker-compose的路径权限chmod +x /usr/local/bin/docker-compose
4.5 检查版本
[root@Harbor ~]# docker-compose -v
Docker Compose version v2.26.0
[root@Harbor ~]# 5 Harbor部署
必须优先部署docker环境，才进行harbor的安装。
5.1 下载harbor
wget https://github.com/goharbor/harbor/releases/download/v2.9.4/harbor-offline-installer-v2.9.4.tgz5.2 创建家目录
mkdir -p /opt/harbor5.3 解压安装包
tar zxf harbor-offline-installer-v2.9.4.tgz 5.4 修改配置文件
cd /opt/harbor/harbor/
cp harbor.yml.tmpl harbor.yml修改的地方有：
hostname: 192.168.1.100
注释掉协议：
https:
修改登陆admin的密码和数据库的密码。
上图中4个箭头分别对应4个需要修改的部分
（1）访问IP（务必改成当前服务器的IP或域名，不可用localhost或127.0.0.1）
（2）http访问端口（如果这服务器是专用于私有镜像仓库的，可设port为80，通过IP即可访问harbor服务）
（3）初次安装harbor时的admin账号密码（启动服务后，登录并修改即可，之后就会以修改后的密码为准），默认初始密码：harbor2024devops
（4）修改数据库的密码，注意记录新密码的值，默认数据库密码：harbor2024DevOps
5.5 修改后的配置文件
[root@harbor harbor]# egrep -v "#|^$" harbor.yml
hostname: 192.168.1.100
http:
port: 80
harbor_admin_password: harbor2024devops
database:
password: harbor2024DevOps
max_idle_conns: 100
max_open_conns: 900
conn_max_lifetime: 5m
conn_max_idle_time: 0
data_volume: /data
trivy:
ignore_unfixed: false
skip_update: false
skip_java_db_update: false
offline_scan: false
security_check: vuln
insecure: false
jobservice:
max_job_workers: 10
job_loggers:
- STD_OUTPUT
- FILE
notification:
webhook_job_max_retry: 3
log:
level: info
local:
rotate_count: 50
rotate_size: 200M
location: /var/log/harbor
_version: 2.9.0
proxy:
http_proxy:
https_proxy:
no_proxy:
components:
- core
- jobservice
- trivy
upload_purging:
enabled: true
age: 168h
interval: 24h
dryrun: false
cache:
enabled: false
expire_hours: 24
[root@harbor harbor]# 5.6 检查环境
检查相关组件的镜像列表，如果本地没有相关镜像，则会自动联网下载。没网的话会报一堆依赖找不到，这里已提前全部下载。
执行命令：
./prepare
#检查过程；
[root@harbor harbor]# ./prepare
prepare base dir is set to /opt/harbor/harbor
Unable to find image 'goharbor/prepare:v2.9.4' locally
v2.9.4: Pulling from goharbor/prepare
5c70ea440659: Pull complete
97b717f3829c: Pull complete
8e3e2393f1f5: Pull complete
922fd4ce5cf9: Pull complete
6d5dc2e2a2f6: Pull complete
2e06411d314c: Pull complete
b129f0ef1377: Pull complete
4dcbfb4c87cd: Pull complete
792e707a745f: Pull complete
c25b0082aebd: Pull complete
Digest: sha256:8525fb155c3471a624b9e736a1ab26298e574450f045a96efd85862e8a873711
Status: Downloaded newer image for goharbor/prepare:v2.9.4
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
[root@harbor harbor]# 检查环境时，这里是联网状态，所以会自动下载该镜像：goharbor/prepare:v2.9.4，如果是公司内网，需要提前导入该镜像，才能执行环境的检查，生成docker-compose.yml。
5.7 查看依赖的镜像
以下harbor依赖的镜像，联网情况下执行安装会自动下载，如果是内网环境，需要提前下载，然后上传至部署的离线环境中，导入即可。
cd /opt/harbor/harbor/
[root@harbor harbor]# cat docker-compose.yml |grep image|uniq
image: goharbor/harbor-log:v2.9.4
image: goharbor/registry-photon:v2.9.4
image: goharbor/harbor-registryctl:v2.9.4
image: goharbor/harbor-db:v2.9.4
image: goharbor/harbor-core:v2.9.4
image: goharbor/harbor-portal:v2.9.4
image: goharbor/harbor-jobservice:v2.9.4
image: goharbor/redis-photon:v2.9.4
image: goharbor/nginx-photon:v2.9.4
[root@harbor harbor]# 5.8 执行安装脚本
[root@harbor harbor]# ./install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 26.1.2
[Step 1]: checking docker-compose is installed ...
Note: Docker Compose version v2.27.0
[Step 2]: loading Harbor images ...
1ffa30c514e7: Loading layer [==================================================>] 115.6MB/115.6MB
Loaded image: goharbor/nginx-photon:v2.9.4
fe9d27457645: Loading layer [==================================================>] 9.117MB/9.117MB
8a1bd330387a: Loading layer [==================================================>] 4.096kB/4.096kB
de5aec39cfef: Loading layer [==================================================>] 3.072kB/3.072kB
b2dc5f9b0adc: Loading layer [==================================================>] 209.9MB/209.9MB
867f6775e77c: Loading layer [==================================================>] 17.45MB/17.45MB
3d23e7d1c061: Loading layer [==================================================>] 228.1MB/228.1MB
Loaded image: goharbor/trivy-adapter-photon:v2.9.4
8e8d9c5fab4c: Loading layer [==================================================>] 115.6MB/115.6MB
8c2100da7b2f: Loading layer [==================================================>] 6.46MB/6.46MB
beb75a4af778: Loading layer [==================================================>] 245.8kB/245.8kB
8b93536ed20c: Loading layer [==================================================>] 1.233MB/1.233MB
Loaded image: goharbor/harbor-portal:v2.9.4
303f42b51cc2: Loading layer [==================================================>] 11.61MB/11.61MB
f6d69dd1d0ad: Loading layer [==================================================>] 3.584kB/3.584kB
8dbc774dc94d: Loading layer [==================================================>] 2.56kB/2.56kB
baba5716726a: Loading layer [==================================================>] 58.75MB/58.75MB
5b67417f435d: Loading layer [==================================================>] 5.632kB/5.632kB
31a7f7742aaa: Loading layer [==================================================>] 122.4kB/122.4kB
a9bc112e775d: Loading layer [==================================================>] 80.38kB/80.38kB
3c10f76aad84: Loading layer [==================================================>] 59.74MB/59.74MB
6c5eb2684647: Loading layer [==================================================>] 2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.9.4
807b050bbb83: Loading layer [==================================================>] 125.3MB/125.3MB
00b81cc4331d: Loading layer [==================================================>] 3.584kB/3.584kB
73d4601a8fa0: Loading layer [==================================================>] 3.072kB/3.072kB
873b556fed34: Loading layer [==================================================>] 2.56kB/2.56kB
d34eaff1dda9: Loading layer [==================================================>] 3.072kB/3.072kB
875416323a99: Loading layer [==================================================>] 3.584kB/3.584kB
04668820172e: Loading layer [==================================================>] 20.48kB/20.48kB
Loaded image: goharbor/harbor-log:v2.9.4
fc9708739fba: Loading layer [==================================================>] 11.61MB/11.61MB
5fbf23947d53: Loading layer [==================================================>] 3.584kB/3.584kB
8980553d20e3: Loading layer [==================================================>] 2.56kB/2.56kB
394a06e3fead: Loading layer [==================================================>] 44.64MB/44.64MB
0b32832b0417: Loading layer [==================================================>] 45.43MB/45.43MB
Loaded image: goharbor/harbor-jobservice:v2.9.4
649c2fb761b1: Loading layer [==================================================>] 11.61MB/11.61MB
dee3ec490421: Loading layer [==================================================>] 27.58MB/27.58MB
ce1ae31f2333: Loading layer [==================================================>] 4.608kB/4.608kB
67b43b0afe41: Loading layer [==================================================>] 28.37MB/28.37MB
Loaded image: goharbor/harbor-exporter:v2.9.4
Loaded image: goharbor/prepare:v2.9.4
9680f9b537e4: Loading layer [==================================================>] 16.08MB/16.08MB
5c80f459c1fa: Loading layer [==================================================>] 173.8MB/173.8MB
b4a8872dec5a: Loading layer [==================================================>] 25.53MB/25.53MB
273925008692: Loading layer [==================================================>] 18.3MB/18.3MB
61756a3742aa: Loading layer [==================================================>] 5.12kB/5.12kB
6aef2ba5f411: Loading layer [==================================================>] 6.144kB/6.144kB
06b8309f6264: Loading layer [==================================================>] 3.072kB/3.072kB
2378215d8cc7: Loading layer [==================================================>] 2.048kB/2.048kB
61639698f424: Loading layer [==================================================>] 2.56kB/2.56kB
11138989c354: Loading layer [==================================================>] 7.68kB/7.68kB
Loaded image: goharbor/harbor-db:v2.9.4
5476b1e98825: Loading layer [==================================================>] 8.603MB/8.603MB
21306bf47b19: Loading layer [==================================================>] 4.096kB/4.096kB
b6a29409bcb0: Loading layer [==================================================>] 17.39MB/17.39MB
f923c3b9c6b2: Loading layer [==================================================>] 3.072kB/3.072kB
b65b25d422a3: Loading layer [==================================================>] 32.69MB/32.69MB
f33443527772: Loading layer [==================================================>] 50.88MB/50.88MB
Loaded image: goharbor/harbor-registryctl:v2.9.4
297dfef001b5: Loading layer [==================================================>] 16.08MB/16.08MB
628e609117e3: Loading layer [==================================================>] 110.6MB/110.6MB
d3164a9024b3: Loading layer [==================================================>] 3.072kB/3.072kB
3bbb99540044: Loading layer [==================================================>] 59.9kB/59.9kB
0b8ee78ceda9: Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.9.4
a4efec05118e: Loading layer [==================================================>] 8.603MB/8.603MB
fc5a28753b96: Loading layer [==================================================>] 4.096kB/4.096kB
e7ebcb1d0d37: Loading layer [==================================================>] 3.072kB/3.072kB
f5a739f25b3b: Loading layer [==================================================>] 17.39MB/17.39MB
3bdc4fd383a7: Loading layer [==================================================>] 18.19MB/18.19MB
Loaded image: goharbor/registry-photon:v2.9.4
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /opt/harbor/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Note: stopping existing Harbor instance ...
WARN[0000] /opt/harbor/harbor/docker-compose.yml: `version` is obsolete
[Step 5]: starting Harbor ...
WARN[0000] /opt/harbor/harbor/docker-compose.yml: `version` is obsolete
[+] Running 10/10
✔ Network harbor_harbor Created 0.1s
✔ Container harbor-log Started 0.3s
✔ Container harbor-portal Started 0.9s
✔ Container registry Started 0.9s
✔ Container registryctl Started 1.0s
✔ Container harbor-db Started 0.8s
✔ Container redis Started 0.9s
✔ Container harbor-core Started 1.2s
✔ Container nginx Started 1.7s
✔ Container harbor-jobservice Started 1.8s
✔ ----Harbor has been installed and started successfully.----
[root@harbor harbor]# 上述安装过程介绍：前面是检查docker和docker-compose部署情况，以及镜像包的获取情况。环境检查完全没问题后，创建一些配置文件，最后部署成功之后自动启动harbor及相关组件。
5.9 查看harbor web端口
[root@harbor ~]# ss -antl|grep 80
LISTEN 0 128 *:80 *:*
LISTEN 0 128 :::80 :::*
[root@harbor ~]# 5.10 配置自启动
添加启动服务文件，设置好之后记得重启，检测效果。
cat << END > /lib/systemd/system/harbor.service
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor
[Service]
Type=simple
Restart=on-failure
RestartSec=60
ExecStart=/usr/local/bin/docker-compose -f /opt/harbor/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /opt/harbor/harbor/docker-compose.yml stop
[Install]
WantedBy=multi-user.target
END更新配置信息，更新 ExecStart 与 ExecStop 的值，前半段是服务器安装 docker-compose的位置，后半段是 harbor的docker-compose文件的位置。其中，/usr/local/bin/docker-compose 的位置需要根据安装的docker-compose的位置自行更新，要在对应的位置找到该文件才行，否则不生效。
注意：其中的 RestartSec 的值记得设置长一些，太短的话容易因重启时服务没启动全，导致部分服务不能正常启动，还需要事后手动启动。
5.11 重启harbor服务
# 加载配置文件的更新
systemctl daemon-reload && systemctl enable harbor
# 开机自启
systemctl enable harbor
# 启动服务（已经启动就不用执行了）
systemctl start harbor
# 检查状态
systemctl status harbor
# 检查状态
systemctl stop harbor使用Harbor的注意以下几点：
在客户端上传镜像时一定要记得执行docker login进行用户认证，否则无法直接push。
在客户端使用的时候如果不是用的https则必须要在客户端的/etc/docker/daemon.json配置文件中配置insecure-registries参数。
数据存放路径应在配置文件中配置到一个容量比较充足的共享存储中。
Harbor是使用docker-compose命令来管理的，如果需要停止Harbor也应用 docker-compose stop 来停止，其他参数请--help。
到此，harbor镜像仓库部署完成，进入下一步使用及测试验证功能。
6 登陆 Harbor web界面
#登陆地址：
http://192.168.1.100/
#登陆账号和密码
admin/harbor2024devops
6.1 进入harbor主页面
输入登陆账号和密码，进入下一步
6.2 新增开发测试用户
#新用户和密码
testimage/Himage123#@！
可以进一步设置为管理员，这里就不做操作啦。
6.3 创建公有仓库
新增项目和设置项目成员权限，镜像都需要根据项目名做分类索引进行存放，新建一个项目，点击项目->新建项目。将项目设置为公开，所有用户都可以读取，如果设置了私有项目，只有登陆了相关用户才可以操作，默认是私有项目。
6.3.1 镜像项目配置解释
1）项目名称：nc_public，也可以叫它公有镜像仓库名称。
2）访问级别：当项目设为公开后，任何人都有此项目下镜像的读权限。命令行用户不需要“docker login”就可以拉取此项目下的镜像。
3）项目配额限制：项目可以使用的最大逻辑空间，如果您想要不设置上限，配置值为‘-1’.
4）镜像代理：开启此项，以使得该项目成为目标仓库的镜像代理，仅支持DockerHub, Docker Registry, Harbor, Aws ECR, AzureACR, Quay, Google GcR,JFrog Artifactory.和GithubGHCR 类型的仓库。
这里就不配置镜像代理了，以后有机会用到，在更新分享。
6.3.2 配置普通用户
添加用户 testimage，角色是开发者，管理员对项目拥有所有操作权限，push/pull/delete/复制策略等操作，开发人员只能push/pull，访客只能pull。点击项目名称 nc_public 就会跳转弹窗下图，把之前创建好的用户名称 testimage添加 角色配置成‘开发者’即可。
添加完成后项目拥有两个项目成员，分别是管理员admin和开发者 testimage，可以点击其他操作修改成员的权限或者移除成员。
6.4 公有仓库测试
6.4.1 推送镜像
上传镜像需要在docker主机进行操作，对于私有项目必须登录Harbor私服才能进行pull和push操作，push到Harbor的镜像的命名必须按照如下规则。
{Harbor地址}:{端口}/{Harbor项目名}/{自定义镜像名}:{自定义tag}下一步测试登录私服：
登录成功，进行镜像推送至 harbor 的nc_public仓库中。
6.4.2 获取镜像
在k8s的工作节点导入镜像，以 NGINX 为例：
[root@k8s-slave01 ~]# docker load -i nginx-latest.tar
08249ce7456a: Loading layer [==================================================>] 83.9MB/83.9MB
d5b40e80384b: Loading layer [==================================================>] 62.01MB/62.01MB
b2f82de68e0d: Loading layer [==================================================>] 3.072kB/3.072kB
41451f050aa8: Loading layer [==================================================>] 4.096kB/4.096kB
44193d3f4ea2: Loading layer [==================================================>] 3.584kB/3.584kB
e7344f8a29a3: Loading layer [==================================================>] 7.168kB/7.168kB
Loaded image: nginx:latest
[root@k8s-slave01 ~]# docker images|grep nginx
nginx latest 55f4b40fe486 23 months ago 142MB
[root@k8s-slave01 ~]# 6.4.3 修改镜像标签
将获取到的镜像，先使用docker tag创建一个新的镜像并且重命名为harbor指定的 ip:host/项目名格式，然后直接push。
将导入默认版本nginx镜像为例：
在项目中标记镜像：
#模板
docker tag SOURCE_IMAGE[:TAG] 192.168.1.100/nc_public/REPOSITORY[:TAG]
#执行的命令行
docker tag nginx:latest 192.168.1.100/nc_public/nginx:v1.0.0
[root@k8s-slave01 ~]# docker tag nginx:latest 192.168.1.100/nc_public/nginx:v1.0.0
[root@k8s-slave01 ~]# docker images|grep nginx
nginx latest 55f4b40fe486 23 months ago 142MB
192.168.1.100/nc_public/nginx v1.0.0 55f4b40fe486 23 months ago 142MB
[root@k8s-slave01 ~]# 6.4.4 push推送镜像
Docker 推送命令，推送镜像到当前项目：
#模板
docker push 192.168.1.100/nc_public/REPOSITORY[:TAG]
#执行的命令行
docker push 192.168.1.100/nc_public/nginx:v1.0.0直接执行推送命令，查看过程发现没有推送成功，因为我在执行这一步的时候，没有提前进行"docker login"登陆到harbor 所以报了为授权错误，该工作节点没有权限推送镜像到nc_public/nginx项目路径下，具体报错如下。
6.4.4.1 解决授权问题
在k8s的工作节点上，通过开发者用户 testimage 登陆到harbor镜像仓库，在进行推送。
登陆成功，执行推送命令，发现镜像已经push成功：
[root@k8s-slave01 ~]# docker push 192.168.1.100/nc_public/nginx:v1.0.0
The push refers to repository [192.168.1.100/nc_public/nginx]
e7344f8a29a3: Pushed
44193d3f4ea2: Pushed
41451f050aa8: Pushed
b2f82de68e0d: Pushed
d5b40e80384b: Pushed
08249ce7456a: Pushed
v1.0.0: digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1 size: 1570
[root@k8s-slave01 ~]# 6.4.5 验证镜像推送
登陆或刷新harbor界面，查看工作节点k8s-slave01推送的NGINX镜像是否上传至192.168.1.100/nc_public/项目路径下，如果有存在，则表示镜像推送成功，反之失败。近一步点进去可以查看镜像的标签信息，点击操作可以对镜像进行复制摘要、添加标签、复制镜像、删除镜像等操作。
如图所示，NGINX镜像已经存放在192.168.1.100/nc_public/项目路径下，说明我在工作节点k8s-slave01使用开发者用户 testimage 推送镜像已经成功，也可以使用该用户登陆进行查看。当前界面是以 testimage 用户（开发者）进行登录的，因此没有删除权限，也没有左侧的系统管理栏。
6.4.6 pull镜像
下一步在另一台工作节点k8s-slave02主机上从私服拉去镜像，确保这台机器和私服ip互通，并且将私服地址添加到docker的不安全仓库地址配置文件中，直接使用docker pull命令拉取私服镜像。
在工作节点拉取镜：
k8s-slave02:
docker pull 192.168.1.100/nc_public/nginx:v1.0.0
#执行命令，可以直接拉取镜像：
[root@k8s-slave02 ~]# docker pull 192.168.1.100/nc_public/nginx:v1.0.0
v1.0.0: Pulling from nc_public/nginx
b85a868b505f: Pull complete
f4407ba1f103: Pull complete
4a7307612456: Pull complete
935cecace2a0: Pull complete
8f46223e4234: Pull complete
fe0ef4c895f5: Pull complete
Digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1
Status: Downloaded newer image for 192.168.1.100/nc_public/nginx:v1.0.0
192.168.1.100/nc_public/nginx:v1.0.0
#镜像拉取成功
[root@k8s-slave02 ~]# docker images|grep nginx
192.168.1.100/nc_public/nginx v1.0.0 55f4b40fe486 23 months ago 142MB
[root@k8s-slave02 ~]# 6.4.6.1 运行公有镜像
以此镜像在k8s-slave02工作节点（192.168.1.8）启动容器，并访问测试：
192.168.1.100/nc_public/nginx:v1.0.0
docker run -itd -p 8088:80 192.168.1.100/nc_public/nginx:v1.0.0
[root@k8s-slave02 ~]# docker images|grep nginx
192.168.1.100/nc_public/nginx v1.0.0 55f4b40fe486 23 months ago 142MB
[root@k8s-slave02 ~]#
[root@k8s-slave02 ~]# docker run -itd -p 8088:80 192.168.1.100/nc_public/nginx:v1.0.0
a8fc741cff8ac673bbce865f56b8bf4503cc839c28368053775f60edf9187b77
[root@k8s-slave02 ~]#
[root@k8s-slave02 ~]# docker ps -a|grep nginx
a8fc741cff8a 192.168.1.100/nc_public/nginx:v1.0.0 "/docker-entrypoint.…" 4 minutes ago Up 4 minutes 0.0.0.0:8088->80/tcp gallant_bose
[root@k8s-slave02 ~]# 6.4.6.2 访问nginx-web
http://192.168.1.8:8088/
服务访问正常，说明我的公共镜像的推送和拉取整个环节都形成完整的闭环，则表示功能没问题。
6.4.7 Harbor服务器上的测试
#先打标签，在执行 “docker login”的情况下执行push
docker tag nginx:latest 192.168.1.100/nc_public/nginx:v1.0.1
docker push 192.168.1.100/nc_public/nginx:v1.0.1
[root@harbor ~]# docker push 192.168.1.100/nc_public/nginx:v1.0.1
The push refers to repository [192.168.1.100/nc_public/nginx]
e7344f8a29a3: Pushed
44193d3f4ea2: Pushed
41451f050aa8: Pushed
b2f82de68e0d: Pushed
d5b40e80384b: Pushed
08249ce7456a: Pushed
v1.0.1: digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1 size: 1570
[root@harbor ~]#
#如果退出“docker login”，执行“docker push”就会失败报错如下：
[root@harbor ~]# docker logout 192.168.1.100
Removing login credentials for 192.168.1.100
[root@harbor ~]#
[root@harbor ~]# docker push 192.168.1.100/nc_public/nginx:v1.0.2
The push refers to repository [192.168.1.100/nc_public/nginx]
e7344f8a29a3: Layer already exists
44193d3f4ea2: Layer already exists
41451f050aa8: Layer already exists
b2f82de68e0d: Layer already exists
d5b40e80384b: Layer already exists
08249ce7456a: Layer already exists
unauthorized: unauthorized to access repository: nc_public/nginx, action: push: unauthorized to access repository: nc_public/nginx, action: push
[root@harbor ~]#
[root@harbor ~]# docker login 192.168.1.100 -u testimage
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@harbor ~]#
[root@harbor ~]# docker push 192.168.1.100/nc_public/nginx:v1.0.2
The push refers to repository [192.168.1.100/nc_public/nginx]
e7344f8a29a3: Layer already exists
44193d3f4ea2: Layer already exists
41451f050aa8: Layer already exists
b2f82de68e0d: Layer already exists
d5b40e80384b: Layer already exists
08249ce7456a: Layer already exists
v1.0.2: digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1 size: 1570
[root@harbor ~]# 6.4.7.1 查看镜像上传
192.168.1.100/nc_public/nginx:v1.0.1 镜像存在，则表示上传成功。
6.4.7.2 拉取镜像并运行
#先把harbor上原有的镜像删除，在验证拉取上传的nginx:v1.0.1镜像
[root@harbor ~]# docker rmi 55f4b40fe486 -f
Untagged: 192.168.1.100/nc_public/nginx:v1.0.1
Untagged: 192.168.1.100/nc_public/nginx@sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1
Untagged: nginx:latest
Deleted: sha256:55f4b40fe486a5b734b46bb7bf28f52fa31426bf23be068c8e7b19e58d9b8deb
Deleted: sha256:5f58fed9b4d8e6c09cdc42eed6de6df7a7e35b40d92c98f30f8ecad4960fb7a0
Deleted: sha256:8bb72c1d014292ebf1ae348a77624c536e766757356c6dbb0de75122a94b445d
Deleted: sha256:cc9ac0adbded956d924bcf6c26ffbc93ea070019be1437d204b530a033ff4b16
Deleted: sha256:30f210588f35917f0edb5a2465db7ad60e4ef3b6ac74fe155474e14e6f0995c5
Deleted: sha256:5ecd5431cf49a2a11115844de1e7b23b9535be8789add9ab50973867db5f7d36
Deleted: sha256:08249ce7456a1c0613eafe868aed936a284ed9f1d6144f7d2d08c514974a2af9
[root@harbor ~]#
#在不执行 “docker login”的情况下执行pull
docker pull 192.168.1.100/nc_public/nginx:v1.0.1
[root@harbor ~]# docker pull 192.168.1.100/nc_public/nginx:v1.0.1
v1.0.1: Pulling from nc_public/nginx
b85a868b505f: Pull complete
f4407ba1f103: Pull complete
4a7307612456: Pull complete
935cecace2a0: Pull complete
8f46223e4234: Pull complete
fe0ef4c895f5: Pull complete
Digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1
Status: Downloaded newer image for 192.168.1.100/nc_public/nginx:v1.0.1
192.168.1.100/nc_public/nginx:v1.0.1
[root@harbor ~]# docker images|grep nginx
192.168.1.100/nc_public/nginx v1.0.1 55f4b40fe486 23 months ago 142MB
[root@harbor ~]#
#运行nginx镜像
docker run -itd -p 8089:80 192.168.1.100/nc_public/nginx:v1.0.1
[root@harbor ~]# docker run -itd -p 8089:80 192.168.1.100/nc_public/nginx:v1.0.1
c76090c2fa2dfe488611af0821a6c24f742d88be948e24a0c6c9c19e5a7c36a1
[root@harbor ~]#
#访问nginx-web镜像
curl -I http://192.168.1.100:8089/
[root@k8s-slave02 ~]# curl -I http://192.168.1.100:8089/
HTTP/1.1 200 OK
Server: nginx/1.23.0
Date: Wed, 29 May 2024 20:11:43 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 21 Jun 2022 14:25:37 GMT
Connection: keep-alive
ETag: "62b1d4e1-267"
Accept-Ranges: bytes
[root@k8s-slave02 ~]# 经过一番测试，在harbor服务器上执行，对于公开项目不需要执行“docker login”就可以访问到私服的用户pull拉取，最后运行服务，能正常访问，则满足功能需求。
6.5 私有仓库测试
对于私有项目需要“docker login”才能进行操作，先创建一个私有仓库 nc_private 进行验证。登陆harborWEB主页面，单击项目，选择新建项目如下：
6.5.1 添加新成员
6.5.2 用普通用户push镜像
验证使用普通用户进行tag和push，是否报错未授权访问私有仓库。
6.5.2.1 Harbor主机上测试
在harbor服务器上测试，先不登陆普通用户 testimage ，检验镜像是否顺利推送至私有仓库 nc_private。
#docker推送命令模板
docker tag SOURCE_IMAGE[:TAG] 192.168.1.100/nc_private/REPOSITORY[:TAG]
docker push 192.168.1.100/nc_private/REPOSITORY[:TAG]
#先打标签
docker tag nginx:latest 192.168.1.100/nc_private/nginx:v1.0.1
#在执行推送
docker push 192.168.1.100/nc_private/nginx:v1.0.1
[root@harbor ~]# docker tag nginx:latest 192.168.1.100/nc_private/nginx:v1.0.1
[root@harbor ~]# docker images|grep nginx
goharbor/nginx-photon v2.9.4 7d707a6be242 6 weeks ago 153MB
192.168.1.100/nc_private/nginx v1.0.1 55f4b40fe486 23 months ago 142MB
nginx latest 55f4b40fe486 23 months ago 142MB
[root@harbor ~]#
#推送报权限错误问题
[root@harbor ~]# docker push 192.168.1.100/nc_private/nginx:v1.0.1
The push refers to repository [192.168.1.100/nc_private/nginx]
e7344f8a29a3: Preparing
44193d3f4ea2: Preparing
41451f050aa8: Preparing
b2f82de68e0d: Preparing
d5b40e80384b: Preparing
08249ce7456a: Waiting
unauthorized: unauthorized to access repository: nc_private/nginx, action: push: unauthorized to access repository: nc_private/nginx, action: push
[root@harbor ~]#
#解决问题，执行“docker login” 登陆用户，在执行 push
[root@harbor ~]# docker push 192.168.1.100/nc_private/nginx:v1.0.1
The push refers to repository [192.168.1.100/nc_private/nginx]
e7344f8a29a3: Mounted from nc_public/nginx
44193d3f4ea2: Mounted from nc_public/nginx
41451f050aa8: Mounted from nc_public/nginx
b2f82de68e0d: Mounted from nc_public/nginx
d5b40e80384b: Mounted from nc_public/nginx
08249ce7456a: Mounted from nc_public/nginx
v1.0.1: digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1 size: 1570
[root@harbor ~]# 如图镜像已经推送成功：
6.5.2.2 拉取镜像
在harbor主机上拉取镜像，具体操作如下：
#先删除镜像
[root@harbor ~]# docker rmi -f 55f4b40fe486
Untagged: 192.168.1.100/nc_private/nginx:v1.0.1
Untagged: 192.168.1.100/nc_private/nginx@sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1
Untagged: nginx:latest
Deleted: sha256:55f4b40fe486a5b734b46bb7bf28f52fa31426bf23be068c8e7b19e58d9b8deb
[root@harbor ~]#
#在执行拉取镜像命令
docker pull 192.168.1.100/nc_private/nginx:v1.0.1
#不执行登陆，就会报以下错误
[root@harbor ~]# docker pull 192.168.1.100/nc_private/nginx:v1.0.1
Error response from daemon: unauthorized: unauthorized to access repository: nc_private/nginx, action: pull: unauthorized to access repository: nc_private/nginx, action: pull
[root@harbor ~]#
#执行登陆解决该问题
[root@harbor ~]# docker pull 192.168.1.100/nc_private/nginx:v1.0.1
v1.0.1: Pulling from nc_private/nginx
b85a868b505f: Already exists
f4407ba1f103: Already exists
4a7307612456: Already exists
935cecace2a0: Already exists
8f46223e4234: Already exists
fe0ef4c895f5: Already exists
Digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1
Status: Downloaded newer image for 192.168.1.100/nc_private/nginx:v1.0.1
192.168.1.100/nc_private/nginx:v1.0.1
[root@harbor ~]#
[root@harbor ~]#
[root@harbor ~]# docker images|grep nginx
goharbor/nginx-photon v2.9.4 7d707a6be242 6 weeks ago 153MB
192.168.1.100/nc_private/nginx v1.0.1 55f4b40fe486 23 months ago 142MB
[root@harbor ~]# 6.5.2.3 运行镜像
在harbor主机上运行上一步拉取的镜像，执行命令如下。
#docker运行命令
docker run -itd -p 8099:80 192.168.1.100/nc_private/nginx:v1.0.1
[root@harbor ~]# docker ps -all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e9c3e6cdf7c1 192.168.1.100/nc_private/nginx:v1.0.1 "/docker-entrypoint.…" 10 seconds ago Up 8 seconds 0.0.0.0:8099->80/tcp, :::8099->80/tcp upbeat_benz
[root@harbor ~]#
#访问测试
curl -I http://192.168.1.100:8099/
[root@k8s-slave01 ~]# curl -I http://192.168.1.100:8099/
HTTP/1.1 200 OK
Server: nginx/1.23.0
Date: Sun, 02 Jun 2024 18:24:53 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 21 Jun 2022 14:25:37 GMT
Connection: keep-alive
ETag: "62b1d4e1-267"
Accept-Ranges: bytes
[root@k8s-slave01 ~]# 
6.5.3 在工作节点测试
在k8s-slave01工作节点测试，验证使用普通用户进行tag和push以及pull镜像，是否报错未授权访问私有仓库。
#检查工作节点是否有nginx镜像，这里没有，那先测试在不登陆用户的情况下，执行pull拉取之前在harbor推送的nginx:v1.0.1镜像
[root@k8s-slave01 ~]# docker images|grep nginx
[root@k8s-slave01 ~]#
#执行命令
docker pull 192.168.1.100/nc_private/nginx:v1.0.1
[root@k8s-slave01 ~]# docker pull 192.168.1.100/nc_private/nginx:v1.0.1
Error response from daemon: unauthorized: unauthorized to access repository: nc_private/nginx, action: pull: unauthorized to access repository: nc_private/nginx, action: pull
[root@k8s-slave01 ~]#
#这里报错了，大致意思，就是没有权限访问私有仓库地址
#解决方法：登陆用户testimage
[root@k8s-slave01 ~]# docker login 192.168.1.100 -u testimage
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@k8s-slave01 ~]#
#再次执行，pull镜像拉取
[root@k8s-slave01 ~]# docker pull 192.168.1.100/nc_private/nginx:v1.0.1
v1.0.1: Pulling from nc_private/nginx
b85a868b505f: Pull complete
f4407ba1f103: Pull complete
4a7307612456: Pull complete
935cecace2a0: Pull complete
8f46223e4234: Pull complete
fe0ef4c895f5: Pull complete
Digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1
Status: Downloaded newer image for 192.168.1.100/nc_private/nginx:v1.0.1
192.168.1.100/nc_private/nginx:v1.0.1
#镜像拉取成功
[root@k8s-slave01 ~]# docker images|grep nginx
192.168.1.100/nc_private/nginx v1.0.1 55f4b40fe486 23 months ago 142MB
[root@k8s-slave01 ~]#
#退出 docker login
[root@k8s-slave01 ~]# docker logout 192.168.1.100
Removing login credentials for 192.168.1.100
[root@k8s-slave01 ~]#
#修改标签
[root@k8s-slave01 ~]# docker tag 192.168.1.100/nc_private/nginx:v1.0.1 192.168.1.100/nc_private/nginx:v1.0.2
[root@k8s-slave01 ~]# docker images|grep nginx
192.168.1.100/nc_private/nginx v1.0.1 55f4b40fe486 23 months ago 142MB
192.168.1.100/nc_private/nginx v1.0.2 55f4b40fe486 23 months ago 142MB
[root@k8s-slave01 ~]#
#推送镜像 nginx:v1.0.2
[root@k8s-slave01 ~]# docker push 192.168.1.100/nc_private/nginx:v1.0.2
The push refers to repository [192.168.1.100/nc_private/nginx]
e7344f8a29a3: Preparing
44193d3f4ea2: Preparing
41451f050aa8: Preparing
b2f82de68e0d: Preparing
d5b40e80384b: Preparing
08249ce7456a: Waiting
unauthorized: unauthorized to access repository: nc_private/nginx, action: push: unauthorized to access repository: nc_private/nginx, action: push
[root@k8s-slave01 ~]#
#果然推送失败了，没有登陆docker login，是没有权限访问私有镜像仓库nc_private的
#再次执行登陆
docker login 192.168.1.100 -u testimage
#登陆成功之后，执行push命令
[root@k8s-slave01 ~]# docker push 192.168.1.100/nc_private/nginx:v1.0.2
The push refers to repository [192.168.1.100/nc_private/nginx]
e7344f8a29a3: Layer already exists
44193d3f4ea2: Layer already exists
41451f050aa8: Layer already exists
b2f82de68e0d: Layer already exists
d5b40e80384b: Layer already exists
08249ce7456a: Layer already exists
v1.0.2: digest: sha256:3536d368b898eef291fb1f6d184a95f8bc1a6f863c48457395aab859fda354d1 size: 1570
[root@k8s-slave01 ~]# 
这一次nginx:v1.0.2镜像推送成功了。
6.6 部署总结
学而时习之，不亦说乎？有朋自远方来，不亦乐乎？人不知而不愠，不亦君子乎？
