Harbor镜像仓库（含clair镜像扫描） - 完整部署记录
Harbor环境部署的要求：系统版本在Centos7.5以上、内核版本在4.4X以上、ip_forward路由转发功能要打开。
一、环境准备
[root@k8s-harbor01 ~]# cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) [root@k8s-harbor01 ~]# uname -r 4.4.232-1.el7.elrepo.x86_64 [root@k8s-harbor01 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward [root@k8s-harbor01 ~]# vim /etc/sysctl.conf net.ipv4.ip_forward = 1 [root@k8s-harbor01 ~]# sysctl -p [root@k8s-harbor01 ~]# systemctl stop firewalld && systemctl disable firewalld && firewall-cmd --state [root@k8s-harbor01 ~]# vim /etc/sysconfig/selinux SELINUX=disabled [root@k8s-harbor01 ~]# getenforce Disabled [root@k8s-harbor01 ~]# python --version Python 2.7.5
二、安装Docker
提前下载二进制安装包docker-18.09.6.tgz到/usr/local/src路径下，解压安装
[root@k8s-harbor01 ~]# cd /usr/local/src/
[root@k8s-harbor01 src]# ll docker-18.09.6.tgz
-rw-r--r-- 1 root root 48047231 Oct 19 2019 docker-18.09.6.tgz
[root@k8s-harbor01 src]# tar -zvxf docker-18.09.6.tgz
[root@k8s-harbor01 src]# cp docker/* /usr/local/bin/
[root@k8s-harbor01 src]# chmod 755 /usr/local/bin/*
/usr/local/bin默认已经加到系统环境变量中
[root@k8s-harbor01 src]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
编辑docker启动文件
注意"WorkingDirectory"路径要和/etc/docker/daemon.json文件中的data-root、exec-root路径一致
[root@k8s-harbor01 src]# cat > /etc/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
[Service]
WorkingDirectory=/data/docker
Environment="PATH=/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/local/bin/dockerd
ExecReload=/bin/kill -s HUP
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
授执行权限
[root@k8s-harbor01 src]# chmod 755 /etc/systemd/system/docker.service
编辑docker 配置文件
编辑docker 配置文件
[root@k8s-harbor01 src]# mkdir -p /etc/docker && mkdir -p /data/docker/data && mkdir -p /data/docker/exec
[root@k8s-harbor01 src]# cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
"insecure-registries": ["docker02:35000"],
"max-concurrent-downloads": 20,
"live-restore": true,
"max-concurrent-uploads": 10,
"debug": true,
"data-root": "/data/docker/data",
"exec-root": "/data/docker/exec",
"log-opts": {
"max-size": "100m",
"max-file": "5"
}
}
EOF
启动 docker 服务
[root@k8s-harbor01 src]# systemctl daemon-reload && systemctl enable docker && systemctl restart docker
[root@k8s-harbor01 src]# systemctl status docker|grep Active
Active: active (running) since Wed 2020-08-12 13:41:07 CST; 28s ago
查看 Docker 版本号
[root@k8s-harbor01 src]# docker --version
Docker version 18.09.6, build 481bc77
三、安装Docker-Compose
下载docker-compose二进制执行文件 百度网盘下载地址：https://pan.baidu.com/s/1er0rM0vxEubYOLHx7LI62A 提取密码：eer9 [root@k8s-harbor01 ~]# cd /usr/local/src/ [root@k8s-harbor01 src]# curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose [root@k8s-harbor01 src]# cp docker-compose /usr/local/bin/ [root@k8s-harbor01 src]# chmod 755 /usr/local/bin/* 查看 docker-compose 版本号 [root@k8s-harbor01 ~]# docker-compose --version docker-compose version 1.26.0, build d4451659
四、部署Harbor镜像仓库
1）HTTPS证书自签
如果线上环境有已购买好的HTTPS证书可以直接拿过来用，如果没有，就在Harbor本机进行HTTPS证书自签。这里Harbor本机ip地址是172.16.60.238
生成CA证书私钥
[root@k8s-harbor01 ~]# openssl genrsa -out ca.key 4096
生成CA证书
[root@k8s-harbor01 ~]# openssl req -x509 -new -nodes -sha512 -days 3650 \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.60.238" \
-key ca.key \
-out ca.crt
生成服务器证书
1）生成私钥
[root@k8s-harbor01 ~]# openssl genrsa -out 172.16.60.238.key 4096
2）生成证书签名请求（CSR）
[root@k8s-harbor01 ~]# openssl req -sha512 -new \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.60.238" \
-key 172.16.60.238.key \
-out 172.16.60.238.csr
3）生成一个x509 v3扩展文件（两种方式根据情况二选一）
####################################################################################
第一种方式：域名
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=172.16.60.238
DNS.2=yourdomain
DNS.3=hostname
EOF
####################################################################################
第二种方式：IP
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = IP:172.16.60.238
EOF
####################################################################################
这里选择第二种的IP方式
[root@k8s-harbor01 ~]# cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = IP:172.16.60.238
EOF
4）使用该v3.ext文件为您的Harbor主机生成证书
[root@k8s-harbor01 ~]# openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in 172.16.60.238.csr \
-out 172.16.60.238.crt
2）提供证书给Harbor和Docker
1）将服务器证书和密钥复制到Harbor主机上的certficates文件夹中 根据自己实际环境需求创建Harbor的certficates文件夹 [root@k8s-harbor01 ~]# mkdir -p /data/cert/ [root@k8s-harbor01 ~]# cp 172.16.60.238.crt /data/cert/ [root@k8s-harbor01 ~]# cp 172.16.60.238.key /data/cert/ 2）转换 172.16.60.238.crt 为172.16.60.238.cert，供Docker使用。 Docker守护程序将.crt文件解释为CA证书，并将.cert文件解释为客户端证书。 [root@k8s-harbor01 ~]# openssl x509 -inform PEM -in 172.16.60.238.crt -out 172.16.60.238.cert 3）将服务器证书，密钥和CA文件复制到Harbor主机上的Docker certificate文件夹中。 记住必须首先创建适当的文件夹 [root@k8s-harbor01 ~]# mkdir -p /etc/docker/certs.d/172.16.60.238/ [root@k8s-harbor01 ~]# cp 172.16.60.238.cert /etc/docker/certs.d/172.16.60.238/ [root@k8s-harbor01 ~]# cp 172.16.60.238.key /etc/docker/certs.d/172.16.60.238/ [root@k8s-harbor01 ~]# cp ca.crt /etc/docker/certs.d/172.16.60.238/ 4）重新启动Docker [root@k8s-harbor01 ~]# systemctl restart docker [root@k8s-harbor01 ~]# systemctl status docker 5）将名为"ca.crt"的CA证书下载到本地电脑，然后安装证书。 这样就可以在本地电脑的浏览器里正常访问https地址的Harbor了（证书可被信任）
3）安装Harbor
到 Harbor的GitHub仓库的Release页面 , 下载最新的在线安装包
这里下载Harbor V2.0.2版本的安装包
[root@k8s-harbor01 ~]# cd /usr/local/src/
[root@k8s-harbor01 src]# wget https://github.com/goharbor/harbor/releases/download/v2.0.2/harbor-online-installer-v2.0.2.tgz
[root@k8s-harbor01 src]# tar -zvxf harbor-online-installer-v2.0.2.tgz
[root@k8s-harbor01 src]# mv harbor /opt/
修改harbor配置信息
[root@k8s-harbor01 src]# cd /opt/harbor/
[root@k8s-harbor01 harbor]# cp harbor.yml.tmpl harbor.yml
.........
........
hostname: 172.16.60.238
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
certificate: /data/cert/172.16.60.238.crt
private_key: /data/cert/172.16.60.238.key
........
........
harbor_admin_password: Harbor@123456
........
........
data_volume: /data
运行install.sh, 注意运行时加上--with-clair 选项，启动clair镜像扫描功能
[root@k8s-harbor01 harbor]# ./install.sh --with-clair
........
........
✔ ----Harbor has been installed and started successfully.----
出现上面的信息，说明Harbor已经安装完成了。
查看harbor启动情况。
docker-compose 命令必须要在harbor安装目录 (这里就是/opt/harbor)路径下才能执行。
[root@k8s-harbor01 harbor]# docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------------------
clair ./docker-entrypoint.sh Up (healthy) 6060/tcp, 6061/tcp
clair-adapter /home/clair-adapter/entryp ... Up (healthy) 8080/tcp
harbor-core /harbor/entrypoint.sh Up (healthy)
harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcp
harbor-jobservice /harbor/entrypoint.sh Up (healthy)
harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp
nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
redis redis-server /etc/redis.conf Up (healthy) 6379/tcp
registry /home/harbor/entrypoint.sh Up (healthy) 5000/tcp
registryctl /home/harbor/start.sh Up (healthy)
查看harbor镜像
[root@k8s-harbor01 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
goharbor/redis-photon v2.0.2 e547529bb6a1 3 weeks ago 72.3MB
goharbor/clair-adapter-photon v2.0.2 9ec8853dc3cb 3 weeks ago 62MB
goharbor/clair-photon v2.0.2 73885002dda7 3 weeks ago 171MB
goharbor/harbor-registryctl v2.0.2 9f8b7bb0f1ff 3 weeks ago 101MB
goharbor/registry-photon v2.0.2 eac8c5fc9ca8 3 weeks ago 83.6MB
goharbor/nginx-photon v2.0.2 eee4771b916c 3 weeks ago 43.6MB
goharbor/harbor-log v2.0.2 b2db762a6c3a 3 weeks ago 82.1MB
goharbor/harbor-jobservice v2.0.2 3960e027ccb9 3 weeks ago 164MB
goharbor/harbor-core v2.0.2 de2495b944cf 3 weeks ago 145MB
goharbor/harbor-portal v2.0.2 90088a0e64a9 3 weeks ago 52.5MB
goharbor/harbor-db v2.0.2 81e98a7af097 3 weeks ago 161MB
goharbor/prepare v2.0.2 7e804db05454 3 weeks ago 160MB
确保harbpr启动后的80和443端口都起来了
[root@k8s-harbor01 harbor]# lsof -i:80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 3095 root 4u IPv6 26027 0t0 TCP *:http (LISTEN)
[root@k8s-harbor01 harbor]# lsof -i:443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 3082 root 4u IPv6 26015 0t0 TCP *:https (LISTEN)
到这里就可以访问harbor了,访问地址为：https://172.16.60.238
用户名为admin，密码为配置文件中定义的"Harbor@123456"
查看clair镜像扫描器
Habor 服务启停
注意：如果harbor.yml配置修改了，要先执行"./prepare"命令进行配置载入，然后再重启harbor服务。
查看Habror docker-compose ps 启动Harbor docker-compose start 停止Harbor docker-compose stop 重启Harbor docker-compose restart 另外： Harbor还可以通过down和up命令去停止和启动， 只不过这种方式是删除、创建的关停和启动。 docker-compose down -v docker-compose up -d
五、客户端登录Harbor
在Habror客户端机器（如k8s的node节点、harbor节点）配置登录：
默认情况下，在客户端登录Habor是会报错的：
[root@k8s-node01 ~]# docker login 172.16.60.238
Authenticating with existing credentials...
Login did not succeed, error: Error response from daemon: Get https://172.16.60.238/v2/: x509: certificate signed by unknown authority
原因： 客户端登录Harbor，https证书不被信任。
解决办法：下面两种方法选其一
1）方法一
将Harbor服务器证书，密钥和CA文件复制到Harbor客户主机上的Docker certificate文件夹中
[root@k8s-node01 ~]# mkdir -p /etc/docker/certs.d/172.16.60.238/
[root@k8s-node01 ~]# cd /etc/docker/certs.d/172.16.60.238/
[root@k8s-node01 172.16.60.238]# rsync -e "ssh -p22" -avpgolr 172.16.60.238:/etc/docker/certs.d/172.16.60.238/* ./
[root@k8s-node01 172.16.60.238]# ll
total 12
-rw-r--r-- 1 root root 2053 Aug 19 14:34 172.16.60.238.cert
-rw-r--r-- 1 root root 3243 Aug 19 14:34 172.16.60.238.key
-rw-r--r-- 1 root root 2033 Aug 19 14:34 ca.crt
重启docker服务
[root@k8s-node01 172.16.60.238]# systemctl restart docker
[root@k8s-node01 172.16.60.238]# systemctl status docker
再次验证登录harbor
[root@k8s-node01 172.16.60.238]# docker login 172.16.60.238
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
2）方法二
配置docker服务的daemon.json文件，添加"insecure-registries"参数，表示忽略ssl证书认证。
[root@k8s-node01 ~]# vim /etc/docker/daemon.json
........
"insecure-registries": ["https://172.16.60.238"],
重启docker服务
[root@k8s-node01 ~]# systemctl restart docker
[root@k8s-node01 ~]# systemctl status docker
再次验证登录harbor
[root@k8s-node01 ~]# docker login 172.16.60.238
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
========================================================================
另外，注意客户端机器登录Harbor时，只要首次登录需要输入用户名和密码。
登录成功后的信息默认保存到/root/.docker/config.json文件里。
下次登录时就不用再输入harbor用户名和密码了，直接读取config.json文件内容
[root@k8s-node01 ~]# cat /root/.docker/config.json
{
"auths": {
"172.16.60.238": {
"auth": "YWRtaW46SGFyYm9yQDEyMzQ1Ng=="
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.09.6 (linux)"
}
[root@k8s-node01 ~]# docker login 172.16.60.238
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
六、Harbor镜像扫描
选中镜像，进行漏洞扫描
如果扫描出漏洞，在漏洞报告了会告知漏洞当前版本和修复版本，按照修复版本修复即可。
修复方法：
可以依据当前基础镜像做Dockerfile，使用"yum update -y 漏洞所属软件名" 进行升级操作，然后再重新做一个基础镜像。
1) 编译Dockerfile 升级原来centos7.7基础镜像里报出来漏洞的软件 [root@k8s-harbor01 ~]# cat Dockerfile FROM 172.16.60.238/kevin/centos7.7:latest RUN yum update -y sqlite \ && yum update -y nss-util \ && yum update -y nss-sysinit \ && yum update -y dbus-libs \ && yum update -y bind-license \ && yum update -y nss \ && yum update -y nss-softokn \ && yum update -y dbus \ && yum update -y nss-softokn-freebl \ && yum update -y nss-tools \ && yum update -y bash \ && yum update -y python-libs \ && yum update -y python \ && yum update -y bind-license \ && yum update -y expat \ && yum update -y libxml2-python \ && yum update -y libxml2 \ && yum update -y shared-mime-info \ && yum update -y libcurl \ && yum update -y file-libs \ && yum update -y curl 2）制作新的基础镜像 [root@k8s-harbor01 ~]# docker build -t 172.16.60.238/kevin/centos7.7:updatev1 . 3）上传到Harbor仓库 [root@k8s-harbor01 ~]# docker push 172.16.60.238/kevin/centos7.7:updatev1
将修复好漏洞的新基础镜像上传到Harbor仓库，再扫描新镜像，发现漏洞已修复。
*************** 当你发现自己的才华撑不起野心时，就请安静下来学习吧！***************
