Harbor私有镜像仓库（上）

一，Harbor简介

• Vmware的开源项目https://github.com/VMware/harbor

• Harbor可帮助用户迅速搭建企业级的注册服务,它提供了管理图形界面,基于角色的访问控制(Role Based Access Control),镜像远程复制(同步),AD/LDAP集成,以及审计日志等企业用户需求的功能,同时还原支持中文,深受中国用户的喜爱

• 该项目自推出一来,在GitHub获得了超过3300多个star和900多个forks.

 

1.1 基于角色的访问控制

用户与Docker镜像仓库通过"项目"进行组织管理,一个用户可以对多个镜像仓库在统一命名空间(projec)里有不同的权限

 

1.2 图形化用户界面

用户可以通过浏览器来浏览,检索当前Docker镜像仓库,管理项目和命名空间

 

1.3 审计管理

所有针对镜像仓库的错做都可以被记录追溯,用于审计管理

 

1.4 国际化

基于英文与中文语言进行了本地化.可以增加更多的语言支持.

 

1.5 RESTful API:

提供给管理员对于Harbor更多的操控,使得与其他管理软件集成变得更容易.

 

1.6 LDAP认证

 

1.7 镜像复制

基于策略的Docker镜像复制功能,可在不同的数据中心,不同的运行环境之间同步镜像,并提供友好的管理界面,大大简化了实际运维中的镜像管理工作.

 

1.8 与Clair集成

与Clair集成,添加漏洞扫描功能,CLair是coreos开源的容器漏洞扫描工具,在容器逐渐普及的今天,容器镜像安全问题日益严重,Clair是目前少数的开源安全扫描工具.

 

1.9 Notary签名工具

Notary是Docker镜像的签名工具,用来保证镜像在pull,push和传输工程中的一致性和完整性,避免中间人攻击,避免非法的镜像更新和运行.

 

二，为Harbor签发域名证书

openssl是目前最流行的SSl密码库工具,提供了一个通用,功能完备的工具套件,用以支持SSL/TLS协议的实现.官网:https://www.openssl.org/source/

 

环境准备

官方文档: https://github.com/vmware/harbor/blob/master/docs/configure_https.md

主机名	IP	用途	最小资源配比	最佳资源配比
harbor-master	192.168.200.70	harbor私有镜像仓库	2CPU-4GBMEM	4CPU-8GBMEM
harbor-slave	192.168.200.109	harbor从库	2CPU-4GBMEM	4CPU-8GBMEM

hostname -I

uname -r

cat /etc/redhat-release

 

创建自己的CA证书

mkdir -p /data/ssl

cd /data/ssl

which openssl

openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crt

 

1. Generating a 4096 bit RSA private key
2. ............................................................................................++
3. ......++
4. writing new private key to 'ca.key'
5. -----
6. You are about to be asked to enter information that will be incorporated
7. into your certificate request.
8. What you are about to enter is what is called a Distinguished Name or a DN.
9. There are quite a few fields but you can leave some blank
10. For some fields there will be a default value,
11. If you enter '.', the field will be left blank.
12. -----
13. Country Name (2 letter code) [XX]:CN #国家
14. State or Province Name (full name) []:Beijing #地区
15. Locality Name (eg, city) [Default City]:Beijing #城市
16. Organization Name (eg, company) [Default Company Ltd]:yunjisuan #公司名称
17. Organizational Unit Name (eg, section) []:yunjisuan
18. Common Name (eg, your name or your server's hostname) []:www.yunjisuan.com
19. Email Address []: #回车

 

生成证书签名请求

openssl req -newkey rsa:4096 -nodes -sha256 -keyout www.yunjisuan.com.key -out www.yunjisuan.com.csr

 

1. Generating a 4096 bit RSA private key
2. ...........................................................................................................................................................................................................++
3. ..................................................................................................................................................................................++
4. writing new private key to 'www.yunjisuan.com.key'
5. -----
6. You are about to be asked to enter information that will be incorporated
7. into your certificate request.
8. What you are about to enter is what is called a Distinguished Name or a DN.
9. There are quite a few fields but you can leave some blank
10. For some fields there will be a default value,
11. If you enter '.', the field will be left blank.
12. -----
13. Country Name (2 letter code) [XX]:CN
14. State or Province Name (full name) []:Beijing
15. Locality Name (eg, city) [Default City]:Beijing
16. Organization Name (eg, company) [Default Company Ltd]:yunjisuan
17. Organizational Unit Name (eg, section) []:yunjisuan
18. Common Name (eg, your name or your server's hostname) []:www.yunjisuan.com
19. Email Address []:
21. Please enter the following 'extra' attributes
22. to be sent with your certificate request
23. A challenge password []:
24. An optional company name []:
25. [root@harbor-master ssl]#

 

生成注册表主机的证书

openssl x509 -req -days 365 -in www.yunjisuan.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out www.yunjisuan.com.crt

 

1. Signature ok
2. subject=/C=CN/ST=Beijing/L=Beijing/O=yunjisuan/OU=yunjisuan/CN=www.yunjisuan.com
3. Getting CA Private Key

 

查看证书情况

ls

 

1. ca.crt ca.srl www.yunjisuan.com.csr
2. ca.key www.yunjisuan.com.crt www.yunjisuan.com.key

 

三，信任自签发的域名证书

由于CA证书是我们自己签发的Linux操作系统是不信任的,因此我们需要把证书加入到系统的信任证书里

 

将自签ca证书添加到系统信任

pwd

cp www.yunjisuan.com.crt /etc/pki/ca-trust/source/anchors/ --->复制到这个下面是让本机先信任这个证书

 

让系统ca信任设置立即生效

update-ca-trust enable

update-ca-trust extract

 

四，Harbor1.4版本配置与安装

 

4.1 安装docker-ce社区版(master和slave都安)

 

安装依赖包

yum -y install yum-utils device-mapper-persistent-data lvm2

 

添加docker的CE版本的yum源配置文件

curl https://download.docker.com/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo

ll /etc/yum.repos.d/docker-ce.repo

 

安装CE版本的docker

yum -y install docker-ce

systemctl start docker #启动docker

systemctl enable docker #添加开机启动

docker version #查看docker版本

which docker

 

看一眼子防火墙的状态(提前安装过docker得话需要重启docker)

sestatus -->防火墙什么的都关上! 过程略

 

4.2 下载并安装harbor私有仓库

 

harbor所有包

链接：https://pan.baidu.com/s/1MSfSWLBsUrvXv3USv233Cg 
提取码：zmom

 

创建harbor的证书目录,并复制

mkdir -p /etc/ssl/harbor

cp /data/ssl/www.yunjisuan.com.key /etc/ssl/harbor/

cp /data/ssl/www.yunjisuan.com.crt /etc/ssl/harbor/

ll /etc/ssl/harbor/

 

创建harbor下载目录并下载harbor-offline-installer-v1.5.0.tgz

mkdir -p /data/install

cd /data/install

pwd

which wget --->这里没有wget下载，上文有下载链接

wget http://harbor.orientsoft.cn/harbor-v1.5.0/harbor-offline-installer-v1.5.0.tgz

ls

tar xf harbor-offline-installer-v1.5.0.tgz

ls

cd harbor

ll

cp harbor.cfg{,.bak}

 

修改harbor.cfg配置文件

cat -n harbor.cfg | sed -n '7p;11p;23p;24p;68p'

 

1. 7 hostname = reg.mydomain.com #要修改成我们证书的域名
2. 11 ui_url_protocol = http #启用加密传输协议https
3. 23 ssl_cert = /data/cert/server.crt #证书的位置
4. 24 ssl_cert_key = /data/cert/server.key #证书密钥位置
5. 68 harbor_admin_password = Harbor12345 #默认管理员及密码

 

修改成如下配置

vim harbor.cfg

cat -n harbor.cfg | sed -n '7p;11p;23p;24p;68p'

 

1. 7 hostname = www.yunjisuan.com
2. 11 ui_url_protocol = https
3. 23 ssl_cert = /etc/ssl/harbor/www.yunjisuan.com.crt
4. 24 ssl_cert_key = /etc/ssl/harbor/www.yunjisuan.com.key
5. 68 harbor_admin_password = Harbor12345

 

安装命令docker-compose(需要1.21版本)

curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-(uname -m) -o /usr/local/bin/docker-compose --->上文有下载链接

cd /usr/local/bin/

ll /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

which docker-compose

docker-compose --version

 

安装harbor私有镜像仓库

cd /data/install/harbor

./install.sh --with-notary --with-clair

 

1. #--with-notary启用镜像签名; --with-clair启用漏洞扫描

 

查看harbor启动的镜像

docker ps -a

 

1. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2. be5fcf3a1930 vmware/harbor-jobservice:v1.5.0 "/harbor/start.sh" About a minute ago Up 50 seconds harbor-jobservice
3. ad2f90bb84c3 vmware/nginx-photon:v1.5.0 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp nginx
4. d782fcc8ce01 vmware/notary-server-photon:v0.5.1-v1.5.0 "/bin/server-start.sh" About a minute ago Up About a minute notary-server
5. a8edc1ccca5f vmware/clair-photon:v2.0.1-v1.5.0 "/docker-entrypoint.…" About a minute ago Up 46 seconds (healthy) 6060-6061/tcp clair
6. edb4a9a4c11e vmware/harbor-ui:v1.5.0 "/harbor/start.sh" About a minute ago Up About a minute (healthy) harbor-ui
7. 2db41616f8e3 vmware/notary-signer-photon:v0.5.1-v1.5.0 "/bin/signer-start.sh" About a minute ago Up About a minute notary-signer
8. fa7b3e5f6f24 vmware/postgresql-photon:v1.5.0 "/entrypoint.sh post…" About a minute ago Up About a minute (healthy) 5432/tcp clair-db
9. 1401200b682e vmware/harbor-adminserver:v1.5.0 "/harbor/start.sh" About a minute ago Up About a minute (healthy) harbor-adminserver
10. 0b37da8995e3 vmware/redis-photon:v1.5.0 "docker-entrypoint.s…" About a minute ago Up About a minute 6379/tcp redis
11. aab6d323c577 vmware/harbor-db:v1.5.0 "/usr/local/bin/dock…" About a minute ago Up About a minute (healthy) 3306/tcp harbor-db
12. bfbdb8a623e4 vmware/registry-photon:v2.6.2-v1.5.0 "/entrypoint.sh serv…" About a minute ago Up About a minute (healthy) 5000/tcp registry
13. f29345823b5a vmware/mariadb-photon:v1.5.0 "/usr/local/bin/dock…" About a minute ago Up About a minute 3306/tcp notary-db
14. 3756f90a2271 vmware/harbor-log:v1.5.0 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log

 

4.3 通过浏览器进行访问测试(需要火狐浏览器才能跳过)

https://192.168.200.70

输入账号密码登录:admin harbor12345

---

 

4.4 做一些安全设置

---

---

 

4.5 本地docker上传测试

 

先做映射

vim /etc/hosts

cat /etc/hosts

 

用docker登录(只能域名访问，IP是加密的不认识)

docker login www.yunjisuan.com

admin Harbor12345

 

也可以面交互登录

docker login -uadmin -pHarbor12345 www.yunjisuan.com

 

先得把镜像改名加路径(就是相当于添加)

docker pull centos

docker images

docker tag centos:latest www.yunjisuan.com/library/centos:v1

docker images

docker push www.yunjisuan.com/library/centos:v1

---

 

刷新页面，然后查看就有了

---

---

---

 

在slave服务器上访问并下载私有镜像仓库里刚上传的那个镜像

 

先映射

echo "192.168.200.70 www.yunjisuan.com" >> /etc/hosts --->映射的是Harbor的那台

cat /etc/hosts

 

然后把主的那个证书复制过去并立即生效

cd /data/ssl

ls

scp www.yunjisuan.com.crt 192.168.200.109:/etc/pki/ca-trust/source/anchors/

---

update-ca-trust enable

update-ca-trust extract

 

重启docker否则没有用

systemctl restart docker

docker login -uadmin -pHarbor12345 www.yunjisuan.com

 

下载私有镜像仓库里的那个刚刚上传的镜像文件

docker pull www.yunjisuan.com/library/centos:v1

docker images

 

五，镜像管理与安全:漏洞扫描和镜像签名

 

5.1 添加docker国内公有镜像源

vim /etc/docker/daemon.json

cat /etc/docker/daemon.json

 

1. {
2. "registry-mirrors":[ "https://registry.docker-cn.com" ]
3. }

systemctl daemon-reload

systemctl restart docker

 

5.2 重新启动Harbor私有镜像仓库

 

让harbor修改过的配置立刻生效

 

1. [root@harbor-master harbor]# pwd
2. /data/install/harbor
3. [root@harbor-master harbor]# ./prepare

 

清理所有harbor容器进程

 

1. [root@harbor-master harbor]# docker-compose down

 

后台重新启动所有harbor容器进程

 

1. [root@harbor-master harbor]# docker-compose up -d
3. **docker-compose命令必须在/data/install/harbor目录下使用，否则找不到**

 

5.3 FAQ:问题解答

 

5.3.1 windows10最新版本默认拒绝非认证的域名证书

如果启动harbor采用https加密证书的方式,最新版本window10浏览器访问的化,默认会直接说"站点不安全,拒绝连接" 
那么我们可以采用非https的方式启动harbor

 

1. [root@harbor-master harbor]# sed -n '11p' /data/install/harbor/harbor.cfg

但是我们要是采用非https加密方式启动harbor的化,最新版本的docker是登录不了的,这是因为新版本docker默认是以https方式登录harbor

为了解决登录问题,我们需要在/etc/docker/下创建一个daemon.json名字的文件,加入http方式登录的harbor域名

 

1. [root@harbor-slave ~]# cat /etc/docker/daemon.json
2. {
3. "insecure-registries":[ "www.yunjisuan.com" ]
4. }
5. [root@harbor-slave ~]# systemctl restart docker #需要重启

然后就能登录成功了

 

六，harbor镜像的复制与同步

harbor私有仓库的主从复制,类似于MySQL,属于1对多的复制

主机名	IP	用途	最小资源配比	最佳资源配比
harbor-master	192.168.200.70	harbor私有镜像仓库	2CPU-4GBMEM	4CPU-8GBMEM
harbor-slave	192.168.200.109	harbor从库	2CPU-4GBMEM	4CPU-8GBMEM

 

6.1 部署Harbor-Slave

请安装一个harbor私有仓库作为harbor的从库,域名为www2.yunjisuan.com

请看上文master部署步骤

 

主找从所以把证书给主一份

cd /data/ssl

ls

scp www2.yunjisuan.com.crt 192.168.200.70:/etc/pki/ca-trust/source/anchors/

 

然后主得立即生效并重启dcoker.然后harbor还得重启启动进程

cd /data/install/harbor

update-ca-trust enable

update-ca-trust extract

systemctl restart docker

docker-compose down --->必须在harbor目录里（/data/install/harbor）

./prepare

./install.sh --with-clair

---

 

在主上做映射

vim /etc/hosts

cat /etc/hosts

 

查看网页主从情况

---

 

6.2 搭建LDNS域名解析服务器

主机名	IP	用途	最小资源配比	最佳资源配比
harbor-master	192.168.200.70	harbor私有镜像仓库	2CPU-4GBMEM	4CPU-8GBMEM
harbor-slave	192.168.200.109	harbor从库	2CPU-4GBMEM	4CPU-8GBMEM
LDNS	192.168.200.110	本地DNS

yum -y install bind bind-chroot bind-utils

cd /etc/

cp named.conf{,.bak}

 

把配置文件修改成如下:

vim named.conf

cat named.conf

 

1. options {
2. listen-on port 53 { 192.168.200.110; }; #监听本机IP地址
3. // listen-on-v6 port 53 { ::1; }; #注释掉
4. directory "/var/named";
5. dump-file "/var/named/data/cache_dump.db";
6. statistics-file "/var/named/data/named_stats.txt";
7. memstatistics-file "/var/named/data/named_mem_stats.txt";
8. allow-query { any; }; #any允许任意客户端
9. forwarders { 192.168.200.2; }; #加一段话,网关的位置
11. recursion yes;
13. dnssec-enable no; #改成no不验证
14. dnssec-validation no; #改成no不验证
16. /* Path to ISC DLV key */
17. bindkeys-file "/etc/named.iscdlv.key";
19. managed-keys-directory "/var/named/dynamic";
21. pid-file "/run/named/named.pid";
22. session-keyfile "/run/named/session.key";
23. };
25. logging {
26. channel default_debug {
27. file "data/named.run";
28. severity dynamic;
29. };
30. };
32. zone "." IN {
33. type hint;
34. file "named.ca";
35. };
37. zone "yunjisuan.com" IN { #加一个维护域名
38. type master; #类型
39. file "yunjisuan.com.zone"; #域名文件
40. }; #内容
41. include "/etc/named.rfc1912.zones";
42. include "/etc/named.root.key";

---

 

检查配置文件是否有错

named-checkconf /etc/named.conf

 

创建正向解析文件

cd /var/named/

ls

cp -p named.empty yunjisuan.com.zone

 

把yunjisuan.com.zone修改成如下

vim yunjisuan.com.zone

cat yunjisuan.com.zone

 

1. $TTL 1D
2. @ IN SOA yunjisuan.com. root.ns1.yunjisuan.com. (
3. 0 ; serial
4. 1D ; refresh
5. 1H ; retry
6. 1W ; expire
7. 3H ) ; minimum
8. NS ns1.yunjisuan.com.
9. ns1 A 192.168.200.110 #LDNS的IP地址,也就是自己
10. www A 192.168.200.70 #harbor主的ip地址
11. www2 A 192.168.200.109 #harbor从的ip地址

 

测试正向解析文件是否有错

named-checkzone yunjisuan.com yunjisuan.com.zone

 

启动域名解析服务

systemctl start named

ss -antup | grep named

 

改一下DNS改成自己本机

vim /etc/resolv.conf

cat /etc/resolv.conf

 

1. #nameserver 192.168.200.2
2. nameserver 192.168.200.110

 

用nslookup域名解析命令测试一下

nslookup www.yunjisuan.com

nslookup www2.yunjisuan.com

nslookup www.baidu.com

 

6.3 建立主从复制

然后开始建立主从复制

---

目前测试连接失败,没关系,我们先点击确定把这个保存一下

 

先清空一下日志

cd /var/log/harbor/

ls

> ui.log

 

然后在点击一下测试连接之后查看日志

cat ui.log

因此发现,Harbor的主从复制是不找本地的hosts文件的,映射了也没有用. 它直接找DNS

 

把主的DNS改成LDNS服务器的IP

vim /etc/resolv.conf

cat /etc/resolv.conf

 

1. nameserver 192.168.200.110

 

然后得重启harbor否则也不行 ，因为它读到缓存去了。

cd /data/install/harbor

docker-compose down --->需要在harbor目录下(/data/install/harbor)

./prepare

./install.sh --with-clair

---

 

刷新网页，在测试连接

 

6.4启用主从复制

---

---

 

在看从的，就把镜像复制过去了 一开始没有任何镜像的

 

在上传一个镜像 看看效果

docker pull hello-world

docker images

docker tag hello-world:latest www.yunjisuan.com/library/hello-world:v1

REPOSITORY TAG IMAGE ID CREATED SIZE 
centos latest 75835a67d134 8 weeks ago 200MB 
www.yunjisuan.com/library/centos v1 75835a67d134 8 weeks ago 200MB 
hello-world latest 4ab4c602aa5e 2 months ago 1.84kB 
www.yunjisuan.com/library/hello-world v1 4ab4c602aa5e 2 months ago 1.84kB

 

上传

[root@wbq-harbor-master harbor]# docker push www.yunjisuan.com/library/hello-world:v1 
The push refers to repository [www.yunjisuan.com/library/hello-world] 
428c97da766c: Pushed 
v1: digest: sha256:1a6fd470b9ce10849be79e99529a88371dff60c60aab424c077007f6979b4812 size: 524

 

主的已经是2个镜像了

---

 

在看从的也变成2个镜像了

这样主从复制就搭建完成了