Linux-docker仓库-分布式Harbor

Harbor

Harbor是一个用于存储和分发Docker镜像的企业级Registry 服务器，由VMware开源，其通过添加一些企业必须的功能特性，例如安全、标识和管理等，扩展了开源Docker Distribution。作为一个企业级私有Registry服务器，Harbor提供了更好的性能和安全，提升用户使用Registry构建和运行环境传输镜像的效率。Hardor支持安装多个Registry节点的镜像资源复制，镜像全部保存在私有Registry中，确保数据和知识产权在公司内部网络中管控，另外，Harbor也提供了高级的安全特性，诸如用户管理，访问控制和活动审计等。

Harbor功能官方介绍：

基于角色的访问控制：角色与Docker镜像仓库通过“项目”进行组织管理，一个用户可以对多个镜像仓库在同一命令空间(project)里有不同的权限。镜像复制：镜像可以在多个Registry实例中复制(同步)。尤其适合于负载均衡，高可用，混合云和多云的场景。
图形化用户里面：用户可以通过浏览器来浏览，检索当前Docker镜像仓库，管理项目和命名空间。
AD/LDAP支：Harbor可以继承企业内部已有的AD/LADAP，用于鉴权认证管理。审计管理：所有针对镜像仓库的操作都可以被记录追溯，用于审计管理。
国际化：已拥有英文，中文，德文，日文和俄文的本地化版本。更多的语言将会添加进来。
RESTful API-RESTful API：提供给管理对于Harbor更多的操控，使得与其他管理软件继承变得更容易。
简单部署：提供在线和离线两种安装工具，也可以安装到vSphere 平台(OVA方式)虚拟设备。

nginx：harbor的一个反向代理组件，代理registry、ui、token等服务。这个代理会转发harbor web 和docker client 的各种请求到后端服务上。

harbor-adminserver：harbor系统管理接口，可以修改系统配置以及获取系统信息

harbor-db：存储项目的元数据，用户、规则、复制策略等信息。

harbor-jobservice：harbor里面主要是为了镜像仓库之间同步使用的。

harbor-log：收集其他harbor的日志信息。

harbor-ui：一个用户界面模块，用来管理registry。

registry：存储docker images的服务，并且提供pull/push服务。

redis；存储缓存信息

webhook：当registry中的image状态发生变化的时候去记录更新日志，复制等操作。

token service：在docker client 进行pull/push的时候负责token的发放。

hardor部署

部署环境

Ubuntu 10.0.0.100 docker hardor

Ubuntu 10.0.0.101 docker hardor

解压harbor安装包

[14:07:36 root@ubuntu-lyj ~]#ls    #准备harbor安装包
docker-compose-linux-x86_64  docker-in.sh  harbor-offline-installer-v2.0.0.tgz      
[14:35:25 root@ubuntu-lyj ~]#tar xvf harbor-offline-installer-v2.0.0.tgz -C /usr/local/src/  #解压缩安装包
harbor/harbor.v2.0.0.tar.gz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl
[14:39:34 root@ubuntu-lyj ~]#cd /usr/local/src/
[14:39:48 root@ubuntu-lyj /usr/local/src]#ls
harbor
[14:39:49 root@ubuntu-lyj /usr/local/src]#ln -sv /usr/local/src/harbor/ /usr/local/         #创建软连接
'/usr/local/harbor' -> '/usr/local/src/harbor/'
[14:40:06 root@ubuntu-lyj /usr/local/src]#cd /usr/local/harbor
[14:40:19 root@ubuntu-lyj /usr/local/harbor]#ls
common.sh  harbor.v2.0.0.tar.gz  harbor.yml.tmpl  install.sh  LICENSE  prepare
[14:40:21 root@ubuntu-lyj /usr/local/harbor]#apt install python-pip -y                      #安装python

修改harbor.yml.tmpl ，并改名为harbor.yml

hostname: hardor.jing.com        #要域名解析，访问端host文件和本机host文件添加域名解析

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
#https:          #注释https相关内容
  # https port for harbor, default is 443
 # port: 443
  # The path of cert and key files for nginx
 # certificate: /your/certificate/path
 # private_key: /your/private/key/path
......
harbor_admin_password: 123456  #hardor WEB界面登录密码

复制更名为hardor.yml

[15:02:12 root@ubuntu-lyj /usr/local/harbor]#cp harbor.yml.tmpl harbor.yml
[15:03:12 root@ubuntu-lyj /usr/local/harbor]#ls
common.sh  harbor.v2.0.0.tar.gz  harbor.yml  harbor.yml.tmpl  install.sh  LICENSE  prepare

安装docker-compose

在线安装速度慢，而且网络访问的官网很不稳定

[15:16:17 root@ubuntu-lyj /usr/local/harbor]#pip install docker-compose

到官网下载docker-compose文件，我下载了docker-compose1.26.2

上传的Linux系统

[15:50:05 root@ubuntu-lyj ~]#ls
docker-compose-Linux-x86_64  harbor-offline-installer-v2.0.0.tgz
[15:48:07 root@ubuntu-lyj ~]#chmod a+x docker-compose-Linux-x86_64                   #加执行全新啊
[15:49:04 root@ubuntu-lyj ~]#cp docker-compose-Linux-x86_64 /usr/bin/docker-compose  #拷贝到/usr/bin/下命名为docker-compose

hardor安装：执行install.sh安装

[15:29:56 root@ubuntu-lyj /usr/local/harbor]#./install.sh

查看镜像

harbor构建的镜像

[19:14:53 root@ubuntu-lyj ~]#docker images
REPOSITORY                      TAG        IMAGE ID       CREATED         SIZE
tomcat-web                      app2       455fc2e0e6ea   9 hours ago     1.04GB
tomcat-web                      app1       da199854bb49   9 hours ago     1.04GB
tomcat-base                     8.5.81     6375807c58e9   11 hours ago    1.02GB
jdk-centos-base                 8u291      17d323ce7653   21 hours ago    1.01GB
centos-base                     7.8.2003   3a7c6d1c0eef   25 hours ago    651MB
martonyang/centos7.8.2003       latest     0c0f2dcf7afd   12 months ago   268MB
goharbor/chartmuseum-photon     v2.0.0     4db8d6aa63e9   2 years ago     127MB
goharbor/redis-photon           v2.0.0     c89ea2e53cc0   2 years ago     72.2MB
goharbor/trivy-adapter-photon   v2.0.0     6122c52b7e48   2 years ago     103MB
goharbor/clair-adapter-photon   v2.0.0     dd2210cb7f53   2 years ago     62MB
goharbor/clair-photon           v2.0.0     f7c7fcc52278   2 years ago     171MB
goharbor/notary-server-photon   v2.0.0     983ac10ed8be   2 years ago     143MB
goharbor/notary-signer-photon   v2.0.0     bee1b6d75e0d   2 years ago     140MB
goharbor/harbor-registryctl     v2.0.0     c53c32d58d04   2 years ago     102MB
goharbor/registry-photon        v2.0.0     afdc1b7ada36   2 years ago     84.5MB
goharbor/nginx-photon           v2.0.0     17892f03e56c   2 years ago     43.6MB
goharbor/harbor-log             v2.0.0     5f8ff08e795c   2 years ago     82MB
goharbor/harbor-jobservice      v2.0.0     c68a2495bf55   2 years ago     116MB
goharbor/harbor-core            v2.0.0     3aa3af64baf8   2 years ago     138MB
goharbor/harbor-portal          v2.0.0     e0b1d3c894c4   2 years ago     52.4MB
goharbor/harbor-db              v2.0.0     5c76f0296cec   2 years ago     154MB
goharbor/prepare                v2.0.0     7266d49995ed   2 years ago     158MB

web访问harbor管理界面

登录成功后的界面

上传镜像到hardor仓库

10.0.0.100 机器上传镜像

*编辑docker.service文件添加域名信任 --insecure-registry hardor.jing.com

[19:05:25 root@ubuntu-lyj /]#vim /lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry hardor.jing.com --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

重启docker服务

[18:40:47 root@ubuntu-lyj ~]#systemctl daemon-reload  #修改了service文件必须重新加载配置文件
[18:40:48 root@ubuntu-lyj ~]#systemctl daemon-reload 
[18:40:49 root@ubuntu-lyj ~]#systemctl restart docker

重启harbor服务

[18:41:36 root@ubuntu-lyj /usr/local/harbor]#docker-compose stop
/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.hazmat.backends import default_backend
Stopping nginx       ... done
Stopping harbor-core ... done
Stopping registry    ... done
Stopping harbor-log  ... done
[18:43:26 root@ubuntu-lyj /usr/local/harbor]#docker-compose start
/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.hazmat.backends import default_backend
Starting log         ... done
Starting postgresql  ... done
Starting redis       ... done
Starting portal      ... done
Starting registry    ... done
Starting core        ... done
Starting jobservice  ... done
Starting proxy       ... done
Starting registryctl ... done

*验证能否登录hardor

必须验证后才能上传镜像

[18:43:52 root@ubuntu-lyj ~]#docker login hardor.jing.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

hardor web端

新建一个项目目录，存储容量 -1 标识不限制

上传镜像

给镜像做tag，并上传

hardor访问域名+项目+镜像名称：tag

hardor.jing.com/n65/centos-base:7.8.2003

[19:14:58 root@ubuntu-lyj ~]#docker tag centos-base:7.8.2003  hardor.jing.com/n65/centos-base:7.8.2003
[19:18:35 root@ubuntu-lyj ~]#docker push hardor.jing.com/n65/centos-base:7.8.2003
The push refers to repository [hardor.jing.com/n65/centos-base]
8d9cb3505aea: Pushed 
816ac87d2f0f: Pushing [=====================>                             ]  161.3MB/382MB
c0cd79243356: Pushed 
fb82b029bea0: Pushing [===========================>                       ]  110.5MB/203.3MB

上传成功

[19:18:35 root@ubuntu-lyj ~]#docker push hardor.jing.com/n65/centos-base:7.8.2003
The push refers to repository [hardor.jing.com/n65/centos-base]
8d9cb3505aea: Pushed 
816ac87d2f0f: Pushed 
c0cd79243356: Pushed 
fb82b029bea0: Pushed 
7.8.2003: digest: sha256:a6bb0d82a47af8cc6e6b09fec575d7a2e94d6813e107ea6791d39756e701a289 size: 1162

10.0.0.101 下载镜像

验证hardor服务器现在镜像并启动容器

更改docker.service配置文件

添加hardor仓库域名信任

[19:51:41 root@ubuntu-lyj ~]#cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry hardor.jing.com
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

重启docker服务

[19:50:44 root@ubuntu-lyj ~]#systemctl daemon-reload 
[19:51:05 root@ubuntu-lyj ~]#systemctl daemon-reload 
[19:51:07 root@ubuntu-lyj ~]#systemctl restart docker

添加域名解析

[19:52:14 root@ubuntu-lyj ~]#cat  /etc/hosts
127.0.0.1    localhost
127.0.1.1    ubuntu-lyj

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.0.0.100 hardor.jing.com

web客户端提取命令

镜像下载

[19:52:18 root@ubuntu-lyj ~]#docker pull hardor.jing.com/n65/tomcat-web@sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e
hardor.jing.com/n65/tomcat-web@sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e: Pulling from n65/tomcat-web
9b4ebb48de8d: Extracting [=========>                                         ]  14.48MB/75.84MB
0483f20596d4: Download complete 
3f8a43825096: Download complete 
ea68556c87dc: Download complete 
8b923d9bf3c0: Downloading [============================>                      ]  82.29MB/146.6MB
308c3b8ba259: Download complete 
adb9d1d96836: Download complete 
dab736b8cd99: Download complete 
a2abdb0eae49: Download complete 
b17bc2f7802a: Download complete 
59667dd5891f: Download complete 
4d7130eb6dff: Download complete 
7ce7f7ca6b8a: Download complete 
973ed29f7bb8: Download complete

下载完成

[19:52:18 root@ubuntu-lyj ~]#docker pull hardor.jing.com/n65/tomcat-web@sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e
hardor.jing.com/n65/tomcat-web@sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e: Pulling from n65/tomcat-web
9b4ebb48de8d: Pull complete 
0483f20596d4: Pull complete 
3f8a43825096: Pull complete 
ea68556c87dc: Pull complete 
8b923d9bf3c0: Pull complete 
308c3b8ba259: Pull complete 
adb9d1d96836: Pull complete 
dab736b8cd99: Pull complete 
a2abdb0eae49: Pull complete 
b17bc2f7802a: Pull complete 
59667dd5891f: Pull complete 
4d7130eb6dff: Pull complete 
7ce7f7ca6b8a: Pull complete 
973ed29f7bb8: Pull complete 
Digest: sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e
Status: Downloaded newer image for hardor.jing.com/n65/tomcat-web@sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e
hardor.jing.com/n65/tomcat-web@sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e

启动镜像测试

[19:57:23 root@ubuntu-lyj ~]#docker run -it --rm -p8081:8080 hardor.jing.com/n65/tomcat-web@sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e
Using CATALINA_BASE:   /apps/tomcat
Using CATALINA_HOME:   /apps/tomcat
Using CATALINA_TMPDIR: /apps/tomcat/temp
Using JRE_HOME:        /usr/local/jdk
Using CLASSPATH:       /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:   
Tomcat started.
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.2    c99dba79e98c

web端测试

多个hardor服务器同步数据

10.0.0.100 harbor1

10.0.0.101 harbor2

10.0.0.100

10.0.0.101

两台服务器设置镜像同步

[22:26:36 root@ubuntu-lyj ~]#vim /lib/systemd/system/docker.service 

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry hardor.jing.com  --insecure-registry hardor.jing.org --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

重启docker服务

[16:12:30 root@ubuntu-lyj /usr/local/harbor]#systemctl daemon-reload 
[16:12:42 root@ubuntu-lyj /usr/local/harbor]#systemctl restart docker
[16:14:22 root@ubuntu-lyj /usr/local/harbor]#ps aux|grep "docker"
root      24161  0.7  4.3 1954768 88336 ?       Ssl  16:13   0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry 10.0.0.100 --insecure-registry 10.0.0.101
root      24448  0.0  0.1 1226668 3604 ?        Sl   16:13   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 1514 -container-ip 172.18.0.6 -container-port 10514
root      26478  0.0  0.1 1078948 3684 ?        Sl   16:14   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.3 -container-port 8080
root      26484  0.0  0.1 1152936 3776 ?        Sl   16:14   0:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 80 -container-ip 172.18.0.3 -container-port 8080
root      26621  0.0  0.0  14436  1092 pts/0    R+   16:14   0:00 grep --color=auto docker

重启hardor服务

[16:15:36 root@ubuntu-lyj /usr/local/harbor]#docker-compose stop
/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.hazmat.backends import default_backend
Stopping nginx       ... done
Stopping harbor-core ... done
Stopping registry    ... done
Stopping harbor-log  ... done
[16:17:26 root@ubuntu-lyj /usr/local/harbor]#docker-compose start
/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.hazmat.backends import default_backend
Starting log         ... done
Starting postgresql  ... done
Starting redis       ... done
Starting portal      ... done
Starting registry    ... done
Starting core        ... done
Starting jobservice  ... done
Starting proxy       ... done
Starting registryctl ... done

shell端登录验证

[16:21:29 root@ubuntu-lyj /usr/local/harbor]#docker login 10.0.0.100
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[16:19:15 root@ubuntu-lyj /usr/local/harbor]#docker login 10.0.0.101
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

hardor Web端设置仓库同步

10.0.0.100-Weg端配置

仓库管理--新建目标

提供者：hardor

目标名：baseimages（项目名称）

目标：http://10.0.0.101/（同步机器的访问地址）

访问ID：admin（登录的用户名）

访问密码：（用户的密码）

点击测试连接：测试连接成功，即可创建。测试连接失败则是没有添加仓库地址信任

复制管理--新建规则

名称：baseimages-sync

复制模式：push-based （push-based是有新的镜像上传后直接推送目标服务器同步，pull-based是自己到目标服务器拉取惊醒同步）

目的Registry：选择建好的目标仓库域名

触发模式：选择事件驱动（只要上传一个镜像就触发同步规则）

10.0.0.101-web端设置

仓库管理--新建目标

提供者：hardor

目标名：baseimages（项目名称）

目标：http://10.0.0.100/（同步机器的访问地址）

访问ID：admin（登录的用户名）

访问密码：（用户的密码）

点击测试连接：测试连接成功，即可创建。测试连接失败则是没有添加仓库地址信任

复制管理--新建规则

名称：baseimages-sync

复制模式：push-based （push-based是有新的镜像上传后直接推送目标服务器同步，pull-based是自己到目标服务器拉取惊醒同步）

目的Registry：选择建好的目标仓库域名

触发模式：选择事件驱动（只要上传一个镜像就触发同步规则）

hardor镜像仓库同步创见完毕。

上传镜像测试同步是否正常

[16:51:43 root@ubuntu-lyj /usr/local/harbor]#docker tag tomcat-web:app2 10.0.0.100/baseimages/tomcat-web:app2
[16:53:44 root@ubuntu-lyj /usr/local/harbor]#docker push 10.0.0.100/baseimages/tomcat-web:app2
The push refers to repository [10.0.0.100/baseimages/tomcat-web]
33974f0793c5: Pushed 
5e6ff1963642: Pushed 
1d6dad8a4caf: Pushed 
82f5a76483d8: Pushed 
2f5511ad49e2: Pushed 
225d79b4ec1b: Pushed 
ddb9acd95f3a: Pushed 
bd952ab4c733: Pushed 
e66a786eb1fd: Pushed 
b596827ea508: Pushed 
8d9cb3505aea: Pushed 
816ac87d2f0f: Pushed 
c0cd79243356: Pushed 
fb82b029bea0: Pushed 
app2: digest: sha256:c6cb282c89eebdddb099b80b3c1db298a5e3abeae403ecc2f2288f51c7e6324e size: 3250