Kubernetes入门（五）
部署Harbor
Harbor是由VMWare中国团队开源的容器镜像仓库。事实上，harbor是在Docker Registry上进行了相应的企业级扩展，从而获得了更加广泛的应用，这些新的企业级特性包括：管理用户界面，基于角色的访问控制，水平扩展，同步，AD/LDAP集成以及日志审计等。
github地址：https://github.com/goharbor/harbor/releases
手动部署
之前已经搭建了kubernetes集群，其中192.168.30.150作为master2节点，我们把它作为harbor服务器。
下载最新的docker-compose二进制文件：
[root@master2 ~]# vim /etc/hosts #添加一行
54.231.48.160 github-production-release-asset-2e65be.s3.amazonaws.com
[root@master2 ~]# curl -L https://github.com/docker/compose/releases/download/1.24.0-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/bin/docker-compose
[root@master2 ~]# chmod +x /usr/bin/docker-compose
下载harbor离线安装包：
[root@master2 ~]# wget https://storage.googleapis.com/harbor-releases/release-1.7.0/harbor-offline-installer-v1.7.0-rc2.tgz
[root@master2 ~]# tar zxf harbor-offline-installer-v1.7.0-rc2.tgz
准备CA证书
[root@master2 ~]# mkdir /data/ && cd /data
上传证书 harbor.lzxlinux.com.key.tar.gz #自制证书
[root@master2 data]# tar zxf harbor.lzxlinux.com.key.tar.gz
[root@master2 data]# ls cert/
ca.crt server.crt server.key
修改配置文件：
[root@master2 cert]# cd harbor/
[root@master2 harbor]# vim harbor.cfg #做下面修改
hostname = harbor.lzxlinux.com
ui_url_protocol = https
执行安装：
[root@master2 harbor]# sh install.sh
[root@master2 harbor]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5cad50fe8189 goharbor/nginx-photon:v1.7.0 "nginx -g 'daemon of…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp nginx
7f536f3285ff goharbor/harbor-jobservice:v1.7.0 "/harbor/start.sh" 5 minutes ago Up 5 minutes harbor-jobservice
f4c2f03e9315 goharbor/harbor-portal:v1.7.0 "nginx -g 'daemon of…" 5 minutes ago Up 5 minutes (healthy) 80/tcp harbor-portal
45755eaf06e3 goharbor/harbor-core:v1.7.0 "/harbor/start.sh" 5 minutes ago Up 5 minutes (healthy) harbor-core
d1b57df6021c goharbor/harbor-adminserver:v1.7.0 "/harbor/start.sh" 5 minutes ago Up 5 minutes (healthy) harbor-adminserver
f65f40cac8cc goharbor/harbor-db:v1.7.0 "/entrypoint.sh post…" 5 minutes ago Up 5 minutes (healthy) 5432/tcp harbor-db
dd50b19871b8 goharbor/harbor-registryctl:v1.7.0 "/harbor/start.sh" 5 minutes ago Up 5 minutes (healthy) registryctl
034de27d8a84 goharbor/registry-photon:v2.6.2-v1.7.0 "/entrypoint.sh /etc…" 5 minutes ago Up 5 minutes (healthy) 5000/tcp registry
dd0c1abf428c goharbor/redis-photon:v1.7.0 "docker-entrypoint.s…" 5 minutes ago Up 5 minutes 6379/tcp redis
97530153c3b7 goharbor/harbor-log:v1.7.0 "/bin/sh -c /usr/loc…" 5 minutes ago Up 5 minutes (healthy) 127.0.0.1:1514->10514/tcp harbor-log
0baf1dbfb8d0 ff281650a721 "/opt/bin/flanneld -…" 2 hours ago Up 2 hours k8s_kube-flannel_kube-flannel-ds-amd64-lfgv5_kube-system_f40a3fe8-34d6-11e9-83ed-000c297ff3a2_0
96ce131643a0 mirrorgooglecontainers/pause-amd64:3.1 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-flannel-ds-amd64-lfgv5_kube-system_f40a3fe8-34d6-11e9-83ed-000c297ff3a2_0
浏览器访问：
编辑Windows上的hosts文件，路径：C:\Windows\System32\drivers\etc\hosts，增加一行：192.168.30.150 harbor.lzxlinux.com 。
用户名：admin ，密码：Harbor12345（默认密码）
新建项目：
一般项目不设为公开
拉取公共镜像:
[root@master2 harbor]# docker pull busybox
[root@master2 harbor]# docker pull tomcat
打标签：
[root@master2 harbor]# docker tag busybox harbor.lzxlinux.com/lzx/busybox
[root@master2 harbor]# echo 192.168.30.150 harbor.lzxlinux.com >> /etc/hosts
[root@master2 harbor]# docker login https://harbor.lzxlinux.com
Username: admin
Password:
Error response from daemon: Get https://harbor.lzxlinux.com/v2/: x509: certificate signed by unknown authority
解决509问题：
[root@master2 harbor]# echo -n | openssl s_client -showcerts -connect harbor.lzxlinux.com:443 2>/dev/null |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/P' >> /etc/ssl/certs/ca-bundle.trust.crt
[root@master2 harbor]# systemctl restart docker
[root@master2 harbor]# docker-compose up -d # -d 后台启动
如果上面方法还是不行，可以这样操作
# mkdir -p /etc/docker/certs.d/harbor.lzxlinux.com
# cp /data/cert/ca.crt /etc/docker/certs.d/harbor.lzxlinux.com/ca.crt
# chmod 400 !$
# systemctl restart docker
再次登录：
[root@master2 harbor]# docker login https://harbor.lzxlinux.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
推送busybox镜像：
[root@master2 harbor]# docker push harbor.lzxlinux.com/lzx/busybox
The push refers to repository [harbor.lzxlinux.com/lzx/busybox]
683f499823be: Pushed
latest: digest: sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510 size: 527
浏览器查看：
刷新一下，可以查看到新传上来的镜像。
推送tomcat镜像：
[root@master2 harbor]# docker tag tomcat harbor.lzxlinux.com/lzx/tomcat
[root@master2 harbor]# docker push harbor.lzxlinux.com/lzx/tomcat
再次刷新查看
在Kubernetes中使用harbor
废了很大劲部署harbor，那它有什么作用呢？接下来在kubernetes中使用它。
以下操作在master节点中执行。
创建secret：
[root@master ~]# kubectl create secret docker-registry my-secret --docker-server=harbor.lzxlinux.com --docker-username=admin --docker-password=Harbor12345
[root@master ~]# kubectl get secret
NAME TYPE DATA AGE
default-token-lfx98 kubernetes.io/service-account-token 3 1d
my-secret kubernetes.io/dockerconfigjson 1 38s # secret可创建多个，在拉取镜像时指定
推送httpd镜像：
[root@master ~]# docker pull httpd
[root@master ~]# docker tag httpd harbor.lzxlinux.com/lzx/httpd
[root@master ~]# docker login https://harbor.lzxlinux.com
[root@master ~]# docker push harbor.lzxlinux.com/lzx/httpd
刷新查看
定义一个pod：
[root@master ~]# vim httpd-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: httpd-pod
spec:
containers:
- image: harbor.lzxlinux.com/lzx/httpd:latest
name: httpd-pod
imagePullSecrets:
- name: my-secret #引用my-secret
[root@master ~]# kubectl create -f httpd-pod.yaml
[root@master ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
httpd-pod 0/1 ImagePullBackOff 0 53s #状态不对
nginx-6f858d4d45-wwnm9 1/1 Running 0 4h
[root@master ~]# kubectl describe pod httpd-pod
Warning Failed 25s (x2 over 53s) kubelet, 192.168.30.130 Error: ErrImagePull
Normal BackOff 11s (x2 over 52s) kubelet, 192.168.30.130 Back-off pulling image "harbor.lzxlinux.com/lzx/httpd:latest"
Warning Failed 11s (x2 over 52s) kubelet, 192.168.30.130 Error: ImagePullBackOff
Normal Pulling 0s (x3 over 1m) kubelet, 192.168.30.130 pulling image "harbor.lzxlinux.com/lzx/httpd:latest"
解决问题：
[root@master2 harbor]# scp /data/cert/ca.crt 192.168.30.129:/root/
[root@master2 harbor]# scp /data/cert/ca.crt 192.168.30.130:/root/
[root@node1 ~]# mkdir -p /etc/docker/certs.d/harbor.lzxlinux.com
[root@node1 ~]# cp ca.crt /etc/docker/certs.d/harbor.lzxlinux.com/ca.crt
[root@node1 ~]# chmod 400 !$
[root@node1 ~]# systemctl restart docker
[root@node1 ~]# vim /etc/hosts #增加一行
192.168.30.150 harbor.lzxlinux.com
[root@node2 ~]# mkdir -p /etc/docker/certs.d/harbor.lzxlinux.com
[root@node2 ~]# cp ca.crt /etc/docker/certs.d/harbor.lzxlinux.com/ca.crt
[root@node2 ~]# chmod 400 !$
[root@node2 ~]# systemctl restart docker
[root@node2 ~]# vim /etc/hosts #增加一行
192.168.30.150 harbor.lzxlinux.com
继续创建pod：
[root@master ~]# kubectl delete pod httpd-pod
[root@master ~]# kubectl create -f httpd-pod.yaml
[root@master ~]# kubectl describe pod httpd-pod
Normal Scheduled <invalid> default-scheduler Successfully assigned default/httpd-pod to 192.168.30.130
Normal Pulling <invalid> kubelet, 192.168.30.130 pulling image "harbor.lzxlinux.com/lzx/httpd:latest"
Normal Pulled <invalid> kubelet, 192.168.30.130 Successfully pulled image "harbor.lzxlinux.com/lzx/httpd:latest"
Normal Created <invalid> kubelet, 192.168.30.130 Created container
Normal Started <invalid> kubelet, 192.168.30.130 Started container
[root@master ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
httpd-pod 1/1 Running 0 3m
nginx-6f858d4d45-wwnm9 1/1 Running 0 4h
这次成功创建，正在运行中。刷新浏览器，可以看到下载次数已更新。
免费证书申请：https://freessl.cn/
harbor443: connect: connection refused问题解决：https://blog.csdn.net/miss1181248983/article/details/88661825
