Kubernetes部署harbor

环境

centos 7.9
k8s 1.24
containerd 1.6.4

一、下载

mkdir harbor && wget https://hub.fastgit.xyz/goharbor/harbor-helm/archive/refs/tags/v1.9.0.tar.gz
tar -zxf v1.9.0.tar.gz && cd harbor-helm-1.9.0

二、自定义配置文件

这里使用的postgresql和redis是我们在安装gitlab时创建的

cat > values-prod.yml << EOF
externalURL: https://harbor.filchaser.com
harborAdminPassword: Harbor12345
logLevel: debug
expose:
  type: ingress
  tls:
    enabled: true
  ingress:
    hosts:
      core: harbor.filchaser.com
      notary: notary.filchaser.com
    annotations:
      ingress.kubernetes.io/ssl-redirect: "true"
      ingress.kubernetes.io/proxy-body-size: "2048m"
      kubernetes.io/ingress.class: "nginx"
persistence:
    enabled: true
    resourcePolicy: "keep"
    persistentVolumeClaim:
      registry:
          storageClass: "harbor-nfs-sc"
          chartmuseum:
          storageClass: "harbor-nfs-sc"
          jobservice:
          storageClass: "harbor-nfs-sc"
          trivy:
          storageClass: "harbor-nfs-sc"
database:
  type: external
  external:
    host: "postgresql.ops.svc.cluster.local"
    port: "5432"
    username: "gitlab"
    password: "passw0rd"
    coreDatabase: "harbor"
    notaryServerDatabase: "notary_server"
    notarySignerDatabase: "notary_signer"
redis:
  type: external
  external:
    addr: "redis.ops.svc.cluster.local:6379"
EOF

三、创建StrogeClass和ns

由于harbor创建的服务比较多，我们单独建一个harbor的ns，和sc

#创建ns
kubectl create ns harbor

#创建sc
cat > harbor-data-sc.yml << EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: harbor-nfs-sc
provisioner: fuseim.pri/ifs
EOF

kubectl apply -f harbor-data-sc.yml

四、给harbor创建三个数据库

$ kubectl get pods -n ops -l name=postgresql

$ kubectl exec -it gitlab-postgresql-5b8969757f-txbjt -n kube-ops /bin/bash

# psql -U postgres

1、harbor
postgres=# CREATE DATABASE harbor OWNER postgres; # 创建 harbor 数据库
postgres=# GRANT ALL PRIVILEGES ON DATABASE harbor to postgres; # 授权给 postgres 用户
postgres=# GRANT ALL PRIVILEGES ON DATABASE harbor to gitlab; # 授权给 gitlab 用户


2、notary_server
postgres=# CREATE DATABASE notary_server OWNER postgres; # 创建 notary_server 数据库
postgres=# GRANT ALL PRIVILEGES ON DATABASE notary_server to postgres; # 授权给 postgres 用户
postgres=# GRANT ALL PRIVILEGES ON DATABASE notary_server to gitlab; # 授权给 gitlab 用户

2、notary_server
postgres=# CREATE DATABASE notary_signer OWNER postgres; # 创建 notary_server 数据库
postgres=# GRANT ALL PRIVILEGES ON DATABASE notary_signer to postgres; # 授权给 postgres 用户
postgres=# GRANT ALL PRIVILEGES ON DATABASE notary_signer to gitlab; # 授权给 gitlab 用户

五、应用

[root@k8s-master harbor-helm-1.9.0]# helm install --namespace harbor harbor -f values-prod.yml .
NAME: harbor
LAST DEPLOYED: Tue May 10 16:46:28 2022
NAMESPACE: harbor
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at https://harbor.filchaser.com
For more details, please visit https://github.com/goharbor/harbor

上面是我们通过 Helm 安装所有涉及到的一些资源对象，稍微等一会儿，就可以安装成功了，查看对应的 Pod 状态：

[root@k8s-master harbor-helm-1.9.0]# kubectl get pod -n harbor 
NAME                                    READY   STATUS    RESTARTS      AGE
harbor-chartmuseum-544899c459-v9rdv     1/1     Running   0             16m
harbor-core-56959fd6f4-lbn65            1/1     Running   0             16m
harbor-jobservice-c79c78867-vbzxf       1/1     Running   0             16m
harbor-notary-server-5888f5c94f-j4llb   1/1     Running   1 (10m ago)   16m
harbor-notary-signer-7f47cfd868-ndz99   1/1     Running   0             16m
harbor-portal-5859b4474b-ptsjj          1/1     Running   0             57m
harbor-registry-5788b67758-6p8db        2/2     Running   0             16m
harbor-trivy-0                          1/1     Running   0             63m

现在都是Running状态了，都成功运行起来了，查看下对应的 Ingress 对象：

[root@k8s-master harbor-helm-1.9.0]# kubectl get ingress -n harbor 
NAME                    CLASS    HOSTS                  ADDRESS                        PORTS     AGE
harbor-ingress          <none>   harbor.filchaser.com   192.168.100.7,192.168.100.79   80, 443   63m
harbor-ingress-notary   <none>   notary.filchaser.com   192.168.100.7,192.168.100.79   80, 443   63m

如果你有自己的真正的域名，则将上面的两个域名解析到你的任意一个 Ingress Controller 的 Pod 所在的节点即可，我们这里为了演示方便，还是自己在本地的/etc/hosts里面添加上harbor.filchaser.com和notary.filchaser.com的映射。

如果发现https证书没有生效，查看相应域名的ingress并检查

$ kubectl edit ingress harbor-ingress -n harbor

...
  tls:
  - hosts:
    - harbor.filchaser.com
    secretName: who-tls
...

添加完成后，在浏览器中输入harbor.filchaser.com就可以打开熟悉的 Harbor 的 Portal 界面了，当然我们配置的 Ingress 中会强制跳转到 https，所以如果你的浏览器有什么安全限制的话，需要信任我们这里 Ingress 对应的证书，证书文件可以通过查看 Secret 资源对象获取：

然后输入用户名：admin，密码：Harbor12345（当然我们也可以通过 Helm 安装的时候自己覆盖 harborAdminPassword）即可登录进入 Portal 首页：

然后我们来测试下使用 docker cli 来进行 pull/push 镜像，这里会看到如下报错

[root@k8s-node2 ~]# docker login harbor.filchaser.com -u admin -pHarbor12345Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "http://harbor.filchaser.com/v2/": dial tcp 192.168.100.79:80: connect: connection refused

这个时候需要修改 /etc/docker/daemon.json加入如下配置,然后重启docker

{
"registry-mirrors": ["https://7vnz06qj.mirror.aliyuncs.com"],
"insecure-registries": ["harbor.filchaser.com"], #新增
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
  "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
    ]
}

测试推送镜像到私有仓库

[root@k8s-node2 ~]# docker tag busybox:latest harbor.filchaser.com/library/busybox:latest
[root@k8s-node2 ~]# docker push harbor.filchaser.com/library/busybox:latest
The push refers to repository [harbor.filchaser.com/library/busybox]
01fd6df81c8e: Pushed 
latest: digest: sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee size: 527

推送完成后，我们同样可以在 Portal 页面上看到这个镜像的信息