helm 安装极狐 gitlab（集成外部 kong ingress controller + ceph + 对象存储）

helm 安装极狐 gitlab（集成外部 kong ingress controller + ceph + 对象存储）

1. 环境信息

1.1 主机信息

主机	配置	角色
10.10.10.51(ceph01)	1核3GB，100GB系统盘，5*50GB数据裸盘	mon,osd,mgr,admin
10.10.10.52(ceph02)	1核3GB，100GB系统盘，5*50GB数据裸盘	mon,osd,mgr,mds
10.10.10.53(ceph03)	1核3GB，100GB系统盘，5*50GB数据裸盘	mon,osd,rgw
10.10.10.60	1核4GB，100GB系统盘	minio，提供兼容 s3 的对象存储
10.10.10.61	8核32GB，100G系统盘	k8s，helm，安装极狐gitlab v15.0.0，内存建议16GB以上

• 操作系统 centos 7.9 x64

1.2 环境初始化

echo "设置时钟同步"
yum install -y chrony
systemctl start chronyd.service
systemctl enable chronyd.service

echo "关闭防火墙"
systemctl stop firewalld
systemctl disable firewalld

echo "关闭 selinux"
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
getenforce

echo "优化 ssh 登录速度"
sed -i 's/#UseDNS yes/UseDNS no/g' /etc/ssh/sshd_config
systemctl restart sshd

echo "安装基础软件 lrzsz, epel-release"
yum install epel-release lrzsz -y

1.3 升级内核

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
# 安装ELRepo
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
# 载入elrepo-kernel元数据
yum --disablerepo=\* --enablerepo=elrepo-kernel repolist
# 查看可用的rpm包
yum --disablerepo=\* --enablerepo=elrepo-kernel list kernel*
# 安装长期支持版本的kernel
yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt.x86_64
# 删除旧版本工具包
yum remove kernel-tools-libs.x86_64 kernel-tools.x86_64 -y
# 安装新版本工具包
yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt-tools.x86_64

#查看默认启动顺序
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg  

#默认启动的顺序是从0开始，新内核是从头插入（目前位置在0，而4.4.4的是在1），所以需要选择0。
grub2-set-default 0

reboot

• 当前最新稳定内核版本 CentOS Linux (5.4.180-1.el7.elrepo.x86_64) 7 (Core)

2. 安装 minio

涉及主机：

• 10.10.10.60

2.1 安装

下载 minio server

yum install wget -y
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
mv minio /usr/local/bin/minio
minio -v

minio version RELEASE.2022-02-26T02-54-46Z

下载 minio client

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/mc
mc -v

mc version RELEASE.2022-02-26T03-58-31Z

配置 minio 数据目录

mkdir -p /opt/minio/data
mkdir -p /opt/minio/logs

创建 start_minio_server.sh 启动脚本

#!/bin/bash
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123456
nohup /usr/local/bin/minio server /opt/minio/data > /opt/minio/logs/minio.log 2>&1 &

启动 minio server

chmod +x start_minio_server.sh
./start_minio_server.sh

• 默认对外服务端口 9000

为客户端设置别名

mc alias set myminio http://10.10.10.60:9000 minio minio123456

2.2 配置桶

配置 gitlab 需要用到的存储桶

mc mb myminio/gitlab-cache # runner cache
mc mb myminio/gitlab-artifacts-storage
mc mb myminio/gitlab-external-diffs
mc mb myminio/gitlab-lfs-storage
mc mb myminio/gitlab-uploads-storage
mc mb myminio/gitlab-packages-storage
mc mb myminio/gitlab-dependency-proxy
mc mb myminio/gitlab-terraform-state
# mc mb myminio/gitlab-pages # pages 功能一般是关闭的
mc mb myminio/gitlab-backup-upload
mc mb myminio/gitlab-tmp-storage
mc mb myminio/gitlab-registry

3. 安装 ceph 集群

涉及主机：

• 10.10.10.51
• 10.10.10.52
• 10.10.10.53

版本：luminous

参考：ceph-deploy 安装 ceph - leffss - 博客园 (cnblogs.com)

4. 安装 k8s 集群

涉及主机：

• 10.10.10.61

版本：v1.22.9+k3s1

参考：k8s 测试环境搭建（k3s） - leffss - 博客园 (cnblogs.com)

如果需要搭建完整版集群，请参考：easzlab/kubeasz

5. k8s集成ceph rbd(StorageClass方式)

首先删除 k3s 默认集成的 StorageClass：

kubectl delete sc local-path
kubectl -n kube-system delete deployment local-path-provisioner

然后参考：k8s集成ceph rbd(StorageClass方式)

最后验证 pvc 时，发现新建的 pvc 一直 pending 状态：

[root@localhost ~]# kubectl get pvc
NAME             STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS        AGE
ceph-pvc-test1   Pending                                      ceph-storageclass   7s

[root@localhost ~]# kubectl describe pvc ceph-pvc-test1
Name:          ceph-pvc-test1
Namespace:     default
StorageClass:  ceph-storageclass
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: ceph-storageclass
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  7s (x2 over 15s)  persistentvolume-controller  no volume plugin matched name: kubernetes.io/rbd

• 原因是相对于标准 k8s 集群，k3s 为了精简二进制文件大小删除了不少 volume plugins ，这里就缺少插件：kubernetes.io/rbd，参考：Rancher Docs: Volumes and Storage

这里我们就不采用上面的方法，而是安装 ceph 官方的插件：ceph/ceph-csi: CSI driver for Ceph 解决

5.1 部署 RBD provisioner

参考：ceph-csi/deploy-rbd.md#deployment-with-kubernetes

删除前面创建的 pvc 与 StorageClass，根据 k8s 集群版本下载对应 ceph-csi 版本，这里我们下载最新版本：v3.6.1

yum install wget -y
wget https://github.com/ceph/ceph-csi/archive/refs/tags/v3.6.1.tar.gz
mv v3.6.1.tar.gz ceph-csi-3.6.1.tar.gz
tar zxvf ceph-csi-3.6.1.tar.gz
cd ceph-csi-3.6.1/deploy/rbd/kubernetes

查看需要用到的镜像：

[root@localhost kubernetes]# grep -rn image .|grep -v imagePullPolicy
./csi-rbdplugin-provisioner.yaml:50:          image: k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
./csi-rbdplugin-provisioner.yaml:70:          image: k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
./csi-rbdplugin-provisioner.yaml:84:          image: k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
./csi-rbdplugin-provisioner.yaml:98:          image: k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
./csi-rbdplugin-provisioner.yaml:115:          image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin-provisioner.yaml:169:          image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin-provisioner.yaml:189:          image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin.yaml:31:          image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
./csi-rbdplugin.yaml:53:          image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin.yaml:125:          image: quay.io/cephcsi/cephcsi:v3.6.1
[root@localhost kubernetes]# 

去重整理：

k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
quay.io/cephcsi/cephcsi:v3.6.1

其中 k8s.gcr.io 开头的因为网络原因，都下载不下来，我们可以使用国内镜像替代：

docker pull registry.aliyuncs.com/google_containers/csi-provisioner:v3.1.0
docker pull registry.aliyuncs.com/google_containers/csi-snapshotter:v5.0.1
docker pull registry.aliyuncs.com/google_containers/csi-attacher:v3.4.0
docker pull registry.aliyuncs.com/google_containers/csi-resizer:v1.4.0
docker pull registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.4.0

然后重新打标签：

docker tag registry.aliyuncs.com/google_containers/csi-provisioner:v3.1.0 k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
docker tag registry.aliyuncs.com/google_containers/csi-snapshotter:v5.0.1 k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
docker tag registry.aliyuncs.com/google_containers/csi-attacher:v3.4.0 k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
docker tag registry.aliyuncs.com/google_containers/csi-resizer:v1.4.0 k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
docker tag registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.4.0 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0

应用 RBAC 权限：

kubectl create -f csi-provisioner-rbac.yaml
kubectl create -f csi-nodeplugin-rbac.yaml

应用 PSP 权限：

kubectl create -f csi-provisioner-psp.yaml
kubectl create -f csi-nodeplugin-psp.yaml

修改 csi-config-map.yaml 配置：

#
# /!\ DO NOT MODIFY THIS FILE
#
# This file has been automatically generated by Ceph-CSI yamlgen.
# The source for the contents can be found in the api/deploy directory, make
# your modifications there.
#
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "ceph-csi-config"
data:
  config.json: |-
    [
      {
        "clusterID": "c1dda3a3-c30a-4594-b87c-3de2a89d0214",
        "monitors": [
          "10.10.10.51:6789",
          "10.10.10.52:6789",
          "10.10.10.53:6789"
        ]
      }
    ]

• clusterID 在 ceph 集群配置文件 /etc/ceph/ceph.conf 中获得

应用：

kubectl create -f csi-config-map.yaml

创建 ceph-config.yaml 配置：

---
# This is a sample configmap that helps define a Ceph configuration as required
# by the CSI plugins.

# Sample ceph.conf available at
# https://github.com/ceph/ceph/blob/master/src/sample.ceph.conf Detailed
# documentation is available at
# https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/
apiVersion: v1
kind: ConfigMap
data:
  ceph.conf: |
    [global]
    fsid = c1dda3a3-c30a-4594-b87c-3de2a89d0214
    public_network = 10.10.10.0/24
    cluster_network = 10.10.10.0/24
    mon_initial_members = ceph01
    mon_host = 10.10.10.51,10.10.10.52,10.10.10.53
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx

  # keyring is a required key and its value should be empty
  keyring: |
metadata:
  name: ceph-config

• ceph.conf 的内容就是 ceph 节点上 /etc/ceph/ceph.conf

kubectl create -f ceph-config.yaml

注释掉 ceph-csi-encryption-kms-config 相关内容

[root@localhost kubernetes]# grep -rn 'ceph-csi-encryption-kms-config' .
./csi-rbdplugin-provisioner.yaml:160:            - name: ceph-csi-encryption-kms-config
./csi-rbdplugin-provisioner.yaml:161:              mountPath: /etc/ceph-csi-encryption-kms-config/
./csi-rbdplugin-provisioner.yaml:230:        - name: ceph-csi-encryption-kms-config
./csi-rbdplugin-provisioner.yaml:232:            name: ceph-csi-encryption-kms-config
./csi-rbdplugin.yaml:107:            - name: ceph-csi-encryption-kms-config
./csi-rbdplugin.yaml:108:              mountPath: /etc/ceph-csi-encryption-kms-config/
./csi-rbdplugin.yaml:188:        - name: ceph-csi-encryption-kms-config
./csi-rbdplugin.yaml:190:            name: ceph-csi-encryption-kms-config

应用 provisioner：

kubectl create -f csi-rbdplugin-provisioner.yaml
kubectl create -f csi-rbdplugin.yaml

查看验证

[root@localhost kubernetes]# kubectl get all
NAME                                             READY   STATUS    RESTARTS   AGE
pod/csi-rbdplugin-provisioner-74888444cc-d5kxf   0/7     Pending   0          20m
pod/csi-rbdplugin-provisioner-74888444cc-vqcrw   0/7     Pending   0          20m
pod/csi-rbdplugin-ds7w7                          3/3     Running   0          20m
pod/csi-rbdplugin-provisioner-74888444cc-b6dbn   7/7     Running   0          20m

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                  ClusterIP   10.43.0.1      <none>        443/TCP    31h
service/csi-rbdplugin-provisioner   ClusterIP   10.43.21.47    <none>        8080/TCP   20m
service/csi-metrics-rbdplugin       ClusterIP   10.43.154.94   <none>        8080/TCP   20m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/csi-rbdplugin   1         1         1       1            1           <none>          20m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/csi-rbdplugin-provisioner   1/3     3            1           20m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/csi-rbdplugin-provisioner-74888444cc   3         3         1       20m

• 其中 csi-rbdplugin-provisioner 有 2 个 pod 一直是 pending 状态，原因是设置了 pod 亲和性，3个副本不能调度到同一 node 上运行，我们这里的 k8s 集群为单节点，仅测试而已，忽略即可

5.2 创建 StorageClass

创建 csi-rbd-secret.yaml

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  # Key values correspond to a user name and its key, as defined in the
  # ceph cluster. User ID should have required access to the 'pool'
  # specified in the storage class
  userID: kubernetes
  userKey: AQAnEZ9ivF6ZExAAnrZhVVxwtSSoaoWM69SuTA==

  # Encryption passphrase
  encryptionPassphrase: test_passphrase

其中的userID与userKey通过以下方式获取

创建存储池：

ceph osd pool create k8s 128 128
rbd pool init k8s

创建并授权用户

$ ceph auth get-or-create client.kubernetes mon 'allow r' osd 'profile rbd pool=k8s'
[client.kubernetes]
        key = AQAnEZ9ivF6ZExAAnrZhVVxwtSSoaoWM69SuTA==

• kubernetes 为 userID，key 为 userKey

应用：

kubectl apply -f csi-rbd-secret.yaml

创建 storageclass.yaml

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
# If topology based provisioning is desired, delayed provisioning of
# PV is required and is enabled using the following attribute
# For further information read TODO<doc>
# volumeBindingMode: WaitForFirstConsumer
parameters:
   # (required) String representing a Ceph cluster to provision storage from.
   # Should be unique across all Ceph clusters in use for provisioning,
   # cannot be greater than 36 bytes in length, and should remain immutable for
   # the lifetime of the StorageClass in use.
   # Ensure to create an entry in the configmap named ceph-csi-config, based on
   # csi-config-map-sample.yaml, to accompany the string chosen to
   # represent the Ceph cluster in clusterID below
   clusterID: c1dda3a3-c30a-4594-b87c-3de2a89d0214

   # (optional) If you want to use erasure coded pool with RBD, you need to
   # create two pools. one erasure coded and one replicated.
   # You need to specify the replicated pool here in the `pool` parameter, it is
   # used for the metadata of the images.
   # The erasure coded pool must be set as the `dataPool` parameter below.
   # dataPool: <ec-data-pool>

   # (required) Ceph pool into which the RBD image shall be created
   # eg: pool: rbdpool
   pool: k8s

   # (optional) RBD image features, CSI creates image with image-format 2 CSI
   # RBD currently supports `layering`, `journaling`, `exclusive-lock`,
   # `object-map`, `fast-diff`, `deep-flatten` features.
   # Refer https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#image-features
   # for image feature dependencies.
   # imageFeatures: layering,journaling,exclusive-lock,object-map,fast-diff
   imageFeatures: "layering"

   # (optional) Specifies whether to try other mounters in case if the current
   # mounter fails to mount the rbd image for any reason. True means fallback
   # to next mounter, default is set to false.
   # Note: tryOtherMounters is currently useful to fallback from krbd to rbd-nbd
   # in case if any of the specified imageFeatures is not supported by krbd
   # driver on node scheduled for application pod launch, but in the future this
   # should work with any mounter type.
   # tryOtherMounters: false

   # (optional) mapOptions is a comma-separated list of map options.
   # For krbd options refer
   # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
   # For nbd options refer
   # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
   # Format:
   # mapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
   # An empty mounter field is treated as krbd type for compatibility.
   # eg:
   # mapOptions: "krbd:lock_on_read,queue_depth=1024;nbd:try-netlink"

   # (optional) unmapOptions is a comma-separated list of unmap options.
   # For krbd options refer
   # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
   # For nbd options refer
   # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
   # Format:
   # unmapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
   # An empty mounter field is treated as krbd type for compatibility.
   # eg:
   # unmapOptions: "krbd:force;nbd:force"

   # The secrets have to contain Ceph credentials with required access
   # to the 'pool'.
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: default
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: default

   # (optional) Specify the filesystem type of the volume. If not specified,
   # csi-provisioner will set default as `ext4`.
   csi.storage.k8s.io/fstype: ext4

   # (optional) uncomment the following to use rbd-nbd as mounter
   # on supported nodes
   # mounter: rbd-nbd

   # (optional) ceph client log location, eg: rbd-nbd
   # By default host-path /var/log/ceph of node is bind-mounted into
   # csi-rbdplugin pod at /var/log/ceph mount path. This is to configure
   # target bindmount path used inside container for ceph clients logging.
   # See docs/rbd-nbd.md for available configuration options.
   # cephLogDir: /var/log/ceph

   # (optional) ceph client log strategy
   # By default, log file belonging to a particular volume will be deleted
   # on unmap, but you can choose to just compress instead of deleting it
   # or even preserve the log file in text format as it is.
   # Available options `remove` or `compress` or `preserve`
   # cephLogStrategy: remove

   # (optional) Prefix to use for naming RBD images.
   # If omitted, defaults to "csi-vol-".
   # volumeNamePrefix: "foo-bar-"

   # (optional) Instruct the plugin it has to encrypt the volume
   # By default it is disabled. Valid values are "true" or "false".
   # A string is expected here, i.e. "true", not true.
   # encrypted: "true"

   # (optional) Use external key management system for encryption passphrases by
   # specifying a unique ID matching KMS ConfigMap. The ID is only used for
   # correlation to configmap entry.
   # encryptionKMSID: <kms-config-id>

   # Add topology constrained pools configuration, if topology based pools
   # are setup, and topology constrained provisioning is required.
   # For further information read TODO<doc>
   # topologyConstrainedPools: |
   #   [{"poolName":"pool0",
   #     "dataPool":"ec-pool0" # optional, erasure-coded pool for data
   #     "domainSegments":[
   #       {"domainLabel":"region","value":"east"},
   #       {"domainLabel":"zone","value":"zone1"}]},
   #    {"poolName":"pool1",
   #     "dataPool":"ec-pool1" # optional, erasure-coded pool for data
   #     "domainSegments":[
   #       {"domainLabel":"region","value":"east"},
   #       {"domainLabel":"zone","value":"zone2"}]},
   #    {"poolName":"pool2",
   #     "dataPool":"ec-pool2" # optional, erasure-coded pool for data
   #     "domainSegments":[
   #       {"domainLabel":"region","value":"west"},
   #       {"domainLabel":"zone","value":"zone1"}]}
   #   ]

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard

• clusterID 与 pool 根据实际情况填写

应用：

kubectl apply -f storageclass.yaml

查看 sc

[root@localhost ~]# kubectl get sc
NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   3s

创建 pvc 测试

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc

查看 pvc 是否正常

kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-pvc   Bound    pvc-8256d0e9-95be-43f3-882c-31997fd161f8   1Gi        RWO            csi-rbd-sc     21s

创建 pod 验证 pvc

---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: docker.io/library/nginx:latest
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

进入 pod 查看是否正常挂载

[root@localhost kubernetes]# kubectl get pod
NAME                                         READY   STATUS    RESTARTS   AGE
csi-rbdplugin-provisioner-74888444cc-wj5nv   0/7     Pending   0          5m43s
csi-rbdplugin-provisioner-74888444cc-tgmj6   0/7     Pending   0          5m43s
csi-rbdplugin-bdgnx                          3/3     Running   0          5m43s
csi-rbdplugin-provisioner-74888444cc-mjb9d   7/7     Running   0          5m43s
csi-rbd-demo-pod                             1/1     Running   0          112s
[root@localhost kubernetes]# kubectl exec -it csi-rbd-demo-pod -- /bin/bash
root@csi-rbd-demo-pod:/# df -h
Filesystem               Size  Used Avail Use% Mounted on
overlay                   50G  4.8G   46G  10% /
tmpfs                     64M     0   64M   0% /dev
tmpfs                     16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G  4.8G   46G  10% /etc/hosts
shm                       64M     0   64M   0% /dev/shm
/dev/rbd0                976M  2.6M  958M   1% /var/lib/www/html
tmpfs                     32G   12K   32G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                     16G     0   16G   0% /proc/acpi
tmpfs                     16G     0   16G   0% /proc/scsi
tmpfs                     16G     0   16G   0% /sys/firmware
root@csi-rbd-demo-pod:/# ls -l /var/lib/www/html/
total 16
drwx------ 2 root root 16384 Jun  7 09:57 lost+found
root@csi-rbd-demo-pod:/# cd /var/lib/www/html/      
root@csi-rbd-demo-pod:/var/lib/www/html# 
root@csi-rbd-demo-pod:/var/lib/www/html# echo test > test.txt
root@csi-rbd-demo-pod:/var/lib/www/html# 
root@csi-rbd-demo-pod:/var/lib/www/html# cat test.txt 
test
root@csi-rbd-demo-pod:/var/lib/www/html# 

更多示例：https://github.com/ceph/ceph-csi/tree/devel/examples/rbd

5.3 设置为默认 StorageClass

gitlab chart 安装时创建 pvc 需要调用默认 StorageClass

[root@localhost ~]# kubectl get sc

NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   8m37s
[root@localhost ~]# 
[root@localhost ~]# kubectl patch storageclass csi-rbd-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/csi-rbd-sc patched
[root@localhost ~]# 
[root@localhost ~]# kubectl get sc
NAME                   PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc (default)   rbd.csi.ceph.com   Delete          Immediate           true                   9m5s

6. 添加极狐gitlab chart 仓库

涉及主机：

• 10.10.10.61

6.1 安装 helm v3

安装 helm v3（v3.3.1 或者更新），参考官方安装文档：Helm | Installing Helm

这里使用手动下载安装方式：

# 发布地址：https://github.com/helm/helm/releases
curl -O https://get.helm.sh/helm-v3.8.0-linux-amd64.tar.gz
tar zxvf helm-v3.8.0-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
chmod +x /usr/local/bin/helm
helm version

version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

6.2 添加 chart 仓库

chart 官方地址：gitlab/gitlab-jh

helm repo add gitlab-jh https://charts.gitlab.cn

# 更新仓库
helm repo update gitlab-jh

# 查看镜像仓库列表
helm search repo gitlab-jh

# 查看完整的镜像仓库列表
helm search repo -l gitlab-jh

7. 安装极狐gitlab

涉及主机：

• 10.10.10.61

域名为 gitlab.example.com，不启用 https，如果启用，请使用有效的证书，自签证书会遇到很多配置问题

7.1 前置准备

1. 修改 ssh 端口

   gitlab-shell 需要使用 22 端口（ssh 方式拉取和提交时），所以修改端口为 2222

   vi /etc/ssh/sshd_config

   ...
   Port 2222
   ...
   

2. 设置 k3s 集群连接配置文件

   原因是：helm v3 版本不再需要 Tiller，而是直接访问 ApiServer 来与 k8s 交互，通过环境变量 KUBECONFIG 来读取存有 ApiServer 的地址与 token 的配置文件地址，默认地址为 ~/.kube/config，但是 k3s 的 KUBECONFIG 默认为 /etc/rancher/k3s/k3s.yaml

   临时解决

   export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
   

   永久解决

   $ vi .bash_profile # 添加
   export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
   
   $ source .bash_profile
   

3. 删除 k3s 集群自带的 traefik ingress controller

   这里使用 kong ingress controller，所以需要删除集群中自带的 traefik ingress controller

   $ kubectl get all -n kube-system
   NAME                                          READY   STATUS      RESTARTS   AGE
   pod/helm-install-traefik-g4xsp                0/1     Completed   0          136m
   pod/metrics-server-86cbb8457f-8wghh           1/1     Running     0          136m
   pod/local-path-provisioner-5ff76fc89d-r87zg   1/1     Running     0          136m
   pod/svclb-traefik-24nj7                       2/2     Running     0          136m
   pod/coredns-854c77959c-nm28h                  1/1     Running     0          136m
   pod/traefik-6f9cbd9bd4-lm4xf                  1/1     Running     0          136m
   
   NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
   service/kube-dns             ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       137m
   service/metrics-server       ClusterIP      10.43.36.234    <none>        443/TCP                      137m
   service/traefik-prometheus   ClusterIP      10.43.47.8      <none>        9100/TCP                     136m
   service/traefik              LoadBalancer   10.43.154.114   10.10.10.60   80:30104/TCP,443:31153/TCP   136m
   
   NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/svclb-traefik   1         1         1       1            1           <none>          136m
   
   NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/metrics-server           1/1     1            1           137m
   deployment.apps/local-path-provisioner   1/1     1            1           137m
   deployment.apps/coredns                  1/1     1            1           137m
   deployment.apps/traefik                  1/1     1            1           136m
   
   NAME                                                DESIRED   CURRENT   READY   AGE
   replicaset.apps/metrics-server-86cbb8457f           1         1         1       136m
   replicaset.apps/local-path-provisioner-5ff76fc89d   1         1         1       136m
   replicaset.apps/coredns-854c77959c                  1         1         1       136m
   replicaset.apps/traefik-6f9cbd9bd4                  1         1         1       136m
   
   NAME                             COMPLETIONS   DURATION   AGE
   job.batch/helm-install-traefik   1/1           19s        137m
   
   # 删除 traefik
   $ kubectl -n kube-system delete service traefik-prometheus
   $ kubectl -n kube-system delete service traefik
   $ kubectl -n kube-system delete deployment.apps/traefik
   
   $ kubectl get po -n kube-system
   NAME                                      READY   STATUS      RESTARTS   AGE
   helm-install-traefik-g4xsp                0/1     Completed   0          139m
   metrics-server-86cbb8457f-8wghh           1/1     Running     0          139m
   local-path-provisioner-5ff76fc89d-r87zg   1/1     Running     0          139m
   coredns-854c77959c-nm28h                  1/1     Running     0          139m
   

7.2 安装 kong-ingress-controller

官方地址：Kong/kubernetes-ingress-controller

安装：

kubectl apply -f https://bit.ly/k4k8s

• 也可以使用 helm 安装，参考 Kong/kubernetes-ingress-controller: get-started

确认安装情况：

[root@localhost ~]# kubectl -n kong get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/svclb-kong-proxy-f5t6g          2/2     Running   0          14h
pod/ingress-kong-5c9f6f86bc-scnkx   2/2     Running   0          14h

NAME                              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/kong-validation-webhook   ClusterIP      10.43.205.159   <none>        443/TCP                      14h
service/kong-proxy                LoadBalancer   10.43.155.217   10.10.10.61   80:31859/TCP,443:32035/TCP   14h

NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-kong-proxy   1         1         1       1            1           <none>          14h

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-kong   1/1     1            1           14h

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-kong-5c9f6f86bc   1         1         1       14h

查看 ingress 相关资源：

[root@localhost ~]# kubectl api-resources|grep -i ingress
kongingresses                     ki           configuration.konghq.com/v1            true         KongIngress
tcpingresses                                   configuration.konghq.com/v1beta1       true         TCPIngress
udpingresses                                   configuration.konghq.com/v1beta1       true         UDPIngress
ingressclasses                                 networking.k8s.io/v1                   false        IngressClass
ingresses                         ing          networking.k8s.io/v1                   true         Ingress
ingressroutes                                  traefik.containo.us/v1alpha1           true         IngressRoute
ingressroutetcps                               traefik.containo.us/v1alpha1           true         IngressRouteTCP
ingressrouteudps                               traefik.containo.us/v1alpha1           true         IngressRouteUDP
[root@localhost ~]# 
[root@localhost ~]# kubectl get ingressclasses
NAME   CONTROLLER                            PARAMETERS   AGE
kong   ingress-controllers.konghq.com/kong   <none>       14h

• ingressclasses 名称为 kong，后面安装 chart 指定参数需要用到

7.3 设置对象存储连接 secret

参考：doc/charts/globals.md · master · GitLab.org / charts / GitLab Chart · GitLab

rails 、registry 和 runner 使用的对象存储连接配置不一样，所以要创建 3 个 secret

创建 rails 连接 rails_minio.yaml 文件

provider: AWS
region: minio
aws_access_key_id: minio
aws_secret_access_key: minio123456
endpoint: http://10.10.10.60:9000
path_style: true

• 其他 s3 兼容的配置参考：Object storage | GitLab

kubectl create secret generic gitlab-storage --from-file=connection=rails_minio.yaml

创建 registry 连接 registry_minio.yaml 文件

s3:
  v4auth: true
  regionendpoint: "http://10.10.10.60:9000"
  pathstyle: true
  region: minio
  bucket: gitlab-registry
  accesskey: minio
  secretkey: minio123456

应用：

kubectl create secret generic gitlab-registry-storage --from-file=connection=registry_minio.yaml

rails 与 registry 支持的对象存储配置参考：examples/objectstorage · master · GitLab.org / charts / GitLab Chart · GitLab

创建 runner cache 连接 secret

kubectl create secret generic gitlab-minioaccess \
    --from-literal=accesskey="minio" \
    --from-literal=secretkey="minio123456"

• secret 名称 gitlab-minioaccess，后面安装 chart 配置需要用到

runner cache 配置参考：

• Using the GitLab Runner chart | GitLab

• GitLab Runner Helm Chart | GitLab

7.4 安装 chart

这里的--set选项是chart的configure，Helm通过chart+configure的方式实现部署，configure可以视为部署chart时需要定义的变量，这里也可以使用helm install gitlab gitlab-jh/gitlab -f values.yaml的方式批量配置configure，详细内容请参考：values.yaml · master · GitLab.org / charts / GitLab Chart · GitLab

创建 runner cache 配置文件 runner_cache.yaml：

gitlab-runner:
  runners:
    config: |
      [[runners]]
        [runners.kubernetes]
          image = "ubuntu:18.04"
          [runners.cache]
            Type = "s3"
            Path = "gitlab-runner"
            Shared = true
            [runners.cache.s3]
              ServerAddress = "10.10.10.60:9000"
              BucketName = "gitlab-cache"
              Insecure = true
    cache:
      secretName: gitlab-minioaccess

• Insecure = true，使用 http

安装命令：

helm install gitlab gitlab-jh/gitlab \
  -f runner_cache.yaml \
  --set global.hosts.domain=example.com \
  --set global.hosts.https=false \
  --set global.ingress.tls.enabled=false \
  --set certmanager-issuer.email=me@example.com \
  --set global.minio.enabled=false \
  --set global.registry.bucket=gitlab-registry \
  --set global.appConfig.object_store.enabled=true \
  --set global.appConfig.object_store.proxy_download=true \
  --set global.appConfig.object_store.connection.secret=gitlab-storage \
  --set global.appConfig.object_store.connection.key=connection \
  --set global.appConfig.lfs.bucket=gitlab-lfs-storage \
  --set global.appConfig.artifacts.bucket=gitlab-artifacts-storage \
  --set global.appConfig.uploads.bucket=gitlab-uploads-storage \
  --set global.appConfig.packages.bucket=gitlab-packages-storage \
  --set global.appConfig.externalDiffs.bucket=gitlab-external-diffs \
  --set global.appConfig.terraformState.bucket=gitlab-terraform-state \
  --set global.appConfig.dependencyProxy.bucket=gitlab-dependency-proxy \
  --set global.appConfig.backups.bucket=gitlab-backup-upload \
  --set global.appConfig.backups.tmpBucket=gitlab-tmp-storage \
  --set gitlab.toolbox.backups.objectStorage.config.secret=gitlab-storage \
  --set gitlab.toolbox.backups.objectStorage.config.key=connection \
  --set registry.storage.secret=gitlab-registry-storage \
  --set registry.storage.key=connection \
  --set global.ingress.enabled=true \
  --set global.ingress.class=kong \
  --set nginx-ingress.enabled=false \
  --set global.busybox.image.repository=busybox \
  --set global.busybox.image.tag=1.28.4 \
  --version=6.0.0

• 外部 ingress 配置参考：examples/values-traefik-ingress.yaml · master · GitLab.org / charts / GitLab Chart · GitLab
• 外部存储配置参考：examples/values-external-objectstorage.yaml · master · GitLab.org / charts / GitLab Chart · GitLab
• 使用 example.com 作为域名，部署成功后便可通过 http://gitlab.example.com 访问 Gitlab 实例
• 其中的 secret 和 key，就是 7.3 章节设置的对象存储连接 secret 的名称，以及其中的 key，注意 chart 实例和连接 secret 需要在同一 namespace
• 这里对象存储使用统一连接配置，也可以分开配置，参考：doc/charts/globals.md · master · GitLab.org / charts / GitLab Chart · GitLab
• 6.0.0 对应 gitlab-jh v15.0.0 版本
• 默认情况下 initContainers 会使用 busybox:latest 镜像，busybox 新版的镜像部分命令与 bug（至少 1.30 版本 nslookup 命令有问题），所以这里指定使用 1.28.4 版本
• 根据网络拉取镜像的情况，可能时间等待较久或者部分 pod 启动失败，解决镜像拉取问题即可

处理完成后发现 gitlab-runner 的 pod 还是不正常的

$ kubectl get po|grep -v Running
NAME                                                   READY   STATUS             RESTARTS   AGE
gitlab-issuer-1-7dt84                                  0/1     Completed          0          34m
gitlab-minio-create-buckets-1-r8pzf                    0/1     Completed          0          34m
gitlab-migrations-1-2scb6                              0/1     Completed          1          34m
gitlab-gitlab-runner-764cdf7c6d-xfw5t                  0/1     CrashLoopBackOff   7          34m

$ kubectl logs gitlab-gitlab-runner-764cdf7c6d-xfw5t
...                                                
Merging configuration from template file "/configmaps/config.template.toml" 
ERROR: Registering runner... failed                 runner=GgfOlfZW status=couldn't execute POST against http://gitlab.example.com/api/v4/runners: Post "http://gitlab.example.com/api/v4/runners": dial tcp: lookup gitlab.example.com on 10.43.0.10:53: no such host
PANIC: Failed to register the runner.    

原因是 runner 的 pod 无法解析 gitlab.example.com，解决方法如下：

首先查看 gitlab.example.com 地址

$ kubectl get ingress -lrelease=gitlab
NAME                        CLASS   HOSTS                  ADDRESS       PORTS   AGE
gitlab-kas                  kong    kas.example.com        10.10.10.61   80      15h
gitlab-registry             kong    registry.example.com   10.10.10.61   80      15h
gitlab-webservice-default   kong    gitlab.example.com     10.10.10.61   80      15h

然后在集群 coredns 中添加 hosts 解析

$ kubectl -n kube-system edit configmaps coredns
...
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          10.10.10.61 gitlab.example.com
          10.10.10.61 registry.example.com
          10.10.10.61 kas.example.com
          ttl 60    
          reload 15s 
          fallthrough
        }               
...

# 删除 coredns pod 生效
$ kubectl get pod -n kube-system
NAME                                      READY   STATUS      RESTARTS   AGE
helm-install-traefik-g4xsp                0/1     Completed   0          151m
metrics-server-86cbb8457f-8wghh           1/1     Running     0          151m
local-path-provisioner-5ff76fc89d-r87zg   1/1     Running     0          151m
coredns-854c77959c-nm28h                  1/1     Running     0          151m

$ kubectl -n kube-system delete pod coredns-854c77959c-nm28h
pod "coredns-854c77959c-nm28h" deleted

再次查看 runner 状态已正常：

[root@localhost ~]# kubectl get pod|grep runner
gitlab-gitlab-runner-764cdf7c6d-xfw5t                  1/1     Running            5 (2m42s ago)   19m

全部 pod 与 ceph rbd pvc 正常

[root@localhost ~]# kubectl get pod
NAME                                                   READY   STATUS      RESTARTS      AGE
csi-rbdplugin-provisioner-74888444cc-wj5nv             0/7     Pending     0             16h
csi-rbdplugin-provisioner-74888444cc-tgmj6             0/7     Pending     0             16h
csi-rbdplugin-bdgnx                                    3/3     Running     0             16h
csi-rbdplugin-provisioner-74888444cc-mjb9d             7/7     Running     0             16h
svclb-gitlab-nginx-ingress-controller-j2tgw            3/3     Running     0             51m
gitlab-nginx-ingress-controller-6545975dbf-d7kpv       1/1     Running     0             51m
gitlab-certmanager-cainjector-74cbc84b8b-d2c9v         1/1     Running     0             51m
gitlab-certmanager-57c4557849-gb79z                    1/1     Running     0             51m
gitlab-nginx-ingress-defaultbackend-7b4f8d7847-zqrwz   1/1     Running     0             51m
gitlab-nginx-ingress-controller-6545975dbf-wfl2f       1/1     Running     0             51m
gitlab-certmanager-webhook-59d745756c-pr2cr            1/1     Running     0             51m
gitlab-gitlab-exporter-9c99c79cd-7mdwh                 1/1     Running     0             51m
gitlab-toolbox-579746c789-wjhdx                        1/1     Running     0             51m
gitlab-gitlab-shell-77d89bf64-zrrm6                    1/1     Running     0             51m
gitlab-redis-master-0                                  2/2     Running     0             41m
gitlab-gitlab-shell-77d89bf64-hrc9z                    1/1     Running     0             51m
gitlab-prometheus-server-77b5cc946-zsk4k               2/2     Running     0             51m
gitlab-postgresql-0                                    2/2     Running     0             51m
gitlab-gitaly-0                                        1/1     Running     0             51m
gitlab-sidekiq-all-in-1-v2-79df4567db-dskb5            1/1     Running     0             51m
gitlab-webservice-default-5bbc95b7d5-rvwnr             2/2     Running     0             51m
gitlab-webservice-default-5bbc95b7d5-88jxv             2/2     Running     0             51m
gitlab-gitlab-runner-764cdf7c6d-xfw5t                  1/1     Running     5 (35m ago)   51m
gitlab-issuer-2--1-xj6qr                               0/1     Completed   0             46s
gitlab-registry-67fb69c6cf-6jmfw                       1/1     Running     0             47s
gitlab-registry-67fb69c6cf-6bldq                       1/1     Running     0             36s
[root@localhost ~]# 
[root@localhost ~]# kubectl get pvc
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
gitlab-prometheus-server           Bound    pvc-bc5b38ea-966e-4f82-a845-bd01a9181330   8Gi        RWO            csi-rbd-sc     52m
redis-data-gitlab-redis-master-0   Bound    pvc-98cd04fb-272f-4873-b065-a91b4aa8eb74   8Gi        RWO            csi-rbd-sc     52m
repo-data-gitlab-gitaly-0          Bound    pvc-33a0c2f6-b086-4ef6-99dc-f262398468cc   50Gi       RWO            csi-rbd-sc     52m
data-gitlab-postgresql-0           Bound    pvc-88e69f3e-089a-417b-a8b3-43ed9034c858   8Gi        RWO            csi-rbd-sc     52m
[root@localhost ~]# 

可以使用 helm get values gitlab > gitlab.yaml 导出 yaml 资源文件

7.5 访问 gitlab 实例

获取 ip

$ kubectl get ingress -lrelease=gitlab
NAME                        CLASS   HOSTS                  ADDRESS       PORTS   AGE
gitlab-kas                  kong    kas.example.com        10.10.10.61   80      15h
gitlab-registry             kong    registry.example.com   10.10.10.61   80      15h
gitlab-webservice-default   kong    gitlab.example.com     10.10.10.61   80      15h

本机设置 hosts

10.10.10.61 gitlab.example.com

获取 root 初始密码

kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode; echo

访问地址：http://gitlab.example.com

gitlab runner 正常注册

7.6 测试 runner

再创建测试项目 p1

添加 .gitlab-ci.yml 文件，提交：

stages:
  - build
  - deploy

variables:
  projectname: "p1"

build:
  stage: build
  retry: 2
  before_script:
    - echo "before_script in build job"
  script:
    - df && ls -l
    - echo ${projectname}
    - echo "script in build job"
  after_script:
    - echo "after_script in build job"

deploy:
  stage: deploy
  script:
    - echo "deploy"
  when: manual
  only:
    - tags

提交后 runner 会自动运行流水线，运行成功：

流水线结果：

Running with gitlab-runner 15.0.0 (c6bb62f6)
  on gitlab-gitlab-runner-764cdf7c6d-xfw5t xfXku1BF
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: default
Using Kubernetes executor with image ubuntu:18.04 ...
Using attach strategy to execute scripts...
Preparing environment
00:27
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-xfxku1bf-project-2-concurrent-0j8gsv via gitlab-gitlab-runner-764cdf7c6d-xfw5t...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/g1/p1/.git/
Created fresh repository.
Checking out b23aedc7 as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ echo "before_script in build job"
before_script in build job
$ df && ls -l
Filesystem              1K-blocks     Used Available Use% Mounted on
overlay                  52403200 11324604  41078596  22% /
tmpfs                       65536        0     65536   0% /dev
tmpfs                    16430384        0  16430384   0% /sys/fs/cgroup
/dev/mapper/centos-root  52403200 11324604  41078596  22% /builds
shm                         65536        0     65536   0% /dev/shm
tmpfs                    32860768       12  32860756   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    16430384        0  16430384   0% /proc/acpi
tmpfs                    16430384        0  16430384   0% /proc/scsi
tmpfs                    16430384        0  16430384   0% /sys/firmware
total 8
-rw-rw-rw- 1 root root 6180 Jun  8 02:55 README.md
$ echo ${projectname}
p1
$ echo "script in build job"
script in build job
Running after_script
00:01
Running after script...
$ echo "after_script in build job"
after_script in build job
Job succeeded

在 10.10.10.60 主机查看 minio 存储目录，发现已保存 pipeline 日志：

[root@localhost data]# pwd
/opt/minio/data
[root@localhost data]# 
[root@localhost data]# du * -sh
8.0K    gitlab-artifacts-storage
0       gitlab-backup-upload
0       gitlab-dependency-proxy
0       gitlab-external-diffs
0       gitlab-lfs-storage
0       gitlab-packages-storage
0       gitlab-pages
0       gitlab-registry
0       gitlab-terraform-state
0       gitlab-tmp-storage
0       gitlab-uploads-storage
[root@localhost data]# 
[root@localhost data]# cat gitlab-artifacts-storage/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2022_06_08/1/1/job.log 
Running with gitlab-runner 15.0.0 (c6bb62f6)
  on gitlab-gitlab-runner-764cdf7c6d-xfw5t xfXku1BF
Preparing the "kubernetes" executor
Using Kubernetes namespace: default
Using Kubernetes executor with image ubuntu:18.04 ...
Using attach strategy to execute scripts...
Preparing environment
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
        ContainersNotReady: "containers with unready status: [build helper]"
        ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-xfxku1bf-project-2-concurrent-0j8gsv via gitlab-gitlab-runner-764cdf7c6d-xfw5t...

Getting source from Git repository
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/g1/p1/.git/
Created fresh repository.
Checking out b23aedc7 as main...

Skipping Git submodules setup

Executing "step_script" stage of the job script
$ echo "before_script in build job"
before_script in build job
$ df && ls -l
Filesystem              1K-blocks     Used Available Use% Mounted on
overlay                  52403200 11324604  41078596  22% /
tmpfs                       65536        0     65536   0% /dev
tmpfs                    16430384        0  16430384   0% /sys/fs/cgroup
/dev/mapper/centos-root  52403200 11324604  41078596  22% /builds
shm                         65536        0     65536   0% /dev/shm
tmpfs                    32860768       12  32860756   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    16430384        0  16430384   0% /proc/acpi
tmpfs                    16430384        0  16430384   0% /proc/scsi
tmpfs                    16430384        0  16430384   0% /sys/firmware
total 8
-rw-rw-rw- 1 root root 6180 Jun  8 02:55 README.md
$ echo ${projectname}
p1
$ echo "script in build job"
script in build job

Running after_script
Running after script...
$ echo "after_script in build job"
after_script in build job

Job succeeded
[root@localhost data]# 

7.7 测试 Registry 服务

修改 .gitlab-ci.yml 文件：

build:
  stage: build
  image:
    # name: gcr.io/kaniko-project/executor:debug
    name: willdockerhub/kaniko-executor:debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
      --skip-tls-verify
  rules:
    - if: $CI_COMMIT_TAG

• 使用 kaniko 打包并提交镜像到 Registry

添加 Dockerfile：

FROM nginx:latest
ENV TEST=1

添加一个 tag，触发 pipeline：

job 运行结果：

Running with gitlab-runner 15.0.0 (c6bb62f6)
  on gitlab-gitlab-runner-764cdf7c6d-xfw5t xfXku1BF
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: default
Using Kubernetes executor with image willdockerhub/kaniko-executor:debug ...
Using attach strategy to execute scripts...
Preparing environment
00:03
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0hmp5l to be running, status is Pending
Running on runner-xfxku1bf-project-2-concurrent-0hmp5l via gitlab-gitlab-runner-764cdf7c6d-xfw5t...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/g1/p1/.git/
Created fresh repository.
Checking out 6516a100 as v1.0.2...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:32
$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}" --skip-tls-verify
INFO[0000] Retrieving image manifest nginx:latest       
INFO[0000] Retrieving image nginx:latest from registry index.docker.io 
INFO[0006] Built cross stage deps: map[]                
INFO[0006] Retrieving image manifest nginx:latest       
INFO[0006] Returning cached image manifest              
INFO[0006] Executing 0 build triggers                   
INFO[0006] Skipping unpacking as no commands require it. 
INFO[0006] ENV LEFFSS=1                                 
INFO[0006] Pushing image to registry.example.com/g1/p1:v1.0.2 
INFO[0032] Pushed registry.example.com/g1/p1@sha256:0a33f3ad16af8e151072c89d98a624f017c9eb36c60a4651f2cc73a76f63acaf 
Job succeeded

正常提交镜像：

在 10.10.10.60 主机查看 minio 存储目录，发现已保存 registry 数据：

[root@localhost data]# pwd
/opt/minio/data
[root@localhost data]# 
[root@localhost data]# du * -sh
60K     gitlab-artifacts-storage
0       gitlab-backup-upload
0       gitlab-dependency-proxy
0       gitlab-external-diffs
0       gitlab-lfs-storage
0       gitlab-packages-storage
0       gitlab-pages
55M     gitlab-registry
0       gitlab-terraform-state
0       gitlab-tmp-storage
0       gitlab-uploads-storage

7.8 暴露 gitlab-shell tcp 服务端口

在 10.10.10.60 主机上设置 hosts，并生成 ssh key：

[root@localhost ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# ceph-node
10.10.10.51 ceph01
10.10.10.52 ceph02
10.10.10.53 ceph03
10.10.10.61 gitlab.example.com
[root@localhost ~]# 
[root@localhost ~]# ssh-keygen -t rsa -b 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:a5ZMokwuQfNysoUk2zKeub9WiIqt27o/vyH76sMmFrk root@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|. +              |
| * +             |
|+ O * . S        |
|.*o% o + o       |
|o** *   *        |
|+EB= . o         |
|*@OO=.           |
+----[SHA256]-----+
[root@localhost ~]# 

把公钥 ~/.ssh/id_rsa.pub 添加到 gitlab 中：

在 10.10.10.60 主机上测试联通性：

[root@localhost ~]# ssh -T git@gitlab.example.com
ssh: connect to host gitlab.example.com port 22: Connection refused

ssh方式无法连接，原因是当使用 gitlab chart 自带的 nginx-ingress-controller 时，是通过 configmap 暴露 gitlab-shell 的 tcp 端口 22 的：

[root@localhost ~]# kubectl get configmaps gitlab-nginx-ingress-tcp -o yaml
apiVersion: v1
data:
  "22": 'default/gitlab-gitlab-shell:22::'
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: gitlab
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2022-06-08T07:56:27Z"
  labels:
    app: gitlab-shell
    app.kubernetes.io/managed-by: Helm
    chart: gitlab-shell-6.0.0
    heritage: Helm
    release: gitlab
  name: gitlab-nginx-ingress-tcp
  namespace: default
  resourceVersion: "91909"
  uid: 049e3173-ed02-4b2b-8d71-0a43b49017b4
[root@localhost ~]# 

而换用 kong-ingress-controller 后，虽然 configmap 配置文件 gitlab-nginx-ingress-tcp 还是创建了，但是 kong-ingress-controller 是无法识别的，这也导致 gitlab-shell 的 22 端口未暴露出来，解决方法是使用 kong-ingress-controller 支持的方式重新暴露 tcp 端口。

参考文档：TCPIngress with Kong - v2.3.x | Kong Docs (konghq.com)

7.8.1 配置 kong-ingress-controller

首先配置 kong-ingress-controller，新增 22 tcp 端口

$ kubectl patch deploy -n kong ingress-kong --patch '{
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "proxy",
            "env": [
              {
                "name": "KONG_STREAM_LISTEN",
                "value": "0.0.0.0:22"
              }
            ],
            "ports": [
              {
                "containerPort": 22,
                "name": "stream22",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    }
  }
}'

$ kubectl patch service -n kong kong-proxy --patch '{
  "spec": {
    "ports": [
      {
        "name": "stream22",
        "port": 22,
        "protocol": "TCP",
        "targetPort": 22
      }
    ]
  }
}'

7.8.2 创建 tcpingress

$ echo "apiVersion: configuration.konghq.com/v1beta1
kind: TCPIngress
metadata:
  name: gitlab-shell
  annotations:
    kubernetes.io/ingress.class: kong
spec:
  rules:
  - port: 22
    backend:
      serviceName: gitlab-gitlab-shell
      servicePort: 22
" | kubectl apply -f -

$ kubectl get tcpingress
NAME           ADDRESS       AGE
gitlab-shell   10.10.10.61   27m

$ kubectl get tcpingress gitlab-shell -o yaml
apiVersion: configuration.konghq.com/v1beta1
kind: TCPIngress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"configuration.konghq.com/v1beta1","kind":"TCPIngress","metadata":{"annotations":{"kubernetes.io/ingress.class":"kong"},"name":"gitlab-shell","namespace":"default"},"spec":{"rules":[{"backend":{"serviceName":"gitlab-gitlab-shell","servicePort":22},"port":22}]}}
    kubernetes.io/ingress.class: kong
  creationTimestamp: "2022-06-09T00:30:42Z"
  generation: 1
  name: gitlab-shell
  namespace: default
  resourceVersion: "258906"
  uid: 865f3885-f365-4d0d-980d-7c84f6524509
spec:
  rules:
  - backend:
      serviceName: gitlab-gitlab-shell
      servicePort: 22
    port: 22
status:
  loadBalancer:
    ingress:
    - ip: 10.10.10.61

7.8.3 测试 ssh 拉取提交

再次在 10.10.10.60 主机上测试：

[root@localhost ~]# ssh -T git@gitlab.example.com
Welcome to GitLab, @root!
[root@localhost ~]# 
[root@localhost ~]# git clone git@gitlab.example.com:gitlab-instance-4bfa23d0/p1.git
Cloning into 'p1'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (6/6), done.
[root@localhost ~]# 
[root@localhost ~]# cd p1/
[root@localhost p1]# echo xx > test.txt
[root@localhost p1]# 
[root@localhost p1]# git status
# On branch main
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       test.txt
nothing added to commit but untracked files present (use "git add" to track)
[root@localhost p1]# 
[root@localhost p1]# git add .
[root@localhost p1]# git commit -m 'add test.txt'
[main 99e3e3c] add test.txt
 1 file changed, 1 insertion(+)
 create mode 100644 test.txt
[root@localhost p1]# 
[root@localhost p1]# git push origin main
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 279 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To git@gitlab.example.com:gitlab-instance-4bfa23d0/p1.git
   c11db43..99e3e3c  main -> main
[root@localhost p1]# 

可以看到已经能够成功使用 ssh 方式拉取与提交了

8. 使用外部服务

可使用外部 PostgreSQL、Redis、Gitaly 等，具体参考文档：Advanced configuration | GitLab

8.1 使用外部 redis

1）搭建 redis

仅测试，所以直接在 centos 7.9 下使用 yum 安装

yum install epel-release -y
yum install redis -y

修改配置，开启远程访问

vi /etc/redis.conf，修改或者添加以下选项

bind 10.10.10.61
requirepass redis123456
protected-mode no

重启 redis 服务

$ systemctl restart redis

$ ss -tnlp|grep 6379
LISTEN     0      511    10.10.10.61:6379                     *:*                   users:(("redis-server",pid=1621,fd=4))

2）连接外部 redis

官方文档：Configure this chart with External Redis | GitLab

首先创建存储 redis 密码的 secret

方式一：yaml 文件

$ echo -n redis123456 | base64
cmVkaXMxMjM0NTY=

vi gitlab-redis-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-redis-secret
data:
  redis-password: cmVkaXMxMjM0NTY=

$ kubectl apply -f gitlab-redis-secret.yaml

$ kubectl get secret gitlab-redis-secret -ojsonpath='{.data.redis-password}' | base64 --decode; echo
redis123456

• data 中可以指定多个 key:value

方式二：文件形式

$ echo -n redis123456 > redis-password

$ kubectl create secret generic gitlab-redis-secret --from-file=./redis-password

$ kubectl get secret gitlab-redis-secret1 -ojsonpath='{.data.redis-password}' | base64 --decode; echo
redis123456

• --from-file 可以指定多个文件，文件名为 key，文件内容为 value（不需转 base64 编码）

安装 chart

helm install gitlab gitlab/gitlab \
  --set global.hosts.domain=example.com \
  --set global.hosts.https=false \
  --set global.ingress.tls.enabled=false \
  --set redis.install=false \
  --set global.redis.host=10.10.10.61 \
  --set global.redis.port=6379 \
  --set global.redis.password.secret=gitlab-redis-secret \
  --set global.redis.password.key=redis-password \
  --set certmanager-issuer.email=me@example.com

• 也可以使用哨兵模式，具体参考官方文档

其他 PostgreSQL、Gitaly 基本类似，具体参考官方文档即可，这里就不演示了

9. 升级极狐gitlab

首先导出配置

helm get values gitlab > gitlab.yaml

然后升级到 6.0.2（对应 gitlab-jh v15.0.1 版本）

helm upgrade gitlab gitlab-jh/gitlab \
  --version 6.0.2 \
  -f gitlab.yaml \
  --set gitlab.migrations.enabled=true \
  --set ...

• --set ... 可选新增参数

如果只是更新下配置则可以使用以下命令：

helm upgrade gitlab gitlab-jh/gitlab \
  --version <原版本> \
  -f gitlab.yaml \
  --set gitlab.migrations.enabled=false