helm 安装极狐 gitlab(集成外部 kong ingress controller + ceph + 对象存储)
helm 安装极狐 gitlab(集成外部 kong ingress controller + ceph + 对象存储)
1. 环境信息
1.1 主机信息
主机 | 配置 | 角色 |
---|---|---|
10.10.10.51(ceph01) | 1核3GB,100GB系统盘,5*50GB数据裸盘 | mon,osd,mgr,admin |
10.10.10.52(ceph02) | 1核3GB,100GB系统盘,5*50GB数据裸盘 | mon,osd,mgr,mds |
10.10.10.53(ceph03) | 1核3GB,100GB系统盘,5*50GB数据裸盘 | mon,osd,rgw |
10.10.10.60 | 1核4GB,100GB系统盘 | minio,提供兼容 s3 的对象存储 |
10.10.10.61 | 8核32GB,100G系统盘 | k8s,helm,安装极狐gitlab v15.0.0,内存建议16GB以上 |
- 操作系统 centos 7.9 x64
1.2 环境初始化
echo "设置时钟同步"
yum install -y chrony
systemctl start chronyd.service
systemctl enable chronyd.service
echo "关闭防火墙"
systemctl stop firewalld
systemctl disable firewalld
echo "关闭 selinux"
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
getenforce
echo "优化 ssh 登录速度"
sed -i 's/#UseDNS yes/UseDNS no/g' /etc/ssh/sshd_config
systemctl restart sshd
echo "安装基础软件 lrzsz, epel-release"
yum install epel-release lrzsz -y
1.3 升级内核
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
# 安装ELRepo
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
# 载入elrepo-kernel元数据
yum --disablerepo=\* --enablerepo=elrepo-kernel repolist
# 查看可用的rpm包
yum --disablerepo=\* --enablerepo=elrepo-kernel list kernel*
# 安装长期支持版本的kernel
yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt.x86_64
# 删除旧版本工具包
yum remove kernel-tools-libs.x86_64 kernel-tools.x86_64 -y
# 安装新版本工具包
yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt-tools.x86_64
#查看默认启动顺序
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
#默认启动的顺序是从0开始,新内核是从头插入(目前位置在0,而4.4.4的是在1),所以需要选择0。
grub2-set-default 0
reboot
- 当前最新稳定内核版本
CentOS Linux (5.4.180-1.el7.elrepo.x86_64) 7 (Core)
2. 安装 minio
涉及主机:
- 10.10.10.60
2.1 安装
下载 minio server
yum install wget -y
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
mv minio /usr/local/bin/minio
minio -v
minio version RELEASE.2022-02-26T02-54-46Z
下载 minio client
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/mc
mc -v
mc version RELEASE.2022-02-26T03-58-31Z
配置 minio 数据目录
mkdir -p /opt/minio/data
mkdir -p /opt/minio/logs
创建 start_minio_server.sh 启动脚本
#!/bin/bash
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123456
nohup /usr/local/bin/minio server /opt/minio/data > /opt/minio/logs/minio.log 2>&1 &
启动 minio server
chmod +x start_minio_server.sh
./start_minio_server.sh
- 默认对外服务端口 9000
为客户端设置别名
mc alias set myminio http://10.10.10.60:9000 minio minio123456
2.2 配置桶
配置 gitlab 需要用到的存储桶
mc mb myminio/gitlab-cache # runner cache
mc mb myminio/gitlab-artifacts-storage
mc mb myminio/gitlab-external-diffs
mc mb myminio/gitlab-lfs-storage
mc mb myminio/gitlab-uploads-storage
mc mb myminio/gitlab-packages-storage
mc mb myminio/gitlab-dependency-proxy
mc mb myminio/gitlab-terraform-state
# mc mb myminio/gitlab-pages # pages 功能一般是关闭的
mc mb myminio/gitlab-backup-upload
mc mb myminio/gitlab-tmp-storage
mc mb myminio/gitlab-registry
3. 安装 ceph 集群
涉及主机:
- 10.10.10.51
- 10.10.10.52
- 10.10.10.53
版本:luminous
参考:ceph-deploy 安装 ceph - leffss - 博客园 (cnblogs.com)
4. 安装 k8s 集群
涉及主机:
- 10.10.10.61
版本:v1.22.9+k3s1
参考:k8s 测试环境搭建(k3s) - leffss - 博客园 (cnblogs.com)
如果需要搭建完整版集群,请参考:easzlab/kubeasz
5. k8s集成ceph rbd(StorageClass方式)
首先删除 k3s 默认集成的 StorageClass:
kubectl delete sc local-path
kubectl -n kube-system delete deployment local-path-provisioner
然后参考:k8s集成ceph rbd(StorageClass方式)
最后验证 pvc 时,发现新建的 pvc 一直 pending 状态:
[root@localhost ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-pvc-test1 Pending ceph-storageclass 7s
[root@localhost ~]# kubectl describe pvc ceph-pvc-test1
Name: ceph-pvc-test1
Namespace: default
StorageClass: ceph-storageclass
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-class: ceph-storageclass
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 7s (x2 over 15s) persistentvolume-controller no volume plugin matched name: kubernetes.io/rbd
- 原因是相对于标准 k8s 集群,k3s 为了精简二进制文件大小删除了不少 volume plugins ,这里就缺少插件:
kubernetes.io/rbd
,参考:Rancher Docs: Volumes and Storage
这里我们就不采用上面的方法,而是安装 ceph 官方的插件:ceph/ceph-csi: CSI driver for Ceph 解决
5.1 部署 RBD provisioner
参考:ceph-csi/deploy-rbd.md#deployment-with-kubernetes
删除前面创建的 pvc 与 StorageClass,根据 k8s 集群版本下载对应 ceph-csi 版本,这里我们下载最新版本:v3.6.1
yum install wget -y
wget https://github.com/ceph/ceph-csi/archive/refs/tags/v3.6.1.tar.gz
mv v3.6.1.tar.gz ceph-csi-3.6.1.tar.gz
tar zxvf ceph-csi-3.6.1.tar.gz
cd ceph-csi-3.6.1/deploy/rbd/kubernetes
查看需要用到的镜像:
[root@localhost kubernetes]# grep -rn image .|grep -v imagePullPolicy
./csi-rbdplugin-provisioner.yaml:50: image: k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
./csi-rbdplugin-provisioner.yaml:70: image: k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
./csi-rbdplugin-provisioner.yaml:84: image: k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
./csi-rbdplugin-provisioner.yaml:98: image: k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
./csi-rbdplugin-provisioner.yaml:115: image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin-provisioner.yaml:169: image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin-provisioner.yaml:189: image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin.yaml:31: image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
./csi-rbdplugin.yaml:53: image: quay.io/cephcsi/cephcsi:v3.6.1
./csi-rbdplugin.yaml:125: image: quay.io/cephcsi/cephcsi:v3.6.1
[root@localhost kubernetes]#
去重整理:
k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
quay.io/cephcsi/cephcsi:v3.6.1
其中 k8s.gcr.io 开头的因为网络原因,都下载不下来,我们可以使用国内镜像替代:
docker pull registry.aliyuncs.com/google_containers/csi-provisioner:v3.1.0
docker pull registry.aliyuncs.com/google_containers/csi-snapshotter:v5.0.1
docker pull registry.aliyuncs.com/google_containers/csi-attacher:v3.4.0
docker pull registry.aliyuncs.com/google_containers/csi-resizer:v1.4.0
docker pull registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.4.0
然后重新打标签:
docker tag registry.aliyuncs.com/google_containers/csi-provisioner:v3.1.0 k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
docker tag registry.aliyuncs.com/google_containers/csi-snapshotter:v5.0.1 k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
docker tag registry.aliyuncs.com/google_containers/csi-attacher:v3.4.0 k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
docker tag registry.aliyuncs.com/google_containers/csi-resizer:v1.4.0 k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
docker tag registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.4.0 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
应用 RBAC 权限:
kubectl create -f csi-provisioner-rbac.yaml
kubectl create -f csi-nodeplugin-rbac.yaml
应用 PSP 权限:
kubectl create -f csi-provisioner-psp.yaml
kubectl create -f csi-nodeplugin-psp.yaml
修改 csi-config-map.yaml
配置:
#
# /!\ DO NOT MODIFY THIS FILE
#
# This file has been automatically generated by Ceph-CSI yamlgen.
# The source for the contents can be found in the api/deploy directory, make
# your modifications there.
#
---
apiVersion: v1
kind: ConfigMap
metadata:
name: "ceph-csi-config"
data:
config.json: |-
[
{
"clusterID": "c1dda3a3-c30a-4594-b87c-3de2a89d0214",
"monitors": [
"10.10.10.51:6789",
"10.10.10.52:6789",
"10.10.10.53:6789"
]
}
]
- clusterID 在 ceph 集群配置文件 /etc/ceph/ceph.conf 中获得
应用:
kubectl create -f csi-config-map.yaml
创建 ceph-config.yaml
配置:
---
# This is a sample configmap that helps define a Ceph configuration as required
# by the CSI plugins.
# Sample ceph.conf available at
# https://github.com/ceph/ceph/blob/master/src/sample.ceph.conf Detailed
# documentation is available at
# https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/
apiVersion: v1
kind: ConfigMap
data:
ceph.conf: |
[global]
fsid = c1dda3a3-c30a-4594-b87c-3de2a89d0214
public_network = 10.10.10.0/24
cluster_network = 10.10.10.0/24
mon_initial_members = ceph01
mon_host = 10.10.10.51,10.10.10.52,10.10.10.53
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
# keyring is a required key and its value should be empty
keyring: |
metadata:
name: ceph-config
- ceph.conf 的内容就是 ceph 节点上 /etc/ceph/ceph.conf
kubectl create -f ceph-config.yaml
注释掉 ceph-csi-encryption-kms-config
相关内容
[root@localhost kubernetes]# grep -rn 'ceph-csi-encryption-kms-config' .
./csi-rbdplugin-provisioner.yaml:160: - name: ceph-csi-encryption-kms-config
./csi-rbdplugin-provisioner.yaml:161: mountPath: /etc/ceph-csi-encryption-kms-config/
./csi-rbdplugin-provisioner.yaml:230: - name: ceph-csi-encryption-kms-config
./csi-rbdplugin-provisioner.yaml:232: name: ceph-csi-encryption-kms-config
./csi-rbdplugin.yaml:107: - name: ceph-csi-encryption-kms-config
./csi-rbdplugin.yaml:108: mountPath: /etc/ceph-csi-encryption-kms-config/
./csi-rbdplugin.yaml:188: - name: ceph-csi-encryption-kms-config
./csi-rbdplugin.yaml:190: name: ceph-csi-encryption-kms-config
应用 provisioner:
kubectl create -f csi-rbdplugin-provisioner.yaml
kubectl create -f csi-rbdplugin.yaml
查看验证
[root@localhost kubernetes]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/csi-rbdplugin-provisioner-74888444cc-d5kxf 0/7 Pending 0 20m
pod/csi-rbdplugin-provisioner-74888444cc-vqcrw 0/7 Pending 0 20m
pod/csi-rbdplugin-ds7w7 3/3 Running 0 20m
pod/csi-rbdplugin-provisioner-74888444cc-b6dbn 7/7 Running 0 20m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 31h
service/csi-rbdplugin-provisioner ClusterIP 10.43.21.47 <none> 8080/TCP 20m
service/csi-metrics-rbdplugin ClusterIP 10.43.154.94 <none> 8080/TCP 20m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/csi-rbdplugin 1 1 1 1 1 <none> 20m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/csi-rbdplugin-provisioner 1/3 3 1 20m
NAME DESIRED CURRENT READY AGE
replicaset.apps/csi-rbdplugin-provisioner-74888444cc 3 3 1 20m
- 其中 csi-rbdplugin-provisioner 有 2 个 pod 一直是 pending 状态,原因是设置了 pod 亲和性,3个副本不能调度到同一 node 上运行,我们这里的 k8s 集群为单节点,仅测试而已,忽略即可
5.2 创建 StorageClass
创建 csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
namespace: default
stringData:
# Key values correspond to a user name and its key, as defined in the
# ceph cluster. User ID should have required access to the 'pool'
# specified in the storage class
userID: kubernetes
userKey: AQAnEZ9ivF6ZExAAnrZhVVxwtSSoaoWM69SuTA==
# Encryption passphrase
encryptionPassphrase: test_passphrase
其中的userID与userKey通过以下方式获取
创建存储池:
ceph osd pool create k8s 128 128
rbd pool init k8s
创建并授权用户
$ ceph auth get-or-create client.kubernetes mon 'allow r' osd 'profile rbd pool=k8s'
[client.kubernetes]
key = AQAnEZ9ivF6ZExAAnrZhVVxwtSSoaoWM69SuTA==
- kubernetes 为 userID,key 为 userKey
应用:
kubectl apply -f csi-rbd-secret.yaml
创建 storageclass.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
# If topology based provisioning is desired, delayed provisioning of
# PV is required and is enabled using the following attribute
# For further information read TODO<doc>
# volumeBindingMode: WaitForFirstConsumer
parameters:
# (required) String representing a Ceph cluster to provision storage from.
# Should be unique across all Ceph clusters in use for provisioning,
# cannot be greater than 36 bytes in length, and should remain immutable for
# the lifetime of the StorageClass in use.
# Ensure to create an entry in the configmap named ceph-csi-config, based on
# csi-config-map-sample.yaml, to accompany the string chosen to
# represent the Ceph cluster in clusterID below
clusterID: c1dda3a3-c30a-4594-b87c-3de2a89d0214
# (optional) If you want to use erasure coded pool with RBD, you need to
# create two pools. one erasure coded and one replicated.
# You need to specify the replicated pool here in the `pool` parameter, it is
# used for the metadata of the images.
# The erasure coded pool must be set as the `dataPool` parameter below.
# dataPool: <ec-data-pool>
# (required) Ceph pool into which the RBD image shall be created
# eg: pool: rbdpool
pool: k8s
# (optional) RBD image features, CSI creates image with image-format 2 CSI
# RBD currently supports `layering`, `journaling`, `exclusive-lock`,
# `object-map`, `fast-diff`, `deep-flatten` features.
# Refer https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#image-features
# for image feature dependencies.
# imageFeatures: layering,journaling,exclusive-lock,object-map,fast-diff
imageFeatures: "layering"
# (optional) Specifies whether to try other mounters in case if the current
# mounter fails to mount the rbd image for any reason. True means fallback
# to next mounter, default is set to false.
# Note: tryOtherMounters is currently useful to fallback from krbd to rbd-nbd
# in case if any of the specified imageFeatures is not supported by krbd
# driver on node scheduled for application pod launch, but in the future this
# should work with any mounter type.
# tryOtherMounters: false
# (optional) mapOptions is a comma-separated list of map options.
# For krbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
# For nbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
# Format:
# mapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
# An empty mounter field is treated as krbd type for compatibility.
# eg:
# mapOptions: "krbd:lock_on_read,queue_depth=1024;nbd:try-netlink"
# (optional) unmapOptions is a comma-separated list of unmap options.
# For krbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
# For nbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
# Format:
# unmapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
# An empty mounter field is treated as krbd type for compatibility.
# eg:
# unmapOptions: "krbd:force;nbd:force"
# The secrets have to contain Ceph credentials with required access
# to the 'pool'.
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
# (optional) Specify the filesystem type of the volume. If not specified,
# csi-provisioner will set default as `ext4`.
csi.storage.k8s.io/fstype: ext4
# (optional) uncomment the following to use rbd-nbd as mounter
# on supported nodes
# mounter: rbd-nbd
# (optional) ceph client log location, eg: rbd-nbd
# By default host-path /var/log/ceph of node is bind-mounted into
# csi-rbdplugin pod at /var/log/ceph mount path. This is to configure
# target bindmount path used inside container for ceph clients logging.
# See docs/rbd-nbd.md for available configuration options.
# cephLogDir: /var/log/ceph
# (optional) ceph client log strategy
# By default, log file belonging to a particular volume will be deleted
# on unmap, but you can choose to just compress instead of deleting it
# or even preserve the log file in text format as it is.
# Available options `remove` or `compress` or `preserve`
# cephLogStrategy: remove
# (optional) Prefix to use for naming RBD images.
# If omitted, defaults to "csi-vol-".
# volumeNamePrefix: "foo-bar-"
# (optional) Instruct the plugin it has to encrypt the volume
# By default it is disabled. Valid values are "true" or "false".
# A string is expected here, i.e. "true", not true.
# encrypted: "true"
# (optional) Use external key management system for encryption passphrases by
# specifying a unique ID matching KMS ConfigMap. The ID is only used for
# correlation to configmap entry.
# encryptionKMSID: <kms-config-id>
# Add topology constrained pools configuration, if topology based pools
# are setup, and topology constrained provisioning is required.
# For further information read TODO<doc>
# topologyConstrainedPools: |
# [{"poolName":"pool0",
# "dataPool":"ec-pool0" # optional, erasure-coded pool for data
# "domainSegments":[
# {"domainLabel":"region","value":"east"},
# {"domainLabel":"zone","value":"zone1"}]},
# {"poolName":"pool1",
# "dataPool":"ec-pool1" # optional, erasure-coded pool for data
# "domainSegments":[
# {"domainLabel":"region","value":"east"},
# {"domainLabel":"zone","value":"zone2"}]},
# {"poolName":"pool2",
# "dataPool":"ec-pool2" # optional, erasure-coded pool for data
# "domainSegments":[
# {"domainLabel":"region","value":"west"},
# {"domainLabel":"zone","value":"zone1"}]}
# ]
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
- clusterID 与 pool 根据实际情况填写
应用:
kubectl apply -f storageclass.yaml
查看 sc
[root@localhost ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-rbd-sc rbd.csi.ceph.com Delete Immediate true 3s
创建 pvc 测试
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: csi-rbd-sc
查看 pvc 是否正常
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Bound pvc-8256d0e9-95be-43f3-882c-31997fd161f8 1Gi RWO csi-rbd-sc 21s
创建 pod 验证 pvc
---
apiVersion: v1
kind: Pod
metadata:
name: csi-rbd-demo-pod
spec:
containers:
- name: web-server
image: docker.io/library/nginx:latest
volumeMounts:
- name: mypvc
mountPath: /var/lib/www/html
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: rbd-pvc
readOnly: false
进入 pod 查看是否正常挂载
[root@localhost kubernetes]# kubectl get pod
NAME READY STATUS RESTARTS AGE
csi-rbdplugin-provisioner-74888444cc-wj5nv 0/7 Pending 0 5m43s
csi-rbdplugin-provisioner-74888444cc-tgmj6 0/7 Pending 0 5m43s
csi-rbdplugin-bdgnx 3/3 Running 0 5m43s
csi-rbdplugin-provisioner-74888444cc-mjb9d 7/7 Running 0 5m43s
csi-rbd-demo-pod 1/1 Running 0 112s
[root@localhost kubernetes]# kubectl exec -it csi-rbd-demo-pod -- /bin/bash
root@csi-rbd-demo-pod:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 50G 4.8G 46G 10% /
tmpfs 64M 0 64M 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-root 50G 4.8G 46G 10% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/rbd0 976M 2.6M 958M 1% /var/lib/www/html
tmpfs 32G 12K 32G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 16G 0 16G 0% /proc/acpi
tmpfs 16G 0 16G 0% /proc/scsi
tmpfs 16G 0 16G 0% /sys/firmware
root@csi-rbd-demo-pod:/# ls -l /var/lib/www/html/
total 16
drwx------ 2 root root 16384 Jun 7 09:57 lost+found
root@csi-rbd-demo-pod:/# cd /var/lib/www/html/
root@csi-rbd-demo-pod:/var/lib/www/html#
root@csi-rbd-demo-pod:/var/lib/www/html# echo test > test.txt
root@csi-rbd-demo-pod:/var/lib/www/html#
root@csi-rbd-demo-pod:/var/lib/www/html# cat test.txt
test
root@csi-rbd-demo-pod:/var/lib/www/html#
更多示例:https://github.com/ceph/ceph-csi/tree/devel/examples/rbd
5.3 设置为默认 StorageClass
gitlab chart 安装时创建 pvc 需要调用默认 StorageClass
[root@localhost ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-rbd-sc rbd.csi.ceph.com Delete Immediate true 8m37s
[root@localhost ~]#
[root@localhost ~]# kubectl patch storageclass csi-rbd-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/csi-rbd-sc patched
[root@localhost ~]#
[root@localhost ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-rbd-sc (default) rbd.csi.ceph.com Delete Immediate true 9m5s
6. 添加极狐gitlab chart 仓库
涉及主机:
- 10.10.10.61
6.1 安装 helm v3
安装 helm v3(v3.3.1 或者更新),参考官方安装文档:Helm | Installing Helm
这里使用手动下载安装方式:
# 发布地址:https://github.com/helm/helm/releases
curl -O https://get.helm.sh/helm-v3.8.0-linux-amd64.tar.gz
tar zxvf helm-v3.8.0-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
chmod +x /usr/local/bin/helm
helm version
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}
6.2 添加 chart 仓库
chart 官方地址:gitlab/gitlab-jh
helm repo add gitlab-jh https://charts.gitlab.cn
# 更新仓库
helm repo update gitlab-jh
# 查看镜像仓库列表
helm search repo gitlab-jh
# 查看完整的镜像仓库列表
helm search repo -l gitlab-jh
7. 安装极狐gitlab
涉及主机:
- 10.10.10.61
域名为 gitlab.example.com,不启用 https,如果启用,请使用有效的证书,自签证书会遇到很多配置问题
7.1 前置准备
-
修改 ssh 端口
gitlab-shell 需要使用 22 端口(ssh 方式拉取和提交时),所以修改端口为 2222
vi /etc/ssh/sshd_config
... Port 2222 ...
-
设置 k3s 集群连接配置文件
原因是:helm v3 版本不再需要 Tiller,而是直接访问 ApiServer 来与 k8s 交互,通过环境变量
KUBECONFIG
来读取存有 ApiServer 的地址与 token 的配置文件地址,默认地址为~/.kube/config
,但是 k3s 的KUBECONFIG
默认为/etc/rancher/k3s/k3s.yaml
临时解决
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
永久解决
$ vi .bash_profile # 添加 export KUBECONFIG=/etc/rancher/k3s/k3s.yaml $ source .bash_profile
-
删除 k3s 集群自带的 traefik ingress controller
这里使用 kong ingress controller,所以需要删除集群中自带的 traefik ingress controller
$ kubectl get all -n kube-system NAME READY STATUS RESTARTS AGE pod/helm-install-traefik-g4xsp 0/1 Completed 0 136m pod/metrics-server-86cbb8457f-8wghh 1/1 Running 0 136m pod/local-path-provisioner-5ff76fc89d-r87zg 1/1 Running 0 136m pod/svclb-traefik-24nj7 2/2 Running 0 136m pod/coredns-854c77959c-nm28h 1/1 Running 0 136m pod/traefik-6f9cbd9bd4-lm4xf 1/1 Running 0 136m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 137m service/metrics-server ClusterIP 10.43.36.234 <none> 443/TCP 137m service/traefik-prometheus ClusterIP 10.43.47.8 <none> 9100/TCP 136m service/traefik LoadBalancer 10.43.154.114 10.10.10.60 80:30104/TCP,443:31153/TCP 136m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/svclb-traefik 1 1 1 1 1 <none> 136m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/metrics-server 1/1 1 1 137m deployment.apps/local-path-provisioner 1/1 1 1 137m deployment.apps/coredns 1/1 1 1 137m deployment.apps/traefik 1/1 1 1 136m NAME DESIRED CURRENT READY AGE replicaset.apps/metrics-server-86cbb8457f 1 1 1 136m replicaset.apps/local-path-provisioner-5ff76fc89d 1 1 1 136m replicaset.apps/coredns-854c77959c 1 1 1 136m replicaset.apps/traefik-6f9cbd9bd4 1 1 1 136m NAME COMPLETIONS DURATION AGE job.batch/helm-install-traefik 1/1 19s 137m # 删除 traefik $ kubectl -n kube-system delete service traefik-prometheus $ kubectl -n kube-system delete service traefik $ kubectl -n kube-system delete deployment.apps/traefik $ kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE helm-install-traefik-g4xsp 0/1 Completed 0 139m metrics-server-86cbb8457f-8wghh 1/1 Running 0 139m local-path-provisioner-5ff76fc89d-r87zg 1/1 Running 0 139m coredns-854c77959c-nm28h 1/1 Running 0 139m
7.2 安装 kong-ingress-controller
官方地址:Kong/kubernetes-ingress-controller
安装:
kubectl apply -f https://bit.ly/k4k8s
- 也可以使用 helm 安装,参考 Kong/kubernetes-ingress-controller: get-started
确认安装情况:
[root@localhost ~]# kubectl -n kong get all
NAME READY STATUS RESTARTS AGE
pod/svclb-kong-proxy-f5t6g 2/2 Running 0 14h
pod/ingress-kong-5c9f6f86bc-scnkx 2/2 Running 0 14h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kong-validation-webhook ClusterIP 10.43.205.159 <none> 443/TCP 14h
service/kong-proxy LoadBalancer 10.43.155.217 10.10.10.61 80:31859/TCP,443:32035/TCP 14h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/svclb-kong-proxy 1 1 1 1 1 <none> 14h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-kong 1/1 1 1 14h
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-kong-5c9f6f86bc 1 1 1 14h
查看 ingress 相关资源:
[root@localhost ~]# kubectl api-resources|grep -i ingress
kongingresses ki configuration.konghq.com/v1 true KongIngress
tcpingresses configuration.konghq.com/v1beta1 true TCPIngress
udpingresses configuration.konghq.com/v1beta1 true UDPIngress
ingressclasses networking.k8s.io/v1 false IngressClass
ingresses ing networking.k8s.io/v1 true Ingress
ingressroutes traefik.containo.us/v1alpha1 true IngressRoute
ingressroutetcps traefik.containo.us/v1alpha1 true IngressRouteTCP
ingressrouteudps traefik.containo.us/v1alpha1 true IngressRouteUDP
[root@localhost ~]#
[root@localhost ~]# kubectl get ingressclasses
NAME CONTROLLER PARAMETERS AGE
kong ingress-controllers.konghq.com/kong <none> 14h
- ingressclasses 名称为 kong,后面安装 chart 指定参数需要用到
7.3 设置对象存储连接 secret
参考:doc/charts/globals.md · master · GitLab.org / charts / GitLab Chart · GitLab
rails 、registry 和 runner 使用的对象存储连接配置不一样,所以要创建 3 个 secret
创建 rails 连接 rails_minio.yaml
文件
provider: AWS
region: minio
aws_access_key_id: minio
aws_secret_access_key: minio123456
endpoint: http://10.10.10.60:9000
path_style: true
- 其他 s3 兼容的配置参考:Object storage | GitLab
kubectl create secret generic gitlab-storage --from-file=connection=rails_minio.yaml
创建 registry 连接 registry_minio.yaml
文件
s3:
v4auth: true
regionendpoint: "http://10.10.10.60:9000"
pathstyle: true
region: minio
bucket: gitlab-registry
accesskey: minio
secretkey: minio123456
应用:
kubectl create secret generic gitlab-registry-storage --from-file=connection=registry_minio.yaml
rails 与 registry 支持的对象存储配置参考:examples/objectstorage · master · GitLab.org / charts / GitLab Chart · GitLab
创建 runner cache 连接 secret
kubectl create secret generic gitlab-minioaccess \
--from-literal=accesskey="minio" \
--from-literal=secretkey="minio123456"
- secret 名称
gitlab-minioaccess
,后面安装 chart 配置需要用到
runner cache 配置参考:
7.4 安装 chart
这里的--set
选项是chart的configure,Helm通过chart+configure的方式实现部署,configure可以视为部署chart时需要定义的变量,这里也可以使用helm install gitlab gitlab-jh/gitlab -f values.yaml
的方式批量配置configure,详细内容请参考:values.yaml · master · GitLab.org / charts / GitLab Chart · GitLab
创建 runner cache 配置文件 runner_cache.yaml
:
gitlab-runner:
runners:
config: |
[[runners]]
[runners.kubernetes]
image = "ubuntu:18.04"
[runners.cache]
Type = "s3"
Path = "gitlab-runner"
Shared = true
[runners.cache.s3]
ServerAddress = "10.10.10.60:9000"
BucketName = "gitlab-cache"
Insecure = true
cache:
secretName: gitlab-minioaccess
- Insecure = true,使用 http
安装命令:
helm install gitlab gitlab-jh/gitlab \
-f runner_cache.yaml \
--set global.hosts.domain=example.com \
--set global.hosts.https=false \
--set global.ingress.tls.enabled=false \
--set certmanager-issuer.email=me@example.com \
--set global.minio.enabled=false \
--set global.registry.bucket=gitlab-registry \
--set global.appConfig.object_store.enabled=true \
--set global.appConfig.object_store.proxy_download=true \
--set global.appConfig.object_store.connection.secret=gitlab-storage \
--set global.appConfig.object_store.connection.key=connection \
--set global.appConfig.lfs.bucket=gitlab-lfs-storage \
--set global.appConfig.artifacts.bucket=gitlab-artifacts-storage \
--set global.appConfig.uploads.bucket=gitlab-uploads-storage \
--set global.appConfig.packages.bucket=gitlab-packages-storage \
--set global.appConfig.externalDiffs.bucket=gitlab-external-diffs \
--set global.appConfig.terraformState.bucket=gitlab-terraform-state \
--set global.appConfig.dependencyProxy.bucket=gitlab-dependency-proxy \
--set global.appConfig.backups.bucket=gitlab-backup-upload \
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage \
--set gitlab.toolbox.backups.objectStorage.config.secret=gitlab-storage \
--set gitlab.toolbox.backups.objectStorage.config.key=connection \
--set registry.storage.secret=gitlab-registry-storage \
--set registry.storage.key=connection \
--set global.ingress.enabled=true \
--set global.ingress.class=kong \
--set nginx-ingress.enabled=false \
--set global.busybox.image.repository=busybox \
--set global.busybox.image.tag=1.28.4 \
--version=6.0.0
- 外部 ingress 配置参考:examples/values-traefik-ingress.yaml · master · GitLab.org / charts / GitLab Chart · GitLab
- 外部存储配置参考:examples/values-external-objectstorage.yaml · master · GitLab.org / charts / GitLab Chart · GitLab
- 使用
example.com
作为域名,部署成功后便可通过http://gitlab.example.com
访问 Gitlab 实例 - 其中的 secret 和 key,就是 7.3 章节设置的对象存储连接 secret 的名称,以及其中的 key,注意 chart 实例和连接 secret 需要在同一 namespace
- 这里对象存储使用统一连接配置,也可以分开配置,参考:doc/charts/globals.md · master · GitLab.org / charts / GitLab Chart · GitLab
- 6.0.0 对应 gitlab-jh v15.0.0 版本
- 默认情况下 initContainers 会使用 busybox:latest 镜像,busybox 新版的镜像部分命令与 bug(至少 1.30 版本 nslookup 命令有问题),所以这里指定使用 1.28.4 版本
- 根据网络拉取镜像的情况,可能时间等待较久或者部分 pod 启动失败,解决镜像拉取问题即可
处理完成后发现 gitlab-runner 的 pod 还是不正常的
$ kubectl get po|grep -v Running
NAME READY STATUS RESTARTS AGE
gitlab-issuer-1-7dt84 0/1 Completed 0 34m
gitlab-minio-create-buckets-1-r8pzf 0/1 Completed 0 34m
gitlab-migrations-1-2scb6 0/1 Completed 1 34m
gitlab-gitlab-runner-764cdf7c6d-xfw5t 0/1 CrashLoopBackOff 7 34m
$ kubectl logs gitlab-gitlab-runner-764cdf7c6d-xfw5t
...
Merging configuration from template file "/configmaps/config.template.toml"
ERROR: Registering runner... failed runner=GgfOlfZW status=couldn't execute POST against http://gitlab.example.com/api/v4/runners: Post "http://gitlab.example.com/api/v4/runners": dial tcp: lookup gitlab.example.com on 10.43.0.10:53: no such host
PANIC: Failed to register the runner.
原因是 runner 的 pod 无法解析 gitlab.example.com,解决方法如下:
首先查看 gitlab.example.com 地址
$ kubectl get ingress -lrelease=gitlab
NAME CLASS HOSTS ADDRESS PORTS AGE
gitlab-kas kong kas.example.com 10.10.10.61 80 15h
gitlab-registry kong registry.example.com 10.10.10.61 80 15h
gitlab-webservice-default kong gitlab.example.com 10.10.10.61 80 15h
然后在集群 coredns 中添加 hosts 解析
$ kubectl -n kube-system edit configmaps coredns
...
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
hosts /etc/coredns/NodeHosts {
10.10.10.61 gitlab.example.com
10.10.10.61 registry.example.com
10.10.10.61 kas.example.com
ttl 60
reload 15s
fallthrough
}
...
# 删除 coredns pod 生效
$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
helm-install-traefik-g4xsp 0/1 Completed 0 151m
metrics-server-86cbb8457f-8wghh 1/1 Running 0 151m
local-path-provisioner-5ff76fc89d-r87zg 1/1 Running 0 151m
coredns-854c77959c-nm28h 1/1 Running 0 151m
$ kubectl -n kube-system delete pod coredns-854c77959c-nm28h
pod "coredns-854c77959c-nm28h" deleted
再次查看 runner 状态已正常:
[root@localhost ~]# kubectl get pod|grep runner
gitlab-gitlab-runner-764cdf7c6d-xfw5t 1/1 Running 5 (2m42s ago) 19m
全部 pod 与 ceph rbd pvc 正常
[root@localhost ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
csi-rbdplugin-provisioner-74888444cc-wj5nv 0/7 Pending 0 16h
csi-rbdplugin-provisioner-74888444cc-tgmj6 0/7 Pending 0 16h
csi-rbdplugin-bdgnx 3/3 Running 0 16h
csi-rbdplugin-provisioner-74888444cc-mjb9d 7/7 Running 0 16h
svclb-gitlab-nginx-ingress-controller-j2tgw 3/3 Running 0 51m
gitlab-nginx-ingress-controller-6545975dbf-d7kpv 1/1 Running 0 51m
gitlab-certmanager-cainjector-74cbc84b8b-d2c9v 1/1 Running 0 51m
gitlab-certmanager-57c4557849-gb79z 1/1 Running 0 51m
gitlab-nginx-ingress-defaultbackend-7b4f8d7847-zqrwz 1/1 Running 0 51m
gitlab-nginx-ingress-controller-6545975dbf-wfl2f 1/1 Running 0 51m
gitlab-certmanager-webhook-59d745756c-pr2cr 1/1 Running 0 51m
gitlab-gitlab-exporter-9c99c79cd-7mdwh 1/1 Running 0 51m
gitlab-toolbox-579746c789-wjhdx 1/1 Running 0 51m
gitlab-gitlab-shell-77d89bf64-zrrm6 1/1 Running 0 51m
gitlab-redis-master-0 2/2 Running 0 41m
gitlab-gitlab-shell-77d89bf64-hrc9z 1/1 Running 0 51m
gitlab-prometheus-server-77b5cc946-zsk4k 2/2 Running 0 51m
gitlab-postgresql-0 2/2 Running 0 51m
gitlab-gitaly-0 1/1 Running 0 51m
gitlab-sidekiq-all-in-1-v2-79df4567db-dskb5 1/1 Running 0 51m
gitlab-webservice-default-5bbc95b7d5-rvwnr 2/2 Running 0 51m
gitlab-webservice-default-5bbc95b7d5-88jxv 2/2 Running 0 51m
gitlab-gitlab-runner-764cdf7c6d-xfw5t 1/1 Running 5 (35m ago) 51m
gitlab-issuer-2--1-xj6qr 0/1 Completed 0 46s
gitlab-registry-67fb69c6cf-6jmfw 1/1 Running 0 47s
gitlab-registry-67fb69c6cf-6bldq 1/1 Running 0 36s
[root@localhost ~]#
[root@localhost ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gitlab-prometheus-server Bound pvc-bc5b38ea-966e-4f82-a845-bd01a9181330 8Gi RWO csi-rbd-sc 52m
redis-data-gitlab-redis-master-0 Bound pvc-98cd04fb-272f-4873-b065-a91b4aa8eb74 8Gi RWO csi-rbd-sc 52m
repo-data-gitlab-gitaly-0 Bound pvc-33a0c2f6-b086-4ef6-99dc-f262398468cc 50Gi RWO csi-rbd-sc 52m
data-gitlab-postgresql-0 Bound pvc-88e69f3e-089a-417b-a8b3-43ed9034c858 8Gi RWO csi-rbd-sc 52m
[root@localhost ~]#
可以使用 helm get values gitlab > gitlab.yaml
导出 yaml 资源文件
7.5 访问 gitlab 实例
获取 ip
$ kubectl get ingress -lrelease=gitlab
NAME CLASS HOSTS ADDRESS PORTS AGE
gitlab-kas kong kas.example.com 10.10.10.61 80 15h
gitlab-registry kong registry.example.com 10.10.10.61 80 15h
gitlab-webservice-default kong gitlab.example.com 10.10.10.61 80 15h
本机设置 hosts
10.10.10.61 gitlab.example.com
获取 root 初始密码
kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode; echo
访问地址:http://gitlab.example.com
gitlab runner 正常注册
7.6 测试 runner
再创建测试项目 p1
添加 .gitlab-ci.yml
文件,提交:
stages:
- build
- deploy
variables:
projectname: "p1"
build:
stage: build
retry: 2
before_script:
- echo "before_script in build job"
script:
- df && ls -l
- echo ${projectname}
- echo "script in build job"
after_script:
- echo "after_script in build job"
deploy:
stage: deploy
script:
- echo "deploy"
when: manual
only:
- tags
提交后 runner 会自动运行流水线,运行成功:
流水线结果:
Running with gitlab-runner 15.0.0 (c6bb62f6)
on gitlab-gitlab-runner-764cdf7c6d-xfw5t xfXku1BF
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: default
Using Kubernetes executor with image ubuntu:18.04 ...
Using attach strategy to execute scripts...
Preparing environment
00:27
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-xfxku1bf-project-2-concurrent-0j8gsv via gitlab-gitlab-runner-764cdf7c6d-xfw5t...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/g1/p1/.git/
Created fresh repository.
Checking out b23aedc7 as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ echo "before_script in build job"
before_script in build job
$ df && ls -l
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 52403200 11324604 41078596 22% /
tmpfs 65536 0 65536 0% /dev
tmpfs 16430384 0 16430384 0% /sys/fs/cgroup
/dev/mapper/centos-root 52403200 11324604 41078596 22% /builds
shm 65536 0 65536 0% /dev/shm
tmpfs 32860768 12 32860756 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 16430384 0 16430384 0% /proc/acpi
tmpfs 16430384 0 16430384 0% /proc/scsi
tmpfs 16430384 0 16430384 0% /sys/firmware
total 8
-rw-rw-rw- 1 root root 6180 Jun 8 02:55 README.md
$ echo ${projectname}
p1
$ echo "script in build job"
script in build job
Running after_script
00:01
Running after script...
$ echo "after_script in build job"
after_script in build job
Job succeeded
在 10.10.10.60 主机查看 minio 存储目录,发现已保存 pipeline 日志:
[root@localhost data]# pwd
/opt/minio/data
[root@localhost data]#
[root@localhost data]# du * -sh
8.0K gitlab-artifacts-storage
0 gitlab-backup-upload
0 gitlab-dependency-proxy
0 gitlab-external-diffs
0 gitlab-lfs-storage
0 gitlab-packages-storage
0 gitlab-pages
0 gitlab-registry
0 gitlab-terraform-state
0 gitlab-tmp-storage
0 gitlab-uploads-storage
[root@localhost data]#
[root@localhost data]# cat gitlab-artifacts-storage/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2022_06_08/1/1/job.log
Running with gitlab-runner 15.0.0 (c6bb62f6)
on gitlab-gitlab-runner-764cdf7c6d-xfw5t xfXku1BF
Preparing the "kubernetes" executor
Using Kubernetes namespace: default
Using Kubernetes executor with image ubuntu:18.04 ...
Using attach strategy to execute scripts...
Preparing environment
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0j8gsv to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-xfxku1bf-project-2-concurrent-0j8gsv via gitlab-gitlab-runner-764cdf7c6d-xfw5t...
Getting source from Git repository
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/g1/p1/.git/
Created fresh repository.
Checking out b23aedc7 as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
$ echo "before_script in build job"
before_script in build job
$ df && ls -l
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 52403200 11324604 41078596 22% /
tmpfs 65536 0 65536 0% /dev
tmpfs 16430384 0 16430384 0% /sys/fs/cgroup
/dev/mapper/centos-root 52403200 11324604 41078596 22% /builds
shm 65536 0 65536 0% /dev/shm
tmpfs 32860768 12 32860756 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 16430384 0 16430384 0% /proc/acpi
tmpfs 16430384 0 16430384 0% /proc/scsi
tmpfs 16430384 0 16430384 0% /sys/firmware
total 8
-rw-rw-rw- 1 root root 6180 Jun 8 02:55 README.md
$ echo ${projectname}
p1
$ echo "script in build job"
script in build job
Running after_script
Running after script...
$ echo "after_script in build job"
after_script in build job
Job succeeded
[root@localhost data]#
7.7 测试 Registry 服务
修改 .gitlab-ci.yml
文件:
build:
stage: build
image:
# name: gcr.io/kaniko-project/executor:debug
name: willdockerhub/kaniko-executor:debug
entrypoint: [""]
script:
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
- >-
/kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
--skip-tls-verify
rules:
- if: $CI_COMMIT_TAG
- 使用 kaniko 打包并提交镜像到 Registry
添加 Dockerfile
:
FROM nginx:latest
ENV TEST=1
添加一个 tag,触发 pipeline:
job 运行结果:
Running with gitlab-runner 15.0.0 (c6bb62f6)
on gitlab-gitlab-runner-764cdf7c6d-xfw5t xfXku1BF
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: default
Using Kubernetes executor with image willdockerhub/kaniko-executor:debug ...
Using attach strategy to execute scripts...
Preparing environment
00:03
Waiting for pod default/runner-xfxku1bf-project-2-concurrent-0hmp5l to be running, status is Pending
Running on runner-xfxku1bf-project-2-concurrent-0hmp5l via gitlab-gitlab-runner-764cdf7c6d-xfw5t...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/g1/p1/.git/
Created fresh repository.
Checking out 6516a100 as v1.0.2...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:32
$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}" --skip-tls-verify
INFO[0000] Retrieving image manifest nginx:latest
INFO[0000] Retrieving image nginx:latest from registry index.docker.io
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest nginx:latest
INFO[0006] Returning cached image manifest
INFO[0006] Executing 0 build triggers
INFO[0006] Skipping unpacking as no commands require it.
INFO[0006] ENV LEFFSS=1
INFO[0006] Pushing image to registry.example.com/g1/p1:v1.0.2
INFO[0032] Pushed registry.example.com/g1/p1@sha256:0a33f3ad16af8e151072c89d98a624f017c9eb36c60a4651f2cc73a76f63acaf
Job succeeded
正常提交镜像:
在 10.10.10.60 主机查看 minio 存储目录,发现已保存 registry 数据:
[root@localhost data]# pwd
/opt/minio/data
[root@localhost data]#
[root@localhost data]# du * -sh
60K gitlab-artifacts-storage
0 gitlab-backup-upload
0 gitlab-dependency-proxy
0 gitlab-external-diffs
0 gitlab-lfs-storage
0 gitlab-packages-storage
0 gitlab-pages
55M gitlab-registry
0 gitlab-terraform-state
0 gitlab-tmp-storage
0 gitlab-uploads-storage
7.8 暴露 gitlab-shell tcp 服务端口
在 10.10.10.60 主机上设置 hosts,并生成 ssh key:
[root@localhost ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# ceph-node
10.10.10.51 ceph01
10.10.10.52 ceph02
10.10.10.53 ceph03
10.10.10.61 gitlab.example.com
[root@localhost ~]#
[root@localhost ~]# ssh-keygen -t rsa -b 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:a5ZMokwuQfNysoUk2zKeub9WiIqt27o/vyH76sMmFrk root@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
| |
| |
|. + |
| * + |
|+ O * . S |
|.*o% o + o |
|o** * * |
|+EB= . o |
|*@OO=. |
+----[SHA256]-----+
[root@localhost ~]#
把公钥 ~/.ssh/id_rsa.pub
添加到 gitlab 中:
在 10.10.10.60 主机上测试联通性:
[root@localhost ~]# ssh -T git@gitlab.example.com
ssh: connect to host gitlab.example.com port 22: Connection refused
ssh方式无法连接,原因是当使用 gitlab chart 自带的 nginx-ingress-controller 时,是通过 configmap 暴露 gitlab-shell 的 tcp 端口 22 的:
[root@localhost ~]# kubectl get configmaps gitlab-nginx-ingress-tcp -o yaml
apiVersion: v1
data:
"22": 'default/gitlab-gitlab-shell:22::'
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: gitlab
meta.helm.sh/release-namespace: default
creationTimestamp: "2022-06-08T07:56:27Z"
labels:
app: gitlab-shell
app.kubernetes.io/managed-by: Helm
chart: gitlab-shell-6.0.0
heritage: Helm
release: gitlab
name: gitlab-nginx-ingress-tcp
namespace: default
resourceVersion: "91909"
uid: 049e3173-ed02-4b2b-8d71-0a43b49017b4
[root@localhost ~]#
而换用 kong-ingress-controller 后,虽然 configmap 配置文件 gitlab-nginx-ingress-tcp
还是创建了,但是 kong-ingress-controller 是无法识别的,这也导致 gitlab-shell 的 22 端口未暴露出来,解决方法是使用 kong-ingress-controller 支持的方式重新暴露 tcp 端口。
参考文档:TCPIngress with Kong - v2.3.x | Kong Docs (konghq.com)
7.8.1 配置 kong-ingress-controller
首先配置 kong-ingress-controller,新增 22 tcp 端口
$ kubectl patch deploy -n kong ingress-kong --patch '{
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "proxy",
"env": [
{
"name": "KONG_STREAM_LISTEN",
"value": "0.0.0.0:22"
}
],
"ports": [
{
"containerPort": 22,
"name": "stream22",
"protocol": "TCP"
}
]
}
]
}
}
}
}'
$ kubectl patch service -n kong kong-proxy --patch '{
"spec": {
"ports": [
{
"name": "stream22",
"port": 22,
"protocol": "TCP",
"targetPort": 22
}
]
}
}'
7.8.2 创建 tcpingress
$ echo "apiVersion: configuration.konghq.com/v1beta1
kind: TCPIngress
metadata:
name: gitlab-shell
annotations:
kubernetes.io/ingress.class: kong
spec:
rules:
- port: 22
backend:
serviceName: gitlab-gitlab-shell
servicePort: 22
" | kubectl apply -f -
$ kubectl get tcpingress
NAME ADDRESS AGE
gitlab-shell 10.10.10.61 27m
$ kubectl get tcpingress gitlab-shell -o yaml
apiVersion: configuration.konghq.com/v1beta1
kind: TCPIngress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"configuration.konghq.com/v1beta1","kind":"TCPIngress","metadata":{"annotations":{"kubernetes.io/ingress.class":"kong"},"name":"gitlab-shell","namespace":"default"},"spec":{"rules":[{"backend":{"serviceName":"gitlab-gitlab-shell","servicePort":22},"port":22}]}}
kubernetes.io/ingress.class: kong
creationTimestamp: "2022-06-09T00:30:42Z"
generation: 1
name: gitlab-shell
namespace: default
resourceVersion: "258906"
uid: 865f3885-f365-4d0d-980d-7c84f6524509
spec:
rules:
- backend:
serviceName: gitlab-gitlab-shell
servicePort: 22
port: 22
status:
loadBalancer:
ingress:
- ip: 10.10.10.61
7.8.3 测试 ssh 拉取提交
再次在 10.10.10.60 主机上测试:
[root@localhost ~]# ssh -T git@gitlab.example.com
Welcome to GitLab, @root!
[root@localhost ~]#
[root@localhost ~]# git clone git@gitlab.example.com:gitlab-instance-4bfa23d0/p1.git
Cloning into 'p1'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (6/6), done.
[root@localhost ~]#
[root@localhost ~]# cd p1/
[root@localhost p1]# echo xx > test.txt
[root@localhost p1]#
[root@localhost p1]# git status
# On branch main
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# test.txt
nothing added to commit but untracked files present (use "git add" to track)
[root@localhost p1]#
[root@localhost p1]# git add .
[root@localhost p1]# git commit -m 'add test.txt'
[main 99e3e3c] add test.txt
1 file changed, 1 insertion(+)
create mode 100644 test.txt
[root@localhost p1]#
[root@localhost p1]# git push origin main
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 279 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To git@gitlab.example.com:gitlab-instance-4bfa23d0/p1.git
c11db43..99e3e3c main -> main
[root@localhost p1]#
可以看到已经能够成功使用 ssh 方式拉取与提交了
8. 使用外部服务
可使用外部 PostgreSQL、Redis、Gitaly 等,具体参考文档:Advanced configuration | GitLab
8.1 使用外部 redis
1)搭建 redis
仅测试,所以直接在 centos 7.9 下使用 yum 安装
yum install epel-release -y
yum install redis -y
修改配置,开启远程访问
vi /etc/redis.conf
,修改或者添加以下选项
bind 10.10.10.61
requirepass redis123456
protected-mode no
重启 redis 服务
$ systemctl restart redis
$ ss -tnlp|grep 6379
LISTEN 0 511 10.10.10.61:6379 *:* users:(("redis-server",pid=1621,fd=4))
2)连接外部 redis
官方文档:Configure this chart with External Redis | GitLab
首先创建存储 redis 密码的 secret
方式一:yaml 文件
$ echo -n redis123456 | base64
cmVkaXMxMjM0NTY=
vi gitlab-redis-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: gitlab-redis-secret
data:
redis-password: cmVkaXMxMjM0NTY=
$ kubectl apply -f gitlab-redis-secret.yaml
$ kubectl get secret gitlab-redis-secret -ojsonpath='{.data.redis-password}' | base64 --decode; echo
redis123456
- data 中可以指定多个 key:value
方式二:文件形式
$ echo -n redis123456 > redis-password
$ kubectl create secret generic gitlab-redis-secret --from-file=./redis-password
$ kubectl get secret gitlab-redis-secret1 -ojsonpath='{.data.redis-password}' | base64 --decode; echo
redis123456
- --from-file 可以指定多个文件,文件名为 key,文件内容为 value(不需转 base64 编码)
安装 chart
helm install gitlab gitlab/gitlab \
--set global.hosts.domain=example.com \
--set global.hosts.https=false \
--set global.ingress.tls.enabled=false \
--set redis.install=false \
--set global.redis.host=10.10.10.61 \
--set global.redis.port=6379 \
--set global.redis.password.secret=gitlab-redis-secret \
--set global.redis.password.key=redis-password \
--set certmanager-issuer.email=me@example.com
- 也可以使用哨兵模式,具体参考官方文档
其他 PostgreSQL、Gitaly 基本类似,具体参考官方文档即可,这里就不演示了
9. 升级极狐gitlab
首先导出配置
helm get values gitlab > gitlab.yaml
然后升级到 6.0.2(对应 gitlab-jh v15.0.1 版本)
helm upgrade gitlab gitlab-jh/gitlab \
--version 6.0.2 \
-f gitlab.yaml \
--set gitlab.migrations.enabled=true \
--set ...
- --set ... 可选新增参数
如果只是更新下配置则可以使用以下命令:
helm upgrade gitlab gitlab-jh/gitlab \
--version <原版本> \
-f gitlab.yaml \
--set gitlab.migrations.enabled=false
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· 清华大学推出第四讲使用 DeepSeek + DeepResearch 让科研像聊天一样简单!
· 实操Deepseek接入个人知识库
· CSnakes vs Python.NET:高效嵌入与灵活互通的跨语言方案对比
· Plotly.NET 一个为 .NET 打造的强大开源交互式图表库