使用 operator 为极狐 gitlab 部署 runner

使用 operator 为极狐 gitlab 部署 runner

极狐gitlab v14.10.0版本新增 gitlab runner operator 方式部署 runner

项目地址：GitLab.org / OpenShift / GitLab Runner Operator

文档地址：Install GitLab Runner Operator | GitLab

1. 安装 k8s

版本：v1.23.5

参考：k8s 测试环境搭建（k3s） - leffss - 博客园 (cnblogs.com)

在集群 coredns 中添加极狐gitlab 主机 hosts 解析

$ kubectl -n kube-system edit configmaps coredns
...
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          10.10.10.60 gitlab.leffss.cn
          ttl 60    
          reload 15s 
          fallthrough
        }               
...

# 删除 coredns pod 生效
$ kubectl get pod -n kube-system
NAME                                      READY   STATUS      RESTARTS   AGE
metrics-server-86cbb8457f-hqt9p           1/1     Running     0          5h6m
local-path-provisioner-5ff76fc89d-cz9hj   1/1     Running     0          5h6m
helm-install-traefik-lmt7r                0/1     Completed   0          5h6m
coredns-854c77959c-w4t7j                  1/1     Running     0          5h6m
svclb-traefik-fp7f9                       2/2     Running     0          5h5m
traefik-6f9cbd9bd4-l7h45                  1/1     Running     0          5h5m

$ kubectl -n kube-system delete pod coredns-854c77959c-w4t7j
pod "coredns-854c77959c-nm28h" deleted

2. 安装 operator

安装 Operator Lifecycle Manager (OLM),

$ curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.20.0/install.sh | bash -s v0.20.0

安装 cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml

安装 operator

$ kubectl create -f https://operatorhub.io/install/gitlab-runner-operator.yaml
  • 当前版本 1.8.0

可以使用以下命令查看当前进度

$ kubectl get csv -n operators

kubectl get pod -n operators
NAME                                                READY   STATUS             RESTARTS   AGE
gitlab-runner-controller-manager-54ddcd566d-hrsc5   1/2     ImagePullBackOff   0          5m38s

Back-off pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"

查看 pod，发现报错：

$ kubectl -n operators describe pod gitlab-runner-controller-manager-54ddcd566d-hrsc5
...
...
...
  Warning  Failed     3m47s                  kubelet            Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://gcr.io/v2/": context deadline exceeded
  Normal   BackOff    81s (x14 over 5m38s)   kubelet            Back-off pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"

原因是无法拉取镜像：gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0

临时解决方法：

docker pull kubesphere/kube-rbac-proxy:v0.8.0
docker tag kubesphere/kube-rbac-proxy:v0.8.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0

3. 部署 runner

3.1 创建 runner

创建 runner Registration token secret 文件：

cat > gitlab-runner-secret.yml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-runner-secret
type: Opaque
stringData:
  runner-registration-token: <runner Registration token> # your project runner secret
EOF

创建 runner crd yaml 文件：

cat > gitlab-runner.yml << EOF
apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
  name: gitlab-runner
spec:
  gitlabUrl: https://gitlab.leffss.cn
  buildImage: alpine
  token: gitlab-runner-secret
EOF

创建：

kubectl apply -f gitlab-runner-secret.yml
kubectl apply -f gitlab-runner.yml

查看：

$ kubectl get pod
NAME                                    READY   STATUS    RESTARTS   AGE
gitlab-runner-runner-587447dfbc-hk2q9   1/1     Running   0          2m18s

$ kubectl get runner
NAME            AGE
gitlab-runner   2m21s

注册成功：

3.2 运行任务测试

创建 .gitlab-ci.yml

stages:
  - test

test_script-section:
  script:
    - echo foo
  needs: []
  variables:
    FF_SCRIPT_SECTIONS: "true"

test:
  stage: test
  script:
  - env

运行时会为每个 job 创建一个 pod

$ kubectl get pod -A
NAMESPACE      NAME                                                              READY   STATUS      RESTARTS   AGE
default        gitlab-runner-runner-587447dfbc-hk2q9                             1/1     Running     0          8m11s
default        runner-8hnnqcxg-project-7-concurrent-04x5gv                       0/2     Init:0/1    0          18s
default        runner-8hnnqcxg-project-7-concurrent-19tn8m                       0/2     Init:0/1    0          18s

4. 卸载 operator

  1. 删除 crd

    kubectl delete -f gitlab-runner.yml

  2. 删除 secret

    kubectl delete -f gitlab-runner-secret.yml

  3. 删除 Operator subscription

    kubectl delete subscription my-gitlab-runner-operator -n operators

  4. 查看 clusterserviceversion

    $ kubectl get clusterserviceversion -n operators
NAME                            DISPLAY         VERSION   REPLACES                        PHASE
gitlab-runner-operator.v1.8.0   GitLab Runner   1.8.0     gitlab-runner-operator.v1.7.0   Succeeded

  5. 删除 clusterserviceversion

    kubectl delete clusterserviceversion gitlab-runner-operator.v1.8.0 -n operators

5. 配置 operator

参考：Configuring GitLab Runner on OpenShift | GitLab

5.1 支持的配置

配置项 Operator版本 描述
gitlabUrl all 极狐 GitLab 实例地址，例如：https://gitlab.example.com
token all Secret 名称，其中包含 runner-registration-token
tags all 设置 runner tags
concurrent all 任务并发数设置。0 无限制，默认 10
interval all interval 设置，默认 30
locked 1.8 是否锁定，默认 false
runUntagged 1.8 是否能运行 untags 的 job，如果未定义 tags 则默认 true，否则 false
protected 1.8 是否只运行保护分支 job，默认 false
cloneURL all 覆盖 gitlabUrl 。 在 runner 不能连接 gitlabUrl 时使用
env all ConfigMap 名称，其中包含的 key-value 键值对将在 pod 作为环境变量
runnerImage 1.7 设置 gitlab runner 镜像。默认是 operator 版本绑定的镜像
helperImage all 设置 GitLab Runner helper 默认镜像
buildImage all 设置 build job 默认镜像
cacheType all 设置 cache 类型，可选项：gcs, s3, azure
cachePath all 设置 cache 目录
cacheShared all 设置 cache shared 模式
s3 all S3 cache 设置。 关联 Cache properties
gcs all GCS cache 设置。关联 Cache properties
azure all Azure cache设置。 关联 Cache properties
ca all TLS secret 名称，其中包括自签 CA 证书
serviceAccount all Runner pod serviceAccount 设置
config all configmap名称，其中包含 configuration template.

5.2 设置 HTTP_PROXY 环境变量

  1. 新增 custom-env.yml

    apiVersion: v1
data:
  NO_PROXY: 172.21.0.1
  HTTP_PROXY: example.com
kind: ConfigMap
metadata:
  name: custom-env

  2. 应用

    kubectl apply -f custom-env.yaml

  3. 更新 gitlab-runner.yml

    apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
  name: dev
spec:
  gitlabUrl: https://gitlab.example.com
  token: gitlab-runner-secret
  env: custom-env

5.3 自定义 config.toml

配置参考：Registering runners | GitLab

  1. 新增模板文件 custom-config.toml

    [[runners]]
  [runners.kubernetes]
    [runners.kubernetes.volumes]
      [[runners.kubernetes.volumes.empty_dir]]
        name = "empty-dir"
        mount_path = "/path/to/empty_dir"
        medium = "Memory"

  2. 创建 ConfigMap

    kubectl create configmap custom-config-toml --from-file config.toml=custom-config.toml

  3. 更新 gitlab-runner.yml

    apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
  name: dev
spec:
  gitlabUrl: https://gitlab.example.com
  token: gitlab-runner-secret
  config: custom-config-toml

  4. 查看配置

    $ kubectl exec -it gitlab-runner-runner-5ff5b95967-z9nbp -- cat /home/gitlab-runner/.gitlab-runner/config.toml
...
...
...
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "alpine"
    namespace = "default"
    namespace_overwrite_allowed = ""
    helper_image = "registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:x86_64-v14.10.0"
    poll_timeout = 180
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.build_container_security_context]
      [runners.kubernetes.build_container_security_context.capabilities]
    [runners.kubernetes.helper_container_security_context]
      [runners.kubernetes.helper_container_security_context.capabilities]
    [runners.kubernetes.service_container_security_context]
      [runners.kubernetes.service_container_security_context.capabilities]
    [runners.kubernetes.volumes]

      [[runners.kubernetes.volumes.empty_dir]]
        name = "empty-dir"
        mount_path = "/path/to/empty_dir"
        medium = "Memory"
    [runners.kubernetes.dns_config]
    [runners.kubernetes.container_lifecycle]

5.4 配置自签 TLS 证书

  1. 创建包含 ca 证书的 secret：custom-tls-ca-secret.yml

    apiVersion: v1
kind: Secret
metadata:
  name: custom-tls-ca
type: Opaque
stringData:
  tls.crt: |
  -----BEGIN CERTIFICATE-----
  MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix
  .....
  7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX
  -----END CERTIFICATE-----

  2. 应用

    kubectl apply -f custom-tls-ca-secret.yaml

  3. 更新 gitlab-runner.yml

    apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
  name: dev
spec:
  gitlabUrl: https://gitlab.example.com
  token: gitlab-runner-secret
  ca: custom-tls-ca

5.5 配置 cpu 和 mem 限制

可以通过 config.toml 配置 cpu limit 与 mem limit ，具体参考 5.3 章节

posted @ 2022-08-11 10:56  leffss  阅读(348)  评论(0)  编辑  收藏  举报