kubernetes(36)：持续集成(5)-k8s集群中搭建gitlab
k8s集群中搭建gitlab
https://www.qikqiak.com/k8s-book/docs/64.Gitlab.html
https://www.cnblogs.com/fuyuteng/p/11418734.html
gitlab有没有必要部署在k8s集群中？ gitlab占用的资源较多，依赖组建复杂。部署在k8s集群中也是一个不错的选择。
依赖组件：ruby 1.9.3+，MySQL，git，redis， Sidekiq。
最低配置CPU 1G，RAM 1G+swap可以支持100用户。
Gitlab官方提供了 Helm 的方式在 Kubernetes 集群中来快速安装，但是在使用的过程中发现 Helm 提供的 Chart 包中有很多其他额外的配置，所以我们这里使用自定义的方式来安装，也就是自己来定义一些资源清单文件。
Gitlab主要涉及到3个应用：Redis、Postgresql、Gitlab 核心程序，实际上我们只要将这3个应用分别启动起来，然后加上对应的配置就可以很方便的安装 Gitlab 了，我们这里选择使用的镜像不是官方的，而是 Gitlab 容器化中使用非常多的一个第三方镜像：sameersbn/gitlab，基本上和官方保持同步更新，地址：http://www.damagehead.com/docker-gitlab/
如果我们已经有可使用的 Redis 或 Postgresql 服务的话，那么直接配置在 Gitlab 环境变量中即可，如果没有的话就单独部署。
Ingress采用的Nginx
参考
https://www.cnblogs.com/wangxu01/articles/11670857.html
# kubectl get pods -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-controller-8n6fz 1/1 Running 0 8d 10.6.76.23 k8s-node-1 <none> <none> nginx-ingress-controller-jt82z 1/1 Running 0 8d 10.6.76.24 k8s-node-2 <none> <none>
1 创建PVC和storageclass做持久化
redis、postgresql、gitlab都需要持久化
1.1 创建持久化StorageClass
#cat gitlab-sc.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gitlab-storageclass provisioner: fuseim.pri/ifs
1.2 vim gitlab-redis-pvc.yaml
#vim gitlab-redis-pvc.yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gitlab-redis-pvc namespace: kube-ops annotations: volume.beta.kubernetes.io/storage-class: "gitlab-storageclass" spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi
1.3 vim gitlab-postgresql-pvc.yaml:
#vim gitlab-postgresql-pvc.yaml: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gitlab-postgresql-pvc namespace: kube-ops annotations: volume.beta.kubernetes.io/storage-class: "gitlab-storageclass" spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi
1.4 vim gitlab-pvc.yaml
#vim gitlab-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gitlab-pvc namespace: kube-ops annotations: volume.beta.kubernetes.io/storage-class: "gitlab-storageclass" spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi
2 gitlab-redis.yaml
首先部署需要的 Redis 服务，对应的资源清单文件如下：(gitlab-redis.yaml)
apiVersion: apps/v1beta1 kind: Deployment metadata: name: redis namespace: kube-ops labels: name: redis spec: template: metadata: name: redis labels: name: redis spec: containers: - name: redis image: sameersbn/redis imagePullPolicy: IfNotPresent ports: - name: redis containerPort: 6379 volumeMounts: - mountPath: /var/lib/redis name: data livenessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 30 timeoutSeconds: 5 readinessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 5 timeoutSeconds: 1 volumes: - name: data persistentVolumeClaim: claimName: gitlab-redis-pvc --- apiVersion: v1 kind: Service metadata: name: redis namespace: kube-ops labels: name: redis spec: ports: - name: redis port: 6379 targetPort: redis selector: name: redis
3 gitlab-postgresql.yaml
然后是数据库 Postgresql，对应的资源清单文件如下：(gitlab-postgresql.yaml)
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gitlab-storageclass provisioner: fuseim.pri/ifs [root@k8s-master gitlab]# cat gitlab-postgresql.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: postgresql namespace: kube-ops labels: name: postgresql spec: template: metadata: name: postgresql labels: name: postgresql spec: containers: - name: postgresql image: sameersbn/postgresql imagePullPolicy: IfNotPresent env: - name: DB_USER value: gitlab - name: DB_PASS value: passw0rd - name: DB_NAME value: gitlab_production - name: DB_EXTENSION value: pg_trgm ports: - name: postgres containerPort: 5432 volumeMounts: - mountPath: /var/lib/postgresql name: data livenessProbe: exec: command: - pg_isready - -h - localhost - -U - postgres initialDelaySeconds: 30 timeoutSeconds: 5 readinessProbe: exec: command: - pg_isready - -h - localhost - -U - postgres initialDelaySeconds: 5 timeoutSeconds: 1 volumes: - name: data persistentVolumeClaim: claimName: gitlab-postgresql-pvc --- apiVersion: v1 kind: Service metadata: name: postgresql namespace: kube-ops labels: name: postgresql spec: ports: - name: postgres port: 5432 targetPort: postgres selector: name: postgresql
4 gitlab.yaml
然后就是我们最核心的 Gitlab 的应用，对应的资源清单文件如下：(gitlab.yaml)
apiVersion: apps/v1beta1 kind: Deployment metadata: name: gitlab namespace: kube-ops labels: name: gitlab spec: template: metadata: name: gitlab labels: name: gitlab spec: containers: - name: gitlab image: sameersbn/gitlab:12.1.6 imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: GITLAB_TIMEZONE value: Beijing - name: GITLAB_SECRETS_DB_KEY_BASE value: long-and-random-alpha-numeric-string - name: GITLAB_SECRETS_SECRET_KEY_BASE value: long-and-random-alpha-numeric-string - name: GITLAB_SECRETS_OTP_KEY_BASE value: long-and-random-alpha-numeric-string - name: GITLAB_ROOT_PASSWORD value: admin321 - name: GITLAB_ROOT_EMAIL value: 314144952@qq.com - name: GITLAB_HOST value: gitlab.wangxu.com - name: GITLAB_PORT value: "80" - name: GITLAB_SSH_PORT value: "22" - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS value: "true" - name: GITLAB_NOTIFY_PUSHER value: "false" - name: GITLAB_BACKUP_SCHEDULE value: daily - name: GITLAB_BACKUP_TIME value: 01:00 - name: DB_TYPE value: postgres - name: DB_HOST value: postgresql - name: DB_PORT value: "5432" - name: DB_USER value: gitlab - name: DB_PASS value: passw0rd - name: DB_NAME value: gitlab_production - name: REDIS_HOST value: redis - name: REDIS_PORT value: "6379" ports: - name: http containerPort: 80 - name: ssh containerPort: 22 volumeMounts: - mountPath: /home/git/data name: data livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 180 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 timeoutSeconds: 1 volumes: - name: data persistentVolumeClaim: claimName: gitlab-pvc --- apiVersion: v1 kind: Service metadata: name: gitlab namespace: kube-ops labels: name: gitlab spec: ports: - name: http port: 80 targetPort: http - name: ssh port: 22 targetPort: ssh nodePort: 30022 type: NodePort selector: name: gitlab --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: gitlab namespace: kube-ops annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: gitlab.wangxu.com http: paths: - backend: serviceName: gitlab servicePort: 80 #apiVersion: extensions/v1beta1 #kind: Ingress #metadata: # name: gitlab # namespace: kube-ops # annotations: # kubernetes.io/ingress.class: traefik #spec: # rules: # - host: gtlab.wangxu.com # http: # paths: # - backend: # serviceName: gitlab # servicePort: http
我们这里应用数据都做数据持久化，添加 PV/PVC 或者 StorageClass。
5 查看Pod的部署状态
#等5分钟
# kubectl get pods -n kube-ops| grep -E "gitlab|^redis-|postgresql" gitlab-687c54659f-bm69j 1/1 Running 1 38m postgresql-8644bdb9c5-qww4v 1/1 Running 0 107m redis-6fcdb86497-2bshn 1/1 Running 0 100m [root@k8s-master gitlab]#
6 页面访问
我们可以通过 Ingress 中定义的域名gitlab.wangxu.com(需要做 DNS 解析或者在本地 /etc/hosts 中添加映射)来访问 Portal：
10.6.76.23 gitlab.wangxu.com 10.6.76.24 gitlab.wangxu.com
使用部署指定的用户密码root/admin321 登陆后请修改
如果在部署中反复操作，导致登录密码错误之类，请kubectl delete -f gitlab.yaml，并删除gitlab持久化数据目录再重新部署
设置成中文
7 配置ssh下载上传
点击Create a project创建一个新的项目
创建完成后，我们可以添加本地用户的一个SSH-KEY，这样我们就可以通过 SSH 来拉取或者推送代码了。SSH 公钥通常包含在~/.ssh/id_rsa.pub 文件中，并以ssh-rsa开头。如果没有的话可以使用ssh-keygen命令来生成，id_rsa.pub里面的内容就是我们需要的 SSH 公钥，然后添加到 Gitlab 中。
由于平时使用的 ssh 默认是 22 端口，现在如果用默认的 22 端口去连接，是没办法和 Gitlab 容器中的 22 端口进行映射的，因为我们只是通过 Service 的 22 端口进行了映射，要想通过节点去进行 ssh 链接就需要在节点上一个端口和容器内部的22端口进行绑定，所以这里我们可以通过 NodePort 去映射 Gitlab 容器内部的22端口，比如我们将环境变量设置为GITLAB_SSH_PORT=30022，将 Gitlab 的 Service 也设置为 NodePort 类型：
前面配置文件已经添加
# kubectl -n kube-ops get svc|grep gitlab gitlab NodePort 10.108.77.201 <none> 80:30419/TCP,22:30022/TCP 11m
我们在项目上面 Clone 的时候使用 ssh 就会带上端口号了：
配置公钥
https://www.cnblogs.com/wangxu01/articles/11058659.html
[root@k8s-master test]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:XP5breGwn4wNJrp9sPfmOGj2MeSzZrr23N6pxbsQW2o root@k8s-master The key's randomart image is: +---[RSA 2048]----+ | | | | | . | | . o | | S . .. . | | .+ *. | | .+XEoo.| | o*+B^+=o| | o=+XXO&+o| +----[SHA256]-----+ [root@k8s-master test]# cd ~/.ssh/ [root@k8s-master .ssh]# ls id_rsa id_rsa.pub known_hosts [root@k8s-master .ssh]# cat id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQ/mJYwmzINjNKfFL+o91WqQk27ciaEAuZkrpJCUPGzkyOf8m3YawqR6K2U5AV14lVpRhkHBsPuMrYRNH9h41LOExXEd9w+sni340hjg4bL+N7yPygct3yr3vnEurh8+CuICldkw4MZNPxLiPoCYJrXtNHQd7pMW0vbf6OyRigOjRgQ8VO0oKNUJ/WKeneKUNcOceyJaIh3lSUvkkQYUKyip9v9gVaDRv7BAdanhwmQu0LWiCdZSguYEwq0+DTIiHr1/GyaZB1bdEgQO4Fp8sryNHeFWfPvIWiKEt0mo/YuIF5DttahEIxrdjOoEjG6c09DTumf7r9fWlChNWjfBm1 root@k8s-master [root@k8s-master .ssh]#
下载
[root@k8s-master test]# git clone ssh://git@gitlab.wangxu.com:30022/root/gitlab-demo.git 正克隆到 'gitlab-demo'... The authenticity of host '[gitlab.wangxu.com]:30022 ([10.6.76.24]:30022)' can't be established. ECDSA key fingerprint is SHA256:wFtUeSNMyj0tav79o7IynrNlt7wNV57ADLbGRh0ZXgg. ECDSA key fingerprint is MD5:cb:3f:6f:78:15:7f:11:9e:7b:75:46:23:7d:b1:90:2b. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[gitlab.wangxu.com]:30022,[10.6.76.24]:30022' (ECDSA) to the list of known hosts. remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0) 接收对象中: 100% (3/3), done. [root@k8s-master test]# ls gitlab-demo [root@k8s-master test]#
上传
上传 [root@k8s-master gitlab-demo]# echo '123' >> test.index [root@k8s-master gitlab-demo]# git add . [root@k8s-master gitlab-demo]# git commit -m 'add test.index' *** Please tell me who you are. Run git config --global user.email "you@example.com" git config --global user.name "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. fatal: unable to auto-detect email address (got 'root@k8s-master.(none)') [root@k8s-master gitlab-demo]# echo $? 128 [root@k8s-master gitlab-demo]# git config --global user.email "314144952@qq.com" [root@k8s-master gitlab-demo]# git config --global user.name "wx" [root@k8s-master gitlab-demo]# git commit -m 'add test.index' [master 2570c66] add test.index 1 file changed, 1 insertion(+) create mode 100644 test.index [root@k8s-master gitlab-demo]# git push origin master Counting objects: 4, done. Delta compression using up to 4 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 273 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://git@gitlab.wangxu.com:30022/root/gitlab-demo.git 631fb39..2570c66 master -> master
