Gitlab 快速部署及日常维护 (二)
一、概述
上一篇我们将Gitlab的安装部署和初始化设置部分全部讲解完成了,接下来我们介绍Gitlab在日常工作中常遇见的问题进行梳理说明。
二、Gitlab的安装和维护过程中常见问题
1、Gitlab访问出现403"Forbidden"现象
问题原因分析:
可能因较多的并发导致的访问被拒绝, Gitlab使用rack_attack做了并发访问的限制!
解决办法:
打开/etc/gitlab/gitlab.rb文件,查找 gitlab_rails['rack_attack_git_basic_auth'] 关键词,取消注释,
修改ip_whitelist白名单属性,加入Gitlab部署的IP地址。
[root@gitlab ~]# vim /etc/gitlab/gitlab.rb ...... gitlab_rails['rack_attack_git_basic_auth'] = { 'enabled' => true, 'ip_whitelist' => ["127.0.0.1","172.16.60.222"], //把gitlab服务器IP地址添加 'maxretry' => 10, 'findtime' => 60, 'bantime' => 3600 }
然后进行重新配置
[root@gitlab ~]# gitlab-ctl reconfigure
2、Gitlab访问出现502的现象
Gitlab访问出现:Whoops, GitLab is taking too much time to respond.
问题原因分析:
1)unicorn原8080默认端口被容器中别的进程已经占用,必须调整为没用过的
2)gitlab的timeout设置过小,默认为60
解决办法:
1)关闭gitlab服务
[root@gitlab ~]# gitlab-ctl stop
2)选择一个没有被系统占用的端口作为unicorn端口,比如8877端口(lsof -i:8877 确认此端口没有被占用)
[root@gitlab ~]# vim /etc/gitlab/gitlab.rb unicorn['port'] = 8877 gitlab_workhorse['auth_backend'] = "http://localhost:8877"
3)重新生成配置,并进行重启。
[root@gitlab ~]# gitlab-ctl reconfigure [root@gitlab ~]# gitlab-ctl restart
3、Gitlab启动失败,或重新安装时出现卡的状态
问题现象:在卸载gitlab然后再次安装执行sudo gitlab-ctl reconfigure的时候往往会出现:ruby_block[supervise_redis_sleep] action run,会一直卡无法往下进行!
解决办法:
1)按ctrl + c 强制结束
2)执行"systemctl restart gitlab-runsvdir" 命令
3)接着再执行"gitlab-ctl reconfigure"
如果Gitlab服务器重启后,启动"gitlab-ctl start"失败,解决办法相同。
[root@gitlab ~]# systemctl restart gitlab-runsvdir [root@gitlab ~]# gitlab-ctl reconfigure [root@gitlab ~]# gitlab-ctl start
4、Gitlab异常关机,导致gitlab启动失败!gitlab-runsvdir方式启动没反应(僵尸状态)
问题现象:Gitlab部署的服务器异常断电,机器重启后,尝试启动gitlab服务,启动失败!通过gitlab-runsvdir方式启动一直没有反应!一直在卡顿状态!日志也没有任务输入!
执行下面的启动命令报错: [root@gitlab ~]# gitlab-ctl start // 或者 "gitlab-ctl restart" fail: alertmanager: runsv not running fail: gitaly: runsv not running fail: gitlab-monitor: runsv not running fail: gitlab-workhorse: runsv not running fail: logrotate: runsv not running fail: nginx: runsv not running fail: node-exporter: runsv not running fail: postgres-exporter: runsv not running fail: postgresql: runsv not running fail: prometheus: runsv not running
报错说
"runsv not running"
那么尝试通过supervisor进程方式启动gitlab,发现一直在卡顿中,根本没有任何反应!
[root@gitlab ~]# systemctl restart gitlab-runsvdir
查看日志,发现也没有任务启动信息打印到日志中 (日志都是之前的)
[root@gitlab ~]# /usr/bin/gitlab-ctl tail
gitlab-runsvdir启动在卡顿中,gitlab服务也没有起来
[root@gitlab ~]# ps -ef|grep gitlab
解决方法:
通过Gitlab自己原生命令去启动服务:
/opt/gitlab/embedded/bin/runsvdir-start
root@gitlab ~]# cat /etc/systemd/system/multi-user.target.wants/gitlab-runsvdir.service [Unit] Description=GitLab Runit supervision process After=multi-user.target [Service] ExecStart=/opt/gitlab/embedded/bin/runsvdir-start #最后通过这条命令启动了Gitlab Restart=always [Install] WantedBy=multi-user.target 执行下面的启动,虽然发现这个也会一直在卡顿中,但是不影响gitlab服务启动。 [root@gitlab ~]# /opt/gitlab/embedded/bin/runsvdir-start 重新打开一个终端窗口,发现gitlab已经有新的日志信息打入了,gitlab也服务已经起来了 [root@gitlab ~]# /usr/bin/gitlab-ctl tail [root@gitlab ~]# ps -ef|grep gitlab 这时候关闭上面执行"/opt/gitlab/embedded/bin/runsvdir-start"的卡顿的终端窗口,发现gitlab也还是启动状态(ps -ef|grep gitlab) [root@gitlab ~]# ps -ef|grep gitlab [root@gitlab ~]# lsof -i:80 [root@gitlab ~]# gitlab-ctl status run: alertmanager: (pid 29804) 1640s; run: log: (pid 29789) 1640s run: gitaly: (pid 29795) 1640s; run: log: (pid 29781) 1640s run: gitlab-monitor: (pid 29799) 1640s; run: log: (pid 29785) 1640s run: gitlab-workhorse: (pid 29794) 1640s; run: log: (pid 29780) 1640s run: logrotate: (pid 29798) 1640s; run: log: (pid 29783) 1640s run: nginx: (pid 29800) 1640s; run: log: (pid 29786) 1640s run: node-exporter: (pid 29802) 1640s; run: log: (pid 29788) 1640s run: postgres-exporter: (pid 29805) 1640s; run: log: (pid 29790) 1640s run: postgresql: (pid 29796) 1640s; run: log: (pid 29782) 1640s run: prometheus: (pid 29797) 1640s; run: log: (pid 29784) 1640s run: redis: (pid 29818) 1640s; run: log: (pid 29793) 1640s run: redis-exporter: (pid 29817) 1640s; run: log: (pid 29792) 1640s run: sidekiq: (pid 29801) 1640s; run: log: (pid 29787) 1640s run: unicorn: (pid 29807) 1640s; run: log: (pid 29791) 1640s 查看日志也有新信息写入,一切正常了! [root@gitlab ~]# /usr/bin/gitlab-ctl tail
5、Gitlab重新安装,在执行"gitlab-ctl reconfigure"配置环节出现了下面报错:
[root@gitlab ~]# gitlab-ctl reconfigure ......... ......... STDERR: sysctl: cannot open "/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf": No such file or directory sysctl: cannot open "/etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf": No such file or directory ---- End output of sysctl -e --system ---- Ran sysctl -e --system returned 255
问题原因分析:
丢失了报错中的这两个配置文件,进入/etc/sysctl.d目录发现,这两个文件都是通过链接到/opt/gitlab/embedded/etc/目录下。
然而/opt/gitlab/embedded/etc/确实没有这两个文件。
[root@gitlab ~]# ll /etc/sysctl.d/ total 0 lrwxrwxrwx 1 root root 58 Nov 10 22:23 90-omnibus-gitlab-kernel.sem.conf -> /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf lrwxrwxrwx 1 root root 61 Nov 10 22:23 90-omnibus-gitlab-kernel.shmall.conf -> /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf lrwxrwxrwx 1 root root 61 Nov 10 22:23 90-omnibus-gitlab-kernel.shmmax.conf -> /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf lrwxrwxrwx 1 root root 66 Nov 10 22:25 90-omnibus-gitlab-net.core.somaxconn.conf -> /opt/gitlab/embedded/etc/90-omnibus-gitlab-net.core.somaxconn.conf lrwxrwxrwx. 1 root root 14 Oct 30 09:13 99-sysctl.conf -> ../sysctl.conf [root@gitlab ~]# ll /opt/gitlab/embedded/etc total 12 -rw-r--r-- 1 root root 24 Apr 12 23:18 90-omnibus-gitlab-kernel.shmall.conf -rw-r--r-- 1 root root 28 Apr 12 23:17 90-omnibus-gitlab-kernel.shmmax.conf -rwxr-xr-x 1 root root 196 Apr 12 23:16 gitconfig [root@gitlab ~]# ll /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf ls: cannot access /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf: No such file or directory [root@gitlab ~]# ll /opt/gitlab/embedded/etc/90-omnibus-gitlab-net.core.somaxconn.conf ls: cannot access /opt/gitlab/embedded/etc/90-omnibus-gitlab-net.core.somaxconn.conf: No such file or directory 解决方法一: 从别的备份机(或者在别的机器上重新安装一次,"gitlab-ctl reconfigure"之后生成这两个文件)将这两个文件拷贝回来! 解决方法二: [root@gitlab ~]# vim /etc/gitlab/gitlab.rb # unicorn['port'] = 8080 修改为: unicorn['port'] = 8090 之后重新加载配置文件 [root@gitlab ~]# gitlab-ctl reconfigure 再次会报错,然后再修改/etc/gitlab/gitlab.rb,修改为原来的配置 [root@gitlab ~]# vim /etc/gitlab/gitlab.rb # unicorn['port'] = 8080 再次重新加载配置文件就OK了! [root@gitlab ~]# gitlab-ctl reconfigure 再次查看,发现上面配置中报错的两个文件已经存在了 [root@gitlab ~]# ll /opt/gitlab/embedded/etc/ total 20 -rw-r--r-- 1 root root 30 Apr 12 23:33 90-omnibus-gitlab-kernel.sem.conf -rw-r--r-- 1 root root 24 Apr 12 23:18 90-omnibus-gitlab-kernel.shmall.conf -rw-r--r-- 1 root root 28 Apr 12 23:17 90-omnibus-gitlab-kernel.shmmax.conf -rw-r--r-- 1 root root 26 Apr 12 23:35 90-omnibus-gitlab-net.core.somaxconn.conf -rwxr-xr-x 1 root root 196 Apr 12 23:16 gitconfig [root@gitlab ~]# ll /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf -rw-r--r-- 1 root root 30 Apr 12 23:33 /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf [root@gitlab ~]# ll /opt/gitlab/embedded/etc/90-omnibus-gitlab-net.core.somaxconn.conf -rw-r--r-- 1 root root 26 Apr 12 23:35 /opt/gitlab/embedded/etc/90-omnibus-gitlab-net.core.somaxconn.conf 最后再启动gitlab [root@gitlab ~]# gitlab-ctl start
6、Gitlab更改默认Nginx
更换gitlab自带Nginx,使用自行编译Nginx来管理gitlab服务。
自行编译的nginx服务和gitlab在同一台机器上 1)编辑gitlab配置文件禁用自带Nignx服务器 [root@gitlab ~]# vim /etc/gitlab/gitlab.rb ... #设置nginx为false,关闭自带Nginx nginx['enable'] = false ... 2)检查默认nginx配置文件,并迁移至新Nginx服务 (即将下面两个gitlab自带nginx的配置文件迁移到自行编译的新的nginx配置中) /var/opt/gitlab/nginx/conf/nginx.conf #nginx配置文件,包含gitlab-http.conf文件 /var/opt/gitlab/nginx/conf/gitlab-http.conf #gitlab核心nginx配置文件 [root@gitlab ~]# cp /var/opt/gitlab/nginx/conf/nginx.conf /etc/nginx/conf.d/ [root@gitlab ~]# cp /var/opt/gitlab/nginx/conf/gitlab-http.conf /etc/nginx/conf.d/ 3)重启gitlab服务 [root@gitlab ~]# gitlab-ctl reconfigure [root@gitlab ~]# gitlab-ctl restart 重启自行编译的nginx服务 [root@gitlab ~]# service nginx restart 如果访问报502。原因是nginx用户无法访问gitlab用户的socket文件。 重启gitlab需要重新授权 [root@gitlab ~]# chmod -R o+x /var/opt/gitlab/gitlab-rails
-----------------------------------------------------------书山有路勤为径,学海无涯苦作舟-------------------------------------------------------------