gitlab 502问题解决
问题现象：
gitlab 默认启用 Prometheus，数据存储时长是 15天，经观察磁盘占用较大（我这边平均每天1G）
如果磁盘紧张的情况下可以修改数据保留时长，或直接关闭 Prometheus 监控，修改配置文件的方法如下：
配置文件位置 gitlab/config/gitlab.rb
1、修改保留天数，下面的配置内容默认全部是注释掉的，可以去掉前面的 # ，然后将15d修改为对应所需的数值即可。
prometheus['flags'] = {
'storage.tsdb.path' => "/var/opt/gitlab/prometheus/data",
# 'storage.tsdb.retention.time' => "15d",
'storage.tsdb.retention.time' => "2d",
'config.file' => "/var/opt/gitlab/prometheus/prometheus.yml"
2、直接关闭该功能，将配置文件中的 # prometheus['enable'] = true 取消注释后修改为 prometheus['enable'] = false
# prometheus['enable'] = true
prometheus['enable'] = false
最后重启 gitlab 即可，多余的文件会被自动删除。
二、还是502，进一步排查
root@gitlab:/# gitlab-ctl status
run: alertmanager: (pid 347) 15s; run: log: (pid 343) 15s
run: gitaly: (pid 474) 14s; run: log: (pid 329) 15s
run: gitlab-exporter: (pid 356) 15s; run: log: (pid 354) 15s
run: gitlab-kas: (pid 346) 15s; run: log: (pid 338) 15s
run: gitlab-workhorse: (pid 345) 15s; run: log: (pid 336) 15s
run: grafana: (pid 342) 15s; run: log: (pid 333) 15s
run: logrotate: (pid 353) 15s; run: log: (pid 349) 15s
run: nginx: (pid 352) 15s; run: log: (pid 348) 15s
run: postgres-exporter: (pid 330) 15s; run: log: (pid 324) 15s
down: postgresql: 1s, normally up, want up; run: log: (pid 332) 15s
run: prometheus: (pid 355) 15s; run: log: (pid 351) 15s
run: puma: (pid 602) 2s; run: log: (pid 328) 15s
run: redis: (pid 331) 15s; run: log: (pid 325) 15s
run: redis-exporter: (pid 341) 15s; run: log: (pid 334) 15s
run: sidekiq: (pid 344) 15s; run: log: (pid 335) 15s
run: sshd: (pid 37) 26s; run: log: (pid 36) 26s
root@gitlab:/# root@gitlab:~# docker exec -it gitlab bash
root@gitlab:/# gitlab-ctl tail postgresql
postgresql没有正常，查看日志
root@gitlab:/# gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/postgresql/current <==
2024-02-04_03:30:08.79426 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:08.79427 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:09.79872 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:09.79874 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:10.80367 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:10.80368 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:11.80861 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:11.80862 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:12.81352 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:12.81353 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:13.81798 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:13.81800 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:14.82294 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:14.82296 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:15.82814 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:15.82817 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
2024-02-04_03:30:16.83341 FATAL: lock file "postmaster.pid" is empty
2024-02-04_03:30:16.83343 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
^CTraceback (most recent call last):
之前的服务器启动崩溃：如果之前的 PostgreSQL 服务器启动过程中发生了崩溃，可能会留下一个空的锁文件，这会阻止新的 PostgreSQL 实例启动。
查找并删除空的锁文件：在 PostgreSQL 的数据目录中查找名为 postmaster.pid 的文件，该文件是用于跟踪 PostgreSQL 进程的锁文件。如果发现该文件为空，则可以尝试删除它。
cd /var/opt/gitlab/postgresql/data/
rm -fr postmaster.pid
启动 postgresql服务
root@gitlab:~# gitlab-ctl start postgresql
ok: run: postgresql: (pid 767) 25s
检测服务，服务正常，服务恢复
root@gitlab:~# gitlab-ctl status
run: alertmanager: (pid 343) 91s; run: log: (pid 341) 91s
run: gitaly: (pid 518) 88s; run: log: (pid 315) 91s
run: gitlab-exporter: (pid 332) 91s; run: log: (pid 322) 91s
run: gitlab-kas: (pid 334) 91s; run: log: (pid 325) 91s
run: gitlab-workhorse: (pid 342) 91s; run: log: (pid 338) 91s
run: grafana: (pid 330) 91s; run: log: (pid 319) 91s
run: logrotate: (pid 340) 91s; run: log: (pid 336) 91s
run: nginx: (pid 329) 91s; run: log: (pid 320) 91s
run: postgres-exporter: (pid 324) 91s; run: log: (pid 316) 91s
run: postgresql: (pid 767) 39s; run: log: (pid 318) 91s
run: prometheus: (pid 333) 91s; run: log: (pid 326) 91s
run: puma: (pid 761) 41s; run: log: (pid 317) 91s
run: redis: (pid 335) 91s; run: log: (pid 313) 91s
run: redis-exporter: (pid 331) 91s; run: log: (pid 321) 91s
run: sidekiq: (pid 756) 41s; run: log: (pid 337) 91s
run: sshd: (pid 29) 101s; run: log: (pid 28) 101s
root@gitlab:~#
