项目环境出现PHP 502 Bad Gateway 问题排查
一、现象
昨天运维人员被告知，在升级完客户集群环境后，访问管理页面偶尔会报 502 Bad Gateway。登录客户环境，发现只要请求分发到 node2，就会报 502，开始解决问题...
二、排查思路
1、看到 502 第一时间想到的应该是 php-fpm 出问题了，先看下 nginx 日志，连接被对端关闭，说明 php-fpm 进程提前退出了，那是什么原因导致 php-fpm 退出了呢？
2、查看 php-fpm 日志，有很多 warning 级别日志，而且都是重复的退出、重建。网上有很多文章说调整 php-fpm 执行时间之类，作用不大，如果报 504 可以考虑适当调整执行时间参数。这时修改 php-fpm 为如下配置，保存重启 php-fpm
pm.max_children = 1 pm.start_servers = 1 pm.min_spare_servers = 1 pm.max_spare_servers = 1
3、使用 strace 命令追踪 php-fpm worker 进程
[mlcdm@node2 ~]$ sudo strace -p 9277 strace: Process 9277 attached accept(5, {sa_family=AF_INET, sin_port=htons(46636), sin_addr=inet_addr("127.0.0.1")}, [112->16]) = 3 poll([{fd=3, events=POLLIN}], 1, 5000) = 1 ([{fd=3, revents=POLLIN}]) times({tms_utime=0, tms_stime=0, tms_cutime=0, tms_cstime=0}) = 8082094440 read(3, "\1\1\0\1\0\10\0\0", 8) = 8 read(3, "\0\1\0\0\0\0\0\0", 8) = 8 read(3, "\1\4\0\1\3a\7\0", 8) = 8 read(3, "\0174SCRIPT_FILENAME/var/www/mlclou"..., 872) = 872 read(3, "\1\4\0\1\0\0\0\0", 8) = 8 lstat("/var/www/mlcloud/public//index.php/api/appfile/pluto", 0x7ffd136a20b0) = -1 ENOTDIR (Not a directory) stat("/var/www/mlcloud/public//index.php/api/appfile", 0x7ffd136a44a0) = -1 ENOTDIR (Not a directory) stat("/var/www/mlcloud/public//index.php/api", 0x7ffd136a44a0) = -1 ENOTDIR (Not a directory) stat("/var/www/mlcloud/public//index.php", {st_mode=S_IFREG|0666, st_size=1604, ...}) = 0 read(3, "\1\5\0\1\0\275\3\0", 8) = 8 read(3, "method=dbf.task.offlinetimeoutta"..., 189) = 189 read(3, "\0\0\0", 3) = 3 stat("/var/www/mlcloud/public/.user.ini", 0x7ffd136a2fc0) = -1 ENOENT (No such file or directory) rt_sigaction(SIGPROF, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGPROF, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGHUP, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fc5c4ae3630}, 8) = 0 rt_sigaction(SIGINT, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 rt_sigaction(SIGQUIT, NULL, {sa_handler=0xa3ac20, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fc5c4ae3630}, 8) = 0 rt_sigaction(SIGQUIT, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fc5c4ae3630}, 8) = 0 rt_sigaction(SIGTERM, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 rt_sigaction(SIGUSR1, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fc5c4ae3630}, 8) = 0 rt_sigaction(SIGUSR1, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 rt_sigaction(SIGUSR2, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fc5c4ae3630}, 8) = 0 rt_sigaction(SIGUSR2, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 setitimer(ITIMER_PROF, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=60, tv_usec=0}}, NULL) = 0 rt_sigaction(SIGPROF, {sa_handler=0x9d26a0, sa_mask=~[ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fc5c4ae3630}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0 lstat("/var/www/mlcloud/public//index.php", {st_mode=S_IFREG|0666, st_size=1604, ...}) = 0 lstat("/var/www/mlcloud/public", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0 lstat("/var/www/mlcloud", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 lstat("/var/www", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 open("/var/www/mlcloud/public/index.php", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0666, st_size=1604, ...}) = 0 getcwd("/home/mlcdm", 4095) = 12 chdir("/var/www/mlcloud/public/") = 0 setitimer(ITIMER_PROF, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=30, tv_usec=0}}, NULL) = 0 stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffd136a1470) = -1 ENOENT (No such file or directory) open("/var/www/mlcloud/public//index.php", O_RDONLY) = 6 fstat(6, {st_mode=S_IFREG|0666, st_size=1604, ...}) = 0 fstat(6, {st_mode=S_IFREG|0666, st_size=1604, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc5c5145000 read(6, "<?php\n\n// Valid PHP Version?\n$mi"..., 4096) = 1604 close(6) = 0 munmap(0x7fc5c5145000, 4096) = 0 stat("/tmp", {st_mode=S_IFDIR|0700, st_size=61440, ...}) = 0 open("/tmp/tmpftRfMVP", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EACCES (Permission denied) --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} --- +++ killed by SIGSEGV +++
找到问题了，php-fpm 在 /tmp/ 目录创建临时文件失败，导致进程退出，页面响应 502
4、看下 /tmp/ 目录 属主及权限，OMG，运维部署时忘记修改 node2 /tmp/ 目录权限了，因为 php-fpm 进程属主是 www，导致无法在此目录写文件
[mlcdm@node2 ~]$ stat /tmp/ File: ‘/tmp/’ Size: 61440 Blocks: 128 IO Block: 4096 directory Device: fd01h/64769d Inode: 524291 Links: 12 Access: (0700/drwx------) Uid: ( 1003/ mlcdm) Gid: ( 1003/ mlcdm) Access: 2024-07-24 15:00:24.774052000 +0800 Modify: 2024-07-24 15:16:39.705052000 +0800 Change: 2024-07-24 15:16:39.705052000 +0800 Birth: -
5、修改 /tmp/ 目录权限，这里为什么是 1777 ，是因为需要让 /tmp/ 保持粘滞位特权，表示任何用户都可以在 tmp 目录创建文件，但是只能由文件的创建者或 root 才能删除，防止文件被意外删除导致服务异常
[mlcdm@node2 ~]$ sudo chmod 1777 /tmp/
三、总结
1、当页面响应 502 时，多半是网关程序（CGI）异常，比如服务未启动，异常退出等原因导致的，可着重排查CGI程序
2、strace 命令功能强大，对了解进程具体做了哪些系统调用很有帮助
3、linux 用户权限是遇到问题时第一个想到一个点，可以参考鸟哥的私房菜用户权限章节
