crontab 定时任务执行失败

Linux操作系统中利用ulimit限制shell启动进程占用的资源，ulimit支持对用户的打开进程数、进程打开文件句柄数、进程打开文件的大小、进程coredump文件大小等资源进行限制，从而防止某个用户进程过度占用系统资源，避免影响整个操作系统和其他应用的正常运行;

• 说说Linux操作系统中nproc的那些事
• Centos ulimit设置
• 修改linux 最大进程数

问题描述

Apr 20 15:38:01 localhost crond[5248]: (xd) RELOAD (/var/spool/cron/xd)
Apr 20 15:38:02 localhost crond[50591]: (CRON) ERROR (setreuid failed): Resource temporarily unavailable
Apr 20 15:38:02 localhost crond[50591]: (xd) ERROR (failed to change user)

问题分析

# 报错详情
[xd@im2 log]$ sudo cat cron |grep ERROR
Apr 20 15:38:01 localhost crond[5248]: (xd) RELOAD (/var/spool/cron/xd)
Apr 20 15:38:02 localhost crond[50591]: (CRON) ERROR (setreuid failed): Resource temporarily unavailable
Apr 20 15:38:02 localhost crond[50591]: (xd) ERROR (failed to change user)

# 排查
[xd@im2 log]$ ps -U xd -u xd -Lf |wc -l  # 查看当前用户进程线程数
2345
[xd@im2 log]$ ps -eLo ruser| awk 'NR>1'| sort -nr| uniq -c
   2346 xd
     64 weihu
     21 toptea
      3 rtkit
      1 rpcuser
      1 rpc
   1210 root
      2 postfix
      1 ntp
    922 mq
      3 haldaemon
     21 gdm
      2 dbus

[xd@im2 log]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 4134578
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024          # nofile
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4134578       # npro
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

# npro root 默认是cat /proc/sys/kernel/threads-max 的值 / 2
# npro 普通用户 默认是 /etc/security/limits.d/20-nproc.conf (centos6 是 90-nproc.conf)
# 注意：普通用户 max user processes 值是不生效的，需要修改 /etc/security/limits.d/20-nproc.conf 文件中的值。

问题解决

[xd@im2 log]$ more /etc/security/limits.conf
# End of file

[xd@im2 log]$ more /etc/security/limits.d/90-nproc.conf  # 针对普通用户
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited

*          soft    nproc     10240
root       soft    nproc     unlimited
*          soft    nofile    10240
*          hard    nofile    10240