kernel: audit: printk limit exceeded

问题：

小长假的第一天早上8：18一个数据，被定时任务中的脚本漏处理；

查定时任务的日志，发现调度异常

 

查var messages-20171231 日志信息，排查问题。

 

http://man7.org/linux/man-pages/man3/audit_set_rate_limit.3.html

 

 

 

87884 Dec 30 08:18:01 hadoop1 kernel: type=1006 audit(1514593081.573:140874): pid=13198 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18447 res=1
87888 Dec 30 08:18:01 hadoop1 systemd: Started Session 18447 of user root.
87889 Dec 30 08:18:01 hadoop1 systemd: Starting Session 18447 of user root.
87890 Dec 30 08:18:01 hadoop1 systemd: Started Session 18448 of user root.
87902 Dec 30 08:19:01 hadoop1 kernel: type=1006 audit(1514593141.037:140895): pid=16717 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18450 res=1
87905 Dec 30 08:19:01 hadoop1 kernel: type=1006 audit(1514593141.037:140898): pid=16716 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18451 res=1
87906 Dec 30 08:19:01 hadoop1 systemd: Started Session 18450 of user root.
87907 Dec 30 08:19:01 hadoop1 systemd: Starting Session 18450 of user root.
87913 Dec 30 08:19:11 hadoop1 kernel: audit_printk_skb: 9 callbacks suppressed
87921 Dec 30 08:20:01 hadoop1 kernel: audit: audit_lost=40257 audit_rate_limit=0 audit_backlog_limit=320
87938 Dec 30 08:20:01 hadoop1 systemd: Started Session 18455 of user root.
87939 Dec 30 08:20:01 hadoop1 systemd: Starting Session 18455 of user root.
87940 Dec 30 08:20:01 hadoop1 systemd: Started Session 18457 of user root.
87949 Dec 30 08:21:01 hadoop1 kernel: type=1006 audit(1514593261.259:140948): pid=22702 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18458 res=1
87951 Dec 30 08:21:01 hadoop1 kernel: type=1006 audit(1514593261.260:140950): pid=22703 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18459 res=1
87952 Dec 30 08:21:01 hadoop1 systemd: Started Session 18458 of user root.
87953 Dec 30 08:21:01 hadoop1 systemd: Starting Session 18458 of user root.
87954 Dec 30 08:21:01 hadoop1 systemd: Started Session 18459 of user root.
87955 Dec 30 08:21:01 hadoop1 systemd: Starting Session 18459 of user root.
87969 Dec 30 08:22:01 hadoop1 kernel: audit: audit_lost=40286 audit_rate_limit=0 audit_backlog_limit=320
87970 Dec 30 08:22:01 hadoop1 kernel: audit: printk limit exceeded
87977 Dec 30 08:22:01 hadoop1 kernel: type=1006 audit(1514593321.110:140973): pid=26096 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18460 res=1
87980 Dec 30 08:22:01 hadoop1 systemd: Started Session 18460 of user root.
87981 Dec 30 08:22:01 hadoop1 systemd: Starting Session 18460 of user root.
87982 Dec 30 08:22:01 hadoop1 systemd: Started Session 18463 of user root.
87992 Dec 30 08:23:01 hadoop1 kernel: type=1006 audit(1514593381.445:140992): pid=29014 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18464 res=1
87994 Dec 30 08:23:01 hadoop1 kernel: type=1006 audit(1514593381.445:140994): pid=29015 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18465 res=1
88011 Dec 30 08:24:01 hadoop1 kernel: audit: audit_lost=40303 audit_rate_limit=0 audit_backlog_limit=320
88021 Dec 30 08:24:01 hadoop1 systemd: Starting Session 18467 of user root.
88026 Dec 30 08:24:01 hadoop1 systemd: Started Session 18469 of user root.
88031 Dec 30 08:24:36 hadoop1 kernel: type=1701 audit(1514593476.620:141031): auid=0 uid=0 gid=0 ses=17824 pid=1962 comm="phantomjs" reason="memory violation" sig=11
88033 Dec 30 08:25:01 hadoop1 kernel: audit: audit_lost=40319 audit_rate_limit=0 audit_backlog_limit=320
88034 Dec 30 08:25:01 hadoop1 kernel: audit: printk limit exceeded
88036 Dec 30 08:25:01 hadoop1 kernel: type=1006 audit(1514593501.674:141035): pid=2897 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18470 res=1
88054 Dec 30 08:26:01 hadoop1 kernel: type=1006 audit(1514593561.575:141052): pid=6260 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18472 res=1
88060 Dec 30 08:26:01 hadoop1 kernel: type=1006 audit(1514593561.575:141058): pid=6258 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18474 res=1
88066 Dec 30 08:26:01 hadoop1 systemd: Started Session 18475 of user root.
88069 Dec 30 08:26:01 hadoop1 systemd: Starting Session 18474 of user root.
88074 Dec 30 08:27:01 hadoop1 kernel: audit: audit_lost=40336 audit_rate_limit=0 audit_backlog_limit=320
88075 Dec 30 08:27:01 hadoop1 kernel: audit: printk limit exceeded
88077 Dec 30 08:27:01 hadoop1 kernel: type=1006 audit(1514593621.900:141077): pid=9667 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18476 res=1
88079 Dec 30 08:27:01 hadoop1 kernel: type=1006 audit(1514593621.900:141079): pid=9668 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18477 res=1
88080 Dec 30 08:27:01 hadoop1 systemd: Started Session 18476 of user root.
88081 Dec 30 08:27:01 hadoop1 systemd: Starting Session 18476 of user root.
88099 Dec 30 08:28:01 hadoop1 kernel: type=1006 audit(1514593681.844:141097): pid=12579 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18479 res=1
88104 Dec 30 08:28:01 hadoop1 systemd: Started Session 18478 of user root.
88119 Dec 30 08:29:01 hadoop1 kernel: type=1006 audit(1514593741.079:141120): pid=15407 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18482 res=1
88120 Dec 30 08:29:01 hadoop1 kernel: type=1006 audit(1514593741.079:141121): pid=15406 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18483 res=1
88121 Dec 30 08:29:01 hadoop1 systemd: Started Session 18482 of user root.
88122 Dec 30 08:29:01 hadoop1 systemd: Starting Session 18482 of user root.
88123 Dec 30 08:29:01 hadoop1 systemd: Started Session 18483 of user root.
88124 Dec 30 08:29:01 hadoop1 systemd: Starting Session 18483 of user root.
88136 Dec 30 08:29:46 hadoop1 kernel: type=1701 audit(1514593786.318:141134): auid=0 uid=0 gid=0 ses=17824 pid=17285 comm="phantomjs" reason="memory violation" sig=11
88143 Dec 30 08:30:01 hadoop1 kernel: type=1006 audit(1514593801.098:141141): pid=17956 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18484 res=1
88156 Dec 30 08:30:01 hadoop1 systemd: Started Session 18484 of user root.
88164 Dec 30 08:31:01 hadoop1 kernel: audit: audit_lost=40382 audit_rate_limit=0 audit_backlog_limit=320
88167 Dec 30 08:31:01 hadoop1 kernel: type=1006 audit(1514593861.301:141177): pid=19500 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18491 res=1
88169 Dec 30 08:31:01 hadoop1 kernel: type=1006 audit(1514593861.302:141179): pid=19501 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18492 res=1
88170 Dec 30 08:31:01 hadoop1 systemd: Started Session 18491 of user root.
88171 Dec 30 08:31:01 hadoop1 systemd: Starting Session 18491 of user root.
88172 Dec 30 08:31:01 hadoop1 systemd: Started Session 18492 of user root.
88173 Dec 30 08:31:01 hadoop1 systemd: Starting Session 18492 of user root.
88177 Dec 30 08:31:11 hadoop1 kernel: audit_printk_skb: 9 callbacks suppressed
88190 Dec 30 08:32:01 hadoop1 kernel: type=1006 audit(1514593921.370:141199): pid=20838 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18493 res=1
88194 Dec 30 08:32:01 hadoop1 kernel: type=1006 audit(1514593921.370:141203): pid=20839 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18495 res=1
88202 Dec 30 08:32:01 hadoop1 systemd: Started Session 18496 of user root.
88203 Dec 30 08:32:01 hadoop1 systemd: Starting Session 18496 of user root.
88208 Dec 30 08:33:01 hadoop1 kernel: audit: audit_lost=40399 audit_rate_limit=0 audit_backlog_limit=320
88209 Dec 30 08:33:01 hadoop1 kernel: audit: printk limit exceeded
88211 Dec 30 08:33:01 hadoop1 kernel: type=1006 audit(1514593981.642:141221): pid=22168 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18497 res=1
88213 Dec 30 08:33:01 hadoop1 kernel: type=1006 audit(1514593981.643:141223): pid=22167 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18498 res=1
88214 Dec 30 08:33:01 hadoop1 systemd: Started Session 18497 of user root.
88221 Dec 30 08:33:11 hadoop1 kernel: audit_printk_skb: 9 callbacks suppressed
88232 Dec 30 08:34:01 hadoop1 kernel: type=1006 audit(1514594041.842:141240): pid=23466 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18499 res=1
88236 Dec 30 08:34:01 hadoop1 kernel: type=1006 audit(1514594041.843:141244): pid=23467 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18501 res=1
88238 Dec 30 08:34:01 hadoop1 systemd: Started Session 18499 of user root.
88239 Dec 30 08:34:01 hadoop1 systemd: Starting Session 18499 of user root.
88240 Dec 30 08:34:01 hadoop1 systemd: Started Session 18500 of user root.
88250 Dec 30 08:35:01 hadoop1 kernel: audit: audit_lost=40415 audit_rate_limit=0 audit_backlog_limit=320
88251 Dec 30 08:35:01 hadoop1 kernel: audit: printk limit exceeded
88253 Dec 30 08:35:01 hadoop1 kernel: type=1006 audit(1514594101.911:141263): pid=24756 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18503 res=1
88271 Dec 30 08:36:01 hadoop1 kernel: audit: printk limit exceeded
88286 Dec 30 08:36:01 hadoop1 systemd: Started Session 18508 of user root.
88287 Dec 30 08:36:01 hadoop1 systemd: Starting Session 18508 of user root.
88288 Dec 30 08:36:12 hadoop1 kernel: audit_printk_skb: 39 callbacks suppressed
88292 Dec 30 08:37:01 hadoop1 kernel: audit: audit_lost=40434 audit_rate_limit=0 audit_backlog_limit=320
88297 Dec 30 08:37:01 hadoop1 kernel: type=1006 audit(1514594221.133:141307): pid=26783 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18510 res=1
88300 Dec 30 08:37:01 hadoop1 systemd: Started Session 18510 of user root.
88301 Dec 30 08:37:01 hadoop1 systemd: Starting Session 18510 of user root.
88305 Dec 30 08:37:11 hadoop1 kernel: audit_printk_skb: 9 callbacks suppressed
88328 Dec 30 08:38:11 hadoop1 kernel: audit_printk_skb: 39 callbacks suppressed
88332 Dec 30 08:39:01 hadoop1 kernel: audit: audit_lost=40452 audit_rate_limit=0 audit_backlog_limit=320
88333 Dec 30 08:39:01 hadoop1 kernel: audit: printk limit exceeded
88335 Dec 30 08:39:01 hadoop1 kernel: type=1006 audit(1514594341.403:141345): pid=28795 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18515 res=1
88337 Dec 30 08:39:01 hadoop1 kernel: type=1006 audit(1514594341.404:141347): pid=28794 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18516 res=1
88338 Dec 30 08:39:01 hadoop1 systemd: Started Session 18515 of user root.
88339 Dec 30 08:39:01 hadoop1 systemd: Starting Session 18515 of user root.
88340 Dec 30 08:39:01 hadoop1 systemd: Started Session 18516 of user root.
88341 Dec 30 08:39:01 hadoop1 systemd: Starting Session 18516 of user root.
88345 Dec 30 08:39:11 hadoop1 kernel: audit_printk_skb: 9 callbacks suppressed
88378 Dec 30 08:40:11 hadoop1 kernel: audit_printk_skb: 81 callbacks suppressed
88382 Dec 30 08:41:01 hadoop1 kernel: audit: audit_lost=40483 audit_rate_limit=0 audit_backlog_limit=320
88383 Dec 30 08:41:01 hadoop1 kernel: audit: printk limit exceeded
88385 Dec 30 08:41:01 hadoop1 kernel: type=1006 audit(1514594461.906:141404): pid=31145 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18524 res=1
88248 Dec 30 08:34:11 hadoop1 kernel: type=1106 audit(1514594051.909:141259): pid=22167 uid=0 auid=0 ses=18498 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/c rond" hostname=? addr=? terminal=cron res=success'
88249 Dec 30 08:35:01 hadoop1 kernel: type=1101 audit(1514594101.911:141260): pid=24757 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/cron d" hostname=? addr=? terminal=cron res=success'
88250 Dec 30 08:35:01 hadoop1 kernel: audit: audit_lost=40415 audit_rate_limit=0 audit_backlog_limit=320
88251 Dec 30 08:35:01 hadoop1 kernel: audit: printk limit exceeded
88252 Dec 30 08:35:01 hadoop1 kernel: type=1103 audit(1514594101.911:141262): pid=24756 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88253 Dec 30 08:35:01 hadoop1 kernel: type=1006 audit(1514594101.911:141263): pid=24756 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18503 res=1
88254 Dec 30 08:35:01 hadoop1 kernel: type=1103 audit(1514594101.912:141264): pid=24757 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88255 Dec 30 08:35:01 hadoop1 kernel: type=1006 audit(1514594101.912:141265): pid=24757 uid=0 old-auid=4294967295 auid=0 old-ses=4294967295 ses=18504 res=1
88256 Dec 30 08:35:01 hadoop1 systemd: Started Session 18503 of user root.
88257 Dec 30 08:35:01 hadoop1 systemd: Starting Session 18503 of user root.
88258 Dec 30 08:35:01 hadoop1 systemd: Started Session 18504 of user root.
88259 Dec 30 08:35:01 hadoop1 systemd: Starting Session 18504 of user root.
88260 Dec 30 08:35:01 hadoop1 kernel: type=1105 audit(1514594101.914:141266): pid=24756 uid=0 auid=0 ses=18503 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/cr ond" hostname=? addr=? terminal=cron res=success'
88261 Dec 30 08:35:01 hadoop1 kernel: type=1110 audit(1514594101.914:141267): pid=24756 uid=0 auid=0 ses=18503 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88262 Dec 30 08:35:01 hadoop1 kernel: type=1105 audit(1514594101.914:141268): pid=24757 uid=0 auid=0 ses=18504 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/cr ond" hostname=? addr=? terminal=cron res=success'
88263 Dec 30 08:35:12 hadoop1 kernel: audit_printk_skb: 9 callbacks suppressed
88264 Dec 30 08:35:12 hadoop1 kernel: type=1104 audit(1514594112.096:141272): pid=23465 uid=0 auid=0 ses=18500 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88265 Dec 30 08:35:12 hadoop1 kernel: type=1106 audit(1514594112.097:141273): pid=23465 uid=0 auid=0 ses=18500 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/c rond" hostname=? addr=? terminal=cron res=success'
88266 Dec 30 08:35:12 hadoop1 kernel: type=1104 audit(1514594112.497:141274): pid=23464 uid=0 auid=0 ses=18502 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88267 Dec 30 08:35:12 hadoop1 kernel: type=1106 audit(1514594112.497:141275): pid=23464 uid=0 auid=0 ses=18502 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/c rond" hostname=? addr=? terminal=cron res=success'
88268 Dec 30 08:35:18 hadoop1 kernel: type=1104 audit(1514594118.160:141276): pid=23466 uid=0 auid=0 ses=18499 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88269 Dec 30 08:35:18 hadoop1 kernel: type=1106 audit(1514594118.160:141277): pid=23466 uid=0 auid=0 ses=18499 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/c rond" hostname=? addr=? terminal=cron res=success'
88270 Dec 30 08:36:01 hadoop1 kernel: type=1101 audit(1514594161.162:141279): pid=25781 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/cron d" hostname=? addr=? terminal=cron res=success'
88271 Dec 30 08:36:01 hadoop1 kernel: audit: printk limit exceeded
88272 Dec 30 08:36:01 hadoop1 kernel: audit: audit_lost=40420 audit_rate_limit=0 audit_backlog_limit=320
88273 Dec 30 08:36:01 hadoop1 kernel: audit: printk limit exceeded
88274 Dec 30 08:36:01 hadoop1 kernel: type=1101 audit(1514594161.182:141281): pid=25780 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/cron d" hostname=? addr=? terminal=cron res=success'
88275 Dec 30 08:36:01 hadoop1 kernel: type=1103 audit(1514594161.193:141282): pid=25780 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88276 Dec 30 08:36:01 hadoop1 kernel: type=1103 audit(1514594161.193:141283): pid=25782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
88277 Dec 30 08:36:01 hadoop1 kernel: type=1103 audit(1514594161.193:141284): pid=25781 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'