转:Red Hat Enterprise Linux systems using the TSC clock source reboots or panics when 'sched_clock()' overflows after an uptime of 208.5 days
虽说是老贴了,但是最近还接触到
原文:https://access.redhat.com/site/solutions/68466
问题
- Linux Kernel panics when
sched_clock()
overflows after an uptime of around 208.5 days - Red Hat Enterprise Linux system reboots with sched_clock() overflow after an uptime of around 208.5 days
- This symptom may happen on a system using the Time Stamp Counter (TSC) clock source
- Some processes may generate following log message:
BUG: soft lockup - CPU#N stuck for 4278190091s!
环境
- Red Hat Enterprise Linux (RHEL) 6
- Red Hat Enterprise Linux 6.0 (some kernels)
- Red Hat Enterprise Linux 6.1 (some kernels)
- Red Hat Enterprise Linux 6.2 (some kernels)
- With TSC clock source
- Red Hat Enterprise Linux (RHEL) 5
- Red Hat Enterprise Linux 5.0
- Red Hat Enterprise Linux 5.1
- Red Hat Enterprise Linux 5.2
- Red Hat Enterprise Linux 5.3 (some kernels)
- Red Hat Enterprise Linux 5.4
- Red Hat Enterprise Linux 5.5
- Red Hat Enterprise Linux 5.6 (some kernels)
- Red Hat Enterprise Linux 5.7
- Red Hat Enterprise Linux 5.8 (some kernels)
- With TSC clock source
- Red Hat Enterprise MRG 1.3 Realtime kernel
- An approximate uptime of around 208.5 days
决议
- Red Hat Enterprise Linux 6
- RHEL 6.x
- Update to
kernel-2.6.32-279.el6
or later (RHSA-2012-0862) - This kernel is already part of Red Hat Enterprise Linux 6.3 GA
- Update to
- RHEL 6.2
- Update to
kernel-2.6.32-220.4.2.el6
or later (RHBA-2012-0124)
- Update to
- RHEL 6.1 Extended Update Support
- Update to
kernel-2.6.32-131.26.1.el6
or later (RHBA-2012-0424)
- Update to
- RHEL 6.x
- Red Hat Enterprise Linux 5 x86_64 (64bit)
- RHEL 5.x
- Update to
kernel-2.6.18-348.el5
or later (RHBA-2013-0006) - Red Hat Enterprise Linux 5.9 GA and later already contain this fix
- Update to
- RHEL 5.8.z
- Update to
kernel-2.6.18-308.11.1.el5
or later (RHSA-2012-1061)
- Update to
- RHEL 5.6.z
- Update to
kernel-2.6.18-238.40.1.el5
or later (RHSA-2012-1087)
- Update to
- RHEL 5.3.z
- Update to
kernel-2.6.18-128.39.1.el5
or later (RHBA-2012-1093)
- Update to
- RHEL 5.x
- Red Hat Enterprise Linux 5 x86 (32bit)
- RHEL 5.x
- Update to
kernel-2.6.18-348.el5
or later (RHBA-2013-0006) - Red Hat Enterprise Linux 5.9 GA and later already contain this fix
- Update to
- RHEL 5.8.z
- Update to
kernel-2.6.18-308.13.1.el5
or later (RHSA-2012-1174)
- Update to
- RHEL 5.6.z
- Update to kernel-2.6.18-238.40.1.el5 or later (RHSA-2012-1087)
- RHEL 5.3.z
- Update to kernel-2.6.18-128.39.1.el5 or later (RHBA-2012-1093)
- RHEL 5.x
- Red Hat Enterprise 1.3 Realtime kernel
- Red Hat Enterprise MRG 1.3 Realtime kernel
- Update to kernel kernel-rt-2.6.33.9-rt31.86.el5rt or later (RHBA-2013:0927)
- Red Hat Enterprise MRG 1.3 Realtime kernel
根源
-
An insufficiently designed calculation in the CPU accelerator in the previous kernel caused an arithmetic overflow in the
sched_clock()
function- This overflow led to a kernel panic or any other unpredictable trouble on the systems using the TSC clock source
- This problem will occur only when system uptime reaches or exceeds 208.5 days
- This update corrects the aforementioned calculation so that this arithmetic overflow and kernel panic can no longer occur under these circumstances
-
On RHEL5, this problem is a timing issue and is very unlikely to be encountered.
-
Switching to another clocksource is usually not a workaround for most workloads
- The TSC is a fast access clock, whereas the HPET and PMTimer are both slow access clocks
- Using notsc would be a significant performance hit
- In RHEL5, the affected
sched_clock()
uses the TSC regardless of clock source selection. - Also, in some situation, the system may hit this issue even if you set notsc to current_clocksource.
诊断步骤
-
Note: this issue could occur in numerous locations that deal with time in the kernel
- For example, a user running a non-Red Hat kernel could have a kernel panic with a soft lockup in
__ticket_spin_lock
- For example, a user running a non-Red Hat kernel could have a kernel panic with a soft lockup in
-
The system must be booting a kernel that is a version prior to releases mentioned in the "Resolution" field
=============================