Is your Redhat Linux Environment ready for Leap Second Insertion on Jun 30, 2015 ?
What is a Leap Second?
A leap second is a second which is added to Coordinated Universal Time (UTC) in order to synchronize atomic clocks with astronomical time to within 0.9 seconds.A leap second is added (can be removed but this never actually happens in reality) on the last day of June or the last day of December as dictatedby the IERS (International Earth Rotation and Reference Systems Service).
Why Do We Need Leap Seconds?
The reason we have to add a second every now and then, is that Earth’s rotation around its own axis, is gradually slowing down, although very slowly.
Atomic clocks however, are programmed to tick away at pretty much the same speed over millions of years. Compared to the Earth’s rotation – which determines the length of a day – the atomic clocks are simply too accurate.
The difference between UTC and the International Atomic Time (UTC-TAI) after the next leap second has been added on June 30, 2015, will be 36 sec.
Handling of the Leap Second:
Redhat Linux Systems (RHEL 4/5/6/7) running NTP :
Systems running any version of Red Hat Enterprise Linux should automatically account for leap second corrections if they are using the NTP (Network Time Protocol) daemon to synchronize their local timekeeping with an NTP server. During the last day before a leap second correction, NTP servers should notify their clients that a leap second will occur, and at 23:59:59 UTC, the Linux kernel should add or remove an extra second by making the 60th second occur twice or removing it entirely.
Example of leap second insertion happened on 31,Dec-2008
2008-12-31 23:59:59:052549000 UTC <– 1st occurrence of the 60th second
2008-12-31 23:59:59:259988000 UTC
2008-12-31 23:59:59:465214000 UTC
2008-12-31 23:59:59:669629000 UTC
2008-12-31 23:59:59:873936000 UTC
2008-12-31 23:59:59:079184000 UTC <– 2nd occurrence of the 60th second
2008-12-31 23:59:59:284011000 UTC
2008-12-31 23:59:59:488648000 UTC
2008-12-31 23:59:59:692691000 UTC
2008-12-31 23:59:59:896577000 UTC
2009-01-01 00:00:00:052378000 UTC
Redhat Linux Systems (RHEL 4/5/6/7) not running NTP :
Linux systems not using NTP to synchronize their timekeeping will not correct for leap seconds, and the time reported by these systems will have a one-second difference relative to UTC after the leap second correction. To Insert/remove leap second on these hosts, You should either “reset the clock manually after leap seconds occur” or “configure these systems to report time corrected for leap seconds by updating the tzdata package to the latest version available, copying the appropriate file from the /usr/share/zoneinfo/right directory hierarchy to /etc/localtime, and resetting the clock to the correct local time”.
The files in /usr/share/zoneinfo/right contain local time information correctedfor all leap seconds that have occurred since the beginning of the Epoch on 1970-01-01 00:00:00 UTC. The other time zone files in /usr/share/zoneinfo do not have leap second corrections added. After the 2008 leap second, there were 24 leap seconds added since the Epoch. Any application that expects the time to be in UTC will have issues if a right/* timezone is used.
As an example, if a system is in the America/Los_Angeles (US Pacific) time zone, you can reconfigure the system to report leap-second-corrected time by running the following and resetting the clock to Pacific Time:
cp /usr/share/zoneinfo/right/America/Los_Angeles /etc/localtime
After /etc/localtime has been changed glibc will reload this file automatically; in addition, note that adjusting timezones does not change the system time, only the conversion from the system time to local time that happens in applications via glibc functions, such as localtime(), ctime(), etc. A system restart is not required after this update; however, if any applications are caching results from these functions then these applications may need to be restarted after tzdata has been updated.
latest tzdata updates with leap second fix:
for RHEL 4 install tzdata-2015a-1.el4 or later
for RHEL 5 install tzdata-2015a-1.el5 or later
for RHEL 6 install tzdata-2015a-1.el6 or later
for RHEL 7 install tzdata-2015a-1.el7 or later
How to verify Whether your Linux System is Vulnerable to leap second or not?
$ gpg –keyserver pgp.mit.edu –recv 8366b0d9
$ gpg –verify leap_vulnerability.sh.asc leap_vulnerability.sh
If the script is authentic, you should see output similar to this:
gpg: Signature made Mon 23 Feb 2015 12:22:15 PM EST using RSA key ID 8366B0D9
gpg: Good signature from “Red Hat, Inc. (tools key) ”
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 8B12 20FC 564E 9583 2002 05FF 7514 F77D 8366 B0D9
And Use the script as below
$ chmod +x leap_vulnerability.sh
If the target is vulnerable, you will see output similar to:
This system is vulnerable to a performance degradation after the Leap Second Insertion of June 30, 2015.
If the target is not vulnerable, you will see output similar to:
How to run NTP in slew mode and what does it mean?
Running ntpd command with -x will cause the ntp daemon to run in slew mode.
e.g ntpd -x
“-x”: Normally, the time is slewed if the offset is less than the step threshold, which is 128 ms by default, and stepped if above the threshold. This option sets the threshold to 600 s,which is well within the accuracy window to set the clock manually.
Note: Since the slew rate of typical Unix kernels is limited to 0.5 ms/s, each second of adjustment requires an amortization interval of 2000 s. Thus, an adjustment as much as 600 s will take almost 14 days to complete. This option can be used with the -g and -q options. See the tinker command for otheroptions. Note: The kernel time discipline is disabled with this option.
The slew mode is a valid mode, from support view it is as valid as using the default mode. It seems like most systems run in default NTP mode though at customers. To use slew mode or not should be dictated by the environment, its like “using swap or not, and how much”.
Factors to consider before using slew mode:
- If Slew mode is used, then applications do not have to deal with the “abrupt” leap second, since it is done over time
- If Slew mode is used and a leapsecond is occuring, then the system time will differ from the official time by ~1 second.
- This means that applications here interacting with other systems have either to cope with the 1 second difference, or the other systems also have to run in slew mode (and have the same slew speed)
- It is also important to understant that your hardware has to be able to keep relatively good time, that part is sometimes an issue.Every second to correct takes over a half hour (2000s) to actually correct in slew mode.
Known Issues of Leap Second Insertion:
Issue 1: System hangs on printing the leap second insertion message
#0 ktime_get_ts (ts=0xffffffff8158bb30) at include/asm/processor.h:691
#1 0xffffffff8104c09a in ktime_get () at kernel/hrtimer.c:59
#2 0xffffffff8102a39a in hrtick_start_fair (rq=0xffff810009013880,
p=<value optimized out>) at kernel/sched.c:1064
#3 0xffffffff8102decc in enqueue_task_fair (rq=0xffff810009013880,
p=0xffff81003fb02d40, wakeup=1) at kernel/sched_fair.c:863
#4 0xffffffff81029a08 in enqueue_task (rq=0xffffffff8158bb30,
p=0xffff81003b8ac418, wakeup=-994836480) at kernel/sched.c:1550
#5 0xffffffff81029a39 in activate_task (rq=0xffff810009013880,
p=0xffff81003b8ac418, wakeup=20045) at kernel/sched.c:1614
#6 0xffffffff8102be38 in try_to_wake_up (p=0xffff81003fb02d40,
state=<value optimized out>, sync=0) at kernel/sched.c:2173
#7 0xffffffff8102be9c in default_wake_function (curr=<value optimized out>,
mode=998949912, sync=20045, key=0x4c4b40000) at kernel/sched.c:4366
#8 0xffffffff810492ed in autoremove_wake_function (wait=0xffffffff8158bb30,
mode=998949912, sync=20045, key=0x4c4b40000) at kernel/wait.c:132
#9 0xffffffff810296a2 in __wake_up_common (q=0xffffffff813d3180, mode=1,
nr_exclusive=1, sync=0, key=0x0) at kernel/sched.c:4387
#10 0xffffffff8102b97b in __wake_up (q=0xffffffff813d3180, mode=1,
nr_exclusive=1, key=0x0) at kernel/sched.c:4406
#11 0xffffffff8103692f in wake_up_klogd () at kernel/printk.c:1005
#12 0xffffffff81036abb in release_console_sem () at kernel/printk.c:1051
#13 0xffffffff81036fd1 in vprintk (fmt=<value optimized out>,
args=<value optimized out>) at kernel/printk.c:789
#14 0xffffffff81037081 in printk (
fmt=0xffffffff8158bb30 “yj$\201????\2008\001\t”) at kernel/printk.c:613
#15 0xffffffff8104ec16 in ntp_leap_second (timer=<value optimized out>)
#16 0xffffffff8104b7a6 in run_hrtimer_pending (cpu_base=0xffff81000900f740)
#17 0xffffffff8104b86a in run_hrtimer_softirq (h=<value optimized out>)
#18 0xffffffff8103b31f in __do_softirq () at kernel/softirq.c:234
#19 0xffffffff8100d52c in call_softirq () at include/asm/current_64.h:10
#20 0xffffffff8100ed5e in do_softirq () at arch/x86/kernel/irq_64.c:262
#21 0xffffffff8103b280 in irq_exit () at kernel/softirq.c:310
#22 0xffffffff8101b0fe in smp_apic_timer_interrupt (regs=<value optimized out>)
#23 0xffffffff8100cf52 in apic_timer_interrupt ()
#24 0xffff81003b9d5a90 in ?? ()
#25 0x0000000000000000 in ?? ()
RHEL 4 : Upgrade to kernel-2.6.9-89.EL or later which contains the fix;
RHEL 5 : Upgrade to kernel-2.6.18-164.el5 or later which contains the fix;
RHEL5.3 : Long Life channel subscriptions, upgrade to kernel-2.6.18-128.37.1.el5 or later which contains the fix;
Issue 2: In RHEL 6.1 to 6.3 , Systems hang due to leap-second livelock.
The problem occurs when the system receives the announcement of the leapsecond, not when the leapsecond insertion actually happens. The problem is timing-dependent, so it is possible for a system to successfully queue the insertion.
However, the request for leapsecond insertion has to be cancelled and re-queued every time ntp adjusts the system time, so there is a chance of the problem happening continuously if ntpd is running.
For Diagnosis purpose, Obtain vmcores from NMI Watchdog crashing the box due to lockups and saw the call traces:
and then check for below two backtraces which will appear similar
- – The first backtrace appears to be in ktime_get.
- – The second backtrace appears to be in getnstimeofday.
if you are not sure on how to backtrace the core files, take help from redhat support.
Upgrade to Latest Version of Kernel
The Fix for RHEL 6.1 available in 6.1.z EUS kernel version 2.6.32-131.30.2.el6
The Fix for RHEL 6.2 available in 6.2.z EUS kernel version 2.6.32-220.25.1.el6
The Fix for RHEL 6.3 available in 6.3 kernel version 2.6.32-279.5.2.el6
The Fix for RHEL 6.2 available in in RHEL 6.4 GA kernel version 2.6.32-358.el6
aside from upgrading to the fixed kernels, the current known workarounds are:
- – Disable ntp & reboot so that the leap announcement is cleared from the kernel.
- – Set forward the time past local midnight and let ntp adjust it over time (slow correct)
- – Setting “tinker step” above 0.5 seconds within ntp.conf and restart ntp
- – Use the -x option within /etc/sysconfig/ntpd and restart ntp service.
This works only with ntp-4.2.6p5-3.el6_6 or later OR earlier versions than 4.2.6.
ISSUE 3: in RHEL 6.1/6.2/6.3 High CPU usage after inserting the leap second?
Sometimes, After the insertion of the leap second several processes, notably java, were reporting a high CPU usage.
Logged messages shows back trace from ktime_get.
You will see Following messages frequently in the logs:
Jul 1 02:23:12 server kernel: [1121001.934357] INFO: task java:60028 blocked for more than 120 seconds.
Jul 1 02:23:12 server kernel: [1121001.937203] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
Jul 1 02:23:12 server kernel: [1121001.940147] java D 0000000000000007 0 60028 1 0x00000080
Jul 1 02:23:12 server kernel: [1121001.943176] ffff880d775419c8 0000000000000082 0000000501bfe000 ffffc900180cb058
Jul 1 02:23:12 server kernel: [1121001.946293] ffffc900180cb020 ffffea002b821aa0 ffff880d34fe9580 000000d00002dc40
Jul 1 02:23:12 server kernel: [1121001.949488] ffff880d34fe9b38 ffff880d77541fd8 000000000000f4e8 ffff880d34fe9b38
Jul 1 02:23:12 server kernel: [1121001.952757] Call Trace:
Jul 1 02:23:12 server kernel: [1121001.956042] [<ffffffff8109b949>] ? ktime_get_ts+0xa9/0xe0
Jul 1 02:23:21 server kernel: [1121001.959347] [<ffffffffa04bac10>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Jul 1 02:23:21 server kernel: [1121001.962724] [<ffffffff814ed833>] io_schedule+0x73/0xc0
Jul 1 02:23:21 server kernel: [1121001.966138] [<ffffffffa04bac1e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs]
Jul 1 02:23:21 server kernel: [1121001.969643] [<ffffffff814ee1ef>] __wait_on_bit+0x5f/0x90
Jul 1 02:23:21 server kernel: [1121001.973190] [<ffffffffa04bac10>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Jul 1 02:23:21 server kernel: [1121001.976738] [<ffffffff814ee298>] out_of_line_wait_on_bit+0x78/0x90
Jul 1 02:23:21 server kernel: [1121001.980253] [<ffffffff81090d70>] ? wake_bit_function+0x0/0x50
Jul 1 02:23:21 server kernel: [1121001.983813] [<ffffffff81110b6e>] ? find_get_page+0x1e/0xa0
Jul 1 02:23:21 server kernel: [1121001.987403] [<ffffffffa04babff>] nfs_wait_on_request+0x2f/0x40 [nfs]
Jul 1 02:23:21 server kernel: [1121001.990991] [<ffffffffa04c145e>] nfs_updatepage+0x28e/0x590 [nfs]
Jul 1 02:23:21 server kernel: [1121001.994526] [<ffffffffa04af5c2>] nfs_write_end+0x152/0x2b0 [nfs]
Jul 1 02:23:21 server kernel: [1121001.998009] [<ffffffff811116c4>] generic_file_buffered_write+0x174/0x2a0
Jul 1 02:23:21 server kernel: [1121002.001488] [<ffffffff81112fb0>] __generic_file_aio_write+0x250/0x480
Jul 1 02:23:21 server kernel: [1121002.004944] [<ffffffff8111324f>] generic_file_aio_write+0x6f/0xe0
Jul 1 02:23:21 server kernel: [1121002.008370] [<ffffffffa04aeffe>] nfs_file_write+0xde/0x1f0 [nfs]
Jul 1 02:23:21 server kernel: [1121002.011760] [<ffffffff8117661a>] do_sync_write+0xfa/0x140
Jul 1 02:23:21 server kernel: [1121002.015276] [<ffffffff81090d30>] ? autoremove_wake_function+0x0/0x40
Jul 1 02:23:21 server kernel: [1121002.018657] [<ffffffff8120c646>] ? security_file_permission+0x16/0x20
Jul 1 02:23:21 server kernel: [1121002.022012] [<ffffffff81176918>] vfs_write+0xb8/0x1a0
Jul 1 02:23:21 server kernel: [1121002.025341] [<ffffffff810d4932>] ? audit_syscall_entry+0x272/0x2a0
Jul 1 02:23:21 server kernel: [1121002.028661] [<ffffffff81177321>] sys_write+0x51/0x90
Jul 1 02:23:21 server kernel: [1121002.031953] [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
As temporary workaround, Manually reset the date by executing any of the below commands, resetting the date will resolve any present issues.
date -s “`LC_ALL=C date`”
date `date +’%m%d%H%M%C%y.%S’`
To Prevent the issue from reoccurring:
For RHEL 6.1 EUS: Update to kernel-2.6.32-131.30.2 (from RHBA-2012-1197) or later.
For RHEL 6.2 EUS: Update to kernel-2.6.32-220.25.1.el6 (from RHBA-2012-1198) or later.
For RHEL 6.3: Update to kernel-2.6.32-279.5.2 (from RHBA-2012-1199) or later.
For RHEL 6.4GA and later: already contain the fix for this issue.