CRS-2409:The clock on host is not synchronous with the mean cluster time

SYMPTOMS

CTSSD runs in observer mode even though no time sync software is running.

or

Cluvfy comp clocksync fails with:

$ ./cluvfy comp clocksync
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
​
NTP Configuration file check started...
NTP Configuration file check passed
​
Checking daemon liveness...
Liveness check failed for "ntpd"
Check failed on nodes:
racbde1
PRVF-5415 : Check to see if NTP daemon is running failed
Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652 : Cluster Time Synchronization Services check failed
Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.
​

or

The clusterware alert log reports errors like:

2009-12-23 20:06:53.974 
[ctssd(13443)]CRS-2409:The clock on host racbde2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

　　Note you could also see the CRS-2409 error if you have a vendor time synchronization service running but the times are sync'd yet.

CAUSE

NTP or some other time sync service is configured but not running. The install, CVU, and CTSSD check for the following :

• There is only one and only one time synchronization service running at a time.

• The time sync service can be a vendor service (NTP, Windows Time Service, etc...) or CTSS.

• If vendor time service is configured, it needs to be correctly configured AND active.

• For CTSS to be in active mode, a vendor time service must not be running, and must not be configured (correctly or otherwise). This is important. CTSS is conservative and will switch to OBSERVER mode the moment it discovers that a vendor service is running or configured on even one node in the cluster. This is to prevent multiple active time sync services running on the cluster, potentially changing clocks.

SOLUTION

1. If you choose to use a vendor time sync service (like ntp), make sure it is configured AND running.

or

2. If you choose to let CTSSD handle time synchronization, de-configure the vendor time sync service. For example, for NTP you may need to move or remove /etc/ntp.conf or /etc/xntp.conf.

After choosing option 2 (and removing /etc/ntp.conf in my case), I now see a good time status reported by cluvfy:

$ ./cluvfy comp clocksync
​
Verifying Clock Synchronization across the cluster nodes
​
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
​
Checking if CTSS Resource is running on all nodes...
CTSS resource check passed
​
​
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
​
Check CTSS state started...
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Check of clock time offsets passed
​
​
Oracle Cluster Time Synchronization Services check passed
​
Verification of Clock Synchronization across the cluster nodes was successful.