After a successful execution Root scripts, the Grid Infrastructure 12.1.0.2 Installation failed at “Configure Oracle Grid Infrastructure for a Cluster” stage with following error –
[INS-20802] Oracle Cluster Verification Utility failed.
When looked into installation log file, Cluster Time Synchronization Service (CTSS) is started in Observer state.
Which means CTSS is expecting NTPD service to be up and running.
INFO: CTSS is in Observer state. Switching over to clock synchronization checks using NTP
INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)...
INFO: NTP configuration file "/etc/ntp.conf" existence check passed
INFO: PRVF-7590 : "ntpd" is not running on node "stdby02"
INFO: PRVF-7590 : "ntpd" is not running on node "stdby01"
INFO: PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
INFO: PRVF-5415 : Check to see if NTP daemon or service is running failed
INFO: Clock synchronization check using Network Time Protocol(NTP) failed
INFO: PRVF-9652 : Cluster Time Synchronization Services check failed
So, I enabled NTPD service on both the nodes using following commands.
service ntpd start
Starting ntpd: [ OK ]
chkconfig ntpd on
Clicked “Retry” on the Installation window.
This time it failed with following errors saying that the NTP Time Server is different on both the nodes.
PRVF-9652 : Cluster Time Synchronization Services check failed
It is required to have atleast one NTP Time Server to be common both nodes for a successful Clock Synchronization check.
root@stdby01 logs]# tail -100f installActions2017-06-22_09-33-49PM.log
INFO: CTSS is in Observer state. Switching over to clock synchronization checks using NTP
INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)...
INFO: NTP configuration file "/etc/ntp.conf" existence check passed
INFO: Liveness check passed for "ntpd"
INFO: Check for NTP daemon or service alive passed on all nodes
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "104.171.118.241" is common only to the following nodes "stdby02"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "132.239.1.6" is common only to the following nodes "stdby02"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "129.6.15.29" is common only to the following nodes "stdby01"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server ".GPS." is common only to the following nodes "stdby02"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "129.6.15.27" is common only to the following nodes "stdby01"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "139.78.97.128" is common only to the following nodes "stdby01"
INFO: PRVF-5416 : Query of NTP daemon failed on all nodes
INFO: Clock synchronization check using Network Time Protocol(NTP) failed
INFO: PRVF-9652 : Cluster Time Synchronization Services check failed
NTP Time servers that are being used can be seen issuing ntpq -pn command. The server with “*” is currently in use.
[root@stdby02 app]# /usr/sbin/ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
+173.230.144.109 132.239.1.6 2 u 44 64 177 123.116 6.012 7.763
+104.171.118.254 104.171.118.241 3 u 1 64 177 51.457 7.346 19.068
*185.140.51.3 .GPS. 1 u 55 64 177 70.576 10.809 26.393 <---
[root@stdby01 logs]# /usr/sbin/ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
+104.131.53.252 129.6.15.29 2 u 63 64 77 50.038 1.200 4.071
+66.85.74.226 192.5.41.40 2 u - 64 177 82.885 5.497 4.595
*108.61.73.244 129.6.15.27 2 u 33 64 177 39.497 1.523 9.439 <---
If we want to use CTSS anyway instead of NTPD for time synchronization, disable NTPD service and remove /etc/ntp.conf file on all the nodes.
[root@stdby01 bin]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@stdby01 bin]# chkconfig ntpd off
[root@stdby01 bin]# mv /etc/ntp.conf /etc/ntp.conf.back
Before clicking “Retry” on the installation window, just wanted to perform clocksync check using cluvfy utility.
The cluvy utility always looks for NTPD service to be up and running and CTSS demon to be started in Observer Mode. So, ignore the below error.
[root@stdby01 bin]# su grid
[grid@stdby01 bin]$ ./cluvfy comp clocksync -n all -verbose
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
stdby01 passed
stdby02 passed
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
stdby02 Observer
stdby01 Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
Node Name File exists?
------------------------------------ ------------------------
stdby02 no
stdby01 no
PRVG-1019 : The NTP configuration file "/etc/ntp.conf" does not exist on nodes "stdby02,stdby01"
PRVF-5414 : Check of NTP Config file failed on all nodes. Cannot proceed further for the NTP tests
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
stdby02 no
stdby01 no
PRVF-7590 : "ntpd" is not running on node "stdby02"
PRVF-7590 : "ntpd" is not running on node "stdby01"
PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652 : Cluster Time Synchronization Services check failed
Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.
Clicked “Retry” on the installation window.
This time installation completed successfully and noticed that CTSS is started in Active state.
[grid@stdby01 bin]$ tail -100f /grid/app/oraInventory/logs/installActions2017-06-22_09-33-49PM.log
INFO: Oracle Clusterware is installed on all nodes.
INFO: CTSS resource check passed
INFO: Query of CTSS for time offset passed
INFO: CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
INFO: Check of clock time offsets passed
INFO: Oracle Cluster Time Synchronization Services check passed
INFO: Checking VIP configuration.
INFO: Checking VIP Subnet configuration.
Thanks 🙂
#12-1-0-2, #12c, #12cr1, #20802, #clocksync, #cluster, #cluster-time-synchronization-services, #cluster-time-synchronization-services-check-failed, #cluvfy, #crsctl, #ctss, #failed, #grid, #ins, #ins-20802, #installation, #ntp, #ntp-time-server, #ntp-time-server-is-common-only-to-the-following, #ntpd, #oracle, #oracle-grid-infrastructure-12-1-0-2-installation-ins-20802-oracle-cluster-verification-utility-failed, #prvf-5408, #prvf-5416, #prvf-9652, #rac, #service, #utility, #verification
Qloud66 