After a successful execution Root scripts, the Grid Infrastructure 12.1.0.2 Installation failed at “Configure Oracle Grid Infrastructure for a Cluster” stage with following error –
[INS-20802] Oracle Cluster Verification Utility failed.
When looked into installation log file, Cluster Time Synchronization Service (CTSS) is started in Observer state.
Which means CTSS is expecting NTPD service to be up and running.
INFO: CTSS is in Observer state. Switching over to clock synchronization checks using NTP INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)... INFO: NTP configuration file "/etc/ntp.conf" existence check passed INFO: PRVF-7590 : "ntpd" is not running on node "stdby02" INFO: PRVF-7590 : "ntpd" is not running on node "stdby01" INFO: PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes. INFO: PRVF-5415 : Check to see if NTP daemon or service is running failed INFO: Clock synchronization check using Network Time Protocol(NTP) failed INFO: PRVF-9652 : Cluster Time Synchronization Services check failed
So, I enabled NTPD service on both the nodes using following commands.
service ntpd start Starting ntpd: [ OK ] chkconfig ntpd on
Clicked “Retry” on the Installation window.
This time it failed with following errors saying that the NTP Time Server is different on both the nodes.
PRVF-9652 : Cluster Time Synchronization Services check failed
It is required to have atleast one NTP Time Server to be common both nodes for a successful Clock Synchronization check.
root@stdby01 logs]# tail -100f installActions2017-06-22_09-33-49PM.log INFO: CTSS is in Observer state. Switching over to clock synchronization checks using NTP INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)... INFO: NTP configuration file "/etc/ntp.conf" existence check passed INFO: Liveness check passed for "ntpd" INFO: Check for NTP daemon or service alive passed on all nodes INFO: ERROR: INFO: PRVF-5408 : NTP Time Server "104.171.118.241" is common only to the following nodes "stdby02" INFO: ERROR: INFO: PRVF-5408 : NTP Time Server "132.239.1.6" is common only to the following nodes "stdby02" INFO: ERROR: INFO: PRVF-5408 : NTP Time Server "129.6.15.29" is common only to the following nodes "stdby01" INFO: ERROR: INFO: PRVF-5408 : NTP Time Server ".GPS." is common only to the following nodes "stdby02" INFO: ERROR: INFO: PRVF-5408 : NTP Time Server "129.6.15.27" is common only to the following nodes "stdby01" INFO: ERROR: INFO: PRVF-5408 : NTP Time Server "139.78.97.128" is common only to the following nodes "stdby01" INFO: PRVF-5416 : Query of NTP daemon failed on all nodes INFO: Clock synchronization check using Network Time Protocol(NTP) failed INFO: PRVF-9652 : Cluster Time Synchronization Services check failed
NTP Time servers that are being used can be seen issuing ntpq -pn command. The server with “*” is currently in use.
[root@stdby02 app]# /usr/sbin/ntpq -pn remote refid st t when poll reach delay offset jitter ============================================================================== +173.230.144.109 132.239.1.6 2 u 44 64 177 123.116 6.012 7.763 +104.171.118.254 104.171.118.241 3 u 1 64 177 51.457 7.346 19.068 *185.140.51.3 .GPS. 1 u 55 64 177 70.576 10.809 26.393 <--- [root@stdby01 logs]# /usr/sbin/ntpq -pn remote refid st t when poll reach delay offset jitter ============================================================================== +104.131.53.252 129.6.15.29 2 u 63 64 77 50.038 1.200 4.071 +66.85.74.226 192.5.41.40 2 u - 64 177 82.885 5.497 4.595 *108.61.73.244 129.6.15.27 2 u 33 64 177 39.497 1.523 9.439 <---
If we want to use CTSS anyway instead of NTPD for time synchronization, disable NTPD service and remove /etc/ntp.conf file on all the nodes.
[root@stdby01 bin]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@stdby01 bin]# chkconfig ntpd off
[root@stdby01 bin]# mv /etc/ntp.conf /etc/ntp.conf.back
Before clicking “Retry” on the installation window, just wanted to perform clocksync check using cluvfy utility.
The cluvy utility always looks for NTPD service to be up and running and CTSS demon to be started in Observer Mode. So, ignore the below error.
[root@stdby01 bin]# su grid [grid@stdby01 bin]$ ./cluvfy comp clocksync -n all -verbose Verifying Clock Synchronization across the cluster nodes Checking if Clusterware is installed on all nodes... Oracle Clusterware is installed on all nodes. Checking if CTSS Resource is running on all nodes... Check: CTSS Resource running on all nodes Node Name Status ------------------------------------ ------------------------ stdby01 passed stdby02 passed CTSS resource check passed Querying CTSS for time offset on all nodes... Query of CTSS for time offset passed Check CTSS state started... Check: CTSS state Node Name State ------------------------------------ ------------------------ stdby02 Observer stdby01 Observer CTSS is in Observer state. Switching over to clock synchronization checks using NTP Starting Clock synchronization checks using Network Time Protocol(NTP)... Checking existence of NTP configuration file "/etc/ntp.conf" across nodes Node Name File exists? ------------------------------------ ------------------------ stdby02 no stdby01 no PRVG-1019 : The NTP configuration file "/etc/ntp.conf" does not exist on nodes "stdby02,stdby01" PRVF-5414 : Check of NTP Config file failed on all nodes. Cannot proceed further for the NTP tests Checking daemon liveness... Check: Liveness for "ntpd" Node Name Running? ------------------------------------ ------------------------ stdby02 no stdby01 no PRVF-7590 : "ntpd" is not running on node "stdby02" PRVF-7590 : "ntpd" is not running on node "stdby01" PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes. PRVF-5415 : Check to see if NTP daemon or service is running failed Result: Clock synchronization check using Network Time Protocol(NTP) failed PRVF-9652 : Cluster Time Synchronization Services check failed Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.
Clicked “Retry” on the installation window.
This time installation completed successfully and noticed that CTSS is started in Active state.
[grid@stdby01 bin]$ tail -100f /grid/app/oraInventory/logs/installActions2017-06-22_09-33-49PM.log INFO: Oracle Clusterware is installed on all nodes. INFO: CTSS resource check passed INFO: Query of CTSS for time offset passed INFO: CTSS is in Active state. Proceeding with check of clock time offsets on all nodes... INFO: Check of clock time offsets passed INFO: Oracle Cluster Time Synchronization Services check passed INFO: Checking VIP configuration. INFO: Checking VIP Subnet configuration.
Thanks 🙂
Qloud66 