Execution of root.sh script failed for Oracle Grid Infrastructure 19c.

[Rabia]

Hi Erman,

   
I am currently in the process of installing Oracle RAC 19c on a Linux machine, and I have encountered an issue while executing the root.sh script on node 1. The installation has completed all prerequisite checks, but during the execution of the root.sh script, I am receiving an error that causes the process to fail. Below are the details:

 
[root@rac1 ~]# /u01/app/19c/grid/root.sh


Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19c/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/rac1/crsconfig/rootcrs_rac1_2024-11-21_03-20-25PM.log
2024/11/21 15:20:30 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2024/11/21 15:20:30 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2024/11/21 15:20:30 CLSRSC-363: User ignored prerequisites during installation
2024/11/21 15:20:30 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2024/11/21 15:20:31 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2024/11/21 15:20:32 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2024/11/21 15:20:33 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2024/11/21 15:20:33 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2024/11/21 15:20:33 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2024/11/21 15:20:34 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2024/11/21 15:20:40 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2024/11/21 15:20:41 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2024/11/21 15:21:14 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2024/11/21 15:21:14 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2024/11/21 15:21:17 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2024/11/21 15:21:22 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2024/11/21 15:21:28 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2024/11/21 15:21:33 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.

CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2674: Start of 'ora.cssdmonitor' on 'rac1' failed
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'rac1'
CRS-2678: 'ora.cssdmonitor' on 'rac1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-5802: Unable to start the agent process
CRS-4000: Command Start failed, or completed with errors.
2024/11/21 15:38:50 CLSRSC-119: Start of the exclusive mode cluster failed

Died at /u01/app/19c/grid/crs/install/crsinstall.pm line 2439.

[Rabia]

I have attached the output logfile.

rootcrs_rac1_2024-11-21_03-20-25PM.log

[ErmanArslansOracleBlog]

Seems like an OS side issue. A missing RPM, an unnecessary file or lack of a permission..

But we need more data to diagnose.

**We need the trace/log of the cssdmonitor...


Gathered info:

2024-11-21 15:38:50: CLSRSC-119: Start of the exclusive mode cluster failed
2024-11-21 15:38:50: ###### Begin DIE Stack Trace ######
2024-11-21 15:38:50:     Package         File                 Line Calling  
2024-11-21 15:38:50:     --------------- -------------------- ---- ----------
2024-11-21 15:38:50:  1: main            rootcrs.pl            355 crsutils::dietrap
2024-11-21 15:38:50:  2: crsinstall      crsinstall.pm        2439 main::__ANON__
2024-11-21 15:38:50:  3: crsinstall      crsinstall.pm        2334 crsinstall::perform_initial_config
2024-11-21 15:38:50:  4: crsinstall      crsinstall.pm        1026 crsinstall::perform_init_config
2024-11-21 15:38:50:  5: crsinstall      crsinstall.pm        1184 crsinstall::init_config
2024-11-21 15:38:50:  6: crsinstall      crsinstall.pm         446 crsinstall::CRSInstall
2024-11-21 15:38:50:  7: main            rootcrs.pl            552 crsinstall::new
2024-11-21 15:38:50: ####### End DIE Stack Trace #######


Gathered info:

>  CRS-2674: Start of 'ora.cssdmonitor' on 'rac1' failed
>  CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'rac1'
>  CRS-2678: 'ora.cssdmonitor' on 'rac1' has experienced an unrecoverable failure


*Similar error stack /but unrelated issue -> Installation Grid Fails With: ORA-29783: GPnP Attribute SET Failed With Error [CLSGPNP_RD_ERR] (Doc ID 2981876.1)

[Rabia]

Hi Erman,

The issue was not resolved . When I again run root.sh script on node 1 , it encountered the following error :


[root@rac1 oraInventory]# /u01/app/19c/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19c/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/rac1/crsconfig/rootcrs_rac1_2024-11-25_03-18-22PM.log
2024/11/25 15:18:33 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2024/11/25 15:18:33 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2024/11/25 15:18:34 CLSRSC-363: User ignored prerequisites during installation
2024/11/25 15:18:34 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2024/11/25 15:18:34 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2024/11/25 15:18:35 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2024/11/25 15:18:36 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2024/11/25 15:18:36 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2024/11/25 15:18:37 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2024/11/25 15:18:37 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2024/11/25 15:18:42 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2024/11/25 15:18:43 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2024/11/25 15:19:15 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2024/11/25 15:19:16 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2024/11/25 15:19:16 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2024/11/25 15:20:48 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2024/11/25 15:20:57 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2024/11/25 15:21:03 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2024/11/25 15:21:09 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2674: Start of 'ora.gipcd' on 'rac1' failed
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded


CRS-4000: Command Start failed, or completed with errors.
2024/11/25 15:22:39 CLSRSC-119: Start of the exclusive mode cluster failed
Died at /u01/app/19c/grid/crs/install/crsinstall.pm line 2439

I have attached output  logfile.

rootcrs_rac1_2024-11-25_03-18-22PM.log

[ErmanArslansOracleBlog]

Send me the gipcd.trc under <ORACLE_BASE>/diag/crs/<HOSTNAME>/crs/trace folder

reference and possible solution: CLSRSC-119: Start of the exclusive mode cluster failed While Running root.sh While Installing Grid Infrastructure 19c (Doc ID 2568395.1) -- but your cluster name is rac-cluster (11 chars long)

anyways lets see that trace..

[Rabia]

I have removed the previous failed installation. I will proceed with the installation again and will share the results with you once it's complete.