RAC node eviction

[Roshan]

SOlaris 11.4
RAC 12.1.0.2 database 2 nodes
Grid 19c

Dear Erman,

I noticed replication golden gate stopped yesterday. When I checked all processes were abended. No errors on DB and GG logs. I checked OS logs and found the following:

7d96f9e29.5ccdcff8, target address 011.001.001.001
Mar  3 15:24:02 dware1 savecore: [ID 570001 daemon.notice] reboot after panic: Oracle RAC node eviction
Mar  3 15:24:02 dware1 savecore: [ID 702911 daemon.warning] System dump time: Thu Mar  3 15:21:41 2022
Mar  3 15:24:02 dware1 savecore: [ID 702911 daemon.warning] Dump found on /dev/defdump
Mar  3 15:24:02 dware1 savecore: [ID 702911 daemon.error] System startup delayed until crashdump is saved to disk
Mar  3 15:24:02 dware1 savecore: [ID 702911 daemon.error] Saving compressed system crash dump files in directory /var/crash/data/2bf18f
d6-519c-418d-bac2-94e7ab916a59

d6-519c-418d-bac2-94e7ab916a59
Mar  3 15:24:09 dware1 iscsi: [ID 644432 kern.info] NOTICE: iscsi connection(23) unable to connect to target iqn.1986-03.com.sun:01:167
453be8fc7.5d024a93, target address 010.001.001.001
Mar  3 15:24:09 dware1 iscsi: [ID 644432 kern.info] NOTICE: iscsi connection(50) unable to connect to target iqn.1986-03.com.sun:01:f96
efbff2016.5d1f4145, target address 010.001.001.001
Mar  3 15:24:09 dware1 iscsi: [ID 644432 kern.info] NOTICE: iscsi connection(34) unable to connect to target iqn.1986-03.com.sun:01:87a
b0e26a851.5cce01cf, target address 011.001.001.001
Mar  3 15:24:09 dware1 iscsi: [ID 644432 kern.info] NOTICE: iscsi connection(61) unable to connect to target iqn.1986-03.com.sun:01:fd3
f959d3859.5d024afe, target address 011.001.001.001
Mar  3 15:24:09 dware1 iscsi: [ID 644432 kern.info] NOTICE: iscsi connection(59) unable to connect to target iqn.1986-03.com.sun:01:fd3
f959d3859.5d024afe, target address 010.001.001.001
Mar  3 15:24:09 dware1 iscsi: [ID 644432 kern.info] NOTICE: iscsi connection(25) unable to connect to target iqn.1986-03.com.sun:01:167
453be8fc7.5d024a93, target address 011.001.001.001


How can I open the dump file?

messages.messages


Regards,

Roshan
Roshan

[ErmanArslansOracleBlog]

It seems you have a RAC panic .

we have the following Mos DOC for this ->
RAC Panic Due To ASM I/O Failure (Doc ID 2729852.1)

But! it seems you have a HW problem as well..
SCSI Driver reports error for reconnecting to a Target..
Probably the target is is temporarily disrupted, check the IO subsystem as well..

For analyzing those types of  crash dump, you 'll use the mdb tool.

See -> https://docs.oracle.com/cd/E23824_01/html/821-1451/tscrashdumps-31.html