Re: node eviction

Posted by ErmanArslansOracleBlog on Dec 02, 2020; 5:21pm
URL: http://erman-arslan-s-oracle-forum.124.s1.nabble.com/node-eviction-tp9069p9077.html

You mean private interconnect check, right?

Did you see thid document? -> Oracle Grid Infrastructure: How to Troubleshoot Missed Network Heartbeat Evictions (Doc ID 1534949.1)

But it seems we checked the wrong alert messages ...
You have another eviction in the year of 2019 and we actually take it into consideration..

Your current eviction is caused by "abnormal instance termination"

Waiting for instances to leave: 2
2020-11-29T06:25:59.926754+04:00
Dumping diagnostic data in directory=[cdmp_20201129062559], requested by (instance=2, osid=12884907632 (LMS0)), summary=[abnormal instance termination].

Well, in node 2 you have lots of error during tat time.. (Nov 29 around 6 am)

ORA-04030: out of process memory when trying to allocate 1052696 bytes (klcliti:kghds,kllcqgf:kllsltba)
ORA-07445: exception encountered: core dump [lbivffs()+28]
ORA-04030: out of process memory when trying to allocate 33584 bytes (qmxtcxEvalXMLE,kghsseg:qmxtixBuildXQ)
ORA-48132: requested file lock is busy, [INCIDENT] [/u01/app/ora12c/diag/rdbms/dware1/dware12/lck/AM_1762783_4031814035.lck]
ORA-48170: unable to lock file - already in use
SVR4 Error: 11: Resource temporarily unavailable

These Ora-04030s and ORA-48170s are already looking related with OS..

Seems like your actual problem is ORA-04030 and the other are the results.. (probably!)

This is IBM right?

On alert log 2, it all starts with "ORA-04030: out of process memory when trying to allocate 311832 bytes (kxs-heap-w,control file i/o buffer)"

Check -> "How to resolve ORA-04030: out of process memory when trying to allocate 248 bytes (kxs-heap-w,ctx:keswxCurPrepare) Errors on IBM AIX Platforms (Doc ID 1934141.1)"

Note that, this is a quick response..  I hope it will help you and make your problem dissapear.
But, also check the situation with all its details, work hard on your alert logs and the errors that you see at the time of eviction.. Check the OS logs as well.. Your cause is there..

So if it is IBM , then we may be hitting