EBS r12.1.3 Failing concurrent managers

[latifa]

Hi Erman,

we have 3 servers CM (concurrent managers). The operator found in the workflow dashbord that 2 CMS  among 3 have problems.  He tried to stop CMs but they restarted automatically. He stopped the listener and he could stop all servers even web, forms, etc ... (3 other servers). This was done around 8:00 AM on 11-FEB-2019.

So to give the root cause, some errors were detected in the log files of the ICM (Internal Concurrent Manager) as shown in the attached file.

I tried to find out which message was most significant and I think it's this one:

Unsuccessful in ping of node RAC_DB_SERVER_2, error reported: Node Timed Out.
Internal Concurrent Manager found node  RAC_DB_SERVER_1 to be down. Adding it to the list of unavailable nodes.

And I looked after this error message and I found this Oracle Doc ID: 2189822.1

My questions:
- What is your opinion about this choice ?
- Could it be something else ? (alert_log doesn't show any error of database down even listeners logs RAC, SCAN_1, SCAN_2 and SCAN_3 were OK)

I attached ICM logs and listener logfile.

Thank you in advance for your help.

Regards,
Latifa

Incidents.zip 

[ErmanArslansOracleBlog]

In that note you sent, Oracle Support says : "That simply means ICM sent a ping to the remote node and response was not received."

Oracle Support also says : " I believe we do both ping and tnsping."

So this ICM ping process uses the tnsping utility as well..
If it uses tnsping, then it wont be successful when your listener is down..

Do you get the idea?

so in your case these errors seem normal, as you stopped the listener..

[latifa]

Hi Erman,
Thank you for your reply.

I understand that you said. However, the issue happened before stopping listener.
Stopping the listener has been done later to restart all CMs.

1. I submitted the Doc ID 2189822.1 to oracle suuport and they don't see that blocking saying:
"this message will be reported when the concurrent node is down,
so this is just stating that the node went down." ?

2. Oracle support pointed out the following error giving the Doc ID describing it: Concurrent Processing - Internal Concurrent Manager ICM log file shows 'CONC-SM TNS FAIL', 'Call to PingProcess failed', and/or 'Call to StopProcess failed' for FNDCPGSC/FNDOPP ( Doc ID 1312632.1 ).

3. They asked also to use first : FNDCPPUR - Purge Concurrent Request and/or Manager Data on a regular basis: because  The FND_CONCURRENT_REQUESTS Table has 163,272 rows of completed runtime data which exceeds the acceptable thresholds that were entered to run this Analyzer. Please find CP Anlyzer.

CP_Analyzer_$NODE_2019-02-28_05_04.html

In this case what the most convenient parameters to submit this request in a regular basis ? Is there any impact of this request ?

Please let me know your opinion.

Regards,
Latifa

[ErmanArslansOracleBlog]

So, Oracle Support is engaged then.
Your question is now "In this case what the most convenient parameters to submit this request in a regular basis ? Is there any impact of this request ? "

My answer is;

MOS tech note “Purging Strategy for E-Business Suite” (Doc ID 732713.1) is helpful for purging EBS data using seeded concurrent programs.

So,

Schedule standard concurrent purge requests with the correct age parameter as per the SLA and business requirements.
To schedule and the purging concurrent request, please refer to these MOS tech notes:

Concurrent Processing - How to Optimize the Process of Running Purge Concurrent Request and/or Manager Data (FNDCPPUR) (Doc ID 92333.1)
Concurrent Processing - Purge Concurrent Request and/or Manager Data Program (FNDCPPUR) (Doc ID 104282.1)