FNDSM down

[satish]

Dear erman,

We have faced the issue where reports are stuck in running status.
When we verify the logs, we could see service manager, opp and workflow managers are down.
Standard managers are up and running and the requests are processing fine.

We have raised sr but no solution yet.

Process monitor session started : 25-JUN-2024 07:45:14
 
                     Process monitor session ended : 25-JUN-2024 07:45:14
 
                   Process monitor session started : 25-JUN-2024 07:47:14
 
The Internal Concurrent Manager has encountered an error.

Review concurrent manager log file for more detailed information. : 25-JUN-2024 07:47:14 -

Shutting down Internal Concurrent Manager : 25-JUN-2024 07:47:14

List of errors encountered:

.............................................................................

_ 1 _

Routine AFPCMT encountered an ORACLE error. ORA-03106: fatal two-task

communication protocol error

.

Review your error messages for the cause of the error. (=<POINTER>)

_ 2 _

Routine AFPGMG encountered an ORACLE error. ORA-03106: fatal two-task

communication protocol error

Review your error messages for the cause of the error. (=<POINTER>)

.............................................................................

APP-FND-01564: ORACLE error 1007 in chk_init_mgr_instances

Cause: chk_init_mgr_instances failed due to ORA-01007: variable not in select list.

The SQL statement being executed at the time of the error was: SELECT DISTINCT NODE_NAME, DB_INSTANCE   FROM FND_CONCURRENT_PROCESSES  WHERE PROCESS_STATUS_CODE = 'Z'    AND MANAGER_TYPE < 6    AND LAST_UPDATE_DATE > SYSDATE - (4 * :pmon_secs)/86400   AND NODE_NAME IS NOT NULL   AND DB_INSTANCE IS NOT NULL and was executed from the file &ERRFILE.

List of errors encountered:

.............................................................................

_ 1 _

Routine AFPCAL received failure code while parsing or running your

concurrent program CPMGR

Review your concurrent request log file for more detailed information.

Make sure you are passing arguments in the correct format.

.............................................................................

The PRODDB_0621@PRODDB_BALANCE internal concurrent manager has terminated with status 1 - giving up.


Pls need your expertise.

We fixed the issue by shutting down the managers and ran cpadmin.sh,then started the managers.

Thank you,
SG

[ErmanArslansOracleBlog]

Interesting.

This is a standard routine and the query gets 01007  , which is due to "a reference was made to a variable not listed in the SELECT clause of the statement."

Failing query: SELECT DISTINCT NODE_NAME, DB_INSTANCE   FROM FND_CONCURRENT_PROCESSES  WHERE PROCESS_STATUS_CODE = 'Z'    AND MANAGER_TYPE < 6    AND LAST_UPDATE_DATE > SYSDATE - (4 * :pmon_secs)/86400   AND NODE_NAME IS NOT NULL   AND DB_INSTANCE IS NOT NULL

The error is probably due to :pmon_secs

These things ((4 * :pmon_secs)/86400) are hardcoded in the standard code. So, SR should resolve this.

1)Does that happen only once? Or does it happen time to time?
2)ORA-03106 seems like a result, but I m not quite sure.. What about patch levels? Both APPS side and DB Side?