Pending OPP issue with custom program

[VinodN]

Greetings to all! Our environment is EBS 12.2.10, DB 19.9.0.0.0 on RHEL 8. We are facing this issue with a custom program where the concurrent request is taking more than 48 hours to complete. Almost all of this time it has no database session and is in "Pending OPP" status/phase.. Any suggestions on how to troubleshoot this would be greatly appreciated.

[ErmanArslansOracleBlog]

Is your OPP (output post processor) service/concurrent manager working ? With enough workers? Any errors in the logs of OPP?

[VinodN]

Thanks for the reply, dear Erman. It's an honour to talk to you. No, there are no errors in the OPP logs and also, the OPPs are processing other requests normally. The actual and target processes are same in number.

[ErmanArslansOracleBlog]

Hi,

1)Send me the OPP logfile

2)Send me a screenshot of those conc request. I need to see the phase and status of them.

3)Send me the output of the following query:
 SELECT service_id, service_handle, developer_parameters
                       FROM fnd_cp_services
                  WHERE service_id = (SELECT manager_type
                                                          FROM fnd_concurrent_queues
                                                   WHERE concurrent_queue_name = 'FNDCPOPP');

4)What is the Actual and Target count of OPP processes? - at any given time and at the problematic time.

5)Are there any conflicting requests in the confict resolution manager's queue during the problematic time interval?

Any blocking request that you can find using the below query;
select * from fnd_concurrent_requests where concurrent_program_id in
(SELECT p.concurrent_program_id FROM FND_CONCURRENT_PROGRAM_SERIAL s, fnd_concurrent_programs_tl p WHERE
(s.to_run_application_id in (select application_id from fnd_concurrent_programs_tl where user_concurrent_program_name='<Your Program Name>'))
and
(s.to_run_concurrent_program_id in (select concurrent_program_id from fnd_concurrent_programs_tl where user_concurrent_program_name='<Your Program Name>'))
and s.running_concurrent_program_id=p.concurrent_program_id)
and phase_code <> 'C';

6).Using the conc request form.. View the details of the pending request and Click Diagnostics button. What do you see as the diag message?

[VinodNN]

opplogs.gzOPP_Issue.docx

Hi Erman,

Kindly review the uploaded files.
There are always specified number of actual and target processes 5 - 5.
There are no blocking requests - the query did not return any rows.
There are no requests against the CRM.

This is a new login id (VinodNN) I had to create for myself for this forum as my old login (VinodN)was not working (password was not working and I was not getting the password reset link too..)

Best Regards,
Vinod

[ErmanArslansOracleBlog]

Hi,

Please restart OPP and retest.. (nothing weird in the logs.. OPP doesn't get those pending requests..
Nothing about those requests in the log and no errors)

How To Restart Output Post Processor (OPP) Via Linux Command? (Doc ID 2440363.1)

You can also try increasing the number of OPP processes and retest..

Bytheway; for querying those requests, use SYSADMIN .. you may not have permission to see those requests..

If the issue persist after all your solution attempts, run CP analyzer and check its output.. E-Business Suite Concurrent Processing Analyzer ( CP Analyzer ) (Doc ID 1411723.1)

[VinodNN]

I already tried restarting the OPPs and also increased their number from 2 to 5. Did not work. CP Analyzer too did not report any issue which could be relevant.

I queried the request as SYSADMIN and I am able to view it now. But I had logged in with an account which had System Administrator privileges.. still I was not able to view it.. Do I need any other responsibility?

Here is the output I got after clicking on Diagnostics:

This request is currently processing normally. Processing began on 11-OCT-2023 11:51:45.

No action required. You may examine the request log to determine its processing status.

[ErmanArslansOracleBlog]

1)Check the configuration of OPP processes using Concurrent Managers Define Form. Any configuration that may prevent them getting those request from their queue?
2)Find and check database sessions of "OPP processes". See what they are doing in the database? Are they bust and ACTIVE? What are their current waits?

Some references for you to check ->

R12: Troubleshooting Known XML Publisher and E-Business Suite (EBS) Integration Issues (Doc ID 1410160.1)
Steps To Analyze XML Publisher Requests With The Message: Unable To Find An Output Post Processor Service To Post-Process Request (Doc ID 1065722.1)

3)Is this issue specific to only some types of concurrent requests? or is it a general OPP issue ? If it is specific, what do these concurrent programs which are waiting for OPP have in common?

4)When did this issue started? After what big change(s) or what kinds of patches?

[VinodNN]

Hi Erman,

There is no special configuration which could prevent OPP from picking up those requests.
This issue is observed for only one particular report and it has been so from the very begging of the introduction of that report. It has always taken so much time.

[ErmanArslansOracleBlog]

Interesting.. What happens, if you recreate a new custom program (using EBS screens) identical to this problematic one and retest this issue by submitting a new concurrent request using that concurrent program?

[VinodNN]

I tried it. Created a new concurrent program by copying the problematic program. Also created a custom manager and assigned this program to it. Still facing the same issue. The CR is in Pending OPP from many hours.

[ErmanArslansOracleBlog]

I wonder if there might be a specialization rule for OPP that prevents these request from being processed? Have you checked this?

[VinodNN]

Yes, but there is no specialization rule for OPP.

[ErmanArslansOracleBlog]

1)What does that custom concurrent program do? Did you review the code and especially the OPP-related part of it? Anything seemed weird, anything that may prevent it being processed by OPP?

2)Review the following MOS note (especially step 2,3 and 4) with your Functional admin or developer, and consider taking actions accordingly..

3)Anything in the Internal Manager log file?

4)This may be related with the relevant QUEUES.. Rebuilding the queues may help.. Check the following -> OPP Manager Takes Long Time To Start (Doc ID 2224189.1) -- Don't do anything critical on PROD before taking a backup and before testing the actions on TEST.

Some additional references : R12 E-Business Suite Output Post Processor (OPP) Fails To Pick Up Concurrent Requests With Error 'Unable to find an Output Post Processor service to post-process request nnnnn' (Doc ID 2215371.1)
EBS: Output Post Processor (OPP) Takes Long Time To Pick Up Requests After Concurrent Managers Start (Doc ID 2969738.1)

review the above ones also.

5)also go through -> Tuning Output Post Processor (OPP) to Improve Performance (Doc ID 1399454.1)

[ErmanArslansOracleBlog]

Rebuilding queues may also be considered , but be careful.. The risk is your.. Always test these types of things on TEST, take your backup for PROD and then take the actions on PROD -> How To Purge FND_AQ Tables (Doc ID 1156523.1).

*Also getting Oracle's approval is a good at this point..