We are running EBS R12.2.3 against 11.2.0.4 RAC database.
The RAC database is running on 3 nodes: Node 1 ,2 and 3. For some reason, most probably due to our configuration, the Standard Managers and Inventory Managers always start running on Node 3 of the database tier. So when our database instance is not running on Node 3, the standard manager and inventory managers, won't start. We need to make sure , even if our database instance is not running on Node 3, our standard and inventory managers will start on the other nodes of the RAC database . Thanks, Ashish |
Administrator
|
Hi, Please take a look at the Paralel Concurrent Processing (PCP). It is what you look for. 7 Ara 2016 01:39 tarihinde "ashish_oracleguy [via Erman Arslan's Oracle Forum]" <[hidden email]> yazdı: We are running EBS R12.2.3 against 11.2.0.4 RAC database. |
Administrator
|
In reply to this post by ashish_oracleguy
But one important question : how many apps nodes do you have? 7 Ara 2016 08:53 tarihinde "Erman Arslan" <[hidden email]> yazdı:
|
Administrator
|
Also, send me the following info:
the value of cp_two_task which is written in your apps tier context file and the tnsnames.ora stored in your Application tier fileystem. |
Hi Erman,
Thanks for the reply. First , we have only one App node. Also the issue I am trying to address right now, is that: We are running 3 node RAC db for our EBS 12.2 If we do NOT bring up the instance on 3rd node, and only start the DB on node 1 and 2, and then start the EBS app tier, the standard mgr and inventory mgr do not start. So it seems the starting of standard mgr and inventory mgr, are tied up with the DB instance running on node 3. Here is the copy of the tnsnames.ora on the EBS app tier VCPP= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) ) ) VCPP1= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) (INSTANCE_NAME=VCPP1) ) ) VCPP1_FO= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) (INSTANCE_NAME=VCPP1) ) ) VCPP2= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) (INSTANCE_NAME=VCPP2) ) ) VCPP2_FO= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) (INSTANCE_NAME=VCPP2) ) ) VCPP3= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) (INSTANCE_NAME=VCPP3) ) ) VCPP3_FO= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) (INSTANCE_NAME=VCPP3) ) ) VCPP_FO= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VCPP) (INSTANCE_NAME=VCPP1) ) ) VCPP_BALANCE= (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=YES) (FAILOVER=YES) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) ) (CONNECT_DATA= (SERVICE_NAME=VCPP) ) ) FNDFS_PVCPAP01= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=PVCPAP01.xxx.xx)(PORT=1666)) (CONNECT_DATA= (SID=FNDFS) ) ) FNDFS_PVCPAP01.xxx.xx= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=PVCPAP01.xxx.xx)(PORT=1666)) (CONNECT_DATA= (SID=FNDFS) ) ) FNDFS_VCPP_PVCPAP01= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=PVCPAP01.xxx.xx)(PORT=1666)) (CONNECT_DATA= (SID=FNDFS) ) ) FNDFS_VCPP_PVCPAP01.xxx.xx= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=PVCPAP01.xxx.xx)(PORT=1666)) (CONNECT_DATA= (SID=FNDFS) ) ) FNDSM_PVCPAP01_VCPP= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=PVCPAP01.xxx.xx)(PORT=1666)) (CONNECT_DATA= (SID=FNDSM) ) ) FNDFS_APPLTOP_pvcpap01= (DESCRIPTION= (ADDRESS_LIST= (ADDRESS=(PROTOCOL=tcp)(HOST=PVCPAP01.xxx.xx)(PORT=1666)) ) (CONNECT_DATA= (SID=FNDFS) ) ) VCPP_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP1) ) ) VCPP1_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP1) ) ) VCPP1_FO_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP1) ) ) VCPP2_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP2) ) ) VCPP2_FO_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP2) ) ) VCPP3_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP3) ) ) VCPP3_FO_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP3) ) ) VCPP_FO_patch= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.218)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.219)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.xx.217)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=ebs_patch) (INSTANCE_NAME=VCPP1) ) ) |
Administrator
|
1)If this problem is encountered for specific managers, then check the following node. How To Run a Concurrent Program Against a Specific RAC Instance with PCP/RAC Setup? (Doc ID 1129203.1) Maybe those concurrent managers are configured to run specifically on node 3. Read the note above, and you will understand what to check / what do do.. 2)Also , you didn't send me the value of s_cp_two_task per requested in my previous comment... 2016-12-08 22:27 GMT+03:00 ashish_oracleguy [via Erman Arslan's Oracle Forum] <[hidden email]>: Hi Erman, Erman Arslan, MBA Applications and Database Operations Manager Oracle Certified Expert, Certified Exadata and Linux Administrator Author, Practical Oracle E-Business Suite Blog: ermanarslan.blogspot.com Forum: http://ermanarslan.blogspot.com/p/forum.html Mobile: 05301567803 |
Hello Erman,
Thank you for sharing the document number. I will review it. Here is the value of the s_cp_twotask variable in the CONTEXT_FILE it is VCPP_BALANCE Does the tnsnames.ora file look okay ? Regards, Ashish |
Hello Erman,
I reviewed the document you had mentioned 1129203.1. This documents, is about running a specific concurrent request on a specific node. My issue is that, my standard manager and inventory manager, are always running on a specific node. From what I understand and know, those are 2 different things : a specific concurrent request vs standard manager and inventory manager Thanks, Ashish |
Administrator
|
Hi,
Yes, that document is in concurrent program level. (my mistake :), still it is a good info :) Anyways, your cp_twotask seems okay and your apps tier tnsnames.ora looks okay. There must be some other problem. 1)The listeners in DB Node 1 and 2 listens for the SERVICE NAME VCPP right? 2)Send me the internal manager and standard manager for further analysis. (ensure , the log contains the log records during the time when the problem happened) 3)Send me the output of listener status command from all your db nodes. |
Hi Erman,
This is the message i see in the manager log file 3 Starting VCPP_1209@VCPP_BALANCE Internal Concurrent Manager -- shell process ID 6576 4 5 logfile=/u02/mclvcp/VCPP/fs_ne/inst/VCPP_pvcpap01/logs/appl/conc/log/VCPP_1209.mgr 6 PRINTER=noprint 7 mailto=applmgr 8 restart=N 9 diag=N 10 sleep=30 11 pmon=4 12 quesiz=1 13 Reviver is ENABLED 14 15 16 Adding Node:(PVCPAP01), Instance:(VCPP3) to unavailable list 17 No such process 18 An error occured in client-side routine afpsmckp for Service Manager FNDSM_PVCPAP01_VCPP. The routine returned code 1. 19 20 Check for preceding errors and as well as the service manager log file for further details." 21 22 09-DEC-2016 15:40:13 - Could not submit job to kill concurrent process 9825: Oracle error 100: ORA-01403: no data found has been d etected in FND_CONC_RAC_UTILS.SUBMIT_MANAGER_KILL_SESSION. 23 24 25 Found dead process: spid=(18846), cpid=(9825), ORA pid=(68), manager=(401/11) 26 27 28 29 Found dead process: spid=(18474), cpid=(9824), ORA pid=(53), manager=(0/1) 30 31 32 +---------------------------------------------------------------------------+ 33 Application Object Library: Concurrent Processing version 11.5 ------ |
Administrator
|
Hi, What is the value of the profile setting? "Concurrent PCP Instance Check" Set it to "off", and restart all your concurrent managers. Then retry. Note: With system profile 'Concurrent:PCP Instance Check' set to ON, managers try to move to a secondary Concurrent Tier when a RAC Instance fails.. But you don't have PCP right? So set it to off, restart conc managers and retry. 2016-12-10 7:47 GMT+03:00 Ashish [via Erman Arslan's Oracle Forum] <[hidden email]>: Hi Erman, Erman Arslan, MBA Applications and Database Operations Manager Oracle Certified Expert, Certified Exadata and Linux Administrator Author, Practical Oracle E-Business Suite Blog: ermanarslan.blogspot.com Forum: http://ermanarslan.blogspot.com/p/forum.html Mobile: 05301567803 |
Administrator
|
Any updates?
|
Hi Erman,
Thanks for the update, I will check the profile setting and provide an update. I just woke up. Regards, Ashish |
Hello Erman,
Latest update, yes the system profile "Concurrent PCP Instance Check" value is ON, right now. Also how do i check if my instance is PCP ready ? Thanks Ashish |
Administrator
|
This post was updated on .
do you have multiple apps nodes or not? PCP can be implemented only when you have multiple apps nodes; in other words; multiple concurrent tiers. So, the actual question is do you have multiple Concurrent tiers or not?
12 Ara 2016 19:07 tarihinde "ashish_oracleguy [via Erman Arslan's Oracle Forum]" <ml-node+s2340467n1725h59@n4.nabble.com> yazdı: > Hello Erman, > > Latest update, yes the system profile "Concurrent PCP Instance Check" > value is ON, right now. > > Also how do i check if my instance is PCP ready ? > > Thanks > > Ashish > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/ > Standard-Mgr-and-Inventory-always-running-on-one- > particular-node-of-RAC-db-tp1693p1725.html > To start a new topic under EBS 12.2, email ml-node+s2340467n6h70@n4. > nabble.com > To unsubscribe from Erman Arslan's Oracle Forum, click here > <http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=ZWFyc2xhbkBwYXJ0bmVyYS5jb20udHJ8MXwxODQ5OTg2NjU2> > . > NAML > <http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > |
do you have multiple apps nodes and do you have multiple Concurrent tiers or not?
No I don't have multiple Apps node nor Multiple Concurrent Tiers, just one App node and 3 node DB RAC. Thanks |
Administrator
|
then you dont have pcp.. set that profile to off and retest.. 12 Ara 2016 22:07 tarihinde "ashish_oracleguy [via Erman Arslan's Oracle Forum]" <[hidden email]> yazdı: do you have multiple apps nodes and do you have multiple Concurrent tiers or not? |
Hello Erman,
I don't think, the profile value of ON, for profile "Concurrent PCP Instance Check" is the issue in my case. In a different system, where I don't have this issue, the value of the profile "Concurrent PCP Instance Check" is ON, and it is also not configured for PCP. ------ I ran the cp_analyzer.sql script as apps, sqlplus apps@VCPP cp_analyzer.sql but in the output html, it showed that database as VCPP3, which is the instance name , on the which the standard mgr and inventory managers are dependent and did not say VCPP, which is the database name. So to me, it seems it is hardcoded somewhere to always look for VCPP3, not sure where to look to find that out. Thans |
Administrator
|
can you send me the output of cp analyzer? 14 Ara 2016 23:26 tarihinde "Ashish YU [via Erman Arslan's Oracle Forum]" <[hidden email]> yazdı: Hello Erman, |
Administrator
|
bytheway, The instance name in the cp analyzer output is not important. It gets the instance_name from v$instance. So it is based on the instance on which your sqlplus session which executes the cpanalyzer, resides connected.
Answer the following questions. 1)Are you sure that all the other manager except standard manager and inventory manager running? -- please test this by submitting some requests to them .. I m asking this, because there can be a more general and the active/actual values in concurrent manager administer screen may be misleading you. 2) I think the problem is happening while your managers are going down. I think it is a bug. The key is managers will have to go down first before they can come back up. So, we need to concantrate on the problem logged in your manager's logfile, which is -> Could not submit job to kill concurrent process 9825: Oracle error 100: ORA-01403: no data found has been detected in FND_CONC_RAC_UTILS.SUBMIT_MANAGER_KILL_SESSION. -> set "Concurrent PCP Instance Check" to OFF. ->Check this note for the patches: Concurrent Programs Fail With: Oracle Error 100: ORA-01403: No Data Found Has Been Detected In FND_CONC_RAC_UTILS. (Doc ID 1988419.1) -> Check this note for the patches as well : Concurrent Manager FNDSM intermittently Crashes / Shutting Down Abnormally When a Concurrent Request is Cancelled (Doc ID 1577982.1) ->Check this note for the patches as well: Concurrent Managers Are Not Going Down Properly After Applying Patch 16735285 (Doc ID 2063862.1) |
Free forum by Nabble | Edit this page |