oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer

[Laurel]

Hi!
12.2.6 EBS.
running system is fs1
as soon as system started, all runing on fs1 , except oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer
starts using fs2. Concurrent Managers are using binaries from fs2.
If adop empty cycle ->cutover then all is OK -> as using fs2.
as again empty cycle -> again all fs1 except those using fs2.

Running adconfig not helping and checked that in Context file  of fs1 none of parameters  refernce to fs2.
None of the env parameters pointing to fs2.
Where  it takes that fs2 pointer the ?
:)

What is this ? Any ideas ?
Thank you,Laurel

[Laurel]

if I do  env |grep fs2 -> the patch_base is fs2.
So meaning conc starting from patch base..
Any hints would be appreciated :)
ps -ef |grep fs2
/testapp/oracle/TEST/fs2/EBSapps/comn/util/jdk64/bin/java -DCLIENT_PROCESSID=24445080 -Dhttp.proxyHost= -Dhttp.proxyPort= -Dhttp.nonProxyHosts= -Djava.awt.headless=true -Xmx384m -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -Ddbcfile=/testapp/oracle/TEST/fs2/inst/apps/TEST_ebstestapps/appl/fnd/12.0.0/secure/TEST.dbc -Dcpid=9758 -Dconc_queue_id=1140 -Dqueue_appl_id=0 -Dlogfile=/testout/oracle/TEST/conc/log/FNDCPGSC9758.txt -DLONG_RUNNING_JVM=true -DEBS_HOSTNAME=ebstestapps -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer

So where is in ebs the reference to use this /testapp/oracle/TEST/fs2/EBSapps/comn/util/jdk64/bin/java
and not the java from fs1
Its the same dir and exist.

/testapp/oracle/test/fs2/EBSapps/comn/util/jdk64/jre/bin]> ./java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pap6470sr9fp10-20150708_01(SR9 FP10))
IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References 20150701_255667 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR9_20150701_0050_B255667
JIT  - tr.r11_20150626_95120.01
GC   - R26_Java726_SR9_20150701_0050_B255667_CMPRSS
J9CL - 20150701_255667)
JCL - 20150628_01 based on Oracle jdk7u85-b15


/testapp/oracle/test/fs2/EBSapps/comn/util/jdk64/jre/bin]> cd /testapp/oracle/test/fs1/EBSapps/comn/util/jdk64/jre/bin
/testapp/oracle/test/fs1/EBSapps/comn/util/jdk64/jre/bin]> ./java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pap6470sr9fp10-20150708_01(SR9 FP10))
IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References 20150701_255667 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR9_20150701_0050_B255667
JIT  - tr.r11_20150626_95120.01
GC   - R26_Java726_SR9_20150701_0050_B255667_CMPRSS
J9CL - 20150701_255667)
JCL - 20150628_01 based on Oracle jdk7u85-b15


Thanks!

[Laurel]

And now I know why :
- autoconfig not populate fnd_env_context  table
so majority  of parameters there refer to fs2 .
So thats why have this problem.

tried to update contex_version  to higher. The same issue

So, I guess to fix our issue we need somehow to clean fnd_env_context  ?
How we can recreate autoconfig files and re-load fnd_env_context ?
Any hints ? :)
Thanks!

[ErmanArslansOracleBlog]

Hi Laurel,

INFO :FND_ENV_CONTEXT Table stores information about environment name and value for each of the concurrent process

Action plan:
1)Run the "Purge Concurrent Requests and/or Manager Data"  concurrent program with the necessary -parameters to clear that records from fnd_env_context.  
2)Stop the concurrent processing
3) Run the Concurrent Manager Recovery Troubleshooting Wizard with the internal manager down as detailed in the following document:
Concurrent Processing - Concurrent Manager Recovery Troubleshooting Wizard (Note 134007.1)
4) Start the concurrent processing.

Reference:
MAIN REF: Workflow Mailer Won't Start In Cloned Environment (Doc ID 1953629.1) -- this for 12.1 but, it seems it is applicable for 12.2 as well.
Concurrent Processing - Purge Concurrent Request and/or Manager Data Program (FNDCPPUR) (Doc ID 104282.1)

[ErmanArslansOracleBlog]

Be careful while running this program. Don't break any business rules (i.e if there is a need to save the concurrent request log and out files for 30 days, then you will need to get te approval of business side for this...)

You can backup the following tables before running the conc program:

FND_CONCURRENT_REQUESTS
                           This table contains a complete history of all
                           concurrent requests. This table should ideally be kept at around
                           4 - 5k records and should be regularly purged as part of normal
                           'housekeeping' by running the standard FNDCPPUR program.
                           This is a part of basic system administration and it is recommended
                           that for the average instance, it should be run every 30 days or so.
                           Allowing the table ( and others ) to increase in size can affect performance.
         
FND_RUN_REQUESTS
                           When a user submits a report set, this table stores
                           information about the reports in the report set and
                           the parameter values for each report.

FND_CONC_REQUEST_ARGUMENTS
                           This table records arguments passed by the concurrent
                           manager to each program it starts running.

FND_DUAL                  
                           This table records when requests do not update
                           database tables.

FND_CONCURRENT_PROCESSES  
                           This table records information about Oracle
                           Applications and operating system processes.

FND_CONC_STAT_LIST
                           This table collects runtime performance statistics
                           for concurrent requests.

FND_CONC_STAT_SUMMARY      
                           This table contains the concurrent program
                           performance statistics generated by the Purge.

FND_ENV_CONTEXT            
                            Table stores information about environment name and value for each of the concurrent process

[ErmanArslansOracleBlog]

Any updates?

[Laurel]

Hi Erman!
Last time I managed to fix-  so conc mgrs started on correct file system -  by updating the s_contextserial to higher than in the database and re-run autoconfig.
 select serial_number from FND_OAM_CONTEXT_FILES where status in ('S','F') and name not in
('METADATA','TEMPLATE') and ctx_type='A';

This time I am again having this issue after cutover.
The adop cycle+cutover went without errors, but after cutover -> again started on the patch filesystem the conc.mgrs.
This time updating the context_serial to higher than in db, is not helping.
autoconfig completes without any errors..
But the serial_number 65 when in context file I put 70.
Its not changing after autoconfig.


Any hints ? I think of cleanuping  the fnd_oam_context_files ?
Br,Laurel

[ErmanArslansOracleBlog]

It seems, although there is no errors report in cutover, the cutover actually can't complete properly.
Probably, there is a misdirection in the env files that your concurrent managers are getting their inputs.

Is your apps tier context file steady? I mean , are the fs-related values stored in context file correct?
What do you see in cutover logs?

[Laurel]

Hi Erman!
Thx for replay.
context file fs1 has only references fs1 and fs2-> fs2.
The only what has starting on patch filesystem is concurrents.Basically they using patch system binaries.
If I  source run file system env and do
env |grep fs2
Its only PATCH_BASE has the reference to fs2.
I think i should force the context file to  upload to the database , as the conc mgrs take values from there..
So, I updated the contextserial to higher number 70, in db its 64. But its not updating serial_number in the FND_OAM_CONTEXT_FILES .
I changed the profile s_sitename in Context file-> run autoconfig -> the value in fnd_profile_option_values got updated but not the serial_number in the file ...

So, what I am  missing here ?
Cutover -> no erros. All looked normal.
What do you think about cleaning up FND_OAM_CONTEXT_FILES..  This then should load from context file values ? Thank you!
Br,Laurel

[ErmanArslansOracleBlog]

I think there is some mis environment variable left in FND tables.
FND_ENV_CONTEXT may be the cause.

After taking your backup, you can try truncating FND_OAM_CONTEXT_FILES and running autoconfig (first in db then in apps tier)afterwards.. But, what I recommend you is following;

INFO :FND_ENV_CONTEXT Table stores information about environment name and value for each of the concurrent process

Action plan:
1)Run the "Purge Concurrent Requests and/or Manager Data"  concurrent program with the necessary -parameters to clear that records from fnd_env_context.  
2)Stop the concurrent processing
3) Run the Concurrent Manager Recovery Troubleshooting Wizard with the internal manager down as detailed in the following document:
Concurrent Processing - Concurrent Manager Recovery Troubleshooting Wizard (Note 134007.1)
4) Start the concurrent processing.

Reference:
MAIN REF: Workflow Mailer Won't Start In Cloned Environment (Doc ID 1953629.1) -- this for 12.1 but, it seems it is applicable for 12.2 as well.
Concurrent Processing - Purge Concurrent Request and/or Manager Data Program (FNDCPPUR) (Doc ID 104282.1)

[Laurel]

Hi Erman!
Yes I agree the Conc mgrs are picking up wrong fs2  from database table..
I did  the action plan  (purge+recovery Conc + even rerun cmclean) -> still wrong filesystem.
Next I will do backup &  the cleanup of the tables:
FND_ENV_CONTEXT, FND_OAM_CONTEXT_FILES and running autoconfig.

The system is working fine. JUST Conc are using the binaries from patching system.. :)
Will let you know how it turns out.

[ErmanArslansOracleBlog]

Okay.. lets see..

[Laurel]

Hi Erman!
Update and FYI:
we fixed the issue by :
shudtown apps tier
truncating the  applsys.fnd_env_context
run autoconfig on db+appstier
restart

Actually its comes up now and then same issue after ALOT of  patching, not everytime..
When alot of online patching cycles done.
So, seems like some bug in EBS that its GSM confused and start on wrong filesystem.
 Take care!
Laurel

[ErmanArslansOracleBlog]

Thanks for the update Laurel..