ebs 12.2 -- fs_clone phase - adPostDomainCloneCfg.pl failed

[RG]

Hi,

All stages of phase= fs_clone were successful.  

adPostDomainCloneCfg.pl   failed   -- could not connect to Admin Server

Question : Do I need to run fs-clone  again or is there a shortcut to run only adPostDomainCloneCfg.pl  

Thank you

[ErmanArslansOracleBlog]

There are some documents about this problem in Oracle Support...

But we need to see the whole log..

At the same time, you can also try taking the action specified in document below;

12.2 E-Business Suite Fsclone Attempts Fail After An Upgrade With Error 'weblogic.security.internal.encryption.EncryptionServiceException' 'in setNodeManagerCredentials java.lang.AssertionError' Running adPostDomainCloneCfg.pl (Doc ID 2232990.1)

[RG]

Thank you Erman.

I reran fs_clone and it went successful but oacore server and forms server come up only in admin mode.

here are my logs.  --removing my server names from path


You are running admanagedsrvctl.sh version 120.14.12020000.11

Starting forms_server1...

Server specific logs are located at /u01/oracle/TST/fs1/FMW_Home/user_projects/domains/EBS_domain/servers/forms_server1/logs

admanagedsrvctl.sh: exiting with status 1

admanagedsrvctl.sh: check the logfile /u01/oracle/TST/fs1/inst/apps/TST/logs/appl/admin/log/adformsctl.txt for more information ...


------------------------------------------------


Forms Log:
forms_server1.log -- Error related to jdbc configuration (userid/passwd )
forms_server1.out -- Error related to jdbc configuration (userid/passwd )
forms_server1-diagnostic.log -- No Error


Error from forms_server1.log

####< <Error> <Deployer> <host name> <forms_server1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1533738339613> <BEA-149205> <Failed to initialize the application 'EBSDataSource' due to error weblogic.application.ModuleException: .
weblogic.application.ModuleException:
        at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:327)
        at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
        at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
        at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)
        at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:45)
        at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:648)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)


Caused By: weblogic.common.resourcepool.ResourceSystemException:
 Could not connect to 'oracle.jdbc.OracleDriver'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in
 the configuration or that the database is not available.
        at weblogic.jdbc.common.internal.JDBCUtil.parseException(JDBCUtil.java:301)
        at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:383)



-------------------------------------------------------


Error from forms_server1.out

nvoking startupclass EBSDataSourceStartup
Setting CLIENT_PROCESS_ID = 454
=======================
< <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Aug 8, 2018 2:25:15 PM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Aug 8, 2018 2:25:24 PM GMT> <Warning> <oracle.as.jmx.framework.generic.spi.BaseJMXFrameworkProviderImpl> <BEA-000000> <The configuration at URI "/u01/oracle/TST/fs1/FMW_Home/user_projects/domains/TST/config/fmwconfig/mbeans/../default-keystore.jks" cannot be loaded.>
<Aug 8, 2018 2:25:39 PM GMT> <Error> <Deployer> <BEA-149205> <Failed to initialize the application 'EBSDataSource' due to error weblogic.application.ModuleException: .
weblogic.application.ModuleException:
        at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:327)
        at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
        at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
        at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)
        Truncated. see log file for complete stacktrace
Caused By: weblogic.common.resourcepool.ResourceSystemException:
 Could not connect to 'oracle.jdbc.OracleDriver'.


---------------------------------------

No Error / Fail in AdminServer.log
No Error / Fail in AdminServer-diagnostic.log

Issues in

/u01/oracle/TST/fs1/FMW_Home/user_projects/domains/EBS_domain_/servers/AdminServer/logs
File : EBS_domain.log


####<Aug 8, 2018 5:40:25 PM GMT> <Error> <HTTP> 
<oacore_server1> 
<[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <>
<0000MKQc1N96yGpfGP2FTB1RQmdR000003> <1533750025430> <BEA-101165> 
<Could not load user defined filter in web.xml: oracle.apps.fnd.security.AppsFrmservletFilter.
oracle.apps.fnd.common.AppsException: java.lang.RuntimeException: javax.naming.NameNotFoundException:
While trying to lookup 'jdbc.TST' didn't find subcontext 'jdbc'. Resolved ''; remaining name 'jdbc/TST'
        at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1513)
        at oracle.apps.fnd.profiles.Profiles.getProfile(Profiles.java:366)
        at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfileFromDB(Unknown Source)

###<Aug 8, 2018 5:40:25 PM GMT> <Error> <HTTP> <oacore_server1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000MKQc1N96yGpfGP2FTB1RQmdR000003> <1533750025430> <BEA-101165> <Could not load user defined filter in web.xml: oracle.apps.fnd.security.AppsFrmservletFilter.
oracle.apps.fnd.common.AppsException: java.lang.RuntimeException: javax.naming.NameNotFoundException: While trying to lookup 'jdbc.TST' didn't find subcontext 'jdbc'. Resolved ''; remaining name 'jdbc/TST'
        at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1513)
        at oracle.apps.fnd.profiles.Profiles.getProfile(Profiles.java:366)
        at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfileFromDB(Unknown Source)
        at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfile(Unknown Source)

       at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.RuntimeException: javax.naming.NameNotFoundException: While trying to lookup 'jdbc.TST' didn't find subcontext 'jdbc'. Resolved ''; remaining name 'jdbc/TST'

[ErmanArslansOracleBlog]

Check whether you have this autoconfig parameter is set to FALSE -> s_jdbc_connect_descriptor_generation

if it is set to FALSE, set it to TRUE and run autoconfig, as explained in the following note->

Forms and OACORE Not Starting After Clone Generates weblogic.jdbc.common.internal.JDBCUtil.parseException (Doc ID 2294909.1)

1. Set the parameter  s_jdbc_connect_descriptor_generation to TRUE
2. Run autoconfig for the affected parameters to reference Target instance
3. Re-test issue

[RG]

Thank you so much for your quick help.

1. Doc ID 2294909.1
parameter  s_jdbc_connect_descriptor_generation  is set to  TRUE and I just did another autoconfig and also restarted completed app but OACORE server and FORMS server still coming in ADMIN mode.


2. I also cleared the tmp areas for forms and oacore with restarting and that did not work either.


I will provide more logs .    Thank you again.

[ErmanArslansOracleBlog]

Send me the logs of OACORE and FORMS managed weblogic servers.. (the start logs of them)

[RG]

Thank you for checking back.  Its weird. I  had to apply Apr Quarterly patches. After applying the ebs/weblogic patches and finishing the adop cycle after cutover the forms and oacore server are coming back up properly  in normal mode.   More APPS DBA on my servers so I am not sure if  anyone else changed anything.    

Whats the best way to track down what got changed and how it is working now.  Any way to track these..I will appreciate  help on this.  I will check the logs and upload new logs later this evening

Thank you