Perl adcfgclone appsTier dualfs errors out at Instantiating the OHS Config state

[mathewjn]

Hello and Good Afternoon Erman:

While I have followed your site for at least 5 years now, I have not needed to register and ask for help until now - since my SR seems to be progressing nowhere.

I am running a clone from source to target - and everything happens as it should (config) until it reaches the OHS config stage.

After looking through some previous posts on here, I find one thread - that is similar in the error being received but not in the previous steps.

Appears that the Admin server is all configured - but then - OHS fails.

When you have some time, could you please help? I will also email you the clone logs - if necessary.

Thanks so much


Script Executed in 78058 milliseconds, returning status 0
AdminServer is in RUNNING mode.
Verified that AdminServer is Up and Running
Inside invokeOHSClone
Checking whether instance EBS_web_VCPFD1_OHS1 is registered with domain...
RegisteredInstanceNames MBean attribute value is NULL
Instance EBS_web_VCPFD1_OHS1 is not registered with Domain
/u02/mclvcp/VCPFD1/fs1/FMW_Home/webtier/instances/EBS_web_VCPFD1_OHS1 directory deleted successfully.

START: Instantiating the OHS Config.
Running /u02/mclvcp/VCPFD1/fs1/FMW_Home/webtier/perl/bin/perl  -I/u02/mclvcp/VCPFD1/fs1/FMW_Home/webtier/perl/lib/5.10.0 -I/u02/mclvcp/VCPFD1/fs1/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I/u02/mclvcp/VCPFD1/fs1/EBSapps/appl/au/12.0.0/perl -I/u02/mclvcp/VCPFD1/fs1/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u02/mclvcp/VCPFD1/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl cfgCloneInstance -ctxfile=/u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/appl/admin/VCPFD1_lddscvcpfsap01.xml -archvloc=/u02/mclvcp/VCPFD1/fs1/EBSapps/comn/clone/FMW/OHS/ohsarchive.jar -mvpln=/u02/mclvcp/VCPFD1/fs1/EBSapps/comn/clone/FMW/OHS/moveplan.xml -t2plogloc=/u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/admin/log/clone/run/ohsT2PApply -logdir=/u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/admin/log/clone/run -console=off -promptmsg=hide
Script Executed in 2503 milliseconds, returning status 1
ERROR: Script failed, exit code 1

[ErmanArslansOracleBlog]

Hi Mathew,

Your problem is in the execution of txkSetOHSConfig.pl.
It tries to the cfgCloneInstance and it can not do it in a given time.
This one is similar to yours -> Clone EBS 12.2.3, At Config Target Application Tier (run Edition File System) Step, Get Error (Doc ID 1683658.1)

But ofcourse, we have to check.
We might get some clues from the FMW components logs.

What do you have in EBS_web_<SID>.log and  CLONEXXX-xxxxxxxxxxx.error logs?

[mathewjn]

Good Morning Erman and thank you for your response.

I will upload the entire zip files of logs to my google drive and then post the link here.

Thanks again for agreeing to take a look.

Joseph

[mathewjn]

Logs have been uploaded to this location

https://drive.google.com/open?id=19TZm7DiPeWf4E0mvWkbMjdsMT-oLcCKf

Thanks so much again.

[ErmanArslansOracleBlog]

I see the following null pointer error in the logs ->


StackTrace:
java.lang.NullPointerException
"oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSDirectives(UpdateOhsT2PMovePlan.java:832)"

Check comn/clone/FMW/OHS/moveplan.xml.. It may de due to some customizations there.
Search the string  /YMS-UI-context-root.  (cause you are failing at this point)

Also check the following->
Do you have any customization in the console url?
Any customization related with the FMW cores components?

So if you see anything to be modified in moveplan.xml (anything custom thing that you may remove); please take a backup of the file , do your corrections and retry the failing clone activity.

You may also remove those customizations in the console url or similar FMW core components. (if there are any..)

NOTE: Take actions after ensuring about the business impact! Don't modify or remove anything that may cause your production to fail.

This guy wrote a blog post about it -> http://tk-appsdba.blogspot.com/2017/06/
The issue in that post is similar to yours.

[mathewjn]

I will take a look at the moveplan.xml.

I had noticed the post you have referred to, and on the suggestion of the analyst, removed that section from the moveplan.

The result was that the scripts (adcfgclone) then ended in in an error 255 - looking at the OHS error logs, there was notation that the moveplan was invalid and was inconsistent with the OHSarchive.jar file.

I will however, look at this again today.

Thanks for your help.

Joseph

[ErmanArslansOracleBlog]

Okay.. Try to remove that added sections "correctly"..

[mathewjn]

I don't see any customizations on the moveplan.xml

Here is the relevant portion of the moveplan.xml - at the point where the clone fails.
**********************************************************************
                        <name>Location</name>
                        <value>/YMS-UI-context-root</value>
                        <itemMetadata>
                            <dataType>STRING</dataType>
                            <scope>READ_ONLY</scope>
                        </itemMetadata>
                        <configProperty>
                            <name>WebLogicCluster</name>
                            <value>lpdstvcpfsap01.companyname.com:6801</value>
                            <itemMetadata>
                                <dataType>STRING</dataType>
                                <scope>READ_WRITE</scope>
                            </itemMetadata>
                        </configProperty>
**********************************************************************

[mathewjn]

After the RDA was run and all values were PASSED - re-ran the clone - with same results.

/u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/admin/log/clone/run/txkSetOHSConfig_04141000/provision_04141000.log shows the following entries
************************************************************************************
Updating /YMS-UI-context-root and setting WeblogicCluster to : lddscvcpfsap01.mclaneco.com:6801
Inside run context

StackTrace:
java.lang.NullPointerException
at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSDirectives(UpdateOhsT2PMovePlan.java:832)
at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSConfig(UpdateOhsT2PMovePlan.java:742)
at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSConfig(UpdateOhsT2PMovePlan.java:719)
at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSConfig(UpdateOhsT2PMovePlan.java:719)
at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.processMoveplan(UpdateOhsT2PMovePlan.java:652)
at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.main(UpdateOhsT2PMovePlan.java:1083)
************************************************************************************

/u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/admin/log/clone/run/txkSetOHSConfig_04141000/cfgCloneInstance.log shows the following entries
************************************************************************************

##############################################################
Modifying /u02/mclvcp/VCPFD1/fs1/EBSapps/comn/clone/FMW/OHS/moveplan.xml to provide target host info
##############################################################
Running : /u02/mclvcp/VCPFD1/fs1/EBSapps/comn/util/jdk32/jre/bin/java -classpath /u02/mclvcp/VCPFD1/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u02/mclvcp/VCPFD1/fs1/EBSapps/comn/java/classes -mx256m -classpath /u02/mclvcp/VCPFD1/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u02/mclvcp/VCPFD1/fs1/EBSapps/comn/java/classes:/u02/mclvcp/VCPFD1/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u02/mclvcp/VCPFD1/fs1/EBSapps/comn/java/classes/oracle/apps/fnd/jar/adall.jar:/u02/mclvcp/VCPFD1/fs1/FMW_Home/webtier/opmn/lib/wlfullclient.jar oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan -contextfile /u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/appl/admin/VCPFD1_lddscvcpfsap01.xml -moveplan /u02/mclvcp/VCPFD1/fs1/EBSapps/comn/clone/FMW/OHS/moveplan.xml -logfile /u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/admin/log/clone/run/txkSetOHSConfig_04141000/provision_04141000.log -promptmsg hideModifying T2P plan is failed.
Check the /u02/mclvcp/VCPFD1/fs1/inst/apps/VCPFD1_lddscvcpfsap01/admin/log/clone/run/txkSetOHSConfig_04141000/provision_04141000.log for more details.
txkSetOHSConfig.pl exiting with ERROR.
************************************************************************************

[ErmanArslansOracleBlog]

It seems it fails duing updating the moveplan.
It wants the modify the plan to customize or modify the component (in this case, OHS)  according to the destination environment.

Failing code is the line 832 of UpdateOhsT2PMovePlan.java.
It gets a nullpointer and it should be directly related with the moveplan.xml.

Can you please check this moveplan.xml with another instance? Another running and stable instance?


that YMS-UI section especially.
The code says -> Updating /YMS-UI-context-root and setting WeblogicCluster to : lddscvcpfsap01.mclaneco.com:6801

But it reads the value correctly actually..
there may be another value in another file, which causes this..
I mean, maybe it reads the moveplan correctly but can not set the values of the destination files because of their contents.

What is written in mod_wl_ohs.conf ?
There should be an entry there as well. entry for for the oaea type ManagedServer YMS application Location directive, the context root is /YMS-UI-context-root.

By the way, why do you use oaea? İt is normally used when installing additional Oracle applications such as Oracle E-Business Suite AccessGate, eKanban, and Spatial.


{ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/mod_wl_ohs.conf

[mathewjn]

Thanks Erman Aslan:

This is an EBS Value Chain Planning instance with a Demantra schema installed. I have asked a colleague for answers as he is in charge of configuring the VCP and Demantra instances.

[mathewjn]

Is it possible that the implementation of SSO for Demantra on the source guest could be causing this?

[ErmanArslansOracleBlog]

Yes it seems related with the SSO implementation. It is because the configuration it fails, is related to oaea.

[mathewjn]

There were three entries in the following files that were causing the clone to fail at the OHSconfig stage.

----------------------------------------------------------------------------------------------------------------
 
In httpd.conf
 
Added this last line   ( I comment out this line , to run fs_clone )
 
include "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/webgate.conf"
 
----------------------------------------------------------------------------------------------------------------

In webgate.conf
 
I added these line in the middle,
 
<Location /demantra>
AuthType Oblix
require valid-user
</Location>

----------------------------------------------------------------------------------------------------------------
 
In mod_wl_ohs.conf
 
I added these lines in the end
 
<Location /demantra>
                SetHandler weblogic-handler
                WebLogicHost HOSTNAME.domain.com
                WebLogicPort 7003
        </Location>
 ----------------------------------------------------------------------------------------------------------------

Once these lines were commented out, I ran the adpreclone.pl script, took a fresh .tar file copy of the source (PROD) instance, and used that on the target instance (adcfgclone.pl) to complete the clone successfully.

[ErmanArslansOracleBlog]

Good for you mathewjn.
Thanks for sharing.