eBS R12.2.4 Cloning Issue

[nauman]

Dears, I am facing following error during cloning R12.2.4. START: Creating new WLS domain.Running /erp/fs/oracle/DEV/fs2/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /erp/fs/oracle/DEV/fs2/EBSapps/comn/util/jdk -al /erp/fs/oracle/DEV/fs2/EBSapps/comn/clone/FMW/WLS/EBSdomain.jar -tdl /erp/fs/oracle/DEV/fs2/FMW_Home/user_projects/domains/EBS_domain_DEV -tmw /erp/fs/oracle/DEV/fs2/FMW_Home -mpl /erp/fs/oracle/DEV/fs2/EBSapps/comn/clone/FMW/WLS/plan/moveplan.xml -ldl /erp/fs/oracle/DEV/fs2/inst/apps/DEV_dev1/admin/log/clone/run/wlsT2PApply -silent true -debug true -domainAdminPassword /erp/fs/oracle/DEV/fs2/EBSapps/comn/clone/FMW/tempinfo.txtScript Executed in 121482 milliseconds, returning status 255ERROR: Script failed, exit code 255 wlsT2PApply logs: SEVERE - ERROR - CLONE-20365  Error in executing WebLogic script for configuring server, cluster , datasource and authenticator.SEVERE - CAUSE - CLONE-20365  Either the values specified in the move plan are not correct or some "READ_ONLY" propertiesin the move plans are modified or some internal error happened.SEVERE - ACTION - CLONE-20365  Check the t2p logs for more details. Make sure that values in the move plans are correct.oracle.as.t2p.exceptions.FMWT2PPasteConfigException: Paste config in offline mode failed.        at oracle.as.clone.cloner.component.J2EEComponentApplyCloner.doClone(J2EEComponentApplyCloner.java:257)        at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)        at oracle.as.clone.request.ApplyCloneRequest.applyArchive(ApplyCloneRequest.java:188)        at oracle.as.clone.request.ApplyCloneRequest._clone(ApplyCloneRequest.java:69)        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)        at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:257)        at oracle.as.clone.client.CloningClient.main(CloningClient.java:119)oracle.as.t2p.exceptions.FMWT2PPasteConfigException: PasteConfig failed. Make sure that the move plan and the values specified in moveplan are correct.        at oracle.as.clone.cloner.component.J2EEComponentApplyCloner.doClone(J2EEComponentApplyCloner.java:302)        at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)        at oracle.as.clone.request.ApplyCloneRequest.applyArchive(ApplyCloneRequest.java:188)        at oracle.as.clone.request.ApplyCloneRequest._clone(ApplyCloneRequest.java:69)        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)        at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:257)        at oracle.as.clone.client.CloningClient.main(CloningClient.java:119)SEVERE : Sep 5, 2016 08:44:54 - ERROR - CLONE-20237   Restoring the sourceid "J2EECOMPONENT#EBS_domain_PROD" has failed.SEVERE : Sep 5, 2016 08:44:54 - CAUSE - CLONE-20237   An internal operation failed.SEVERE : - ACTION - CLONE-20237   Check the clone log for more details.SEVERE : Sep 5, 2016 08:44:54 - ERROR - CLONE-20218   Cloning is not successful.SEVERE : 08:44:54 - CAUSE - CLONE-20218   An internal operation failed.SEVERE : Sep 5, 2016 08:44:54 - ACTION - CLONE-20218   Provide the clone log and error file for investigation. Note: Database version is 11g 11.2.0.4 and db cloning completed normal. I am following the standard oracle document for cloning R12.2 Kindly help me in resolving this issue.

[nauman]

I need an urgent help. Would be obliged if someone can assist me please.

[nauman]

Dear Erman,

Making my issue more clear to you to get the assistance.

I am in a process of Cloning R12.2.4 from a 2 x Node RAC DB / 2 x node Appl to 4 x Node RAC DB/ 2 x Node Appl.

Database part went well. Autoconfig is enabled and context file entries are generated.

Base OS being used is RHEL 5.8 for application and for database it is OEL 6.8

I am on txk, code 9 on application.

While running the adcfgclone Appstier (havent used dual FS because i am not in a practice to do it) the first issue that i observed is

ORA 12505 - while making the connection with database using SID. But connection was obtained using the service name and connection was successful so i ignored the issue.

The adcfgclone script keeps going and then it stuck while creating the wlst domain with the fol error

Executed in 121482 milliseconds, returning status 255ERROR: Script failed, exit code 255 wlsT2PApply

i went to the error logs and observe the fol msgs

CLONE-20365  Error in executing WebLogic script for configuring server, cluster , datasource and authenticator.SEVERE - CAUSE - CLONE-20365  Either the values specified in the move plan are not correct or some "READ_ONLY" propertiesin the move plans are modified or some internal error happened.SEVERE - ACTION - CLONE-20365  Check the t2p logs for more details. Make sure that values in the move plans are correct.oracle.as.t2p.exceptions.FMWT2PPasteConfigException: Paste config in offline mode failed.

and

ERROR - CLONE-20237   Restoring the sourceid "J2EECOMPONENT#EBS_domain_PROD" has failed.SEVERE : Sep 5, 2016 08:44:54 - CAUSE - CLONE-20237   An internal operation failed.SEVERE : - ACTION - CLONE-20237   Check the clone log for more details.SEVERE : Sep 5, 2016 08:44:54 - ERROR - CLONE-20218   Cloning is not successful.SEVERE : 08:44:54 - CAUSE - CLONE-20218   An internal operation failed.SEVERE : Sep 5, 2016 08:44:54 - ACTION - CLONE-20218   Provide the clone log and error file for investigation


Moreover one of the error that i observed in the logfile is

error:setlistenaddress()

The machine which is currently my target instance is previously running the eBS Appl R12.2.4. To avoid any port conflict issue i have stopped the application svcs and rebooted the machine and assumed that ports are now free.

However, i am not getting the port conflict issue error in my log files. the port related issue that i found in my log files is att as screenshot.


Moreover, in the recent past i added some of the forms servers on my source instance. These forms servers were added successfully but sessions are not getting connected to newly added forms servers. I am adding this details to know whether these may be cause of the issue.

Hoping for your assistance.

Thanks & Regards

[ErmanArslansOracleBlog]

Hi,

I m working on it. I will send my update soon.

[ErmanArslansOracleBlog]

Normally, we should look at the moveplan.xml and see if all the managed servers are in correct port range.

The issue looks related with Bug 22840524. So your first try should be either the following action plan or the workaround;

Action plan:

Ensure a valid backup exists before applying the recommended patch.
Apply Patch 27212806 along with any required prerequisite patches per the readme file instructions.
Retest the cloning process and confirm the error is resolved.

A workaround may be running preclone and adcfgclone again.. But this time with a port pool which is in a higher range than the SOURCE instance.

Except the failing lines, your error stack is similar to the one given in the following note ->

12.2 Creating New WLS Domain Fails At Script Adcfgclone.pl With "oracle.as.t2p.exceptions.FMWT2PPasteConfigException: PasteConfig failed. Make sure that the move plan and the values specified in moveplan are correct." (Doc ID 2215729.1)

[nauman]

Dear Erman

Thanks for your reply

I would like if you can clarify the following ambiguities

1. Can you please let me know the location of moveplan.xml. And do i need to check the moveplan.xml at the source machine. What i know about the moveplan.xml that one should not edit the file manually.

2. How to change the port pool. During the adcfgclone process i am selecting

Do you want the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 0

The reason of selecting 0 is that my database is configured on 1521 port. What i know is that since i have configured the database on 1521 therefore i need to give port 0 on application side.

The source side is also having the same port range 1521(DB) and 8000 (appl).

Now can you please let me know how to change the port pool?

Thanks for your anticipation.

Thanks & Regards

[nauman]

Moreover, I dont have any issue to give the higher port pool range till the time application get connection with database port 1521 after giving the higher value.

In addition to it clarity is req that why i need to run preclone again on source machine since nothing is going to be change on source instance.

Your kind assistance is requested for the resolution of issue.

Thanks

[ErmanArslansOracleBlog]

For changing the port pool; you will first change your target db listener port (according to the EBS cloning documentation) and then supply a different port pool during your adcfgclone.pl run.

As for moveplan.xml;

The Oracle E-Business Suite WebLogic domain's configuration template:
<COMMON_TOP>/clone/FMW/WLS/plan/moveplan.xml

The Oracle HTTP Server configuration instance's configuration template:
<COMMON_TOP>/clone/FMW/OHS/moveplan.xml

But I already mentioned, the Patch 27212806 seem promising.. You can first apply this patch on your TEST and then PROD, and then retest your cloning.

Reference for the patch: Oracle - MOS DOCUMENT ->

12.2 Creating New WLS Domain Fails At Script Adcfgclone.pl With "oracle.as.t2p.exceptions.FMWT2PPasteConfigException: PasteConfig failed. Make sure that the move plan and the values specified in moveplan are correct." (Doc ID 2215729.1)

[nauman]

Dear Erman,

Thanks for your valuable time and update.

I use to clone the R12.2.4 instance in my test instance and never found a bug like this. Therefore i intend to go for a workaround first and if no luck then will got to apply the patch. I am going to follow the following action plan. Please let me know if any change is suggested by you:-

1. Change the listener port of database on target machine
2. Run the autoconfig on all nodes of target database
3. Run the adcfgclone on the application with the Changed port pool that coincide with the changed db port

Pl highlight if any change is suggested.

Thanks & Regards

[ErmanArslansOracleBlog]

okay.. let's see.. This is the workaround documented in MOS note 2215729.1.
See if it works..
If it won't solve the issue, then you may consider patching.

[nauman]

Hi,

Yeah i will update you accordingly.

Following action plan i am going to follow to change the listener port of Target DB running in RAC

1. Change the listener port in Database Context file manually in all DB nodes
2. Change the port in tnsnames.ora and listener.ora files
3. Run the autoconfig of all nodes of DBs

Pleas let me know if any change is suggested.

Thanks & Regards

[nauman]

Dear Erman,

Can you please let me know how to change the database listener port on eBS RAC environment.

Regards

[ErmanArslansOracleBlog]

Currently there is no clear documentation for that.
What I would do is, change the listener configuration manually.
clear the setup using fnd_conc_clone.setup_clean
use adbldxml to rebuild the contenxt file and run autoconfig on all the rac nodes.
After running autoconfig I would check the local_listener and remote_listener parameters to see if they are okay. Then I would check the tnsnames files of the db tier to see if they are then.
Lastly I would check the scan listeners and local listeners using lsrcntl status. I would see if the registered services are okay or not.. IF all are okay, then that 's it.
Note that, scan listener should have entries for all the rac nodes.. A local listener should have one entry per node. (an entry for the instance it belongs to)

These are TEST database right? Just act like they are new home.. Act with the logic documented in See the Patching AutoConfig subsection of the Technical Configuration section of the Oracle E-Business Suite Setup Guide Release 12.2, especially the "Enabling AutoConfig on a New Oracle Home" section for instructions on how to implement and run AutoConfig.

Read this as well -> Using Oracle 12c Release 1 (12.1) Real Application Clusters with Oracle E-Business Suite Release 12.2 (Doc ID 1626606.1)

[nauman]

Dear Erman Thanks for your reply.

Please see and confirm that i have found a different patch regarding my problem which exactly match my issue being faced.

Patch 28708563: MERGE REQUEST ON TOP OF 11.1.1.9.0 FOR BUGS 20484781 28371189

This patch resolve the following bugs:-

20484781 PASTECONFIG ERRORS WHILE CONFIGURING MACHINE BECAUSE OF SWAPPING MACHINE NAMES >> In my case hostname of the machine is changed
28371189 T2P WLS DOMAIN CLONE FAILED WHILE SETTING SERVER LISTENADDRESS ON TARGET DOMAIN >> Error:Setlistenaddress() is also observed in my log files

Need your guidance to validate my finding. Moreover, we havent done patching on FMW home before. I hope patching can be done in hot mode and can also be reverted.

Thanks & Regards

[ErmanArslansOracleBlog]

Hi,

I already recommended you a patch , patch 27212806. This patch is recommended in DOC ID 2215729.1.

Except the failing lines, your error stack is similar to the one given in the following note ->

12.2 Creating New WLS Domain Fails At Script Adcfgclone.pl With "oracle.as.t2p.exceptions.FMWT2PPasteConfigException: PasteConfig failed. Make sure that the move plan and the values specified in moveplan are correct." (Doc ID 2215729.1)

The logs you sent to me are just a small part of the actual logs, so if you see the same pattern in your logs, then you can also apply Patch 28708563.

You can try the patches one by one. (apply the first patch, retest, and if it won't work, apply the second patch and retest)

In order to patch the FMW, follow the document named -> Oracle E-Business Suite 12.2 Patching Technology Components Guide (Doc ID 1355068.1)

[nauman]

Dear Erman,

I would like to thank you. My issue has been resolved by applying the workaround. :)


Now i am in a process of changing the physical application server. Need to do the same procedure. Since current application is attached with the database so i am going to follow the below action plan. Please confirm the plan:-

1. De register the application from db by EXEC FND_CONC_CLONE.SETUP_CLEAN and clear the fnd node table.
2. Run the autoconfig on all db nodes
3. Move the application tar on new machine and will run adcfgclone

Please confirm the action plan and suggest if any change is recommended.

Thanks & Regards

[ErmanArslansOracleBlog]

Good :)

Yes.. After running adcfgclone.pl, you may need to do some extras.. It depends on your env, however; the general steps are ok.