Clone issue

[satish]

Dear Erman,

We have cloned our TEST from PROD today.While starting the services,we observed the service control script exited with status 1

Analysis from our end:
We have 4 oacore,4 forms in prod,After clone we could see only 3 oacores and 3 forms appear in TEST.Below is the error from service control script

03/19/18-17:58:19 :: admanagedsrvctl.sh version 120.14.12020000.11


Calling txkChkEBSDependecies.pl to perform dependency checks for oacore_server4
Perl script txkChkEBSDependecies.pl got executed successfully


Validated the passed arguments for the option ebs-get-serverstatus
ERROR: Invalid managed server oacore_server4

Server specific logs are located at /u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server4/logs

03/19/18-17:58:27 :: admanagedsrvctl.sh: exiting with status 1

================================================================================

In xml file,we have the reference to oacore_server4 as below

[appluat@rookuatapp01 admin]$ grep -i oacore_server UATDB_rookuatapp02.xml
      <oacore_server>
      </oacore_server>
         <oacore_server_ports oa_var="s_oacore_server_ports">oacore_server3:7206,oacore_server4:7209</oacore_server_ports>
            <oa_service_name oa_var="s_oacorename" type="managed_server">oacore_server3,oacore_server4</oa_service_name>
            <oa_managed_server_name oa_var="s_oacore_managed_servers">oacore_server3,oacore_server4</oa_managed_server_name>

We dont see any directories for oacore_server4 under EBS_DOMAIN_HOME

[appluat@rookuatapp01 servers]$ pwd
/u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers
[appluat@rookuatapp01 servers]$ ls -ld oacore*
drwxr-xr-x 9 appluat dba 3864 Mar 19 10:48 oacore_server1
drwxr-xr-x 9 appluat dba 3864 Mar 19 10:48 oacore_server2
drwxr-xr-x 9 appluat dba 3864 Mar 19 10:52 oacore_server3
[appluat@rookuatapp01 servers]$

could you please help us on why we are unable to start oacore_Serevr4

Thanks,
satish

[satish]

Please note the instance was released and users are able to access with no errors reported,but we just wanted to know the issue

[ErmanArslansOracleBlog]

How did you add this managed server to your "source" instance?
Did you follow EBS documents for adding it properly?
Probably, it was not added to your source instance properly in the first place and that's why you are getting this in your cloned instance..

For adding this managed server in your source instance, you needed to follow ->

Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2 (Doc ID 1905593.1) -- 4.4.1 Adding a new managed server

So, did you add it properly? First answer this question..

[satish]

Thank you for the updates erman.

We have followed the same process as in the document "1905593.1) -- 4.4.1 Adding a new managed server"

PROD-2 application nodes(oacore1,oacore2 on one node and oacore3,oacore4 on other node)
TEST-2 application nodes

After clone,our test server has oacore1,oacore2 on one node and only oacore3 on other node

could you please let us know what might be the reasons for this issue

Thanks,
satish

[ErmanArslansOracleBlog]

I suspect that, oacore_server4 was not properly added into your source environment (somehow different than the others) and that's why after cloning , it was not created in the cloned environment.

Anyways, as for the solution, you can add it to your cloned environment;
Simple read it by following the note below (in your cloned env);

Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2 (Doc ID 1905593.1) -- 4.4.1 Adding a new managed server

[ErmanArslansOracleBlog]

What I also see from your outputs is that, your oacore_server4 is partially there in your cloned environment.. I mean, your cloned EBS knows its existence but Weblogic side of your cloned EBS lacks the related managed server..

So, it may be a timing issue, as well..

When did you executed this clone process? Exactly what did you do for cloning?
When did you executed your preclone script?
When did you add the oacore_server4 into your source environment?
Did you run fs_clone after adding the managed server in your source env? (your source fs1 and fs2 might not be sync, while your were taking a clone from your source env..)

[satish]

Thanks for the update erman

When did you add the oacore_server4 into your source environment? -1 year back
Did you run fs_clone after adding the managed server in your source env? (your source fs1 and fs2 might not be sync, while your were taking a clone from your source env..)

Ran fs_clone and they are in sync

When did you executed your preclone script?
Just before copying the EBSapps folder

When did you executed this clone process? Exactly what did you do for cloning?

on target primary node,we have ran

adcfgclone.pl appsTier <backup context file of primary node> dualfs

Then on secondary node,we have ran adclonectx.pl addnode <primarynode_xml> pairsfile dualfs

while running this command on secondary node,i faced the perl related error below

java.io.IOException: Cannot run program "/u01/PRODAPPS/fs1/FMW_Home/webtier/perl/bin/perl": error=2, No such file or directory which was taking prod's perl location even i have exported test's perl location.

To avoid this error i have ran adpreclone on test system primary node and then ran adclonectx.pl again and then addnode completed successfully


I have reviewed the logs that day but today i am unable to find the logfiles..not sure how they got deleted

[appluat@erpuatapp01 ~]$ cat /u01/UATAPPS/fs2/EBSapps/comn/clone/bin/../FMW/logs/sfslog/run/RCloneSFSAddNode_03181720.log
cat: /u01/UATAPPS/fs2/EBSapps/comn/clone/bin/../FMW/logs/sfslog/run/RCloneSFSAddNode_03181720.log: No such file or directory

[ErmanArslansOracleBlog]

So, this oacore_server4 should be running on your target/clone secondary node then.

So you are following;

Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1)..
As for adding the secondary target application node you should follow -"5.3.2 Steps for adding a node on a non-shared file system"

Are you following step 5.3.2 and doing exactly what is written there?

[satish]

Thanks for the update erman.

oacore_server4 was not found in domain.we are using shared appltop

[ErmanArslansOracleBlog]

So, as your env is based on the shared Application Filesystem then you jumped to "Section 4 of My Oracle Support Knowledge Document 1375769.1, Sharing the Application Tier File System in Oracle E-Business Suite Release 12.2"..

okay..
As your env is based on a shared Application Filesystem, the problem is not related with adding the secondary target/clone application node..
The problem is there even when creating the primary target/clone application node.

So it is probably on preclone phase, which was executed in the source (just before starting the cloning process)

So we start by preparing our Source system;
     1a)For preparing our application tier; we basically run adpreclone.pl from the Run file system.     Adpreclone creates  compressed archives of Oracle Fusion Middleware components as follows;
A compressed archive of the Oracle WebLogic Server home, Oracle Web Tier Utilities home, Oracle Common Utilities home and the Oracle E-Business Suite home:<COMMON_TOP>/clone/FMW/FMW_Home.jar
A compressed archive of the Oracle E-Business Suite Weblogic domain:<COMMON_TOP>/clone/FMW/WLS/EBSdomain.jar
The Oracle E-Business Suite Weblogic domain's configuration template:<COMMON_TOP>/clone/FMW/WLS/plan/moveplan.xml
A compressed archive of the Oracle Web Tier/Oracle HTTP Server configuration instance:<COMMON_TOP>/clone/FMW/OHS/ohsarchive.jar
The Oracle HTTP Server configuration instance's configuration template:<COMMON_TOP>/clone/FMW/OHS/moveplan.xml



Questions:

1)Did you run preclone on apps tier before starting your cloning operations?
2)Any errors during preclone? Any error in its logfile?