fs_clone failed for post upgrade 12.2.0 to 12.2.9

[shahzad53]

Dear team,

I have successfully done a ebs upgrade 12.2.0 to 12.2.9 , but post upgrade is given error during fs_clone .

we have tried several time to complete fs_clone , but every time it is aborted.

Below is the error logs messages.


START: Instantiating the OHS Config.
Running /u01/oracle/PROD/fs2/FMW_Home/webtier/perl/bin/perl  -I/u01/oracle/PROD/fs2/FMW_Home/webtier/perl/lib/5.10.0 -I/u01/oracle/PROD/fs2/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I/u01/oracle/PROD/fs2/EBSapps/appl/au/12.0.0/perl -I/u01/oracle/PROD/fs2/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u01/oracle/PROD/fs2/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl cfgCloneInstance -ctxfile=/u01/oracle/PROD/fs2/inst/apps/PROD_appsnode1229/appl/admin/PROD_appsnode1229.xml -archvloc=/u01/oracle/PROD/fs1/EBSapps/comn/adopclone_appsnode1229/FMW/OHS/ohsarchive.jar -mvpln=/u01/oracle/PROD/fs1/EBSapps/comn/adopclone_appsnode1229/FMW/OHS/moveplan.xml -t2plogloc=/u01/oracle/PROD/fs1/inst/apps/PROD_appsnode1229/admin/log/clone/ohsT2PApply -logdir=/u01/oracle/PROD/fs1/inst/apps/PROD_appsnode1229/admin/log/clone -console=off -promptmsg=hide
Script Executed in 35450 milliseconds, returning status 255
ERROR: Script failed, exit code 255

START: Inside exitClone....
Updating status INCOMPLETE for ApplyohsTechStack

[applmgr@appsnode1229 ~]$ adopscanlog -latest=yes




Scanning /u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/ directory ...

/u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/20191120_095107/fs_clone/appsnode1229/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_11201014.log:
------------------------------------------------------------------------------------------------------------------------------------------------

Lines #(2741-2745):

*** LOG FILE: /u01/oracle/PROD/fs2/inst/apps/PROD_appsnode1229/logs/appl/rgf/TXK/txkSetAppsConf_11201121.log ***
ERROR: The value </u01/oracle/PROD/fs2/FMW_Home/webtier/instances/EBS_web_OHS1> for s_ohs_instance_loc in /u01/oracle/PROD/fs2/inst/apps/PROD_appsnode1229/appl/admin/PROD_appsnode1229.xml isn't a directory.

Instantiating template: mod_wl_ohs_conf_FMW.tmp

Lines #(5459-5463):
Running /u01/oracle/PROD/fs2/FMW_Home/webtier/perl/bin/perl  -I/u01/oracle/PROD/fs2/FMW_Home/webtier/perl/lib/5.10.0 -I/u01/oracle/PROD/fs2/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I/u01/oracle/PROD/fs2/EBSapps/appl/au/12.0.0/perl -I/u01/oracle/PROD/fs2/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u01/oracle/PROD/fs2/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl cfgCloneInstance -ctxfile=/u01/oracle/PROD/fs2/inst/apps/PROD_appsnode1229/appl/admin/PROD_appsnode1229.xml -archvloc=/u01/oracle/PROD/fs1/EBSapps/comn/adopclone_appsnode1229/FMW/OHS/ohsarchive.jar -mvpln=/u01/oracle/PROD/fs1/EBSapps/comn/adopclone_appsnode1229/FMW/OHS/moveplan.xml -t2plogloc=/u01/oracle/PROD/fs1/inst/apps/PROD_appsnode1229/admin/log/clone/ohsT2PApply -logdir=/u01/oracle/PROD/fs1/inst/apps/PROD_appsnode1229/admin/log/clone -console=off -promptmsg=hide
Script Executed in 35450 milliseconds, returning status 255
ERROR: Script failed, exit code 255

START: Inside exitClone....

/u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/20191120_095107/fs_clone/appsnode1229/TXK_SYNC_create/fsclone_apply/ohsT2PApply/CLONE2019-11-20_11-46-31_1557899290.log:
------------------------------------------------------------------------------------------------------------------------------------------------------------------

Lines #(1117-1121):
---------------------------------------------------
Error Message  :1
  [PLUGIN][OHS] - ERROR - Nov 20, 2019 11:47:01 - CLONE-26009   OHS T2P failed.
  [PLUGIN][OHS] - CAUSE - Nov 20, 2019 11:47:01 - CLONE-26009   Unable to start OS component.
  [PLUGIN][OHS] - ACTION - Nov 20, 2019 11:47:01 - CLONE-26009   Check clone log and error file and ohs log file /u01/oracle/PROD/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/console~OHS~1.log for root cause.

/u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/20191120_095107/fs_clone/appsnode1229/TXK_SYNC_create/txkADOPPreparePhaseSynchronize.log:
-----------------------------------------------------------------------------------------------------------------------------------

Lines #(501-503):
TIME    : Wed Nov 20 11:47:02 2019
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /u01/oracle/PROD/fs1/EBSapps/comn/adopclone_appsnode1229/bin/adclone.pl did not go through successfully.

/u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/20191120_095107/adop.log:
-------------------------------------------------------------------

Lines #(107-112):
    [PROCEDURE] Calling: /u01/oracle/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
    [EVENT]     Log: /u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/20191120_095107/fs_clone/appsnode1229
    [UNEXPECTED]Error occurred running "perl /u01/oracle/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/PROD/fs1/inst/apps/PROD_appsnode1229/appl/admin/PROD_appsnode1229.xml -patchcontextfile=/u01/oracle/PROD/fs2/inst/apps/PROD_appsnode1229/appl/admin/PROD_appsnode1229.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=3 -timestamp=20191120_095107 -outdir=/u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/20191120_095107/fs_clone/appsnode1229"
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u01/oracle/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/PROD/fs1/inst/apps/PROD_appsnode1229/appl/admin/PROD_appsnode1229.xml -patchcontextfile=/u01/oracle/PROD/fs2/inst/apps/PROD_appsnode1229/appl/admin/PROD_appsnode1229.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=3 -timestamp=20191120_095107 -outdir=/u01/oracle/PROD/fs_ne/EBSapps/log/adop/3/20191120_095107/fs_clone/appsnode1229".
    [EVENT]     Releasing managed server ports.
        [PROCEDURE] [START 2019/11/20 11:47:02] Releasing ports

Lines #(135-139):
            [PROCEDURE] [END   2019/11/20 11:47:40] Check and Stop patch Node Manager
            [PROCEDURE] [END   2019/11/20 11:47:40] Check and stop patch Node Manager
    [UNEXPECTED]fs_clone has failed.
    [PROCEDURE] [START 2019/11/20 11:47:40] Unlocking sessions table
    [STATEMENT] Unlocking ad_adop_sessions table for appsnode1229 with wait interval of 60 seconds and number of tries 2
[applmgr@appsnode1229 ~]$

[ErmanArslansOracleBlog]

Your OHS can not be started.

Check OHS related log files and then take actions.

Logs files like,  CLONE-timestamp.log, console~OHS~1.log, EBS_web_*.log should be reviewed.
Check this note as well -> Fs_clone Is Failing on DMZ Node With Error "(13)Permission denied error" (Doc ID 2410217.1)

It can even related with your OS..  It may be related with a privileged port configuration or some other thing.

We need to see those logs and decide.

[shahzad53]

Hi Brother Erman,

all log files are attached, please check & advise.

Regards,
shahzadCLONE2019-11-20_11-46-31_1557899290.logconsole~OHS~1.logEBS_web.log

[ErmanArslansOracleBlog]

It seems you have configured SSL for this environment?
Is this right? Is it how it needs to be?

You have "Unable to initialize SSL environment, nzos call nzosSetCredential returned 43084" error in your HTTP Server's log file.

So your issue is related with your SSL setup.

Here is a known issue for that -> OHS Fails to Start With Error "Unable to initialize SSL environment, nzos call nzosSetCredential returned 43084" (Doc ID 2245311.1)

Please review your SSL configuration.. Or disable SSL if you don't need it in this clone environment..

[shahzad53]

Dear Erman,

I am not enable any SSL setup. I am just upgrade from ebs 12.2.0 to 12.2.9 and post step is to run

fs_clone , so during fs_clone we found this error.

How we disable SSL in setup , Kindly advice.


Kindly help us to solve it.

Regards,

[ErmanArslansOracleBlog]

Are you sure?
Because even your http port gives us a clue.. It is 443.. This is default.
Please check your change history accordingly.

In order to setup SSL in 12.2, we follow the document below;

Enabling TLS in Oracle E-Business Suite Release 12.2 (Doc ID 1367293.1)
There is section named Appendix A: Disabling TLS..

anyways, again , check your change history.. Don't do anything without being sure.
But as your log files say ; your OHS is trying to start itself with SSL...