FS_CLONE failure in 12.2.7

[karthikmca.bhc]

Hi Support,

Please provide a fix for the below issue. 

  [applmgr@1227demo scripts]$ adop phase=fs_clone force=yes workers=10
===========================================================================
ADOP (C.Delta.12)
Session ID: 13
Node: 1227demo
Phase: fs_clone
Log: /u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/adop.log
===========================================================================

Validating configuration on node: [1227demo].
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/validate/1227demo
        [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log fil           details.

Checking for pending cleanup actions.
    No pending cleanup actions found.

Blocking managed server ports.
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo/txkCloneAcquirePort.log

Performing CLONE steps.
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo

Beginning application tier FSCloneStage - Sun Aug 23 18:14:24 2020

/u02/applmgr/fs2/EBSapps/comn/util/jdk32/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/           ath /u02/applmgr/fs2/FMW_Home/webtier/lib/xmlparserv2.jar:/u02/applmgr/fs2/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/u02/applmgr/fs2/EBSapps/comn/java/classes:/u           fs2/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/u02/applmgr/fs2/FMW_Home/webtier/oui/jlib/ewt3.jar:/u02/applmgr/fs2/FMW_Home/webtier/oui/jlib/share.jar:/u02/           /FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u02/applmgr/fs2/FMW_Home/webtier/jlib/ojmisc.jar:/u02/applmgr/fs2/FMW_Home/wlserver_10.3/server/lib/w           :/u02/applmgr/fs2/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneStageAppsTier -e /u02/applmgr/fs2/inst/apps/R1227_1227demo/a           1227_1227demo.xml -targ /u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -stage /u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo -tmp            nent appsTier -nopromptmsg
Log file located at /u02/applmgr/fs2/inst/apps/R1227_1227demo/admin/log/clone/FSCloneStageAppsTier_08231814.log
Completed FSCloneStage...
Sun Aug 23 18:33:59 2020
*******FATAL ERROR*******
PROGRAM : (/u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Sun Aug 23 18:34:00 2020
FUNCTION: TXK::IO::open [ Level 3 ]
MESSAGES:
error = Unable to open file
errorno = No such file or directory
file = </u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo/TXK_SYNC_create/fsclone_stage/NONE

STACK TRACE
 at /u02/applmgr/fs2/EBSapps/appl/au/12.0.0/perl/TXK/Error.pm line 168
        TXK::Error::abort('TXK::Error', 'HASH(0x25756c8)') called at /u02/applmgr/fs2/EBSapps/appl/au/12.0.0/perl/TXK/Common.pm line 299
        TXK::Common::doError('TXK::IO=HASH(0x3cd2200)', 'HASH(0x3187e80)', undef) called at /u02/applmgr/fs2/EBSapps/appl/au/12.0.0/perl/TXK/Common.pm line 3           14
        TXK::Common::setError('TXK::IO=HASH(0x3cd2200)', 'HASH(0x3187e80)') called at /u02/applmgr/fs2/EBSapps/appl/au/12.0.0/perl/TXK/IO.pm line 291
        TXK::IO::open('TXK::IO=HASH(0x3cd2200)', 'HASH(0x3243570)') called at /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchr           onize.pl line 6109
        main::searchFileContents('/u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clo...', 'Status: Completed Successfully') called at /u02/applmgr           /fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl line 5988
        main::runFSCloneStage() called at /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl line 2229
        main::createPatchApplTop() called at /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl line 645
    [UNEXPECTED]Error occurred running "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmg           r/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promp           tmsg=hide -console=off -mode=create -sessionid=13 -timestamp=20200823_181133 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo           "
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/tx           kADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/           apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=13 -timestamp=20200823_181133 -outdir=/u02/applmgr/fs_           ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo".
    Releasing managed server ports.
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo/txkCloneAcquirePort.log
    Stopping services on patch file system.
    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.11

Environment file /u02/applmgr/fs1/EBSapps/appl/APPSR1227_1227demo.env is not found.
/u02/applmgr/fs1/inst/apps/R1227_1227demo/admin/scripts/adadminsrvctl.sh: line 161: $LOGFILE: ambiguous redirect
    [UNEXPECTED]Error occurred running "sh /u02/applmgr/fs1/inst/apps/R1227_1227demo/admin/scripts/adadminsrvctl.sh  stop -nopromptmsg"
    [UNEXPECTED]Error running adadminsrvctl command
[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

[applmgr@1227demo ~]$ adopscanlog -latest=yes

Scanning /u02/applmgr/fs_ne/EBSapps/log/adop/13/ directory ...

/u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/adop.log:
----------------------------------------------------------------
Lines #(111-116):
    [PROCEDURE] Calling: /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
    [EVENT]     Log: /u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo
    [UNEXPECTED]Error occurred running "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=13 -timestamp=20200823_181133 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo"
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=13 -timestamp=20200823_181133 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo".
    [EVENT]     Releasing managed server ports.
        [PROCEDURE] [START 2020/08/23 18:34:00] Releasing ports
Lines #(132-137):
            [PROCEDURE] [START 2020/08/23 18:34:02] Check and stop patch Admin Server
            [PROCEDURE] [START 2020/08/23 18:34:03] Check and Stop patch FS Admin Server
            [UNEXPECTED]Error occurred running "sh /u02/applmgr/fs1/inst/apps/R1227_1227demo/admin/scripts/adadminsrvctl.sh  stop -nopromptmsg"
            [UNEXPECTED]Error running adadminsrvctl command
        [PROCEDURE] [START 2020/08/23 18:34:03] Unlocking sessions table
        [STATEMENT] Unlocking ad_adop_sessions table for 1227demo with wait interval of 60 seconds and number of tries 2

[ErmanArslansOracleBlog]

Hmm that "unable to file - NONE" issue again.

Check this thread- > http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/Fs-clone-failed-td8430.html

Please review the issue above..
As I said there ->

Ensure there are no adop/adworker or other  adop related processes running in the systems..
The following note may help you on this ->

Script to Diagnose adop and Other AD-TXK Issues in Oracle E-Business Suite Release 12.2 [ Patch 19045166:R12.AD.C ] (Doc ID 1901242.1)
Details of running adop, adpatch and adworker processes..

However; most probably , you will get approval from SR and perform the manual update I mentioned....

Lastly, the following is worth trying.

adop phase=abort
adop phase=cleanup cleanup_mode=full
adop phase=fs_clone force=yes

If the issue persist, a manual update is required.. ( an update to the adop tables.. (ad_adop_session_patches))

[karthikmca.bhc]

Dear Erman,

Thaks much. I am sure and have confirmed that there is no adop/adworker or other  adop related processes running.

I tried 3 times and ended with the same issue. I can see that the failure happening during the FMW phase as adop not able to perform in that phase.

This is my test instance, so i can try manual update on ad_adop_session_patches table. Please tell me what exactly i should update on this table to fix this issue?

I performed below but still the fs_clone failed with same issue.

[applmgr@122demo scripts]$ ./adadminsrvctl.sh stop

[applmgr@122demo ~]$ adop -status -detail
===============================================================
Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
1227demo        master     FS_CLONE        FAILED          2020/08/23 18:12:16                       4:26:35


SQL> select adop_session_id,status from ad_adop_session_patches where status in('F') order by ADOP_SESSION_ID desc;

ADOP_SESSION_ID STATUS
--------------- --------------------
             13 F
             11 F
              9 F
              8 F
              7 F
              5 F
              5 F
              5 F

[applmgr@122demo ~]$ sqlplus apps/apps

SQL> create table adop_bkp_4 as (select * from ad_adop_session_patches);

Table created.

SQL> update ad_adop_session_patches set status='Y' where status='F' and adop_session_id='5';
SQL> update ad_adop_session_patches set status='Y' where status='F' and adop_session_id='7';
SQL> update ad_adop_session_patches set status='Y' where status='F' and adop_session_id='8';
SQL> update ad_adop_session_patches set status='Y' where status='F' and adop_session_id='9';
SQL> update ad_adop_session_patches set status='Y' where status='F' and adop_session_id='11';
SQL> update ad_adop_session_patches set status='Y' where status='F' and adop_session_id='13';

SQL> commit;

[applmgr@122demo scripts]$ pkill -u applmgr

unset module

[applmgr@1227demo ~]$ adop phase=cleanup cleanup_mode=full workers=10

[applmgr@122demo scripts]$ adop -status -detail
===============================================================
Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
1227demo        master     FS_CLONE                        2020/08/23 23:00:56                       0:30:50


[applmgr@122demo scripts]$ adop phase=fs_clone force=yes workers=10


Please let me know if anything else i should try to make it resolve.

Note: I cannot run 'abort' because it says there is no active adop cycle.

Thanks,
Karthik

[ErmanArslansOracleBlog]

Those status values may be interpreted as follows;
                     'N','Applied on other nodes'
                     'R','Running'
                     'H','Failed Hard'
                     'F','Failed Jobs Skipped'
                     'S','Success Jobs Skipped'
                     'Y','Success'
                     'C','Clone Complete'

What is the output of the following;

SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' ;

What happens when you backup that table again and update it according to the where condition of the above query?

[karthikmca.bhc]

Hi,

Please find below the output of your requested query,

[applmgr@1227demo ~]$ adop -status -detail

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 16
Command: status
Output: /u02/applmgr/fs_ne/EBSapps/log/adop/16/20200825_141507/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
1227demo        master     FS_CLONE        FAILED          2020/08/24 00:17:02                       37:83:14


SQL> SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' ;

CLONE_STATUS||','
-------------------------------
REMOVE-FMW-HOME,


Please let me kow which column i should update on the table ad_adop_session_patches after taking backup of the table?

Please provide me the sequence that you want me to try to resolve this.


below is my latest adop scan log output:

[applmgr@1227demo ~]$ adopmon
Enter the APPS password:

Validating credentials...
Printing the log statements starting from sequence #825878

Timestamp           Node name           Message Type Message Text
---------------------------------------------------------------------------------------------------------------------------
2020/08/25 03:05:53 1227demo            UNEXPECTED   Error occurred running "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/p
                                                     atch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applm
                                                     gr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchco
                                                     ntextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1
                                                     227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=16 -t
                                                     imestamp=20200825_010237 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/1
                                                     6/20200825_010237/fs_clone/1227demo"
2020/08/25 03:05:53 1227demo            UNEXPECTED   occurred during CLONE Patch File System from Run File System, running
                                                     command: "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/t
                                                     xkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/ap
                                                     ps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02
                                                     /applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -p
                                                     romptmsg=hide -console=off -mode=create -sessionid=16 -timestamp=20200
                                                     825_010237 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/16/20200825_010
                                                     237/fs_clone/1227demo".
2020/08/25 03:05:53 1227demo            EVENT        Releasing managed server ports.
2020/08/25 03:05:54 1227demo            EVENT        Log: /u02/applmgr/fs_ne/EBSapps/log/adop/16/20200825_010237/fs_clone/1
                                                     227demo/txkCloneAcquirePort.log
2020/08/25 03:05:55 1227demo            EVENT        Stopping services on patch file system.
2020/08/25 03:05:55 1227demo            EVENT        Stopping admin server.
2020/08/25 03:06:24 1227demo            EVENT        Stopping node manager.
2020/08/25 03:06:31 1227demo            UNEXPECTED   fs_clone has failed.
2020/08/25 14:15:17 1227demo            ERROR        Unable to set edition. Invalid edition type : PATCH


[applmgr@1227demo ~]$ adopscanlog -latest=yes

Scanning /u02/applmgr/fs_ne/EBSapps/log/adop/16/ directory ...

No Errors.

[ErmanArslansOracleBlog]

Backup the adop session table and update it as you already did earlier. This time use the following conditions in your update;

 status in ('R','F','N') AND bug_number = 'CLONE' ;

This is what we can do at the moment.. Give it a try.. (but take your backup before that)

I think that some records which are left there by an earlier adop session is causing the issue.

Read the following thread line by line to understand what I mean - >

http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/Fs-clone-failed-td8430.html

[ErmanArslansOracleBlog]

After trying the action that I suggested in the previous update, please send me the output of the following;

SELECT * FROM ad_adop_session_patches WHERE status in ('R','F','N');   -- format the output properly, so I can review.

[karthikmca.bhc]

Failed again... Performed below steps as you advised. attached the query output post failure.


[applmgr@1227demo applmgr]$ . ./EBSapps.env run
[applmgr@1227demo applmgr]$ adadminsrvctl.sh stop forcerunfs
[applmgr@1227demo applmgr]$ adnodemgrctl.sh stop forcerunfs

[applmgr@1227demo applmgr]$ . ./EBSapps.env patch
[applmgr@1227demo applmgr]$ adadminsrvctl.sh stop forcepatchfs
[applmgr@1227demo applmgr]$ adnodemgrctl.sh stop forcepatchfs
[applmgr@1227demo scripts]$ adopmnctl.sh stop forcepatchfs

[applmgr@1227demo scripts]$ ps ufx
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
applmgr   6105  0.1  0.0 108800  2232 pts/2    S    18:04   0:00 -bash
applmgr   6752  3.0  0.0 110252  1068 pts/2    R+   18:08   0:00  \_ ps ufx


SQL> SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' ;
CLONE_STATUS||','
-------------------------------
REMOVE-FMW-HOME,

SQL> create table adop_bkp_5 as (select * from ad_adop_session_patches);
Table created.

SQL> update ad_adop_session_patches set status='Y' where status in ('R','F','N') AND bug_number = 'CLONE' ;
1 row updated.
SQL> commit;

-open fresh new terminal
-source run-fs env file

[applmgr@1227demo applmgr]$ adop phase=fs_clone force=yes workers=10

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials.
Warning: WebLogic Admin Server is not running.
Trying to start Admin Server.
You are running adadminsrvctl.sh version 120.10.12020000.11
Starting WLS Admin Server...
Refer /u02/applmgr/fs2/inst/apps/R1227_1227demo/logs/appl/admin/log/adadminsrvctl.txt for details
AdminServer logs are located at /u02/applmgr/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs
adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /u02/applmgr/fs2/inst/apps/R1227_1227demo/logs/appl/admin/log/adadminsrvctl.txt for more information ...
Initializing.
    Run Edition context  : /u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml
    Patch edition context: /u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml
    Patch file system free space: 65.25 GB

Validating system setup.
    Node registry is valid.
    [WARNING]   ETCC: The following required database fixes have not been applied to node 1227demo:
                  8975044
                  30758943
                Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.
Checking for existing adop sessions.
    No pending session exists.
    Starting new adop session.
===============================================
ADOP (C.Delta.12)
Session ID: 17
Node: 1227demo
Phase: fs_clone
Log: /u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/adop.log
==================================================
Validating configuration on node: [1227demo].
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/validate/1227demo
        [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
Checking for pending cleanup actions.
    No pending cleanup actions found.
Blocking managed server ports.
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo/txkCloneAcquirePort.log
Performing CLONE steps.
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo

Beginning application tier FSCloneStage - Tue Aug 25 18:24:35 2020
/u02/applmgr/fs2/EBSapps/comn/util/jdk32/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /u02/applmgr/fs2/FMW_Home/webtier/lib/xmlparserv2.jar:/u02/applmgr/fs2/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/u02/applmgr/fs2/EBSapps/comn/java/classes:/u02/applmgr/fs2/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/u02/applmgr/fs2/FMW_Home/webtier/oui/jlib/ewt3.jar:/u02/applmgr/fs2/FMW_Home/webtier/oui/jlib/share.jar:/u02/applmgr/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u02/applmgr/fs2/FMW_Home/webtier/jlib/ojmisc.jar:/u02/applmgr/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u02/applmgr/fs2/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneStageAppsTier -e /u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -targ /u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -stage /u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo -tmp /tmp -component appsTier -nopromptmsg
Log file located at /u02/applmgr/fs2/inst/apps/R1227_1227demo/admin/log/clone/FSCloneStageAppsTier_08251824.log
Completed FSCloneStage...
Tue Aug 25 18:44:11 2020

Beginning application tier FSCloneApply - Tue Aug 25 18:49:02 2020
/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/FMW/t2pjdk/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/xmlparserv2.jar:/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/ojdbc6.jar:/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/java:/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/oui/OraInstaller.jar:/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/oui/ewt3.jar:/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/oui/share.jar:/u02/applmgr/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/ojmisc.jar:/u02/applmgr/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneApplyAppsTier -e /u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -targ /u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -stage /u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo -tmp /u02/applmgr/fs2/inst/apps/R1227_1227demo/appltmp -nodbac -component appsTier -nopromptmsg
Log file located at /u02/applmgr/fs2/inst/apps/R1227_1227demo/admin/log/clone/FSCloneApplyAppsTier_08251849.log
Log file located at /u02/applmgr/fs2/inst/apps/R1227_1227demo/admin/log/clone/FSCloneApplyAppsTier_08251849.log
Target System Fusion Middleware Home set to /u02/applmgr/fs1/FMW_Home
Target System Web Oracle Home set to /u02/applmgr/fs1/FMW_Home/webtier
Target System Appl TOP set to /u02/applmgr/fs1/EBSapps/appl
Target System COMMON TOP set to /u02/applmgr/fs1/EBSapps/comn
Target System Instance Top set to /u02/applmgr/fs1/inst/apps/R1227_1227demo
Report file located at /u02/applmgr/fs1/inst/apps/R1227_1227demo/temp/portpool.lst
The new APPL_TOP context file has been created :
  /u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml
contextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml

Enabling EBS_LOGON trigger
ERROR while running FSCloneApply...
Tue Aug 25 20:21:42 2020
*******FATAL ERROR*******
PROGRAM : (/u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Tue Aug 25 20:21:42 2020
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/bin/adclone.pl did not go through successfully.

    [UNEXPECTED]Error occurred running "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=17 -timestamp=20200825_181813 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo"
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=17 -timestamp=20200825_181813 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo".
    Releasing managed server ports.
    Log: /u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo/txkCloneAcquirePort.log
    Stopping services on patch file system.
    Stopping admin server.
You are running adadminsrvctl.sh version 120.10.12020000.11
Stopping WLS Admin Server...
Refer /u02/applmgr/fs1/inst/apps/R1227_1227demo/logs/appl/admin/log/adadminsrvctl.txt for details
AdminServer logs are located at /u02/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs
adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /u02/applmgr/fs1/inst/apps/R1227_1227demo/logs/appl/admin/log/adadminsrvctl.txt for more information ...
    Stopping node manager.
You are running adnodemgrctl.sh version 120.11.12020000.12
The Node Manager is already shutdown
NodeManager log is located at /u02/applmgr/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1
adnodemgrctl.sh: exiting with status 2
adnodemgrctl.sh: check the logfile /u02/applmgr/fs1/inst/apps/R1227_1227demo/logs/appl/admin/log/adnodemgrctl.txt for more information ...
    [UNEXPECTED]fs_clone has failed.
[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 2 (Fail)

#######################################################
[applmgr@1227demo ~]$ adopscanlog -latest=yes

Scanning /u02/applmgr/fs_ne/EBSapps/log/adop/17/ directory ...

/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo/TXK_SYNC_create/fsclone_apply/ohsT2PApply/CLONE2020-08-25_20-21-07_946846209.log:
--------------------------------------------------------------------------------------
Lines #(1109-1113):
----------------------------
Error Message  :1
  [PLUGIN][OHS] - ERROR - Aug 25, 2020 20:21:41 - CLONE-26009   OHS T2P failed.
  [PLUGIN][OHS] - CAUSE - Aug 25, 2020 20:21:41 - CLONE-26009   Unable to start OS component.
  [PLUGIN][OHS] - ACTION - Aug 25, 2020 20:21:41 - CLONE-26009   Check clone log and error file and ohs log file /u02/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/console~OHS~1.log for root cause.

/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_08251849.log:
-----------------------------------------------------------------------
Lines #(5897-5901):
Running /u02/applmgr/fs1/FMW_Home/webtier/perl/bin/perl  -I/u02/applmgr/fs1/FMW_Home/webtier/perl/lib/5.10.0 -I/u02/applmgr/fs1/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I/u02/applmgr/fs1/EBSapps/appl/au/12.0.0/perl -I/u02/applmgr/fs1/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u02/applmgr/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl cfgCloneInstance -ctxfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -archvloc=/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/FMW/OHS/ohsarchive.jar -mvpln=/u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/FMW/OHS/moveplan.xml -t2plogloc=/u02/applmgr/fs2/inst/apps/R1227_1227demo/admin/log/clone/ohsT2PApply -logdir=/u02/applmgr/fs2/inst/apps/R1227_1227demo/admin/log/clone -console=off -promptmsg=hide
Script Executed in 44709 milliseconds, returning status 255
ERROR: Script failed, exit code 255

START: Inside exitClone....

/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo/TXK_SYNC_create/txkADOPPreparePhaseSynchronize.log:
------------------------------------------------------------------------------------
Lines #(514-516):
TIME    : Tue Aug 25 20:21:42 2020
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /u02/applmgr/fs2/EBSapps/comn/adopclone_1227demo/bin/adclone.pl did not go through successfully.

/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/adop.log:
----------------------------------------------------------------
Lines #(115-120):
    [PROCEDURE] Calling: /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
    [EVENT]     Log: /u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo
    [UNEXPECTED]Error occurred running "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=17 -timestamp=20200825_181813 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo"
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=17 -timestamp=20200825_181813 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1227demo".
    [EVENT]     Releasing managed server ports.
        [PROCEDURE] [START 2020/08/25 20:21:43] Releasing ports
Lines #(143-147):
            [PROCEDURE] [END   2020/08/25 20:22:18] Check and Stop patch Node Manager
            [PROCEDURE] [END   2020/08/25 20:22:18] Check and stop patch Node Manager
    [UNEXPECTED]fs_clone has failed.
    [PROCEDURE] [START 2020/08/25 20:22:19] Unlocking sessions table
    [STATEMENT] Unlocking ad_adop_sessions table for 1227demo with wait interval of 60 seconds and number of tries 2
[applmgr@1227demo ~]$

[applmgr@1227demo ~]$ adopmon
Timestamp           Node name           Message Type Message Text
---------------------------------------------------------------------------------------------------------------------------
2020/08/25 20:21:41                     EVENT        Alter logon trigger: ENABLE
2020/08/25 20:21:43 1227demo            UNEXPECTED   Error occurred running "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/p
                                                     atch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applm
                                                     gr/fs2/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchco
                                                     ntextfile=/u02/applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1
                                                     227demo.xml -promptmsg=hide -console=off -mode=create -sessionid=17 -t
                                                     imestamp=20200825_181813 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/1
                                                     7/20200825_181813/fs_clone/1227demo"
2020/08/25 20:21:43 1227demo            UNEXPECTED   occurred during CLONE Patch File System from Run File System, running
                                                     command: "perl /u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/t
                                                     xkADOPPreparePhaseSynchronize.pl -contextfile=/u02/applmgr/fs2/inst/ap
                                                     ps/R1227_1227demo/appl/admin/R1227_1227demo.xml -patchcontextfile=/u02
                                                     /applmgr/fs1/inst/apps/R1227_1227demo/appl/admin/R1227_1227demo.xml -p
                                                     romptmsg=hide -console=off -mode=create -sessionid=17 -timestamp=20200
                                                     825_181813 -outdir=/u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181
                                                     813/fs_clone/1227demo".
2020/08/25 20:21:43 1227demo            EVENT        Releasing managed server ports.
2020/08/25 20:21:43 1227demo            EVENT        Log: /u02/applmgr/fs_ne/EBSapps/log/adop/17/20200825_181813/fs_clone/1
                                                     227demo/txkCloneAcquirePort.log
2020/08/25 20:21:45 1227demo            EVENT        Stopping services on patch file system.
2020/08/25 20:21:45 1227demo            EVENT        Stopping admin server.
2020/08/25 20:22:12 1227demo            EVENT        Stopping node manager.
2020/08/25 20:22:18 1227demo            UNEXPECTED   fs_clone has failed.

[karthikmca.bhc]

Hi Erman,

Would like to update you that the issue has been resolved. Please see the workaround that i followed. I remember this issue i got after i applied latest psu/cpu patches on appsTier. Below note ID's helped me to fix this issue. Moreover your instant support on troubleshooting and fixing issues in EBS is much useful and easily understandable. Thanks once again your instant support on this case.

Error:
=====
[root@1227demo ~]# vi /u02/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/console~OHS~1.log
[root@1227demo ~]# vi /u02/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log
[2020-08-22T16:56:38.2643+04:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: 1227demo.energy-esi-dubai.com] [host_addr: 192.168.1.133] [pid: 12168] [tid: 140657640527680] [user: applmgr] [VirtualHost: main]  ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

[2020-08-22T16:56:38.2649+04:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: 1227demo.energy-esi-dubai.com] [host_addr: 192.168.1.133] [pid: 12168] [tid: 140657640527680] [user: applmgr] [VirtualHost: main]  ModSecurity: APR compiled version="1.4.5"; loaded version="1.4.5"
[2020-08-22T16:56:38.2651+04:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: 1227demo.energy-esi-dubai.com] [host_addr: 192.168.1.133] [pid: 12168] [tid: 140657640527680] [user: applmgr] [VirtualHost: main]  ModSecurity: PCRE compiled version="5.0 "; loaded version="5.0 13-Sep-2004"
[2020-08-22T16:56:38.2652+04:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: 1227demo.energy-esi-dubai.com] [host_addr: 192.168.1.133] [pid: 12168] [tid: 140657640527680] [user: applmgr] [VirtualHost: main]  ModSecurity: LIBXML compiled version="2.7.2"
[2020-08-22T16:56:39.8264+04:00] [OHS] [ERROR:32] [] [core.c] [host_id: 1227demo.energy-esi-dubai.com] [host_addr: 192.168.1.133] [pid: 12168] [tid: 140657640527680] [user: applmgr] [VirtualHost: 1227demo.energy-esi-dubai.com:0]  Init: (1227demo.energy-esi-dubai.com:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 43084
[2020-08-22T16:56:39.8268+04:00] [OHS] [ERROR:32] [] [core.c] [host_id: 1227demo.energy-esi-dubai.com] [host_addr: 192.168.1.133] [pid: 12168] [tid: 140657640527680] [user: applmgr] [VirtualHost: 1227demo.energy-esi-dubai.com:0]  NZ Library Error: Unknown error


Solution:
=========
Note ID:
Oracle Security Service (OSS) Patch Removes MD5 - Steps to Evaluate and Update SSL Wallet for FMW 11.1.1.9 (Doc ID 2572809.1)
Oracle HTTP Server Fails to Start With the Error "NZ Library Error: Failed to open the wallet" (Doc ID 414175.1)
OHS Mod Proxy SSL Connection Fails With Error "OHS:2171 NZ Library Error: Invalid X509 certificate chain" (Doc ID 2246359.1)

[ErmanArslansOracleBlog]

Good for you Karthik..

So, that "unable to file - NONE" issue has been resolved and then you encountered this last "NZ Library Error: Unknown error" and followed the documented resources to solve it.
So the first error was resolved by those ad_adop_session_patches actions and adop fs_clone force.. and then you got the second error, which was  "NZ Library Error: Unknown error" and you solved it using the MOS notes.

Am I correct?
We need to conclude this because other followers may get help from this thread and from the solution that we give here..

[karthikmca.bhc]

Yes Erman. Here is the workaround which will help our followers when they get the same issue.

Issue 1:
=====
*******FATAL ERROR*******
PROGRAM : (/u02/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Sun Aug 23 18:34:00 2020
FUNCTION: TXK::IO::open [ Level 3 ]
MESSAGES:
error = Unable to open file
errorno = No such file or directory
file = </u02/applmgr/fs_ne/EBSapps/log/adop/13/20200823_181133/fs_clone/1227demo/TXK_SYNC_create/fsclone_stage/NONE

<b>Fix:
===
SQL> create table adop_bkp_5 as (select * from ad_adop_session_patches);

SQL> update ad_adop_session_patches set status='Y' where status in ('R','F','N') AND bug_number = 'CLONE' ;
SQL> commit;

-open fresh new terminal
-source run-fs env file
-run fs_clone again
$ adop phase=fs_clone force=yes workers=10

Issue 2:
=====
Unable to initialize SSL environment, nzos call nzosSetCredential returned 43084
NZ Library Error: Unknown error

Fix:
==
Oracle Security Service (OSS) Patch Removes MD5 - Steps to Evaluate and Update SSL Wallet for FMW 11.1.1.9 (Doc ID 2572809.1)
Oracle HTTP Server Fails to Start With the Error "NZ Library Error: Failed to open the wallet" (Doc ID 414175.1)
OHS Mod Proxy SSL Connection Fails With Error "OHS:2171 NZ Library Error: Invalid X509 certificate chain" (Doc ID 2246359.1)

Thanks for your continuous support!

[ErmanArslansOracleBlog]

Bless your hands :)  
That's what I was talking about.