FS_CLONE ERROR in EBS 12.2.5 - [UNEXPECTED]Error occurred running "perl /home/../txkADOPValidations.pl

[karthikmca.bhc]

Hi Erman,

Getting below error when running fs_clone:


$adop phase=fs_clone force=yes

Validating configuration on node: [pieslxtst].
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/15/20171003_161154/fs_clone/validate/pieslxtst
                [ERROR]: JDK TOP (or FMW JDK TOP) is not set up correctly.
                [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
                [WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.
                [ERROR]: The value of s_ohs_installed is incorrect.
    [UNEXPECTED]Error occurred running "perl /home/applprd/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -patchctxfile=/home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -phase=fs_clone -logloc=/home/applprd/PROD/fs_ne/EBSapps/log/adop/15/20171003_161154/fs_clone/validate/pieslxtst -promptmsg=hide"
    [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on pieslxtst
 
 
[STATEMENT] Please run adopscanlog utility, using the command
 
"adopscanlog -latest=yes"
 
to get the list of the log files along with snippet of the error message corresponding to each log file.
 
 
adop exiting with status = 1 (Fail)

Please advise for a solution.

Thanks
Karthik

[ErmanArslansOracleBlog]

Take a look at this note: ADOP Errors During Prepare (Doc ID 2107688.1)

[karthik]

Hi Erman

Thanks a lot for the instant advise and support.

I have gone through the note ID that you advised and as per the note I have checked the table FND_OAM_CONTEXT_FILES which were having multiple context entries.

after removing those entries ran autoconfig on db and application tiers and then tried fs_clone and now the fs_clone is running without those errors.

Thanks a lot for your support and sharing knowledge which is helping us a lot to be much familiar with 12.2.x.

Best Regards.

[karthik]

I got the below issue as the fs_clone didn't get complete successfully and when I ran clean up that ends with below error. Please advise me for fix.

[applprd@pieslxtst scripts]$ adop -status

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.9)
Session Id: 15
Command: status
Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/15/20171004_124013/adzdshowstatus.out
===============================================================
Node Name Node Type  Phase   Status   Started Finished     Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
pieslxtst master   FS_CLONE   FAILED   2017/10/03 13:18:53     23:14:27



File System Synchronization Type: Full
*******************
[applprd@pieslxtst scripts]$ adop phase=cleanup cleanup_mode=quick workers=10

nter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
    Run Edition context  : /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml
    Patch edition context: /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml
    Patch file system free space: 98.92 GB

Validating system setup.
    Node registry is valid.
Checking for existing adop sessions.

===========================================================================
ADOP (C.Delta.9)
Session ID: 10
Node: pieslxtst
Phase: cleanup
Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/10/20171004_113353/adop.log
===========================================================================

Processing cleanup actions in parallel.
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/10/20171004_113353/cleanup/pieslxtst/log/cleanup.log

Generating report of queued DDL actions.
    Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/10/20171004_113353/cleanup/pieslxtst/adzdallddls.out

Running cleanup in QUICK mode.
Log: @ADZDSHOWLOG.sql "2017/10/04 12:21:11"

[ERROR]     [CLEANUP 1:1 ddl_id=36623] ORA-04021: timeout occurred while waiting to lock object  SQL: begin sys.ad_grants.cleanup; end;
Generating log report.
Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/10/20171004_113353/cleanup/pieslxtst/adzdshowlog.out

The cleanup phase completed successfully.
adop exiting with status = 0 (Success)


[applprd@pieslxtst scripts]$ adop -status

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.9)
Session Id: 15
Command: status
Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/15/20171004_124013/adzdshowstatus.out
===============================================================
Node Name Node Type  Phase   Status   Started Finished     Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
pieslxtst master   FS_CLONE   FAILED   2017/10/03 13:18:53     23:14:27

File System Synchronization Type: Full


Regards

[ErmanArslansOracleBlog]

Shutdown your apps services, shutdown your db, start your db and start your apps services again .
Retry after that.. (I want you to do this, because there seems to be a locking issue in the db level)

If the above actions don't solve the problem, send me the file ->"home/applprd/PROD/fs_ne/EBSapps/log/adop/10/20171004_113353/cleanup/pieslxtst/adzdshowlog.out "

If we can't find anything to be fixed, you may need to apply R12.AD.C.Delta.10 and R12.TXK.C.Delta.10 (Doc ID 2295390.1), as these Deltas solve lots of "sys.ad_grants.cleanup"-related cleanup errors.

[karthik]

Thanks Erman for your advise. I will work on your workaround and will update you the status.

Regards

[karthik]

Erman

cleanup phase completed successfully without any errors after bouncing db and application services. but adop -status still shows the failed fs_clone entry. do I need to run cleanup mode as full and then I have to run below fs_clone command?

$adop phase=fs_clone force=yes workers=10


**************************

Checking for existing adop sessions.

===========================================================================
ADOP (C.Delta.9)
Session ID: 7
Node: pieslxtst
Phase: cleanup
Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/7/20171004_145115/adop.log
===========================================================================
Processing cleanup actions in parallel.  Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/7/20171004_145115/cleanup/pieslxtst/log/cleanup.log
Generating report of queued DDL actions.  Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/7/20171004_145115/cleanup/pieslxtst/adzdallddls.out
Running cleanup in QUICK mode.
    Log: @ADZDSHOWLOG.sql "2017/10/04 14:54:22"
Generating log report.   Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/7/20171004_145115/cleanup/pieslxtst/adzdshowlog.out

The cleanup phase completed successfully.
adop exiting with status = 0 (Success)


[applprd@pieslxtst scripts]$ adop -status
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.9)
Session Id: 15
Command: status
Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/15/20171004_145753/adzdshowstatus.out
===============================================================
Node Name Node Type  Phase   Status   Started Finished     Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
pieslxtst master   FS_CLONE   FAILED   2017/10/03 13:18:53     25:99:06

File System Synchronization Type: Full
adop exiting with status = 0 (Success)


Regards

[ErmanArslansOracleBlog]

Good. Your fsclone problem solved.
"adop -status" is just looking at the old info and showing the old info.
run adop prepare phase and then check the adop -status and see if it is showing the updated info.

[karthik]

Erman,

Prepare phase failed:

[applprd@pieslxtst pieslxtst]$ adop -status

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.9)
Session Id: 16
Command: status
Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_152324/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
pieslxtst       master     PREPARE         RUNNING         2017/10/04 15:21:00                       0:02:30
                           APPLY           NOT STARTED
                           FINALIZE        NOT STARTED
                           CUTOVER         NOT STARTED
                           CLEANUP         NOT STARTED

File System Synchronization Type: Fulll
adop exiting with status = 0 (Success)

***************

[applprd@pieslxtst scripts]$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials.
Warning: WebLogic Admin Server is not running.
Trying to start Admin Server.
You are running adadminsrvctl.sh version 120.10.12020000.10
Starting WLS Admin Server...
Refer /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/logs/appl/admin/log/adadminsrvctl.txt for details
AdminServer logs are located at /home/applprd/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs
adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/logs/appl/admin/log/adadminsrvctl.txt for more information ...  
Initializing.
    Run Edition context  : /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml
    Patch edition context: /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml
    Patch file system free space: 98.90 GB

Checking for existing adop sessions.
    No pending session exists.
    Starting new adop session.
=================================================================
ADOP (C.Delta.9)
Session ID: 16
Node: pieslxtst
Phase: prepare
Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/adop.log
=================================================================
Validating configuration on node: [pieslxtst].
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/validate/pieslxtst
        [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
        [WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.
        [WARNING]: Found invalid cross references in FS config files.
        [WARNING]: Found invalid cross references in OHS config files.
        [WARNING]: Found invalid cross references in node manager and domain config files.

Verifying data dictionary.
Validating system is ready to prepare.
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst

Detecting configuration changes.
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst/adConfigChanges.log
The log file is /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst/adConfigChanges.log

/home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/cutover directory not found, so configuration changes cannot be detected.
The directory may have been deleted intentionally or accidentally. Or, if this is the first adop cycle, the directory will only be created after cutover.

Without this directory, the configuration changes will be propagated to the patch file system forcefully.

Configuration change detected.
    File system configuration changes were detected.

Detecting ad/txk codelevel bumpup.
    ad/txk codelevels are bumped.
    Checking for CLONE record existence for pieslxtst
    CLONE record already exist for pieslxtst
Checking for pending cleanup actions.
    No pending cleanup actions found.
Validating database is ready to prepare.
Generating report for key tablespaces.
Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst/adzdshowts.out
Starting Apps listener.
adalnctl.sh version 120.3.12020000.4
Checking for FNDFS executable.
Starting listener process APPS_PROD.
adalnctl.sh: exiting with status 0
adalnctl.sh: check the logfile /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/logs/appl/admin/log/adalnctl.txt for more information ...  
Submitting ADZDPATCH concurrent program.
Saving database logs from previous session.
Output: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst/adzdshowlog_cleanup.out
Creating database patch edition.
Log: @ADZDSHOWLOG.sql "2017/10/04 15:21:22"
Running steps for CLONE.
    Checking for pending cleanup actions.
    No pending cleanup actions found.
    Blocking managed server ports.
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst/txkCloneAcquirePort.log
Performing CLONE steps.
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst

Beginning application tier FSCloneApply - Wed Oct  4 15:25:25 2017

/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/FMW/t2pjdk/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/xmlparserv2.jar:/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/ojdbc6.jar:/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/java:/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/oui/OraInstaller.jar:/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/oui/ewt3.jar:/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/oui/share.jar:/home/applprd/PROD/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/ojmisc.jar:/home/applprd/PROD/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneApplyAppsTier -e /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -targ /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -stage /home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst -tmp /tmp -nodbac -component appsTier -nopromptmsg
Log file located at /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/admin/log/clone/FSCloneApplyAppsTier_10041525.log

Enabling EBS_LOGON trigger

ERROR while running FSCloneApply...
Wed Oct  4 15:25:53 2017
*******FATAL ERROR*******
PROGRAM : (/home/applprd/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Wed Oct  4 15:25:54 2017
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/bin/adclone.pl did not go through successfully.

    [UNEXPECTED]Error occurred running "perl /home/applprd/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -patchcontextfile=/home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=16 -timestamp=20171004_151529 -outdir=/home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst"
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /home/applprd/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -patchcontextfile=/home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=16 -timestamp=20171004_151529 -outdir=/home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst".
    Releasing managed server ports.
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/16/20171004_151529/prepare/pieslxtst/txkCloneAcquirePort.log
    [UNEXPECTED]fs_clone has failed.
    [UNEXPECTED]Error calling runPendingClone subroutine.
    Stopping services on patch file system.
    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.10

 The AdminServer is already shutdown

AdminServer logs are located at /home/applprd/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 2

adadminsrvctl.sh: check the logfile /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/logs/appl/admin/log/adadminsrvctl.txt for more information ...  

    Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12

The Node Manager is already shutdown

NodeManager log is located at /home/applprd/PROD/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/logs/appl/admin/log/adnodemgrctl.txt for more information ...  
   
[ERROR]     Prepare phase has failed.

[STATEMENT] Please run adopscanlog utility, using the command
"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 1 (Fail)

Regards.

[karthik]

[applprd@pieslxtst pieslxtst]$ vi /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/admin/log/clone/FSCloneApplyAppsTier_10041525.log

+++++++++++++++++++++++++
configProperty id = Server4
Count for NodeIterator nextNode = 3
Input for oafm_server1 Listen Address OR Port is NULL
        Updating oafm_server1 Listen Address to pieslxtst.ad.adco.net
        Updating oafm_server1 Listen Port to 7602
+++++++++++++++++++++++++
configProperty id = Server5
Count for NodeIterator nextNode = 3
Input for forms-c4ws_server1 Listen Address OR Port is NULL
        Updating forms-c4ws_server1 Listen Address to pieslxtst.ad.adco.net
        Updating forms-c4ws_server1 Listen Port to 7802
+++++++++++++++++++++++++
 configGroup Type name = MACHINE_CONFIG
configProperty id = Machine1
Count for NodeIterator nextNode = 3
ERROR: Machine configurations are not in sync between file system context and DB context
ERROR: Update Moveplan Fail

START: Inside exitClone....
Updating status INCOMPLETE for ApplywlsTechStack

START: Updating status INCOMPLETE for action ApplywlsTechStack

END: Updated status INCOMPLETE for action ApplywlsTechStack

[ErmanArslansOracleBlog]

Follow ->EBS R12.2: FS_CLONE Failing With ERROR: Update Moveplan Fail (Doc ID 2296036.1)

[karthik]

Thanks Erman for the guidance.

Referred the metalink 2296036.1 and found some of old hostname entries presented in the movplan.xml file. I have updated those entries and moving forward to fs_clone run. Will update you the status shortly.

Best regards

[karthik]

Erman

Again it got failed with the same error,

+++++++++++++++++++++++++
configProperty id = Server3
Count for NodeIterator nextNode = 3
Input for forms_server1 Listen Address OR Port is NULL
        Updating forms_server1 Listen Address to pieslxtst.ad.adco.net
        Updating forms_server1 Listen Port to 7402
+++++++++++++++++++++++++
configProperty id = Server4
Count for NodeIterator nextNode = 3
Input for oafm_server1 Listen Address OR Port is NULL
        Updating oafm_server1 Listen Address to pieslxtst.ad.adco.net
        Updating oafm_server1 Listen Port to 7602
+++++++++++++++++++++++++
configProperty id = Server5
Count for NodeIterator nextNode = 3
Input for forms-c4ws_server1 Listen Address OR Port is NULL
        Updating forms-c4ws_server1 Listen Address to pieslxtst.ad.adco.net
        Updating forms-c4ws_server1 Listen Port to 7802
+++++++++++++++++++++++++
 configGroup Type name = MACHINE_CONFIG
configProperty id = Machine1
Count for NodeIterator nextNode = 3
ERROR: Machine configurations are not in sync between file system context and DB context
ERROR: Update Moveplan Fail

START: Inside exitClone....
Updating status INCOMPLETE for ApplywlsTechStack

START: Updating status INCOMPLETE for action ApplywlsTechStack

END: Updated status INCOMPLETE for action ApplywlsTechStack

****************
Enabling EBS_LOGON trigger

ERROR while running FSCloneApply...
Wed Oct  4 20:42:13 2017
*******FATAL ERROR*******
PROGRAM : (/home/applprd/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Wed Oct  4 20:42:14 2017
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /home/applprd/PROD/fs1/EBSapps/comn/adopclone_pieslxtst/bin/adclone.pl did not go through successfully.

    [UNEXPECTED]Error occurred running "perl /home/applprd/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -patchcontextfile=/home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=17 -timestamp=20171004_180052 -outdir=/home/applprd/PROD/fs_ne/EBSapps/log/adop/17/20171004_180052/fs_clone/pieslxtst"
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /home/applprd/PROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -patchcontextfile=/home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=17 -timestamp=20171004_180052 -outdir=/home/applprd/PROD/fs_ne/EBSapps/log/adop/17/20171004_180052/fs_clone/pieslxtst".
    Releasing managed server ports.
    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/17/20171004_180052/fs_clone/pieslxtst/txkCloneAcquirePort.log
    Stopping services on patch file system.
    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.10


 The AdminServer is already shutdown

AdminServer logs are located at /home/applprd/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 2

adadminsrvctl.sh: check the logfile /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/logs/appl/admin/log/adadminsrvctl.txt for more information ...  

    Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12


The Node Manager is already shutdown

NodeManager log is located at /home/applprd/PROD/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/logs/appl/admin/log/adnodemgrctl.txt for more information ...  

    [UNEXPECTED]fs_clone has failed.


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)

[ErmanArslansOracleBlog]

What does the log of txkADOPPreparePhaseSynchronize.pl say?

[Mani]

/u32/oracle/NIBDEV/fs2/EBSapps/comn/adopclone_nibddbs02/bin/adclone.pl did not go through successfully.
LOG DIRECTORY: /u32/oracle/NIBDEV/fs_ne/EBSapps/log/adop/32/prepare_20190407_224210/NIBDEV_nibddbs02/TXK_SYNC_create/fsclone_apply.
*******FATAL ERROR*******
PROGRAM : (/u32/oracle/NIBDEV/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Sun Apr  7 22:57:40 2019
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /u32/oracle/NIBDEV/fs2/EBSapps/comn/adopclone_nibddbs02/bin/adclone.pl did not go through successfully.
~

[ErmanArslansOracleBlog]

These log lines are so generic.

I need to review all the adop logs..
If it is okay for your security policy, please send me the all the files under ->

"/home/applprd/PROD/fs_ne/EBSapps/log/adop/17/20171004_180052/fs_clone/pieslxts"

[Akash Dalmia]

Were you able to fix this problem? I am facing a similar issue and I have followed all the steps in this thread.

[ErmanArslansOracleBlog]

We need the logs, as I already stated in my previous update.
Mani didn't send the logs, so we could not proceed.