fs_clone fails on Slave node

[Mohsin]

Hi Emran

We have a Master slave node on UAT environment. When trting to do fs_clone the same fails on the slave node as below :

[appluat@SVREBSUAT03 appl]$ adop phase=fs_clone

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
    Run Edition context  : /u01/UAT/oracle/fs1/inst/apps/UAT_svrebsuat03/appl/admin/UAT_svrebsuat03.xml
    Patch edition context: /u01/UAT/oracle/fs2/inst/apps/UAT_svrebsuat03/appl/admin/UAT_svrebsuat03.xml
    Patch file system free space: 767.84 GB

Validating system setup.
    Node registry is valid.
    Log: /u01/UAT/oracle/fs1/inst/apps/UAT_svrebsuat03/logs/appl/rgf/TXK/verifyssh.log
    Output: /u01/UAT/oracle/fs1/inst/apps/UAT_svrebsuat03/logs/appl/rgf/TXK/out.xml
    Remote execution is operational.

Checking for existing adop sessions.
    No pending session exists.
    Starting new adop session.

===========================================================================
ADOP (C.Delta.11)
Session ID: 19
Node: svrebsuat03
Phase: fs_clone
Log: /u01/UAT/oracle/fs_ne/EBSapps/log/adop/19/20190917_141951/adop.log
===========================================================================

Verifying existence of context files in database.

Checking if adop can continue with available nodes in the configuration.
    Log: /u01/UAT/oracle/fs_ne/EBSapps/log/adop/19/20190917_141951/fs_clone/svrebsuat03
        txkADOPEvalSrvStatus.pl returned SUCCESS
Skipping configuration validation on admin node: [svrebsuat03]

Validating configuration on node(s): [svrebsuat08].
    Output: /u01/UAT/oracle/fs_ne/EBSapps/log/adop/19/20190917_141951/fs_clone/validate/remote_execution_result_level2.xm        l
    Log: /u01/UAT/oracle/fs_ne/EBSapps/log/adop/19/20190917_141951/fs_clone/svrebsuat03
        txkADOPEvalSrvStatus.pl returned SUCCESS

Starting admin server on patch file system.

Running fs_clone on node(s): [svrebsuat08].
    Output: /u01/UAT/oracle/fs_ne/EBSapps/log/adop/19/20190917_141951/fs_clone/remote_execution_result_level2.xml
    [ERROR]     adop phase=fs_clone failed on Node: "svrebsuat08"
    Log: /u01/UAT/oracle/fs_ne/EBSapps/log/adop/19/20190917_141951/fs_clone/svrebsuat03
        txkADOPEvalSrvStatus.pl returned SUCCESS

Stopping services on patch file system.
    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.11

Stopping WLS Admin Server...
.Refer /u01/UAT/oracle/fs2/inst/apps/UAT_svrebsuat03/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at /u01/UAT/oracle/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile /u01/UAT/oracle/fs2/inst/apps/UAT_svrebsuat03/logs/appl/admin/log/adadminsrvctl.txt f        or more information ...

    Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12


NodeManager log is located at /u01/UAT/oracle/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 0

adnodemgrctl.sh: check the logfile /u01/UAT/oracle/fs2/inst/apps/UAT_svrebsuat03/logs/appl/admin/log/adnodemgrctl.txt for         more information ...


Summary report for current adop session:
    Node svrebsuat03:
       - Fs_clone status:   Completed successfully
    Node svrebsuat08:
       - Fs_clone status:   Failed
    For more details, run the command: adop -status -detail

adop exiting with status = 0 (Success)



the below files stats as follows:
cat /u01/UAT/oracle/fs1/inst/apps/UAT_svrebsuat08/logs/appl/rgf/txkCloneAcquirePort.txt
svrebsuat08.pitnet.com:6105:17134
svrebsuat08.pitnet.com:6205:17139
svrebsuat08.pitnet.com:10004:PORTBUSY



But netstats does not show the PORT 10004 as busy?
netstat -anp|grep 10004

[root@SVREBSUAT08 ~]# netstat -anp|grep 10004
[root@SVREBSUAT08 ~]#


Can you help us find the issue although i have bounce the services but still the same error.


Thanks and Regards
Mohsin

[ErmanArslansOracleBlog]

Please execute the following command with root and update me ->

netstat -putan | grep 10004

As you see I use the -putan argument.. for instance "u" is to list the udp sockets in addition to the others... Please update me with the findings..