fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

dbapro
Hi Arslan,

I am trying apply fs_clone on my run file system but i am having an error.
The autoconfig terminated succesfully
My appliaction is an freshinstall 12.2.4
Do you have any sugestion?

[oracle@myhost ERP]$ . EBSapps.env

  E-Business Suite Environment Information
  ----------------------------------------
  RUN File System           : /u01/oracle/apps/ERP/fs1/EBSapps/appl
  PATCH File System         : /u01/oracle/apps/ERP/fs2/EBSapps/appl
  Non-Editioned File System : /u01/oracle/apps/ERP/fs_ne


  DB Host: myhost.test.com  Service/SID: ERP


  E-Business Suite Environment Setting
  ------------------------------------
  - Enter [R/r] for sourcing Run File System Environment file, or
  - Enter [P/p] for sourcing Patch File System Environment file, or
  - Enter anything else to exit

  Please choose the environment file you wish to source [R/P]:R

  Sourcing the RUN File System ...

[oracle@myhost ERP]$ echo $FILE_EDITION
run
[oracle@myhost ERP]$ adop phase=fs_clone force=yes

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials...

Initializing...
    Run Edition context  : /u01/oracle/apps/ERP/fs1/inst/apps/ERP_myhost/appl/admin/ERP_myhost.xml
    Patch edition context: /u01/oracle/apps/ERP/fs2/inst/apps/ERP_myhost/appl/admin/ERP_myhost.xml
    Patch file system freespace: 90.92 GB

Validating system setup...
    Node registry is valid.

Checking for pending adop sessions...
    No pending session exists.
    Staging new adop session...

===========================================================================
ADOP (C.Delta.6)
Session ID: 5
Node: myhost
Phase: fs_clone
Log: /u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/adop_20150917_002111.log
===========================================================================

Validating configuration on node: [myhost]...
    Log: /u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost
        [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

Checking if there are pending cleanup actions...
    No pending cleanup actions exist.

Blocking the managed servers ports...
    Log: /u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost/txkCloneAcquirePort.log
    [UNEXPECTED]Error occurred while executing "perl /u01/oracle/apps/ERP/fs1/inst/apps/ERP_myhost/admin/install/txkCloneAcquirePort.pl  -option=listen -server=myhost.test.com -ports=5557,7002,7602,7402,7202,7802,6101,6201 -log=/u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost/txkCloneAcquirePort.log"
    [UNEXPECTED]Error 1 occurred while Blocking ports on myhost
    Stopping services on patch file system...
    Stopping admin server...

You are running adadminsrvctl.sh version 120.10.12020000.2


 The AdminServer is already shutdown

adadminsrvctl.sh: exiting with status 2

adadminsrvctl.sh: check the logfile /u01/oracle/apps/ERP/fs2/inst/apps/ERP_myhost/logs/appl/admin/log/adadminsrvctl.txt for more information ...  

    Stopping node manager...

You are running adnodemgrctl.sh version 120.11.12020000.4


Either the Node Manager is not up or the credentials are incorrect.

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile /u01/oracle/apps/ERP/fs2/inst/apps/ERP_myhost/logs/appl/admin/log/adnodemgrctl.txt for more information ...  

    [UNEXPECTED]fs_clone phase completed with errors/warnings. Please check logfiles


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 2 (Fail)
[oracle@myhost ERP]$


[oracle@myhost ERP]$ adopscanlog -latest=yes

Scanning /u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/ directory ...

/u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/adop_20150917_002111.log:
---------------------------------------------------------------------------

Lines #(84-88):
    [PROCEDURE] Calling: /u01/oracle/apps/ERP/fs1/inst/apps/ERP_myhost/admin/install/txkCloneAcquirePort.pl
    [EVENT]     Log: /u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost/txkCloneAcquirePort.log
    [UNEXPECTED]Error occurred while executing "perl /u01/oracle/apps/ERP/fs1/inst/apps/ERP_myhost/admin/install/txkCloneAcquirePort.pl  -option=listen -server=myhost.test.com -ports=5557,7002,7602,7402,7202,7802,6101,6201 -log=/u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost/txkCloneAcquirePort.log"
    [UNEXPECTED]Error 1 occurred while Blocking ports on myhost
    [EVENT]     Stopping services on patch file system...

Lines #(85-89):
    [EVENT]     Log: /u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost/txkCloneAcquirePort.log
    [UNEXPECTED]Error occurred while executing "perl /u01/oracle/apps/ERP/fs1/inst/apps/ERP_myhost/admin/install/txkCloneAcquirePort.pl  -option=listen -server=myhost.test.com -ports=5557,7002,7602,7402,7202,7802,6101,6201 -log=/u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost/txkCloneAcquirePort.log"
    [UNEXPECTED]Error 1 occurred while Blocking ports on myhost
    [EVENT]     Stopping services on patch file system...
        [EVENT]     Stopping admin server...

Lines #(97-101):
            [PROCEDURE] [END   2015/09/17 00:23:49] Check and Stop patch Node Manager
            [PROCEDURE] [END   2015/09/17 00:23:49] Check and Stop patch Node Manager
    [UNEXPECTED]fs_clone phase completed with errors/warnings. Please check logfiles
    [PROCEDURE] [START 2015/09/17 00:23:49] Unlocking sessions table
    [STATEMENT] Unlocking ad_adop_sessions table for myhost with wait interval of 60 seconds and number of tries 22.
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

dbapro
the ports are free in my system.
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

ErmanArslansOracleBlog
Administrator
send me this log file:

/u01/oracle/apps/ERP/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/ERP_myhost/txkCloneAcquirePort.log
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

ErmanArslansOracleBlog
Administrator
Analyzing the perl script...

RElated info:

sub errorOnOpen{
        my $msg = $_[0];
        logMessage("\n$msg\n");
        exitFromScript (1);
}




So, the problem is probably in listenPorts or validateLocalHost perl function used in the txkCloneAcquirePort.pl


Please send the log file, so I can see
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

ErmanArslansOracleBlog
Administrator
 ("$gConfig{$option}" eq "listen"){
                deletePIDFile();
                createPIDFile();
                listenPorts();  -> creates the socket.


Okay. Checketed deletePIDFILE and createPIDFile function, these function are not related.
The problem is in listenPorts.

Possible causes:

fork may have failed.
ports may be busy .(isAnyPortsBusy)


-->

createSocket:

  #create socket object
  $socket = new IO::Socket::INET (
        LocalHost => $server ,
        LocalPort => $port,
        Proto => 'tcp',
        Listen => 5,
        Reuse => 1
  ) ;
  if(! defined $socket){
        open (FILE, ">> $PIDFILE") or errorOnOpen("ERROR: Could not open $PIDFILE to write");
        print FILE "$server:$port:PORTBUSY\n";
        close (FILE

 if (scalar(@lines) > 0){
                foreach my $templine (@lines){
                        my @temp = split(':',$templine);
                        my $tempPort = $temp[1];
                        my $tempPid = $temp[2];

                        chomp($tempPid);
                        if ("$tempPid" eq "PORTBUSY"){
                                logMessage ("\nERROR : port $tempPort is BUSY\n");
                                $isAnyPortBusy = "true";



So, the decision is made using the txkCloneAcquirePort.txt file.
In this file, there are lines like hostname:port:pid

Please open that file and check that if there are any lines written as busy hostname:port:PORTBUSY for the ports that tried to be open.

So , if there are lines for the ports that adop tries to open / listen,

check that port using netstat -anp |grep <portnumber> and find the process holding that port.

Write me then.

Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

dbapro
Hi Arslan,

I solved the issue.
into the log:
/u01/oracle/apps/EBSUITE/fs_ne/EBSapps/log/adop/5/fs_clone_20150917_002111/EBSUITE_myhost/txkCloneAcquirePort.log

Script Name    : txkCloneAcquirePort.pl
Started        : Thu Sep 17 00:23:39 CST 2015

ERROR : Input server name does not match with system host name

Completed      : Thu Sep 17 00:23:39 CST 2015

and then I modified the file /etc/sysconfig/network
HOSTNAME=MYHOST.test.com
for
HOSTNAME=myhost.test.com
I bounced the system.

Now the phase fs_clone terminated successfully.

    [PROCEDURE] [END   2015/09/17 06:04:29] Generating AD_ZD_LOGS Report
[PROCEDURE] [END   2015/09/17 06:04:29] Executing steps for FS_CLONE phase...
[PROCEDURE] [END   2015/09/17 06:04:29] adzdoptl.pl run

adop phase=fs_clone - Completed Successfully

Thank you so much Arslan.
Regards.
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

ErmanArslansOracleBlog
Administrator
Yeah yeah.
Always analyze the logs even if the terminal output seems sufficient in some cases, as some error messages may be red herring :) When the logs are also not enough, open the script / java or whatever it is and analyze the code yourself.

In my previous update , as I analyzed the perl script and said  "the problem is probably in listenPorts or validateLocalHost perl function used in the txkCloneAcquirePort.pl "..

so there were 2 possibilities and for this the problem it was the validateLocalHost :)

sub validateLocalHost{

        my @srvArray1 = split('\.', $gConfig{$server});
        my $inputSrvHostName = $srvArray1[0];

        my $systemHost = hostname();
        my @srvArray2 = split('\.', $systemHost);
        my $systemHostName = $srvArray2[0];

        if ("$inputSrvHostName" ne "$systemHostName"){
                logMessage ("\nERROR : Input server name does not match with system host name\n");
                exitFromScript(1);
        }
}


You see exitFromScript(1) there and this function gets the hostname using the hostname() function.


standard hostname() function in perl :

The Sys::Hostname module supplies just one function, hostname, which makes up for that fact by busting its behind to try to figure out what your current host calls itself. On those systems that support the standard gethostname(2) syscall, this is used, as it's the most efficient method.

So, if you re interested , see this.

http://ermanarslan.blogspot.com.tr/2014/11/linux-setting-hostname-fqdn-or-short.html


Anyways, Good for you.
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

relmaazouz
Hi, I have the same problem.
However when executing fs_clone, everything went well and I finished the phase of cutover. I since I have been  appying hot patches. Now that I want to apply a new patch with with adop phase= prepare I have this issue [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
Please let me know if I can re-run fs_clonne with force option
B.R
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

ErmanArslansOracleBlog
Administrator
There are known issue in Oracle Support.
Such as: ADOP Errors During Prepare (Doc ID 2107688.1)

did you check those?

Send me the log for further analysis.
Raj
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

Raj
in the Web node  - do a ps -ef|grep txkCloneAcquirePort.pl  and kill any such process to fix this issue
Reply | Threaded
Open this post in threaded view
|

Re: fs_clone fail [UNEXPECTED]Error 1 occurred while Blocking ports on myhots

ErmanArslansOracleBlog
Administrator
It depends Raj.. I guess you have the same port related problem, right?
Do you see that process in netstat -anp|grep <your_port> output?