Fs_clone failed in my production instance

[syed]

Hi Emran,
we have EBS having db version 11.2.0.3 and R12.2.4 application last week i am apply about 15 patches in my production instance at that time I ran fs?_clone and it takes 40 minutes to complete and do the remaining adop cycle like prepare,finalize,cutover ,and cleanup .
Now after that when we run fs_clone to apply more patches then it hangs it create the dir in patch fs like EBSapps only in 636 MB and FMW around 5.9 GB and inst around 2.3 GB while running the fs_clone my vm machine is rebooting so after that we run fs_clone with force option but it failed from last 4 days i am facing the issues
adopscanlog -latest=yes

Scanning /app/apps/fs_ne/EBSapps/log/adop/79/ directory ...

/app/apps/fs_ne/EBSapps/log/adop/79/adop_20180502_004546.log:
-------------------------------------------------------------

Lines #(103-107):
    [PROCEDURE] Calling: /app/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
    [EVENT]     Log: /app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp
    [UNEXPECTED]Error occurred while executing "perl /app/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/app/apps/fs1/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -patchcontextfile=/app/apps/fs2/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -promptmsg=hide -console=off -mode=create -sessionid=79 -timestamp=20180502_004546 -outdir=/app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp"
    [UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: "perl /app/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/app/apps/fs1/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -patchcontextfile=/app/apps/fs2/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -promptmsg=hide -console=off -mode=create -sessionid=79 -timestamp=20180502_004546 -outdir=/app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp".
    [EVENT]     Releasing the managed servers ports...

Lines #(104-108):
    [EVENT]     Log: /app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp
    [UNEXPECTED]Error occurred while executing "perl /app/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/app/apps/fs1/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -patchcontextfile=/app/apps/fs2/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -promptmsg=hide -console=off -mode=create -sessionid=79 -timestamp=20180502_004546 -outdir=/app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp"
    [UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: "perl /app/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/app/apps/fs1/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -patchcontextfile=/app/apps/fs2/inst/apps/NODPROD_noidaapp/appl/admin/NODPROD_noidaapp.xml -promptmsg=hide -console=off -mode=create -sessionid=79 -timestamp=20180502_004546 -outdir=/app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp".
    [EVENT]     Releasing the managed servers ports...
        [PROCEDURE] [START 2018/05/02 05:33:16] Releasing the ports

Lines #(124-128):
            [PROCEDURE] [START 2018/05/02 05:33:17] Check and Stop patch Admin Server
            [PROCEDURE] [START 2018/05/02 05:33:17] Check and Stop patch FS Admin Server
            [UNEXPECTED]Error occurred while executing "sh /app/apps/fs2/inst/apps/NODPROD_noidaapp/admin/scripts/adadminsrvctl.sh  stop -nopromptmsg"
            [UNEXPECTED]Error executing adadminsrvctl
        [PROCEDURE] [START 2018/05/02 05:33:18] Unlocking sessions table

Lines #(125-129):
            [PROCEDURE] [START 2018/05/02 05:33:17] Check and Stop patch FS Admin Server
            [UNEXPECTED]Error occurred while executing "sh /app/apps/fs2/inst/apps/NODPROD_noidaapp/admin/scripts/adadminsrvctl.sh  stop -nopromptmsg"
            [UNEXPECTED]Error executing adadminsrvctl
        [PROCEDURE] [START 2018/05/02 05:33:18] Unlocking sessions table
        [STATEMENT] Unlocking ad_adop_sessions table for noidaapp with wait interval of 60 seconds and number of tries 2

/app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_05020333.log:
----------------------------------------------------------------------------------------------------------------------------------------------

Lines #(811-815):
Running /app/apps/fs1/EBSapps/comn/adopclone_noidaapp/FMW/pasteBinary.sh -javaHome /app/apps/fs2/EBSapps/comn/util/jdk64 -al /app/apps/fs1/EBSapps/comn/adopclone_noidaapp/FMW/FMW_Home.jar -tl /app/apps/fs2/FMW_Home -invPtrLoc /etc/oraInst.loc -ldl /app/apps/fs1/inst/apps/NODPROD_noidaapp/admin/log/clone/fmwT2PApply -silent true -debug true -executeSysPrereqs false
Script Executed in 7200059 milliseconds, returning status -1
ERROR: Script timed out.

START: Inside exitClone....

/app/apps/fs_ne/EBSapps/log/adop/79/fs_clone_20180502_004546/NODPROD_noidaapp/TXK_SYNC_create/txkADOPPreparePhaseSynchronize.log:
---------------------------------------------------------------------------------------------------------------------------------

Lines #(462-464):
TIME    : Wed May  2 05:33:16 2018
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /app/apps/fs1/EBSapps/comn/adopclone_noidaapp/bin/adclone.pl did not go through successfully.

[syed]

Hi Emran ,
Now i am running the the fs_clone with force option
]$ export TIMEDPROCESS_TIMEOUT=-1
]$ export CONFIG_JVM_ARGS="-Xms2048m -Xmx2048m"
]$ adop phase=fs_clone force=yes
END: FSCloneStage - AppsTier Finished at Wed May 02 09:29:29 IST 2018
Status: Completed Successfully
Total Time Taken : 8.164 mins
#############################################################



END: Calling StageAppsTier...
It completed the stage within 8 to 9 minutes and after that cursor will blink after after 5 to 6 hours it failed with error as above

[ErmanArslansOracleBlog]

1. Open a new terminal.

2. Execute:
      export TIMEDPROCESS_TIMEOUT=-1

3. Execute:
      adop phase=fs_clone

Note that, fs_clone may take some time depending on the server resources (CPU and RAM).

[ErmanArslansOracleBlog]

Aha.. You already did this.

You shouldn't get timeout error after these actions..

You must be getting something different.

So send me the error you getting after these actions..

[ErmanArslansOracleBlog]

Also, just in case -> in addition to the actions above, you can run;

export CONFIG_JVM_ARGS="-Xms1500m -Xmx3000m"    before running adop fs_clone.

Anyways, send me the error you getting after these actions.

[syed]

Hi Emran , Now fs_clone completed successfiully thanks for your help but can we find out why it take more than 9 hous to complete adzdshowstatus_latest.out

[syed]

Hi Emran,
can  you find why fa_clone take 9 hours to complete in my production instance and in my clone instance it takes only 42 minutes to complete the fs_Clone

[ErmanArslansOracleBlog]

The problem you are asking about it very generic.
You need to investigate, where the fs_clone operation loses its time..

Only after that, I can make a proper comment..

There are issues on this. But they are very specific . For example, check this one ->

DOP FS_CLONE Is Taking Long Hours To Complete Or Hangs (Doc ID 2189964.1)

[syed]

vi txkCloneAcquirePort.log

PID listening port 7812 is 1686286

PID listening port 6111 is 1686287

PID listening port 6211 is 1686293

Completed      : Fri May  4 15:10:39 IST 2018

Script Name    : txkCloneAcquirePort.pl
Started        : Fri May  4 22:44:08 IST 2018

Attempting to close socket on port 5567

WARNING: PID 1686112 does not exist, either the port 5567 might have been already released or the PID might have been killed

Attempting to close socket on port 7012

WARNING: PID 1686230 does not exist, either the port 7012 might have been already released or the PID might have been killed

Attempting to close socket on port 7612

WARNING: PID 1686273 does not exist, either the port 7612 might have been already released or the PID might have been killed

Attempting to close socket on port 7412

WARNING: PID 1686274 does not exist, either the port 7412 might have been already released or the PID might have been killed

Attempting to close socket on port 7212

WARNING: PID 1686282 does not exist, either the port 7212 might have been already released or the PID might have been killed

Attempting to close socket on port 7812

WARNING: PID 1686286 does not exist, either the port 7812 might have been already released or the PID might have been killed

Attempting to close socket on port 6111

WARNING: PID 1686287 does not exist, either the port 6111 might have been already released or the PID might have been killed

Attempting to close socket on port 6211

WARNING: PID 1686293 does not exist, either the port 6211 might have been already released or the PID might have been killed

Completed      : Fri May  4 22:44:08 IST 2018

"txkCloneAcquirePort.log" 76L, 2247C                        

[ErmanArslansOracleBlog]

What is this?

I said -> you need to investigate, where the fs_clone operation loses its time..

[syed]

Hi Emran,
If you see txkCloneAcquirePort.log it completed Completed      : Fri May  4 15:10:39 IST 2018

and second script Script Name    : txkCloneAcquirePort.pl
Started        : Fri May  4 22:44:08 IST 2018
it is started at 22:44:08
it shows some warning for pids so is this issue

[ErmanArslansOracleBlog]

Those are just warnings.
It shows warning, but it completed at 22:44:08 .
So it started at 22:44:08 and completed very quickly.

So that's not the cause.
You need to find the cause.. The script that takes long time to complete.
Check your fsclone related logfiles, accordingly..

[Roja]

Hi,

I am facing the same error messages with fs_clone since 1 week.

How did you resolved it? Please help me.

Regards,
Roja

[syed]

After rebooting the application server it resolved

On Mon 24 Dec, 2018, 10:53 PM Roja [via Erman Arslan's Oracle Forum] <[hidden email] wrote:

Hi,

I am facing the same error messages with fs_clone since 1 week.

How did you resolved it? Please help me.

Regards,
Roja

---

If you reply to this email, your message will be added to the discussion below:

http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/Fs-clone-failed-in-my-production-instance-tp5842p7020.html

To unsubscribe from Fs_clone failed in my production instance, click here.
NAML