adop phase=cutover failed :: [UNEXPECTED]Error occurred running "perl txkADOPCutOverPhaseCtrlScript.pl"

===========================================================================
ADOP (C.Delta.10)
Session ID: 11
Node: sun3app2
Phase: cutover
Log: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/adop.log
===========================================================================

Verifying existence of context files in database.

Checking for failed nodes in the configuration.

Checking if adop can continue with available nodes in the configuration.
    Log: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/cutover/sun3app2
        txkADOPEvalSrvStatus.pl returned SUCCESS

Validating configuration on admin node: [sun3app2].
    Output: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/cutover/validate/remote_execution_result_level1.xml
    Log: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/cutover/sun3app2
        txkADOPEvalSrvStatus.pl returned SUCCESS

Validating configuration on node(s): [sun3app1].
    Output: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/cutover/validate/remote_execution_result_level2.xml
    Log: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/cutover/sun3app2
        txkADOPEvalSrvStatus.pl returned SUCCESS

Creating list of nodes where cutover needs to be run.
    The cutover phase needs to be run on node: sun3app1
    The cutover phase needs to be run on node: sun3app2

Running cutover phase on node(s): [sun3app1 and sun3app2].
Broadcast Message from appluat (???) on sun3app2 Sat Jan 13 14:46:34...
UAT_BALANCE environment has changed.
All users must re-source the environment.
    Output: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/cutover/remote_execution_result_level1.xml
    [ERROR]     adop phase=cutover failed on Node: "sun3app1"
    [ERROR]     adop phase=cutover failed on Node: "sun3app2"
    [UNEXPECTED]adop phase=cutover on all the nodes returned failure.
    [UNEXPECTED]Unable to continue processing.
    [UNEXPECTED]Error running "adop phase=cutover" on node(s): sun3app1 and sun3app2.

Summary report for current adop session:
    Node sun3app2: Failed
       - Cutover status:    ADMIN_STARTUP_COMPLETED
    Node sun3app1: Failed
       - Cutover status:    FORCE_STARTUP_COMPLETED
    For more details, run the command: adop -status -detail


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"


Thanks,
Prafull

Hi Erman,

I have 2 node of database and 2 node of application as our EBS architecture.
I have successfully completed the PREPARE | APPLY | FINALIZE phase , but in CUTOVER i faced above error.

But there are below things which needs consideration
>> file system switch over happened and if I try to resource environment it goes with new run file system
>> i have checked patch from ad_bugs table it shows that patch has been applied
>> also we have retested the issue for which we have applied the patch, it also has been resolved.

From above, one thing is clear that patch has been applied successfully. Now coming back to error, I think it got failed on startup of admin services. Then , I tried to start the application tier services manually and it was successful.

But for adop -status -details am getting cutover status failed

appluat@sun3app2:~$ adop -status -details

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.10)
Session Id: 11
Command: status
Output: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180114_123704/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
sun3app2        master     PREPARE         COMPLETED       2018/01/11 14:35:42  2018/01/11 17:18:02  2:42:20
                           APPLY           COMPLETED       2018/01/13 13:00:11  2018/01/13 13:09:50  0:09:39
                           FINALIZE        COMPLETED       2018/01/13 13:24:57  2018/01/13 13:44:08  0:19:11
                           CUTOVER         FAILED          2018/01/13 13:54:51  2018/01/13 14:53:53  0:59:02
                           CLEANUP         NOT STARTED

sun3app1        slave      PREPARE         COMPLETED       2018/01/11 14:35:38  2018/01/13 12:23:27  45:47:49
                           APPLY           COMPLETED       2018/01/13 13:11:54  2018/01/13 13:16:12  0:04:18
                           FINALIZE        COMPLETED       2018/01/13 13:24:57  2018/01/13 13:44:08  0:19:11
                           CUTOVER         FAILED          2018/01/13 13:55:00  2018/01/13 14:54:38  0:59:38
                           CLEANUP         NOT STARTED



File System Synchronization Type: Light




Generating full adop status report: /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180114_123704/adzdshowstatus.out

adop exiting with status = 0 (Success)
appluat@sun3app2:~$


and

-bash-4.1$ adopscanlog -latest=yes

Scanning /u01/oracle/uat/fs_ne/EBSapps/log/adop/11/ directory ...

/u01/oracle/uat/fs_ne/EBSapps/log/adop/11/20180113_134644/cutover/sun3app1/adop.                                                                             log:
--------------------------------------------------------------------------------                                                                             ----

Lines #(609-616):
    Starting Middle Tier Services did not complete on node(s): <sun3app2>
    Will wait for another minute and check again.
    [UNEXPECTED]Detected a failed session on another node(s): sun3app2 and
    [UNEXPECTED]This session will also fail.
    [UNEXPECTED]Middle tier services did not start on all nodes. Refer to log fi                                                                             les for details.
    [UNEXPECTED]Cutover phase has failed.
    [PROCEDURE] [START 2018/01/13 14:54:37] Updating session timestamps
    [STATEMENT] CUTOVER Phase END TIME: 13-01-2018 14:54:38
-bash-4.1$





Can you please advise what action should I take in this scenario ? Should I run cutover again ?


Thanks,
Prafull
adop_cutover_issue.rar

[ErmanArslansOracleBlog]

This is only for internal research, don't respond.

---------------------------------
Key info: file system switch over happened and if I try to resource environment it goes with new run file system

[ErmanArslansOracleBlog]

Check the log file of the problematic components that could not be started during your last cutover..
For ex: Nodemanager, logfile: /u01/oracle/uat/fs1/inst/apps/UAT_sun3app2/logs/appl/admin/log/adnodemgrctl.txt

After checking the errors and taking the corrective actions; do the following to be in a clean state..


-Please create a new online patching cycle by runing adop prepare phase..
-Reapply the patch
-reexecute the cutover (this time ensure it is not failed) and update me.

Hi Erman,

Thanks for your inputs.

After 2 days of research and review all the logs, I can to conclusion and able to resolve the issue
followed below steps along with situational considirations

>> though cutover failed, before cutover failed, the file system switch happend
>> then i login with new putty session
>> resourced env file (this brought me to newly switched file system, but this can be ignore as it has been switch in last cutover run and cutover failed at the middle tier services start)
>> rerun adop phase=cutover
>> completed successfully on both nodes
>> then adop phase=cleanup , completed successfully on both nodes.


Thanks,
Prafull

[Venkat]

Thanks Praful. That helped for me also. I had just migrated existing single node application node to 2 node load balanced. And this was the first adop cycle I had started which failed in the cutover. Your post helped me to complete the cutover and cleanup on both the nodes successfully.