adop hotpatch

[satish]

CONTENTS DELETED

The author has deleted this message.

[satish]

CONTENTS DELETED

The author has deleted this message.

[satish]

Dear Erman,

Adop -status was not showing the hot patch details

[applprod@erpnode2 ~]$ adop -status -detail


Enter the APPS password:
Connected.

==============================================================
ADOP (C.Delta.7)
Session Id: 17
Command: status
Output: /u01/ROOKAPPS/fs_ne/EBSapps/log/status_20180415_164136/adzdshowstatus.out
===============================================================

Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
erpnode1 master PREPARE COMPLETED 2017/12/06 15:29:13 2017/12/06 16:08:36 0:39:23
APPLY COMPLETED 2017/12/06 16:18:16 2017/12/06 16:45:18 0:27:02
FINALIZE COMPLETED 2017/12/06 16:35:41 2017/12/06 16:47:12 0:11:31
CUTOVER COMPLETED 2017/12/07 06:35:23 2017/12/07 06:51:58 0:16:35
CLEANUP COMPLETED 2017/12/07 06:57:36 2017/12/07 06:58:16 0:00:40

erpnode2 slave PREPARE COMPLETED 2017/12/06 15:29:11 2017/12/06 16:10:29 0:41:18
APPLY COMPLETED 2017/12/06 16:21:31 2017/12/06 16:45:49 0:24:18
FINALIZE COMPLETED 2017/12/06 16:35:41 2017/12/06 16:47:12 0:11:31
CUTOVER COMPLETED 2017/12/07 06:35:23 2017/12/07 06:52:03 0:16:40
CLEANUP COMPLETED 2017/12/07 06:57:36 2017/12/07 06:58:16 0:00:40



File System Synchronization Type: Full




Generating full adop status report: /u01/ROOKAPPS/fs_ne/EBSapps/log/status_20180415_164136/adzdshowstatus.out

adop exiting with status = 0 (Success)
[applprod@erpnode2 ~]$

[satish]

we really need your suggestion/advice in this situation

[ErmanArslansOracleBlog]

Aha.. it seems, adop thinks that patch is already applied, that's why tells you use forceapply.

1)What is the adop command that you are using? (the full command?)
2)Is it a shared or non-shared application fs env?

[satish]

Thanks a lot for the update erman.

1)We are using shared appltop
2)Here is the log

From primrary node
==================

[applprod@node1 23021248]$ adop phase=apply patches=23021248 hotpatch=yes

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Validating credentials.

Initializing.
    Run Edition context  : /u02/PRODINST/fs2/inst/apps/PRODDB_node1/appl/admin/PRODDB_node1.xml
    Patch edition context: /u02/PRODINST/fs1/inst/apps/PRODDB_node1/appl/admin/PRODDB_node1.xmlw
eReading driver file (up to 50000000 bytes).
    Patch file system free space: 35.63 GB

Validating system setup.                                                                               Node registry is valid.
    Log: /u02/PRODINST/fs2/inst/apps/PRODDB_node1/logs/appl/rgf/TXK/verifyssh.log
    Output: /u02/PRODINST/fs2/inst/apps/PRODDB_node1/logs/appl/rgf/TXK/out.xml
    Remote execution is operational.

Checking for existing adop sessions.
    Application tier services are down.
    Continuing with the existing session [Session ID: 11].

===========================================================================
ADOP (C.Delta.7)
Session ID: 11
Node: node1
Phase: apply
Log: /u01/ROOKAPPS/fs_ne/EBSapps/log/adop/11/adop_20180415_103926.log
===========================================================================

Verifying existence of context files in database.

Checking for failed nodes in the configuration.

Checking if adop can continue with available nodes in the configuration.
    Log: /u01/ROOKAPPS/fs_ne/EBSapps/log/adop/11/apply_20180415_103926/PRODDB_node1
Returning SUCCESS


Applying <23021248> patch(es) on admin node: [node1].
    Output: /u01/ROOKAPPS/fs_ne/EBSapps/log/adop/11/apply_20180415_103926/PRODDB_node1/remote_execution_result_20180415_104030.xml
    Log: /u01/ROOKAPPS/fs_ne/EBSapps/log/adop/11/apply_20180415_103926/PRODDB_node1
Returning SUCCESS


Applying <23021248> patch(es) on node(s): [node2].
    Output: /u01/ROOKAPPS/fs_ne/EBSapps/log/adop/11/apply_20180415_103926/PRODDB_node1/remote_execution_result_20180415_104302.xml
    [ERROR]     adop phase=apply failed on Node: "node2"
    Log: /u01/ROOKAPPS/fs_ne/EBSapps/log/adop/11/apply_20180415_103926/PRODDB_node1
Returning SUCCESS


Summary report for current adop session:
    Node node1: COMPLETED
       - Apply status:      COMPLETED
    Node node2: FAILED
       - Apply status:      NOT STARTED
       [ERROR]      You must specify options=forceapply, to apply the patch <23021248> to
                    master node before it can be applied to a slave node.
    For more details, run the command: adop -status -detail

adop exiting with status = 0 (Success)

[satish]

CONTENTS DELETED

The author has deleted this message.

[satish]

Dear erman,

Patch was not getting started on node2,not sure why it was so.

Thank you for all the support

[ErmanArslansOracleBlog]

What are the outputs of the following queries?

select ADOP_SESSION_ID,PREPARE_STATUS,APPLY_STATUS,FINALIZE_STATUS,CUTOVER_STATUS,CLEANUP_STATUS,ABORT_STATUS,STATUS,ABANDON_FLAG,NODE_NAME from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;

select ADOP_SESSION_ID,BUG_NUMBER,CLONE_STATUS,STATUS,NODE_NAME from AD_ADOP_SESSION_PATCHES order by ADOP_SESSION_ID;

Also you can try : forceapply option in conjunction with the hotpatch option..

example: adop phase=apply patches=17204589 hotpatch=yes options=forceapply

[satish]

CONTENTS DELETED

The author has deleted this message.

[ErmanArslansOracleBlog]

What are the outputs of the following queries?
Do you see EXPLICIT = applied there?

1)
select ad_patch.is_patch_applied('R12',-1,23021248) from dual;

2)
Example sql using the APPL_TOP ID for a specific node in a multinode environment (1045 is the APPL_TOP ID -- check yours and update below query accordingly):

SELECT adb.bug_number,ad_patch.is_patch_applied('R12', 1045, adb.bug_number)
FROM ad_bugs adb
WHERE adb.bug_number in (1,23021248)

[Satish]

Dear erman,

Thanks for the update. We could see the query returns explicit.It is also returned in logs

STATEMENT] [START 2018/04/15 10:43:27] Check if patch has been applied to master node
[STATEMENT] Release : 12.0.0
[STATEMENT] Appltop Id : 1196
[STATEMENT] Bug number : 23021248
[STATEMENT] Language : US
[STATEMENT] Is patch applied on master:EXPLICIT
[STATEMENT] [END 2018/04/15 10:43:27] Check if patch has been applied to master node

Thanks again for your valuable updates

[ErmanArslansOracleBlog]

Okay.. At this point, we may look at the codes..
As far as I can see, adop goes into the adzdoptl.pl.. So we should investigate it from there.. But that perl, may also using other perls or java codes.. So invesigating it may be a little long and complex process.
Currently I don't have an EBS 12.2 to play with.  But, I m wondering, in what conditions it goes to the following ->

[ERROR] You must specify options=forceapply, to apply the patch <23021248> to master node before it can be applied to a slave node.

This seems like a bug..
You are in C.Delta.7

What Oracle says is :

If a hotpatch session fails for an approved patch (NLS, Online Help, or documented in the patch readme) you should:

1. Investigate the error and try to complete the hotpatch session by re-applying the hotpatch with restart=yes  (You can try this restart=yes argument)

or

2. Apply a replacement patch with abandon=yes.
It should be possible to just abandon the hotpatch session, and then start a new ADOP online patching cycle (Make sure you are on AD/TXK Delta 6 or higher).

[Satish]

Tried but didn't work.

[ErmanArslansOracleBlog]

which one?

17 Nis 2018 Sal 18:44 tarihinde Satish [via Erman Arslan's Oracle Forum] <[hidden email]> şunu yazdı:

Tried but didn't work.

---

If you reply to this email, your message will be added to the discussion below:

http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/adop-hotpatch-tp5667p5698.html

To start a new topic under EBS 12.2, email [hidden email]
To unsubscribe from Erman Arslan's Oracle Forum, click here.
NAML

[satish]

Dear Erman,

Thanks for the update.I have tried using below option but the same error repeats.I think some inconsistency with adop tables

1. Investigate the error and try to complete the hotpatch session by re-applying the hotpatch with restart=yes  (You can try this restart=yes argument)

[ErmanArslansOracleBlog]

What about the other options?

1)2. Apply a replacement patch with abandon=yes.
It should be possible to just abandon the hotpatch session, and then start a new ADOP online patching cycle (Make sure you are on AD/TXK Delta 6 or higher).

2)As far as I can see, adop goes into the adzdoptl.pl.. So we should investigate it from there.

[satish]

Thanks for the update erman.Issue is that patch was not able to start on second node,It didnt fail in the middle of applying.

We have deleted the entries related to patch in ad_adop_session and ad_adop_session_patches table after taking backup.Then tried to apply the patch which exited by saying "patch was already applied and use restart=yes option"

Then we used forceapply option and this time patch was applied on both the nodes successfully.But this is our personal system.Not sure if we can implement the same in our erp server

we have also ran prepare,finalize and cutover which went successfully

could you please let us know if this will create any problems

[ErmanArslansOracleBlog]

If you reproduced the issue and tested it in a clone environment(clone of your erp server), I think you can implement the same in your erp server.
However, I can't guarantee or approve this. Please create an SR in Oracle Support and ask for the approval.
you can create a SVR 1 SR for this. (it is a production)

[ErmanArslansOracleBlog]

You can also check the patches applied using the following query with the following query..
It may produce a better output, maybe we can identify the problematic record there..

SELECT adop_session_id, bug_number, session_type,
       DECODE(status,'N','Applied on other nodes',
                     'R','Running',
                     'H','Failed (Hard)',
                     'F','Failed (Jobs Skipped)',
                     'S','Success (Jobs Skipped)',
                     'Y','Success',
                     'C','Clone Complete') status,
       applied_file_system_base, patch_file_system_base,
       node_name, start_date, end_date,
       ROUND((end_date - start_date) * 24*60,2) exec_time,
       adpatch_options, autoconfig_status, driver_file_name
FROM ad_adop_session_patches
WHERE session_type IN ('ADPATCH','HOTPATCH','DOWNTIME','ONLINE')
ORDER BY adop_session_id, start_date, end_date;