adop phase=cleanup is failing post 12.2.6 upgrade of 12.2 vision instance

[samdba]

Hello,

I installed 12.2 vision instance on a single node and upgraded to 12.2.6. Next, I am running adop phase=cleanup as per note # 2114016.1 and it shows error [ERROR] [CLEANUP 1:1 ddl_id=26481] ORA-04021: timeout occurred while waiting to lock object SQL: begin sys.ad_grants.cleanup; end;

Then, I performed steps as per note # 2285537.1 to fix this issue and then re-ran adop phase=cleanup again but now it is failing with [UNEXPECTED]Unrecoverable error occurred. Exiting current adop session.

I have uploaded the text file adop_cleanup_errors_NEW.txt which shows all the messages.

I have opened SR with Oracle but not got any good response yet.

Please help.

Thanks,

Samadop_cleanup_errors_NEW.txt

[samdba]

More Info:
===========
In the uploaded text, file you can see that it shows [WARNING]   MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh but I have already installed this patch and latest run of checkMTpatch.sh shows "All required one-offs are confirmed as present. "

select release_name from apps.fnd_product_groups;
== 12.2.6

select ad_patch.is_patch_applied('R12',-1,21841299,21900901) from dual;
== NOT_APPLIED   <<- Even though I have applied these patches. See below.

select bug_number, creation_date from ad_bugs where bug_number in (21841299,21900901)
21841299 7/11/2017 6:52:38 PM
21900901 7/17/2017 10:54:55 PM

[ErmanArslansOracleBlog]

The error seems to be occured while checking the adop sessions.

It should write , " No pending session exists.", but it just fails.

Two tables are suspicious: ad_adop_sessions and ad_adop_session_patches

I recommend ,  adop phase=abort,cleanup cleanup_mode=full.
adop abort will deletes the rows inserted for the pending session ID from the ad_adop_sessions and ad_adop_session_patches tables.

[samdba]

Thanks!!!

As per my SR analyst's action plan, I ran adadmin and updated the snapshot information and re-tried adop phase=cleanup but it shows different errors now. I will ask Oracle if I can try your solution.

$ adop phase=cleanup

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials.

Initializing.
    Run Edition context  : /ord80/ohfvis/ora/app/fs1/inst/apps/OHFVIS_oaortest/appl/admin/OHFVIS_oaortest.xml
    Patch edition context: /ord80/ohfvis/ora/app/fs2/inst/apps/OHFVIS_oaortest/appl/admin/OHFVIS_oaortest.xml
    Patch file system free space: 113.86 GB

Validating system setup.
    Node registry is valid.

Checking for existing adop sessions.
    [UNEXPECTED]Unrecoverable error occurred. Exiting current adop session.


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 2 (Fail)

$  adopscanlog -latest=yes

Scanning /ord80/ohfvis/ora/app/fs_ne/EBSapps/log/adop/2/ directory ...

No Errors.

[ErmanArslansOracleBlog]

Again, it fails during ->Checking for existing adop sessions.

Okay, let's analyze those tables, get the approval from SR and update me accordingly.

[Sai Sandeep Punukollu]

Hi,

I'm facing the same issue while trying to upgrade to 12.2.7. But i have faced it during the patch cycle of ad/txk delta 10 patching.

Could you please let me know if the issue has been identified and any possible solution.


Thank You,
Sai

[ErmanArslansOracleBlog]

Said,

First, run  "adop phase=abort + adop phase=cleanup cleanup_mode=full"  and retry.

[karthikmca.bhc]

Hi Erman,

As you advised, abort and full mode cleanup has been ran and performed below tasks as oracle advised. After the below fsclone completed successfully.

"On the application server can you please do a "ps -eaf | grep <PID>" where the value of PID is the list:
20923,20928, 20933,20938,20943,20948,20953,20963,20976, 20981
Do all these PIDs still exist? if yes, kill them all by using "kill
-9 <pid>".

If the PIDs do not exist, do "netstat -an | grep <port number>" for these
ports?
5557,7002 ,17002,7602 ,7402 ,7202 ,7802 ,6101 ,6201 ,10000

This should list the port number and its status. For Example "netstat
-an | grep 5556". If there is a response for this command, do "/usr/sbin/lsof
| grep <port_number>".. This will show the actual command with PIDs and what
their current status is. Kill these processes using "kill -9 <PID>"

Execute

    export TIMEDPROCESS_TIMEOUT=-1 (infinite)
    export TIMEDPROCESS_DEBUG=YES    <------Stack Trace

Reattempt clone as such

assure there is an empty fs2 directory
adop phase=fs_clone force=yes

[ErmanArslansOracleBlog]

Good for you!
Thanks for the update.

[karthikmca.bhc]

Thanks Erman for the instant support and advise on this issue.

How I can do a adop cycle test without applying any patch to make sure adop functions without any issues.

Thanks.

[ErmanArslansOracleBlog]

There is a mode named test mode.

See - -> How To Run Adop Like - Adpatch - With Option Apply=No? (Doc ID 2067371.1)

[karthikmca.bhc]

Thanks Erman,

How I can cancel the running adop session?

I ran actualize_all as below and it seems running for very long time and not sure it got hanged somewhere.

Any workaround to make the adop session to move forward?

Thanks

[karthikmca.bhc]

[applprd@pieslxtst scripts]$ adop phase=actualize_all

 

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

 

Validating credentials.

 

Initializing.

    Run Edition context  : /home/applprd/PROD/fs1/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml

    Patch edition context: /home/applprd/PROD/fs2/inst/apps/PROD_pieslxtst/appl/admin/PROD_pieslxtst.xml

    Patch file system free space: 124.62 GB

 

Validating system setup.

    Node registry is valid.

 

Checking for existing adop sessions.

    Continuing with existing session [Session ID: 18].

===========================================================================

ADOP (C.Delta.9)

Session ID: 18

Node: pieslxtst

Phase: actualize_all

Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/18/20171010_103439/adop.log

===========================================================================

Populating actualize actions in DDL handler table.

    Clearing old rows from DDL handler table.

    Populating DDLs.

    Log: @ADZDSHOWLOG.sql "2017/10/10 10:36:03"

Running actualize_all DDL in parallel.

    Log: /home/applprd/PROD/fs_ne/EBSapps/log/adop/18/20171010_103439/actualize_all/pieslxtst/log/actualize_all.log

 

[root@pieslxtst ~]# tail -500f /home/applprd/PROD/fs_ne/EBSapps/log/adop/18/20171010_103439/actualize_all/pieslxtst/log/actualize_all.log

 

Connecting to APPS......Connected successfully.

Completed: file ADZDWRKR.sql on worker  3 for product ad  username APPLSYS.

Time is: Tue Oct 10 2017 12:36:31

Assigned: file ADZDWRKR.sql on worker  2 for product ad  username APPLSYS.

Time is: Tue Oct 10 2017 12:36:31

Assigned: file ADZDWRKR.sql on worker  3 for product ad  username APPLSYS.

Time is: Tue Oct 10 2017 12:36:32

Completed: file ADZDWRKR.sql on worker  1 for product ad  username APPLSYS.

Time is: Tue Oct 10 2017 12:38:05

Completed: file ADZDWRKR.sql on worker  4 for product ad  username APPLSYS.

Time is: Tue Oct 10 2017 12:38:06

Assigned: file ADZDWRKR.sql on worker  1 for product ad  username APPLSYS.

Time is: Tue Oct 10 2017 12:38:06

Assigned: file ADZDWRKR.sql on worker  4 for product ad  username APPLSYS.

Time is: Tue Oct 10 2017 12:38:06

 

[karthikmca.bhc]

Thanks Erman,

How I can cancel the running adop session?

I ran actualize_all as below and it seems running for very long time and not sure it got hanged somewhere.

Any workaround to make the adop session to move forward?

Thanks

[ErmanArslansOracleBlog]

AD Controller (adctrl) Manages parallel workers in AD Administration and AutoPatch.

You source "patch env". and use adctrl. (adctrl should be used from the patch env)

Read this -> https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T530135.htm (read the subtitle : Terminating a Hanging Worker Process)

[ErmanArslansOracleBlog]

After you terminate the process from the OS. Your adwork will be failed.
Then you can follow subtitle named "Restarting a Failed Worker" in the same document.