Adop session

[Laurel]

Hi Erman!
I hope you had a nice vacation :)
Back to the problems..

adop phase=prepare
adop phase=apply patches=12345
Went fine
adop phase=apply patches=23456
failed adworkers on tablespace issue
Fixed issue
adctrl on patching system -> fixed restart ( option 4)
NOTHING happens, adop just was not giving prompt back..
Tried different adctrl  to quit etc -> nope, nothing happens and adop not doing anything..

I killed the adop session from prompt. ( Ctrl C, adop no more as a process : ps -ef |grep adop

Now adop phase=apply patches =23456 restart=yes abandon=no
Error -> Unable to continue as an existing adop session was found.
Tried
adop phase=apply patches =23456 restart=no abandon=yes
Error :Unable to continue as an existing adop session was found.

adop -status

xxx      master     PREPARE         COMPLETED      
                           APPLY           RUNNING        
                           FINALIZE        NOT STARTED
                           CUTOVER         NOT STARTED
                           CLEANUP         NOT STARTED

SO.. Should I run adop phase=abort and cleanup  ? and from begining  of first patch?
OR can I just manually update the table that Apply  phase failed and to restart ?

Any suggestions ?
Thanks,Laurel

[ErmanArslansOracleBlog]

Hi Laurel,

Another vacation is on its ways:)

Anyways, did you check the worker's status ? (adctrl)

Did you check the solution provided by Latifa in this thread -> http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/All-phasis-are-aborted-only-cleanup-is-completed-td1900.html

I can suggest you ,  adop phase=abort + adop phase=cleanup cleanup_mode=full

But if the issue persist, a manual update may be required (update status column in the ad_adop_sessions table to 'C'), but this should be approved by Oracle Support.

[Laurel]

Hi Erman!
I wish You  a good vacation :)

Yes I did check  adctrl and ask to restart ( option 4), so it stayed that way .
Adop did not do anything just hanging  and I killed it after 15 min.
the adop -status showed running.
So I asked you here.
BUT
after 3H I noticed that on the  unix came this  message
$ Out of memory!
Callback called exit at /xx/fs1/EBSapps/appl/au/12.0.0/perl/ADOP/Phase.pm line 215.

Haha..
Checking the adop -status -> apply status changed FROM Running  to FAILED.
Now i could run adop restart=yes abandon=no
and it completed successfully..

So, there is somekind of timeout mechanism adop and it changes status RUNNING->FAILED
why in the earth adop  is  not continuing after you do in adctrl  (option4)
I actually noticed that before. adctrl usage with adop, adop is not continuing with it

Some undocumented feature .. Have you used successfully in the same session with adop adctrl to restart failed worker ?

Thanks,Laurel

[ErmanArslansOracleBlog]

good news :)
3h timeout .. interesting.
I didn't face it actually.

But how do you see this OS message if  there is no adop related processes running in your system? ( Callback called exit at /xx/fs1/EBSapps/appl/au/12.0.0/perl/ADOP/Phase.pm line 215. )

[ErmanArslansOracleBlog]

Note: "Callback called exit"  comes from the perl (that pm..) , it is caused by an infinite loop or something.

This is not a timeout I guess, but the code just fails after a while, after consuming somethings..

[ErmanArslansOracleBlog]

adctrl can be used with adop. That is documented.
Option 4 should work as well.
However, in your situation, it seems adop and adworkers were just stuck. That's why maybe they didn't processed your adctrl commands...


Were there any processes left, that may cause that callback message? Did you use os utils, like ps -ef , to check them?

[Laurel]

HI Erman!
I did specifically checked that no any adop processes left by ps -ef.
Its AIX 7.1

The message appeared in the same unix session  where I was running  adop.
I killed in another window the  adop session.

$Killed
$Out of memory!
Callback called exit at /romapp/oracle/ROM/fs1/EBSapps/appl/au/12.0.0/perl/ADOP/Phase.pm line 215.


I think I have seeing before somebody was writing that after while the adop status by itself was changing from Running to Failed..  :)

Thanks for help!

[ErmanArslansOracleBlog]

interesting.. why your terminal?
I think something left there in the background..