12.2.5 patch apply failed

[karthikmca.bhc]

Hi Erman,

Im applying RUP 12.2.5 as per oracle ID  1983050.1. Section 8: Apply Oracle E-Business Suite 12.2.5 Release Update Pack

$ adop phase=apply apply_mode=downtime patches=19676458

ATTENTION: All workers either have failed or are waiting:

           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  1.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  2.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  3.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  4.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  7.

ATTENTION: Please fix the above failed worker(s) so the manager can continue.

Worker Log:
-------------

BEGIN AD_ZD_TABLE.APPLY('FND_EID_GROUPS_B_F1'); END;

*
ERROR at line 1:
ORA-20001: Could not apply crossedition trigger "APPS"."FND_EID_GROUPS_B_F1":
ORA-20001: Parallel update task failed: "FND_EID_GROUPS_B_F1" status=8 Error:
ORA-28115: policy with check option violation
ORA-06512: at "APPS.FND_EID_GROUPS_B_F1", line 29
ORA-04088: error during execution of trigger 'APPS.FND_EID_GROUPS_B_F1'
ORA-06512: at "APPS.AD_ZD", line 26
ORA-06512: at "APPS.AD_ZD_TABLE", line 19
ORA-06512: at "APPS.AD_ZD_TABLE", line 1784
ORA-06512: at line 1

===============

BEGIN AD_ZD_TABLE.APPLY('FND_EID_GROUPS_TL_F1'); END;

*
ERROR at line 1:
ORA-20001: Could not apply crossedition trigger "APPS"."FND_EID_GROUPS_TL_F1":
ORA-20001: Parallel update task failed: "FND_EID_GROUPS_TL_F1" status=8 Error:
ORA-28115: policy with check option violation
ORA-06512: at "APPS.FND_EID_GROUPS_TL_F1", line 27
ORA-04088: error during execution of trigger 'APPS.FND_EID_GROUPS_TL_F1'
ORA-06512: at "APPS.AD_ZD", line 26
ORA-06512: at "APPS.AD_ZD_TABLE", line 19
ORA-06512: at "APPS.AD_ZD_TABLE", line 1784
ORA-06512: at line 1

===========

BEGIN AD_ZD_TABLE.APPLY('FND_EID_PDR_ATTRS_B_F1'); END;

*
ERROR at line 1:
ORA-20001: Could not apply crossedition trigger
"APPS"."FND_EID_PDR_ATTRS_B_F1": ORA-20001: Parallel update task failed:
"FND_EID_PDR_ATTRS_B_F1" status=8 Error: ORA-04063: package body
"APPS.FND_EID_PDR_ATTRS_PKG" has errors
ORA-06508: PL/SQL: could not find program unit being called:
"APPS.FND_EID_PDR_ATTRS_PKG"
ORA-06512: at "APPS.FND_EID_PDR_ATTRS_B_F1", line 39
ORA-04088: error during execution of trigger 'APPS.FND_EID_PDR_ATTRS_B_F1'
ORA-06512: at "APPS.AD_ZD", line 26
ORA-06512: at "APPS.AD_ZD_TABLE", line 19
ORA-06512: at "APPS.AD_ZD_TABLE", line 1784
ORA-06512: at line 1

==============

BEGIN AD_ZD_TABLE.APPLY('FND_EID_PDR_ATTRS_TL_F1'); END;

*
ERROR at line 1:
ORA-20001: Could not apply crossedition trigger
"APPS"."FND_EID_PDR_ATTRS_TL_F1": ORA-20001: Parallel update task failed:
"FND_EID_PDR_ATTRS_TL_F1" status=8 Error: ORA-04063: package body
"APPS.FND_EID_PDR_ATTRS_PKG" has errors
ORA-06508: PL/SQL: could not find program unit being called:
"APPS.FND_EID_PDR_ATTRS_PKG"
ORA-06512: at "APPS.FND_EID_PDR_ATTRS_TL_F1", line 39
ORA-04088: error during execution of trigger 'APPS.FND_EID_PDR_ATTRS_TL_F1'
ORA-06512: at "APPS.AD_ZD", line 26
ORA-06512: at "APPS.AD_ZD_TABLE", line 19
ORA-06512: at "APPS.AD_ZD_TABLE", line 1784
ORA-06512: at line 1

================

BEGIN AD_ZD_TABLE.APPLY('FND_EID_ATTR_GROUPS_F1'); END;

*
ERROR at line 1:
ORA-20001: Could not apply crossedition trigger
"APPS"."FND_EID_ATTR_GROUPS_F1": ORA-20001: Parallel update task failed:
"FND_EID_ATTR_GROUPS_F1" status=8 Error: ORA-28115: policy with check option
violation
ORA-06512: at "APPS.FND_EID_ATTR_GROUPS_F1", line 30
ORA-04088: error during execution of trigger 'APPS.FND_EID_ATTR_GROUPS_F1'
ORA-06512: at "APPS.AD_ZD", line 26
ORA-06512: at "APPS.AD_ZD_TABLE", line 19
ORA-06512: at "APPS.AD_ZD_TABLE", line 1784
ORA-06512: at line 1


Please provide workaround/fix to continue the patch apply.

Thanks
Karthik

[ErmanArslansOracleBlog]

It seems to have  a package missing or invalid.

Do you have this object in your database ? > APPS.FND_EID_PDR_ATTRS_PKG
Is it valid?
If it is not valid -> what is its error?

[Karthik]

Hi Erman,

I checked the status of the failing triggers. I have enabled those failing triggers and the reapplied the patch with restart=yes option and now it again failed with below error,

Reading jobs from FND_INSTALL_PROCESSES table ...


Evaluating symbolic arguments for worker 1
   FAILED: file AD_ZD_TABLE_APPLY.sql on worker  1 for product ad  username APPLSYS.

Time is: Sun Jan 06 2019 19:00:37


Evaluating symbolic arguments for worker 2
   FAILED: file AD_ZD_TABLE_APPLY.sql on worker  2 for product ad  username APPLSYS.

Time is: Sun Jan 06 2019 19:00:37


Evaluating symbolic arguments for worker 3
   FAILED: file AD_ZD_TABLE_APPLY.sql on worker  3 for product ad  username APPLSYS.

Time is: Sun Jan 06 2019 19:00:37


Evaluating symbolic arguments for worker 4
   FAILED: file AD_ZD_TABLE_APPLY.sql on worker  4 for product ad  username APPLSYS.

Time is: Sun Jan 06 2019 19:00:37


Evaluating symbolic arguments for worker 5
   FAILED: file AD_ZD_TABLE_APPLY.sql on worker  5 for product ad  username APPLSYS.

Time is: Sun Jan 06 2019 19:00:37


Done reading jobs from FND_INSTALL_PROCESSES table ...

Telling workers to read 'todo' restart file.
Done.

ATTENTION: All workers either have failed or are waiting:

           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  1.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  2.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  3.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  4.
           FAILED: file AD_ZD_TABLE_APPLY.sql on worker  5.

ATTENTION: Please fix the above failed worker(s) so the manager can continue.

Current time is: Sun Jan 06 2019 19:00:37


Telling workers to quit...

5 workers have quit.  Waiting for 11 more.

All workers have quit.

Connecting to APPS......Connected successfully.

AutoPatch error:

Error running SQL and EXEC commands in parallel

 Freeing includes hash table
Freeing fixes hash table
 Freeing basedons hash table
Freeing entities hash table

You should check the file
/u02/applmgr/fs_ne/EBSapps/log/adop/7/20190106_185942/apply/1227demo/19676458/log/u19676458.log


Worker1 log:
--------------

************* Start of AD Worker session *************
AD Worker version: 12.2.0
AD Worker started at: Sun Jan 06 2019 19:00:37

APPL_TOP is set to /u02/applmgr/fs2/EBSapps/appl

Worker id = 1
Started by: adpatch

NLS_LANG value from the environment is    : AMERICAN_AMERICA.AL32UTF8
NLS_LANG value for this AD utility run is : AMERICAN_AMERICA.AL32UTF8

Connecting to SYSTEM......Connected successfully.

Already asked for the ORACLE username and password
of Application Object Library.
(The given username/password was: APPLSYS/*****)

Connecting to APPS......Connected successfully.


The status of various features in this run of AD Worker is:

                                           <-Feature version in->
Feature                          Active?   APPLTOP    Data model    Flags
------------------------------   -------   --------   -----------   -----------
CHECKFILE                        Yes       1          1             Y N N Y N Y
PREREQ                           Yes       6          6             Y N N Y N Y
CONCURRENT_SESSIONS              No        2          2             Y Y N Y Y N
PATCH_TIMING                     Yes       2          2             Y N N Y N Y
PATCH_HIST_IN_DB                 Yes       6          6             Y N N Y N Y
SCHEMA_SWAP                      Yes       1          1             Y N N Y Y Y
JAVA_WORKER                      Yes       1          1             Y N N Y N Y
CODELEVEL                        Yes       1          1             Y N N Y N Y



Reading product information from file...

Reading language and territory information from file...

Reading language information from applUS.txt ...

Currently, the following language is installed:

Code   Language                                Status
----   --------------------------------------- ---------
US     American English                        Base        

Your base language will be AMERICAN.


Manager says to quit.


Time when worker quit: Sun Jan 06 2019 19:00:38

AD Worker is complete.

Errors and warnings are listed in the log file
/u02/applmgr/fs_ne/EBSapps/log/adop/7/20190106_185942/apply/1227demo/19676458/log/adwork001.log

and in other log files in the same directory.

[ErmanArslansOracleBlog]

We need to check the workers log once more..
Send me adworker1 log.
Also please answer my following question:

Do you have this object in your database ? > APPS.FND_EID_PDR_ATTRS_PKG
Is it valid?
If it is not valid -> what is its error?

[Karthik]

Hi Erman,

I would like to update you that the trigger issue persists when applying 12.2.5 RUP. So what I did is instead of applying 12.2.5 i applied 12.2.7 RUP on top of 12.2.0 and that successfully applied without those trigger errors. It seems the 12.2.5 RUP has the bug which raised when applying and that must have fixed on 12.2.7 version or higher.

Thank you for your instant support and advice in fixing issues and learning.

[ErmanArslansOracleBlog]

Thanks for the info Karthik..
I upgraded EBS 12.2.0 to 12.2.5 lots of times, but never hit that bug..
But maybe as you say, it is a bug that appears in certain conditions..
Good for you.