prepare phase failed after upgrade 19c

[satish]

Hi Erman,

We have upgraded 12c database to 19c. This is the first patching cycle after the upgrade.
prepare phase failed while running autoconfig.

Prepare phase erroed out with


FNDCPUCF Executable : /u01/PRODAPPS/fs2/EBSapps/appl/fnd/12.0.0/bin/FNDCPUCF


Uploading Context file /u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/appl/admin/PRODDB_erpprodapp01.xml
ORACLE Password:
Context File upload for /u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/appl/admin/PRODDB_erpprodapp01.xml failed.
Check log file created by FNDCPUCF program.

SQL*Plus: Release 10.1.0.5.0 - Production on Thu Feb 9 11:30:57 2023

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Enter value for 1: Enter value for 2: Enter value for 3: ERROR:
ORA-12521: TNS:listener does not currently know of instance requested in
connect descriptor


SP2-0640: Not connected

Uploading Metadata file /u01/PRODAPPS/fs2/EBSapps/appl/ad/12.0.0/admin/template/adctxinf.tmp
ORACLE Password:
Metadata file /u01/PRODAPPS/fs2/EBSapps/appl/ad/12.0.0/admin/template/adctxinf.tmp upload failed.
Check log file created by FNDCPUCF program.
ERRORCODE = 1 ERRORCODE_END
.end std out.
APP-FND-01564: ORACLE error 12521 in AFPCOA

Cause: AFPCOA failed due to ORA-12521: TNS:listener does not currently know of instance requested in connect descriptor
.

The SQL statement being executed at the time of the error was:  and was executed from the file .
APP-FND-01564: ORACLE error 12521 in AFPCOA

  [APPLY PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/PRODAPPS/fs2/FMW_Home/webtier/perl/bin/perl -I /u01/PRODAPPS/fs2/FMW_Home/webtier/perl/lib/5.10.0 -I /u01/PRODAPPS/fs2/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl -I /u01/PRODAPPS/fs2/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/admin/install
      txkGenADOPWrapper.pl    INSTE8_APPLY       1



AutoConfig is exiting with status 28

AutoConfig execution completed on Thu Feb  9 11:31:20 2023

Time taken for AutoConfig execution to complete : 4 mins  0 secs
                                                                       

We have 2 node rac. We have verified listener status and it is listening for patch service.

Services Summary...
Service "28182bc4cf6c4887e053b340d20a8b55" has 1 instance(s).
  Instance "prodcdb1", status READY, has 1 handler(s) for this service...
Service "86b637b62fdf7a65e053f706e80a27ca" has 1 instance(s).
  Instance "prodcdb1", status READY, has 1 handler(s) for this service...
Service "PRODDBXDB" has 1 instance(s).
  Instance "prodcdb1", status READY, has 1 handler(s) for this service...
Service "PRODDB_ebs_patch" has 1 instance(s).
  Instance "prodcdb1", status READY, has 1 handler(s) for this service...
Service "ebs_PRODDB" has 1 instance(s).
  Instance "prodcdb1", status READY, has 1 handler(s) for this service...
Service "prodcdb" has 1 instance(s).
  Instance "prodcdb1", status READY, has 1 handler(s) for this service...
Service "proddb" has 1 instance(s).
  Instance "prodcdb1", status READY, has 1 handler(s) for this service...
The command completed successfully

What might be wrong here.can you please guide.

[satish]

Dear Erman,

env: 2node application tier and 2 nonde RAc.

We also verified local listener. 1528 is database port. We tried running autoconfig manually on node1 and node2.On node1, It completed in run edition but failed on patch edition with same error. On node2,it completed on both run and patch filesystems.

SQL> sho parameter local

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string      erpproddb2:1528
parallel_force_local                 boolean     TRUE
SQL> alter session set container=PRODDB;

Session altered.

SQL> sho parameter local

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string      erpproddb2:1528

[ErmanArslansOracleBlog]

This seems to be related with the TNS_ADMIN configuration files on Run or Patch fs. Please check accordingly.

Here is a ref: Adop Prepare Failure With Error 'ORA-12521' (Doc ID 1998916.1)

[satish]

Thanks for the update. We tried but unable to get connection..No errors in logfiles.

[applprod@erpprodapp01 ~]$ . /u01/PRODAPPS/EBSapps.env PATCH

  E-Business Suite Environment Information
  ----------------------------------------
  RUN File System           : /u01/PRODAPPS/fs1/EBSapps/appl
  PATCH File System         : /u01/PRODAPPS/fs2/EBSapps/appl
  Non-Editioned File System : /u01/PRODAPPS/fs_ne


  DB Host: erpproddb1.ttd.com  Service/SID: PRODDB


  Sourcing the PATCH File System ...

[applprod@erpprodapp01 ~]$ perl $AD_TOP/bin/adgentns.pl contextfile=$CONTEXT_FILE appspass=ap9l1cati0n
##########################################################################
                   Generate Tns Names
##########################################################################
Logfile:  /u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/admin/log/02091338/NetServiceHandler.log
        Classpath                   : /u01/PRODAPPS/fs2/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/PRODAPPS/fs2/EBSapps/comn/java/classes

Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0

adgentns.pl exiting with status 0
ERRORCODE = 0 ERRORCODE_END


[applprod@erpprodapp01 ~]$ sqlplus apps/**********

SQL*Plus: Release 10.1.0.5.0 - Production on Thu Feb 9 13:38:59 2023

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

ERROR:
ORA-12521: TNS:listener does not currently know of instance requested in
connect descriptor


Log

ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : (/u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/admin/install/txkGenADOPWrapper.pl)
TIME    : Sun Feb  5 20:10:04 2023
FUNCTION: TXK::SQLPLUS::_doExecute [ Level 3 ]
MESSAGES:
Invalid Connect string - cannot connect to database
ERROR:
ORA-12521: TNS:listener does not currently know of instance requested in
connect descriptor





STACK TRACE
 at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/Error.pm line 168
        TXK::Error::abort('TXK::Error', 'HASH(0x13f1da8)') called at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/Common.pm line 299
        TXK::Common::doError('TXK::SQLPLUS=HASH(0x241b428)', 'Invalid Connect string - cannot connect to database\x{a}ERROR:\x{a}OR...', undef) called at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/Common.pm line 314
        TXK::Common::setError('TXK::SQLPLUS=HASH(0x241b428)', 'Invalid Connect string - cannot connect to database\x{a}ERROR:\x{a}OR...') called at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/SQLPLUS.pm line 741
        TXK::SQLPLUS::_doExecute('TXK::SQLPLUS=HASH(0x241b428)', 'true') called at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/SQLPLUS.pm line 376
        TXK::SQLPLUS::validateConnectInfo('TXK::SQLPLUS=HASH(0x241b428)') called at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/SQLPLUS.pm line 267
        TXK::SQLPLUS::setConnectInfo('TXK::SQLPLUS=HASH(0x241b428)', 'HASH(0x1b155a8)') called at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/TechstackDB.pm line 608
        TXK::TechstackDB::_getSQLValue('TXK::TechstackDB=HASH(0x1b15350)', 'SELECT ad_zd_adop.get_node_type(\'erpprodapp01\') FROM DUAL') called at /u01/PRODAPPS/fs2/EBSapps/appl/au/12.0.0/perl/TXK/TechstackDB.pm line 573
        TXK::TechstackDB::getSQLValue('TXK::TechstackDB=HASH(0x1b15350)', 'SELECT ad_zd_adop.get_node_type(\'erpprodapp01\') FROM DUAL') called at /u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/admin/install/txkGenADOPWrapper.pl line 506
        main::getNodeType() called at /u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/admin/install/txkGenADOPWrapper.pl line 202
        eval {...} called at /u02/PRODINST/fs2/inst/apps/PRODDB_erpprodapp01/admin/install/txkGenADOPWrapper.pl line 143
)
ERRORCODE = 1 ERRORCODE_END
.end std out.

[ErmanArslansOracleBlog]

How do your tnsnames files look like (both in patch and run fs) .. Is they wrong? What is wrong with them?

In case, check your database listener as well. Ensure the listener have PDB service registered. (but I guess you have ebs_PRODDB already there)