ORA-12547: TNS:lost contact

[tyagi.ankur07]

 I'm encountering below issue while upgrading my EBS database from 11.1.0.7 to 11.2.0.3 version.

-bash-4.2$ pwd
/u01/oracle/VIS/db/tech_st/11.2.0
-bash-4.2$ cat 11gR2.env
export ORACLE_BASE=/u01/oracle/VIS
export ORACLE_HOME=/u01/oracle/VIS/db/tech_st/11.2.0
export LD_LIBRARY_PATH=/u01/oracle/VIS/db/tech_st/11.2.0/lib
export ORA_NLS10=/u01/oracle/VIS/db/tech_st/11.2.0/nls/data/9idata
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/perl/bin:$PATH
export PERL5LIB=$ORACLE_HOME/perl/lib/5.10.0:$ORACLE_HOME/perl/lib/site_perl/5.10.0

-bash-4.2$ . 11gR2.env
-bash-4.2$ export ORACLE_SID=VIS
-bash-4.2$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Wed Aug 16 00:51:47 2017
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
ERROR:
ORA-12547: TNS:lost contact


-bash-4.2$ ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x    1 oravis   oinstall          0 Oct 21 2011  /u01/oracle/VIS/db/tech_st/11.2.0/bin/oracle


Followed below steps but still getting the same error.
$ cd $ORACLE_HOME/bin
$ chmod 6751 oracle

$ cd $ORACLE_HOME/rdbms/lib
$ mv config.o config.o.bad
$ relink all

Below is the error message while trying to relink all 11203 oracle binaries:

SEVERE: oracle.sysman.oii.oiil.OiilActionException: Error in invoking target 'irman ioracle' of makefile '/u01/oracle/VIS/db/tech_st/11.2.0/rdbms/lib/ins_rdbms.mk'. See '/u01/oracle/VIS/db/tech_st/11.2.0/ins
tall/relinkActions2017-08-16_12-01-53-AM.log' for details.
        at oracle.sysman.oii.oiis.OiisMakeDeps.invokeMakefile(OiisMakeDeps.java:544)
        at oracle.sysman.oii.oiis.OiisMakeDeps.doRelink(OiisMakeDeps.java:621)
        at oracle.sysman.oii.oiis.OiisMakeDeps.doOperation(OiisMakeDeps.java:802)
        at oracle.sysman.oii.oiis.OiisMakeDeps.main(OiisMakeDeps.java:812)


Verified all interop. patches are in place.
Log files:
http://www.filedropper.com/opatchlinv
http://www.filedropper.com/relink
http://www.filedropper.com/relinkactions2017-08-1511-57-41-pm

Environment Details
------------------------
EBS 12.1.3
RDBMS - 11.0.1.7 -> Upgrade to 11.2.0.3
OS: AIX 7.1

Thanks,
Ankur

[ErmanArslansOracleBlog]

Hi,

Did you check this note? -> Relink Reports "Undefined Reference" Errors While Cloning 11.2 (Doc ID 1367164.1)

[tyagi.ankur07]

Hi Erman,

I followed the note and unset LD_LIBRARY_PATH and other environment variables with ORACLE_HOME, but still I'm getting the same error when trying to link ORACLE_HOME for 11.2.0.3

[ErmanArslansOracleBlog]

This is a clone right?

Please take "all the actions" documented in that document.. A db tier oracle home recloning is involved here as awell.

I mean;

1 - Delete target Oracle_Home directory (rm -rf)
2 - Unzip the source to the target Oracle_Home (step 2 of Note:1221705.1)
3 - Unset LD_LIBRARY_PATH and any other variables that point to a different $ORACLE_HOME
4 - Clone the home (step 3 of Note:1221705.1)
5 - Run the command (Against binaries in $ORACLE_HOME/bin )

% readelf -d $ORACLE_HOME/bin/oracle | grep -i rpath

to check that there are no more references to the source environment

Alternatively, check the timestamps of the files under $ORACLE_HOME/bin to make sure that all binaries have been successfully relinked

ldd command can also be used to check the binaries for library path used for compilation.
library should point to cloned  ORACLE_HOME/lib

[tyagi.ankur07]

Hi Erman,

This is not clone, I'm upgrading my EBS database from 11.1.0.7 to 11.2.0.3 version.
I'm getting this error while trying to install 11.2.0.3 software only, I skipped and continued further.
But now when I connect to database "as sysdba" I'm getting this error "ORA-12547"
Please review the log file for relink, opatch, installAction - attached earlier.

Thanks,
Ankur

[ErmanArslansOracleBlog]

Can you relink your Oracle home binaries using relink all successfully?

$ORACLE_HOME/bin/relink all;

[tyagi.ankur07]

relink all is failing.
Attached the make.log output.
http://www.filedropper.com/relink_1

[ErmanArslansOracleBlog]

Gathered info from the logfile:

ld: 0711-317 ERROR: Undefined symbol: .decimal128ToNumber
ld: 0711-317 ERROR: Undefined symbol: .decContextDefault
ld: 0711-317 ERROR: Undefined symbol: .decNumberExtractBiasedExponent
ld: 0711-317 ERROR: Undefined symbol: .decNumberZero
ld: 0711-317 ERROR: Undefined symbol: .decNumberInsertBiasedExponent
ld: 0711-317 ERROR: Undefined symbol: .decNumberAdd
ld: 0711-317 ERROR: Undefined symbol: .decimal64FromNumber
ld: 0711-317 ERROR: Undefined symbol: .decimal64ToNumber
ld: 0711-317 ERROR: Undefined symbol: .decNumberMultiply
ld: 0711-317 ERROR: Undefined symbol: .decNumberToIntegralValue
ld: 0711-317 ERROR: Undefined symbol: .decimal128FromNumber
ld: 0711-317 ERROR: Undefined symbol: .decNumberToLongLong
ld: 0711-317 ERROR: Undefined symbol: .decNumberSubtract
ld: 0711-317 ERROR: Undefined symbol: .decNumberCompare
ld: 0711-317 ERROR: Undefined symbol: .decimal32ToNumber
ld: 0711-317 ERROR: Undefined symbol: .decimal32FromNumber
make: 1254-004 The error code from the last command is 8.


Stop.
Error in invoking target 'irman ioracle' of makefile

[ErmanArslansOracleBlog]

Seems to be a "gcc" related thing..
The errors are produced some gcc related definitions.

Did you installed all the required OS packages? (especially gcc and its version.)
You should review the MOS document written for Database 11gR2 and AIX -- your version to ensure.

Your 11gR2 Binaries can be corrupted as well, but this is a low possibility.
Check, ensure and then update me.

AIX 7.1 required packages:

bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools
xlC.rte.11.1.0.2 or later
gpfs.base 3.3.0.11 or later (Only for RAC systems that will use GPFS cluster filesystems)


APARs for AIX 7.1:
IZ87216
IZ87564
IZ89165
IZ97035
See Note:1264074.1 and Note:1379908.1 for other AIX 7.1 patches that may be required

Refer to the 11.2 AIX Install Guide. Part Number E24332-01

[tyagi.ankur07]

I have reviewed the install document for aix 7.1 and verified all OS Pre requisite, everything looks in place.

Operating System Requirements
The following operating system filesets are required for AIX 7.1:
• bos.adt.base
• bos.adt.lib
• bos.adt.libm
• bos.perf.libperfstat
• bos.perf.perfstat
• bos.perf.proctools
• xlC.rte.11.1.0.2 or later
• gpfs.base 3.3.0.11 or later


root@aixxpp1:/-> lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libperfstat bos.perf.perfstat bos.perf.proctools xlC.rte* gpfs.base
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.adt.base               7.1.4.0  COMMITTED  Base Application Development
                                                 Toolkit
  bos.adt.lib               7.1.2.15  COMMITTED  Base Application Development
                                                 Libraries
  bos.adt.libm               7.1.4.0  COMMITTED  Base Application Development
                                                 Math Library
  bos.perf.libperfstat       7.1.4.0  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat          7.1.4.0  COMMITTED  Performance Statistics
                                                 Interface
  bos.perf.proctools         7.1.4.0  COMMITTED  Proc Filesystem Tools
  xlC.rte                   12.1.0.2  COMMITTED  IBM XL C++ Runtime for AIX

Path: /etc/objrepos
  bos.adt.base               7.1.4.0  COMMITTED  Base Application Development
                                                 Toolkit
  bos.perf.libperfstat       7.1.4.0  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat          7.1.4.0  COMMITTED  Performance Statistics
                                                 Interface
lslpp: 0504-132  Fileset gpfs.base not installed.


Patch Requirements
root@aixxpp1:/-> instfix -i -k "IZ87216 IZ87564 IZ89165 IZ97035"
    All filesets for IZ87216 were found.
    All filesets for IZ87564 were found.
    All filesets for IZ89165 were found.
    All filesets for IZ97035 were found.

[ErmanArslansOracleBlog]

I have started to think that your installation may be corrupt. But , it is to early to be sure.
The Undefined symbol error such as  ".decimal32FromNumber" is unexpected and undocumented.
These are like core symbols.
You have also lots of duplicate symbol warnings in your relink log.

1)What is written in 11gR2.env file? ( this problem may be caused by PATH and LIBPATH environment variables)
2)How did you install this 11gR2 ORacle RDBMS software? (silently or copying or using runInstaller properly)
3)Which link did you use for getting the Oracle RDBMS package installation?
4)Which note did you follow for the installation?
5)Tell me more about this database server env. You have 11.1.0.7 binaries there right? You have 11.2.0.3 also there. What else? Tell me more details.
6)Please send me the log or screenshot representing the errors that you got when you were installing the RDBMS software.. (you said you skipped some errors during the installation) I need to see them as well.

--Please enumerate your answer, like I did above.

[tyagi.ankur07]

1)What is written in 11gR2.env file? ( this problem may be caused by PATH and LIBPATH environment variables)

export ORACLE_BASE=/u01/oracle/VIS
export ORACLE_HOME=/u01/oracle/VIS/db/tech_st/11.2.0
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=/u01/oracle/VIS/db/tech_st/11.2.0/lib
export PERL5LIB=$ORACLE_HOME/perl/lib/5.10.0:$ORACLE_HOME/perl/lib/site_perl/5.10.0
export ORA_NLS10=/u01/oracle/VIS/db/tech_st/11.2.0/nls/data/9idata


2)How did you install this 11gR2 ORacle RDBMS software? (silently or copying or using runInstaller properly)
I installed 11.20.0.3 binaries using runInstaller, verified the software integrity by doing checksum and there are no issues with downloaded .zip files


3)Which link did you use for getting the Oracle RDBMS package installation?
Downloaded Patch 10404530 from oracle support.



4)Which note did you follow for the installation?
Interoperability Notes Oracle EBS R12 with Oracle Database 11gR2 (11.2.0.3) (Doc ID 1585578.1)
http://expertoracle.com/2012/10/13/upgrade-11-1-0-7-database-to-11-2-0-3-database-with-r12-1-3/
http://www.runningoracle.com/product_info.php?products_id=401


5)Tell me more about this database server env. You have 11.1.0.7 binaries there right? You have 11.2.0.3 also there. What else? Tell me more details.
I need to have a working EBS enviornment with DB version 11.2.0.3 and EBS version 12.1.3
Initially I installed Oracle EBS 12.1.1 which comes with 11.1.0.7 database, insatllation was successful.
I installed EBS 12.1.1 to 12.1.3 which was successfull.
Now, I'm working to upgrade DB from 11.1.0.7 to 11.2.0.3 version and gettting this relink error while installing the binaries.
I have mentioned the docs. I'm following for the upgrade & installation in point no. 4

6)Please send me the log or screenshot representing the errors that you got when you were installing the RDBMS software.. (you said you skipped some errors during the installation) I need to see them as well.
Attached the oraInstall.log and the screenshot of the error message from runInstaller.
http://www.filedropper.com/installactions2017-08-1411-15-10am
http://www.filedropper.com/irman

Please let me know if you need any additinoal information, thanks for your time and help on this.

[ErmanArslansOracleBlog]

The same errors that we see in relink...

Which C compiler are you using in this machine?

Send me the following ->

/usr/vacpp/bin/xlC -qversion

/usr/vac/bin/xlc -qversion

lslpp -l | grep "vac.C"; lslpp -l | grep "vacpp.cmp.core"; lslpp -l | grep "xlC.aix*"

Ref: Certified Compilers (Doc ID 43208.1)
Ref: Precompiler Pro*C Compilation On AIX Throws Duplicate Symbol p_xargc p_xargv p_xrcfg p_xrc Warnings (Doc ID 1136533.1)

[ErmanArslansOracleBlog]

send me the output of "env" command as well.
execute env command using oracle RDBMS owner OS user. (after sourcing your environment file, just like you did in your last relink attempt)

[tyagi.ankur07]

Please find the requested details.


-bash-4.2$ /usr/vacpp/bin/xlC -qversion
IBM XL C/C++ for AIX, V11.1 (5724-X13)
Version: 11.01.0000.0011


-bash-4.2$ /usr/vac/bin/xlc -qversion
IBM XL C/C++ for AIX, V11.1 (5724-X13)
Version: 11.01.0000.0011

-bash-4.2$ lslpp -l | grep "vac.C"; lslpp -l | grep "vacpp.cmp.core"; lslpp -l | grep "xlC.aix*"
  vac.C                    11.1.0.11  COMMITTED  IBM XL C Compiler
  vac.C                    11.1.0.11  COMMITTED  IBM XL C Compiler
  vacpp.cmp.core           11.1.0.11  COMMITTED  IBM XL C/C++ Compiler
  vacpp.cmp.core            11.1.0.0  COMMITTED  IBM XL C/C++ Compiler
  xlC.aix61.rte             12.1.0.2  COMMITTED  IBM XL C++ Runtime for AIX 6.1


-bash-4.2$ id
uid=211(oravis) gid=202(oinstall) groups=203(dba)
-bash-4.2$ env
AUTHSTATE=compat
TERM=xterm
SHELL=/usr/bin/bash
SSH_CLIENT=172.16.10.23 49457 22
PERL5LIB=/u01/oracle/VIS/db/tech_st/11.2.0/perl/lib/5.10.0:/u01/oracle/VIS/db/tech_st/11.2.0/perl/lib/site_perl/5.10.0
OLDPWD=/var
SSH_TTY=/dev/pts/1
LOCPATH=/usr/lib/nls/loc
USER=oravis
ORA_NLS10=/u01/oracle/VIS/db/tech_st/11.2.0/nls/data/9idata
ODMDIR=/etc/objrepos
ORACLE_BASE=/u01/oracle/VIS
AIXTHREAD_SCOPE=S
MAIL=/usr/spool/mail/oravis
PATH=/u01/oracle/VIS/db/tech_st/11.2.0/bin:/usr/bin:/etc:/usr/sbin:/usr/vacpp/bin:/usr/ucb:/home/oravis/bin:/usr/bin/X11:/sbin:.
LOGIN=oravis
PWD=/u01/oracle/VIS/db/tech_st/11.2.0
LANG=en_US
TZ=PST8
SHLVL=1
HOME=/home/oravis
LC__FASTMSG=true
MAILMSG=[YOU HAVE NEW MAIL]
LOGNAME=oravis
SSH_CONNECTION=172.16.10.23 49457 172.16.102.20 22
ORACLE_HOME=/u01/oracle/VIS/db/tech_st/11.2.0
_=/usr/bin/env
NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
LD_LIBRARY_PATH=/u01/oracle/VIS/db/tech_st/11.2.0/lib

[ErmanArslansOracleBlog]

Your gcc compiler seems to be certified. XL C/C++ 11.1  ...

However; why do you have such a thing? ->  xlC.aix61.rte  "12.1.0.2  COMMITTED  IBM XL C++ Runtime for AIX 6.1 "

[ErmanArslansOracleBlog]

Anyways, let's delete that 11.2.0.3 Oracle Home properly and reinstall it.
While reinstalling, ensure your env is empty (in terms of oracle related environment variables.. No setting like LD_LIBRARY_PATH or ORACLE_HOME or any related thing should be there.)

Let's see if you will get the same error during the installation.

We can also do a remote session, I want to review this issue by connecting to your server.

[tyagi.ankur07]

Ok, I will get rid of below from the server.
  xlC.aix61.rte             12.1.0.2  COMMITTED  IBM XL C++ Runtime for AIX 6.1

[tyagi.ankur07]

ok, let me cleanup up 11.2.0.3 OH and let's re-install without having any oracle related environment variables set up.
Please let me know your suitable time for a remote session, I will be available as per your availability.

[ErmanArslansOracleBlog]

Okay, do the cleanup, do a clean install as you confirmed and update me..
If the issue persist, I will send you the available time for the remote session.