Oracle EBS 12.2 on VMware

[Muazzam]

Greetings of the day!

I'm trying to install Oracle EBS 12.2 on Oracle Linux 7.7 on VMware platform but I'm getting this error :

RW-50004 : An error has occured. Please check /u01/ORAPROD/PROD/12.1.0/appsutil/log/PROD_erpnode1/12030400.log

The log file has this error :

Failed to rename database
RW-50010: Error: – script has returned an error:   1
RW-50004: Error code received when running external process. Check log file for details. Running Database Install Driver for PROD instance

I checked on the internet regarding this error and the solution I found was to change directory permissions to 777/775 for ORAPROD, APPLPROD and Stage directories. I have done that, but still I'm getting the same error.


The following are my pre-installation steps that I’ve followed
#cat /etc/resolv.conf

# Generated by NetworkManager
search oralabs.com
nameserver 192.168.23.2
options attempts:5
options timeout:15

[root@erpnode1 ~]# cat /etc/sysctl.conf

# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
kernel.msgmax = 8192
net.ipv4.tcp_tw_recycle = 0

# oracle-ebs-server-R12-preinstall setting for fs.file-max is 6815744
fs.file-max = 6815744

# oracle-ebs-server-R12-preinstall setting for kernel.sem is '256 32000 100 142'
kernel.sem = 256 32000 100 142

# oracle-ebs-server-R12-preinstall setting for kernel.shmmni is 4096
kernel.shmmni = 4096

# oracle-ebs-server-R12-preinstall setting for kernel.shmall is 1073741824 on x86_64
# oracle-ebs-server-R12-preinstall setting for kernel.shmall is 2097152 on i386
kernel.shmall = 1073741824

# oracle-ebs-server-R12-preinstall setting for kernel.shmmax is 4398046511104 on x86_64
# oracle-ebs-server-R12-preinstall setting for kernel.shmmax is 4294967295 on i386
kernel.shmmax = 4398046511104

# oracle-ebs-server-R12-preinstall setting for kernel.panic_on_oops is 1
kernel.panic_on_oops = 1

# oracle-ebs-server-R12-preinstall setting for kernel.msgmni is 2878
kernel.msgmni = 2878

# oracle-ebs-server-R12-preinstall setting for kernel.msgmnb is 65535
kernel.msgmnb = 65535

# oracle-ebs-server-R12-preinstall setting for net.core.rmem_default is 262144
net.core.rmem_default = 262144

# oracle-ebs-server-R12-preinstall setting for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304

# oracle-ebs-server-R12-preinstall setting for net.core.wmem_default is 262144
net.core.wmem_default = 262144

# oracle-ebs-server-R12-preinstall setting for net.ipv4.conf.all.rp_filter is 2
net.ipv4.conf.all.rp_filter = 2

# oracle-ebs-server-R12-preinstall setting for net.ipv4.conf.default.rp_filter is 2
net.ipv4.conf.default.rp_filter = 2

# oracle-ebs-server-R12-preinstall setting for net.core.wmem_max is 1048576
net.core.wmem_max = 1048576

# oracle-ebs-server-R12-preinstall setting for fs.aio-max-nr is 1048576
fs.aio-max-nr = 1048576

# oracle-ebs-server-R12-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500


[root@erpnode1 ~]# grep SwapTotal /proc/meminfo
SwapTotal:      17407996 kB


[root@erpnode1 limits.d]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.23.2 erpnode1.oralabs.com erpnode1



[root@erpnode1 limits.d]# cat oracle-ebs-server-R12-preinstall.conf
# oracle-ebs-server-R12-preinstall setting for nofile soft limit is 4096
oracle   soft   nofile    4096
applmgr   soft   nofile    4096

# oracle-ebs-server-R12-preinstall setting for nofile hard limit is 65536
oracle   hard   nofile    65536
applmgr   hard   nofile    65536

# oracle-ebs-server-R12-preinstall setting for nproc soft limit is 16384
# refer orabug15971421 for more info.
oracle   soft   nproc    16384
applmgr   soft   nproc    16384

# oracle-ebs-server-R12-preinstall setting for nproc hard limit is 16384
oracle   hard   nproc    16384
applmgr   hard   nproc    16384

# oracle-ebs-server-R12-preinstall setting for stack soft limit is 10240KB
oracle   soft   stack    10240
applmgr   soft   stack    10240

# oracle-ebs-server-R12-preinstall setting for stack hard limit is 32768KB
oracle   hard   stack    32768
applmgr   hard   stack    32768

# oracle-ebs-server-R12-preinstall setting for memlock hard limit is maximum of 128GB for x86_64 OR 90percent of RAM
oracle   hard   memlock    134217728
applmgr   hard   memlock    134217728

# oracle-ebs-server-R12-preinstall setting for memlock soft limit is maximum of 128GB for x86_64 OR 90percent of RAM
oracle   soft   memlock    134217728
applmgr   soft   memlock    134217728


[root@erpnode1 limits.d]# cat /etc/sysconfig/network
# Created by anaconda
# oracle-ebs-server-R12-preinstall Add NOZEROCONF=yes
NOZEROCONF=yes
NETWORKING=yes
HOSTNAME=erpnode1.oralabs.com
GATEWAY=192.168.23.2
[root@erpnode1 limits.d]# uname -r
3.10.0-1062.7.1.el7.x86_64

[root@erpnode1 limits.d]# rpm -qa --qf "%{n}-%{v}-%{r}.%{arch}\n" | grep libgcc
libgcc-4.8.5-39.0.3.el7.x86_64
libgcc-4.8.5-39.0.3.el7.i686

[ErmanArslansOracleBlog]

I need to see the log file..
This may due to lots of things.. /tmp size is one of them..
Increase the disk space for  "/tmp"  and retry.

But, as I said, I need to see the log file..
Please attach it to here.


In order to uploa a file, you should first create an account .
With your account logged in, click reply. While you are replying you should click on "More>Upload a file" to upload a file.

[Muazzam]

log_file_with_error.txt

Please note the above file is the main file error message refers to.


restore-rac1.log
restore-single2.log
restore-rac3.log

These three files are created in /u01/ORAPROD/PROD/12.1.0/appsutil/out/PROD_erpnode1 directory, to which main log file is pointing to check.

Also, I'm uploading a file in which all pre-requisite checks have been saved before installation
Port_Availability.txt

This file contains all checks performed before installation.

Please go through all (if needed) and let me know.

[ErmanArslansOracleBlog]

It is related with your backup-restore.

restore-single2.log complains about the following ->


RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 12/09/2019 01:47:20
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 405 found to restore
RMAN-06023: no backup or copy of datafile 397 found to restore
RMAN-06023: no backup or copy of datafile 396 found to restore
RMAN-06023: no backup or copy of datafile 395 found to restore
RMAN-06023: no backup or copy of datafile 353 found to restore
RMAN-06023: no backup or copy of datafile 352 found to restore
RMAN-06023: no backup or copy of datafile 32 found to restore
RMAN-06023: no backup or copy of datafile 27 found to restore
RMAN-06023: no backup or copy of datafile 22 found to restore
RMAN-06023: no backup or copy of datafile 18 found to restore
RMAN-06023: no backup or copy of datafile 16 found to restore
RMAN-06023: no backup or copy of datafile 6 found to restore


That's why restore-rac3.log complains about the following;

RMAN-03002: failure of sql statement command at 12/09/2019 01:47:32
ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
ORA-01110: data file 1: '/u01/R122_EBS/data/system01.dbf'

**Ensure your backup is proper one, and retry the operation.

[Muazzam]

Can you please elaborate how to do that? I'm sorry, this may appear to be a basic step, but I'm new to EBS field. Please help me, I'm stuck here for more time than I'd have had liked to.

Any step by step guide would do, or any link/article which may give me a fair idea where I've been going wrong.

[ErmanArslansOracleBlog]

Okay...

you are doing a fresh / new installation.
In EBS installations, the database is actually restored from backup..
That backup is in your stage actually..
The backup is provided with the installation media that you download from Oracle.

So in this your stage seems corrupt, because some datafiles can not be restored during this installation process.. You may even forget to download some of the necessary installation files.

So your solution is there , in your stage , in your installation media.

First validate your Stage area using ->  MD5 Checksums for R12.2 Rapid Install Media (Doc ID 1505510.1)

Check the following document to create a proper stage area ->

R12.2: How To Create the Stage In Preparation For Installation (Doc ID 1596433.1)

Also ensure that you downloaded all the required installation files...

After correcting the issue, use the following document to clean your failed installation and do a proper fresh install again.

12.2 E-Business Suite - How To Clean Up A Rapid Install Failure A Restart A Fresh Install (Doc ID 2091880.1)

[Muazzam]

I'm stuck again in installiation of EBS and this I went upto step 3 (out of 8) and the installation stopped on its own. I'm including the log fileinstall_actions.txtlog.txttnsonames.txttnsora.PNG

If you check properly, the tnsnames is not getting created in $ORACLE_HOME/network/admin location but in ERPNODE1_PROD directory. Is this fine?
I thought that maybe there is a problem with my stage
So I ran the md5sum -c R12_2_-_Linux_x86-64_-_OL7.txt > md5s2.txt command
It says listed files could not be read
Thats why I ran this
md5sum -c StartCD51-Linux_x86-64.txt > md5s.txt
md5sum: WARNING: 11716 listed files could not be read is the output
Where am I going wrong?
Please help me, let me know
I'd be more than grateful
And yes, before starting the listener the installation stops
Second time I tried, listener didn't start at all
Also, the listener.ora is not created in $ORACLE_HOME/network/admin location but in ERPNODE1_PROD

[ErmanArslansOracleBlog]

In EBS DB environments, listeners and tns file are created in $TNS_ADMIN directory.. (not the usual $ORACLE_HOME/network_/admin)

Read my book for further info on this -> Practical Oracle E-Business Suite

You have a db connection problem in general..
your listener is configured to listen on erpnode1.XXX.com , PORT=1521 .. But you get TNS-00511: No listener error mostly.

TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused

Send me the following files->

/u01/oracle/PROD/12.1.0/network/admin/PROD_erpnode1/listener.ora
/u01/oracle/PROD/12.1.0/admin/PROD_erpnode1/diag/tnslsnr/erpnode1/prod/alert/log.xml
/u01/oracle/PROD/12.1.0/appsutil/log/PROD_erpnode1/12311825/adconfig.log
output of hostname command

Note that: You must hide your important information about your environment, while sending the files to me, I do not accept any responsibility for this. I m just trying to help..

[Muazzam]

Here are the files
1. listener.ora file EAlistener.txt

2. log.xml EAlogxml.txt

3. adconfig.log EAadconfig.txt

Note: Please note that this info is not sensitive as it is test, not production and tested on different location/platform. Thanks for your concern and your help as well

[ErmanArslansOracleBlog]

1)switch user (su) to the Oracle RDBMS OS user, set "your environment properly" and send me the following command outputs->

lsnrctl status
ps -ef |grep inh
telnet your_hostname 1521
tnsping PROD
sqlplus apps/<apps_password>@your_hostname:1521/PROD
sqlplus apps/<apps_password>@PROD

2) Send me the link to the document that you follow for this task , and also state in which part  you get this error.

[Muazzam]

Please clarify who is  "Oracle RDBMS OS user". I'm sorry, as I'm new to EBS environment I'm not aware.

Anyways, I tried all the commands as su - oracle and except ps command, all the commands are showing only "command not found" errorEAError.PNG

[ErmanArslansOracleBlog]

As I already mentioned, you need to set the proper shell "environment" after switching to oracle user.
Only then, you will able to execute those commands.

[Muazzam]

1. lsnrctl status
LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 07-JAN-2020 18:22:15

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused

2. ps -ef |grep inh 
oracle    5645  5420  0 18:23 pts/1    00:00:00 grep --color=auto inh


3. telnet erpnode1.oralabs.com 1521
Trying <ip address>...
telnet: connect to address 192.168.23.2: Connection refused (can ping the ip normally)


4. tnsping PROD 

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 07-JAN-2020 18:31:41

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:

Used HOSTNAME adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=IP)(PORT=1521)))
TNS-12541: TNS:no listener

5. sqlplus apps/<passwd>@ernode1.oralabs.com:1521/PROD

SQL*Plus: Release 12.1.0.2.0 Production on Tue Jan 7 18:34:17 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ERROR:
ORA-12541: TNS:no listener (Prompting for username after this)

6. sqlplus apps/<passwd>@PROD

SQL*Plus: Release 12.1.0.2.0 Production on Tue Jan 7 18:35:30 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ERROR:
ORA-12541: TNS:no listener (Again, prompting for username after this)

[ErmanArslansOracleBlog]

You are copy&pasting the IP Addresses here!
Again. Remember -> You must hide your important information about your environment, while sending the files to me, I do not accept any responsibility for this. I m just trying to help..

So, you were trying to install EBS and then you got error and stop there.
At this point, your listener should be up. (unless you deleted the incomplete installation files or reboot your db node..)

However; as I see, your listener is not running.
That's a problem.

Run the following commands with oracle OS user (in EBS DB Server) and see if your listener is started properly->  
First set your TNS_ADMIN accordingly, and then run->

export TNS_ADMIN=blabla (directory pointing EBS's tnsnames.ora files)
lsnrctl start
lsnrctl status
ps -ef |grep inh
tnsping PROD

This is a test to see whether there is a problem preventing your listener to start or not..

Also, as I said earlier -> *) Send me the link to the document that you follow for this task , and also state in which part  you get this error.