Oracle EBS Application not starting

[latifa]

Hi Erman,

I cloned a new environment for oracle EBS 12.1.3 and DB 12c RAC & ASM.
I have 3 OAS & 3 CM servers (Linux).
In the begining I configured all the system with load balancer --> this issue was occuring.
After that I run : exec fnd_conc_clone.setup_clean ;
and I configured only one OAS and CM to check server by server.

Cloning was Ok without errors. However while starting application on OAS1, I have got error messages:

Apache, Oacore, forms and oafm --> failed to start:

Apache error:
*******************

ERROR : Timed out( 200000 ): Interrupted Exception

--------
17/11/21 11:54:50 Start process
--------
/exec/products/OP11ZNCE/nce/OA/12.1/inst/apps/OP11ZNCE_op11zas1/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd

--------
17/11/21 11:56:52 Stop process
--------
.../Apache/Apache/bin/apachectl stop: httpd (no pid file) not running

Oacore error:
********************
17/11/21 11:56:30 Start process
--------
17/11/21 11:56:31 WARNING: ApplicationLogManager is not installed, may result in loader leaks. Set -Djava.util.logging.manager=oracle.classloader.util.ApplicationLogManager
2.714: [GC 33024K->10997K(125952K), 0.0331490 secs]

forms error:
******************

Error
--> Process (index=1,uid=2073649726,pid=21151)
    time out while waiting for a managed process to start
    Log:
    default_group~forms~default_group~1.log
No error message:
17/11/21 11:59:52 Start process
--------
0.551: [GC 16896K->6704K(63232K), 0.0061220 secs]
1.105: [GC 23600K->11254K(80128K), 0.0077110 secs]


oafm error:
******************

Error
--> Process (index=1,uid=2073649695,pid=27099)
    time out while waiting for a managed process to start
    Log:
    default_group~oafm~default_group~1.log

no error in this file:
17/11/21 12:01:30 Start process
--------
1.237: [GC 16896K->6873K(63232K), 0.0073390 secs]
2.805: [GC 23769K->11283K(63232K), 0.0183780 secs]


Any advise for this issue ?

Thank you in adavnce for your help.

Regards,
Latifa

[ErmanArslansOracleBlog]

What is written in OHS log files?

[latifa]

>>I have no log files in :

$INST_TOP/logs/ora/10.1.3/Apache.

Only empty sec_debug.log & sec_audit.log.
No error_log and access.log files !

>>in $INST_TOP/logs/ora/10.1.3/opmn/, I have this one but
opmn.log
----------
17/11/21 12:35:48 [pm-requests] Request 1d Completed. Command: /shutdown
17/11/21 12:35:48 [pm-requests] Http 1d
OP11ZNCE_op11zas1.op11zas1.idf.fr.ftgroup~HTTP_Server~HTTP_Server~HTTP_Server~2073649721:19588
  Status: Stop
  Operation: request (time out while waiting for a managed process to stop)
  ErrFile: /exec/products/OP11ZNCE/nce/OA/12.1/inst/apps/OP11ZNCE_op11zas1/logs/ora/10.1.3/opmn/HTTP_Server~1.log

17/11/21 12:35:48 [pm-process] Stopping Process: HTTP_Server~HTTP_Server~HTTP_Server~1 (2073649721:19588)
17/11/21 12:35:48 [libopmnohs] Forcefully Terminating Process: HTTP_Server~HTTP_Server~HTTP_Server~1 (2073649721:19588)
17/11/21 12:40:48 [pm-scheduler] Sched SyncExit: monitor timed out
17/11/21 12:40:49 [internal] Logging disabled


Regards,
Latifa

[ErmanArslansOracleBlog]

What is written in /exec/products/OP11ZNCE/nce/OA/12.1/inst/apps/OP11ZNCE_op11zas1/logs/ora/10.1.3/opmn/HTTP_Server~1.log ?

Also, please follow this doc to enable logging for Apache and send me the relevant log files:

How To Enable Apache, OC4J and OPMN Logging In Oracle E-Business Suite R12 (Doc ID 419839.1)

[ErmanArslansOracleBlog]

Check this note as well:

Oracle HTTP Server Fails to Start With the Error "Parent process has died... exiting" (Doc ID 821419.1)

[latifa]

Hi Erman,

As I didn't find what was wrong I decided to reclone the OAS1 (only one server without loadbalancer) and after that server by server and I'll finish by the load balancer. Let see if the issue will remain ... I think it's more clean.

Regards,
Latifa

[latifa]

Hi Erman,

The issue is remaining even after recloning !

The error message is same:

for apache:
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adapcctl.sh version 120.7.12010000.2
Starting OPMN managed Oracle HTTP Server (OHS) instance ...
opmnctl: opmn is already running.
opmnctl: starting opmn managed processes...


for oacore:
ERROR : Timed out( 200000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance  ...
for forms:
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh  version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance  ...


for oafm:
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Starting OPMN managed OAFM OC4J instance  

for FORMS-C4WS:
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adforms-c4wsctl.sh version 120.0.12010000.3
Starting OPMN managed FORMS-C4WS OC4J instance


Could it be ports problems ? some conflict ? The other environnment I cloned before has the same ports. (I stopped it but no change).

Regards,
Latifa

[ErmanArslansOracleBlog]

These outputs are not enough for solving it.

please follow this doc to enable logging for Apache and send me the relevant log files:

How To Enable Apache, OC4J and OPMN Logging In Oracle E-Business Suite R12 (Doc ID 419839.1)


Bytheway, I ll be on a vacation till 28.11.2017.. I will have limited access to this.

[latifa]

Hi Erman,

I am sorry, I didn't configure logging mode. I'll do it if we continue to not have informations.
I prefered to reclone the system because it's a production environment. After that I could access to the application. However, the problem appeared again after installing some webservices.

Actually I have access logs and an error logs files.

logs_26_28_112017.zip

 Is it the same issue as previously where we should compile jsp files ?

Regards,
Latifa

[latifa]

Also one more ifnromation, I have a lot of <defunct> processes that I can't kill even with "kill -9".

[latifa]

I configured logging in httpd.conf to debug and I could see an error while

./adopmnctl.sh stop

Error message in default_group~oacore~default_group~1.log:

Error: Could not connect to the remote server. Please check if the server is down or the client is using invalid host, ORMI port or password to connect: Connection refused (Connection refused)

here it is the corresponding logs:

Logs3.7z

Regards,
Latifa

[ErmanArslansOracleBlog]

First start with OHS..
You have the following key errors in the logs;

AppsLogin File does not exists
Input/output error: fcntl: F_SETLKW: Error getting accept lock, exiting!  Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[libopmnohs] Forcefully Terminating Process:
[alert] Child 21066 returned a Fatal error... \nApache is exiting!


Please describe this env->

1)OS vendor and version.
2)Filesystem type
3)Multi node- single node?

*Check this note as well ->Unable To Open The E-Business Suite Login HTTP-404 Error. Error_log Shows "File does not exist: $INST_TOP/portal/OA_HTML/AppsLogin" (Doc ID 1177264.1)

[latifa]

Hi Erman,

1) OS Vendor:

uname -a
Linux op11zas1 4.1.12-94.2.1.el7uek.x86_64 #2 SMP Wed Apr 26 15:32:38 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux

2) it's an NFS file system

3) Multi-node system:
3 OAS and 3 CM

The Doc ID is the same as I ran previously to compile Jsp's ...

[ErmanArslansOracleBlog]

1)Is therer any problems in NFS layer?
2)Do you have AppsLogin in the correct place of the filesystem?

[latifa]

Hi Erman,

1°) I applied the note 1177264.1 but no changes ... the issue is remaining.

2°) What do you mean by "AppsLogin in the correct place of the filesystem" ?

Regards,
Latifa

[latifa]

if you mean the user with which we connect to the application in Linux machines, how to check that ?

[ErmanArslansOracleBlog]

I mean do you have the appslogin file present?

[latifa]

No it doesn't exist

[latifa]

Hi Erman,

About the NFS, I heard that there is a bug on NFS server which disturbs the function of RPC service. It needs to be patched and they are waiting the GO and schedule for patching the NFS server.

I think without this patch the issue will remain actual. What do you think ?

Regards,
Latifa

[latifa]

Hi Erman,

How to check RPC service for Oracle EBS ? What is the role of RPC in EBS in order to be blocked like that if it's the case ?

Regards,
Latifa