We are facing slowness in ERP application

[Mohammed Kamran]

Hi,

we have checked from the database end awr reports and addm reports and all checks for database side all looks fine in DB end but how to we find what is causing the issue in application server and why it is running slow.

Slowness in below.

1. If approver  is trying to approve the request it is taking more time and says loading in the tab of browser waiting for ebs prod.
2. If any reports are been pulled it is not very slow and end it says Apache bride connection error.
3. Applying any leave in employee self service it also take time.
4. there is no specific time for its slowness its all the time after working hours as well.

Please let me know what needs to be checked and what may causing the issue.

[ErmanArslansOracleBlog]

Did you check the application server?

1)check the load and check all the things possible in OS tier..
2)did you check the application server? I don't know the EBS version of yours, but first thing is to check the application server and its configuration.. (EBS 12.2 -> weblogic, EBS R12 and 11i -> IAS)

[Mohammed Kamran]

Hi,

can you please suggest what needs to be checked in OS level.

Version of EBS is 12.2.4

[ErmanArslansOracleBlog]

Os levels check.. You know..  load averages, /var/log/messages and etc...

Aha you are on version 12.2. So your application server is Weblogic.

Of course, you need to check the Weblogic side, as well.

Managed server counts and heap size especially..

You need to tune your application server, according to your load (total user count and concurrent user count)

[ErmanArslansOracleBlog]

Also, I suggest you to enable trace for your problematic processes.
For ex: if you have a slow OAF page, then enable OAF trace..
Likewise, if you have a slow concurrent request, enable trace and check the trace.. Try to find the cause of the slownes..
Likewise, if you have a slow forms, enable forms trace + FRD trace..

You got the idea, right..

[Mohammed Kamran]

/var/log/messages

[root@u01vpodb01 log]# cd messages
-bash: cd: messages: Not a directory
[root@u01vpodb01 log]# cat messages
May 13 03:49:02 u01vpodb01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="5065" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
May 15 22:27:29 u01vpodb01 vmpscan[22195]: Start:DoHostScan
May 15 22:27:29 u01vpodb01 vmpscan[22200]: Using default script settings and running as root
May 15 22:27:30 u01vpodb01 vmpscan[22334]: Start:os
May 15 22:28:10 u01vpodb01 vmpscan[27194]: End:os
May 15 22:28:10 u01vpodb01 vmpscan[27196]: Start:net
May 15 22:28:13 u01vpodb01 modprobe: FATAL: Error inserting bridge (/lib/modules/3.8.13-16.2.1.el6uek.x86_64/kernel/net/bridge/bridge.ko): Unknown symbol in module, or unknown parameter (see dmesg)
May 15 22:28:13 u01vpodb01 kernel: bridge: Unknown symbol ipv6_dev_get_saddr (err 0)
May 15 22:28:23 u01vpodb01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
May 15 22:28:24 u01vpodb01 vmpscan[29114]: Start:DoHostScan
May 15 22:28:24 u01vpodb01 vmpscan[29122]: Using default script settings and running as root
May 15 22:28:24 u01vpodb01 vmpscan[29196]: End:net
May 15 22:28:24 u01vpodb01 vmpscan[29199]: Start:storage
May 15 22:28:24 u01vpodb01 vmpscan[29297]: Start:os
May 15 22:28:37 u01vpodb01 vmpscan[31277]: End:DoHostScan:Elapsed=65 seconds
May 15 22:28:59 u01vpodb01 vmpscan[913]: End:os
May 15 22:28:59 u01vpodb01 vmpscan[918]: Start:net
May 15 22:29:01 u01vpodb01 kernel: bridge: Unknown symbol ipv6_dev_get_saddr (err 0)
May 15 22:29:01 u01vpodb01 modprobe: FATAL: Error inserting bridge (/lib/modules/3.8.13-16.2.1.el6uek.x86_64/kernel/net/bridge/bridge.ko): Unknown symbol in module, or unknown parameter (see dmesg)
May 15 22:29:11 u01vpodb01 vmpscan[1992]: End:net
May 15 22:29:11 u01vpodb01 vmpscan[1997]: Start:storage
May 15 22:29:23 u01vpodb01 vmpscan[3449]: End:DoHostScan:Elapsed=57 seconds
May 16 01:25:37 u01vpodb01 auditd[5025]: Audit daemon rotating log files
[root@u01vpodb01 log]# pwd
/var/log
[root@u01vpodb01 log]#

[ErmanArslansOracleBlog]

/var/log/messages is just a simple check. You check it to whether you have ai important OS error or not.. it looks okay.

I recommend you to follow the approach, below;

You are on version 12.2. So your application server is Weblogic.
Of course, you need to check the Weblogic side, as well.
Managed server counts and heap size especially..
You need to tune your application server, according to your load (total user count and concurrent user count)

Also, I suggest you to enable trace for your problematic processes.
For ex: if you have a slow OAF page, then enable OAF trace..
Likewise, if you have a slow concurrent request, enable trace and check the trace.. Try to find the cause of the slownes..
Likewise, if you have a slow forms, enable forms trace + FRD trace..

[Mohammed Kamran]

How do I check these two below if you can give some high level steps.

Managed server counts and heap size especially..
You need to tune your application server, according to your load (total user count and concurrent user count)

[Mohammed Kamran]

attached weblogic counts

[ErmanArslansOracleBlog]

So you are sure that you don't have a high load in your Operating Systems/nodes..

You have multiple 3 oacore and 3 forms server.. So this seems to be crowded system.. A loaded one.
So your councurrent user count and total user count seems high. (as Oracle recommends up to 150-200 concurrent users per JVM/managed server)

First review your heap sizes; especially for oacore..

Review this note.
Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2 (Doc ID 1905593.1)
 
Check the heap sizes, recommended Xmx is 2GB..

Get help from your Weblogic admin/EBS 12.2 admin(APPS DBA)

Getting every detail, and sending the action plan for every little thing is beyond the scope of this forum.

If your heap sizes are okay, continue with the previous action plan that I sent you earlier.
Try to find the root cause of this performance problem.. After finding the root cause, it is easy to fix.

[Mohammed Kamran]

Hi,


We found below for leak connections-
oacore_server1 -- 1 leak connection
oacore_server2 -- 2 leak connections

Are these patches important and needs to be applied for sure for ERP slowness?

Patch: 19494816 - connection leak on ErrorStack.java
Patch: 24760532 - CONNECTION LEAKS @ ORACLE.APPS.FND.OAM.SERVLET.UI.HANDLERS.OAMLOGIN23624806> (supersedes 23624806 and 20763474>supersedes 19807163) - transactionscope - connection leak in 12.2.4
Patch: 20026276 (supersedes 19858552) - NULLPOINTEREXCEPTION ON OAERRORPAGE AFTER 19858552
Patch: 21619282 (supersedes 21270508) - OOB 20526782 CONN.ISCLOSED() CHECK IN OACONNECTIONPOOLMANAGERIMPL.JAVA
Patch: 20027711 R12.BNE.C LEAKED CONNECTION BNEOASESSIONUTILS
Patch: 20823641 GET A PROFILE LEAVES AN OPEN CONNECTION BEHIND
Patch: 22512856 BneBaseBajaContext CONNECTION LEAK
Patch: 19245366 R12.ATG_PF.C.delta.5 Details


 Patch 10169939 is recommended on weblogic server

[ErmanArslansOracleBlog]

This is again very generic.

applying patches is a good thing, but need to be tested in a test environment first.

I don't think your issue is related with that 1 leak connections, but here is a document for you ->

How To Detect a Connection Leak Using Diagnostic JDBC Dumps (Doc ID 1502054.1) -- to collect an image after enabling JDBC Leak Diagnostics.

I already sent you the way for diagnosing this slowness.

You need to start your investigation by analyzing the OS and its load.. Then continue with the Oracle HTTP Server and the WEBLOGIC -- "managed server stability and performance"

Also check your db tier..

for weblogic side, you can the following command to determine your patch levels and after that you may consider increasing your patch levels accordingly->

$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -contextfile=$CONTEXT_FILE -appspass=<appspassword> -outfile=$APPLTMP/Report_App_Inventory.html

[Mohammed Kamran]

Thanks for your suggests and help but I am bit confused from where to start for the below as we don't have OS Linux support we are using OEL 6.5 if you suggest what needs to be checked the I can start from there.
YYou need to start your investigation by analyzing the OS and its load..

[ErmanArslansOracleBlog]

Check your OS performance and stability.

Check its load, check its free memory, check its I/O performance, check its error logs and etc..

First, ensure that it is not caused by OS..

Then narrow down the issue. Continue with FMW (Weblogic and so on) and DB tier.. (DB performance -- AWR, DB OS load analysis and so on)