question on oacore processes

[Sri]

Hi Erman,

Today we faced a issue in our test environment.we could see 2 oacore processes are consuming high cpu(more than 150%) and we are not able to login to EBS.How to check if the process was doing any garbage collection during that time

Thank you

[ErmanArslansOracleBlog]

If they are spinning, there should be a clear view of that problem in your oacore logs.
Check oacore log and out files first.

[Sri]

oacore.txt

Hi erman,

uploaded the log and output files at the time of issue.can you please let us know what would be the reasons behind this

Thank you for all your support

[ErmanArslansOracleBlog]

You have db connection problem.
Increase your sesssion and process limit.
also consider increasing data source limit.

Finding:
oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.

[Sri]

Thank you very much for your timely help erman

[Sri]

Hi Erman,

We again face the issue now.i dont think the issue is with processes.please see the output from the instance

same errors which i uploaded earlier

SQL> select RESOURCE_NAME,CURRENT_UTILIZATION,MAX_UTILIZATION,LIMIT_VALUE from v$RESOURCE_LIMIT where resource_name in ('processes','sessions');

RESOURCE_NAME                  CURRENT_UTILIZATION MAX_UTILIZATION
------------------------------ ------------------- ---------------
LIMIT_VALUE
----------------------------------------
processes                                      224             432
       800

sessions                                       235             452
      1224


SQL>

[ErmanArslansOracleBlog]

Do you have the same error in oacore log?

Finding:
oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.

[ErmanArslansOracleBlog]

There can be other causes for that as well..

For ex:

Unable To Start Service Components and Workflow Services In Release 12.2.4 - ORA-28001: the password has expired (Doc ID 2052312.1)
Users Receive Error "Exception: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object" During Login To E-Business Suite (Doc ID 1485112.1)
Random oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException Occurs For Login (Doc ID 1937879.1)

You have definetly a db connection problem. You should be checking all the components during the path from apps to db.
Connection pool - db listener - db processes - dbc files and all the other related things which can be used for connecting from apps to db.. Get the idea?

[Sri]

Thanks for your time erman,i have checked the noteids.These are not applicable

But I agree with you there might be the problem with connections.we are bouncing the server when we face the issue and it is getting fixed

From your last updates,I think datasource capacity needs to be increased.How can we increase the datasource capacity in r12.2,i mean do we need it to change anything in dbc file or in weblogic console?

Thank you

[Sri]

i have earlier attached the oacorelogfiles to this post..shortly below are the errors from oacores

oacore_server1 Log

####<9 Nov, 2017 7:15:02 PM IST> <Error> <ServletContext-/OA_HTML> <erpsupport.ttd.com> <oacore_server1> <[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <005NOnk8P7rBT8GqywaeMG0007n00005aW> <1510235102073> <BEA-000000> <chain failed
java.net.SocketException: Connection reset

oacore_server1 out

Caused By: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.

oacore_server2 log

####<9 Nov, 2017 7:18:02 PM IST> <Info> <ServletContext-/OA_HTML> <erpsupport.ttd.com> <oacore_server2> <pool-3-thread-3> <<anonymous>> <> <005NOm_X2jBBT8GqywaeMG0007n000001W> <1510235282359> <BEA-000000> <oracle.apps.fnd.login.LoginHelper.getConnection:END: Connection from WLS pool weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@7474>

oacore_server2 out

Caused By: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.

[Sri]

Hi Erman,

We have the issue again and we could see the java heap space error in logs.can you please suggest what can be done in this case?

[ErmanArslansOracleBlog]

Consider increasing the Xmx , Xms and permsize of the managed server(oacore in this case)

[Sri]

Thanks for your update erman,This issue happened when there are very few users around 10.Uploaded the problematic oacore logs and trace.can we identify from attached logs what would have caused thisstack.zip

[ErmanArslansOracleBlog]

You have the following:

*Exception creating new Poolable object.
*Connection reset at java.net.SocketOutputStream.socketWrite
*java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method)

1)Check your db and listener.
2)Check your network
3)Ask your users.. They should not cancel their requests while they are running. ( I am not talking about concurrent request here.. I m talking about web related request.. in oaf screens)

Looking to your jstack outputs, you have lots of BLOCKED threads.

Here for example:

 java.lang.Thread.State: BLOCKED (on object monitor)
        at com.bea.logging.RotatingFileStreamHandler.publish(RotatingFileStreamHandler.java:74)
        - waiting to lock <0x00000000c0b351c0> (a weblogic.logging.FileStreamHandler)
        at java.util.logging.Logger.log(Logger.java:616)
        at com.bea.logging.BaseLogger.log(BaseLogger.java:51)
        at weblogic.logging.WLLogger.log(WLLogger.java:40)
        at weblogic.logging.j2ee.ServletContextLogger.log(ServletContextLogger.java:84)

Even your connection pool:

pool-2-thread-1" prio=10 tid=0x00007fcfb68c1000 nid=0x248c waiting for monitor entry [0x00007fcff4dcf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at oracle.jdbc.driver.T4CConnection.doPingDatabase(T4CConnection.java:4745)
        - waiting to lock <0x00000000c6c3ad70> (a oracle.jdbc.driver.T4CConnection)
        at oracle.jdbc.driver.PhysicalConnection.pingDatabase(PhysicalConnection.java:12722)
        at oracle.jdbc.driver.PhysicalConnection.pingDatabase(PhysicalConnection.java:12781)
        at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)

You see where your pool thread is waiting ->  at oracle.jdbc.driver.T4CConnection.doPingDatabase

So if your db is okay, if it is not the cause, if your listener is not refusing anyting during the problematic period and if your network is all okay (not dropping packages , not limiting something, not breaking something), then consider the following action plan ->

Read this -> "Stuck Threads On DoPingDatabase (Doc ID 2271927.1)"
Consider -> "Upgrading Oracle Fusion Middleware WebTier of Oracle E-Business Suite Release 12.2 to the latest 11gR1 (11.1.1.x) PatchSet (Doc ID 1590356.1)" for upgrading FMW to 11.1.1.9.0 -- if it is not already 11.1.1.9.0

So for this, I recommend you to apply "Patch 20141119 on top of FMW 11.1.1.9.0."

The document above also says: "If using iRecruitment and intermittently observing stuck threads related with JDBC pingdatabase, even with Patch:20141119, when calling OAError.jsp and trying to open a new connection using gateway userid,  please apply EBS R12 Patch:25969099."

[Sri]

Thank you very much for prompt response working on it

[PBK]

How did you fix this issue?

[ErmanArslansOracleBlog]

We have recommended the solution, but we didn't get the news from Sri.

PBK, consider the actions that I sent in my last update.
If you have a similar problem, please create a seperate issue by referencing this one. Give us the required information, and let's see what we can do.