out of memory error in alert log file.

[shahzad53]

Dear Ersalan,

Today we have found that we have lot of error in alert log file, error are below .

In the error , it has shown the trace file . please check below trace file and advice.

We have 11.2.0.4 database, and ebs 12.1.3 application.

Errors in file /d01/oracle/PROD/db/tech_st/11.2.0/admin/PROD_proddb22/diag/rdbms/prod/PROD/trace/PROD_ora_1747 9.trc (incident=2617113):
ORA-04030: out of process memory when trying to allocate 8232 bytes (TCHK^bcb6174a,kggec.c.kggfa)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /d01/oracle/PROD/db/tech_st/11.2.0/admin/PROD_proddb22/diag/rdbms/prod/PROD/trace/PROD_ora_1747 9.trc (incident=2617114):
ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri May 15 20:30:19 2020
Sweep [inc][2617114]: completed
Sweep [inc][2617113]: completed
Fri May 15 20:34:34 2020


I have attached the trace file with this post, Kindly check & advice us.

Regards,
shahzadPROD_ora_17479.trc

[ErmanArslansOracleBlog]

*You should start checking the OS limits of Oracle User.
it may be related with the max data memory OS limit  for Oracle OS user. (ulimit..)
*Also check syslog for swap space or /tmp full errors -> Increase swap or /tmp space if needed.
*Check the PGA limits as well -> You may need to tune that as well. Maybe you are using lots of memory per process and consume the OS memory resources aggresively..
*There are bugs as well..

Note that, "this is very generic. We need to see those trace files to be more specific."

[shahzad53]

Hi Erman,

Continue found ora-04030 error in production, i have attached the trace file as you have mention.

Errors in file /d01/oracle/PROD/db/tech_st/11.2.0/admin/PROD_proddb22/diag/rdbms/prod/PROD/trace/PROD_ora_13393.trc  (incident=2641444):
ORA-04030: out of process memory when trying to allocate 32792 bytes (QERGH hash-agg,qeshHDAlloc.3)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon May 25 02:10:58 2020
Sweep [inc][2641444]: completed

check below ulimit, free -m command

[oracle@proddb22 bkup]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2918259
max locked memory       (kbytes, -l) 134217728
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16384
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[oracle@proddb22 ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:   364813     363240    1572     115651        134     335537
-/+ buffers/cache:      27568     337245
Swap:        65535       4287      61248


SQL> show parameter pga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target                 big integer 10G
SQL>

kindly help me to solve the issue.


TRACE file are attached.

Regards,
shahzad


PROD_ora_13393.trc

[ErmanArslansOracleBlog]

This may be related with a bug as well.
I need to see the trace.


*Connect the database using sqlplus "/as sysdba"
*SQL> ALTER SYSTEM SET EVENTS '4030 trace name heapdump level 536870917;name errorstack level 3';
*Reproduce the error.. After reproducing the error , continue with the next step->
*SQL>ALTER SYSTEM SET EVENTS '4030 trace name context off; name context off';

Send me the relevant trace file, so I can review.