Requests in errors

[latifa]

Hi Erman,

Customer has 115 AR requests finished with errors Following:

XLAACCPB: Create Accounting
...

Concurrent Manager encountered an error while running the spawned
concurrent program oracle.apps.xdo.oa.cp.JCP4XDODataEngine for your
concurrent request 24388528. usdsop ne peut pas créer de nouveau
traitement

Cause : usdsop detected an error while creating new request …

I looked in the database alert_log however no errors found. In oracle support I couldn't find Something solving ths issue.

Could you help please.

Regards,
Latifa

[latifa]

Hi again,

Customer has resubmitted requests and he has got errors again; some of them succeeded and other not.

hs_err_pid6841.log

Regards,
Latifa

[latifa]

Shull we close the application and reboot the system ?

[ErmanArslansOracleBlog]

Hi Latifa,

Acutally it is already stated that ->

#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=

So, did you check the memory and swap usage?

Your problem is clear..
java wants memory and OS doesn't give that memory to java..
your problem is in java's native memory. (not the heap spaces)
Decreasing the heap space (by the argument -xms and -xmx) and leaving that space for the native memory may be a good idea.
Larger code cache may be a good idea as well -> -XX:ReservedCodeCacheSize= (Setting it as concurrent program argument)

However; it is better to check it with your OS admin. ensure you have enough memory space on your server.
Ensure there is nothing that blocking java for getting enough memory.

[latifa]

Hi Erman,

Actually we maintain rebooting applicatives servers. However, as we have 2 CM servers (Master and Slave), top informations of the system shows that Slave server is not so much busy:

Master:

load averages:  12.6,  12.4,  12.6;               up 54+01:49:31       16:27:16
410 processes: 250 sleeping, 150 zombie, 10 on cpu
CPU states: 84.4% idle, 12.9% user,  2.8% kernel,  0.0% stolen,  0.0% swap
Kernel: 33246 ctxsw, 17596 trap, 12063 intr, 31703 syscall, 89 fork, 10196 flt
Memory: 48G phys mem, 13G free mem, 48G total swap, 9028M free swap

Slave:

load averages:  2.30,  2.54,  2.71;               up 54+01:49:33       16:27:54
293 processes: 292 sleeping, 1 on cpu
CPU states: 98.2% idle,  0.8% user,  1.0% kernel,  0.0% stolen,  0.0% swap
Kernel: 23393 ctxsw, 2807 trap, 16729 intr, 25007 syscall, 5 fork, 1732 flt
Memory: 48G phys mem, 42G free mem, 48G total swap, 41G free swap

So we will move exécute requests of this Concurrent Manager on the Slave server rather than the Master. Do you think that it will be enough ?




Regards
Latifa

 

[ErmanArslansOracleBlog]

It seems like a good plan. Second server seems idle in terms of memory as well.
That move is already mapped to one of the suggested actions for solving this problem -> "Reduce memory load on the system"

[latifa]

Sometimes the solution is so close to our nose and we look for somewhere far :)

Let's see batchs's results of this night :)

Thank you Erman for your advice.

Regards,
Latifa

[ErmanArslansOracleBlog]

Yes :) It s still good, as you saw that opportunity to migrate those concurrent managers to the idle node as a quick win :)

Lets see..

12 Kas 2020 Per 20:11 tarihinde latifa [via Erman Arslan's Oracle Forum] <[hidden email]> şunu yazdı:

Sometimes the solution is so close to our nose and we look for somewhere far :)

Let's see batchs's results of this night :)

Thank you Erman for your advice.

Regards,
Latifa

---

If you reply to this email, your message will be added to the discussion below:

http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/Requests-in-errors-tp8989p8996.html

To start a new topic under EBS 12.2, email [hidden email]
To unsubscribe from Erman Arslan's Oracle Forum, click here.
NAML

[latifa]

Hi Erman,

Again our requests some of them failed and others succeded !!

Same errors !!

Regards,
Latifa

[latifa]

You said:

Decreasing the heap space (by the argument -xms and -xmx) and leaving that space for the native memory may be a good idea. : which values ? in which server ?

Larger code cache may be a good idea as well -> -XX:ReservedCodeCacheSize= (Setting it as concurrent program argument)  : which value ?

Thank you in advance for your advice

Regards,
Latifa

[ErmanArslansOracleBlog]

We do it in the concurrent program definition page.
Concurrent -> Program -> Define ->  Search for the following concurrent program
For the concurrent program;
You check the values in the Options field of Executable region. So if they are high, decrease them a little bit.

Try adding -XX:ReservedCodeCacheSize=512m to that field as well..

You will have Something like the following there

-xms1024m -xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=512m


 

[latifa]

Hi Erman,

Options of the program creating comptabilisation are:

-Xms32m -Xmx128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid%p.hprof

so if I follow what you said, we should have more arguments like this:

-Xms32m -Xmx128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid%p.hprof  -XX:ReservedCodeCacheSize=512m

?

[ErmanArslansOracleBlog]

Yes.. You may increase your Xms and Xmx too.. It seems they are set to very low values.. (it may be better to make those xms and xmx values doubled)

[ErmanArslansOracleBlog]

Actually, one of the suggestions for getting rid of this problem is to reduce xmx and xms values, but your Xmx and xms values are already so low. Anyways, lets don't touch them and just add those additional parameters that I sent to you.

[ErmanArslansOracleBlog]

I would also try adding this one -> -XX:MaxPermSize=1024m

[latifa]

Is it Ok : -Xms32m -Xmx128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid%p.hprof  -XX:ReservedCodeCacheSize=512m  ?

[ErmanArslansOracleBlog]

yes. try..

[latifa]

Erman,

These requests were Ok before 10/11/2020 and I think that Something changed:

 COUNT(*) TRUNC(FNC S
---------- --------- -
       127 01-NOV-20 C
       277 02-NOV-20 C
       254 03-NOV-20 C
       297 04-NOV-20 C
       273 05-NOV-20 C
       289 06-NOV-20 C
       121 07-NOV-20 C
       111 08-NOV-20 C
       275 09-NOV-20 C
       215 10-NOV-20 C
         5 10-NOV-20 G
       115 10-NOV-20 E
       126 11-NOV-20 C
        57 12-NOV-20 E
       319 12-NOV-20 C

How can we track this ?

Regards,
Latifa

[ErmanArslansOracleBlog]

Check the parameters of the concurrent request.. Maybe they use extra parameters now, or they changed the parameters and then the concurrent started to requests lots of memory.
Check your server side. Check if there is a new conf that may cause the memory or swap to be full.

[latifa]

Hi Erman,

I am really sorry for disturbing you because I did mistake checking requests. I checked all the rquests happening on 12/11/2020 and the failing requests were those ran in the day time, so before the server move and reboot applicative servers :(

I apologise :)

Thank you very much for your patience and your help.

Regards,
Latifa