Erman Arslan's Oracle Forum - Re: RAC instance crash

Re: RAC instance crash

Posted by Jabbar on Nov 25, 2019; 10:47am
URL: http://erman-arslan-s-oracle-forum.124.s1.nabble.com/RAC-instance-crash-tp7856p7889.html

--- LMHB (ospid: XXXX): terminating the instance due to error 29770 (Doc ID 2438319.1)

Yes i checked this note. _lm_enable_aff_benefit_stats parameter is also false our database.

Now i load trace and incident files:

TRACE_FILE:

*** 2019-11-15T14:50:20.420113+04:00 (CDB$ROOT(1))
==============================
IPC0 (ospid: 368909) has not moved for 22 sec (1573815015.1573814993)
: heartbeat check status 2 (acceptable) (threshold 70 sec)
: heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10
: heartbeat state 0x2.ffff () pso-flag 0x0
: Not in wait; last wait ended 17 secs ago.
: last wait_id 3106025 at 'rdbms ipc message'.
==============================
Dumping PROCESS IPC0 (ospid: 368909) States
==============================
===[ Callstack ]===

*** 2019-11-15T14:50:20.420233+04:00 (CDB$ROOT(1))
Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909,
pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 14:50:19 ]
NOTE: scheduling delay has not been sampled for 0.488611 secs
0.000000 secs from [ 14:50:15 - 14:50:20 ], 5 sec avg
0.000000 secs from [ 14:49:20 - 14:50:20 ], 1 min avg
0.000000 secs from [ 14:45:20 - 14:50:20 ], 5 min avg

*** 2019-11-15T14:50:22.767007+04:00 (CDB$ROOT(1))
loadavg : 8.14 5.28 4.22
System user time: 0.07 sys time: 0.02 context switch: 61402
Memory (Avail / Total) = 2945.37M / 772256.64M
Swap (Avail / Total) = 24000.20M / 24576.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:07:30 ora_ipc0_cdbiamas1
Name: ora_ipc0_cdbiam
State: R (running)
Tgid: 368909
Ngid: 0
Pid: 368909
PPid: 1
TracerPid: 0
Uid: 1001 1001 1001 1001
Gid: 1001 1001 1001 1001
FDSize: 128
Groups: 1001 1002
NStgid: 368909
NSpid: 368909
NSpgid: 368909
NSsid: 368909
VmPeak: 668175752 kB
VmSize: 668175748 kB
VmLck: 0 kB
VmPin: 562562088 kB
VmHWM: 7413828 kB
VmRSS: 7413828 kB
VmData: 8388 kB
VmStk: 156 kB
VmExe: 343516 kB
VmLib: 32424 kB
VmPTE: 14924 kB
VmPMD: 2172 kB
VmSwap: 0 kB
Threads: 1
SigQ: 1/3088249
SigPnd: 0000000000000800
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000016400207
SigCgt: 00000003c9887cf8
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
Seccomp: 0
Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-111
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4588970
nonvoluntary_ctxt_switches: 11610

*** 2019-11-15T14:50:25.421552+04:00 (CDB$ROOT(1))
Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2653 ms

-------------------------------------------------------------------------------
Process diagnostic dump actual duration=5.002000 sec
(max dump time=5.000000 sec)

*** 2019-11-15T14:50:25.421655+04:00 (CDB$ROOT(1))

*** 2019-11-15T14:50:30.879349+04:00 (CDB$ROOT(1))
kjfmGCR_HBCheckAll: elapse time between calls is 15 secs
----------------------------------------
SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1
SGA version=(1,0)
(session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310
flags: (0x51) USR/- flags2: (0x409) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
DID: 0001-001F-000000040000-0000-00000000, short-term DID:
txn branch: (nil)
con_id/con_uid/con_name: 1/1/CDB$ROOT
con_logonuid: 1 con_logonid: 1
con_scuid: 1 con_scid: 1
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0x55bffcb060, PX stats: 0x110cb824
service name: SYS$BACKGROUND
Current Wait Stack:
Not in wait; last wait ended 0.000211 sec ago
Wait State:
fixed_waits=0 flags=0x21 boundary=(nil)/-1
Session Wait History:
elapsed time of 0.000231 sec since last wait
0: waited for 'GCR CSS group update'
=0x0, =0x0, =0x0
wait_id=14436246 seq_num=18705 snap_id=2
wait times: snap=0.000001 sec, exc=0.000002 sec, total=0.000333 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
1: waited for 'CSS operation: data update'
function_id=0xd, =0x0, =0x0
wait_id=14436247 seq_num=18704 snap_id=1
wait times: snap=0.000331 sec, exc=0.000331 sec, total=0.000331 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
2: waited for 'GCR CSS group update'
=0x0, =0x0, =0x0
wait_id=14436246 seq_num=18703 snap_id=1
wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000073 sec of elapsed time
3: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436245 seq_num=18702 snap_id=1
wait times: snap=0.150164 sec, exc=0.150164 sec, total=0.150164 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000077 sec of elapsed time
4: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436244 seq_num=18701 snap_id=1
wait times: snap=0.149855 sec, exc=0.149855 sec, total=0.149855 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000125 sec of elapsed time
5: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436243 seq_num=18700 snap_id=1
wait times: snap=0.149900 sec, exc=0.149900 sec, total=0.149900 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.005222 sec of elapsed time
6: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436242 seq_num=18699 snap_id=1
wait times: snap=5.001861 sec, exc=5.001861 sec, total=5.001861 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000097 sec of elapsed time
7: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436240 seq_num=18698 snap_id=2
wait times: snap=0.000139 sec, exc=2.347218 sec, total=5.001355 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
8: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436241 seq_num=18697 snap_id=1
wait times: snap=2.654137 sec, exc=2.654137 sec, total=2.654137 sec
wait times: max=2.650000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
9: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436240 seq_num=18696 snap_id=1
wait times: snap=2.347079 sec, exc=2.347079 sec, total=2.347079 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000213 sec of elapsed time
Sampled Session History of session 9560 serial 54991
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).

The history is displayed in reverse chronological order.

sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[5 samples, 14:50:25 - 14:50:29]
waited for 'ksdxexeotherwait', seq_num: 18699
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001861 sec (sample interval: 4 sec)
[2 samples, 14:50:23 - 14:50:24]
waited for 'ksdxexeotherwait', seq_num: 18697
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.654137 sec (sample interval: 1 sec)
[3 samples, 14:50:20 - 14:50:22]
waited for 'process diagnostic dump', seq_num: 18696
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.347079 sec (sample interval: 2 sec)
[5 samples, 14:50:15 - 14:50:19]
waited for 'ksdxexeotherwait', seq_num: 18695
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001145 sec (sample interval: 4 sec)
[105 samples, 14:48:30 - 14:50:14]
idle wait at each sample
---------------------------------------------------
Sampled Session History Summary:
longest_non_idle_wait: 'ksdxexeotherwait'
[5 samples, 14:50:25 - 14:50:29]
time_waited: 5.001861 sec (sample interval: 4 sec)
---------------------------------------------------
----------------------------------------
kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0
kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0
temporary object counter: 0
last pdbid to increment: 0, last function to increment: (null)
BPS: empty
----------------------------------------
Virtual Thread:
kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991
vc: (nil), proc: 0x566000f310, id: 9560
consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig:
vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2
inwait: 0, short wait event: 0 posted_run: 0
lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)'
lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)'
location where insched last set: kgskthrrun
location where insched last cleared: kgskthrrun1
location where inwait last set: NULL
location where inwait last cleared: NULL
is_assigned: 0, in_sched: 0 (0)
qcls: (nil), qlink: FALSE
vt_active: 0 (pending: 0)
vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0)
ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0)
used quanta (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
exec start consumed time: 0
exec start elapsed time: 0
idle time: 0, active time: 0 (cg: 0)
last updnumps: 0, active time (pq: 0 ps: 0)
cpu yields:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu waits:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu wait time (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
ASL queued time outs: 0, time: 0 (cur 0, cg 0)
PQQ queued time outs: 0, time: 0 (cur 0, cg 0)
Queue timeout violation: 0
calls aborted: 0, num est exec limit hit: 0
undo current: 0k max: 0k
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
KKS-UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 0x973f6daae8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab38
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab88
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dabd8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
----------------------------------------
KGL-UOL SO Cache(total=0, free=0)
KGX Atomic Operation Log 0x971f689230
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689298
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689300
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689368
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6893d0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689438
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6894a0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689508
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689570
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
----------------------------------------
SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1
SGA version=(1,0)
(dummy) nxc=0, nlb=0

*** 2019-11-15T14:50:35.882962+04:00 (CDB$ROOT(1))
==============================
IPC0 (ospid: 368909) has not moved for 37 sec (1573815030.1573814993)
: heartbeat check status 2 (acceptable) (threshold 70 sec)
: heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10
: heartbeat state 0x2.ffff () pso-flag 0x0
: Not in wait; last wait ended 33 secs ago.
: last wait_id 3106025 at 'rdbms ipc message'.
==============================
Dumping PROCESS IPC0 (ospid: 368909) States
==============================
===[ Callstack ]===

*** 2019-11-15T14:50:35.883082+04:00 (CDB$ROOT(1))
Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909,
pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 14:50:34 ]
NOTE: scheduling delay has not been sampled for 0.890380 secs
0.000000 secs from [ 14:50:30 - 14:50:35 ], 5 sec avg
0.000000 secs from [ 14:49:36 - 14:50:35 ], 1 min avg
0.000000 secs from [ 14:45:36 - 14:50:35 ], 5 min avg

*** 2019-11-15T14:50:38.199797+04:00 (CDB$ROOT(1))
loadavg : 9.24 5.66 4.36
System user time: 0.05 sys time: 0.02 context switch: 160189
Memory (Avail / Total) = 6366.25M / 772256.64M
Swap (Avail / Total) = 24000.20M / 24576.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:07:51 ora_ipc0_cdbiamas1
Name: ora_ipc0_cdbiam
State: R (running)
Tgid: 368909
Ngid: 0
Pid: 368909
PPid: 1
TracerPid: 0
Uid: 1001 1001 1001 1001
Gid: 1001 1001 1001 1001
FDSize: 128
Groups: 1001 1002
NStgid: 368909
NSpid: 368909
NSpgid: 368909
NSsid: 368909
VmPeak: 668175752 kB
VmSize: 668175748 kB
VmLck: 0 kB
VmPin: 562562088 kB
VmHWM: 7413828 kB
VmRSS: 7413828 kB
VmData: 8388 kB
VmStk: 156 kB
VmExe: 343516 kB
VmLib: 32424 kB
VmPTE: 14924 kB
VmPMD: 2172 kB
VmSwap: 0 kB
Threads: 1
SigQ: 2/3088249
SigPnd: 0000000000000800
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000016400207
SigCgt: 00000003c9887cf8
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
Seccomp: 0
Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-111
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4589413
nonvoluntary_ctxt_switches: 14530

*** 2019-11-15T14:50:40.884556+04:00 (CDB$ROOT(1))
Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2683 ms

-------------------------------------------------------------------------------
Process diagnostic dump actual duration=5.002000 sec
(max dump time=5.000000 sec)

*** 2019-11-15T14:50:40.884646+04:00 (CDB$ROOT(1))

*** 2019-11-15T14:50:45.885958+04:00 (CDB$ROOT(1))
kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat split1 check, id 3 failed

*** 2019-11-15T14:50:45.890690+04:00 (CDB$ROOT(1))
kjgcr_DoPreAction: KJGCR_PREACTION? - id 36

*** 2019-11-15T14:50:45.890721+04:00 (CDB$ROOT(1))
kjgcr_DoAction: KJGCR_ACTION? - id 36

*** 2019-11-15T14:50:45.890738+04:00 (CDB$ROOT(1))
kjgcr_DoAction: KJGCR_ACTION - id 36 ran with status:0 clr:1
kjfmGCR_HBCheckAll: elapse time between calls is 15 secs
----------------------------------------
SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1
SGA version=(1,0)
(session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310
flags: (0x51) USR/- flags2: (0x409) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
DID: 0001-001F-000000040000-0000-00000000, short-term DID:
txn branch: (nil)
con_id/con_uid/con_name: 1/1/CDB$ROOT
con_logonuid: 1 con_logonid: 1
con_scuid: 1 con_scid: 1
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0x55bffcb060, PX stats: 0x110cb824
service name: SYS$BACKGROUND
Current Wait Stack:
Not in wait; last wait ended 0.000190 sec ago
Wait State:
fixed_waits=0 flags=0x21 boundary=(nil)/-1
Session Wait History:
elapsed time of 0.000222 sec since last wait
0: waited for 'GCR CSS group query'
=0x0, =0x0, =0x0
wait_id=14436253 seq_num=18714 snap_id=2
wait times: snap=0.000002 sec, exc=0.000004 sec, total=0.000416 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
1: waited for 'CSS operation: data query'
function_id=0x3, =0x0, =0x0
wait_id=14436254 seq_num=18713 snap_id=1
wait times: snap=0.000412 sec, exc=0.000412 sec, total=0.000412 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
2: waited for 'GCR CSS group query'
=0x0, =0x0, =0x0
wait_id=14436253 seq_num=18712 snap_id=1
wait times: snap=0.000002 sec, exc=0.000002 sec, total=0.000002 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000202 sec of elapsed time
3: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436252 seq_num=18711 snap_id=1
wait times: snap=0.149943 sec, exc=0.149943 sec, total=0.149943 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.004943 sec of elapsed time
4: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436251 seq_num=18710 snap_id=1
wait times: snap=5.001112 sec, exc=5.001112 sec, total=5.001112 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000077 sec of elapsed time
5: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436249 seq_num=18709 snap_id=2
wait times: snap=0.000122 sec, exc=2.317113 sec, total=5.001503 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
6: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436250 seq_num=18708 snap_id=1
wait times: snap=2.684390 sec, exc=2.684390 sec, total=2.684390 sec
wait times: max=2.680000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
7: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436249 seq_num=18707 snap_id=1
wait times: snap=2.316991 sec, exc=2.316991 sec, total=2.316991 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000200 sec of elapsed time
8: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436248 seq_num=18706 snap_id=1
wait times: snap=5.000485 sec, exc=5.000485 sec, total=5.000485 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.003099 sec of elapsed time
9: waited for 'GCR CSS group update'
=0x0, =0x0, =0x0
wait_id=14436246 seq_num=18705 snap_id=2
wait times: snap=0.000001 sec, exc=0.000002 sec, total=0.000333 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
Sampled Session History of session 9560 serial 54991
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).

The history is displayed in reverse chronological order.

sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[5 samples, 14:50:41 - 14:50:45]
waited for 'ksdxexeotherwait', seq_num: 18710
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: >= 4 sec (still in wait)
[3 samples, 14:50:38 - 14:50:40]
waited for 'ksdxexeotherwait', seq_num: 18708
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.684390 sec (sample interval: 2 sec)
[2 samples, 14:50:36 - 14:50:37]
waited for 'process diagnostic dump', seq_num: 18707
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.316991 sec (sample interval: 1 sec)
[5 samples, 14:50:31 - 14:50:35]
waited for 'ksdxexeotherwait', seq_num: 18706
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.000485 sec (sample interval: 4 sec)
[1 sample, 14:50:30]
idle wait at each sample
[5 samples, 14:50:25 - 14:50:29]
waited for 'ksdxexeotherwait', seq_num: 18699
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001861 sec (sample interval: 4 sec)
[2 samples, 14:50:23 - 14:50:24]
waited for 'ksdxexeotherwait', seq_num: 18697
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.654137 sec (sample interval: 1 sec)
[3 samples, 14:50:20 - 14:50:22]
waited for 'process diagnostic dump', seq_num: 18696
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.347079 sec (sample interval: 2 sec)
[5 samples, 14:50:15 - 14:50:19]
waited for 'ksdxexeotherwait', seq_num: 18695
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001145 sec (sample interval: 4 sec)
[90 samples, 14:48:46 - 14:50:14]
idle wait at each sample
---------------------------------------------------
Sampled Session History Summary:
longest_non_idle_wait: 'ksdxexeotherwait'
[5 samples, 14:50:25 - 14:50:29]
time_waited: 5.001861 sec (sample interval: 4 sec)
---------------------------------------------------
----------------------------------------
kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0
kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0
temporary object counter: 0
last pdbid to increment: 0, last function to increment: (null)
BPS: empty
----------------------------------------
Virtual Thread:
kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991
vc: (nil), proc: 0x566000f310, id: 9560
consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig:
vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2
inwait: 0, short wait event: 0 posted_run: 0
lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)'
lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)'
location where insched last set: kgskthrrun
location where insched last cleared: kgskthrrun1
location where inwait last set: NULL
location where inwait last cleared: NULL
is_assigned: 0, in_sched: 0 (0)
qcls: (nil), qlink: FALSE
vt_active: 0 (pending: 0)
vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0)
ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0)
used quanta (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
exec start consumed time: 0
exec start elapsed time: 0
idle time: 0, active time: 0 (cg: 0)
last updnumps: 0, active time (pq: 0 ps: 0)
cpu yields:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu waits:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu wait time (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
ASL queued time outs: 0, time: 0 (cur 0, cg 0)
PQQ queued time outs: 0, time: 0 (cur 0, cg 0)
Queue timeout violation: 0
calls aborted: 0, num est exec limit hit: 0
undo current: 0k max: 0k
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
KKS-UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 0x973f6daae8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab38
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab88
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dabd8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
----------------------------------------
KGL-UOL SO Cache(total=0, free=0)
KGX Atomic Operation Log 0x971f689230
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689298
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689300
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689368
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6893d0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689438
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6894a0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689508
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689570
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
----------------------------------------
SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1
SGA version=(1,0)
(dummy) nxc=0, nlb=0

*** 2019-11-15T14:50:51.045691+04:00 (CDB$ROOT(1))
==============================
IPC0 (ospid: 368909) has not moved for 52 sec (1573815045.1573814993)
: heartbeat check status 2 (acceptable) (threshold 70 sec)
: heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10
: heartbeat state 0x2.ffff () pso-flag 0x0
: Not in wait; last wait ended 48 secs ago.
: last wait_id 3106025 at 'rdbms ipc message'.
==============================
Dumping PROCESS IPC0 (ospid: 368909) States
==============================
===[ Callstack ]===

*** 2019-11-15T14:50:51.045811+04:00 (CDB$ROOT(1))
Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909,
pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 14:50:50 ]
NOTE: scheduling delay has not been sampled for 0.993271 secs
0.000000 secs from [ 14:50:47 - 14:50:51 ], 5 sec avg
0.000000 secs from [ 14:49:51 - 14:50:51 ], 1 min avg
0.000000 secs from [ 14:45:51 - 14:50:51 ], 5 min avg

*** 2019-11-15T14:50:53.367348+04:00 (CDB$ROOT(1))
loadavg : 195.27 48.34 18.43
System user time: 0.05 sys time: 0.02 context switch: 61840
Memory (Avail / Total) = 6364.98M / 772256.64M
Swap (Avail / Total) = 24000.20M / 24576.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:04 ora_ipc0_cdbiamas1
Name: ora_ipc0_cdbiam
State: R (running)
Tgid: 368909
Ngid: 0
Pid: 368909
PPid: 1
TracerPid: 0
Uid: 1001 1001 1001 1001
Gid: 1001 1001 1001 1001
FDSize: 128
Groups: 1001 1002
NStgid: 368909
NSpid: 368909
NSpgid: 368909
NSsid: 368909
VmPeak: 668175752 kB
VmSize: 668175748 kB
VmLck: 0 kB
VmPin: 562562088 kB
VmHWM: 7413828 kB
VmRSS: 7413828 kB
VmData: 8388 kB
VmStk: 156 kB
VmExe: 343516 kB
VmLib: 32424 kB
VmPTE: 14924 kB
VmPMD: 2172 kB
VmSwap: 0 kB
Threads: 1
SigQ: 4/3088249
SigPnd: 0000000000000800
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000016400207
SigCgt: 00000003c9887cf8
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
Seccomp: 0
Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-111
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4589413
nonvoluntary_ctxt_switches: 14559

*** 2019-11-15T14:50:56.047704+04:00 (CDB$ROOT(1))
Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2678 ms

-------------------------------------------------------------------------------
Process diagnostic dump actual duration=5.002000 sec
(max dump time=5.000000 sec)

*** 2019-11-15T14:50:56.047804+04:00 (CDB$ROOT(1))

*** 2019-11-15T14:51:01.049333+04:00 (CDB$ROOT(1))
kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat split2 check, id 2 failed

*** 2019-11-15T14:51:01.049469+04:00 (CDB$ROOT(1))
kjgcr_StatCheckCPU: Current load ratio is high (68672/1369)
kjfmGCR_HBCheckAll: elapse time between calls is 15 secs
----------------------------------------
SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1
SGA version=(1,0)
(session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310
flags: (0x51) USR/- flags2: (0x409) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
DID: 0001-001F-000000040000-0000-00000000, short-term DID:
txn branch: (nil)
con_id/con_uid/con_name: 1/1/CDB$ROOT
con_logonuid: 1 con_logonid: 1
con_scuid: 1 con_scid: 1
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0x55bffcb060, PX stats: 0x110cb824
service name: SYS$BACKGROUND
Current Wait Stack:
Not in wait; last wait ended 0.000191 sec ago
Wait State:
fixed_waits=0 flags=0x21 boundary=(nil)/-1
Session Wait History:
elapsed time of 0.000212 sec since last wait
0: waited for 'GCR CSS group query'
=0x0, =0x0, =0x0
wait_id=14436260 seq_num=18723 snap_id=2
wait times: snap=0.000001 sec, exc=0.000003 sec, total=0.000703 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
1: waited for 'CSS operation: data query'
function_id=0x3, =0x0, =0x0
wait_id=14436261 seq_num=18722 snap_id=1
wait times: snap=0.000700 sec, exc=0.000700 sec, total=0.000700 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
2: waited for 'GCR CSS group query'
=0x0, =0x0, =0x0
wait_id=14436260 seq_num=18721 snap_id=1
wait times: snap=0.000002 sec, exc=0.000002 sec, total=0.000002 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000205 sec of elapsed time
3: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436259 seq_num=18720 snap_id=1
wait times: snap=0.150108 sec, exc=0.150108 sec, total=0.150108 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.005407 sec of elapsed time
4: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436258 seq_num=18719 snap_id=1
wait times: snap=5.001340 sec, exc=5.001340 sec, total=5.001340 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000084 sec of elapsed time
5: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436256 seq_num=18718 snap_id=2
wait times: snap=0.000133 sec, exc=2.321943 sec, total=5.001918 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
6: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436257 seq_num=18717 snap_id=1
wait times: snap=2.679975 sec, exc=2.679975 sec, total=2.679975 sec
wait times: max=2.670000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
7: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436256 seq_num=18716 snap_id=1
wait times: snap=2.321810 sec, exc=2.321810 sec, total=2.321810 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000211 sec of elapsed time
8: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436255 seq_num=18715 snap_id=1
wait times: snap=5.000770 sec, exc=5.000770 sec, total=5.000770 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.003562 sec of elapsed time
9: waited for 'GCR CSS group query'
=0x0, =0x0, =0x0
wait_id=14436253 seq_num=18714 snap_id=2
wait times: snap=0.000002 sec, exc=0.000004 sec, total=0.000416 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
Sampled Session History of session 9560 serial 54991
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).

The history is displayed in reverse chronological order.

sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[5 samples, 14:50:56 - 14:51:00]
waited for 'ksdxexeotherwait', seq_num: 18719
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: >= 4 sec (still in wait)
[3 samples, 14:50:53 - 14:50:55]
waited for 'ksdxexeotherwait', seq_num: 18717
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.679975 sec (sample interval: 2 sec)
[2 samples, 14:50:51 - 14:50:52]
waited for 'process diagnostic dump', seq_num: 18716
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.321810 sec (sample interval: 1 sec)
[5 samples, 14:50:46 - 14:50:50]
waited for 'ksdxexeotherwait', seq_num: 18715
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.000770 sec (sample interval: 4 sec)
[5 samples, 14:50:41 - 14:50:45]
waited for 'ksdxexeotherwait', seq_num: 18710
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001112 sec (sample interval: 4 sec)
[3 samples, 14:50:38 - 14:50:40]
waited for 'ksdxexeotherwait', seq_num: 18708
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.684390 sec (sample interval: 2 sec)
[2 samples, 14:50:36 - 14:50:37]
waited for 'process diagnostic dump', seq_num: 18707
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.316991 sec (sample interval: 1 sec)
[5 samples, 14:50:31 - 14:50:35]
waited for 'ksdxexeotherwait', seq_num: 18706
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.000485 sec (sample interval: 4 sec)
[1 sample, 14:50:30]
idle wait at each sample
[5 samples, 14:50:25 - 14:50:29]
waited for 'ksdxexeotherwait', seq_num: 18699
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001861 sec (sample interval: 4 sec)
[2 samples, 14:50:23 - 14:50:24]
waited for 'ksdxexeotherwait', seq_num: 18697
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.654137 sec (sample interval: 1 sec)
[3 samples, 14:50:20 - 14:50:22]
waited for 'process diagnostic dump', seq_num: 18696
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.347079 sec (sample interval: 2 sec)
[5 samples, 14:50:15 - 14:50:19]
waited for 'ksdxexeotherwait', seq_num: 18695
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001145 sec (sample interval: 4 sec)
[75 samples, 14:49:01 - 14:50:14]
idle wait at each sample
---------------------------------------------------
Sampled Session History Summary:
longest_non_idle_wait: 'ksdxexeotherwait'
[5 samples, 14:50:25 - 14:50:29]
time_waited: 5.001861 sec (sample interval: 4 sec)
---------------------------------------------------
----------------------------------------
kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0
kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0
temporary object counter: 0
last pdbid to increment: 0, last function to increment: (null)
BPS: empty
----------------------------------------
Virtual Thread:
kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991
vc: (nil), proc: 0x566000f310, id: 9560
consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig:
vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2
inwait: 0, short wait event: 0 posted_run: 0
lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)'
lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)'
location where insched last set: kgskthrrun
location where insched last cleared: kgskthrrun1
location where inwait last set: NULL
location where inwait last cleared: NULL
is_assigned: 0, in_sched: 0 (0)
qcls: (nil), qlink: FALSE
vt_active: 0 (pending: 0)
vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0)
ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0)
used quanta (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
exec start consumed time: 0
exec start elapsed time: 0
idle time: 0, active time: 0 (cg: 0)
last updnumps: 0, active time (pq: 0 ps: 0)
cpu yields:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu waits:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu wait time (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
ASL queued time outs: 0, time: 0 (cur 0, cg 0)
PQQ queued time outs: 0, time: 0 (cur 0, cg 0)
Queue timeout violation: 0
calls aborted: 0, num est exec limit hit: 0
undo current: 0k max: 0k
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
KKS-UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 0x973f6daae8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab38
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab88
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dabd8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
----------------------------------------
KGL-UOL SO Cache(total=0, free=0)
KGX Atomic Operation Log 0x971f689230
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689298
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689300
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689368
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6893d0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689438
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6894a0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689508
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689570
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
----------------------------------------
SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1
SGA version=(1,0)
(dummy) nxc=0, nlb=0

*** 2019-11-15T14:51:06.209982+04:00 (CDB$ROOT(1))
==============================
IPC0 (ospid: 368909) has not moved for 67 sec (1573815060.1573814993)
: heartbeat check status 2 (acceptable) (threshold 70 sec)
: heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10
: heartbeat state 0x2.ffff () pso-flag 0x0
: Not in wait; last wait ended 63 secs ago.
: last wait_id 3106025 at 'rdbms ipc message'.
==============================
Dumping PROCESS IPC0 (ospid: 368909) States
==============================
===[ Callstack ]===

*** 2019-11-15T14:51:06.210105+04:00 (CDB$ROOT(1))
Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909,
pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 14:51:06 ]
NOTE: scheduling delay has not been sampled for 0.092426 secs
0.000000 secs from [ 14:51:02 - 14:51:07 ], 5 sec avg
0.000000 secs from [ 14:50:06 - 14:51:07 ], 1 min avg
0.000000 secs from [ 14:46:06 - 14:51:07 ], 5 min avg

*** 2019-11-15T14:51:08.523175+04:00 (CDB$ROOT(1))
loadavg : 300.42 78.79 28.87
System user time: 0.05 sys time: 0.01 context switch: 54992
Memory (Avail / Total) = 6361.30M / 772256.64M
Swap (Avail / Total) = 24000.20M / 24576.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:19 ora_ipc0_cdbiamas1
Name: ora_ipc0_cdbiam
State: R (running)
Tgid: 368909
Ngid: 0
Pid: 368909
PPid: 1
TracerPid: 0
Uid: 1001 1001 1001 1001
Gid: 1001 1001 1001 1001
FDSize: 128
Groups: 1001 1002
NStgid: 368909
NSpid: 368909
NSpgid: 368909
NSsid: 368909
VmPeak: 668175752 kB
VmSize: 668175748 kB
VmLck: 0 kB
VmPin: 562562088 kB
VmHWM: 7413828 kB
VmRSS: 7413828 kB
VmData: 8388 kB
VmStk: 156 kB
VmExe: 343516 kB
VmLib: 32424 kB
VmPTE: 14924 kB
VmPMD: 2172 kB
VmSwap: 0 kB
Threads: 1
SigQ: 6/3088249
SigPnd: 0000000000000800
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000016400207
SigCgt: 00000003c9887cf8
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
Seccomp: 0
Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-111
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4589415
nonvoluntary_ctxt_switches: 14581

*** 2019-11-15T14:51:11.211384+04:00 (CDB$ROOT(1))
Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2687 ms

-------------------------------------------------------------------------------
Process diagnostic dump actual duration=5.001000 sec
(max dump time=5.000000 sec)

*** 2019-11-15T14:51:11.211482+04:00 (CDB$ROOT(1))

*** 2019-11-15T14:51:16.213118+04:00 (CDB$ROOT(1))
kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat split3 check, id 1 failed

*** 2019-11-15T14:51:16.213225+04:00 (CDB$ROOT(1))
kjgcr_StatCheckCPU: Current load ratio is high (91612/1369)

*** 2019-11-15T14:51:16.668235+04:00 (CDB$ROOT(1))
kjfmGCR_HBCheckAll: elapse time between calls is 16 secs
----------------------------------------
SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1
SGA version=(1,0)
(session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310
flags: (0x51) USR/- flags2: (0x409) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
DID: 0001-001F-000000040000-0000-00000000, short-term DID:
txn branch: (nil)
con_id/con_uid/con_name: 1/1/CDB$ROOT
con_logonuid: 1 con_logonid: 1
con_scuid: 1 con_scid: 1
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0x55bffcb060, PX stats: 0x110cb824
service name: SYS$BACKGROUND
Current Wait Stack:
Not in wait; last wait ended 0.000250 sec ago
Wait State:
fixed_waits=0 flags=0x21 boundary=(nil)/-1
Session Wait History:
elapsed time of 0.000272 sec since last wait
0: waited for 'GCR CSS group update'
=0x0, =0x0, =0x0
wait_id=14436269 seq_num=18734 snap_id=2
wait times: snap=0.000001 sec, exc=0.000002 sec, total=0.000452 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
1: waited for 'CSS operation: data update'
function_id=0xd, =0x0, =0x0
wait_id=14436270 seq_num=18733 snap_id=1
wait times: snap=0.000450 sec, exc=0.000450 sec, total=0.000450 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
2: waited for 'GCR CSS group update'
=0x0, =0x0, =0x0
wait_id=14436269 seq_num=18732 snap_id=1
wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000068 sec of elapsed time
3: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436268 seq_num=18731 snap_id=1
wait times: snap=0.149957 sec, exc=0.149957 sec, total=0.149957 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000082 sec of elapsed time
4: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436267 seq_num=18730 snap_id=1
wait times: snap=0.149864 sec, exc=0.149864 sec, total=0.149864 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000110 sec of elapsed time
5: waited for 'GCR sleep'
=0x0, =0x0, =0x0
wait_id=14436266 seq_num=18729 snap_id=1
wait times: snap=0.149669 sec, exc=0.149669 sec, total=0.149669 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.005414 sec of elapsed time
6: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436265 seq_num=18728 snap_id=1
wait times: snap=5.001068 sec, exc=5.001068 sec, total=5.001068 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000079 sec of elapsed time
7: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436263 seq_num=18727 snap_id=2
wait times: snap=0.000137 sec, exc=2.313499 sec, total=5.001312 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
8: waited for 'ksdxexeotherwait'
=0x0, =0x0, =0x0
wait_id=14436264 seq_num=18726 snap_id=1
wait times: snap=2.687813 sec, exc=2.687813 sec, total=2.687813 sec
wait times: max=2.680000 sec
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
9: waited for 'process diagnostic dump'
=0x0, =0x0, =0x0
wait_id=14436263 seq_num=18725 snap_id=1
wait times: snap=2.313362 sec, exc=2.313362 sec, total=2.313362 sec
wait times: max=5.000000 sec
wait counts: calls=0 os=0
occurred after 0.000219 sec of elapsed time
Sampled Session History of session 9560 serial 54991
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).

The history is displayed in reverse chronological order.

sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[5 samples, 14:51:11 - 14:51:15]
waited for 'ksdxexeotherwait', seq_num: 18728
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: >= 4 sec (still in wait)
[3 samples, 14:51:08 - 14:51:10]
waited for 'ksdxexeotherwait', seq_num: 18726
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.687813 sec (sample interval: 2 sec)
[2 samples, 14:51:06 - 14:51:07]
waited for 'process diagnostic dump', seq_num: 18725
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.313362 sec (sample interval: 1 sec)
[5 samples, 14:51:01 - 14:51:05]
waited for 'ksdxexeotherwait', seq_num: 18724
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.000459 sec (sample interval: 4 sec)
[5 samples, 14:50:56 - 14:51:00]
waited for 'ksdxexeotherwait', seq_num: 18719
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001340 sec (sample interval: 4 sec)
[3 samples, 14:50:53 - 14:50:55]
waited for 'ksdxexeotherwait', seq_num: 18717
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.679975 sec (sample interval: 2 sec)
[2 samples, 14:50:51 - 14:50:52]
waited for 'process diagnostic dump', seq_num: 18716
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.321810 sec (sample interval: 1 sec)
[5 samples, 14:50:46 - 14:50:50]
waited for 'ksdxexeotherwait', seq_num: 18715
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.000770 sec (sample interval: 4 sec)
[5 samples, 14:50:41 - 14:50:45]
waited for 'ksdxexeotherwait', seq_num: 18710
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001112 sec (sample interval: 4 sec)
[3 samples, 14:50:38 - 14:50:40]
waited for 'ksdxexeotherwait', seq_num: 18708
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.684390 sec (sample interval: 2 sec)
[2 samples, 14:50:36 - 14:50:37]
waited for 'process diagnostic dump', seq_num: 18707
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.316991 sec (sample interval: 1 sec)
[5 samples, 14:50:31 - 14:50:35]
waited for 'ksdxexeotherwait', seq_num: 18706
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.000485 sec (sample interval: 4 sec)
[1 sample, 14:50:30]
idle wait at each sample
[5 samples, 14:50:25 - 14:50:29]
waited for 'ksdxexeotherwait', seq_num: 18699
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001861 sec (sample interval: 4 sec)
[2 samples, 14:50:23 - 14:50:24]
waited for 'ksdxexeotherwait', seq_num: 18697
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.654137 sec (sample interval: 1 sec)
[3 samples, 14:50:20 - 14:50:22]
waited for 'process diagnostic dump', seq_num: 18696
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 2.347079 sec (sample interval: 2 sec)
[5 samples, 14:50:15 - 14:50:19]
waited for 'ksdxexeotherwait', seq_num: 18695
p1: ''=0x0
p2: ''=0x0
p3: ''=0x0
time_waited: 5.001145 sec (sample interval: 4 sec)
[60 samples, 14:49:16 - 14:50:14]
idle wait at each sample
---------------------------------------------------
Sampled Session History Summary:
longest_non_idle_wait: 'ksdxexeotherwait'
[5 samples, 14:50:25 - 14:50:29]
time_waited: 5.001861 sec (sample interval: 4 sec)
---------------------------------------------------
----------------------------------------
kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0
kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0
temporary object counter: 0
last pdbid to increment: 0, last function to increment: (null)
BPS: empty
----------------------------------------
Virtual Thread:
kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991
vc: (nil), proc: 0x566000f310, id: 9560
consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig:
vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2
inwait: 0, short wait event: 0 posted_run: 0
lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)'
lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)'
location where insched last set: kgskthrrun
location where insched last cleared: kgskthrrun1
location where inwait last set: NULL
location where inwait last cleared: NULL
is_assigned: 0, in_sched: 0 (0)
qcls: (nil), qlink: FALSE
vt_active: 0 (pending: 0)
vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0)
ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0)
used quanta (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
exec start consumed time: 0
exec start elapsed time: 0
idle time: 0, active time: 0 (cg: 0)
last updnumps: 0, active time (pq: 0 ps: 0)
cpu yields:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu waits:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu wait time (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
ASL queued time outs: 0, time: 0 (cur 0, cg 0)
PQQ queued time outs: 0, time: 0 (cur 0, cg 0)
Queue timeout violation: 0
calls aborted: 0, num est exec limit hit: 0
undo current: 0k max: 0k
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
KKS-UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 0x973f6daae8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab38
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dab88
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x973f6dabd8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 9560 efd 0 whr 0 slp 0
----------------------------------------
KGL-UOL SO Cache(total=0, free=0)
KGX Atomic Operation Log 0x971f689230
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689298
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689300
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689368
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6893d0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689438
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f6894a0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689508
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x971f689570
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 9560 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
----------------------------------------
SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1
SGA version=(1,0)
(dummy) nxc=0, nlb=0

*** 2019-11-15T14:51:21.673454+04:00 (CDB$ROOT(1))
==============================
IPC0 (ospid: 368909) has not moved for 83 sec (1573815076.1573814993)
: heartbeat check status 6 (no-heartbeat) (threshold 70 sec)

*** 2019-11-15T14:51:26.675662+04:00 (CDB$ROOT(1))
==================================================
=== IPC0 (ospid: 368909) Heartbeat Report
==================================================
IPC0 (ospid: 368909) has no heartbeats for 93 sec. (threshold 70)
: heartbeat state 0x2.ffff () pso-flag 0x0
: Not in wait; last wait ended 78 secs ago.
: last wait_id 3106025 at 'rdbms ipc message'.
==============================
Dumping PROCESS IPC0 (ospid: 368909) States
==============================
===[ System Load State ]===
CPU Total 96 Raw 96 Core 48 Socket 2
Load normal: Cur 104133 Highmark 122880 (406.76 480.00)
===[ Latch State ]===
Not in Latch Get
===[ Session State Object ]===
----------------------------------------
SO: 0x18e1753030, type: 4, owner: 0x18e0002c50, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x18e0002c50, name=session, file=ksu.h LINE:15737, pg=0, conuid=1
SGA version=(1,0)
(session) sid: 727 ser: 53431 trans: (nil), creator: 0x18e0002c50
flags: (0x51) USR/- flags2: (0x409) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
DID: 0000-0000-000000000000-0000-00000000, short-term DID:
txn branch: (nil)
con_id/con_uid/con_name: 1/1/CDB$ROOT
con_logonuid: 1 con_logonid: 1
con_scuid: 1 con_scid: 1
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0x183fffc798, PX stats: 0x110cb824
service name: SYS$BACKGROUND
Current Wait Stack:
Not in wait; last wait ended 1 min 28 sec ago
Wait State:
fixed_waits=0 flags=0x21 boundary=(nil)/-1
Session Wait History:
elapsed time of 1 min 28 sec since last wait
0: waited for 'rdbms ipc message'
timeout=0x12c, =0x0, =0x0
wait_id=3106025 seq_num=35086 snap_id=1
wait times: snap=3.003118 sec, exc=3.003118 sec, total=3.003118 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.000011 sec of elapsed time
1: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106024 seq_num=35085 snap_id=1
wait times: snap=0.020098 sec, exc=0.020098 sec, total=0.020098 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050228 sec of elapsed time
2: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106023 seq_num=35084 snap_id=1
wait times: snap=0.020110 sec, exc=0.020110 sec, total=0.020110 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000015 sec of elapsed time
3: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106022 seq_num=35083 snap_id=1
wait times: snap=0.020180 sec, exc=0.020180 sec, total=0.020180 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.049185 sec of elapsed time
4: waited for 'rdbms ipc message'
timeout=0x69, =0x0, =0x0
wait_id=3106021 seq_num=35082 snap_id=1
wait times: snap=1.051154 sec, exc=1.051154 sec, total=1.051154 sec
wait times: max=1.050000 sec
wait counts: calls=1 os=1
occurred after 0.000011 sec of elapsed time
5: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106020 seq_num=35081 snap_id=1
wait times: snap=0.020146 sec, exc=0.020146 sec, total=0.020146 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050272 sec of elapsed time
6: waited for 'rdbms ipc message'
timeout=0x12c, =0x0, =0x0
wait_id=3106019 seq_num=35080 snap_id=1
wait times: snap=1.878880 sec, exc=1.878880 sec, total=1.878880 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.000011 sec of elapsed time
7: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106018 seq_num=35079 snap_id=1
wait times: snap=0.020171 sec, exc=0.020171 sec, total=0.020171 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050231 sec of elapsed time
8: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106017 seq_num=35078 snap_id=1
wait times: snap=0.020184 sec, exc=0.020184 sec, total=0.020184 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000016 sec of elapsed time
9: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106016 seq_num=35077 snap_id=1
wait times: snap=0.020206 sec, exc=0.020206 sec, total=0.020206 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050239 sec of elapsed time
Sampled Session History of session 727 serial 53431
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).

The history is displayed in reverse chronological order.

sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[88 samples, 14:49:58 - 14:51:25]
not in wait at each sample
[9 samples, 14:49:49 - 14:49:57]
idle wait at each sample
[1 sample, 14:49:48]
waited for 'RMA: IPC0 completion sync', seq_num: 35075
p1: 'send count'=0x0
p2: ''=0x0
p3: ''=0x23e
time_waited: 0.020107 sec (sample interval: 0 sec)
[4 samples, 14:49:44 - 14:49:47]
idle wait at each sample
[1 sample, 14:49:43]
not in wait at each sample
[13 samples, 14:49:30 - 14:49:42]
idle wait at each sample
[1 sample, 14:49:29]
waited for 'RMA: IPC0 completion sync', seq_num: 35045
p1: 'send count'=0x0
p2: ''=0x0
p3: ''=0x23e
time_waited: 0.020174 sec (sample interval: 0 sec)
[4 samples, 14:49:26 - 14:49:28]
idle wait at each sample
---------------------------------------------------
Sampled Session History Summary:
longest_non_idle_wait: 'RMA: IPC0 completion sync'
[1 sample, 14:49:29 ]
time_waited: 0.020174 sec (sample interval: 0 sec)
---------------------------------------------------
----------------------------------------
kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0
kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0
temporary object counter: 0
last pdbid to increment: 0, last function to increment: (null)
BPS: empty
----------------------------------------
Virtual Thread:
kgskvt: 0x976416cc20, sess: 0x18e1753030/1 sid: 727 ser: 53431
vc: (nil), proc: 0x18e0002c50, id: 727
consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig:
vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2
inwait: 0, short wait event: 0 posted_run: 0
lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)'
lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)'
location where insched last set: kgskthrrun
location where insched last cleared: kgskthrrun1
location where inwait last set: NULL
location where inwait last cleared: NULL
is_assigned: 0, in_sched: 0 (0)
qcls: (nil), qlink: FALSE
vt_active: 0 (pending: 0)
vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0)
ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0)
used quanta (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
exec start consumed time: 0
exec start elapsed time: 0
idle time: 0, active time: 0 (cg: 0)
last updnumps: 0, active time (pq: 0 ps: 0)
cpu yields:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu waits:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu wait time (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
ASL queued time outs: 0, time: 0 (cur 0, cg 0)
PQQ queued time outs: 0, time: 0 (cur 0, cg 0)
Queue timeout violation: 0
calls aborted: 0, num est exec limit hit: 0
undo current: 0k max: 0k
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
KKS-UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 0x977f6f40a0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x977f6f40f0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x977f6f4140
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x977f6f4190
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
----------------------------------------
KGL-UOL SO Cache(total=0, free=0)
KGX Atomic Operation Log 0x975f73a2d8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a340
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a3a8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a410
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a478
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a4e0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a548
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a5b0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a618
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
----------------------------------------
SO: 0x1812141ff8, type: 79, owner: 0x18e1753030, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x18e0002c50, name=dummy, file=ktccts.h LINE:456, pg=0, conuid=1
SGA version=(1,0)
(dummy) nxc=0, nlb=0
===[ Callstack ]===

*** 2019-11-15T14:51:26.683226+04:00 (CDB$ROOT(1))
Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909,
pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 14:51:26 ]
NOTE: scheduling delay has not been sampled for 0.484605 secs
0.000000 secs from [ 14:51:22 - 14:51:27 ], 5 sec avg
0.000000 secs from [ 14:50:26 - 14:51:27 ], 1 min avg
0.000000 secs from [ 14:46:27 - 14:51:27 ], 5 min avg

*** 2019-11-15T14:51:29.002962+04:00 (CDB$ROOT(1))
loadavg : 406.77 117.41 42.64
System user time: 0.07 sys time: 0.02 context switch: 66627
Memory (Avail / Total) = 6340.92M / 772256.64M
Swap (Avail / Total) = 24000.20M / 24576.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:35 ora_ipc0_cdbiamas1
Name: ora_ipc0_cdbiam
State: R (running)
Tgid: 368909
Ngid: 0
Pid: 368909
PPid: 1
TracerPid: 0
Uid: 1001 1001 1001 1001
Gid: 1001 1001 1001 1001
FDSize: 128
Groups: 1001 1002
NStgid: 368909
NSpid: 368909
NSpgid: 368909
NSsid: 368909
VmPeak: 668175752 kB
VmSize: 668175748 kB
VmLck: 0 kB
VmPin: 562562088 kB
VmHWM: 7413828 kB
VmRSS: 7413828 kB
VmData: 8388 kB
VmStk: 156 kB
VmExe: 343516 kB
VmLib: 32424 kB
VmPTE: 14924 kB
VmPMD: 2172 kB
VmSwap: 0 kB
Threads: 1
SigQ: 11/3088249
SigPnd: 0000000000000800
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000016400207
SigCgt: 00000003c9887cf8
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
Seccomp: 0
Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-111
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4589416
nonvoluntary_ctxt_switches: 14614

*** 2019-11-15T14:51:31.684041+04:00 (CDB$ROOT(1))
Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2680 ms

-------------------------------------------------------------------------------
Process diagnostic dump actual duration=5.001000 sec
(max dump time=5.000000 sec)

*** 2019-11-15T14:51:31.684131+04:00 (CDB$ROOT(1))
===[ Heartbeat State Object ]===
(lm heartbeat) state=0x38000000 active
current heartbeat: time 0x5dce82d1 count 0x3403e where 0x1
last poke: time 0x5dce82db req 0x3403b ack 0x3403a freq 10
last wait: state 0x2 id 3106025
last record: time 0x5dce82dd count 0x3403e
last check: time 0x5dce8324
misc: good dur 70 default 70 ckreq 10 tmupdcnt 0 hangcnt 1
Dump of memory from 0x000000979F1E9608 to 0x000000979F1E9818
979F1E9600 9F1D2450 00000097 [P$......]
979F1E9610 E9D93E40 00000096 9F1D2460 00000097 [@>......`$......]
979F1E9620 E9D93E50 00000096 9F1E9628 00000097 [P>......(.......]
979F1E9630 9F1E9628 00000097 E0002C50 00000018 [(.......P,......]
979F1E9640 6000A5A0 00000000 00000001 30435049 [...`........IPC0]
979F1E9650 00000000 38000000 00000000 00000000 [.......8........]
979F1E9660 00000046 00000046 00000000 0003403E [F...F.......>@..]
979F1E9670 5DCE82D1 00000001 00000000 0000000A [...]............]
979F1E9680 5DCE82DB 0003403B 0003403A 0000000A [...];@..:@......]
979F1E9690 0000000A 00000001 5DCE8324 5DCE8324 [........$..]$..]]
979F1E96A0 0003403E 5DCE82DD 5DCE8323 00000000 [>@.....]#..]....]
979F1E96B0 00000076 00000000 00000000 00000000 [v...............]
979F1E96C0 00000000 00000000 00000000 00000006 [................]
979F1E96D0 00000000 00000000 00000000 00000000 [................]
979F1E96E0 00000002 FFFFFFFF 00000000 00000000 [................]
979F1E96F0 00000000 00000000 002F64E9 00000000 [.........d/.....]
979F1E9700 04B4390A 00000000 00000007 00000000 [.9..............]
979F1E9710 00000000 00000000 00000000 00000000 [................]
Repeat 4 times
979F1E9760 74C300A8 00000097 00000004 00000000 [...t............]
979F1E9770 00000000 00000000 00000000 370000FA [...............7]
979F1E9780 00000005 00000961 5DCE8324 00000000 [....a...$..]....]
979F1E9790 00000008 00000000 00000000 00000000 [................]
979F1E97A0 00000000 00000000 00000000 00000000 [................]
Repeat 1 times
979F1E97C0 00000000 00000000 9F1E97C8 00000097 [................]
979F1E97D0 9F1E97C8 00000097 60048728 00000000 [........(..`....]
979F1E97E0 9F1D2620 00000097 00000000 00000000 [ &..............]
979F1E97F0 00000000 00000000 00000001 00000000 [................]
979F1E9800 00000000 00000000 00000000 00000000 [................]
979F1E9810 00000000 00000000 [........]
===[ Trace Buffer ]===

*** 2019-11-15T14:51:36.685443+04:00 (CDB$ROOT(1))
===[ DLM IPC ]===
KJC Communication Dump:
state 0x35 flags 0x0 mode 0x0 inst 1 inc 28
inst=1 inc=28 nrcv=9 nsp=9 nrcvbuf=1000
reg_msg: msgsz=640 mpools=2 cur=688 max=12300 init=17390
mp[0]: cur=685 out=(sga:3602.3530.0 initq:613) max=12297 ini=8695 pavl=0 pmax=0
mp[1]: cur=3 out=(sga:0.0.0 initq:3) max=13 ini=8695 pavl=0 pmax=0
big_msg: msgsz=8504 mpools=2 cur=86 max=947 init=17390
mp[0]: cur=66 out=(sga:0.0.0 initq:66) max=923 ini=8695 pavl=0 pmax=0
mp[1]: cur=46 out=(sga:0.0.0 initq:46) max=132 ini=8695 pavl=0 pmax=0
rsv_msg: msgsz=8504 mpools=1 cur=0 max=1 init=1000
mp[0]: cur=0 out=(sga:0.0.0 initq:0) max=1 tot=1000 pavl=0 pmax=0
rcvr: id=0 (orapid=26 LMD0) grp=1 ospid=368961
rcvr: id=8 (orapid=34 LMD2) grp=1 ospid=368989
rcvr: id=1 (orapid=28 LMS0) grp=2 ospid=368963
rcvr: id=5 (orapid=32 LMS4) grp=2 ospid=368977
rcvr: id=6 (orapid=27 LMS5) grp=2 ospid=368981
rcvr: id=4 (orapid=25 LMS3) grp=2 ospid=368972
rcvr: id=2 (orapid=23 LMS1) grp=2 ospid=368965
rcvr: id=7 (orapid=29 LMD1) grp=1 ospid=368983
rcvr: id=3 (orapid=30 LMS2) grp=2 ospid=368969
send proxy: id=0 orapid=26 ndst=1 (2:0 ) pso=0x18e000d1f0.(nil) state=x0
send proxy: id=8 orapid=34 ndst=1 (2:8 ) pso=0x18e0011430.(nil) state=x0
send proxy: id=1 orapid=28 ndst=1 (2:1 ) pso=0x18e000e280.(nil) state=x0
send proxy: id=5 orapid=32 ndst=1 (2:5 ) pso=0x18e00103a0.(nil) state=x0
send proxy: id=6 orapid=27 ndst=1 (2:6 ) pso=0x566000d1f0.(nil) state=x0
send proxy: id=4 orapid=25 ndst=1 (2:4 ) pso=0x566000c160.(nil) state=x0
send proxy: id=2 orapid=23 ndst=1 (2:2 ) pso=0x566000b0d0.(nil) state=x0
send proxy: id=7 orapid=29 ndst=1 (2:7 ) pso=0x566000e280.(nil) state=x0
send proxy: id=3 orapid=30 ndst=1 (2:3 ) pso=0x18e000f310.(nil) state=x0
GES resource limits:
ges resources: cur=242473 max=503402 ini=241332
ges enqueues: cur=124210 max=157460 ini=363099
ges cresources: cur=124792 max=132896
gcs resources: cur=21223354 max=25452045 ini=77122167
gcs shadows: cur=18628493 max=18812915 ini=77122167
KJCTS state: seq-check=no timeout=yes waitticks=0x3
highload=no comm_interface=msgq-drm (rds)
GES destination context:
GES remote instance per receiver context:
GES destination context:
Dest=2 rcvr=0 inc=28 state=0x10041 tstate=0x0
batch-type=quick bmsg=(nil) tmout=0xd626881 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 81)
sndq_flushs=44664975 last_flush_time=0x89a6388e8 flush_idx=4
sndq_flush_stats=
[4]v:44664975|w:16|f:0|l:1|t:1000|s:1|m:1
[3]v:44664974|w:57|f:0|l:1|t:1000|s:1|m:1
[2]v:44664973|w:68|f:0|l:1|t:1000|s:1|m:1
[1]v:44664972|w:302|f:0|l:1|t:1000|s:1|m:1
[0]v:44664971|w:56|f:0|l:1|t:999|s:1|m:1
[9]v:44664970|w:188|f:0|l:1|t:1000|s:1|m:1
[8]v:44664969|w:81|f:0|l:1|t:1000|s:1|m:1
[7]v:44664968|w:25|f:0|l:1|t:999|s:1|m:1
[6]v:44664967|w:48|f:0|l:1|t:1000|s:1|m:1
[5]v:44664966|w:25|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=168981285 ack=0 snt=168981284 sync seq=0.168981284 inc=0
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.153767097,[PCM]0.0,[ALL]0.168981284
batch_snds=1683692 tot_msgs=3533432 max_sz=38 full_load=0 snd_seq=1683692
pbatch_snds=19271326 tot_msgs=19455116 sndq_msg_tot=46516315 tm=(0 5671)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 214 wqlen 29423807
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=1 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x13b91f9c msg_in_batch=0
tkt_total=1000 (min=1000) avl=748 sp_rsv=249 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 98)
sndq_flushs=43452752 last_flush_time=0x89a6388f0 flush_idx=1
sndq_flush_stats=
[1]v:43452752|w:16|f:0|l:1|t:999|s:1|m:1
[0]v:43452751|w:14|f:0|l:1|t:996|s:1|m:1
[9]v:43452750|w:69|f:0|l:1|t:999|s:1|m:1
[8]v:43452749|w:37|f:0|l:1|t:998|s:1|m:1
[7]v:43452748|w:149|f:0|l:1|t:998|s:1|m:1
[6]v:43452747|w:95|f:0|l:1|t:999|s:1|m:1
[5]v:43452746|w:13|f:0|l:1|t:999|s:1|m:1
[4]v:43452745|w:239|f:0|l:1|t:1000|s:1|m:1
[3]v:43452744|w:76|f:0|l:1|t:997|s:1|m:1
[2]v:43452743|w:36|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=132507620 ack=0 snt=132507619 sync seq=0.132507619 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85809507,[PCM]0.132507619,[ALL]0.132507619
batch_snds=1399886 tot_msgs=5033684 max_sz=96 full_load=642 snd_seq=1399886
pbatch_snds=21225728 tot_msgs=28383594 sndq_msg_tot=46911662 tm=(0 49774149)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41600432
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=2 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x109a8a06 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 97)
sndq_flushs=42914870 last_flush_time=0x89a6388eb flush_idx=9
sndq_flush_stats=
[9]v:42914870|w:26|f:0|l:1|t:999|s:1|m:1
[8]v:42914869|w:46|f:0|l:1|t:1000|s:1|m:1
[7]v:42914868|w:3|f:0|l:1|t:999|s:1|m:1
[6]v:42914867|w:1|f:0|l:1|t:1000|s:1|m:1
[5]v:42914866|w:0|f:0|l:1|t:1000|s:1|m:1
[4]v:42914865|w:13|f:0|l:1|t:999|s:1|m:1
[3]v:42914864|w:38|f:0|l:1|t:1000|s:1|m:1
[2]v:42914863|w:30|f:0|l:1|t:998|s:1|m:1
[1]v:42914862|w:21|f:0|l:1|t:999|s:1|m:1
[0]v:42914861|w:79|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=133268181 ack=0 snt=133268181 sync seq=0.133268181 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.89088728,[PCM]0.133268181,[ALL]0.133268181
batch_snds=1426683 tot_msgs=5103642 max_sz=77 full_load=569 snd_seq=1426683
pbatch_snds=20722409 tot_msgs=27863597 sndq_msg_tot=46412046 tm=(0 50945990)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 364 wqlen 41640453
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=3 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x113ff945 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 95)
sndq_flushs=43539285 last_flush_time=0x89a6388b9 flush_idx=4
sndq_flush_stats=
[4]v:43539285|w:30|f:0|l:1|t:999|s:1|m:1
[3]v:43539284|w:55|f:0|l:1|t:1000|s:1|m:1
[2]v:43539283|w:55|f:0|l:1|t:1000|s:1|m:1
[1]v:43539282|w:39|f:0|l:1|t:998|s:1|m:1
[0]v:43539281|w:27|f:0|l:1|t:1000|s:1|m:1
[9]v:43539280|w:18|f:0|l:1|t:997|s:1|m:1
[8]v:43539279|w:41|f:0|l:1|t:999|s:1|m:1
[7]v:43539278|w:97|f:0|l:1|t:1000|s:1|m:1
[6]v:43539277|w:43|f:0|l:1|t:999|s:1|m:1
[5]v:43539276|w:78|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=134352993 ack=0 snt=134352990 sync seq=0.134352990 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.86869831,[PCM]0.134352990,[ALL]0.134352990
batch_snds=1432672 tot_msgs=5118093 max_sz=91 full_load=605 snd_seq=1432672
pbatch_snds=21305579 tot_msgs=28448192 sndq_msg_tot=47043265 tm=(0 51174746)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 375 wqlen 41622154
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=4 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x1168f832 msg_in_batch=0
tkt_total=1000 (min=1000) avl=747 sp_rsv=249 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 103)
sndq_flushs=43168383 last_flush_time=0x89a6388ef flush_idx=2
sndq_flush_stats=
[2]v:43168383|w:16|f:0|l:1|t:998|s:1|m:1
[1]v:43168382|w:79|f:0|l:1|t:1000|s:1|m:1
[0]v:43168381|w:4|f:0|l:1|t:998|s:1|m:1
[9]v:43168380|w:12|f:0|l:1|t:999|s:1|m:1
[8]v:43168379|w:22|f:0|l:1|t:1000|s:1|m:1
[7]v:43168378|w:3|f:0|l:1|t:1000|s:1|m:1
[6]v:43168377|w:106|f:0|l:1|t:1000|s:1|m:1
[5]v:43168376|w:41|f:0|l:1|t:999|s:1|m:1
[4]v:43168375|w:15|f:0|l:1|t:996|s:1|m:1
[3]v:43168374|w:41|f:0|l:1|t:998|s:1|m:1
seq wrp=0 lst=143196222 ack=0 snt=143196221 sync seq=0.143196221 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.96323855,[PCM]0.143196221,[ALL]0.143196221
batch_snds=1427942 tot_msgs=5134969 max_sz=59 full_load=601 snd_seq=1427942
pbatch_snds=20915126 tot_msgs=28054314 sndq_msg_tot=46695919 tm=(0 50968179)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 346 wqlen 41912172
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=5 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x139c13ee msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 94)
sndq_flushs=43574282 last_flush_time=0x89a6388b5 flush_idx=1
sndq_flush_stats=
[1]v:43574282|w:15|f:0|l:1|t:997|s:1|m:1
[0]v:43574281|w:5|f:0|l:1|t:1000|s:1|m:1
[9]v:43574280|w:42|f:1|l:1|t:997|s:1|m:1
[8]v:43574279|w:85|f:0|l:1|t:999|s:1|m:1
[7]v:43574278|w:68|f:0|l:1|t:1000|s:1|m:1
[6]v:43574277|w:79|f:0|l:1|t:1000|s:1|m:1
[5]v:43574276|w:12|f:0|l:1|t:998|s:1|m:1
[4]v:43574275|w:33|f:0|l:1|t:999|s:1|m:1
[3]v:43574274|w:2|f:0|l:1|t:1000|s:1|m:1
[2]v:43574273|w:59|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=137469430 ack=0 snt=137469427 sync seq=0.137469427 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.93123683,[PCM]0.137469427,[ALL]0.137469427
batch_snds=1415019 tot_msgs=5079675 max_sz=74 full_load=587 snd_seq=1415019
pbatch_snds=21334518 tot_msgs=28460067 sndq_msg_tot=47061098 tm=(0 49472680)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41481937
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=6 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x14b8d66c msg_in_batch=0
tkt_total=1000 (min=1000) avl=748 sp_rsv=249 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 96)
sndq_flushs=43191530 last_flush_time=0x89a6388ed flush_idx=9
sndq_flush_stats=
[9]v:43191530|w:20|f:0|l:1|t:1000|s:1|m:1
[8]v:43191529|w:84|f:0|l:1|t:1000|s:1|m:1
[7]v:43191528|w:31|f:0|l:1|t:1000|s:1|m:1
[6]v:43191527|w:8|f:0|l:1|t:997|s:1|m:1
[5]v:43191526|w:24|f:0|l:1|t:998|s:1|m:1
[4]v:43191525|w:79|f:0|l:1|t:999|s:1|m:1
[3]v:43191524|w:149|f:0|l:1|t:977|s:1|m:1
[2]v:43191523|w:2|f:0|l:1|t:999|s:1|m:1
[1]v:43191522|w:1|f:0|l:1|t:1000|s:1|m:1
[0]v:43191521|w:21|f:0|l:1|t:993|s:1|m:1
seq wrp=0 lst=129460055 ack=0 snt=129460053 sync seq=0.129460053 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85305884,[PCM]0.129460053,[ALL]0.129460053
batch_snds=1414189 tot_msgs=5124380 max_sz=67 full_load=652 snd_seq=1414189
pbatch_snds=20885815 tot_msgs=28060372 sndq_msg_tot=46719857 tm=(0 50088934)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 359 wqlen 42071557
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=7 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0xdc9a426 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 54)
sndq_flushs=46048278 last_flush_time=0x89a6388ea flush_idx=7
sndq_flush_stats=
[7]v:46048278|w:2|f:0|l:1|t:1000|s:1|m:1
[6]v:46048277|w:0|f:0|l:1|t:1000|s:1|m:1
[5]v:46048276|w:48|f:0|l:1|t:999|s:1|m:1
[4]v:46048275|w:24|f:0|l:1|t:1000|s:1|m:1
[3]v:46048274|w:2|f:0|l:1|t:1000|s:1|m:1
[2]v:46048273|w:31|f:0|l:1|t:999|s:1|m:1
[1]v:46048272|w:37|f:0|l:1|t:1000|s:1|m:1
[0]v:46048271|w:85|f:0|l:1|t:1000|s:1|m:1
[9]v:46048270|w:45|f:0|l:1|t:999|s:1|m:1
[8]v:46048269|w:61|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=163062139 ack=0 snt=163062139 sync seq=0.163062139 inc=0
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.210,[PCM]0.0,[ALL]0.163062139
batch_snds=1938346 tot_msgs=3972927 max_sz=39 full_load=0 snd_seq=1938346
pbatch_snds=14296974 tot_msgs=14477098 sndq_msg_tot=48084777 tm=(0 5829)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 110 wqlen 35949749
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=8 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0xcf125a2 msg_in_batch=0
tkt_total=1000 (min=1000) avl=746 sp_rsv=248 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 55)
sndq_flushs=38362337 last_flush_time=0x89a6388e8 flush_idx=6
sndq_flush_stats=
[6]v:38362337|w:1|f:0|l:1|t:997|s:1|m:1
[5]v:38362336|w:69|f:0|l:1|t:998|s:1|m:1
[4]v:38362335|w:166|f:0|l:1|t:997|s:1|m:1
[3]v:38362334|w:68|f:0|l:1|t:1000|s:1|m:1
[2]v:38362333|w:42|f:0|l:1|t:1000|s:1|m:1
[1]v:38362332|w:73|f:0|l:1|t:999|s:1|m:1
[0]v:38362331|w:80|f:0|l:1|t:1000|s:1|m:1
[9]v:38362330|w:31|f:0|l:1|t:999|s:1|m:1
[8]v:38362329|w:43|f:0|l:1|t:997|s:1|m:1
[7]v:38362328|w:42|f:0|l:2|t:998|s:1|m:2
seq wrp=0 lst=161548233 ack=0 snt=161548231 sync seq=0.161548231 inc=0
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.218,[PCM]0.0,[ALL]0.161548231
batch_snds=1587433 tot_msgs=3357402 max_sz=45 full_load=0 snd_seq=1587433
pbatch_snds=15617700 tot_msgs=15800503 sndq_msg_tot=40133301 tm=(0 4194)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 218 wqlen 26651687
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
GES remote instance per receiver context:
Rcvr=0 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.169438649 sync_rcv_seq=0.169438649 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1452889
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.166858040
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=8 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.160789037 sync_rcv_seq=0.160789031 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1208966
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.158381798
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=1 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.141250908 sync_rcv_seq=0.141250907 sync_ack_seq=0.129056901 inc=28 batch_rcv_seq=1845551
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.134058573
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=5 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.141357194 sync_rcv_seq=0.141357194 sync_ack_seq=0.131227202 inc=28 batch_rcv_seq=1858785
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.135396667
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=6 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.150276081 sync_rcv_seq=0.150276079 sync_ack_seq=0.137776658 inc=28 batch_rcv_seq=1893391
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.142779468
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=4 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.125328139 sync_rcv_seq=0.125328139 sync_ack_seq=0.114702593 inc=28 batch_rcv_seq=1863433
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.119202571
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=2 dest=2 inc=28 state=0x103 tstate=0x0 nak=2 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.124480280 sync_rcv_seq=0.124480280 sync_ack_seq=0.114194111 inc=28 batch_rcv_seq=1875123
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.118453984
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=7 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.165115287 sync_rcv_seq=0.165115287 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1989087
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.163643671
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=3 dest=2 inc=28 state=0x103 tstate=0x0 nak=5 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.120369356 sync_rcv_seq=0.120369356 sync_ack_seq=0.110107175 inc=28 batch_rcv_seq=1849776
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.114270602
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=0 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:111)
msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0
rmpool: alloc=1334 (r:1311 s:23) avail=134 (r:111 s:23) max=(s:23)
hist_last_rcv_tm=0x89a6388e9 count=224553208 idx=3
[3]t:x89a635f15|c:791|w:13|x:(668.123.0.0)99|m:552
[2]t:x89a62e9aa|c:2574|w:11|x:(2251.323.0.0)99|m:1817
[1]t:x89a62746e|c:19120|w:1|x:(18828.292.0.0)99|m:14268
[0]t:x89a61ff34|c:2824|w:10|x:(2526.298.0.0)99|m:2057
[9]t:x89a618a02|c:3096|w:9|x:(2785.311.0.0)99|m:2195
[8]t:x89a6114b8|c:3643|w:8|x:(3348.295.0.0)99|m:2806
[7]t:x89a609f72|c:3880|w:7|x:(3598.282.0.0)99|m:3044
[6]t:x89a602a2a|c:17602|w:1|x:(17318.284.0.0)99|m:15328
[5]t:x89a5fb4e7|c:3445|w:8|x:(3155.290.0.0)99|m:2819
[4]t:x89a5f3fa4|c:3861|w:7|x:(3566.295.0.0)98|m:3119
ibatch: 2[1372526] 3[53116] 4[16271] 5[4951] 6[2397] 7[1341]
ibatch: 8[909] 9[761] 10[450] 11[173] 12[87] 13[63]
ibatch: 14[28] 15[15] 16[11] 17[6]
ibatch: sum=1453105 rcvs=1453215 tot_msgs=3042811 max_sz=35 avg_sz=2
pbatch: 2[7672] 3[415] 4[136] 5[61] 6[57] 7[17]
pbatch: 8[4] 9[3] 10[4] 11[1] 12[0] 13[4]
pbatch: 14[1] 15[5] 16[4] 17[1]
pbatch: sum=8385 rcvs=20424692 tot_msgs=20579530 max_sz=103 avg_sz=2
Rcvr=8 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:157)
msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0
rmpool: alloc=1380 (r:1357 s:23) avail=180 (r:157 s:23) max=(s:23)
hist_last_rcv_tm=0x89a6388e8 count=217130446 idx=3
[3]t:x89a6364ac|c:752|w:12|x:(650.102.0.0)99|m:517
[2]t:x89a62ef5b|c:2631|w:11|x:(2307.324.0.0)99|m:1803
[1]t:x89a627a0c|c:16992|w:1|x:(16699.293.0.0)99|m:12695
[0]t:x89a6204bb|c:2630|w:11|x:(2297.333.0.0)99|m:1843
[9]t:x89a618f38|c:3198|w:9|x:(2887.311.0.0)99|m:2092
[8]t:x89a611a02|c:2654|w:11|x:(2321.333.0.0)99|m:1778
[7]t:x89a60a49e|c:2621|w:11|x:(2306.315.0.0)99|m:1828
[6]t:x89a602f6d|c:15008|w:2|x:(14713.295.0.0)99|m:13063
[5]t:x89a5fba3d|c:1997|w:15|x:(1652.345.0.0)99|m:1560
[4]t:x89a5f44ed|c:2134|w:14|x:(1795.339.0.0)99|m:1621
ibatch: 2[1113837] 3[61797] 4[20539] 5[6821] 6[3212] 7[1341]
ibatch: 8[743] 9[391] 10[198] 11[116] 12[59] 13[35]
ibatch: 14[21] 15[15] 16[7] 17[9]
ibatch: sum=1209141 rcvs=1209227 tot_msgs=2573520 max_sz=38 avg_sz=2
pbatch: 2[5158] 3[535] 4[201] 5[91] 6[37] 7[31]
pbatch: 8[20] 9[19] 10[13] 11[3] 12[6] 13[10]
pbatch: 14[5] 15[6] 16[2] 17[9]
pbatch: sum=6146 rcvs=19070248 tot_msgs=19222335 max_sz=103 avg_sz=2
Rcvr=1 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:364)
msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0
rmpool: alloc=1633 (r:1564 s:69) avail=430 (r:364 s:66) max=(s:69)
hist_last_rcv_tm=0x89a6388f0 count=330858922 idx=0
[0]t:x89a635b9b|c:2136|w:5|x:(2136.0.0.0)30|m:853
[9]t:x89a62e661|c:8054|w:3|x:(8054.0.0.0)30|m:3383
[8]t:x89a62712e|c:6558|w:4|x:(6558.0.0.0)30|m:2955
[7]t:x89a61fbfd|c:7561|w:3|x:(7561.0.0.0)30|m:3010
[6]t:x89a6186bf|c:5868|w:5|x:(5868.0.0.0)30|m:2357
[5]t:x89a61118a|c:5499|w:5|x:(5499.0.0.0)30|m:2345
[4]t:x89a609c55|c:6241|w:4|x:(6241.0.0.0)30|m:2705
[3]t:x89a60271c|c:7254|w:4|x:(7254.0.0.0)30|m:2809
[2]t:x89a5fb1e5|c:6797|w:4|x:(6797.0.0.0)30|m:3079
[1]t:x89a5f3cb2|c:6626|w:4|x:(6626.0.0.0)30|m:2850
ibatch: 2[1220363] 3[251612] 4[102831] 5[55882] 6[38196] 7[30964]
ibatch: 8[23999] 9[20125] 10[16840] 11[15151] 12[13532] 13[12454]
ibatch: 14[11550] 15[10833] 16[9778] 17[3805]
ibatch: sum=1837915 rcvs=1849590 tot_msgs=6174214 max_sz=56 avg_sz=3
pbatch: 2[352286] 3[202796] 4[123721] 5[93438] 6[93402] 7[104490]
pbatch: 8[27372] 9[18633] 10[17937] 11[15916] 12[15737] 13[13953]
pbatch: 14[13953] 15[12735] 16[14167] 17[92343]
pbatch: sum=1212879 rcvs=20362030 tot_msgs=32519217 max_sz=74 avg_sz=5
Rcvr=5 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341)
msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0
rmpool: alloc=1610 (r:1541 s:69) avail=407 (r:341 s:66) max=(s:69)
hist_last_rcv_tm=0x89a6388f5 count=328938604 idx=0
[0]t:x89a634fbf|c:2905|w:5|x:(2905.0.0.0)30|m:1253
[9]t:x89a62da8c|c:5846|w:5|x:(5846.0.0.0)30|m:2312
[8]t:x89a626543|c:5009|w:5|x:(5009.0.0.0)30|m:2451
[7]t:x89a61f00d|c:4569|w:6|x:(4569.0.0.0)30|m:1761
[6]t:x89a617ad2|c:3664|w:8|x:(3664.0.0.0)30|m:1473
[5]t:x89a610592|c:4264|w:7|x:(4262.2.0.0)31|m:1713
[4]t:x89a609049|c:4347|w:6|x:(4347.0.0.0)30|m:1854
[3]t:x89a601b18|c:4616|w:6|x:(4616.0.0.0)30|m:1556
[2]t:x89a5fa5e7|c:4752|w:6|x:(4752.0.0.0)30|m:2229
[1]t:x89a5f30ab|c:5274|w:5|x:(5274.0.0.0)30|m:2301
ibatch: 2[1225447] 3[254830] 4[103664] 5[56407] 6[38776] 7[30966]
ibatch: 8[24303] 9[20643] 10[17098] 11[15454] 12[13628] 13[12816]
ibatch: 14[11667] 15[10919] 16[9954] 17[4018]
ibatch: sum=1850590 rcvs=1862824 tot_msgs=6243422 max_sz=52 avg_sz=3
pbatch: 2[350360] 3[200988] 4[125538] 5[93312] 6[94203] 7[104713]
pbatch: 8[28340] 9[18640] 10[18044] 11[15789] 12[16167] 13[13836]
pbatch: 14[13789] 15[12668] 16[13887] 17[92690]
pbatch: sum=1212964 rcvs=19829482 tot_msgs=31998448 max_sz=74 avg_sz=5
Rcvr=6 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318)
msgs_rdyq=0 watq=0 nsq=0 routq=1202 msgq=0
rmpool: alloc=1564 (r:1518 s:46) avail=362 (r:318 s:44) max=(s:46)
hist_last_rcv_tm=0x89a6388ed count=347611873 idx=0
[0]t:x89a63463c|c:4359|w:3|x:(4359.0.0.0)30|m:1782
[9]t:x89a62d10c|c:7888|w:3|x:(7888.0.0.0)30|m:2981
[8]t:x89a625bd0|c:8434|w:3|x:(8434.0.0.0)30|m:3542
[7]t:x89a61e697|c:6469|w:4|x:(6469.0.0.0)30|m:2630
[6]t:x89a617150|c:5699|w:5|x:(5698.1.0.0)31|m:2280
[5]t:x89a60fc19|c:6267|w:4|x:(6267.0.0.0)30|m:2647
[4]t:x89a6086e8|c:5200|w:5|x:(5200.0.0.0)30|m:2163
[3]t:x89a6011b7|c:6625|w:4|x:(6625.0.0.0)30|m:2467
[2]t:x89a5f9c7f|c:5407|w:5|x:(5406.1.0.0)31|m:2495
[1]t:x89a5f273e|c:7966|w:3|x:(7966.0.0.0)30|m:3396
ibatch: 2[1251337] 3[258991] 4[106698] 5[56215] 6[39919] 7[29475]
ibatch: 8[24834] 9[19768] 10[18088] 11[15360] 12[14221] 13[12597]
ibatch: 14[12181] 15[11011] 16[10537] 17[4081]
ibatch: sum=1885313 rcvs=1897647 tot_msgs=6349480 max_sz=54 avg_sz=3
pbatch: 2[357532] 3[210027] 4[123428] 5[95483] 6[91888] 7[107378]
pbatch: 8[25641] 9[20011] 10[16849] 11[16988] 12[14832] 13[14752]
pbatch: 14[13117] 15[13450] 16[13519] 17[93193]
pbatch: sum=1228088 rcvs=19965179 tot_msgs=32152763 max_sz=74 avg_sz=5
Rcvr=4 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341)
msgs_rdyq=0 watq=0 nsq=0 routq=1202 msgq=0
rmpool: alloc=1587 (r:1541 s:46) avail=385 (r:341 s:44) max=(s:46)
hist_last_rcv_tm=0x89a6388ef count=291797547 idx=0
[0]t:x89a638682|c:64|w:10|x:(64.0.0.0)30|m:26
[9]t:x89a63114b|c:6950|w:4|x:(6950.0.0.0)30|m:3238
[8]t:x89a629c1a|c:10508|w:2|x:(10508.0.0.0)30|m:4547
[7]t:x89a6226e8|c:10742|w:2|x:(10742.0.0.0)30|m:3977
[6]t:x89a61b1ad|c:5043|w:5|x:(5043.0.0.0)30|m:1965
[5]t:x89a613c72|c:6451|w:4|x:(6451.0.0.0)30|m:2806
[4]t:x89a60c742|c:5526|w:5|x:(5526.0.0.0)30|m:2345
[3]t:x89a605211|c:7198|w:4|x:(7198.0.0.0)30|m:2769
[2]t:x89a5fdcd1|c:7269|w:4|x:(7269.0.0.0)30|m:3115
[1]t:x89a5f678c|c:5222|w:5|x:(5222.0.0.0)30|m:2593
ibatch: 2[1234999] 3[252810] 4[104488] 5[54128] 6[38980] 7[29147]
ibatch: 8[24513] 9[19667] 10[17880] 11[15098] 12[13992] 13[12561]
ibatch: 14[12222] 15[11097] 16[10306] 17[3819]
ibatch: sum=1855707 rcvs=1867971 tot_msgs=6256370 max_sz=51 avg_sz=3
pbatch: 2[344218] 3[202531] 4[120299] 5[95235] 6[92709] 7[105848]
pbatch: 8[26248] 9[19662] 10[17784] 11[17509] 12[14867] 13[14646]
pbatch: 14[13139] 15[13514] 16[13571] 17[92942]
pbatch: sum=1204722 rcvs=20477905 tot_msgs=32639611 max_sz=74 avg_sz=5
Rcvr=2 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318)
msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0
rmpool: alloc=1587 (r:1518 s:69) avail=384 (r:318 s:66) max=(s:69)
hist_last_rcv_tm=0x89a6388f5 count=278280354 idx=9
[9]t:x89a6315dd|c:6025|w:4|x:(6025.0.0.0)30|m:2729
[8]t:x89a62a09b|c:5382|w:5|x:(5382.0.0.0)30|m:2632
[7]t:x89a622b69|c:4661|w:6|x:(4661.0.0.0)30|m:2142
[6]t:x89a61b61e|c:3939|w:7|x:(3939.0.0.0)30|m:1740
[5]t:x89a6140e4|c:4523|w:6|x:(4521.2.0.0)31|m:1908
[4]t:x89a60cbaa|c:3731|w:8|x:(3731.0.0.0)30|m:1652
[3]t:x89a60567a|c:5259|w:5|x:(5259.0.0.0)30|m:1912
[2]t:x89a5fe148|c:4399|w:6|x:(4399.0.0.0)30|m:1996
[1]t:x89a5f6c02|c:3823|w:7|x:(3823.0.0.0)30|m:1899
[0]t:x89a5ef6bf|c:5992|w:5|x:(5992.0.0.0)30|m:2797
ibatch: 2[1228028] 3[255817] 4[107012] 5[56903] 6[40928] 7[31133]
ibatch: 8[25761] 9[21059] 10[18571] 11[15973] 12[14775] 13[12869]
ibatch: 14[12242] 15[11229] 16[10392] 17[4092]
ibatch: sum=1866784 rcvs=1879395 tot_msgs=6367991 max_sz=54 avg_sz=3
pbatch: 2[353486] 3[206296] 4[122255] 5[94494] 6[91116] 7[105765]
pbatch: 8[25985] 9[19548] 10[16768] 11[17365] 12[14744] 13[14894]
pbatch: 14[13731] 15[13582] 16[13514] 17[92954]
pbatch: sum=1216497 rcvs=19822588 tot_msgs=31993720 max_sz=74 avg_sz=5
Rcvr=7 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:180)
msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0
rmpool: alloc=1403 (r:1380 s:23) avail=203 (r:180 s:23) max=(s:23)
hist_last_rcv_tm=0x89a6388ea count=231318606 idx=2
[2]t:x89a6349c7|c:1286|w:12|x:(1100.186.0.0)99|m:820
[1]t:x89a62d45b|c:2683|w:11|x:(2377.306.0.0)99|m:1711
[0]t:x89a625f09|c:20508|w:1|x:(20227.281.0.0)99|m:14946
[9]t:x89a61e9d2|c:2751|w:10|x:(2419.332.0.0)99|m:1722
[8]t:x89a61745d|c:2983|w:10|x:(2667.316.0.0)99|m:1876
[7]t:x89a60ff24|c:3786|w:7|x:(3473.313.0.0)99|m:2292
[6]t:x89a6089eb|c:5113|w:5|x:(4776.337.0.0)99|m:3577
[5]t:x89a6014a3|c:16750|w:1|x:(16441.309.0.0)99|m:14276
[4]t:x89a5f9f6b|c:3434|w:8|x:(3095.339.0.0)99|m:2341
[3]t:x89a5f2a25|c:3668|w:8|x:(3348.320.0.0)98|m:2414
ibatch: 2[1870524] 3[105708] 4[10127] 5[1884] 6[732] 7[414]
ibatch: 8[170] 9[81] 10[69] 11[30] 12[19] 13[26]
ibatch: 14[4] 15[0] 16[1] 17[1]
ibatch: sum=1989790 rcvs=1989854 tot_msgs=4119282 max_sz=23 avg_sz=2
pbatch: 2[6307] 3[414] 4[92] 5[29] 6[15] 7[5]
pbatch: 8[4] 9[3] 10[5] 11[5] 12[3] 13[8]
pbatch: 14[4] 15[4] 16[6] 17[8]
pbatch: sum=6912 rcvs=16772268 tot_msgs=16925126 max_sz=103 avg_sz=2
Rcvr=3 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341)
msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0
rmpool: alloc=1610 (r:1541 s:69) avail=407 (r:341 s:66) max=(s:69)
hist_last_rcv_tm=0x89a6388f6 count=289399211 idx=9
[9]t:x89a631ef8|c:5444|w:4|x:(5444.0.0.0)30|m:2194
[8]t:x89a62a9be|c:5590|w:5|x:(5590.0.0.0)30|m:2162
[7]t:x89a623484|c:5274|w:5|x:(5274.0.0.0)30|m:2524
[6]t:x89a61bf46|c:5719|w:5|x:(5719.0.0.0)30|m:2388
[5]t:x89a614a01|c:4346|w:6|x:(4346.0.0.0)30|m:1769
[4]t:x89a60d4c5|c:5610|w:5|x:(5610.0.0.0)30|m:2220
[3]t:x89a605f8d|c:4747|w:6|x:(4747.0.0.0)30|m:1892
[2]t:x89a5fea57|c:4861|w:6|x:(4861.0.0.0)30|m:1844
[1]t:x89a5f7526|c:4966|w:6|x:(4966.0.0.0)30|m:2315
[0]t:x89a5effec|c:6927|w:4|x:(6927.0.0.0)30|m:2932
ibatch: 2[1221232] 3[251727] 4[102904] 5[56754] 6[39175] 7[30881]
ibatch: 8[24073] 9[20703] 10[16862] 11[15386] 12[13402] 13[12711]
ibatch: 14[11581] 15[10830] 16[9697] 17[3984]
ibatch: sum=1841902 rcvs=1855190 tot_msgs=6259867 max_sz=48 avg_sz=3
pbatch: 2[360080] 3[204156] 4[124610] 5[92957] 6[94169] 7[105196]
pbatch: 8[28209] 9[17678] 10[18665] 11[15940] 12[15763] 13[13774]
pbatch: 14[13801] 15[12728] 16[13885] 17[93027]
pbatch: sum=1224638 rcvs=20360708 tot_msgs=32542328 max_sz=74 avg_sz=5
GES PGA send message buffer usage:
orapid=23 ospid=368965 pmp: alloc=69 avail=66 max=69
orapid=25 ospid=368972 pmp: alloc=46 avail=44 max=46
orapid=26 ospid=368961 pmp: alloc=23 avail=23 max=23
orapid=27 ospid=368981 pmp: alloc=46 avail=44 max=46
orapid=28 ospid=368963 pmp: alloc=69 avail=66 max=69
orapid=29 ospid=368983 pmp: alloc=23 avail=23 max=23
orapid=30 ospid=368969 pmp: alloc=69 avail=66 max=69
orapid=32 ospid=368977 pmp: alloc=69 avail=66 max=69
orapid=34 ospid=368989 pmp: alloc=23 avail=23 max=23
orapid=71 ospid=369094 pmp: alloc=0 avail=0 max=23
orapid=74 ospid=369090 pmp: alloc=0 avail=0 max=23
orapid=75 ospid=369121 pmp: alloc=0 avail=0 max=23
orapid=78 ospid=369098 pmp: alloc=0 avail=0 max=23
orapid=79 ospid=369123 pmp: alloc=0 avail=0 max=23
orapid=82 ospid=369108 pmp: alloc=0 avail=0 max=23
kjctseventdump-begin tail 220 heads 218 @ 0 220 @ -817966651
#0 id 439 exid 77713 dests 00
#1 id 439 exid 73326 dests 00
#2 id 439 exid 77713 dests 00
#3 id 439 exid 73326 dests 00
#4 id 439 exid 19919 dests 00
#5 id 439 exid 73326 dests 00
#6 id 439 exid 19919 dests 00
#7 id 439 exid 73326 dests 00
#8 id 439 exid 77713 dests 00
#9 id 439 exid 19919 dests 00
#10 id 439 exid 73326 dests 00
#11 id 439 exid 19919 dests 00
#12 id 439 exid 73326 dests 00
#13 id 439 exid 77713 dests 00
#14 id 439 exid 73326 dests 00
#15 id 439 exid 73326 dests 00
#16 id 439 exid 77713 dests 00
#17 id 439 exid 73326 dests 00
#18 id 439 exid 19919 dests 00
#19 id 439 exid 73326 dests 00
#20 id 439 exid 19919 dests 00
#21 id 439 exid 73326 dests 00
#22 id 439 exid 19919 dests 00
#23 id 439 exid 73326 dests 00
#24 id 439 exid 77713 dests 00
#25 id 439 exid 77713 dests 00
#26 id 439 exid 73326 dests 00
#27 id 439 exid 19919 dests 00
#28 id 439 exid 73326 dests 00
#29 id 439 exid 73326 dests 00
#30 id 439 exid 73326 dests 00
#31 id 439 exid 19919 dests 00
#32 id 439 exid 19919 dests 00
#33 id 439 exid 73326 dests 00
#34 id 439 exid 77713 dests 00
#35 id 439 exid 73326 dests 00
#36 id 439 exid 73326 dests 00
#37 id 439 exid 77713 dests 00
#38 id 439 exid 73326 dests 00
#39 id 439 exid 19919 dests 00
#40 id 439 exid 73326 dests 00
#41 id 439 exid 19919 dests 00
#42 id 439 exid 73326 dests 00
#43 id 439 exid 73326 dests 00
#44 id 439 exid 19919 dests 00
#45 id 439 exid 73326 dests 00
#46 id 439 exid 77713 dests 00
#47 id 439 exid 73326 dests 00
#48 id 439 exid 77713 dests 00
#49 id 439 exid 19919 dests 00
#50 id 439 exid 19919 dests 00
#51 id 439 exid 73326 dests 00
#52 id 439 exid 19919 dests 00
#53 id 439 exid 73326 dests 00
#54 id 439 exid 19919 dests 00
#55 id 439 exid 19919 dests 00
#56 id 439 exid 77713 dests 00
#57 id 439 exid 73326 dests 00
#58 id 439 exid 73326 dests 00
#59 id 439 exid 73326 dests 00
#60 id 439 exid 77713 dests 00
#61 id 439 exid 77713 dests 00
#62 id 439 exid 73326 dests 00
#63 id 439 exid 19919 dests 00
#64 id 439 exid 73326 dests 00
#65 id 439 exid 19919 dests 00
#66 id 439 exid 73326 dests 00
#67 id 439 exid 19919 dests 00
#68 id 439 exid 73326 dests 00
#69 id 439 exid 77713 dests 00
#70 id 439 exid 77713 dests 00
#71 id 439 exid 73326 dests 00
#72 id 439 exid 73326 dests 00
#73 id 439 exid 19919 dests 00
#74 id 439 exid 73326 dests 00
#75 id 439 exid 19919 dests 00
#76 id 439 exid 73326 dests 00
#77 id 439 exid 19919 dests 00
#78 id 439 exid 19919 dests 00
#79 id 439 exid 73326 dests 00
#80 id 439 exid 77713 dests 00
#81 id 439 exid 73326 dests 00
#82 id 439 exid 77713 dests 00
#83 id 439 exid 19919 dests 00
#84 id 439 exid 73326 dests 00
#85 id 439 exid 73326 dests 00
#86 id 439 exid 19919 dests 00
#87 id 439 exid 73326 dests 00
#88 id 439 exid 77713 dests 00
#89 id 439 exid 73326 dests 00
#90 id 439 exid 77713 dests 00
#91 id 439 exid 73326 dests 00
#92 id 439 exid 19919 dests 00
#93 id 439 exid 19919 dests 00
#94 id 439 exid 73326 dests 00
#95 id 439 exid 73326 dests 00
#96 id 439 exid 19919 dests 00
#97 id 439 exid 73326 dests 00
#98 id 439 exid 77713 dests 00
#99 id 439 exid 73326 dests 00
#100 id 439 exid 77713 dests 00
#101 id 439 exid 73326 dests 00
#102 id 439 exid 19919 dests 00
#103 id 439 exid 19919 dests 00
#104 id 439 exid 73326 dests 00
#105 id 439 exid 19919 dests 00
#106 id 439 exid 73326 dests 00
#107 id 439 exid 77713 dests 00
#108 id 439 exid 77713 dests 00
#109 id 439 exid 73326 dests 00
#110 id 439 exid 19919 dests 00
#111 id 439 exid 19919 dests 00
#112 id 439 exid 73326 dests 00
#113 id 439 exid 73326 dests 00
#114 id 439 exid 19919 dests 00
#115 id 439 exid 19919 dests 00
#116 id 439 exid 73326 dests 00
#117 id 439 exid 77713 dests 00
#118 id 439 exid 73326 dests 00
#119 id 439 exid 77713 dests 00
#120 id 439 exid 73326 dests 00
#121 id 439 exid 19919 dests 00
#122 id 439 exid 73326 dests 00
#123 id 439 exid 19919 dests 00
#124 id 439 exid 73326 dests 00
#125 id 439 exid 19919 dests 00
#126 id 439 exid 19919 dests 00
#127 id 439 exid 73326 dests 00
#128 id 439 exid 77713 dests 00
#129 id 439 exid 73326 dests 00
#130 id 439 exid 77713 dests 00
#131 id 439 exid 19919 dests 00
#132 id 439 exid 73326 dests 00
#133 id 439 exid 19919 dests 00
#134 id 439 exid 73326 dests 00
#135 id 439 exid 19919 dests 00
#136 id 439 exid 73326 dests 00
#137 id 439 exid 77713 dests 00
#138 id 439 exid 73326 dests 00
#139 id 439 exid 77713 dests 00
#140 id 439 exid 73326 dests 00
#141 id 439 exid 19919 dests 00
#142 id 439 exid 73326 dests 00
#143 id 439 exid 19919 dests 00
#144 id 439 exid 73326 dests 00
#145 id 439 exid 19919 dests 00
#146 id 439 exid 73326 dests 00
#147 id 439 exid 77713 dests 00
#148 id 439 exid 73326 dests 00
#149 id 439 exid 77713 dests 00
#150 id 439 exid 19919 dests 00
#151 id 439 exid 73326 dests 00
#152 id 439 exid 73326 dests 00
#153 id 439 exid 19919 dests 00
#154 id 439 exid 73326 dests 00
#155 id 439 exid 77713 dests 00
#156 id 439 exid 73326 dests 00
#157 id 439 exid 77713 dests 00
#158 id 439 exid 73326 dests 00
#159 id 439 exid 19919 dests 00
#160 id 439 exid 73326 dests 00
#161 id 439 exid 19919 dests 00
#162 id 439 exid 19919 dests 00
#163 id 439 exid 73326 dests 00
#164 id 439 exid 77713 dests 00
#165 id 439 exid 73326 dests 00
#166 id 439 exid 77713 dests 00
#167 id 439 exid 73326 dests 00
#168 id 439 exid 19919 dests 00
#169 id 439 exid 19919 dests 00
#170 id 439 exid 73326 dests 00
#171 id 439 exid 19919 dests 00
#172 id 439 exid 73326 dests 00
#173 id 439 exid 77713 dests 00
#174 id 439 exid 73326 dests 00
#175 id 439 exid 77713 dests 00
#176 id 439 exid 19919 dests 00
#177 id 439 exid 73326 dests 00
#178 id 439 exid 19919 dests 00
#179 id 439 exid 19919 dests 00
#180 id 439 exid 73326 dests 00
#181 id 439 exid 19919 dests 00
#182 id 439 exid 73326 dests 00
#183 id 439 exid 77713 dests 00
#184 id 439 exid 73326 dests 00
#185 id 439 exid 77713 dests 00
#186 id 439 exid 73326 dests 00
#187 id 439 exid 19919 dests 00
#188 id 439 exid 73326 dests 00
#189 id 439 exid 19919 dests 00
#190 id 439 exid 73326 dests 00
#191 id 439 exid 19919 dests 00
#192 id 439 exid 73326 dests 00
#193 id 439 exid 77713 dests 00
#194 id 439 exid 77713 dests 00
#195 id 439 exid 73326 dests 00
#196 id 439 exid 19919 dests 00
#197 id 439 exid 19919 dests 00
#198 id 439 exid 73326 dests 00
#199 id 439 exid 19919 dests 00
#200 id 439 exid 73326 dests 00
#201 id 439 exid 77713 dests 00
#202 id 439 exid 73326 dests 00
#203 id 439 exid 77713 dests 00
#204 id 439 exid 19919 dests 00
#205 id 439 exid 73326 dests 00
#206 id 439 exid 19919 dests 00
#207 id 439 exid 19919 dests 00
#208 id 439 exid 73326 dests 00
#209 id 439 exid 19919 dests 00
#210 id 439 exid 19919 dests 00
#211 id 439 exid 73326 dests 00
#212 id 439 exid 77713 dests 00
#213 id 439 exid 73326 dests 00
#214 id 439 exid 77713 dests 00
#215 id 439 exid 73326 dests 00
#216 id 439 exid 19919 dests 00
#217 id 439 exid 73326 dests 00
#218 id 439 exid 19919 dests 00
#219 id 439 exid 73326 dests 00
#220 id 439 exid 73326 dests 00
#221 id 439 exid 77713 dests 00
#222 id 439 exid 73326 dests 00
#223 id 439 exid 19919 dests 00
#224 id 439 exid 73326 dests 00
#225 id 439 exid 19919 dests 00
#226 id 439 exid 73326 dests 00
#227 id 439 exid 73326 dests 00
#228 id 439 exid 19919 dests 00
#229 id 439 exid 73326 dests 00
#230 id 439 exid 77713 dests 00
#231 id 439 exid 73326 dests 00
#232 id 439 exid 77713 dests 00
#233 id 439 exid 19919 dests 00
#234 id 439 exid 73326 dests 00
#235 id 439 exid 73326 dests 00
#236 id 439 exid 19919 dests 00
#237 id 439 exid 73326 dests 00
#238 id 439 exid 77713 dests 00
#239 id 439 exid 77713 dests 00
#240 id 439 exid 73326 dests 00
#241 id 439 exid 19919 dests 00
#242 id 439 exid 73326 dests 00
#243 id 439 exid 19919 dests 00
#244 id 439 exid 19919 dests 00
#245 id 439 exid 73326 dests 00
#246 id 439 exid 77713 dests 00
#247 id 439 exid 73326 dests 00
#248 id 439 exid 77713 dests 00
#249 id 439 exid 19919 dests 00
#250 id 439 exid 73326 dests 00
#251 id 439 exid 73326 dests 00
#252 id 439 exid 19919 dests 00
#253 id 439 exid 19919 dests 00
#254 id 439 exid 73326 dests 00
#255 id 439 exid 73326 dests 00
kjctseventdump-end tail 220 heads 218 @ 0 220 @ -817966651
End of KJC Communication Dump
kjfmGCR_HBCheckAll: 1 process failed HBStatusCheck
kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat check, id 0 failed

*** 2019-11-15T14:51:36.699115+04:00 (CDB$ROOT(1))
kjgcr_StatCheckCPU: Current load ratio is high (114711/1369)

*** 2019-11-15T14:51:41.704742+04:00 (CDB$ROOT(1))
kjfmGCR_HBAnalyze: IPC0 has no heartbeat

*** 2019-11-15T14:51:46.706154+04:00 (CDB$ROOT(1))
==============================
IPC0 (ospid: 368909) has not moved for 108 sec (1573815101.1573814993)
2019-11-15T14:51:46.746879+04:00
Incident 6943370 created, dump file: /u01/app/oracle/diag/rdbms/cdbiamas/cdbiamas1/incident/incdir_6943370/cdbiamas1_lmhb_368993_i6943370.trc
ORA-29770: global enqueue process IPC0 (OSID 368909) is hung for more than 70 seconds

===[NO-HB PROCESS CALLSTACK]===
<Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 60 ms
>
==============================
==============================
Dumping PROCESS IPC0 (ospid: 368909) States
==============================
===[ System Load State ]===
CPU Total 96 Raw 96 Core 48 Socket 2
Load high: Cur 123801 Highmark 122880 (483.59 480.00)
===[ Latch State ]===
Not in Latch Get
===[ Session State Object ]===
----------------------------------------
SO: 0x18e1753030, type: 4, owner: 0x18e0002c50, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x18e0002c50, name=session, file=ksu.h LINE:15737, pg=0, conuid=1
SGA version=(1,0)
(session) sid: 727 ser: 53431 trans: (nil), creator: 0x18e0002c50
flags: (0x51) USR/- flags2: (0x409) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
DID: 0000-0000-000000000000-0000-00000000, short-term DID:
txn branch: (nil)
con_id/con_uid/con_name: 1/1/CDB$ROOT
con_logonuid: 1 con_logonid: 1
con_scuid: 1 con_scid: 1
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0x183fffc798, PX stats: 0x110cb824
service name: SYS$BACKGROUND
Current Wait Stack:
Not in wait; last wait ended 1 min 49 sec ago
Wait State:
fixed_waits=0 flags=0x21 boundary=(nil)/-1
Session Wait History:
elapsed time of 1 min 49 sec since last wait
0: waited for 'rdbms ipc message'
timeout=0x12c, =0x0, =0x0
wait_id=3106025 seq_num=35086 snap_id=1
wait times: snap=3.003118 sec, exc=3.003118 sec, total=3.003118 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.000011 sec of elapsed time
1: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106024 seq_num=35085 snap_id=1
wait times: snap=0.020098 sec, exc=0.020098 sec, total=0.020098 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050228 sec of elapsed time
2: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106023 seq_num=35084 snap_id=1
wait times: snap=0.020110 sec, exc=0.020110 sec, total=0.020110 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000015 sec of elapsed time
3: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106022 seq_num=35083 snap_id=1
wait times: snap=0.020180 sec, exc=0.020180 sec, total=0.020180 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.049185 sec of elapsed time
4: waited for 'rdbms ipc message'
timeout=0x69, =0x0, =0x0
wait_id=3106021 seq_num=35082 snap_id=1
wait times: snap=1.051154 sec, exc=1.051154 sec, total=1.051154 sec
wait times: max=1.050000 sec
wait counts: calls=1 os=1
occurred after 0.000011 sec of elapsed time
5: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106020 seq_num=35081 snap_id=1
wait times: snap=0.020146 sec, exc=0.020146 sec, total=0.020146 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050272 sec of elapsed time
6: waited for 'rdbms ipc message'
timeout=0x12c, =0x0, =0x0
wait_id=3106019 seq_num=35080 snap_id=1
wait times: snap=1.878880 sec, exc=1.878880 sec, total=1.878880 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.000011 sec of elapsed time
7: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106018 seq_num=35079 snap_id=1
wait times: snap=0.020171 sec, exc=0.020171 sec, total=0.020171 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050231 sec of elapsed time
8: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106017 seq_num=35078 snap_id=1
wait times: snap=0.020184 sec, exc=0.020184 sec, total=0.020184 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.000016 sec of elapsed time
9: waited for 'RMA: IPC0 completion sync'
send count=0x0, =0x0, =0x23e
wait_id=3106016 seq_num=35077 snap_id=1
wait times: snap=0.020206 sec, exc=0.020206 sec, total=0.020206 sec
wait times: max=infinite
wait counts: calls=1 os=1
occurred after 0.050239 sec of elapsed time
Sampled Session History of session 727 serial 53431
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).

The history is displayed in reverse chronological order.

sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[108 samples, 14:49:58 - 14:51:45]
not in wait at each sample
[9 samples, 14:49:49 - 14:49:57]
idle wait at each sample
[1 sample, 14:49:48]
waited for 'RMA: IPC0 completion sync', seq_num: 35075
p1: 'send count'=0x0
p2: ''=0x0
p3: ''=0x23e
time_waited: 0.020107 sec (sample interval: 0 sec)
[3 samples, 14:49:46 - 14:49:47]
idle wait at each sample
---------------------------------------------------
Sampled Session History Summary:
longest_non_idle_wait: 'RMA: IPC0 completion sync'
[1 sample, 14:49:48 ]
time_waited: 0.020107 sec (sample interval: 0 sec)
---------------------------------------------------
----------------------------------------
kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0
kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0
temporary object counter: 0
last pdbid to increment: 0, last function to increment: (null)
BPS: empty
----------------------------------------
Virtual Thread:
kgskvt: 0x976416cc20, sess: 0x18e1753030/1 sid: 727 ser: 53431
vc: (nil), proc: 0x18e0002c50, id: 727
consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig:
vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2
inwait: 0, short wait event: 0 posted_run: 0
lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)'
lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)'
location where insched last set: kgskthrrun
location where insched last cleared: kgskthrrun1
location where inwait last set: NULL
location where inwait last cleared: NULL
is_assigned: 0, in_sched: 0 (0)
qcls: (nil), qlink: FALSE
vt_active: 0 (pending: 0)
vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0)
ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0)
used quanta (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
exec start consumed time: 0
exec start elapsed time: 0
idle time: 0, active time: 0 (cg: 0)
last updnumps: 0, active time (pq: 0 ps: 0)
cpu yields:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu waits:
stmt: 0, accum: 0, mapped: 0, tot: 0
cpu wait time (usecs):
stmt: 0, accum: 0, mapped: 0, tot: 0
ASL queued time outs: 0, time: 0 (cur 0, cg 0)
PQQ queued time outs: 0, time: 0 (cur 0, cg 0)
Queue timeout violation: 0
calls aborted: 0, num est exec limit hit: 0
undo current: 0k max: 0k
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
KKS-UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 0x977f6f40a0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x977f6f40f0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x977f6f4140
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
KGX Atomic Operation Log 0x977f6f4190
Mutex (nil)(0, 0) idn 0 oper NONE(0)
FSO mutex uid 727 efd 0 whr 0 slp 0
----------------------------------------
KGL-UOL SO Cache(total=0, free=0)
KGX Atomic Operation Log 0x975f73a2d8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a340
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a3a8
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a410
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a478
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a4e0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a548
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a5b0
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
KGX Atomic Operation Log 0x975f73a618
Mutex (nil)(0, 0) idn 0 oper NONE(0)
Library Cache uid 727 efd 0 whr 0 slp 0
oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
pt4=(nil) pt5=(nil) ub4=0 flg=0x0
msk=0000-0000-0000-0000-0000
----------------------------------------
SO: 0x1812141ff8, type: 79, owner: 0x18e1753030, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x18e0002c50, name=dummy, file=ktccts.h LINE:456, pg=0, conuid=1
SGA version=(1,0)
(dummy) nxc=0, nlb=0
===[ Callstack ]===

*** 2019-11-15T14:51:46.868697+04:00 (CDB$ROOT(1))
Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909,
pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 14:51:46 ]
NOTE: scheduling delay has not been sampled for 0.587919 secs
0.000000 secs from [ 14:51:42 - 14:51:47 ], 5 sec avg
0.000000 secs from [ 14:50:47 - 14:51:47 ], 1 min avg
0.000000 secs from [ 14:46:47 - 14:51:47 ], 5 min avg

*** 2019-11-15T14:51:49.174275+04:00 (CDB$ROOT(1))
loadavg : 483.60 153.67 56.17
System user time: 0.06 sys time: 0.02 context switch: 54748
Memory (Avail / Total) = 6431.47M / 772256.64M
Swap (Avail / Total) = 24000.20M / 24576.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:54 ora_ipc0_cdbiamas1
Name: ora_ipc0_cdbiam
State: R (running)
Tgid: 368909
Ngid: 0
Pid: 368909
PPid: 1
TracerPid: 0
Uid: 1001 1001 1001 1001
Gid: 1001 1001 1001 1001
FDSize: 128
Groups: 1001 1002
NStgid: 368909
NSpid: 368909
NSpgid: 368909
NSsid: 368909
VmPeak: 668175752 kB
VmSize: 668175748 kB
VmLck: 0 kB
VmPin: 562562088 kB
VmHWM: 7413828 kB
VmRSS: 7413828 kB
VmData: 8388 kB
VmStk: 156 kB
VmExe: 343516 kB
VmLib: 32424 kB
VmPTE: 14924 kB
VmPMD: 2172 kB
VmSwap: 0 kB
Threads: 1
SigQ: 12/3088249
SigPnd: 0000000000000800
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000016400207
SigCgt: 00000003c9887cf8
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
Seccomp: 0
Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-111
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4589423
nonvoluntary_ctxt_switches: 14717

*** 2019-11-15T14:51:51.870518+04:00 (CDB$ROOT(1))
Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2695 ms

-------------------------------------------------------------------------------
Process diagnostic dump actual duration=5.002000 sec
(max dump time=5.000000 sec)

*** 2019-11-15T14:51:51.870615+04:00 (CDB$ROOT(1))
===[ Heartbeat State Object ]===
(lm heartbeat) state=0x38000001 active noheartbeat
current heartbeat: time 0x5dce82d1 count 0x3403e where 0x1
last poke: time 0x5dce82db req 0x3403b ack 0x3403a freq 10
last wait: state 0x2 id 3106025
last record: time 0x5dce82dd count 0x3403e
last check: time 0x5dce833d
misc: good dur 70 default 70 ckreq 10 tmupdcnt 0 hangcnt 2
Dump of memory from 0x000000979F1E9608 to 0x000000979F1E9818
979F1E9600 9F1D2450 00000097 [P$......]
979F1E9610 E9D93E40 00000096 E9D93E60 00000096 [@>......`>......]
979F1E9620 E9D93E60 00000096 9F1E9628 00000097 [`>......(.......]
979F1E9630 9F1E9628 00000097 E0002C50 00000018 [(.......P,......]
979F1E9640 6000A5A0 00000000 00000001 30435049 [...`........IPC0]
979F1E9650 00000000 38000001 00000000 00000000 [.......8........]
979F1E9660 00000046 00000046 00000000 0003403E [F...F.......>@..]
979F1E9670 5DCE82D1 00000001 00000000 0000000A [...]............]
979F1E9680 5DCE82DB 0003403B 0003403A 0000000A [...];@..:@......]
979F1E9690 0000000A 00000002 5DCE833D 5DCE833D [........=..]=..]]
979F1E96A0 0003403E 5DCE82DD 5DCE8338 00000000 [>@.....]8..]....]
979F1E96B0 0000009A 00000000 00000000 00000000 [................]
979F1E96C0 00000000 00000000 00000000 00000006 [................]
979F1E96D0 00000000 00000000 00000000 00000000 [................]
979F1E96E0 00000002 FFFFFFFF 00000000 00000000 [................]
979F1E96F0 00000000 00000000 002F64E9 00000000 [.........d/.....]
979F1E9700 06322FDE 00000000 00000007 00000000 [./2.............]
979F1E9710 00000000 00000000 00000000 00000000 [................]
Repeat 4 times
979F1E9760 74C300A8 00000097 00000004 00000000 [...t............]
979F1E9770 00000000 00000000 00000000 370000FA [...............7]
979F1E9780 00000005 00000961 5DCE833D 00000000 [....a...=..]....]
979F1E9790 0000000A 00000000 00000000 00000000 [................]
979F1E97A0 00000000 00000000 00000000 00000000 [................]
Repeat 1 times
979F1E97C0 00000000 00000000 9F1E97C8 00000097 [................]
979F1E97D0 9F1E97C8 00000097 60048728 00000000 [........(..`....]
979F1E97E0 9F1D2620 00000097 00000000 00000000 [ &..............]
979F1E97F0 00000000 00000000 00000001 00000000 [................]
979F1E9800 00000000 00000000 00000000 00000000 [................]
979F1E9810 00000000 00000000 [........]
===[ Trace Buffer ]===

*** 2019-11-15T14:51:56.871636+04:00 (CDB$ROOT(1))
=====[PROCESS SNAPSHOTS]=====
SNAPSHOT 0
Time: 2019-11-15 14:51:21.673721 +04:00
Stack:
<Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms
>
Wait Stack: not in wait
Wait History:
0 : 'rdbms ipc message' (waitid 3106025)
snap time=3.003118 sec total time=3.003118 sec
time since last wait=0.000011 sec
1 : 'RMA: IPC0 completion sync' (waitid 3106024)
snap time=0.020098 sec total time=0.020098 sec
time since last wait=0.050228 sec
2 : 'RMA: IPC0 completion sync' (waitid 3106023)
snap time=0.020110 sec total time=0.020110 sec
time since last wait=0.000015 sec
3 : 'RMA: IPC0 completion sync' (waitid 3106022)
snap time=0.020180 sec total time=0.020180 sec
time since last wait=0.049185 sec
4 : 'rdbms ipc message' (waitid 3106021)
snap time=1.051154 sec total time=1.051154 sec
time since last wait=0.000011 sec

SNAPSHOT 1
Time: 2019-11-15 14:51:11.211355 +04:00
Stack:
<Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms
>
Wait Stack: not in wait
Wait History:
0 : 'rdbms ipc message' (waitid 3106025)
snap time=3.003118 sec total time=3.003118 sec
time since last wait=0.000011 sec
1 : 'RMA: IPC0 completion sync' (waitid 3106024)
snap time=0.020098 sec total time=0.020098 sec
time since last wait=0.050228 sec
2 : 'RMA: IPC0 completion sync' (waitid 3106023)
snap time=0.020110 sec total time=0.020110 sec
time since last wait=0.000015 sec
3 : 'RMA: IPC0 completion sync' (waitid 3106022)
snap time=0.020180 sec total time=0.020180 sec
time since last wait=0.049185 sec
4 : 'rdbms ipc message' (waitid 3106021)
snap time=1.051154 sec total time=1.051154 sec
time since last wait=0.000011 sec

SNAPSHOT 2
Time: 2019-11-15 14:50:56.047266 +04:00
Stack:
<Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms
>
Wait Stack: not in wait
Wait History:
0 : 'rdbms ipc message' (waitid 3106025)
snap time=3.003118 sec total time=3.003118 sec
time since last wait=0.000011 sec
1 : 'RMA: IPC0 completion sync' (waitid 3106024)
snap time=0.020098 sec total time=0.020098 sec
time since last wait=0.050228 sec
2 : 'RMA: IPC0 completion sync' (waitid 3106023)
snap time=0.020110 sec total time=0.020110 sec
time since last wait=0.000015 sec
3 : 'RMA: IPC0 completion sync' (waitid 3106022)
snap time=0.020180 sec total time=0.020180 sec
time since last wait=0.049185 sec
4 : 'rdbms ipc message' (waitid 3106021)
snap time=1.051154 sec total time=1.051154 sec
time since last wait=0.000011 sec

SNAPSHOT 3
Time: 2019-11-15 14:50:40.883990 +04:00
Stack:
<Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms
>
Wait Stack: not in wait
Wait History:
0 : 'rdbms ipc message' (waitid 3106025)
snap time=3.003118 sec total time=3.003118 sec
time since last wait=0.000011 sec
1 : 'RMA: IPC0 completion sync' (waitid 3106024)
snap time=0.020098 sec total time=0.020098 sec
time since last wait=0.050228 sec
2 : 'RMA: IPC0 completion sync' (waitid 3106023)
snap time=0.020110 sec total time=0.020110 sec
time since last wait=0.000015 sec
3 : 'RMA: IPC0 completion sync' (waitid 3106022)
snap time=0.020180 sec total time=0.020180 sec
time since last wait=0.049185 sec
4 : 'rdbms ipc message' (waitid 3106021)
snap time=1.051154 sec total time=1.051154 sec
time since last wait=0.000011 sec

SNAPSHOT 4
Time: 2019-11-15 14:50:25.421175 +04:00
Stack:
<Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms
>
Wait Stack: not in wait
Wait History:
0 : 'rdbms ipc message' (waitid 3106025)
snap time=3.003118 sec total time=3.003118 sec
time since last wait=0.000011 sec
1 : 'RMA: IPC0 completion sync' (waitid 3106024)
snap time=0.020098 sec total time=0.020098 sec
time since last wait=0.050228 sec
2 : 'RMA: IPC0 completion sync' (waitid 3106023)
snap time=0.020110 sec total time=0.020110 sec
time since last wait=0.000015 sec
3 : 'RMA: IPC0 completion sync' (waitid 3106022)
snap time=0.020180 sec total time=0.020180 sec
time since last wait=0.049185 sec
4 : 'rdbms ipc message' (waitid 3106021)
snap time=1.051154 sec total time=1.051154 sec
time since last wait=0.000011 sec
==============================

===[ DLM IPC ]===
KJC Communication Dump:
state 0x35 flags 0x0 mode 0x0 inst 1 inc 28
inst=1 inc=28 nrcv=9 nsp=9 nrcvbuf=1000
reg_msg: msgsz=640 mpools=2 cur=795 max=12300 init=17390
mp[0]: cur=792 out=(sga:3602.3530.0 initq:720) max=12297 ini=8695 pavl=0 pmax=0
mp[1]: cur=3 out=(sga:0.0.0 initq:3) max=13 ini=8695 pavl=0 pmax=0
big_msg: msgsz=8504 mpools=2 cur=115 max=947 init=17390
mp[0]: cur=88 out=(sga:0.0.0 initq:88) max=923 ini=8695 pavl=0 pmax=0
mp[1]: cur=53 out=(sga:0.0.0 initq:53) max=132 ini=8695 pavl=0 pmax=0
rsv_msg: msgsz=8504 mpools=1 cur=0 max=1 init=1000
mp[0]: cur=0 out=(sga:0.0.0 initq:0) max=1 tot=1000 pavl=0 pmax=0
rcvr: id=0 (orapid=26 LMD0) grp=1 ospid=368961
rcvr: id=8 (orapid=34 LMD2) grp=1 ospid=368989
rcvr: id=1 (orapid=28 LMS0) grp=2 ospid=368963
rcvr: id=5 (orapid=32 LMS4) grp=2 ospid=368977
rcvr: id=6 (orapid=27 LMS5) grp=2 ospid=368981
rcvr: id=4 (orapid=25 LMS3) grp=2 ospid=368972
rcvr: id=2 (orapid=23 LMS1) grp=2 ospid=368965
rcvr: id=7 (orapid=29 LMD1) grp=1 ospid=368983
rcvr: id=3 (orapid=30 LMS2) grp=2 ospid=368969
send proxy: id=0 orapid=26 ndst=1 (2:0 ) pso=0x18e000d1f0.(nil) state=x0
send proxy: id=8 orapid=34 ndst=1 (2:8 ) pso=0x18e0011430.(nil) state=x0
send proxy: id=1 orapid=28 ndst=1 (2:1 ) pso=0x18e000e280.(nil) state=x0
send proxy: id=5 orapid=32 ndst=1 (2:5 ) pso=0x18e00103a0.(nil) state=x0
send proxy: id=6 orapid=27 ndst=1 (2:6 ) pso=0x566000d1f0.(nil) state=x0
send proxy: id=4 orapid=25 ndst=1 (2:4 ) pso=0x566000c160.(nil) state=x0
send proxy: id=2 orapid=23 ndst=1 (2:2 ) pso=0x566000b0d0.(nil) state=x0
send proxy: id=7 orapid=29 ndst=1 (2:7 ) pso=0x566000e280.(nil) state=x0
send proxy: id=3 orapid=30 ndst=1 (2:3 ) pso=0x18e000f310.(nil) state=x0
GES resource limits:
ges resources: cur=242473 max=503402 ini=241332
ges enqueues: cur=124210 max=157460 ini=363099
ges cresources: cur=124812 max=132896
gcs resources: cur=21223354 max=25452045 ini=77122167
gcs shadows: cur=18628493 max=18812915 ini=77122167
KJCTS state: seq-check=no timeout=yes waitticks=0x3
highload=yes comm_interface=msgq-drm (rds)
GES destination context:
GES remote instance per receiver context:
GES destination context:
Dest=2 rcvr=0 inc=28 state=0x10041 tstate=0x0
batch-type=quick bmsg=(nil) tmout=0xd626f14 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=249 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 81)
sndq_flushs=44665413 last_flush_time=0x89a63d5cb flush_idx=2
sndq_flush_stats=
[2]v:44665413|w:114|f:0|l:1|t:1000|s:1|m:1
[1]v:44665412|w:43|f:0|l:1|t:999|s:1|m:1
[0]v:44665411|w:143|f:0|l:1|t:1000|s:1|m:1
[9]v:44665410|w:0|f:0|l:1|t:1000|s:1|m:1
[8]v:44665409|w:4|f:0|l:1|t:1000|s:1|m:1
[7]v:44665408|w:4|f:0|l:1|t:998|s:1|m:1
[6]v:44665407|w:4|f:0|l:1|t:1000|s:1|m:1
[5]v:44665406|w:28|f:0|l:1|t:999|s:1|m:1
[4]v:44665405|w:8|f:0|l:1|t:1000|s:1|m:1
[3]v:44665404|w:36|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=168982587 ack=0 snt=168982587 sync seq=0.168982587 inc=0
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.153767097,[PCM]0.0,[ALL]0.168982587
batch_snds=1683713 tot_msgs=3533474 max_sz=38 full_load=0 snd_seq=1683713
pbatch_snds=19271455 tot_msgs=19455245 sndq_msg_tot=46516774 tm=(0 5671)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 214 wqlen 29424158
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=1 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x13b93631 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 98)
sndq_flushs=43453450 last_flush_time=0x89a63d5cb flush_idx=8
sndq_flush_stats=
[8]v:43453450|w:105|f:0|l:1|t:1000|s:1|m:1
[7]v:43453449|w:7|f:0|l:1|t:997|s:1|m:1
[6]v:43453448|w:28|f:0|l:1|t:998|s:1|m:1
[5]v:43453447|w:7|f:0|l:1|t:1000|s:1|m:1
[4]v:43453445|w:1|f:0|l:1|t:999|s:0|m:0
[3]v:43453444|w:0|f:0|l:3|t:999|s:0|m:0
[2]v:43453443|w:8|f:0|l:8|t:999|s:0|m:0
[1]v:43453442|w:11|f:0|l:1|t:1000|s:1|m:1
[0]v:43453441|w:18|f:0|l:1|t:1000|s:1|m:1
[9]v:43453440|w:9|f:0|l:1|t:999|s:1|m:1
seq wrp=0 lst=132509668 ack=0 snt=132509668 sync seq=0.132509668 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85809507,[PCM]0.132509668,[ALL]0.132509668
batch_snds=1399916 tot_msgs=5033794 max_sz=96 full_load=642 snd_seq=1399916
pbatch_snds=21225985 tot_msgs=28383853 sndq_msg_tot=46912409 tm=(0 49774893)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41601174
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=2 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x109a9ca1 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 97)
sndq_flushs=42915584 last_flush_time=0x89a63d581 flush_idx=0
sndq_flush_stats=
[0]v:42915584|w:74|f:0|l:1|t:1000|s:1|m:1
[9]v:42915582|w:14|f:0|l:5|t:1000|s:0|m:0
[8]v:42915581|w:8|f:0|l:1|t:1000|s:1|m:1
[7]v:42915580|w:15|f:0|l:1|t:999|s:1|m:1
[6]v:42915579|w:12|f:0|l:1|t:1000|s:1|m:1
[5]v:42915578|w:5|f:0|l:1|t:1000|s:1|m:1
[4]v:42915577|w:1|f:0|l:1|t:998|s:1|m:1
[3]v:42915576|w:7|f:0|l:1|t:999|s:1|m:1
[2]v:42915575|w:57|f:0|l:1|t:1000|s:1|m:1
[1]v:42915574|w:15|f:0|l:1|t:994|s:1|m:1
seq wrp=0 lst=133270032 ack=0 snt=133270028 sync seq=0.133270028 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.89088728,[PCM]0.133270028,[ALL]0.133270028
batch_snds=1426711 tot_msgs=5103727 max_sz=77 full_load=569 snd_seq=1426711
pbatch_snds=20722666 tot_msgs=27863854 sndq_msg_tot=46412781 tm=(0 50946416)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 364 wqlen 41640990
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=3 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x11400698 msg_in_batch=0
tkt_total=1000 (min=1000) avl=739 sp_rsv=246 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 95)
sndq_flushs=43540002 last_flush_time=0x89a63d5eb flush_idx=9
sndq_flush_stats=
[9]v:43540002|w:30|f:0|l:1|t:988|s:1|m:1
[8]v:43540001|w:6|f:0|l:1|t:997|s:1|m:1
[7]v:43540000|w:6|f:0|l:1|t:999|s:1|m:1
[6]v:43539999|w:29|f:0|l:1|t:1000|s:1|m:1
[5]v:43539998|w:35|f:0|l:1|t:998|s:1|m:1
[4]v:43539997|w:0|f:0|l:1|t:999|s:1|m:1
[3]v:43539996|w:30|f:0|l:1|t:1000|s:1|m:1
[2]v:43539995|w:23|f:0|l:1|t:998|s:1|m:1
[1]v:43539994|w:9|f:0|l:1|t:999|s:1|m:1
[0]v:43539993|w:16|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=134354755 ack=0 snt=134354753 sync seq=0.134354753 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.86869831,[PCM]0.134354753,[ALL]0.134354753
batch_snds=1432721 tot_msgs=5118238 max_sz=91 full_load=605 snd_seq=1432721
pbatch_snds=21305828 tot_msgs=28448445 sndq_msg_tot=47044023 tm=(0 51175426)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 375 wqlen 41622836
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=4 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x11690c6e msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 103)
sndq_flushs=43169094 last_flush_time=0x89a63d5eb flush_idx=2
sndq_flush_stats=
[2]v:43169094|w:41|f:0|l:1|t:991|s:1|m:1
[1]v:43169093|w:64|f:0|l:1|t:993|s:1|m:1
[0]v:43169092|w:70|f:0|l:1|t:1000|s:1|m:1
[9]v:43169091|w:18|f:0|l:1|t:989|s:1|m:1
[8]v:43169090|w:8|f:0|l:1|t:995|s:1|m:1
[7]v:43169089|w:16|f:0|l:1|t:999|s:1|m:1
[6]v:43169088|w:8|f:0|l:1|t:1000|s:1|m:1
[5]v:43169087|w:1|f:0|l:1|t:998|s:1|m:1
[4]v:43169086|w:4|f:0|l:1|t:999|s:1|m:1
[3]v:43169085|w:5|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=143198820 ack=0 snt=143198818 sync seq=0.143198818 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.96323855,[PCM]0.143198818,[ALL]0.143198818
batch_snds=1427975 tot_msgs=5135084 max_sz=59 full_load=601 snd_seq=1427975
pbatch_snds=20915386 tot_msgs=28054583 sndq_msg_tot=46696674 tm=(0 50968735)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 346 wqlen 41912789
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=5 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x139c2141 msg_in_batch=0
tkt_total=1000 (min=1000) avl=748 sp_rsv=247 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 94)
sndq_flushs=43574952 last_flush_time=0x89a63d5f0 flush_idx=1
sndq_flush_stats=
[1]v:43574952|w:4|f:0|l:1|t:996|s:1|m:1
[0]v:43574951|w:32|f:0|l:1|t:997|s:1|m:1
[9]v:43574950|w:7|f:0|l:1|t:999|s:1|m:1
[8]v:43574949|w:16|f:0|l:1|t:990|s:1|m:1
[7]v:43574948|w:1|f:0|l:1|t:994|s:1|m:1
[6]v:43574947|w:33|f:0|l:1|t:995|s:1|m:1
[5]v:43574946|w:23|f:0|l:1|t:997|s:1|m:1
[4]v:43574945|w:113|f:0|l:1|t:1000|s:1|m:1
[3]v:43574944|w:1|f:0|l:1|t:998|s:1|m:1
[2]v:43574943|w:8|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=137471407 ack=0 snt=137471407 sync seq=0.137471407 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.93123683,[PCM]0.137471407,[ALL]0.137471407
batch_snds=1415057 tot_msgs=5079786 max_sz=74 full_load=587 snd_seq=1415057
pbatch_snds=21334732 tot_msgs=28460294 sndq_msg_tot=47061791 tm=(0 49473126)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41482478
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=6 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0x14b8e6de msg_in_batch=0
tkt_total=1000 (min=1000) avl=749 sp_rsv=246 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 96)
sndq_flushs=43192233 last_flush_time=0x89a63d5e8 flush_idx=0
sndq_flush_stats=
[0]v:43192233|w:30|f:0|l:1|t:996|s:1|m:1
[9]v:43192232|w:5|f:1|l:1|t:998|s:1|m:1
[8]v:43192231|w:2|f:0|l:1|t:999|s:1|m:1
[7]v:43192230|w:145|f:0|l:1|t:1000|s:1|m:1
[6]v:43192229|w:18|f:0|l:1|t:1000|s:1|m:1
[5]v:43192228|w:7|f:0|l:1|t:993|s:1|m:1
[4]v:43192227|w:33|f:0|l:1|t:994|s:1|m:1
[3]v:43192226|w:46|f:0|l:1|t:1000|s:1|m:1
[2]v:43192225|w:12|f:0|l:1|t:1000|s:1|m:1
[1]v:43192224|w:21|f:0|l:1|t:992|s:1|m:1
seq wrp=0 lst=129461745 ack=0 snt=129461745 sync seq=0.129461745 inc=28
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85305884,[PCM]0.129461745,[ALL]0.129461745
batch_snds=1414226 tot_msgs=5124499 max_sz=67 full_load=652 snd_seq=1414226
pbatch_snds=20886062 tot_msgs=28060625 sndq_msg_tot=46720607 tm=(0 50089865)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 359 wqlen 42072275
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=7 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=(nil) tmout=0xdc9ab30 msg_in_batch=0
tkt_total=1000 (min=1000) avl=750 sp_rsv=249 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 54)
sndq_flushs=46048808 last_flush_time=0x89a63d5ef flush_idx=8
sndq_flush_stats=
[8]v:46048808|w:28|f:0|l:1|t:1000|s:1|m:1
[7]v:46048807|w:7|f:0|l:1|t:1000|s:1|m:1
[6]v:46048806|w:58|f:0|l:1|t:1000|s:1|m:1
[5]v:46048805|w:56|f:0|l:1|t:1000|s:1|m:1
[4]v:46048804|w:44|f:0|l:1|t:1000|s:1|m:1
[3]v:46048803|w:54|f:0|l:1|t:1000|s:1|m:1
[2]v:46048802|w:11|f:0|l:1|t:1000|s:1|m:1
[1]v:46048800|w:36|f:0|l:2|t:1000|s:0|m:0
[0]v:46048799|w:29|f:0|l:1|t:1000|s:1|m:1
[9]v:46048798|w:58|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=163063310 ack=0 snt=163063310 sync seq=0.163063310 inc=0
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.210,[PCM]0.0,[ALL]0.163063310
batch_snds=1938366 tot_msgs=3972972 max_sz=39 full_load=0 snd_seq=1938366
pbatch_snds=14297054 tot_msgs=14477178 sndq_msg_tot=48085319 tm=(0 5829)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 110 wqlen 35950230
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
Dest=2 rcvr=8 inc=28 state=0x50041 tstate=0x0
batch-type=auto bmsg=0x7ff182378028 tmout=0xcf12c1c msg_in_batch=2
tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250
tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0)
tkt_inc=0 tkt_dec=0 tkt_rel_time 0
nullreq_sends=0 nullack_rcvs=(0 55)
sndq_flushs=38362705 last_flush_time=0x89a63d537 flush_idx=6
sndq_flush_stats=
[6]v:38362705|w:38|f:0|l:2|t:1000|s:0|m:0
[5]v:38362704|w:42|f:0|l:1|t:1000|s:1|m:1
[4]v:38362703|w:0|f:0|l:1|t:998|s:1|m:1
[3]v:38362702|w:92|f:0|l:1|t:999|s:1|m:1
[2]v:38362701|w:41|f:0|l:1|t:998|s:1|m:1
[1]v:38362700|w:1|f:0|l:1|t:998|s:1|m:1
[0]v:38362699|w:32|f:0|l:1|t:999|s:1|m:1
[9]v:38362698|w:78|f:0|l:1|t:1000|s:1|m:1
[8]v:38362697|w:33|f:0|l:1|t:999|s:1|m:1
[7]v:38362696|w:200|f:0|l:1|t:1000|s:1|m:1
seq wrp=0 lst=161549421 ack=0 snt=161549416 sync seq=0.161549421 inc=0
sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.218,[PCM]0.0,[ALL]0.161549421
batch_snds=1587448 tot_msgs=3357433 max_sz=45 full_load=0 snd_seq=1587448
pbatch_snds=15617786 tot_msgs=15800589 sndq_msg_tot=40133677 tm=(0 4194)
sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 218 wqlen 26651993
sndq msg 0 start_tm 0 end_tm 0
PSends-rcvr/prxy/tkts 0/0/0
GES remote instance per receiver context:
Rcvr=0 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.169439835 sync_rcv_seq=0.169439835 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1452902
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.166858040
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=8 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.160790094 sync_rcv_seq=0.160790090 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1208972
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.158381798
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=1 dest=2 inc=28 state=0x103 tstate=0x0 nak=4 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.141253146 sync_rcv_seq=0.141253146 sync_ack_seq=0.129056901 inc=28 batch_rcv_seq=1845673
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.134058573
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=5 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.141358734 sync_rcv_seq=0.141358734 sync_ack_seq=0.131227202 inc=28 batch_rcv_seq=1858892
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.135396667
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=6 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.150277858 sync_rcv_seq=0.150277856 sync_ack_seq=0.137776658 inc=28 batch_rcv_seq=1893521
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.142779468
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=4 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.125330474 sync_rcv_seq=0.125330474 sync_ack_seq=0.114702593 inc=28 batch_rcv_seq=1863564
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.119202571
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=2 dest=2 inc=28 state=0x103 tstate=0x0 nak=11 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.124482338 sync_rcv_seq=0.124482338 sync_ack_seq=0.114194111 inc=28 batch_rcv_seq=1875249
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.118453984
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=7 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.165116380 sync_rcv_seq=0.165116380 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1989097
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.163643671
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=3 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500
rcv_seq=0.120370819 sync_rcv_seq=0.120370818 sync_ack_seq=0.110107175 inc=28 batch_rcv_seq=1849905
work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.114270602
WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0
alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged
alloc_history: unchanged
cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets
Rcvr=0 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:111)
msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0
rmpool: alloc=1334 (r:1311 s:23) avail=134 (r:111 s:23) max=(s:23)
hist_last_rcv_tm=0x89a63d5cb count=224554961 idx=4
[4]t:x89a63d44b|c:39|w:11|x:(33.6.0.0)57|m:29
[3]t:x89a635f15|c:2505|w:11|x:(2193.312.0.0)99|m:1696
[2]t:x89a62e9aa|c:2574|w:11|x:(2251.323.0.0)99|m:1817
[1]t:x89a62746e|c:19120|w:1|x:(18828.292.0.0)99|m:14268
[0]t:x89a61ff34|c:2824|w:10|x:(2526.298.0.0)99|m:2057
[9]t:x89a618a02|c:3096|w:9|x:(2785.311.0.0)99|m:2195
[8]t:x89a6114b8|c:3643|w:8|x:(3348.295.0.0)99|m:2806
[7]t:x89a609f72|c:3880|w:7|x:(3598.282.0.0)99|m:3044
[6]t:x89a602a2a|c:17602|w:1|x:(17318.284.0.0)99|m:15328
[5]t:x89a5fb4e7|c:3445|w:8|x:(3155.290.0.0)99|m:2819
ibatch: 2[1372539] 3[53116] 4[16271] 5[4951] 6[2397] 7[1341]
ibatch: 8[909] 9[761] 10[450] 11[173] 12[87] 13[63]
ibatch: 14[28] 15[15] 16[11] 17[6]
ibatch: sum=1453118 rcvs=1453228 tot_msgs=3042837 max_sz=35 avg_sz=2
pbatch: 2[7672] 3[415] 4[136] 5[61] 6[57] 7[17]
pbatch: 8[4] 9[3] 10[4] 11[1] 12[0] 13[4]
pbatch: 14[1] 15[5] 16[4] 17[1]
pbatch: sum=8385 rcvs=20424780 tot_msgs=20579618 max_sz=103 avg_sz=2
Rcvr=8 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:157)
msgs_rdyq=0 watq=0 nsq=0 routq=1201 msgq=0
rmpool: alloc=1380 (r:1357 s:23) avail=179 (r:157 s:22) max=(s:23)
hist_last_rcv_tm=0x89a63d59a count=217132059 idx=3
[3]t:x89a6364ac|c:2365|w:12|x:(2057.308.0.0)99|m:1568
[2]t:x89a62ef5b|c:2631|w:11|x:(2307.324.0.0)99|m:1803
[1]t:x89a627a0c|c:16992|w:1|x:(16699.293.0.0)99|m:12695
[0]t:x89a6204bb|c:2630|w:11|x:(2297.333.0.0)99|m:1843
[9]t:x89a618f38|c:3198|w:9|x:(2887.311.0.0)99|m:2092
[8]t:x89a611a02|c:2654|w:11|x:(2321.333.0.0)99|m:1778
[7]t:x89a60a49e|c:2621|w:11|x:(2306.315.0.0)99|m:1828
[6]t:x89a602f6d|c:15008|w:2|x:(14713.295.0.0)99|m:13063
[5]t:x89a5fba3d|c:1997|w:15|x:(1652.345.0.0)99|m:1560
[4]t:x89a5f44ed|c:2134|w:14|x:(1795.339.0.0)99|m:1621
ibatch: 2[1113843] 3[61797] 4[20539] 5[6821] 6[3212] 7[1341]
ibatch: 8[743] 9[391] 10[198] 11[116] 12[59] 13[35]
ibatch: 14[21] 15[15] 16[7] 17[9]
ibatch: sum=1209147 rcvs=1209233 tot_msgs=2573532 max_sz=38 avg_sz=2
pbatch: 2[5158] 3[535] 4[201] 5[91] 6[37] 7[31]
pbatch: 8[20] 9[19] 10[13] 11[3] 12[6] 13[10]
pbatch: 14[5] 15[6] 16[2] 17[9]
pbatch: sum=6146 rcvs=19070301 tot_msgs=19222388 max_sz=103 avg_sz=2
Rcvr=1 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:364)
msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0
rmpool: alloc=1633 (r:1564 s:69) avail=429 (r:364 s:65) max=(s:69)
hist_last_rcv_tm=0x89a63d5f6 count=330864285 idx=1
[1]t:x89a63d0ce|c:397|w:3|x:(397.0.0.0)30|m:178
[0]t:x89a635b9b|c:7102|w:4|x:(7102.0.0.0)30|m:2729
[9]t:x89a62e661|c:8054|w:3|x:(8054.0.0.0)30|m:3383
[8]t:x89a62712e|c:6558|w:4|x:(6558.0.0.0)30|m:2955
[7]t:x89a61fbfd|c:7561|w:3|x:(7561.0.0.0)30|m:3010
[6]t:x89a6186bf|c:5868|w:5|x:(5868.0.0.0)30|m:2357
[5]t:x89a61118a|c:5499|w:5|x:(5499.0.0.0)30|m:2345
[4]t:x89a609c55|c:6241|w:4|x:(6241.0.0.0)30|m:2705
[3]t:x89a60271c|c:7254|w:4|x:(7254.0.0.0)30|m:2809
[2]t:x89a5fb1e5|c:6797|w:4|x:(6797.0.0.0)30|m:3079
ibatch: 2[1220451] 3[251625] 4[102841] 5[55889] 6[38198] 7[30966]
ibatch: 8[23999] 9[20125] 10[16840] 11[15151] 12[13532] 13[12454]
ibatch: 14[11550] 15[10833] 16[9778] 17[3805]
ibatch: sum=1838037 rcvs=1849712 tot_msgs=6174530 max_sz=56 avg_sz=3
pbatch: 2[352287] 3[202797] 4[123721] 5[93438] 6[93402] 7[104490]
pbatch: 8[27372] 9[18633] 10[17937] 11[15917] 12[15737] 13[13953]
pbatch: 14[13953] 15[12735] 16[14167] 17[92343]
pbatch: sum=1212882 rcvs=20362418 tot_msgs=32519618 max_sz=74 avg_sz=5
Rcvr=5 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341)
msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0
rmpool: alloc=1610 (r:1541 s:69) avail=406 (r:341 s:65) max=(s:69)
hist_last_rcv_tm=0x89a63d5f0 count=328941849 idx=1
[1]t:x89a63c4ef|c:936|w:4|x:(936.0.0.0)30|m:424
[0]t:x89a634fbf|c:5214|w:5|x:(5214.0.0.0)30|m:2193
[9]t:x89a62da8c|c:5846|w:5|x:(5846.0.0.0)30|m:2312
[8]t:x89a626543|c:5009|w:5|x:(5009.0.0.0)30|m:2451
[7]t:x89a61f00d|c:4569|w:6|x:(4569.0.0.0)30|m:1761
[6]t:x89a617ad2|c:3664|w:8|x:(3664.0.0.0)30|m:1473
[5]t:x89a610592|c:4264|w:7|x:(4262.2.0.0)31|m:1713
[4]t:x89a609049|c:4347|w:6|x:(4347.0.0.0)30|m:1854
[3]t:x89a601b18|c:4616|w:6|x:(4616.0.0.0)30|m:1556
[2]t:x89a5fa5e7|c:4752|w:6|x:(4752.0.0.0)30|m:2229
ibatch: 2[1225519] 3[254849] 4[103673] 5[56410] 6[38778] 7[30966]
ibatch: 8[24303] 9[20643] 10[17098] 11[15454] 12[13628] 13[12816]
ibatch: 14[11669] 15[10919] 16[9954] 17[4018]
ibatch: sum=1850697 rcvs=1862931 tot_msgs=6243714 max_sz=52 avg_sz=3
pbatch: 2[350367] 3[200988] 4[125539] 5[93312] 6[94203] 7[104713]
pbatch: 8[28340] 9[18640] 10[18044] 11[15789] 12[16168] 13[13836]
pbatch: 14[13789] 15[12668] 16[13887] 17[92690]
pbatch: sum=1212973 rcvs=19829883 tot_msgs=31998870 max_sz=74 avg_sz=5
Rcvr=6 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318)
msgs_rdyq=0 watq=0 nsq=0 routq=1202 msgq=0
rmpool: alloc=1564 (r:1518 s:46) avail=362 (r:318 s:44) max=(s:46)
hist_last_rcv_tm=0x89a63d5e8 count=347615844 idx=1
[1]t:x89a63bb73|c:1545|w:4|x:(1545.0.0.0)30|m:659
[0]t:x89a63463c|c:6785|w:4|x:(6785.0.0.0)30|m:2705
[9]t:x89a62d10c|c:7888|w:3|x:(7888.0.0.0)30|m:2981
[8]t:x89a625bd0|c:8434|w:3|x:(8434.0.0.0)30|m:3542
[7]t:x89a61e697|c:6469|w:4|x:(6469.0.0.0)30|m:2630
[6]t:x89a617150|c:5699|w:5|x:(5698.1.0.0)31|m:2280
[5]t:x89a60fc19|c:6267|w:4|x:(6267.0.0.0)30|m:2647
[4]t:x89a6086e8|c:5200|w:5|x:(5200.0.0.0)30|m:2163
[3]t:x89a6011b7|c:6625|w:4|x:(6625.0.0.0)30|m:2467
[2]t:x89a5f9c7f|c:5407|w:5|x:(5406.1.0.0)31|m:2495
ibatch: 2[1251432] 3[259009] 4[106705] 5[56219] 6[39923] 7[29475]
ibatch: 8[24835] 9[19768] 10[18089] 11[15360] 12[14221] 13[12597]
ibatch: 14[12181] 15[11011] 16[10537] 17[4081]
ibatch: sum=1885443 rcvs=1897777 tot_msgs=6349814 max_sz=54 avg_sz=3
pbatch: 2[357537] 3[210028] 4[123428] 5[95485] 6[91888] 7[107378]
pbatch: 8[25641] 9[20011] 10[16849] 11[16988] 12[14833] 13[14752]
pbatch: 14[13117] 15[13450] 16[13519] 17[93193]
pbatch: sum=1228097 rcvs=19965547 tot_msgs=32153157 max_sz=74 avg_sz=5
Rcvr=4 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341)
msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0
rmpool: alloc=1587 (r:1541 s:46) avail=383 (r:341 s:42) max=(s:46)
hist_last_rcv_tm=0x89a63d5ef count=291802152 idx=0
[0]t:x89a638682|c:4669|w:4|x:(4669.0.0.0)30|m:2161
[9]t:x89a63114b|c:6950|w:4|x:(6950.0.0.0)30|m:3238
[8]t:x89a629c1a|c:10508|w:2|x:(10508.0.0.0)30|m:4547
[7]t:x89a6226e8|c:10742|w:2|x:(10742.0.0.0)30|m:3977
[6]t:x89a61b1ad|c:5043|w:5|x:(5043.0.0.0)30|m:1965
[5]t:x89a613c72|c:6451|w:4|x:(6451.0.0.0)30|m:2806
[4]t:x89a60c742|c:5526|w:5|x:(5526.0.0.0)30|m:2345
[3]t:x89a605211|c:7198|w:4|x:(7198.0.0.0)30|m:2769
[2]t:x89a5fdcd1|c:7269|w:4|x:(7269.0.0.0)30|m:3115
[1]t:x89a5f678c|c:5222|w:5|x:(5222.0.0.0)30|m:2593
ibatch: 2[1235086] 3[252832] 4[104503] 5[54131] 6[38981] 7[29149]
ibatch: 8[24513] 9[19668] 10[17880] 11[15098] 12[13992] 13[12561]
ibatch: 14[12222] 15[11097] 16[10306] 17[3819]
ibatch: sum=1855838 rcvs=1868102 tot_msgs=6256714 max_sz=51 avg_sz=3
pbatch: 2[344222] 3[202533] 4[120299] 5[95235] 6[92709] 7[105848]
pbatch: 8[26248] 9[19663] 10[17784] 11[17509] 12[14867] 13[14646]
pbatch: 14[13139] 15[13514] 16[13571] 17[92942]
pbatch: sum=1204729 rcvs=20478276 tot_msgs=32639998 max_sz=74 avg_sz=5
Rcvr=2 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318)
msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0
rmpool: alloc=1587 (r:1518 s:69) avail=384 (r:318 s:66) max=(s:69)
hist_last_rcv_tm=0x89a63d5ea count=278284752 idx=0
[0]t:x89a638b1b|c:4257|w:4|x:(4257.0.0.0)30|m:1778
[9]t:x89a6315dd|c:6166|w:4|x:(6166.0.0.0)30|m:2788
[8]t:x89a62a09b|c:5382|w:5|x:(5382.0.0.0)30|m:2632
[7]t:x89a622b69|c:4661|w:6|x:(4661.0.0.0)30|m:2142
[6]t:x89a61b61e|c:3939|w:7|x:(3939.0.0.0)30|m:1740
[5]t:x89a6140e4|c:4523|w:6|x:(4521.2.0.0)31|m:1908
[4]t:x89a60cbaa|c:3731|w:8|x:(3731.0.0.0)30|m:1652
[3]t:x89a60567a|c:5259|w:5|x:(5259.0.0.0)30|m:1912
[2]t:x89a5fe148|c:4399|w:6|x:(4399.0.0.0)30|m:1996
[1]t:x89a5f6c02|c:3823|w:7|x:(3823.0.0.0)30|m:1899
ibatch: 2[1228115] 3[255833] 4[107024] 5[56905] 6[40932] 7[31133]
ibatch: 8[25764] 9[21059] 10[18571] 11[15973] 12[14775] 13[12869]
ibatch: 14[12242] 15[11229] 16[10393] 17[4093]
ibatch: sum=1866910 rcvs=1879521 tot_msgs=6368352 max_sz=54 avg_sz=3
pbatch: 2[353488] 3[206297] 4[122256] 5[94495] 6[91116] 7[105765]
pbatch: 8[25985] 9[19548] 10[16768] 11[17365] 12[14744] 13[14894]
pbatch: 14[13731] 15[13582] 16[13514] 17[92954]
pbatch: sum=1216502 rcvs=19822972 tot_msgs=31994115 max_sz=74 avg_sz=5
Rcvr=7 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:180)
msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0
rmpool: alloc=1403 (r:1380 s:23) avail=203 (r:180 s:23) max=(s:23)
hist_last_rcv_tm=0x89a63d5ef count=231320391 idx=3
[3]t:x89a63bef7|c:640|w:9|x:(585.55.0.0)98|m:388
[2]t:x89a6349c7|c:2431|w:12|x:(2082.349.0.0)99|m:1514
[1]t:x89a62d45b|c:2683|w:11|x:(2377.306.0.0)99|m:1711
[0]t:x89a625f09|c:20508|w:1|x:(20227.281.0.0)99|m:14946
[9]t:x89a61e9d2|c:2751|w:10|x:(2419.332.0.0)99|m:1722
[8]t:x89a61745d|c:2983|w:10|x:(2667.316.0.0)99|m:1876
[7]t:x89a60ff24|c:3786|w:7|x:(3473.313.0.0)99|m:2292
[6]t:x89a6089eb|c:5113|w:5|x:(4776.337.0.0)99|m:3577
[5]t:x89a6014a3|c:16750|w:1|x:(16441.309.0.0)99|m:14276
[4]t:x89a5f9f6b|c:3434|w:8|x:(3095.339.0.0)99|m:2341
ibatch: 2[1870533] 3[105709] 4[10127] 5[1884] 6[732] 7[414]
ibatch: 8[170] 9[81] 10[69] 11[30] 12[19] 13[26]
ibatch: 14[4] 15[0] 16[1] 17[1]
ibatch: sum=1989800 rcvs=1989864 tot_msgs=4119303 max_sz=23 avg_sz=2
pbatch: 2[6307] 3[414] 4[92] 5[29] 6[15] 7[5]
pbatch: 8[4] 9[3] 10[5] 11[5] 12[3] 13[8]
pbatch: 14[4] 15[4] 16[6] 17[8]
pbatch: sum=6912 rcvs=16772328 tot_msgs=16925186 max_sz=103 avg_sz=2
Rcvr=3 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341)
msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0
rmpool: alloc=1610 (r:1541 s:69) avail=406 (r:341 s:65) max=(s:69)
hist_last_rcv_tm=0x89a63d5eb count=289402308 idx=0
[0]t:x89a63942a|c:2649|w:6|x:(2649.0.0.0)30|m:1088
[9]t:x89a631ef8|c:5892|w:5|x:(5892.0.0.0)30|m:2360
[8]t:x89a62a9be|c:5590|w:5|x:(5590.0.0.0)30|m:2162
[7]t:x89a623484|c:5274|w:5|x:(5274.0.0.0)30|m:2524
[6]t:x89a61bf46|c:5719|w:5|x:(5719.0.0.0)30|m:2388
[5]t:x89a614a01|c:4346|w:6|x:(4346.0.0.0)30|m:1769
[4]t:x89a60d4c5|c:5610|w:5|x:(5610.0.0.0)30|m:2220
[3]t:x89a605f8d|c:4747|w:6|x:(4747.0.0.0)30|m:1892
[2]t:x89a5fea57|c:4861|w:6|x:(4861.0.0.0)30|m:1844
[1]t:x89a5f7526|c:4966|w:6|x:(4966.0.0.0)30|m:2315
ibatch: 2[1221328] 3[251740] 4[102912] 5[56760] 6[39176] 7[30881]
ibatch: 8[24076] 9[20703] 10[16863] 11[15387] 12[13402] 13[12711]
ibatch: 14[11581] 15[10830] 16[9697] 17[3984]
ibatch: sum=1842031 rcvs=1855319 tot_msgs=6260211 max_sz=48 avg_sz=3
pbatch: 2[360085] 3[204157] 4[124611] 5[92957] 6[94169] 7[105197]
pbatch: 8[28209] 9[17678] 10[18665] 11[15940] 12[15763] 13[13774]
pbatch: 14[13801] 15[12728] 16[13885] 17[93027]
pbatch: sum=1224646 rcvs=20361103 tot_msgs=32542739 max_sz=74 avg_sz=5
GES PGA send message buffer usage:
orapid=23 ospid=368965 pmp: alloc=69 avail=66 max=69
orapid=25 ospid=368972 pmp: alloc=46 avail=42 max=46
orapid=26 ospid=368961 pmp: alloc=23 avail=23 max=23
orapid=27 ospid=368981 pmp: alloc=46 avail=44 max=46
orapid=28 ospid=368963 pmp: alloc=69 avail=65 max=69
orapid=29 ospid=368983 pmp: alloc=23 avail=23 max=23
orapid=30 ospid=368969 pmp: alloc=69 avail=65 max=69
orapid=32 ospid=368977 pmp: alloc=69 avail=65 max=69
orapid=34 ospid=368989 pmp: alloc=23 avail=22 max=23
orapid=71 ospid=369094 pmp: alloc=0 avail=0 max=23
orapid=74 ospid=369090 pmp: alloc=0 avail=0 max=23
orapid=75 ospid=369121 pmp: alloc=0 avail=0 max=23
orapid=78 ospid=369098 pmp: alloc=0 avail=0 max=23
orapid=79 ospid=369123 pmp: alloc=0 avail=0 max=23
orapid=82 ospid=369108 pmp: alloc=0 avail=0 max=23
kjctseventdump-begin tail 231 heads 218 @ 0 231 @ -817936614
#0 id 439 exid 77713 dests 00
#1 id 439 exid 73326 dests 00
#2 id 439 exid 77713 dests 00
#3 id 439 exid 73326 dests 00
#4 id 439 exid 19919 dests 00
#5 id 439 exid 73326 dests 00
#6 id 439 exid 19919 dests 00
#7 id 439 exid 73326 dests 00
#8 id 439 exid 77713 dests 00
#9 id 439 exid 19919 dests 00
#10 id 439 exid 73326 dests 00
#11 id 439 exid 19919 dests 00
#12 id 439 exid 73326 dests 00
#13 id 439 exid 77713 dests 00
#14 id 439 exid 73326 dests 00
#15 id 439 exid 73326 dests 00
#16 id 439 exid 77713 dests 00
#17 id 439 exid 73326 dests 00
#18 id 439 exid 19919 dests 00
#19 id 439 exid 73326 dests 00
#20 id 439 exid 19919 dests 00
#21 id 439 exid 73326 dests 00
#22 id 439 exid 19919 dests 00
#23 id 439 exid 73326 dests 00
#24 id 439 exid 77713 dests 00
#25 id 439 exid 77713 dests 00
#26 id 439 exid 73326 dests 00
#27 id 439 exid 19919 dests 00
#28 id 439 exid 73326 dests 00
#29 id 439 exid 73326 dests 00
#30 id 439 exid 73326 dests 00
#31 id 439 exid 19919 dests 00
#32 id 439 exid 19919 dests 00
#33 id 439 exid 73326 dests 00
#34 id 439 exid 77713 dests 00
#35 id 439 exid 73326 dests 00
#36 id 439 exid 73326 dests 00
#37 id 439 exid 77713 dests 00
#38 id 439 exid 73326 dests 00
#39 id 439 exid 19919 dests 00
#40 id 439 exid 73326 dests 00
#41 id 439 exid 19919 dests 00
#42 id 439 exid 73326 dests 00
#43 id 439 exid 73326 dests 00
#44 id 439 exid 19919 dests 00
#45 id 439 exid 73326 dests 00
#46 id 439 exid 77713 dests 00
#47 id 439 exid 73326 dests 00
#48 id 439 exid 77713 dests 00
#49 id 439 exid 19919 dests 00
#50 id 439 exid 19919 dests 00
#51 id 439 exid 73326 dests 00
#52 id 439 exid 19919 dests 00
#53 id 439 exid 73326 dests 00
#54 id 439 exid 19919 dests 00
#55 id 439 exid 19919 dests 00
#56 id 439 exid 77713 dests 00
#57 id 439 exid 73326 dests 00
#58 id 439 exid 73326 dests 00
#59 id 439 exid 73326 dests 00
#60 id 439 exid 77713 dests 00
#61 id 439 exid 77713 dests 00
#62 id 439 exid 73326 dests 00
#63 id 439 exid 19919 dests 00
#64 id 439 exid 73326 dests 00
#65 id 439 exid 19919 dests 00
#66 id 439 exid 73326 dests 00
#67 id 439 exid 19919 dests 00
#68 id 439 exid 73326 dests 00
#69 id 439 exid 77713 dests 00
#70 id 439 exid 77713 dests 00
#71 id 439 exid 73326 dests 00
#72 id 439 exid 73326 dests 00
#73 id 439 exid 19919 dests 00
#74 id 439 exid 73326 dests 00
#75 id 439 exid 19919 dests 00
#76 id 439 exid 73326 dests 00
#77 id 439 exid 19919 dests 00
#78 id 439 exid 19919 dests 00
#79 id 439 exid 73326 dests 00
#80 id 439 exid 77713 dests 00
#81 id 439 exid 73326 dests 00
#82 id 439 exid 77713 dests 00
#83 id 439 exid 19919 dests 00
#84 id 439 exid 73326 dests 00
#85 id 439 exid 73326 dests 00
#86 id 439 exid 19919 dests 00
#87 id 439 exid 73326 dests 00
#88 id 439 exid 77713 dests 00
#89 id 439 exid 73326 dests 00
#90 id 439 exid 77713 dests 00
#91 id 439 exid 73326 dests 00
#92 id 439 exid 19919 dests 00
#93 id 439 exid 19919 dests 00
#94 id 439 exid 73326 dests 00
#95 id 439 exid 73326 dests 00
#96 id 439 exid 19919 dests 00
#97 id 439 exid 73326 dests 00
#98 id 439 exid 77713 dests 00
#99 id 439 exid 73326 dests 00
#100 id 439 exid 77713 dests 00
#101 id 439 exid 73326 dests 00
#102 id 439 exid 19919 dests 00
#103 id 439 exid 19919 dests 00
#104 id 439 exid 73326 dests 00
#105 id 439 exid 19919 dests 00
#106 id 439 exid 73326 dests 00
#107 id 439 exid 77713 dests 00
#108 id 439 exid 77713 dests 00
#109 id 439 exid 73326 dests 00
#110 id 439 exid 19919 dests 00
#111 id 439 exid 19919 dests 00
#112 id 439 exid 73326 dests 00
#113 id 439 exid 73326 dests 00
#114 id 439 exid 19919 dests 00
#115 id 439 exid 19919 dests 00
#116 id 439 exid 73326 dests 00
#117 id 439 exid 77713 dests 00
#118 id 439 exid 73326 dests 00
#119 id 439 exid 77713 dests 00
#120 id 439 exid 73326 dests 00
#121 id 439 exid 19919 dests 00
#122 id 439 exid 73326 dests 00
#123 id 439 exid 19919 dests 00
#124 id 439 exid 73326 dests 00
#125 id 439 exid 19919 dests 00
#126 id 439 exid 19919 dests 00
#127 id 439 exid 73326 dests 00
#128 id 439 exid 77713 dests 00
#129 id 439 exid 73326 dests 00
#130 id 439 exid 77713 dests 00
#131 id 439 exid 19919 dests 00
#132 id 439 exid 73326 dests 00
#133 id 439 exid 19919 dests 00
#134 id 439 exid 73326 dests 00
#135 id 439 exid 19919 dests 00
#136 id 439 exid 73326 dests 00
#137 id 439 exid 77713 dests 00
#138 id 439 exid 73326 dests 00
#139 id 439 exid 77713 dests 00
#140 id 439 exid 73326 dests 00
#141 id 439 exid 19919 dests 00
#142 id 439 exid 73326 dests 00
#143 id 439 exid 19919 dests 00
#144 id 439 exid 73326 dests 00
#145 id 439 exid 19919 dests 00
#146 id 439 exid 73326 dests 00
#147 id 439 exid 77713 dests 00
#148 id 439 exid 73326 dests 00
#149 id 439 exid 77713 dests 00
#150 id 439 exid 19919 dests 00
#151 id 439 exid 73326 dests 00
#152 id 439 exid 73326 dests 00
#153 id 439 exid 19919 dests 00
#154 id 439 exid 73326 dests 00
#155 id 439 exid 77713 dests 00
#156 id 439 exid 73326 dests 00
#157 id 439 exid 77713 dests 00
#158 id 439 exid 73326 dests 00
#159 id 439 exid 19919 dests 00
#160 id 439 exid 73326 dests 00
#161 id 439 exid 19919 dests 00
#162 id 439 exid 19919 dests 00
#163 id 439 exid 73326 dests 00
#164 id 439 exid 77713 dests 00
#165 id 439 exid 73326 dests 00
#166 id 439 exid 77713 dests 00
#167 id 439 exid 73326 dests 00
#168 id 439 exid 19919 dests 00
#169 id 439 exid 19919 dests 00
#170 id 439 exid 73326 dests 00
#171 id 439 exid 19919 dests 00
#172 id 439 exid 73326 dests 00
#173 id 439 exid 77713 dests 00
#174 id 439 exid 73326 dests 00
#175 id 439 exid 77713 dests 00
#176 id 439 exid 19919 dests 00
#177 id 439 exid 73326 dests 00
#178 id 439 exid 19919 dests 00
#179 id 439 exid 19919 dests 00
#180 id 439 exid 73326 dests 00
#181 id 439 exid 19919 dests 00
#182 id 439 exid 73326 dests 00
#183 id 439 exid 77713 dests 00
#184 id 439 exid 73326 dests 00
#185 id 439 exid 77713 dests 00
#186 id 439 exid 73326 dests 00
#187 id 439 exid 19919 dests 00
#188 id 439 exid 73326 dests 00
#189 id 439 exid 19919 dests 00
#190 id 439 exid 73326 dests 00
#191 id 439 exid 19919 dests 00
#192 id 439 exid 73326 dests 00
#193 id 439 exid 77713 dests 00
#194 id 439 exid 77713 dests 00
#195 id 439 exid 73326 dests 00
#196 id 439 exid 19919 dests 00
#197 id 439 exid 19919 dests 00
#198 id 439 exid 73326 dests 00
#199 id 439 exid 19919 dests 00
#200 id 439 exid 73326 dests 00
#201 id 439 exid 77713 dests 00
#202 id 439 exid 73326 dests 00
#203 id 439 exid 77713 dests 00
#204 id 439 exid 19919 dests 00
#205 id 439 exid 73326 dests 00
#206 id 439 exid 19919 dests 00
#207 id 439 exid 19919 dests 00
#208 id 439 exid 73326 dests 00
#209 id 439 exid 19919 dests 00
#210 id 439 exid 19919 dests 00
#211 id 439 exid 73326 dests 00
#212 id 439 exid 77713 dests 00
#213 id 439 exid 73326 dests 00
#214 id 439 exid 77713 dests 00
#215 id 439 exid 73326 dests 00
#216 id 439 exid 19919 dests 00
#217 id 439 exid 73326 dests 00
#218 id 439 exid 19919 dests 00
#219 id 439 exid 73326 dests 00
#220 id 439 exid 19919 dests 00
#221 id 439 exid 19919 dests 00
#222 id 439 exid 73326 dests 00
#223 id 439 exid 77713 dests 00
#224 id 439 exid 77713 dests 00
#225 id 439 exid 77713 dests 00
#226 id 439 exid 73326 dests 00
#227 id 439 exid 19919 dests 00
#228 id 439 exid 73326 dests 00
#229 id 439 exid 19919 dests 00
#230 id 439 exid 73326 dests 00
#231 id 439 exid 73326 dests 00
#232 id 439 exid 77713 dests 00
#233 id 439 exid 19919 dests 00
#234 id 439 exid 73326 dests 00
#235 id 439 exid 73326 dests 00
#236 id 439 exid 19919 dests 00
#237 id 439 exid 73326 dests 00
#238 id 439 exid 77713 dests 00
#239 id 439 exid 77713 dests 00
#240 id 439 exid 73326 dests 00
#241 id 439 exid 19919 dests 00
#242 id 439 exid 73326 dests 00
#243 id 439 exid 19919 dests 00
#244 id 439 exid 19919 dests 00
#245 id 439 exid 73326 dests 00
#246 id 439 exid 77713 dests 00
#247 id 439 exid 73326 dests 00
#248 id 439 exid 77713 dests 00
#249 id 439 exid 19919 dests 00
#250 id 439 exid 73326 dests 00
#251 id 439 exid 73326 dests 00
#252 id 439 exid 19919 dests 00
#253 id 439 exid 19919 dests 00
#254 id 439 exid 73326 dests 00
#255 id 439 exid 73326 dests 00
kjctseventdump-end tail 231 heads 218 @ 0 231 @ -817936614
End of KJC Communication Dump
kjfmGCR_HBdisambig: action=Inst-kill

*** 2019-11-15T14:51:56.885323+04:00 (CDB$ROOT(1))
kjgcr_DoPreAction: KJGCR_PREACTION? - id 1

*** 2019-11-15T14:51:57.034864+04:00 (CDB$ROOT(1))
GCR_SYS group GR+DB_CDBIAMAS LOCKED by memno 1

*** 2019-11-15T14:51:57.035188+04:00 (CDB$ROOT(1))
kjgcr_querycssginf: grp "GR+N_1" has no private grp data.

*** 2019-11-15T14:51:57.035217+04:00 (CDB$ROOT(1))
kjgcr_pollcssg: grp "GR+DB_CDBIAMAS" mbr 1 LOCKED next state (3) grp1=0x0 grp2=0x7ffe97383098 act=1.

*** 2019-11-15T14:51:57.035234+04:00 (CDB$ROOT(1))
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_CDBIAMAS by memno 1

*** 2019-11-15T14:51:57.338024+04:00 (CDB$ROOT(1))
kjgcr_pollcssg: grp "GR+DB_+ASM" mbr 2 LOCKED next state (3) grp1=0x7ffe97382ff0 grp2=0x0 act=1.

*** 2019-11-15T14:51:57.339033+04:00 (CDB$ROOT(1))
kjgcr_DoAction: KJGCR_ACTION? - id 1

*** 2019-11-15T14:51:57.339066+04:00 (CDB$ROOT(1))
kjgcr_RunCDCallback: Running action callback [kjgcr Dump OS Diagnostics]

*** 2019-11-15T14:52:02.340436+04:00 (CDB$ROOT(1))
==============================
IPC0 (ospid: 368909) has not moved for 123 sec (1573815116.1573814993)

*** 2019-11-15T14:52:02.351421+04:00 (CDB$ROOT(1))
Forcibly terminated 'oracle@exa-hddbadm01.core.iamas.gov.az (LGWR)' process
Forcibly terminated 4 out of 4 LGWR worker processes

*** 2019-11-15T14:52:02.351508+04:00 (CDB$ROOT(1))
LGWR LWN SCN: 0x000000004f84c33e, Issued SCN: 0x000000004f84c33e, On-disk SCN: 0x000000004f84c33eKSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS0)'
KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS1)'
KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS2)'
KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS3)'
KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS4)'
KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS5)'
2019-11-15 14:52:02.351 :kjzduptcctx(): Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+346<-kjzduptcctx()+868<-kjzdicrshnfy()+1113<-ksuitm_opt()+1678<-kjgcr_KillInstance()+170<-kjgcr_RunCallback()+1242<-kjgcr_DoAction()+186<-kjgcr_servicecssg()+856<-kjgcr_Main()+418<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202
<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----

*** 2019-11-15T14:52:02.375872+04:00 (CDB$ROOT(1))
LMHB (ospid: 368993): terminating the instance due to error 29770
ksuitm: waiting up to [5] seconds before killing DIAG(368925)

*** 2019-11-15T14:52:16.532424+04:00 (CDB$ROOT(1))
Instance termination failed to kill one or more processes
ksuitm_check: OS PID=360990 is still alive
ksuitm_check: OS PID=360970 is still alive
ksuitm_check: OS PID=360880 is still alive
ksuitm_check: OS PID=360772 is still alive
ksuitm_check: OS PID=360757 is still alive
ksuitm_check: OS PID=360755 is still alive
ksuitm_check: OS PID=360753 is still alive
ksuitm_check: OS PID=360749 is still alive
ksuitm_check: OS PID=360746 is still alive
ksuitm_check: OS PID=360744 is still alive
ksuitm_check: OS PID=360734 is still alive
ksuitm_check: OS PID=360729 is still alive
ksuitm_check: OS PID=360724 is still alive
ksuitm_check: OS PID=360722 is still alive
ksuitm_check: OS PID=360714 is still alive
ksuitm_check: OS PID=360710 is still alive
ksuitm_check: OS PID=360706 is still alive
ksuitm_check: OS PID=360694 is still alive
ksuitm_check: OS PID=360691 is still alive
ksuitm_check: OS PID=360689 is still alive
ksuitm_check: OS PID=360687 is still alive
ksuitm_check: OS PID=360653 is still alive
ksuitm_check: OS PID=360651 is still alive
ksuitm_check: OS PID=360649 is still alive
ksuitm_check: OS PID=360645 is still alive
ksuitm_check: OS PID=360642 is still alive
ksuitm_check: OS PID=360640 is still alive
ksuitm_check: OS PID=360635 is still alive
ksuitm_check: OS PID=360631 is still alive
ksuitm_check: OS PID=360622 is still alive
ksuitm_check: OS PID=360620 is still alive
ksuitm_check: OS PID=360618 is still alive
ksuitm_check: OS PID=360179 is still alive
ksuitm_check: OS PID=360616 is still alive
ksuitm_check: OS PID=360612 is still alive
ksuitm_check: OS PID=360044 is still alive
ksuitm_check: OS PID=360614 is still alive
ksuitm_check: OS PID=360610 is still alive
ksuitm_check: OS PID=360607 is still alive
ksuitm_check: OS PID=360605 is still alive
ksuitm_check: OS PID=358415 is still alive
ksuitm_check: OS PID=360585 is still alive
ksuitm_check: OS PID=358375 is still alive
ksuitm_check: OS PID=360598 is still alive
ksuitm_check: OS PID=358298 is still alive
ksuitm_check: OS PID=358294 is still alive
ksuitm_check: OS PID=358291 is still alive
ksuitm_check: OS PID=358287 is still alive
ksuitm_check: OS PID=358284 is still alive
ksuitm_check: OS PID=358282 is still alive
ksuitm_check: OS PID=358280 is still alive
ksuitm_check: OS PID=358278 is still alive
ksuitm_check: OS PID=358276 is still alive
ksuitm_check: OS PID=358269 is still alive
ksuitm_check: OS PID=358228 is still alive
ksuitm_check: OS PID=358222 is still alive
ksuitm_check: OS PID=358220 is still alive
ksuitm_check: OS PID=358217 is still alive
ksuitm_check: OS PID=358213 is still alive
ksuitm_check: OS PID=358210 is still alive
ksuitm_check: OS PID=358202 is still alive
ksuitm_check: OS PID=358191 is still alive
ksuitm_check: OS PID=358194 is still alive
ksuitm_check: OS PID=358185 is still alive
ksuitm_check: OS PID=358181 is still alive
ksuitm_check: OS PID=358160 is still alive
ksuitm_check: OS PID=358156 is still alive
ksuitm_check: OS PID=358086 is still alive
ksuitm_check: OS PID=358146 is still alive
ksuitm_check: OS PID=358082 is still alive
ksuitm_check: OS PID=358075 is still alive
ksuitm_check: OS PID=358033 is still alive
ksuitm_check: OS PID=358022 is still alive
ksuitm_check: OS PID=358020 is still alive
ksuitm_check: OS PID=357961 is still alive
ksuitm_check: OS PID=357959 is still alive
ksuitm_check: OS PID=357955 is still alive
ksuitm_check: OS PID=357953 is still alive
ksuitm_check: OS PID=357948 is still alive
ksuitm_check: OS PID=357939 is still alive
ksuitm_check: OS PID=357937 is still alive
ksuitm_check: OS PID=357935 is still alive
ksuitm_check: OS PID=357931 is still alive
ksuitm_check: OS PID=357929 is still alive
ksuitm_check: OS PID=357927 is still alive
ksuitm_check: OS PID=357925 is still alive
ksuitm_check: OS PID=357923 is still alive
ksuitm_check: OS PID=357892 is still alive
ksuitm_check: OS PID=357880 is still alive
ksuitm_check: OS PID=357876 is still alive
ksuitm_check: OS PID=357874 is still alive
ksuitm_check: OS PID=357870 is still alive
ksuitm_check: OS PID=358061 is still alive
ksuitm_check: OS PID=357863 is still alive
ksuitm_check: OS PID=357861 is still alive
ksuitm_check: OS PID=357859 is still alive
ksuitm_check: OS PID=357855 is still alive
ksuitm_check: OS PID=357843 is still alive
ksuitm_check: OS PID=357841 is still alive
ksuitm_check: OS PID=357839 is still alive
ksuitm_check: OS PID=357837 is still alive
ksuitm_check: OS PID=357835 is still alive
ksuitm_check: OS PID=357824 is still alive
ksuitm_check: OS PID=357818 is still alive
ksuitm_check: OS PID=357814 is still alive
ksuitm_check: OS PID=357802 is still alive
ksuitm_check: OS PID=357799 is still alive
ksuitm_check: OS PID=357797 is still alive
ksuitm_check: OS PID=357795 is still alive
ksuitm_check: OS PID=357793 is still alive
ksuitm_check: OS PID=357748 is still alive
ksuitm_check: OS PID=357726 is still alive
ksuitm_check: OS PID=357722 is still alive
ksuitm_check: OS PID=357720 is still alive
ksuitm_check: OS PID=357715 is still alive
ksuitm_check: OS PID=357713 is still alive
ksuitm_check: OS PID=357710 is still alive
ksuitm_check: OS PID=357706 is still alive
ksuitm_check: OS PID=357702 is still alive
ksuitm_check: OS PID=357700 is still alive
ksuitm_check: OS PID=357698 is still alive
ksuitm_check: OS PID=357696 is still alive
ksuitm_check: OS PID=357694 is still alive
ksuitm_check: OS PID=357663 is still alive
ksuitm_check: OS PID=357655 is still alive
ksuitm_check: OS PID=357653 is still alive
ksuitm_check: OS PID=357651 is still alive
ksuitm_check: OS PID=357647 is still alive
ksuitm_check: OS PID=357632 is still alive
ksuitm_check: OS PID=357568 is still alive
ksuitm_check: OS PID=357551 is still alive
ksuitm_check: OS PID=357549 is still alive
ksuitm_check: OS PID=357547 is still alive
ksuitm_check: OS PID=357531 is still alive
ksuitm_check: OS PID=357468 is still alive
ksuitm_check: OS PID=357440 is still alive
ksuitm_check: OS PID=357355 is still alive
ksuitm_check: OS PID=357293 is still alive
ksuitm_check: OS PID=357257 is still alive
ksuitm_check: OS PID=357254 is still alive
ksuitm_check: OS PID=357250 is still alive
ksuitm_check: OS PID=357209 is still alive
ksuitm_check: OS PID=357207 is still alive
ksuitm_check: OS PID=357147 is still alive
ksuitm_check: OS PID=357140 is still alive
ksuitm_check: OS PID=357205 is still alive
ksuitm_check: OS PID=357162 is still alive
ksuitm_check: OS PID=357157 is still alive
ksuitm_check: OS PID=356318 is still alive
ksuitm_check: OS PID=356270 is still alive
ksuitm_check: OS PID=354493 is still alive
ksuitm_check: OS PID=354491 is still alive
ksuitm_check: OS PID=354483 is still alive
ksuitm_check: OS PID=354479 is still alive
ksuitm_check: OS PID=354477 is still alive
ksuitm_check: OS PID=354475 is still alive
ksuitm_check: OS PID=354473 is still alive
ksuitm_check: OS PID=354463 is still alive
ksuitm_check: OS PID=358024 is still alive
ksuitm_check: OS PID=354408 is still alive
ksuitm_check: OS PID=354396 is still alive
ksuitm_check: OS PID=354394 is still alive
ksuitm_check: OS PID=354382 is still alive
ksuitm_check: OS PID=354379 is still alive
ksuitm_check: OS PID=354368 is still alive
ksuitm_check: OS PID=354366 is still alive
ksuitm_check: OS PID=354364 is still alive
ksuitm_check: OS PID=354362 is still alive
ksuitm_check: OS PID=354360 is still alive
ksuitm_check: OS PID=354354 is still alive
ksuitm_check: OS PID=354334 is still alive
ksuitm_check: OS PID=354330 is still alive
ksuitm_check: OS PID=354318 is still alive
ksuitm_check: OS PID=354313 is still alive
ksuitm_check: OS PID=354309 is still alive
ksuitm_check: OS PID=354301 is still alive
ksuitm_check: OS PID=354298 is still alive
ksuitm_check: OS PID=354296 is still alive
ksuitm_check: OS PID=354294 is still alive
ksuitm_check: OS PID=354292 is still alive
ksuitm_check: OS PID=354290 is still alive
ksuitm_check: OS PID=354286 is still alive
ksuitm_check: OS PID=354284 is still alive
ksuitm_check: OS PID=354282 is still alive
ksuitm_check: OS PID=354280 is still alive
ksuitm_check: OS PID=354278 is still alive
ksuitm_check: OS PID=354273 is still alive
ksuitm_check: OS PID=354144 is still alive
ksuitm_check: OS PID=354140 is still alive
ksuitm_check: OS PID=354138 is still alive
ksuitm_check: OS PID=354136 is still alive
ksuitm_check: OS PID=354134 is still alive
ksuitm_check: OS PID=354129 is still alive
ksuitm_check: OS PID=354127 is still alive
ksuitm_check: OS PID=354125 is still alive
ksuitm_check: OS PID=354123 is still alive
ksuitm_check: OS PID=354121 is still alive
ksuitm_check: OS PID=354119 is still alive
ksuitm_check: OS PID=354117 is still alive
ksuitm_check: OS PID=354115 is still alive
ksuitm_check: OS PID=354113 is still alive
ksuitm_check: OS PID=354109 is still alive
ksuitm_check: OS PID=354072 is still alive
ksuitm_check: OS PID=354060 is still alive
ksuitm_check: OS PID=354057 is still alive
ksuitm_check: OS PID=354053 is still alive
ksuitm_check: OS PID=354046 is still alive
ksuitm_check: OS PID=354024 is still alive
ksuitm_check: OS PID=354014 is still alive
ksuitm_check: OS PID=354010 is still alive
ksuitm_check: OS PID=354006 is still alive
ksuitm_check: OS PID=354004 is still alive
ksuitm_check: OS PID=354002 is still alive
ksuitm_check: OS PID=354000 is still alive
ksuitm_check: OS PID=353998 is still alive
ksuitm_check: OS PID=353996 is still alive
ksuitm_check: OS PID=353992 is still alive
ksuitm_check: OS PID=353990 is still alive
ksuitm_check: OS PID=353988 is still alive
ksuitm_check: OS PID=353986 is still alive
ksuitm_check: OS PID=353984 is still alive
ksuitm_check: OS PID=353895 is still alive
ksuitm_check: OS PID=353891 is still alive
ksuitm_check: OS PID=353887 is still alive
ksuitm_check: OS PID=353883 is still alive
ksuitm_check: OS PID=353868 is still alive
ksuitm_check: OS PID=353864 is still alive
ksuitm_check: OS PID=353854 is still alive
ksuitm_check: OS PID=353840 is still alive
ksuitm_check: OS PID=353838 is still alive
ksuitm_check: OS PID=353820 is still alive
ksuitm_check: OS PID=353832 is still alive
ksuitm_check: OS PID=353797 is still alive
ksuitm_check: OS PID=353795 is still alive
ksuitm_check: OS PID=353793 is still alive
ksuitm_check: OS PID=353785 is still alive
ksuitm_check: OS PID=353783 is still alive
ksuitm_check: OS PID=353764 is still alive
ksuitm_check: OS PID=353982 is still alive
ksuitm_check: OS PID=353470 is still alive
ksuitm_check: OS PID=353449 is still alive
ksuitm_check: OS PID=353432 is still alive
ksuitm_check: OS PID=353399 is still alive
ksuitm_check: OS PID=353394 is still alive
ksuitm_check: OS PID=353387 is still alive
ksuitm_check: OS PID=353385 is still alive
ksuitm_check: OS PID=353383 is still alive
ksuitm_check: OS PID=353380 is still alive
ksuitm_check: OS PID=353368 is still alive
ksuitm_check: OS PID=353363 is still alive
ksuitm_check: OS PID=353360 is still alive
ksuitm_check: OS PID=353358 is still alive
ksuitm_check: OS PID=353356 is still alive
ksuitm_check: OS PID=353302 is still alive
ksuitm_check: OS PID=353300 is still alive
ksuitm_check: OS PID=353298 is still alive
ksuitm_check: OS PID=353296 is still alive
ksuitm_check: OS PID=353294 is still alive
ksuitm_check: OS PID=353284 is still alive
ksuitm_check: OS PID=353282 is still alive
ksuitm_check: OS PID=353278 is still alive
ksuitm_check: OS PID=353276 is still alive
ksuitm_check: OS PID=353265 is still alive
ksuitm_check: OS PID=353263 is still alive
ksuitm_check: OS PID=353261 is still alive
ksuitm_check: OS PID=353259 is still alive
ksuitm_check: OS PID=353257 is still alive
ksuitm_check: OS PID=353249 is still alive
ksuitm_check: OS PID=353236 is still alive
ksuitm_check: OS PID=353234 is still alive
ksuitm_check: OS PID=353229 is still alive
ksuitm_check: OS PID=353354 is still alive
ksuitm_check: OS PID=353352 is still alive
ksuitm_check: OS PID=353215 is still alive
ksuitm_check: OS PID=360174 is still alive
ksuitm_check: OS PID=353350 is still alive
ksuitm_check: OS PID=353187 is still alive
ksuitm_check: OS PID=353348 is still alive
ksuitm_check: OS PID=353346 is still alive
ksuitm_check: OS PID=353127 is still alive
ksuitm_check: OS PID=353343 is still alive
ksuitm_check: OS PID=353201 is still alive
ksuitm_check: OS PID=360568 is still alive
ksuitm_check: OS PID=353341 is still alive
ksuitm_check: OS PID=352330 is still alive
ksuitm_check: OS PID=352326 is still alive
ksuitm_check: OS PID=352324 is still alive
ksuitm_check: OS PID=352320 is still alive
ksuitm_check: OS PID=352318 is still alive
ksuitm_check: OS PID=352316 is still alive
ksuitm_check: OS PID=352232 is still alive
ksuitm_check: OS PID=352230 is still alive
ksuitm_check: OS PID=352226 is still alive
ksuitm_check: OS PID=352216 is still alive
ksuitm_check: OS PID=352213 is still alive
ksuitm_check: OS PID=352208 is still alive
ksuitm_check: OS PID=352194 is still alive
ksuitm_check: OS PID=352186 is still alive
ksuitm_check: OS PID=352161 is still alive
ksuitm_check: OS PID=352163 is still alive
ksuitm_check: OS PID=352154 is still alive
ksuitm_check: OS PID=352151 is still alive
ksuitm_check: OS PID=352147 is still alive
ksuitm_check: OS PID=352145 is still alive
ksuitm_check: OS PID=352132 is still alive
ksuitm_check: OS PID=352124 is still alive
ksuitm_check: OS PID=352122 is still alive
ksuitm_check: OS PID=352120 is still alive
ksuitm_check: OS PID=352116 is still alive
ksuitm_check: OS PID=352104 is still alive
ksuitm_check: OS PID=352102 is still alive
ksuitm_check: OS PID=352092 is still alive
ksuitm_check: OS PID=352086 is still alive
ksuitm_check: OS PID=352084 is still alive
ksuitm_check: OS PID=352082 is still alive
ksuitm_check: OS PID=352080 is still alive
ksuitm_check: OS PID=352078 is still alive
ksuitm_check: OS PID=352076 is still alive
ksuitm_check: OS PID=352073 is still alive
ksuitm_check: OS PID=352053 is still alive
ksuitm_check: OS PID=352049 is still alive
ksuitm_check: OS PID=352043 is still alive
ksuitm_check: OS PID=352040 is still alive
ksuitm_check: OS PID=352034 is still alive
ksuitm_check: OS PID=352020 is still alive
ksuitm_check: OS PID=352018 is still alive
ksuitm_check: OS PID=352014 is still alive
ksuitm_check: OS PID=352012 is still alive
ksuitm_check: OS PID=352006 is still alive
ksuitm_check: OS PID=352002 is still alive
ksuitm_check: OS PID=351974 is still alive
ksuitm_check: OS PID=351969 is still alive
ksuitm_check: OS PID=351967 is still alive
ksuitm_check: OS PID=351959 is still alive
ksuitm_check: OS PID=351956 is still alive
ksuitm_check: OS PID=351952 is still alive
ksuitm_check: OS PID=351950 is still alive
ksuitm_check: OS PID=351947 is still alive
ksuitm_check: OS PID=351945 is still alive
ksuitm_check: OS PID=351928 is still alive
ksuitm_check: OS PID=351787 is still alive
ksuitm_check: OS PID=351686 is still alive
ksuitm_check: OS PID=351658 is still alive
ksuitm_check: OS PID=351651 is still alive
ksuitm_check: OS PID=351647 is still alive
ksuitm_check: OS PID=351643 is still alive
ksuitm_check: OS PID=351599 is still alive
ksuitm_check: OS PID=351601 is still alive
ksuitm_check: OS PID=351595 is still alive
ksuitm_check: OS PID=351592 is still alive
ksuitm_check: OS PID=351588 is still alive
ksuitm_check: OS PID=351586 is still alive
ksuitm_check: OS PID=351584 is still alive
ksuitm_check: OS PID=351582 is still alive
ksuitm_check: OS PID=351580 is still alive
ksuitm_check: OS PID=351578 is still alive
ksuitm_check: OS PID=351576 is still alive
ksuitm_check: OS PID=351574 is still alive
ksuitm_check: OS PID=351572 is still alive
ksuitm_check: OS PID=351566 is still alive
ksuitm_check: OS PID=351553 is still alive
ksuitm_check: OS PID=351551 is still alive
ksuitm_check: OS PID=351543 is still alive
ksuitm_check: OS PID=351541 is still alive
ksuitm_check: OS PID=351538 is still alive
ksuitm_check: OS PID=351533 is still alive
ksuitm_check: OS PID=351511 is still alive
ksuitm_check: OS PID=351505 is still alive
ksuitm_check: OS PID=351496 is still alive
ksuitm_check: OS PID=351492 is still alive
ksuitm_check: OS PID=351488 is still alive
ksuitm_check: OS PID=351482 is still alive
ksuitm_check: OS PID=351480 is still alive
ksuitm_check: OS PID=351478 is still alive
ksuitm_check: OS PID=351476 is still alive
ksuitm_check: OS PID=351474 is still alive
ksuitm_check: OS PID=351460 is still alive
ksuitm_check: OS PID=351446 is still alive
ksuitm_check: OS PID=351444 is still alive
ksuitm_check: OS PID=351442 is still alive
ksuitm_check: OS PID=351440 is still alive
ksuitm_check: OS PID=351436 is still alive
ksuitm_check: OS PID=351434 is still alive
ksuitm_check: OS PID=351361 is still alive
ksuitm_check: OS PID=351357 is still alive
ksuitm_check: OS PID=351359 is still alive
ksuitm_check: OS PID=351215 is still alive
ksuitm_check: OS PID=351211 is still alive
ksuitm_check: OS PID=351209 is still alive
ksuitm_check: OS PID=351207 is still alive
ksuitm_check: OS PID=351204 is still alive
ksuitm_check: OS PID=351196 is still alive
ksuitm_check: OS PID=351191 is still alive
ksuitm_check: OS PID=351185 is still alive
ksuitm_check: OS PID=351179 is still alive
ksuitm_check: OS PID=351173 is still alive
ksuitm_check: OS PID=351134 is still alive
ksuitm_check: OS PID=351091 is still alive
ksuitm_check: OS PID=351076 is still alive
ksuitm_check: OS PID=351078 is still alive
ksuitm_check: OS PID=351062 is still alive
ksuitm_check: OS PID=351045 is still alive
ksuitm_check: OS PID=350947 is still alive
ksuitm_check: OS PID=350945 is still alive
ksuitm_check: OS PID=350943 is still alive
ksuitm_check: OS PID=350941 is still alive
ksuitm_check: OS PID=350928 is still alive
ksuitm_check: OS PID=351217 is still alive
ksuitm_check: OS PID=350901 is still alive
ksuitm_check: OS PID=350728 is still alive
ksuitm_check: OS PID=350726 is still alive
ksuitm_check: OS PID=350689 is still alive
ksuitm_check: OS PID=350682 is still alive
ksuitm_check: OS PID=350687 is still alive
ksuitm_check: OS PID=350676 is still alive
ksuitm_check: OS PID=350662 is still alive
ksuitm_check: OS PID=350658 is still alive
ksuitm_check: OS PID=350656 is still alive
ksuitm_check: OS PID=350652 is still alive
ksuitm_check: OS PID=350641 is still alive
ksuitm_check: OS PID=350637 is still alive
ksuitm_check: OS PID=350629 is still alive
ksuitm_check: OS PID=350626 is still alive
ksuitm_check: OS PID=350624 is still alive
ksuitm_check: OS PID=350620 is still alive
ksuitm_check: OS PID=350618 is still alive
ksuitm_check: OS PID=350616 is still alive
ksuitm_check: OS PID=350614 is still alive
ksuitm_check: OS PID=350600 is still alive
ksuitm_check: OS PID=350598 is still alive
ksuitm_check: OS PID=350596 is still alive
ksuitm_check: OS PID=350594 is still alive
ksuitm_check: OS PID=350592 is still alive
ksuitm_check: OS PID=350590 is still alive
ksuitm_check: OS PID=350588 is still alive
ksuitm_check: OS PID=350884 is still alive
ksuitm_check: OS PID=350886 is still alive
ksuitm_check: OS PID=350878 is still alive
ksuitm_check: OS PID=350537 is still alive
ksuitm_check: OS PID=350535 is still alive
ksuitm_check: OS PID=350868 is still alive
ksuitm_check: OS PID=350528 is still alive
ksuitm_check: OS PID=350526 is still alive
ksuitm_check: OS PID=350522 is still alive
ksuitm_check: OS PID=350520 is still alive
ksuitm_check: OS PID=350517 is still alive
ksuitm_check: OS PID=350515 is still alive
ksuitm_check: OS PID=350513 is still alive
ksuitm_check: OS PID=350500 is still alive
ksuitm_check: OS PID=350496 is still alive
ksuitm_check: OS PID=350484 is still alive
ksuitm_check: OS PID=350481 is still alive
ksuitm_check: OS PID=350477 is still alive
ksuitm_check: OS PID=350475 is still alive
ksuitm_check: OS PID=350470 is still alive
ksuitm_check: OS PID=350468 is still alive
ksuitm_check: OS PID=350466 is still alive
ksuitm_check: OS PID=350464 is still alive
ksuitm_check: OS PID=350462 is still alive
ksuitm_check: OS PID=350459 is still alive
ksuitm_check: OS PID=350455 is still alive
ksuitm_check: OS PID=350453 is still alive
ksuitm_check: OS PID=350446 is still alive
ksuitm_check: OS PID=350422 is still alive
ksuitm_check: OS PID=350420 is still alive
ksuitm_check: OS PID=350415 is still alive
ksuitm_check: OS PID=350402 is still alive
ksuitm_check: OS PID=350391 is still alive
ksuitm_check: OS PID=350384 is still alive
ksuitm_check: OS PID=350373 is still alive
ksuitm_check: OS PID=350369 is still alive
ksuitm_check: OS PID=350365 is still alive
ksuitm_check: OS PID=350363 is still alive
ksuitm_check: OS PID=350361 is still alive
ksuitm_check: OS PID=350358 is still alive
ksuitm_check: OS PID=350350 is still alive
ksuitm_check: OS PID=350329 is still alive
ksuitm_check: OS PID=350326 is still alive
ksuitm_check: OS PID=350324 is still alive
ksuitm_check: OS PID=350316 is still alive
ksuitm_check: OS PID=350314 is still alive
ksuitm_check: OS PID=350310 is still alive
ksuitm_check: OS PID=350308 is still alive
ksuitm_check: OS PID=350299 is still alive
ksuitm_check: OS PID=350297 is still alive
ksuitm_check: OS PID=350293 is still alive
ksuitm_check: OS PID=350253 is still alive
ksuitm_check: OS PID=350200 is still alive
ksuitm_check: OS PID=350108 is still alive
ksuitm_check: OS PID=350045 is still alive
ksuitm_check: OS PID=349953 is still alive
ksuitm_check: OS PID=349907 is still alive
ksuitm_check: OS PID=349903 is still alive
ksuitm_check: OS PID=349901 is still alive
ksuitm_check: OS PID=349898 is still alive
ksuitm_check: OS PID=349895 is still alive
ksuitm_check: OS PID=349892 is still alive
ksuitm_check: OS PID=349889 is still alive
ksuitm_check: OS PID=349887 is still alive
ksuitm_check: OS PID=349885 is still alive
ksuitm_check: OS PID=349875 is still alive
ksuitm_check: OS PID=349863 is still alive
ksuitm_check: OS PID=349861 is still alive
ksuitm_check: OS PID=347105 is still alive
ksuitm_check: OS PID=346886 is still alive
ksuitm_check: OS PID=346884 is still alive
ksuitm_check: OS PID=346881 is still alive
ksuitm_check: OS PID=346879 is still alive
ksuitm_check: OS PID=346877 is still alive
ksuitm_check: OS PID=346873 is still alive
ksuitm_check: OS PID=346870 is still alive
ksuitm_check: OS PID=346866 is still alive
ksuitm_check: OS PID=346858 is still alive
ksuitm_check: OS PID=346848 is still alive
ksuitm_check: OS PID=346846 is still alive
ksuitm_check: OS PID=346844 is still alive
ksuitm_check: OS PID=346842 is still alive
ksuitm_check: OS PID=346840 is still alive
ksuitm_check: OS PID=346825 is still alive
ksuitm_check: OS PID=346821 is still alive
ksuitm_check: OS PID=346814 is still alive
ksuitm_check: OS PID=346783 is still alive
ksuitm_check: OS PID=346765 is still alive
ksuitm_check: OS PID=346541 is still alive
ksuitm_check: OS PID=346539 is still alive
ksuitm_check: OS PID=346537 is still alive
ksuitm_check: OS PID=346527 is still alive
ksuitm_check: OS PID=346522 is still alive
ksuitm_check: OS PID=346520 is still alive
ksuitm_check: OS PID=346517 is still alive
ksuitm_check: OS PID=346512 is still alive
ksuitm_check: OS PID=346510 is still alive
ksuitm_check: OS PID=346498 is still alive
ksuitm_check: OS PID=346494 is still alive
ksuitm_check: OS PID=346492 is still alive
ksuitm_check: OS PID=346150 is still alive
ksuitm_check: OS PID=346148 is still alive
ksuitm_check: OS PID=346124 is still alive
ksuitm_check: OS PID=346490 is still alive
ksuitm_check: OS PID=346488 is still alive
ksuitm_check: OS PID=346485 is still alive
ksuitm_check: OS PID=346482 is still alive
ksuitm_check: OS PID=346479 is still alive
ksuitm_check: OS PID=346474 is still alive
ksuitm_check: OS PID=346085 is still alive
ksuitm_check: OS PID=346470 is still alive
ksuitm_check: OS PID=346081 is still alive
ksuitm_check: OS PID=346079 is still alive
ksuitm_check: OS PID=346468 is still alive
ksuitm_check: OS PID=346073 is still alive
ksuitm_check: OS PID=346464 is still alive
ksuitm_check: OS PID=346068 is still alive
ksuitm_check: OS PID=346066 is still alive
ksuitm_check: OS PID=346064 is still alive
ksuitm_check: OS PID=346058 is still alive
ksuitm_check: OS PID=346060 is still alive
ksuitm_check: OS PID=346054 is still alive
ksuitm_check: OS PID=346051 is still alive
ksuitm_check: OS PID=346049 is still alive
ksuitm_check: OS PID=346037 is still alive
ksuitm_check: OS PID=346035 is still alive
ksuitm_check: OS PID=346024 is still alive
ksuitm_check: OS PID=346022 is still alive
ksuitm_check: OS PID=346019 is still alive
ksuitm_check: OS PID=346017 is still alive
ksuitm_check: OS PID=346013 is still alive
ksuitm_check: OS PID=346007 is still alive
ksuitm_check: OS PID=346005 is still alive
ksuitm_check: OS PID=346001 is still alive
ksuitm_check: OS PID=345987 is still alive
ksuitm_check: OS PID=345983 is still alive
ksuitm_check: OS PID=345978 is still alive
ksuitm_check: OS PID=345973 is still alive
ksuitm_check: OS PID=345943 is still alive
ksuitm_check: OS PID=345941 is still alive
ksuitm_check: OS PID=345937 is still alive
ksuitm_check: OS PID=345933 is still alive
ksuitm_check: OS PID=345922 is still alive
ksuitm_check: OS PID=345919 is still alive
ksuitm_check: OS PID=345909 is still alive
ksuitm_check: OS PID=345905 is still alive
ksuitm_check: OS PID=345899 is still alive
ksuitm_check: OS PID=345887 is still alive
ksuitm_check: OS PID=345882 is still alive
ksuitm_check: OS PID=345872 is still alive
ksuitm_check: OS PID=345869 is still alive
ksuitm_check: OS PID=345867 is still alive
ksuitm_check: OS PID=345865 is still alive
ksuitm_check: OS PID=345863 is still alive
ksuitm_check: OS PID=345861 is still alive
ksuitm_check: OS PID=345859 is still alive
ksuitm_check: OS PID=345857 is still alive
ksuitm_check: OS PID=345855 is still alive
ksuitm_check: OS PID=345851 is still alive
ksuitm_check: OS PID=345847 is still alive
ksuitm_check: OS PID=345830 is still alive
ksuitm_check: OS PID=345833 is still alive
ksuitm_check: OS PID=345826 is still alive
ksuitm_check: OS PID=345822 is still alive
ksuitm_check: OS PID=345820 is still alive
ksuitm_check: OS PID=333255 is still alive
ksuitm_check: OS PID=344723 is still alive
ksuitm_check: OS PID=352192 is still alive
ksuitm_check: OS PID=332956 is still alive
ksuitm_check: OS PID=332954 is still alive
ksuitm_check: OS PID=332951 is still alive
ksuitm_check: OS PID=332949 is still alive
ksuitm_check: OS PID=351432 is still alive
ksuitm_check: OS PID=344694 is still alive
ksuitm_check: OS PID=332941 is still alive
ksuitm_check: OS PID=349859 is still alive
ksuitm_check: OS PID=332894 is still alive
ksuitm_check: OS PID=315854 is still alive
ksuitm_check: OS PID=332891 is still alive
ksuitm_check: OS PID=332889 is still alive
ksuitm_check: OS PID=315850 is still alive
ksuitm_check: OS PID=332887 is still alive
ksuitm_check: OS PID=315848 is still alive
ksuitm_check: OS PID=332883 is still alive
ksuitm_check: OS PID=315846 is still alive
ksuitm_check: OS PID=275447 is still alive
ksuitm_check: OS PID=345764 is still alive
ksuitm_check: OS PID=320938 is still alive
ksuitm_check: OS PID=319582 is still alive
ksuitm_check: OS PID=344685 is still alive
ksuitm_check: OS PID=332232 is still alive
ksuitm_check: OS PID=144360 is still alive
ksuitm_check: OS PID=345725 is still alive
ksuitm_check: OS PID=338194 is still alive
ksuitm_check: OS PID=352190 is still alive
ksuitm_check: OS PID=350473 is still alive
ksuitm_check: OS PID=332223 is still alive
ksuitm_check: OS PID=345633 is still alive
ksuitm_check: OS PID=339607 is still alive
ksuitm_check: OS PID=345573 is still alive
ksuitm_check: OS PID=315013 is still alive
ksuitm_check: OS PID=339526 is still alive
ksuitm_check: OS PID=339524 is still alive
ksuitm_check: OS PID=332584 is still alive
ksuitm_check: OS PID=332582 is still alive
ksuitm_check: OS PID=345534 is still alive
ksuitm_check: OS PID=302786 is still alive
ksuitm_check: OS PID=332579 is still alive
ksuitm_check: OS PID=302784 is still alive
ksuitm_check: OS PID=349857 is still alive
ksuitm_check: OS PID=224600 is still alive
ksuitm_check: OS PID=251863 is still alive
ksuitm_check: OS PID=332577 is still alive
ksuitm_check: OS PID=332575 is still alive
ksuitm_check: OS PID=332939 is still alive
ksuitm_check: OS PID=332937 is still alive
ksuitm_check: OS PID=332926 is still alive
ksuitm_check: OS PID=332885 is still alive
ksuitm_check: OS PID=332876 is still alive
ksuitm_check: OS PID=332573 is still alive
ksuitm_check: OS PID=344750 is still alive
ksuitm_check: OS PID=332868 is still alive
ksuitm_check: OS PID=251573 is still alive
ksuitm_check: OS PID=301846 is still alive
ksuitm_check: OS PID=314954 is still alive
ksuitm_check: OS PID=345527 is still alive
ksuitm_check: OS PID=333738 is still alive
ksuitm_check: OS PID=314908 is still alive
ksuitm_check: OS PID=346502 is still alive
ksuitm_check: OS PID=333864 is still alive
ksuitm_check: OS PID=275371 is still alive
ksuitm_check: OS PID=332879 is still alive
ksuitm_check: OS PID=339629 is still alive
ksuitm_check: OS PID=344589 is still alive
ksuitm_check: OS PID=344587 is still alive
ksuitm_check: OS PID=354356 is still alive
ksuitm_check: OS PID=338834 is still alive
ksuitm_check: OS PID=349855 is still alive
ksuitm_check: OS PID=349853 is still alive
ksuitm_check: OS PID=349851 is still alive
ksuitm_check: OS PID=332567 is still alive
ksuitm_check: OS PID=71738 is still alive
ksuitm_check: OS PID=333291 is still alive
ksuitm_check: OS PID=337373 is still alive
ksuitm_check: OS PID=307154 is still alive
ksuitm_check: OS PID=345418 is still alive
ksuitm_check: OS PID=113607 is still alive
ksuitm_check: OS PID=238213 is still alive
ksuitm_check: OS PID=332922 is still alive
ksuitm_check: OS PID=332918 is still alive
ksuitm_check: OS PID=332916 is still alive
ksuitm_check: OS PID=332914 is still alive
ksuitm_check: OS PID=122743 is still alive
ksuitm_check: OS PID=346773 is still alive
ksuitm_check: OS PID=334408 is still alive
ksuitm_check: OS PID=233245 is still alive
ksuitm_check: OS PID=332852 is still alive
ksuitm_check: OS PID=71162 is still alive
ksuitm_check: OS PID=333057 is still alive
ksuitm_check: OS PID=269538 is still alive
ksuitm_check: OS PID=332563 is still alive
ksuitm_check: OS PID=349849 is still alive
ksuitm_check: OS PID=267137 is still alive
ksuitm_check: OS PID=276391 is still alive
ksuitm_check: OS PID=332466 is still alive
ksuitm_check: OS PID=333068 is still alive
ksuitm_check: OS PID=353391 is still alive
ksuitm_check: OS PID=349847 is still alive
ksuitm_check: OS PID=353389 is still alive
ksuitm_check: OS PID=345394 is still alive
ksuitm_check: OS PID=353191 is still alive
ksuitm_check: OS PID=345361 is still alive
ksuitm_check: OS PID=338742 is still alive
ksuitm_check: OS PID=123530 is still alive
ksuitm_check: OS PID=350932 is still alive
ksuitm_check: OS PID=296623 is still alive
ksuitm_check: OS PID=338950 is still alive
ksuitm_check: OS PID=338740 is still alive
ksuitm_check: OS PID=338682 is still alive
ksuitm_check: OS PID=338678 is still alive
ksuitm_check: OS PID=338641 is still alive
ksuitm_check: OS PID=359695 is still alive
ksuitm_check: OS PID=338634 is still alive
ksuitm_check: OS PID=388592 is still alive
ksuitm_check: OS PID=338592 is still alive
ksuitm_check: OS PID=388582 is still alive
ksuitm_check: OS PID=151346 is still alive
ksuitm_check: OS PID=137861 is still alive
ksuitm_check: OS PID=307033 is still alive
ksuitm_check: OS PID=388569 is still alive
ksuitm_check: OS PID=239496 is still alive
ksuitm_check: OS PID=334239 is still alive
ksuitm_check: OS PID=354402 is still alive
ksuitm_check: OS PID=334237 is still alive
ksuitm_check: OS PID=354434 is still alive
ksuitm_check: OS PID=388417 is still alive
ksuitm_check: OS PID=339465 is still alive
ksuitm_check: OS PID=288210 is still alive
ksuitm_check: OS PID=353461 is still alive
ksuitm_check: OS PID=354400 is still alive
ksuitm_check: OS PID=344979 is still alive
ksuitm_check: OS PID=320975 is still alive
ksuitm_check: OS PID=354251 is still alive
ksuitm_check: OS PID=42629 is still alive
ksuitm_check: OS PID=332908 is still alive
ksuitm_check: OS PID=354245 is still alive
ksuitm_check: OS PID=353366 is still alive
ksuitm_check: OS PID=287807 is still alive
ksuitm_check: OS PID=360565 is still alive
ksuitm_check: OS PID=354234 is still alive
ksuitm_check: OS PID=354232 is still alive
ksuitm_check: OS PID=269826 is still alive
ksuitm_check: OS PID=354230 is still alive
ksuitm_check: OS PID=319943 is still alive
ksuitm_check: OS PID=357639 is still alive
ksuitm_check: OS PID=354220 is still alive
ksuitm_check: OS PID=354218 is still alive
ksuitm_check: OS PID=320791 is still alive
ksuitm_check: OS PID=354213 is still alive
ksuitm_check: OS PID=42621 is still alive
ksuitm_check: OS PID=354211 is still alive
ksuitm_check: OS PID=354209 is still alive
ksuitm_check: OS PID=354206 is still alive
ksuitm_check: OS PID=358080 is still alive
ksuitm_check: OS PID=354204 is still alive
ksuitm_check: OS PID=354962 is still alive
ksuitm_check: OS PID=354202 is still alive
ksuitm_check: OS PID=332958 is still alive
ksuitm_check: OS PID=347015 is still alive
ksuitm_check: OS PID=354200 is still alive
ksuitm_check: OS PID=354196 is still alive
ksuitm_check: OS PID=349845 is still alive
ksuitm_check: OS PID=350874 is still alive
ksuitm_check: OS PID=346995 is still alive
ksuitm_check: OS PID=341341 is still alive
ksuitm_check: OS PID=312624 is still alive
ksuitm_check: OS PID=8736 is still alive
ksuitm_check: OS PID=282610 is still alive
ksuitm_check: OS PID=72140 is still alive
ksuitm_check: OS PID=346983 is still alive
ksuitm_check: OS PID=361123 is still alive
ksuitm_check: OS PID=177937 is still alive
ksuitm_check: OS PID=320813 is still alive
ksuitm_check: OS PID=387851 is still alive
ksuitm_check: OS PID=339848 is still alive
ksuitm_check: OS PID=339158 is still alive
ksuitm_check: OS PID=346977 is still alive
ksuitm_check: OS PID=346971 is still alive
ksuitm_check: OS PID=345411 is still alive
ksuitm_check: OS PID=286909 is still alive
ksuitm_check: OS PID=346969 is still alive
ksuitm_check: OS PID=353339 is still alive
ksuitm_check: OS PID=346955 is still alive
ksuitm_check: OS PID=346953 is still alive
ksuitm_check: OS PID=314906 is still alive
ksuitm_check: OS PID=346946 is still alive
ksuitm_check: OS PID=350926 is still alive
ksuitm_check: OS PID=346942 is still alive
ksuitm_check: OS PID=346930 is still alive
ksuitm_check: OS PID=346922 is still alive
ksuitm_check: OS PID=298085 is still alive
ksuitm_check: OS PID=346912 is still alive
ksuitm_check: OS PID=350864 is still alive
ksuitm_check: OS PID=346910 is still alive
ksuitm_check: OS PID=294772 is still alive
ksuitm_check: OS PID=346906 is still alive
ksuitm_check: OS PID=339811 is still alive
ksuitm_check: OS PID=361103 is still alive
ksuitm_check: OS PID=42544 is still alive
ksuitm_check: OS PID=344439 is still alive
ksuitm_check: OS PID=353994 is still alive
ksuitm_check: OS PID=345311 is still alive
ksuitm_check: OS PID=346908 is still alive
ksuitm_check: OS PID=338566 is still alive
ksuitm_check: OS PID=162947 is still alive
ksuitm_check: OS PID=327984 is still alive
ksuitm_check: OS PID=338326 is still alive
ksuitm_check: OS PID=185758 is still alive
ksuitm_check: OS PID=370823 is still alive
ksuitm_check: OS PID=370999 is still alive
ksuitm_check: OS PID=346904 is still alive
ksuitm_check: OS PID=346966 is still alive
ksuitm_check: OS PID=185122 is still alive
ksuitm_check: OS PID=155830 is still alive
ksuitm_check: OS PID=374282 is still alive
ksuitm_check: OS PID=339805 is still alive
ksuitm_check: OS PID=161123 is still alive
ksuitm_check: OS PID=364922 is still alive
ksuitm_check: OS PID=345849 is still alive
ksuitm_check: OS PID=287179 is still alive
ksuitm_check: OS PID=334172 is still alive
ksuitm_check: OS PID=393386 is still alive
ksuitm_check: OS PID=353280 is still alive
ksuitm_check: OS PID=352761 is still alive
ksuitm_check: OS PID=322530 is still alive
ksuitm_check: OS PID=58549 is still alive
ksuitm_check: OS PID=375178 is still alive
ksuitm_check: OS PID=375174 is still alive
ksuitm_check: OS PID=375170 is still alive
ksuitm_check: OS PID=375163 is still alive
ksuitm_check: OS PID=375143 is still alive
ksuitm_check: OS PID=345308 is still alive
ksuitm_check: OS PID=319293 is still alive
ksuitm_check: OS PID=264429 is still alive
ksuitm_check: OS PID=393586 is still alive
ksuitm_check: OS PID=375176 is still alive
ksuitm_check: OS PID=375172 is still alive
ksuitm_check: OS PID=375166 is still alive
ksuitm_check: OS PID=375087 is still alive
ksuitm_check: OS PID=375161 is still alive
ksuitm_check: OS PID=375150 is still alive
ksuitm_check: OS PID=338579 is still alive
ksuitm_check: OS PID=375052 is still alive
ksuitm_check: OS PID=375031 is still alive
ksuitm_check: OS PID=374966 is still alive
ksuitm_check: OS PID=374962 is still alive
ksuitm_check: OS PID=374958 is still alive
ksuitm_check: OS PID=374954 is still alive
ksuitm_check: OS PID=350312 is still alive
ksuitm_check: OS PID=374950 is still alive
ksuitm_check: OS PID=374946 is still alive
ksuitm_check: OS PID=375057 is still alive
ksuitm_check: OS PID=374938 is still alive
ksuitm_check: OS PID=375037 is still alive
ksuitm_check: OS PID=374924 is still alive
ksuitm_check: OS PID=374908 is still alive
ksuitm_check: OS PID=374902 is still alive
ksuitm_check: OS PID=374898 is still alive
ksuitm_check: OS PID=374956 is still alive
ksuitm_check: OS PID=374952 is still alive
ksuitm_check: OS PID=374948 is still alive
ksuitm_check: OS PID=157396 is still alive
ksuitm_check: OS PID=374884 is still alive
ksuitm_check: OS PID=374873 is still alive
ksuitm_check: OS PID=374932 is still alive
ksuitm_check: OS PID=374861 is still alive
ksuitm_check: OS PID=374920 is still alive
ksuitm_check: OS PID=374841 is still alive
ksuitm_check: OS PID=374930 is still alive
ksuitm_check: OS PID=374773 is still alive
ksuitm_check: OS PID=374904 is still alive
ksuitm_check: OS PID=374737 is still alive
ksuitm_check: OS PID=374877 is still alive
ksuitm_check: OS PID=374900 is still alive
ksuitm_check: OS PID=350827 is still alive
ksuitm_check: OS PID=374896 is still alive
ksuitm_check: OS PID=374720 is still alive
ksuitm_check: OS PID=374869 is still alive
ksuitm_check: OS PID=374702 is still alive
ksuitm_check: OS PID=374935 is still alive
ksuitm_check: OS PID=374670 is still alive
ksuitm_check: OS PID=374847 is still alive
ksuitm_check: OS PID=374660 is still alive
ksuitm_check: OS PID=374837 is still alive
ksuitm_check: OS PID=374769 is still alive
ksuitm_check: OS PID=374607 is still alive
ksuitm_check: OS PID=374597 is still alive
ksuitm_check: OS PID=374728 is still alive
ksuitm_check: OS PID=354288 is still alive
ksuitm_check: OS PID=374940 is still alive
ksuitm_check: OS PID=374680 is still alive
ksuitm_check: OS PID=374581 is still alive
ksuitm_check: OS PID=374666 is still alive
ksuitm_check: OS PID=374706 is still alive
ksuitm_check: OS PID=360595 is still alive
ksuitm_check: OS PID=374626 is still alive
ksuitm_check: OS PID=374145 is still alive
ksuitm_check: OS PID=374543 is still alive
ksuitm_check: OS PID=374603 is still alive
ksuitm_check: OS PID=374127 is still alive
ksuitm_check: OS PID=373950 is still alive
ksuitm_check: OS PID=374587 is still alive
ksuitm_check: OS PID=345285 is still alive
ksuitm_check: OS PID=374550 is still alive
ksuitm_check: OS PID=373569 is still alive
ksuitm_check: OS PID=374152 is still alive
ksuitm_check: OS PID=319291 is still alive
ksuitm_check: OS PID=374138 is still alive
ksuitm_check: OS PID=373389 is still alive
ksuitm_check: OS PID=374109 is still alive
ksuitm_check: OS PID=314658 is still alive
ksuitm_check: OS PID=166107 is still alive
ksuitm_check: OS PID=150730 is still alive
ksuitm_check: OS PID=328905 is still alive
ksuitm_check: OS PID=373610 is still alive
ksuitm_check: OS PID=334169 is still alive
ksuitm_check: OS PID=373362 is still alive
ksuitm_check: OS PID=325956 is still alive
ksuitm_check: OS PID=238639 is still alive
ksuitm_check: OS PID=373284 is still alive
ksuitm_check: OS PID=373328 is still alive
ksuitm_check: OS PID=373279 is still alive
ksuitm_check: OS PID=373304 is still alive
ksuitm_check: OS PID=373237 is still alive
ksuitm_check: OS PID=320150 is still alive
ksuitm_check: OS PID=373229 is still alive
ksuitm_check: OS PID=373235 is still alive
ksuitm_check: OS PID=373212 is still alive
ksuitm_check: OS PID=373221 is still alive
ksuitm_check: OS PID=151144 is still alive
ksuitm_check: OS PID=373205 is still alive
ksuitm_check: OS PID=373203 is still alive
ksuitm_check: OS PID=373197 is still alive
ksuitm_check: OS PID=310281 is still alive
ksuitm_check: OS PID=349843 is still alive
ksuitm_check: OS PID=373191 is still alive
ksuitm_check: OS PID=373181 is still alive
ksuitm_check: OS PID=373187 is still alive
ksuitm_check: OS PID=373179 is still alive
ksuitm_check: OS PID=373173 is still alive
ksuitm_check: OS PID=373165 is still alive
ksuitm_check: OS PID=373158 is still alive
ksuitm_check: OS PID=373094 is still alive
ksuitm_check: OS PID=373075 is still alive
ksuitm_check: OS PID=373073 is still alive
ksuitm_check: OS PID=372988 is still alive
ksuitm_check: OS PID=373404 is still alive
ksuitm_check: OS PID=360588 is still alive
ksuitm_check: OS PID=372783 is still alive
ksuitm_check: OS PID=372771 is still alive
ksuitm_check: OS PID=372765 is still alive
ksuitm_check: OS PID=372753 is still alive
ksuitm_check: OS PID=364532 is still alive
ksuitm_check: OS PID=372728 is still alive
ksuitm_check: OS PID=372760 is still alive
ksuitm_check: OS PID=364081 is still alive
ksuitm_check: OS PID=372657 is still alive
ksuitm_check: OS PID=334001 is still alive
ksuitm_check: OS PID=372747 is still alive
ksuitm_check: OS PID=373326 is still alive
ksuitm_check: OS PID=372745 is still alive
ksuitm_check: OS PID=372651 is still alive
ksuitm_check: OS PID=39417 is still alive
ksuitm_check: OS PID=372649 is still alive
ksuitm_check: OS PID=372706 is still alive
ksuitm_check: OS PID=372647 is still alive
ksuitm_check: OS PID=162767 is still alive
ksuitm_check: OS PID=372635 is still alive
ksuitm_check: OS PID=372631 is still alive
ksuitm_check: OS PID=372625 is still alive
ksuitm_check: OS PID=157095 is still alive
ksuitm_check: OS PID=372509 is still alive
ksuitm_check: OS PID=374890 is still alive
ksuitm_check: OS PID=371889 is still alive
ksuitm_check: OS PID=353972 is still alive
ksuitm_check: OS PID=302701 is still alive
ksuitm_check: OS PID=369129 is still alive
ksuitm_check: OS PID=369100 is still alive
ksuitm_check: OS PID=369119 is still alive
ksuitm_check: OS PID=369096 is still alive
ksuitm_check: OS PID=369114 is still alive
ksuitm_check: OS PID=369088 is still alive
ksuitm_check: OS PID=369092 is still alive
ksuitm_check: OS PID=369058 is still alive
ksuitm_check: OS PID=369060 is still alive
ksuitm_check: OS PID=328865 is still alive
ksuitm_check: OS PID=369040 is still alive
ksuitm_check: OS PID=369036 is still alive
ksuitm_check: OS PID=369028 is still alive
ksuitm_check: OS PID=368991 is still alive
ksuitm_check: OS PID=368977 is still alive
ksuitm_check: OS PID=368969 is still alive
ksuitm_check: OS PID=368963 is still alive
ksuitm_check: OS PID=368981 is still alive
ksuitm_check: OS PID=368972 is still alive
ksuitm_check: OS PID=368965 is still alive
ksuitm_check: OS PID=368943 is still alive
ksuitm_check: OS PID=368946 is still alive
ksuitm_check: OS PID=368934 is still alive
ksuitm_check: OS PID=339474 is still alive
ksuitm_check: OS PID=368915 is still alive
ksuitm_check: OS PID=368909 is still alive
ksuitm_check: OS PID=368886 is still alive

*** 2019-11-15T14:52:25.652020+04:00 (CDB$ROOT(1))
Instance termination failed to kill one or more processes
ksuitm_check: OS PID=360990 is still alive
ksuitm_check: OS PID=360970 is still alive
ksuitm_check: OS PID=360880 is still alive
ksuitm_check: OS PID=360772 is still alive
ksuitm_check: OS PID=360757 is still alive
ksuitm_check: OS PID=360755 is still alive
ksuitm_check: OS PID=360753 is still alive
ksuitm_check: OS PID=360749 is still alive
ksuitm_check: OS PID=360746 is still alive
ksuitm_check: OS PID=360744 is still alive
ksuitm_check: OS PID=360734 is still alive
ksuitm_check: OS PID=360729 is still alive
ksuitm_check: OS PID=360724 is still alive
ksuitm_check: OS PID=360722 is still alive
ksuitm_check: OS PID=360714 is still alive
ksuitm_check: OS PID=360710 is still alive
ksuitm_check: OS PID=360706 is still alive
ksuitm_check: OS PID=360694 is still alive
ksuitm_check: OS PID=360691 is still alive
ksuitm_check: OS PID=360689 is still alive
ksuitm_check: OS PID=360687 is still alive
ksuitm_check: OS PID=360653 is still alive
ksuitm_check: OS PID=360651 is still alive
ksuitm_check: OS PID=360649 is still alive
ksuitm_check: OS PID=360645 is still alive
ksuitm_check: OS PID=360642 is still alive
ksuitm_check: OS PID=360640 is still alive
ksuitm_check: OS PID=360635 is still alive
ksuitm_check: OS PID=360631 is still alive
ksuitm_check: OS PID=360622 is still alive
ksuitm_check: OS PID=360620 is still alive
ksuitm_check: OS PID=360618 is still alive
ksuitm_check: OS PID=360179 is still alive
ksuitm_check: OS PID=360616 is still alive
ksuitm_check: OS PID=360612 is still alive
ksuitm_check: OS PID=360044 is still alive
ksuitm_check: OS PID=360614 is still alive
ksuitm_check: OS PID=360610 is still alive
ksuitm_check: OS PID=360607 is still alive
ksuitm_check: OS PID=360605 is still alive
ksuitm_check: OS PID=358415 is still alive
ksuitm_check: OS PID=360585 is still alive
ksuitm_check: OS PID=358375 is still alive
ksuitm_check: OS PID=360598 is still alive
ksuitm_check: OS PID=358298 is still alive
ksuitm_check: OS PID=358294 is still alive
ksuitm_check: OS PID=358291 is still alive
ksuitm_check: OS PID=358287 is still alive
ksuitm_check: OS PID=358284 is still alive
ksuitm_check: OS PID=358282 is still alive
ksuitm_check: OS PID=358280 is still alive
ksuitm_check: OS PID=358278 is still alive
ksuitm_check: OS PID=358276 is still alive
ksuitm_check: OS PID=358269 is still alive
ksuitm_check: OS PID=358228 is still alive
ksuitm_check: OS PID=358222 is still alive
ksuitm_check: OS PID=358220 is still alive
ksuitm_check: OS PID=358217 is still alive
ksuitm_check: OS PID=358213 is still alive
ksuitm_check: OS PID=358210 is still alive
ksuitm_check: OS PID=358202 is still alive
ksuitm_check: OS PID=358191 is still alive
ksuitm_check: OS PID=358194 is still alive
ksuitm_check: OS PID=358185 is still alive
ksuitm_check: OS PID=358181 is still alive
ksuitm_check: OS PID=358160 is still alive
ksuitm_check: OS PID=358156 is still alive
ksuitm_check: OS PID=358086 is still alive
ksuitm_check: OS PID=358146 is still alive
ksuitm_check: OS PID=358082 is still alive
ksuitm_check: OS PID=358075 is still alive
ksuitm_check: OS PID=358033 is still alive
ksuitm_check: OS PID=358022 is still alive
ksuitm_check: OS PID=358020 is still alive
ksuitm_check: OS PID=357961 is still alive
ksuitm_check: OS PID=357959 is still alive
ksuitm_check: OS PID=357955 is still alive
ksuitm_check: OS PID=357953 is still alive
ksuitm_check: OS PID=357948 is still alive
ksuitm_check: OS PID=357939 is still alive
ksuitm_check: OS PID=357937 is still alive
ksuitm_check: OS PID=357935 is still alive
ksuitm_check: OS PID=357931 is still alive
ksuitm_check: OS PID=357929 is still alive
ksuitm_check: OS PID=357927 is still alive
ksuitm_check: OS PID=357925 is still alive
ksuitm_check: OS PID=357923 is still alive
ksuitm_check: OS PID=357892 is still alive
ksuitm_check: OS PID=357880 is still alive
ksuitm_check: OS PID=357876 is still alive
ksuitm_check: OS PID=357874 is still alive
ksuitm_check: OS PID=357870 is still alive
ksuitm_check: OS PID=358061 is still alive
ksuitm_check: OS PID=357863 is still alive
ksuitm_check: OS PID=357861 is still alive
ksuitm_check: OS PID=357859 is still alive
ksuitm_check: OS PID=357855 is still alive
ksuitm_check: OS PID=357843 is still alive
ksuitm_check: OS PID=357841 is still alive
ksuitm_check: OS PID=357839 is still alive
ksuitm_check: OS PID=357837 is still alive
ksuitm_check: OS PID=357835 is still alive
ksuitm_check: OS PID=357824 is still alive
ksuitm_check: OS PID=357818 is still alive
ksuitm_check: OS PID=357814 is still alive
ksuitm_check: OS PID=357802 is still alive
ksuitm_check: OS PID=357799 is still alive
ksuitm_check: OS PID=357797 is still alive
ksuitm_check: OS PID=357795 is still alive
ksuitm_check: OS PID=357793 is still alive
ksuitm_check: OS PID=357748 is still alive
ksuitm_check: OS PID=357726 is still alive
ksuitm_check: OS PID=357722 is still alive
ksuitm_check: OS PID=357720 is still alive
ksuitm_check: OS PID=357715 is still alive
ksuitm_check: OS PID=357713 is still alive
ksuitm_check: OS PID=357710 is still alive
ksuitm_check: OS PID=357706 is still alive
ksuitm_check: OS PID=357702 is still alive
ksuitm_check: OS PID=357700 is still alive
ksuitm_check: OS PID=357698 is still alive
ksuitm_check: OS PID=357696 is still alive
ksuitm_check: OS PID=357694 is still alive
ksuitm_check: OS PID=357663 is still alive
ksuitm_check: OS PID=357655 is still alive
ksuitm_check: OS PID=357653 is still alive
ksuitm_check: OS PID=357651 is still alive
ksuitm_check: OS PID=357647 is still alive
ksuitm_check: OS PID=357632 is still alive
ksuitm_check: OS PID=357568 is still alive
ksuitm_check: OS PID=357551 is still alive
ksuitm_check: OS PID=357549 is still alive
ksuitm_check: OS PID=357547 is still alive
ksuitm_check: OS PID=357531 is still alive
ksuitm_check: OS PID=357468 is still alive
ksuitm_check: OS PID=357440 is still alive
ksuitm_check: OS PID=357355 is still alive
ksuitm_check: OS PID=357293 is still alive
ksuitm_check: OS PID=357257 is still alive
ksuitm_check: OS PID=357254 is still alive
ksuitm_check: OS PID=357250 is still alive
ksuitm_check: OS PID=357209 is still alive
ksuitm_check: OS PID=357207 is still alive
ksuitm_check: OS PID=357147 is still alive
ksuitm_check: OS PID=357140 is still alive
ksuitm_check: OS PID=357205 is still alive
ksuitm_check: OS PID=357162 is still alive
ksuitm_check: OS PID=357157 is still alive
ksuitm_check: OS PID=356318 is still alive
ksuitm_check: OS PID=356270 is still alive
ksuitm_check: OS PID=354493 is still alive
ksuitm_check: OS PID=354491 is still alive
ksuitm_check: OS PID=354483 is still alive
ksuitm_check: OS PID=354479 is still alive
ksuitm_check: OS PID=354477 is still alive
ksuitm_check: OS PID=354475 is still alive
ksuitm_check: OS PID=354473 is still alive
ksuitm_check: OS PID=354463 is still alive
ksuitm_check: OS PID=358024 is still alive
ksuitm_check: OS PID=354408 is still alive
ksuitm_check: OS PID=354396 is still alive
ksuitm_check: OS PID=354394 is still alive
ksuitm_check: OS PID=354382 is still alive
ksuitm_check: OS PID=354379 is still alive
ksuitm_check: OS PID=354368 is still alive
ksuitm_check: OS PID=354366 is still alive
ksuitm_check: OS PID=354364 is still alive
ksuitm_check: OS PID=354362 is still alive
ksuitm_check: OS PID=354360 is still alive
ksuitm_check: OS PID=354354 is still alive
ksuitm_check: OS PID=354334 is still alive
ksuitm_check: OS PID=354330 is still alive
ksuitm_check: OS PID=354318 is still alive
ksuitm_check: OS PID=354313 is still alive
ksuitm_check: OS PID=354309 is still alive
ksuitm_check: OS PID=354301 is still alive
ksuitm_check: OS PID=354298 is still alive
ksuitm_check: OS PID=354296 is still alive
ksuitm_check: OS PID=354294 is still alive
ksuitm_check: OS PID=354292 is still alive
ksuitm_check: OS PID=354290 is still alive
ksuitm_check: OS PID=354286 is still alive
ksuitm_check: OS PID=354284 is still alive
ksuitm_check: OS PID=354282 is still alive
ksuitm_check: OS PID=354280 is still alive
ksuitm_check: OS PID=354278 is still alive
ksuitm_check: OS PID=354273 is still alive
ksuitm_check: OS PID=354144 is still alive
ksuitm_check: OS PID=354140 is still alive
ksuitm_check: OS PID=354138 is still alive
ksuitm_check: OS PID=354136 is still alive
ksuitm_check: OS PID=354134 is still alive
ksuitm_check: OS PID=354129 is still alive
ksuitm_check: OS PID=354127 is still alive
ksuitm_check: OS PID=354125 is still alive
ksuitm_check: OS PID=354123 is still alive
ksuitm_check: OS PID=354121 is still alive
ksuitm_check: OS PID=354119 is still alive
ksuitm_check: OS PID=354117 is still alive
ksuitm_check: OS PID=354115 is still alive
ksuitm_check: OS PID=354113 is still alive
ksuitm_check: OS PID=354109 is still alive
ksuitm_check: OS PID=354072 is still alive
ksuitm_check: OS PID=354060 is still alive
ksuitm_check: OS PID=354057 is still alive
ksuitm_check: OS PID=354053 is still alive
ksuitm_check: OS PID=354046 is still alive
ksuitm_check: OS PID=354024 is still alive
ksuitm_check: OS PID=354014 is still alive
ksuitm_check: OS PID=354010 is still alive
ksuitm_check: OS PID=354006 is still alive
ksuitm_check: OS PID=354004 is still alive
ksuitm_check: OS PID=354002 is still alive
ksuitm_check: OS PID=354000 is still alive
ksuitm_check: OS PID=353998 is still alive
ksuitm_check: OS PID=353996 is still alive
ksuitm_check: OS PID=353992 is still alive
ksuitm_check: OS PID=353990 is still alive
ksuitm_check: OS PID=353988 is still alive
ksuitm_check: OS PID=353986 is still alive
ksuitm_check: OS PID=353984 is still alive
ksuitm_check: OS PID=353895 is still alive
ksuitm_check: OS PID=353891 is still alive
ksuitm_check: OS PID=353887 is still alive
ksuitm_check: OS PID=353883 is still alive
ksuitm_check: OS PID=353868 is still alive
ksuitm_check: OS PID=353864 is still alive
ksuitm_check: OS PID=353854 is still alive
ksuitm_check: OS PID=353840 is still alive
ksuitm_check: OS PID=353838 is still alive
ksuitm_check: OS PID=353820 is still alive
ksuitm_check: OS PID=353832 is still alive
ksuitm_check: OS PID=353797 is still alive
ksuitm_check: OS PID=353795 is still alive
ksuitm_check: OS PID=353793 is still alive
ksuitm_check: OS PID=353785 is still alive
ksuitm_check: OS PID=353783 is still alive
ksuitm_check: OS PID=353764 is still alive
ksuitm_check: OS PID=353982 is still alive
ksuitm_check: OS PID=353470 is still alive
ksuitm_check: OS PID=353449 is still alive
ksuitm_check: OS PID=353432 is still alive
ksuitm_check: OS PID=353399 is still alive
ksuitm_check: OS PID=353394 is still alive
ksuitm_check: OS PID=353387 is still alive
ksuitm_check: OS PID=353385 is still alive
ksuitm_check: OS PID=353383 is still alive
ksuitm_check: OS PID=353380 is still alive
ksuitm_check: OS PID=353368 is still alive
ksuitm_check: OS PID=353363 is still alive
ksuitm_check: OS PID=353360 is still alive
ksuitm_check: OS PID=353358 is still alive
ksuitm_check: OS PID=353356 is still alive
ksuitm_check: OS PID=353302 is still alive
ksuitm_check: OS PID=353300 is still alive
ksuitm_check: OS PID=353298 is still alive
ksuitm_check: OS PID=353296 is still alive
ksuitm_check: OS PID=353294 is still alive
ksuitm_check: OS PID=353284 is still alive
ksuitm_check: OS PID=353282 is still alive
ksuitm_check: OS PID=353278 is still alive
ksuitm_check: OS PID=353276 is still alive
ksuitm_check: OS PID=353265 is still alive
ksuitm_check: OS PID=353263 is still alive
ksuitm_check: OS PID=353261 is still alive
ksuitm_check: OS PID=353259 is still alive
ksuitm_check: OS PID=353257 is still alive
ksuitm_check: OS PID=353249 is still alive
ksuitm_check: OS PID=353236 is still alive
ksuitm_check: OS PID=353234 is still alive
ksuitm_check: OS PID=353229 is still alive
ksuitm_check: OS PID=353354 is still alive
ksuitm_check: OS PID=353352 is still alive
ksuitm_check: OS PID=353215 is still alive
ksuitm_check: OS PID=360174 is still alive
ksuitm_check: OS PID=353350 is still alive
ksuitm_check: OS PID=353187 is still alive
ksuitm_check: OS PID=353348 is still alive
ksuitm_check: OS PID=353346 is still alive
ksuitm_check: OS PID=353127 is still alive
ksuitm_check: OS PID=353343 is still alive
ksuitm_check: OS PID=353201 is still alive
ksuitm_check: OS PID=360568 is still alive
ksuitm_check: OS PID=353341 is still alive
ksuitm_check: OS PID=352330 is still alive
ksuitm_check: OS PID=352326 is still alive
ksuitm_check: OS PID=352324 is still alive
ksuitm_check: OS PID=352320 is still alive
ksuitm_check: OS PID=352318 is still alive
ksuitm_check: OS PID=352316 is still alive
ksuitm_check: OS PID=352232 is still alive
ksuitm_check: OS PID=352230 is still alive
ksuitm_check: OS PID=352226 is still alive
ksuitm_check: OS PID=352216 is still alive
ksuitm_check: OS PID=352213 is still alive
ksuitm_check: OS PID=352208 is still alive
ksuitm_check: OS PID=352194 is still alive
ksuitm_check: OS PID=352186 is still alive
ksuitm_check: OS PID=352161 is still alive
ksuitm_check: OS PID=352163 is still alive
ksuitm_check: OS PID=352154 is still alive
ksuitm_check: OS PID=352151 is still alive
ksuitm_check: OS PID=352147 is still alive
ksuitm_check: OS PID=352145 is still alive
ksuitm_check: OS PID=352132 is still alive
ksuitm_check: OS PID=352124 is still alive
ksuitm_check: OS PID=352122 is still alive
ksuitm_check: OS PID=352120 is still alive
ksuitm_check: OS PID=352116 is still alive
ksuitm_check: OS PID=352104 is still alive
ksuitm_check: OS PID=352102 is still alive
ksuitm_check: OS PID=352092 is still alive
ksuitm_check: OS PID=352086 is still alive
ksuitm_check: OS PID=352084 is still alive
ksuitm_check: OS PID=352082 is still alive
ksuitm_check: OS PID=352080 is still alive
ksuitm_check: OS PID=352078 is still alive
ksuitm_check: OS PID=352076 is still alive
ksuitm_check: OS PID=352073 is still alive
ksuitm_check: OS PID=352053 is still alive
ksuitm_check: OS PID=352049 is still alive
ksuitm_check: OS PID=352043 is still alive
ksuitm_check: OS PID=352040 is still alive
ksuitm_check: OS PID=352034 is still alive
ksuitm_check: OS PID=352020 is still alive
ksuitm_check: OS PID=352018 is still alive
ksuitm_check: OS PID=352014 is still alive
ksuitm_check: OS PID=352012 is still alive
ksuitm_check: OS PID=352006 is still alive
ksuitm_check: OS PID=352002 is still alive
ksuitm_check: OS PID=351974 is still alive
ksuitm_check: OS PID=351969 is still alive
ksuitm_check: OS PID=351967 is still alive
ksuitm_check: OS PID=351959 is still alive
ksuitm_check: OS PID=351956 is still alive
ksuitm_check: OS PID=351952 is still alive
ksuitm_check: OS PID=351950 is still alive
ksuitm_check: OS PID=351947 is still alive
ksuitm_check: OS PID=351945 is still alive
ksuitm_check: OS PID=351928 is still alive
ksuitm_check: OS PID=351787 is still alive
ksuitm_check: OS PID=351686 is still alive
ksuitm_check: OS PID=351658 is still alive
ksuitm_check: OS PID=351651 is still alive
ksuitm_check: OS PID=351647 is still alive
ksuitm_check: OS PID=351643 is still alive
ksuitm_check: OS PID=351599 is still alive
ksuitm_check: OS PID=351601 is still alive
ksuitm_check: OS PID=351595 is still alive
ksuitm_check: OS PID=351592 is still alive
ksuitm_check: OS PID=351588 is still alive
ksuitm_check: OS PID=351586 is still alive
ksuitm_check: OS PID=351584 is still alive
ksuitm_check: OS PID=351582 is still alive
ksuitm_check: OS PID=351580 is still alive
ksuitm_check: OS PID=351578 is still alive
ksuitm_check: OS PID=351576 is still alive
ksuitm_check: OS PID=351574 is still alive
ksuitm_check: OS PID=351572 is still alive
ksuitm_check: OS PID=351566 is still alive
ksuitm_check: OS PID=351553 is still alive
ksuitm_check: OS PID=351551 is still alive
ksuitm_check: OS PID=351543 is still alive
ksuitm_check: OS PID=351541 is still alive
ksuitm_check: OS PID=351538 is still alive
ksuitm_check: OS PID=351533 is still alive
ksuitm_check: OS PID=351511 is still alive
ksuitm_check: OS PID=351505 is still alive
ksuitm_check: OS PID=351496 is still alive
ksuitm_check: OS PID=351492 is still alive
ksuitm_check: OS PID=351488 is still alive
ksuitm_check: OS PID=351482 is still alive
ksuitm_check: OS PID=351480 is still alive
ksuitm_check: OS PID=351478 is still alive
ksuitm_check: OS PID=351476 is still alive
ksuitm_check: OS PID=351474 is still alive
ksuitm_check: OS PID=351460 is still alive
ksuitm_check: OS PID=351446 is still alive
ksuitm_check: OS PID=351444 is still alive
ksuitm_check: OS PID=351442 is still alive
ksuitm_check: OS PID=351440 is still alive
ksuitm_check: OS PID=351436 is still alive
ksuitm_check: OS PID=351434 is still alive
ksuitm_check: OS PID=351361 is still alive
ksuitm_check: OS PID=351357 is still alive
ksuitm_check: OS PID=351359 is still alive
ksuitm_check: OS PID=351215 is still alive
ksuitm_check: OS PID=351211 is still alive
ksuitm_check: OS PID=351209 is still alive
ksuitm_check: OS PID=351207 is still alive
ksuitm_check: OS PID=351204 is still alive
ksuitm_check: OS PID=351196 is still alive
ksuitm_check: OS PID=351191 is still alive
ksuitm_check: OS PID=351185 is still alive
ksuitm_check: OS PID=351179 is still alive
ksuitm_check: OS PID=351173 is still alive
ksuitm_check: OS PID=351134 is still alive
ksuitm_check: OS PID=351091 is still alive
ksuitm_check: OS PID=351076 is still alive
ksuitm_check: OS PID=351078 is still alive
ksuitm_check: OS PID=351062 is still alive
ksuitm_check: OS PID=351045 is still alive
ksuitm_check: OS PID=350947 is still alive
ksuitm_check: OS PID=350945 is still alive
ksuitm_check: OS PID=350943 is still alive
ksuitm_check: OS PID=350941 is still alive
ksuitm_check: OS PID=350928 is still alive
ksuitm_check: OS PID=351217 is still alive
ksuitm_check: OS PID=350901 is still alive
ksuitm_check: OS PID=350728 is still alive
ksuitm_check: OS PID=350726 is still alive
ksuitm_check: OS PID=350689 is still alive
ksuitm_check: OS PID=350682 is still alive
ksuitm_check: OS PID=350687 is still alive
ksuitm_check: OS PID=350676 is still alive
ksuitm_check: OS PID=350662 is still alive
ksuitm_check: OS PID=350658 is still alive
ksuitm_check: OS PID=350656 is still alive
ksuitm_check: OS PID=350652 is still alive
ksuitm_check: OS PID=350641 is still alive
ksuitm_check: OS PID=350637 is still alive
ksuitm_check: OS PID=350629 is still alive
ksuitm_check: OS PID=350626 is still alive
ksuitm_check: OS PID=350624 is still alive
ksuitm_check: OS PID=350620 is still alive
ksuitm_check: OS PID=350618 is still alive
ksuitm_check: OS PID=350616 is still alive
ksuitm_check: OS PID=350614 is still alive
ksuitm_check: OS PID=350600 is still alive
ksuitm_check: OS PID=350598 is still alive
ksuitm_check: OS PID=350596 is still alive
ksuitm_check: OS PID=350594 is still alive
ksuitm_check: OS PID=350592 is still alive
ksuitm_check: OS PID=350590 is still alive
ksuitm_check: OS PID=350588 is still alive
ksuitm_check: OS PID=350884 is still alive
ksuitm_check: OS PID=350886 is still alive
ksuitm_check: OS PID=350878 is still alive
ksuitm_check: OS PID=350537 is still alive
ksuitm_check: OS PID=350535 is still alive
ksuitm_check: OS PID=350868 is still alive
ksuitm_check: OS PID=350528 is still alive
ksuitm_check: OS PID=350526 is still alive
ksuitm_check: OS PID=350522 is still alive
ksuitm_check: OS PID=350520 is still alive
ksuitm_check: OS PID=350517 is still alive
ksuitm_check: OS PID=350515 is still alive
ksuitm_check: OS PID=350513 is still alive
ksuitm_check: OS PID=350500 is still alive
ksuitm_check: OS PID=350496 is still alive
ksuitm_check: OS PID=350484 is still alive
ksuitm_check: OS PID=350481 is still alive
ksuitm_check: OS PID=350477 is still alive
ksuitm_check: OS PID=350475 is still alive
ksuitm_check: OS PID=350470 is still alive
ksuitm_check: OS PID=350468 is still alive
ksuitm_check: OS PID=350466 is still alive
ksuitm_check: OS PID=350464 is still alive
ksuitm_check: OS PID=350462 is still alive
ksuitm_check: OS PID=350459 is still alive
ksuitm_check: OS PID=350455 is still alive
ksuitm_check: OS PID=350453 is still alive
ksuitm_check: OS PID=350446 is still alive
ksuitm_check: OS PID=350422 is still alive
ksuitm_check: OS PID=350420 is still alive
ksuitm_check: OS PID=350415 is still alive
ksuitm_check: OS PID=350402 is still alive
ksuitm_check: OS PID=350391 is still alive
ksuitm_check: OS PID=350384 is still alive
ksuitm_check: OS PID=350373 is still alive
ksuitm_check: OS PID=350369 is still alive
ksuitm_check: OS PID=350365 is still alive
ksuitm_check: OS PID=350363 is still alive
ksuitm_check: OS PID=350361 is still alive
ksuitm_check: OS PID=350358 is still alive
ksuitm_check: OS PID=350350 is still alive
ksuitm_check: OS PID=350329 is still alive
ksuitm_check: OS PID=350326 is still alive
ksuitm_check: OS PID=350324 is still alive
ksuitm_check: OS PID=350316 is still alive
ksuitm_check: OS PID=350314 is still alive
ksuitm_check: OS PID=350310 is still alive
ksuitm_check: OS PID=350308 is still alive
ksuitm_check: OS PID=350299 is still alive
ksuitm_check: OS PID=350297 is still alive
ksuitm_check: OS PID=350293 is still alive
ksuitm_check: OS PID=350253 is still alive
ksuitm_check: OS PID=350200 is still alive
ksuitm_check: OS PID=350108 is still alive
ksuitm_check: OS PID=350045 is still alive
ksuitm_check: OS PID=349953 is still alive
ksuitm_check: OS PID=349907 is still alive
ksuitm_check: OS PID=349903 is still alive
ksuitm_check: OS PID=349901 is still alive
ksuitm_check: OS PID=349898 is still alive
ksuitm_check: OS PID=349895 is still alive
ksuitm_check: OS PID=349892 is still alive
ksuitm_check: OS PID=349889 is still alive
ksuitm_check: OS PID=349887 is still alive
ksuitm_check: OS PID=349885 is still alive
ksuitm_check: OS PID=349875 is still alive
ksuitm_check: OS PID=349863 is still alive
ksuitm_check: OS PID=349861 is still alive
ksuitm_check: OS PID=347105 is still alive
ksuitm_check: OS PID=346886 is still alive
ksuitm_check: OS PID=346884 is still alive
ksuitm_check: OS PID=346881 is still alive
ksuitm_check: OS PID=346879 is still alive
ksuitm_check: OS PID=346877 is still alive
ksuitm_check: OS PID=346873 is still alive
ksuitm_check: OS PID=346870 is still alive
ksuitm_check: OS PID=346866 is still alive
ksuitm_check: OS PID=346858 is still alive
ksuitm_check: OS PID=346848 is still alive
ksuitm_check: OS PID=346846 is still alive
ksuitm_check: OS PID=346844 is still alive
ksuitm_check: OS PID=346842 is still alive
ksuitm_check: OS PID=346840 is still alive
ksuitm_check: OS PID=346825 is still alive
ksuitm_check: OS PID=346821 is still alive
ksuitm_check: OS PID=346814 is still alive
ksuitm_check: OS PID=346783 is still alive
ksuitm_check: OS PID=346765 is still alive
ksuitm_check: OS PID=346541 is still alive
ksuitm_check: OS PID=346539 is still alive
ksuitm_check: OS PID=346537 is still alive
ksuitm_check: OS PID=346527 is still alive
ksuitm_check: OS PID=346522 is still alive
ksuitm_check: OS PID=346520 is still alive
ksuitm_check: OS PID=346517 is still alive
ksuitm_check: OS PID=346512 is still alive
ksuitm_check: OS PID=346510 is still alive
ksuitm_check: OS PID=346498 is still alive
ksuitm_check: OS PID=346494 is still alive
ksuitm_check: OS PID=346492 is still alive
ksuitm_check: OS PID=346150 is still alive
ksuitm_check: OS PID=346148 is still alive
ksuitm_check: OS PID=346124 is still alive
ksuitm_check: OS PID=346490 is still alive
ksuitm_check: OS PID=346488 is still alive
ksuitm_check: OS PID=346485 is still alive
ksuitm_check: OS PID=346482 is still alive
ksuitm_check: OS PID=346479 is still alive
ksuitm_check: OS PID=346474 is still alive
ksuitm_check: OS PID=346085 is still alive
ksuitm_check: OS PID=346470 is still alive
ksuitm_check: OS PID=346081 is still alive
ksuitm_check: OS PID=346079 is still alive
ksuitm_check: OS PID=346468 is still alive
ksuitm_check: OS PID=346073 is still alive
ksuitm_check: OS PID=346464 is still alive
ksuitm_check: OS PID=346068 is still alive
ksuitm_check: OS PID=346066 is still alive
ksuitm_check: OS PID=346064 is still alive
ksuitm_check: OS PID=346058 is still alive
ksuitm_check: OS PID=346060 is still alive
ksuitm_check: OS PID=346054 is still alive
ksuitm_check: OS PID=346051 is still alive
ksuitm_check: OS PID=346049 is still alive
ksuitm_check: OS PID=346037 is still alive
ksuitm_check: OS PID=346035 is still alive
ksuitm_check: OS PID=346024 is still alive
ksuitm_check: OS PID=346022 is still alive
ksuitm_check: OS PID=346019 is still alive
ksuitm_check: OS PID=346017 is still alive
ksuitm_check: OS PID=346013 is still alive
ksuitm_check: OS PID=346007 is still alive
ksuitm_check: OS PID=346005 is still alive
ksuitm_check: OS PID=346001 is still alive
ksuitm_check: OS PID=345987 is still alive
ksuitm_check: OS PID=345983 is still alive
ksuitm_check: OS PID=345978 is still alive
ksuitm_check: OS PID=345973 is still alive
ksuitm_check: OS PID=345943 is still alive
ksuitm_check: OS PID=345941 is still alive
ksuitm_check: OS PID=345937 is still alive
ksuitm_check: OS PID=345933 is still alive
ksuitm_check: OS PID=345922 is still alive
ksuitm_check: OS PID=345919 is still alive
ksuitm_check: OS PID=345909 is still alive
ksuitm_check: OS PID=345905 is still alive
ksuitm_check: OS PID=345899 is still alive
ksuitm_check: OS PID=345887 is still alive
ksuitm_check: OS PID=345882 is still alive
ksuitm_check: OS PID=345872 is still alive
ksuitm_check: OS PID=345869 is still alive
ksuitm_check: OS PID=345867 is still alive
ksuitm_check: OS PID=345865 is still alive
ksuitm_check: OS PID=345863 is still alive
ksuitm_check: OS PID=345861 is still alive
ksuitm_check: OS PID=345859 is still alive
ksuitm_check: OS PID=345857 is still alive
ksuitm_check: OS PID=345855 is still alive
ksuitm_check: OS PID=345851 is still alive
ksuitm_check: OS PID=345847 is still alive
ksuitm_check: OS PID=345830 is still alive
ksuitm_check: OS PID=345833 is still alive
ksuitm_check: OS PID=345826 is still alive
ksuitm_check: OS PID=345822 is still alive
ksuitm_check: OS PID=345820 is still alive
ksuitm_check: OS PID=333255 is still alive
ksuitm_check: OS PID=344723 is still alive
ksuitm_check: OS PID=352192 is still alive
ksuitm_check: OS PID=332956 is still alive
ksuitm_check: OS PID=332954 is still alive
ksuitm_check: OS PID=332951 is still alive
ksuitm_check: OS PID=332949 is still alive
ksuitm_check: OS PID=351432 is still alive
ksuitm_check: OS PID=344694 is still alive
ksuitm_check: OS PID=332941 is still alive
ksuitm_check: OS PID=349859 is still alive
ksuitm_check: OS PID=332894 is still alive
ksuitm_check: OS PID=315854 is still alive
ksuitm_check: OS PID=332891 is still alive
ksuitm_check: OS PID=332889 is still alive
ksuitm_check: OS PID=315850 is still alive
ksuitm_check: OS PID=332887 is still alive
ksuitm_check: OS PID=315848 is still alive
ksuitm_check: OS PID=332883 is still alive
ksuitm_check: OS PID=315846 is still alive
ksuitm_check: OS PID=275447 is still alive
ksuitm_check: OS PID=345764 is still alive
ksuitm_check: OS PID=320938 is still alive
ksuitm_check: OS PID=319582 is still alive
ksuitm_check: OS PID=344685 is still alive
ksuitm_check: OS PID=332232 is still alive
ksuitm_check: OS PID=144360 is still alive
ksuitm_check: OS PID=345725 is still alive
ksuitm_check: OS PID=338194 is still alive
ksuitm_check: OS PID=352190 is still alive
ksuitm_check: OS PID=350473 is still alive
ksuitm_check: OS PID=332223 is still alive
ksuitm_check: OS PID=345633 is still alive
ksuitm_check: OS PID=339607 is still alive
ksuitm_check: OS PID=345573 is still alive
ksuitm_check: OS PID=315013 is still alive
ksuitm_check: OS PID=339526 is still alive
ksuitm_check: OS PID=339524 is still alive
ksuitm_check: OS PID=332584 is still alive
ksuitm_check: OS PID=332582 is still alive
ksuitm_check: OS PID=345534 is still alive
ksuitm_check: OS PID=302786 is still alive
ksuitm_check: OS PID=332579 is still alive
ksuitm_check: OS PID=302784 is still alive
ksuitm_check: OS PID=349857 is still alive
ksuitm_check: OS PID=224600 is still alive
ksuitm_check: OS PID=251863 is still alive
ksuitm_check: OS PID=332577 is still alive
ksuitm_check: OS PID=332575 is still alive
ksuitm_check: OS PID=332939 is still alive
ksuitm_check: OS PID=332937 is still alive
ksuitm_check: OS PID=332926 is still alive
ksuitm_check: OS PID=332885 is still alive
ksuitm_check: OS PID=332876 is still alive
ksuitm_check: OS PID=332573 is still alive
ksuitm_check: OS PID=344750 is still alive
ksuitm_check: OS PID=332868 is still alive
ksuitm_check: OS PID=251573 is still alive
ksuitm_check: OS PID=301846 is still alive
ksuitm_check: OS PID=314954 is still alive
ksuitm_check: OS PID=345527 is still alive
ksuitm_check: OS PID=333738 is still alive
ksuitm_check: OS PID=314908 is still alive
ksuitm_check: OS PID=346502 is still alive
ksuitm_check: OS PID=333864 is still alive
ksuitm_check: OS PID=275371 is still alive
ksuitm_check: OS PID=332879 is still alive
ksuitm_check: OS PID=339629 is still alive
ksuitm_check: OS PID=344589 is still alive
ksuitm_check: OS PID=344587 is still alive
ksuitm_check: OS PID=354356 is still alive
ksuitm_check: OS PID=338834 is still alive
ksuitm_check: OS PID=349855 is still alive
ksuitm_check: OS PID=349853 is still alive
ksuitm_check: OS PID=349851 is still alive
ksuitm_check: OS PID=332567 is still alive
ksuitm_check: OS PID=71738 is still alive
ksuitm_check: OS PID=333291 is still alive
ksuitm_check: OS PID=337373 is still alive
ksuitm_check: OS PID=307154 is still alive
ksuitm_check: OS PID=345418 is still alive
ksuitm_check: OS PID=113607 is still alive
ksuitm_check: OS PID=238213 is still alive
ksuitm_check: OS PID=332922 is still alive
ksuitm_check: OS PID=332918 is still alive
ksuitm_check: OS PID=332916 is still alive
ksuitm_check: OS PID=332914 is still alive
ksuitm_check: OS PID=122743 is still alive
ksuitm_check: OS PID=346773 is still alive
ksuitm_check: OS PID=334408 is still alive
ksuitm_check: OS PID=233245 is still alive
ksuitm_check: OS PID=332852 is still alive
ksuitm_check: OS PID=71162 is still alive
ksuitm_check: OS PID=333057 is still alive
ksuitm_check: OS PID=269538 is still alive
ksuitm_check: OS PID=332563 is still alive
ksuitm_check: OS PID=349849 is still alive
ksuitm_check: OS PID=267137 is still alive
ksuitm_check: OS PID=276391 is still alive
ksuitm_check: OS PID=332466 is still alive
ksuitm_check: OS PID=333068 is still alive
ksuitm_check: OS PID=353391 is still alive
ksuitm_check: OS PID=349847 is still alive
ksuitm_check: OS PID=353389 is still alive
ksuitm_check: OS PID=345394 is still alive
ksuitm_check: OS PID=353191 is still alive
ksuitm_check: OS PID=345361 is still alive
ksuitm_check: OS PID=338742 is still alive
ksuitm_check: OS PID=123530 is still alive
ksuitm_check: OS PID=350932 is still alive
ksuitm_check: OS PID=296623 is still alive
ksuitm_check: OS PID=338950 is still alive
ksuitm_check: OS PID=338740 is still alive
ksuitm_check: OS PID=338682 is still alive
ksuitm_check: OS PID=338678 is still alive
ksuitm_check: OS PID=338641 is still alive
ksuitm_check: OS PID=359695 is still alive
ksuitm_check: OS PID=338634 is still alive
ksuitm_check: OS PID=388592 is still alive
ksuitm_check: OS PID=338592 is still alive
ksuitm_check: OS PID=388582 is still alive
ksuitm_check: OS PID=151346 is still alive
ksuitm_check: OS PID=137861 is still alive
ksuitm_check: OS PID=307033 is still alive
ksuitm_check: OS PID=388569 is still alive
ksuitm_check: OS PID=239496 is still alive
ksuitm_check: OS PID=334239 is still alive
ksuitm_check: OS PID=354402 is still alive
ksuitm_check: OS PID=334237 is still alive
ksuitm_check: OS PID=354434 is still alive
ksuitm_check: OS PID=388417 is still alive
ksuitm_check: OS PID=339465 is still alive
ksuitm_check: OS PID=288210 is still alive
ksuitm_check: OS PID=353461 is still alive
ksuitm_check: OS PID=354400 is still alive
ksuitm_check: OS PID=344979 is still alive
ksuitm_check: OS PID=320975 is still alive
ksuitm_check: OS PID=354251 is still alive
ksuitm_check: OS PID=42629 is still alive
ksuitm_check: OS PID=332908 is still alive
ksuitm_check: OS PID=354245 is still alive
ksuitm_check: OS PID=353366 is still alive
ksuitm_check: OS PID=287807 is still alive
ksuitm_check: OS PID=360565 is still alive
ksuitm_check: OS PID=354234 is still alive
ksuitm_check: OS PID=354232 is still alive
ksuitm_check: OS PID=269826 is still alive
ksuitm_check: OS PID=354230 is still alive
ksuitm_check: OS PID=319943 is still alive
ksuitm_check: OS PID=357639 is still alive
ksuitm_check: OS PID=354220 is still alive
ksuitm_check: OS PID=354218 is still alive
ksuitm_check: OS PID=320791 is still alive
ksuitm_check: OS PID=354213 is still alive
ksuitm_check: OS PID=42621 is still alive
ksuitm_check: OS PID=354211 is still alive
ksuitm_check: OS PID=354209 is still alive
ksuitm_check: OS PID=354206 is still alive
ksuitm_check: OS PID=358080 is still alive
ksuitm_check: OS PID=354204 is still alive
ksuitm_check: OS PID=354962 is still alive
ksuitm_check: OS PID=354202 is still alive
ksuitm_check: OS PID=332958 is still alive
ksuitm_check: OS PID=347015 is still alive
ksuitm_check: OS PID=354200 is still alive
ksuitm_check: OS PID=354196 is still alive
ksuitm_check: OS PID=349845 is still alive
ksuitm_check: OS PID=350874 is still alive
ksuitm_check: OS PID=346995 is still alive
ksuitm_check: OS PID=341341 is still alive
ksuitm_check: OS PID=312624 is still alive
ksuitm_check: OS PID=8736 is still alive
ksuitm_check: OS PID=282610 is still alive
ksuitm_check: OS PID=72140 is still alive
ksuitm_check: OS PID=346983 is still alive
ksuitm_check: OS PID=361123 is still alive
ksuitm_check: OS PID=177937 is still alive
ksuitm_check: OS PID=320813 is still alive
ksuitm_check: OS PID=387851 is still alive
ksuitm_check: OS PID=339848 is still alive
ksuitm_check: OS PID=339158 is still alive
ksuitm_check: OS PID=346977 is still alive
ksuitm_check: OS PID=346971 is still alive
ksuitm_check: OS PID=345411 is still alive
ksuitm_check: OS PID=286909 is still alive
ksuitm_check: OS PID=346969 is still alive
ksuitm_check: OS PID=353339 is still alive
ksuitm_check: OS PID=346955 is still alive
ksuitm_check: OS PID=346953 is still alive
ksuitm_check: OS PID=314906 is still alive
ksuitm_check: OS PID=346946 is still alive
ksuitm_check: OS PID=350926 is still alive
ksuitm_check: OS PID=346942 is still alive
ksuitm_check: OS PID=346930 is still alive
ksuitm_check: OS PID=346922 is still alive
ksuitm_check: OS PID=298085 is still alive
ksuitm_check: OS PID=346912 is still alive
ksuitm_check: OS PID=350864 is still alive
ksuitm_check: OS PID=346910 is still alive
ksuitm_check: OS PID=294772 is still alive
ksuitm_check: OS PID=346906 is still alive
ksuitm_check: OS PID=339811 is still alive
ksuitm_check: OS PID=361103 is still alive
ksuitm_check: OS PID=42544 is still alive
ksuitm_check: OS PID=344439 is still alive
ksuitm_check: OS PID=353994 is still alive
ksuitm_check: OS PID=345311 is still alive
ksuitm_check: OS PID=346908 is still alive
ksuitm_check: OS PID=338566 is still alive
ksuitm_check: OS PID=162947 is still alive
ksuitm_check: OS PID=327984 is still alive
ksuitm_check: OS PID=338326 is still alive
ksuitm_check: OS PID=185758 is still alive
ksuitm_check: OS PID=370823 is still alive
ksuitm_check: OS PID=370999 is still alive
ksuitm_check: OS PID=346904 is still alive
ksuitm_check: OS PID=346966 is still alive
ksuitm_check: OS PID=185122 is still alive
ksuitm_check: OS PID=155830 is still alive
ksuitm_check: OS PID=374282 is still alive
ksuitm_check: OS PID=339805 is still alive
ksuitm_check: OS PID=161123 is still alive
ksuitm_check: OS PID=364922 is still alive
ksuitm_check: OS PID=345849 is still alive
ksuitm_check: OS PID=287179 is still alive
ksuitm_check: OS PID=334172 is still alive
ksuitm_check: OS PID=393386 is still alive
ksuitm_check: OS PID=353280 is still alive
ksuitm_check: OS PID=352761 is still alive
ksuitm_check: OS PID=322530 is still alive
ksuitm_check: OS PID=58549 is still alive
ksuitm_check: OS PID=375178 is still alive
ksuitm_check: OS PID=375174 is still alive
ksuitm_check: OS PID=375170 is still alive
ksuitm_check: OS PID=375163 is still alive
ksuitm_check: OS PID=375143 is still alive
ksuitm_check: OS PID=345308 is still alive
ksuitm_check: OS PID=319293 is still alive
ksuitm_check: OS PID=264429 is still alive
ksuitm_check: OS PID=393586 is still alive
ksuitm_check: OS PID=375176 is still alive
ksuitm_check: OS PID=375172 is still alive
ksuitm_check: OS PID=375166 is still alive
ksuitm_check: OS PID=375087 is still alive
ksuitm_check: OS PID=375161 is still alive
ksuitm_check: OS PID=375150 is still alive
ksuitm_check: OS PID=338579 is still alive
ksuitm_check: OS PID=375052 is still alive
ksuitm_check: OS PID=375031 is still alive
ksuitm_check: OS PID=374966 is still alive
ksuitm_check: OS PID=374962 is still alive
ksuitm_check: OS PID=374958 is still alive
ksuitm_check: OS PID=374954 is still alive
ksuitm_check: OS PID=350312 is still alive
ksuitm_check: OS PID=374950 is still alive
ksuitm_check: OS PID=374946 is still alive
ksuitm_check: OS PID=375057 is still alive
ksuitm_check: OS PID=374938 is still alive
ksuitm_check: OS PID=375037 is still alive
ksuitm_check: OS PID=374924 is still alive
ksuitm_check: OS PID=374908 is still alive
ksuitm_check: OS PID=374902 is still alive
ksuitm_check: OS PID=374898 is still alive
ksuitm_check: OS PID=374956 is still alive
ksuitm_check: OS PID=374952 is still alive
ksuitm_check: OS PID=374948 is still alive
ksuitm_check: OS PID=157396 is still alive
ksuitm_check: OS PID=374884 is still alive
ksuitm_check: OS PID=374873 is still alive
ksuitm_check: OS PID=374932 is still alive
ksuitm_check: OS PID=374861 is still alive
ksuitm_check: OS PID=374920 is still alive
ksuitm_check: OS PID=374841 is still alive
ksuitm_check: OS PID=374930 is still alive
ksuitm_check: OS PID=374773 is still alive
ksuitm_check: OS PID=374904 is still alive
ksuitm_check: OS PID=374737 is still alive
ksuitm_check: OS PID=374877 is still alive
ksuitm_check: OS PID=374900 is still alive
ksuitm_check: OS PID=350827 is still alive
ksuitm_check: OS PID=374896 is still alive
ksuitm_check: OS PID=374720 is still alive
ksuitm_check: OS PID=374869 is still alive
ksuitm_check: OS PID=374702 is still alive
ksuitm_check: OS PID=374935 is still alive
ksuitm_check: OS PID=374670 is still alive
ksuitm_check: OS PID=374847 is still alive
ksuitm_check: OS PID=374660 is still alive
ksuitm_check: OS PID=374837 is still alive
ksuitm_check: OS PID=374769 is still alive
ksuitm_check: OS PID=374607 is still alive
ksuitm_check: OS PID=374597 is still alive
ksuitm_check: OS PID=374728 is still alive
ksuitm_check: OS PID=354288 is still alive
ksuitm_check: OS PID=374940 is still alive
ksuitm_check: OS PID=374680 is still alive
ksuitm_check: OS PID=374581 is still alive
ksuitm_check: OS PID=374666 is still alive
ksuitm_check: OS PID=374706 is still alive
ksuitm_check: OS PID=360595 is still alive
ksuitm_check: OS PID=374626 is still alive
ksuitm_check: OS PID=374145 is still alive
ksuitm_check: OS PID=374543 is still alive
ksuitm_check: OS PID=374603 is still alive
ksuitm_check: OS PID=374127 is still alive
ksuitm_check: OS PID=373950 is still alive
ksuitm_check: OS PID=374587 is still alive
ksuitm_check: OS PID=345285 is still alive
ksuitm_check: OS PID=374550 is still alive
ksuitm_check: OS PID=373569 is still alive
ksuitm_check: OS PID=374152 is still alive
ksuitm_check: OS PID=319291 is still alive
ksuitm_check: OS PID=374138 is still alive
ksuitm_check: OS PID=373389 is still alive
ksuitm_check: OS PID=374109 is still alive
ksuitm_check: OS PID=314658 is still alive
ksuitm_check: OS PID=166107 is still alive
ksuitm_check: OS PID=150730 is still alive
ksuitm_check: OS PID=328905 is still alive
ksuitm_check: OS PID=373610 is still alive
ksuitm_check: OS PID=334169 is still alive
ksuitm_check: OS PID=373362 is still alive
ksuitm_check: OS PID=325956 is still alive
ksuitm_check: OS PID=238639 is still alive
ksuitm_check: OS PID=373284 is still alive
ksuitm_check: OS PID=373328 is still alive
ksuitm_check: OS PID=373279 is still alive
ksuitm_check: OS PID=373304 is still alive
ksuitm_check: OS PID=373237 is still alive
ksuitm_check: OS PID=320150 is still alive
ksuitm_check: OS PID=373229 is still alive
ksuitm_check: OS PID=373235 is still alive
ksuitm_check: OS PID=373212 is still alive
ksuitm_check: OS PID=373221 is still alive
ksuitm_check: OS PID=151144 is still alive
ksuitm_check: OS PID=373205 is still alive
ksuitm_check: OS PID=373203 is still alive
ksuitm_check: OS PID=373197 is still alive
ksuitm_check: OS PID=310281 is still alive
ksuitm_check: OS PID=349843 is still alive
ksuitm_check: OS PID=373191 is still alive
ksuitm_check: OS PID=373181 is still alive
ksuitm_check: OS PID=373187 is still alive
ksuitm_check: OS PID=373179 is still alive
ksuitm_check: OS PID=373173 is still alive
ksuitm_check: OS PID=373165 is still alive
ksuitm_check: OS PID=373158 is still alive
ksuitm_check: OS PID=373094 is still alive
ksuitm_check: OS PID=373075 is still alive
ksuitm_check: OS PID=373073 is still alive
ksuitm_check: OS PID=372988 is still alive
ksuitm_check: OS PID=373404 is still alive
ksuitm_check: OS PID=360588 is still alive
ksuitm_check: OS PID=372783 is still alive
ksuitm_check: OS PID=372771 is still alive
ksuitm_check: OS PID=372765 is still alive
ksuitm_check: OS PID=372753 is still alive
ksuitm_check: OS PID=364532 is still alive
ksuitm_check: OS PID=372728 is still alive
ksuitm_check: OS PID=372760 is still alive
ksuitm_check: OS PID=364081 is still alive
ksuitm_check: OS PID=372657 is still alive
ksuitm_check: OS PID=334001 is still alive
ksuitm_check: OS PID=372747 is still alive
ksuitm_check: OS PID=373326 is still alive
ksuitm_check: OS PID=372745 is still alive
ksuitm_check: OS PID=372651 is still alive
ksuitm_check: OS PID=39417 is still alive
ksuitm_check: OS PID=372649 is still alive
ksuitm_check: OS PID=372706 is still alive
ksuitm_check: OS PID=372647 is still alive
ksuitm_check: OS PID=162767 is still alive
ksuitm_check: OS PID=372635 is still alive
ksuitm_check: OS PID=372631 is still alive
ksuitm_check: OS PID=372625 is still alive
ksuitm_check: OS PID=157095 is still alive
ksuitm_check: OS PID=372509 is still alive
ksuitm_check: OS PID=374890 is still alive
ksuitm_check: OS PID=371889 is still alive
ksuitm_check: OS PID=353972 is still alive
ksuitm_check: OS PID=302701 is still alive
ksuitm_check: OS PID=369129 is still alive
ksuitm_check: OS PID=369100 is still alive
ksuitm_check: OS PID=369119 is still alive
ksuitm_check: OS PID=369096 is still alive
ksuitm_check: OS PID=369114 is still alive
ksuitm_check: OS PID=369088 is still alive
ksuitm_check: OS PID=369092 is still alive
ksuitm_check: OS PID=369058 is still alive
ksuitm_check: OS PID=369060 is still alive
ksuitm_check: OS PID=328865 is still alive
ksuitm_check: OS PID=369040 is still alive
ksuitm_check: OS PID=369036 is still alive
ksuitm_check: OS PID=369028 is still alive
ksuitm_check: OS PID=368991 is still alive
ksuitm_check: OS PID=368977 is still alive
ksuitm_check: OS PID=368969 is still alive
ksuitm_check: OS PID=368963 is still alive
ksuitm_check: OS PID=368981 is still alive
ksuitm_check: OS PID=368972 is still alive
ksuitm_check: OS PID=368965 is still alive
ksuitm_check: OS PID=368943 is still alive
ksuitm_check: OS PID=368946 is still alive
ksuitm_check: OS PID=368934 is still alive
ksuitm_check: OS PID=339474 is still alive
ksuitm_check: OS PID=368915 is still alive
ksuitm_check: OS PID=368909 is still alive
ksuitm_check: OS PID=368886 is still alive
Warning: 686 processes are still attach to shmid 130482192:

*** 2019-11-15T14:52:25.668067+04:00
(size: 2097152 bytes, creator pid: 368747, last attach/detach pid: 366515)

*** 2019-11-15T14:52:25.668095+04:00
*********************** Dumping ipcs output ********************

*** 2019-11-15T14:52:25.675549+04:00
------ Shared Memory Segments --------

*** 2019-11-15T14:52:25.675589+04:00
key shmid owner perms bytes nattch status

*** 2019-11-15T14:52:25.675606+04:00
0x00000000 0 root 644 64 2 dest

*** 2019-11-15T14:52:25.675620+04:00
0x00000000 32769 root 644 16384 2 dest

*** 2019-11-15T14:52:25.675633+04:00
0x00000000 65538 root 644 280 2 dest

*** 2019-11-15T14:52:25.675650+04:00
0x00000000 98307 root 644 80 2

*** 2019-11-15T14:52:25.675663+04:00
0x00000000 131076 root 644 16384 2

*** 2019-11-15T14:52:25.675677+04:00
0x00000000 163845 root 644 280 2

*** 2019-11-15T14:52:25.675690+04:00
0x00000000 1545764870 oracle 600 10485760 47

*** 2019-11-15T14:52:25.675704+04:00
0x00000000 1545797639 oracle 600 3204448256 47

*** 2019-11-15T14:52:25.675717+04:00
0xec5fc108 1545830408 oracle 600 16384 47

*** 2019-11-15T14:52:25.675731+04:00
0x00000000 1545863177 oracle 600 4096 0

*** 2019-11-15T14:52:25.675747+04:00
0x00000000 1545895946 oracle 600 4096 0

*** 2019-11-15T14:52:25.675760+04:00
0xef74da1c 1545928715 oracle 600 24576 31

*** 2019-11-15T14:52:25.675774+04:00
0x00000000 130351116 oracle 600 31457280 686 dest

*** 2019-11-15T14:52:25.675787+04:00
0x00000000 130383885 oracle 600 85899345920 1372 dest

*** 2019-11-15T14:52:25.675801+04:00
0x00000000 130416654 oracle 600 590021132288 3437 dest

*** 2019-11-15T14:52:25.675814+04:00
0x00000000 130449423 oracle 600 278921216 686 dest

*** 2019-11-15T14:52:25.675827+04:00
0x00000000 130482192 oracle 600 2097152 686 dest

*** 2019-11-15T14:52:25.675844+04:00
------ Semaphore Arrays --------

*** 2019-11-15T14:52:25.675857+04:00
key semid owner perms nsems

*** 2019-11-15T14:52:25.675871+04:00
0x61001b17 98306 root 666 1

*** 2019-11-15T14:52:25.675884+04:00
0x00000000 163844 root 666 3

*** 2019-11-15T14:52:25.675897+04:00
0x00000000 1769477 root 666 3

*** 2019-11-15T14:52:25.675911+04:00
0xcfe88130 284229638 oracle 600 514

*** 2019-11-15T14:52:25.675924+04:00
0xcfe88131 284262407 oracle 600 514

*** 2019-11-15T14:52:25.675940+04:00
0xcfe88132 284295176 oracle 600 514

*** 2019-11-15T14:52:25.675954+04:00
0x670d69c8 284426249 oracle 600 124

*** 2019-11-15T14:52:25.675967+04:00
0x00000000 4096019 root 666 3

*** 2019-11-15T14:52:25.675980+04:00
0x00000000 5701652 root 666 3

*** 2019-11-15T14:52:25.675993+04:00
0x00000000 7307285 root 666 3

*** 2019-11-15T14:52:25.676007+04:00
0x00000000 8912918 root 666 3

*** 2019-11-15T14:52:25.676020+04:00
0x00000000 10518551 root 666 3

*** 2019-11-15T14:52:25.676036+04:00
0x00000000 12124184 root 666 3

*** 2019-11-15T14:52:25.676050+04:00
0x00000000 13729817 root 666 3

*** 2019-11-15T14:52:25.676063+04:00
0x00000000 15335450 root 666 3

*** 2019-11-15T14:52:25.676076+04:00
0x00000000 16941083 root 666 3

*** 2019-11-15T14:52:25.676090+04:00
0x00000000 18546716 root 666 3

*** 2019-11-15T14:52:25.676103+04:00
0x00000000 20152349 root 666 3

*** 2019-11-15T14:52:25.676121+04:00
0x00000000 21757982 root 666 3

*** 2019-11-15T14:52:25.676137+04:00
0x00000000 23363615 root 666 3

*** 2019-11-15T14:52:25.676151+04:00
0x00000000 24969248 root 666 3

*** 2019-11-15T14:52:25.676164+04:00
0x00000000 26574881 root 666 3

*** 2019-11-15T14:52:25.676177+04:00
0x00000000 31326242 root 666 3

*** 2019-11-15T14:52:25.676190+04:00
0x00000000 32931875 root 666 3

*** 2019-11-15T14:52:25.676204+04:00
0x00000000 34537508 root 666 3

*** 2019-11-15T14:52:25.676217+04:00
0x00000000 36143141 root 666 3

*** 2019-11-15T14:52:25.676233+04:00
0x00000000 37748774 root 666 3

*** 2019-11-15T14:52:25.676246+04:00
0x00000000 39354407 root 666 3

*** 2019-11-15T14:52:25.676259+04:00
0x00000000 41353256 root 666 3

*** 2019-11-15T14:52:25.676273+04:00
0x00000000 42958889 root 666 3

*** 2019-11-15T14:52:25.676286+04:00
0x00000000 44564522 root 666 3

*** 2019-11-15T14:52:25.676299+04:00
0x00000000 46170155 root 666 3

*** 2019-11-15T14:52:25.676312+04:00
0x00000000 47775788 root 666 3

*** 2019-11-15T14:52:25.676328+04:00
0x00000000 49381421 root 666 3

*** 2019-11-15T14:52:25.676342+04:00
0x00000000 50987054 root 666 3

*** 2019-11-15T14:52:25.676355+04:00
0x00000000 52592687 root 666 3

*** 2019-11-15T14:52:25.676368+04:00
0x00000000 54198320 root 666 3

*** 2019-11-15T14:52:25.676381+04:00
0x00000000 55803953 root 666 3

*** 2019-11-15T14:52:25.676395+04:00
0x00000000 57409586 root 666 3

*** 2019-11-15T14:52:25.676408+04:00
0x00000000 59015219 root 666 3

*** 2019-11-15T14:52:25.676424+04:00
0x00000000 60620852 root 666 3

*** 2019-11-15T14:52:25.676437+04:00
0x00000000 62226485 root 666 3

*** 2019-11-15T14:52:25.676451+04:00
0x00000000 63832118 root 666 3

*** 2019-11-15T14:52:25.676464+04:00
0x00000000 65830967 root 666 3

*** 2019-11-15T14:52:25.676477+04:00
0x00000000 67436600 root 666 3

*** 2019-11-15T14:52:25.676493+04:00
0x00000000 69042233 root 666 3

*** 2019-11-15T14:52:25.676506+04:00
0x00000000 70647866 root 666 3

*** 2019-11-15T14:52:25.676522+04:00
0x00000000 72253499 root 666 3

*** 2019-11-15T14:52:25.676535+04:00
0x00000000 73859132 root 666 3

*** 2019-11-15T14:52:25.676549+04:00
0x00000000 75464765 root 666 3

*** 2019-11-15T14:52:25.676562+04:00
0x00000000 77070398 root 666 3

*** 2019-11-15T14:52:25.676575+04:00
0x00000000 78676031 root 666 3

*** 2019-11-15T14:52:25.676593+04:00
0x00000000 80281664 root 666 3

*** 2019-11-15T14:52:25.676607+04:00
0x00000000 81887297 root 666 3

*** 2019-11-15T14:52:25.676622+04:00
0x00000000 83492930 root 666 3

*** 2019-11-15T14:52:25.676636+04:00
0x00000000 85098563 root 666 3

*** 2019-11-15T14:52:25.676649+04:00
0x00000000 87097412 root 666 3

*** 2019-11-15T14:52:25.676662+04:00
0x00000000 88703045 root 666 3

*** 2019-11-15T14:52:25.676675+04:00
0x00000000 90308678 root 666 3

*** 2019-11-15T14:52:25.676689+04:00
0x00000000 92307527 root 666 3

*** 2019-11-15T14:52:25.676702+04:00
0x00000000 93913160 root 666 3

*** 2019-11-15T14:52:25.676718+04:00
0x00000000 95518793 root 666 3

*** 2019-11-15T14:52:25.676734+04:00
0x00000000 97124426 root 666 3

*** 2019-11-15T14:52:25.676748+04:00
0x00000000 98762827 root 666 3

*** 2019-11-15T14:52:25.676761+04:00
0x00000000 100368460 root 666 3

*** 2019-11-15T14:52:25.676774+04:00
0x00000000 101974093 root 666 3

*** 2019-11-15T14:52:25.676788+04:00
0x00000000 103579726 root 666 3

*** 2019-11-15T14:52:25.676801+04:00
0x00000000 105185359 root 666 3

*** 2019-11-15T14:52:25.676817+04:00
0x00000000 106790992 root 666 3

*** 2019-11-15T14:52:25.676830+04:00
0x00000000 108396625 root 666 3

*** 2019-11-15T14:52:25.676844+04:00
0x00000000 110002258 root 666 3

*** 2019-11-15T14:52:25.676857+04:00
0x00000000 111607891 root 666 3

*** 2019-11-15T14:52:25.676870+04:00
0x00000000 113213524 root 666 3

*** 2019-11-15T14:52:25.676884+04:00
0x00000000 114819157 root 666 3

*** 2019-11-15T14:52:25.676897+04:00
0x00000000 116424790 root 666 3

*** 2019-11-15T14:52:25.676913+04:00
0x00000000 118030423 root 666 3

*** 2019-11-15T14:52:25.676926+04:00
0x00000000 119636056 root 666 3

*** 2019-11-15T14:52:25.676939+04:00
0x00000000 121241689 root 666 3

*** 2019-11-15T14:52:25.676953+04:00
0x00000000 122847322 root 666 3

*** 2019-11-15T14:52:25.676966+04:00
0x00000000 124452955 root 666 3

*** 2019-11-15T14:52:25.676979+04:00
0x00000000 126058588 root 666 3

*** 2019-11-15T14:52:25.676993+04:00
0x00000000 127664221 root 666 3

*** 2019-11-15T14:52:25.677009+04:00
0x00000000 129269854 root 666 3

*** 2019-11-15T14:52:25.677022+04:00
0x00000000 130875487 root 666 3

*** 2019-11-15T14:52:25.677035+04:00
0x00000000 132481120 root 666 3

*** 2019-11-15T14:52:25.677049+04:00
0x00000000 134086753 root 666 3

*** 2019-11-15T14:52:25.677062+04:00
0x00000000 135692386 root 666 3

*** 2019-11-15T14:52:25.677075+04:00
0x00000000 137298019 root 666 3

*** 2019-11-15T14:52:25.677089+04:00
0x00000000 138903652 root 666 3

*** 2019-11-15T14:52:25.677105+04:00
0x00000000 140509285 root 666 3

*** 2019-11-15T14:52:25.677118+04:00
0x00000000 142114918 root 666 3

*** 2019-11-15T14:52:25.677131+04:00
0x00000000 143720551 root 666 3

*** 2019-11-15T14:52:25.677144+04:00
0x00000000 145326184 root 666 3

*** 2019-11-15T14:52:25.677158+04:00
0x00000000 146931817 root 666 3

*** 2019-11-15T14:52:25.677171+04:00
0x00000000 148537450 root 666 3

*** 2019-11-15T14:52:25.677184+04:00
0x00000000 150143083 root 666 3

*** 2019-11-15T14:52:25.677200+04:00
0x00000000 151748716 root 666 3

*** 2019-11-15T14:52:25.677214+04:00
0x00000000 153354349 root 666 3

*** 2019-11-15T14:52:25.677227+04:00
0x00000000 154959982 root 666 3

*** 2019-11-15T14:52:25.677240+04:00
0x00000000 156565615 root 666 3

*** 2019-11-15T14:52:25.677254+04:00
0x00000000 158171248 root 666 3

*** 2019-11-15T14:52:25.677267+04:00
0x00000000 159776881 root 666 3

*** 2019-11-15T14:52:25.677280+04:00
0x00000000 161382514 root 666 3

*** 2019-11-15T14:52:25.677296+04:00
0x00000000 162988147 root 666 3

*** 2019-11-15T14:52:25.677310+04:00
0x00000000 164593780 root 666 3

*** 2019-11-15T14:52:25.677323+04:00
0x00000000 166199413 root 666 3

*** 2019-11-15T14:52:25.677339+04:00
0x00000000 167805046 root 666 3

*** 2019-11-15T14:52:25.677353+04:00
0x00000000 169410679 root 666 3

*** 2019-11-15T14:52:25.677366+04:00
0x00000000 171737208 root 666 3

*** 2019-11-15T14:52:25.677379+04:00
0x00000000 173342841 root 666 3

*** 2019-11-15T14:52:25.677395+04:00
0x00000000 174948474 root 666 3

*** 2019-11-15T14:52:25.677408+04:00
0x00000000 176554107 root 666 3

*** 2019-11-15T14:52:25.677422+04:00
0x00000000 178159740 root 666 3

*** 2019-11-15T14:52:25.677435+04:00
0x00000000 179765373 root 666 3

*** 2019-11-15T14:52:25.677448+04:00
0x00000000 181371006 root 666 3

*** 2019-11-15T14:52:25.677462+04:00
0x00000000 182976639 root 666 3

*** 2019-11-15T14:52:25.677475+04:00
0x00000000 184582272 root 666 3

*** 2019-11-15T14:52:25.677491+04:00
0x00000000 186187905 root 666 3

*** 2019-11-15T14:52:25.677504+04:00
0x00000000 187793538 root 666 3

*** 2019-11-15T14:52:25.677517+04:00
0x00000000 189399171 root 666 3

*** 2019-11-15T14:52:25.677533+04:00
0x00000000 191004804 root 666 3

*** 2019-11-15T14:52:25.677546+04:00
0x00000000 192610437 root 666 3

*** 2019-11-15T14:52:25.677559+04:00
0x00000000 194216070 root 666 3

*** 2019-11-15T14:52:25.677573+04:00
0x00000000 195821703 root 666 3

*** 2019-11-15T14:52:25.677592+04:00
0x00000000 197427336 root 666 3

*** 2019-11-15T14:52:25.677606+04:00
0x00000000 199032969 root 666 3

*** 2019-11-15T14:52:25.677619+04:00
0x00000000 200638602 root 666 3

*** 2019-11-15T14:52:25.677632+04:00
0x00000000 202244235 root 666 3

*** 2019-11-15T14:52:25.677645+04:00
0x00000000 203849868 root 666 3

*** 2019-11-15T14:52:25.677659+04:00
0x00000000 205455501 root 666 3

*** 2019-11-15T14:52:25.677672+04:00
0x00000000 207061134 root 666 3

*** 2019-11-15T14:52:25.677688+04:00
0x00000000 208666767 root 666 3

*** 2019-11-15T14:52:25.677701+04:00
0x00000000 210272400 root 666 3

*** 2019-11-15T14:52:25.677715+04:00
0x00000000 211878033 root 666 3

*** 2019-11-15T14:52:25.677728+04:00
0x00000000 213483666 root 666 3

*** 2019-11-15T14:52:25.677741+04:00
0x00000000 215089299 root 666 3

*** 2019-11-15T14:52:25.677755+04:00
0x00000000 216694932 root 666 3

*** 2019-11-15T14:52:25.677768+04:00
0x00000000 218300565 root 666 3

*** 2019-11-15T14:52:25.677784+04:00
0x00000000 219906198 root 666 3

*** 2019-11-15T14:52:25.677797+04:00
0x00000000 221511831 root 666 3

*** 2019-11-15T14:52:25.677810+04:00
0x00000000 223117464 root 666 3

*** 2019-11-15T14:52:25.677824+04:00
0x00000000 224723097 root 666 3

*** 2019-11-15T14:52:25.677837+04:00
0x00000000 226328730 root 666 3

*** 2019-11-15T14:52:25.677850+04:00
0x00000000 227934363 root 666 3

*** 2019-11-15T14:52:25.677864+04:00
0x00000000 229539996 root 666 3

*** 2019-11-15T14:52:25.677880+04:00
0x00000000 231145629 root 666 3

*** 2019-11-15T14:52:25.677893+04:00
0x00000000 232751262 root 666 3

*** 2019-11-15T14:52:25.677906+04:00
0x00000000 234356895 root 666 3

*** 2019-11-15T14:52:25.677920+04:00
0x00000000 235962528 root 666 3

*** 2019-11-15T14:52:25.677933+04:00
0x00000000 237568161 root 666 3

*** 2019-11-15T14:52:25.677950+04:00
0x00000000 239173794 root 666 3

*** 2019-11-15T14:52:25.677964+04:00
0x00000000 240779427 root 666 3

*** 2019-11-15T14:52:25.677980+04:00
0x00000000 242385060 root 666 3

*** 2019-11-15T14:52:25.677993+04:00
0x00000000 243990693 root 666 3

*** 2019-11-15T14:52:25.678007+04:00
0x00000000 245596326 root 666 3

*** 2019-11-15T14:52:25.678020+04:00
0x00000000 247201959 root 666 3

*** 2019-11-15T14:52:25.678033+04:00
0x00000000 248807592 root 666 3

*** 2019-11-15T14:52:25.678047+04:00
0x00000000 250413225 root 666 3

*** 2019-11-15T14:52:25.678061+04:00
0x00000000 252018858 root 666 3

*** 2019-11-15T14:52:25.678077+04:00
0x00000000 253624491 root 666 3

*** 2019-11-15T14:52:25.678090+04:00
0x00000000 255230124 root 666 3

*** 2019-11-15T14:52:25.678104+04:00
0x00000000 256835757 root 666 3

*** 2019-11-15T14:52:25.678117+04:00
0x00000000 258441390 root 666 3

*** 2019-11-15T14:52:25.678130+04:00
0x00000000 260047023 root 666 3

*** 2019-11-15T14:52:25.678144+04:00
0x00000000 261652656 root 666 3

*** 2019-11-15T14:52:25.678157+04:00
0x00000000 263258289 root 666 3

*** 2019-11-15T14:52:25.678173+04:00
0x00000000 264863922 root 666 3

*** 2019-11-15T14:52:25.678186+04:00
0x00000000 266469555 root 666 3

*** 2019-11-15T14:52:25.678199+04:00
0x00000000 268075188 root 666 3

*** 2019-11-15T14:52:25.678213+04:00
0x00000000 269680821 root 666 3

*** 2019-11-15T14:52:25.678226+04:00
0x00000000 271286454 root 666 3

*** 2019-11-15T14:52:25.678240+04:00
0x00000000 272892087 root 666 3

*** 2019-11-15T14:52:25.678253+04:00
0x00000000 274497720 root 666 3

*** 2019-11-15T14:52:25.678269+04:00
0x00000000 276103353 root 666 3

*** 2019-11-15T14:52:25.678282+04:00
0x00000000 277708986 root 666 3

*** 2019-11-15T14:52:25.678295+04:00
0x00000000 279314619 root 666 3

*** 2019-11-15T14:52:25.678309+04:00
0x00000000 280920252 root 666 3

*** 2019-11-15T14:52:25.678322+04:00
0x00000000 282525885 root 666 3

*** 2019-11-15T14:52:25.678335+04:00
0x00000000 286032062 root 666 3

*** 2019-11-15T14:52:25.678349+04:00
0x00000000 287637695 root 666 3

*** 2019-11-15T14:52:25.678365+04:00
0x00000000 289243328 root 666 3

*** 2019-11-15T14:52:25.678378+04:00
0x00000000 290848961 root 666 3

*** 2019-11-15T14:52:25.678391+04:00
0x00000000 292454594 root 666 3

*** 2019-11-15T14:52:25.678405+04:00
0x00000000 294060227 root 666 3

*** 2019-11-15T14:52:25.678418+04:00
0x00000000 295665860 root 666 3

*** 2019-11-15T14:52:25.678431+04:00
0x00000000 297271493 root 666 3

*** 2019-11-15T14:52:25.678444+04:00
0x00000000 298877126 root 666 3

*** 2019-11-15T14:52:25.678460+04:00
0x00000000 300482759 root 666 3

*** 2019-11-15T14:52:25.678474+04:00
0x00000000 302088392 root 666 3

*** 2019-11-15T14:52:25.678487+04:00
0x00000000 303694025 root 666 3

*** 2019-11-15T14:52:25.678500+04:00
0x00000000 305299658 root 666 3

*** 2019-11-15T14:52:25.678513+04:00
0x00000000 306905291 root 666 3

*** 2019-11-15T14:52:25.678527+04:00
0x00000000 308510924 root 666 3

*** 2019-11-15T14:52:25.678540+04:00
0x00000000 310116557 root 666 3

*** 2019-11-15T14:52:25.678560+04:00
0x00000000 311722190 root 666 3

*** 2019-11-15T14:52:25.678574+04:00
0x00000000 313327823 root 666 3

*** 2019-11-15T14:52:25.678590+04:00
0x00000000 314933456 root 666 3

*** 2019-11-15T14:52:25.678604+04:00
0x00000000 316539089 root 666 3

*** 2019-11-15T14:52:25.678617+04:00
0x00000000 318144722 root 666 3

*** 2019-11-15T14:52:25.678631+04:00
0x00000000 319750355 root 666 3

*** 2019-11-15T14:52:25.678644+04:00
0x00000000 321355988 root 666 3

*** 2019-11-15T14:52:25.678660+04:00
0x00000000 322961621 root 666 3

*** 2019-11-15T14:52:25.678673+04:00
0x00000000 324567254 root 666 3

*** 2019-11-15T14:52:25.678687+04:00
0x00000000 326172887 root 666 3

*** 2019-11-15T14:52:25.678700+04:00
0x00000000 327778520 root 666 3

*** 2019-11-15T14:52:25.678713+04:00
0x00000000 329384153 root 666 3

*** 2019-11-15T14:52:25.678727+04:00
0x00000000 330989786 root 666 3

*** 2019-11-15T14:52:25.678740+04:00
0x00000000 332595419 root 666 3

*** 2019-11-15T14:52:25.678756+04:00
0x00000000 334201052 root 666 3

*** 2019-11-15T14:52:25.678769+04:00
0x00000000 335806685 root 666 3

*** 2019-11-15T14:52:25.678783+04:00
0x00000000 337412318 root 666 3

*** 2019-11-15T14:52:25.678796+04:00
0x00000000 339017951 root 666 3

*** 2019-11-15T14:52:25.678809+04:00
0x00000000 340623584 root 666 3

*** 2019-11-15T14:52:25.678823+04:00
0x00000000 342229217 root 666 3

*** 2019-11-15T14:52:25.678836+04:00
0x00000000 343834850 root 666 3

*** 2019-11-15T14:52:25.678852+04:00
0x00000000 345440483 root 666 3

*** 2019-11-15T14:52:25.678865+04:00
0x00000000 347046116 root 666 3

*** 2019-11-15T14:52:25.678879+04:00
0x00000000 348651749 root 666 3

*** 2019-11-15T14:52:25.678892+04:00
0x00000000 350257382 root 666 3

*** 2019-11-15T14:52:25.678905+04:00
0x00000000 352256231 root 666 3

*** 2019-11-15T14:52:25.678919+04:00
0x00000000 353861864 root 666 3

*** 2019-11-15T14:52:25.678932+04:00
0x00000000 355467497 root 666 3

*** 2019-11-15T14:52:25.678948+04:00
0x00000000 357073130 root 666 3

*** 2019-11-15T14:52:25.678961+04:00
0x00000000 358678763 root 666 3

*** 2019-11-15T14:52:25.678975+04:00
0x00000000 360284396 root 666 3

*** 2019-11-15T14:52:25.678988+04:00
0x00000000 361890029 root 666 3

*** 2019-11-15T14:52:25.679001+04:00
0x00000000 363495662 root 666 3

*** 2019-11-15T14:52:25.679015+04:00
0x00000000 365101295 root 666 3

*** 2019-11-15T14:52:25.679028+04:00
0x00000000 366706928 root 666 3

*** 2019-11-15T14:52:25.679044+04:00
0x00000000 368312561 root 666 3

*** 2019-11-15T14:52:25.679057+04:00
0x00000000 369918194 root 666 3

*** 2019-11-15T14:52:25.679070+04:00
0x00000000 371523827 root 666 3

*** 2019-11-15T14:52:25.679084+04:00
0x00000000 373129460 root 666 3

*** 2019-11-15T14:52:25.679097+04:00
0x00000000 374735093 root 666 3

*** 2019-11-15T14:52:25.679110+04:00
0x00000000 376340726 root 666 3

*** 2019-11-15T14:52:25.679124+04:00
0x00000000 377946359 root 666 3

*** 2019-11-15T14:52:25.679140+04:00
0x00000000 379551992 root 666 3

*** 2019-11-15T14:52:25.679153+04:00
0x00000000 381157625 root 666 3

*** 2019-11-15T14:52:25.679169+04:00
0x00000000 382763258 root 666 3

*** 2019-11-15T14:52:25.679182+04:00
0x00000000 384368891 root 666 3

*** 2019-11-15T14:52:25.679196+04:00
0x00000000 385974524 root 666 3

*** 2019-11-15T14:52:25.679209+04:00
0x00000000 387580157 root 666 3

*** 2019-11-15T14:52:25.679222+04:00
0x00000000 389185790 root 666 3

*** 2019-11-15T14:52:25.679238+04:00
0x00000000 389480703 root 666 3

*** 2019-11-15T14:52:25.679252+04:00
------ Message Queues --------

*** 2019-11-15T14:52:25.679265+04:00
key msqid owner perms used-bytes messages

*** 2019-11-15T14:52:25.679305+04:00
*********************** End of ipcs command dump **************

*** 2019-11-15T14:52:25.679319+04:00