RAC instance crash
Locked
 
|
Hello, Mr. Erman
We are currently using Exadata X7-2, and one of the database-based apps is also called web_cons. One of the cluster nodes turns off when application admin increase the number of read only connections to 3000 which coming application server.. (2 nodes RAC) db_configuration: SGA_MAX_SIZE 630GB pga_aggregate_target = 30GB pga_aggregate_limit = 60GB sessions = 11616 processes = 7680 total physical RAM is 754Gb and we've configured HugePages. HugePage size is 640Gb Thanks in advance |
Re: RAC instance crash
Administrator
|
Hi,
This kind of a reboot can be caused by a hang(as a result of high load -due to cpu or I/O) or a bug. Send me the following files: The cssdagent logs The cssdmonitor logs The ocssd logs The lastgasp log(s) in /etc/oracle/lastgasp or /var/opt/oracle/lastgasp CHM or OS Watcher data 'opatch lsinventory -detail' output for the GRID home Messages files /var/log/messages sar -q output from the node (sar -q -f /file_that_has_the_info_with_the_related_time_interval(reboot)) |
|
|
Hi,
cssdagent log, cssdmonitor log, ocssd log and lastgasp log location is empty. I can give you db and crs alert logs and trace files. sar output: [root@exa-hddbadm01 ~]#sar -q -f /var/log/sa/sa15 Linux 4.1.12-94.8.4.el6uek.x86_64 (exa-hddbadm01.core.iamas.gov.az) 11/15/2019 _x86_64_ (96 CPU) 12:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 12:10:01 AM 9 2852 2.15 2.61 2.53 12:20:01 AM 2 2828 1.35 1.94 2.27 12:30:01 AM 5 2831 1.66 1.95 2.16 12:40:01 AM 3 2822 1.60 1.75 1.98 12:50:01 AM 5 2823 1.39 1.71 1.88 01:00:01 AM 2 2833 1.60 1.76 1.84 01:10:01 AM 2 2826 2.09 1.79 1.80 01:20:01 AM 2 2815 1.52 1.80 1.84 01:30:01 AM 2 2831 1.57 1.82 1.85 01:40:01 AM 3 2828 1.60 1.65 1.74 01:50:01 AM 2 2819 1.91 1.94 1.83 02:00:01 AM 6 2836 1.73 1.79 1.82 02:10:01 AM 5 2814 1.81 2.09 2.04 02:20:01 AM 5 2821 1.31 1.55 1.80 02:30:01 AM 5 2826 1.59 1.54 1.69 02:40:01 AM 3 2831 1.76 1.65 1.65 02:50:01 AM 3 2826 1.84 1.77 1.69 03:00:01 AM 4 2827 1.84 2.46 2.18 03:10:01 AM 8 2836 2.44 2.48 2.29 03:20:01 AM 3 2843 1.95 2.33 2.35 03:30:01 AM 3 2864 3.26 3.38 2.89 03:40:01 AM 4 2860 2.19 3.11 3.07 03:40:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 03:50:01 AM 3 2825 2.73 2.64 2.80 04:00:01 AM 8 2839 2.36 2.36 2.55 04:10:01 AM 5 2816 1.74 2.12 2.36 04:20:01 AM 10 2937 2.05 1.99 2.16 04:30:01 AM 3 2831 2.81 2.73 2.50 04:40:01 AM 5 2844 1.98 2.13 2.28 04:50:01 AM 4 2834 2.42 2.33 2.30 05:00:01 AM 5 2848 1.75 2.12 2.22 05:10:01 AM 4 2825 3.51 2.59 2.35 05:20:01 AM 7 2814 2.11 2.23 2.25 05:30:01 AM 4 2825 2.01 2.15 2.26 05:40:01 AM 7 2810 2.26 2.20 2.22 05:50:01 AM 3 2805 1.95 2.21 2.23 06:00:01 AM 8 2824 2.01 2.15 2.19 06:10:01 AM 3 2831 2.91 2.50 2.34 06:20:01 AM 2 2811 2.12 2.35 2.34 06:30:01 AM 5 2811 2.57 2.58 2.50 06:40:01 AM 7 2816 2.84 2.40 2.46 06:50:01 AM 5 2809 2.52 2.57 2.57 07:00:01 AM 6 2830 3.34 3.22 2.90 07:10:01 AM 4 2820 2.08 2.42 2.65 07:20:01 AM 7 2808 2.81 2.65 2.69 07:20:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 07:30:01 AM 4 2841 3.56 3.56 3.21 07:40:01 AM 1 2842 2.67 3.32 3.36 07:50:01 AM 1 2840 1.21 1.88 2.62 08:00:01 AM 4 2844 2.65 2.00 2.26 08:10:01 AM 2 2823 1.87 1.81 2.03 08:20:01 AM 3 2818 2.41 1.92 1.95 08:30:01 AM 3 2813 2.21 2.57 2.26 08:40:01 AM 4 2842 2.75 2.67 2.42 08:50:01 AM 2 2843 2.10 2.14 2.23 09:00:01 AM 4 2851 2.49 2.36 2.29 09:10:01 AM 7 2860 3.65 3.22 2.74 09:20:01 AM 20 2885 14.27 7.47 4.51 09:30:01 AM 8 2875 2.58 3.60 3.97 09:40:01 AM 2 2877 2.68 3.41 3.76 09:50:01 AM 3 2898 2.86 3.41 3.64 10:00:01 AM 5 2883 3.54 3.49 3.53 10:10:01 AM 7 2882 3.37 3.52 3.52 10:20:01 AM 7 2995 3.88 4.03 3.78 10:30:01 AM 6 2900 3.50 3.91 3.83 10:40:01 AM 8 2874 4.90 5.07 4.49 10:50:01 AM 8 2874 4.31 4.31 4.45 11:00:01 AM 6 3004 6.09 7.00 5.70 11:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 11:10:01 AM 8 2880 6.25 6.61 6.11 11:20:01 AM 16 2884 8.52 7.09 6.49 11:30:01 AM 4 2886 7.26 7.67 7.12 11:40:01 AM 7 2877 6.29 6.65 6.90 11:50:01 AM 7 2877 4.55 5.48 6.22 12:00:01 PM 10 2896 4.96 4.84 5.50 12:10:01 PM 3 2886 3.05 4.19 5.02 12:20:01 PM 0 2880 3.99 4.21 4.66 12:30:01 PM 6 2885 5.33 4.39 4.49 12:40:01 PM 4 2914 4.37 4.50 4.50 12:50:01 PM 7 2900 2.72 3.53 3.97 01:00:01 PM 3 2953 3.08 3.27 3.65 01:10:01 PM 3 2943 2.34 2.67 3.19 01:20:01 PM 3 2927 3.10 2.78 2.98 01:30:01 PM 5 2940 3.38 3.21 3.15 01:40:01 PM 2 2925 3.16 3.08 3.12 01:50:01 PM 6 2926 3.62 3.22 3.17 02:00:01 PM 3 2871 2.92 3.69 3.58 02:10:01 PM 2 2864 2.94 3.05 3.27 02:20:01 PM 5 2943 4.08 3.73 3.55 02:30:01 PM 6 2940 4.50 3.92 3.66 02:40:01 PM 2 2930 3.46 3.44 3.54 02:40:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 02:50:01 PM 8 4036 8.08 5.13 4.16 03:00:01 PM 1 2270 1.69 62.03 63.01 03:10:01 PM 1 2248 1.40 9.70 33.88 03:20:01 PM 0 2241 0.42 1.72 17.99 03:30:01 PM 1 2785 2.24 1.90 10.22 03:40:01 PM 15 2895 7.65 4.35 7.18 03:50:01 PM 5 2892 4.79 7.11 7.31 04:00:01 PM 5 2918 5.25 4.57 5.69 04:10:01 PM 7 2908 2.78 3.22 4.45 04:20:01 PM 12 3112 5.08 3.75 4.04 04:30:01 PM 6 2942 4.79 5.35 5.00 04:40:01 PM 5 2925 3.79 4.87 4.97 04:50:01 PM 4 2932 4.34 3.92 4.34 05:00:01 PM 4 2922 3.90 3.79 4.04 05:10:01 PM 3 2895 3.75 4.09 4.16 05:20:01 PM 6 2931 4.24 4.01 4.14 05:30:01 PM 2 2916 4.37 4.61 4.35 05:40:01 PM 6 2881 3.93 4.39 4.42 05:50:01 PM 5 2883 4.09 3.96 4.19 06:00:01 PM 13 2892 11.74 7.23 5.29 06:10:01 PM 19 2853 10.80 10.65 8.20 06:20:01 PM 15 2874 10.95 11.12 9.63 06:20:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 06:30:01 PM 13 2871 10.99 10.94 10.22 06:40:01 PM 12 2855 10.28 10.61 10.41 06:50:01 PM 11 2849 9.75 9.74 10.02 07:00:01 PM 7 2874 9.23 9.51 9.72 07:10:01 PM 14 2847 10.15 11.03 10.35 07:20:01 PM 11 2864 9.39 9.53 9.96 07:30:01 PM 17 2875 9.95 10.09 10.05 07:40:01 PM 11 2840 10.25 10.28 10.19 07:50:01 PM 8 2857 9.11 9.54 9.81 08:00:01 PM 11 2861 8.58 8.80 9.25 08:10:01 PM 10 2837 9.63 9.17 9.16 08:20:01 PM 11 2841 8.53 8.72 8.94 08:30:01 PM 14 2857 11.03 10.04 9.48 08:40:01 PM 21 2849 10.71 10.36 9.91 08:50:01 PM 14 2844 10.65 10.69 10.29 09:00:01 PM 11 2852 10.50 10.91 10.65 09:10:01 PM 14 2852 9.61 10.15 10.38 09:20:01 PM 11 2832 10.72 10.39 10.43 09:30:01 PM 16 2850 10.81 10.63 10.50 09:40:01 PM 9 2831 10.17 10.42 10.50 09:50:01 PM 11 2839 9.77 10.06 10.30 10:00:01 PM 10 2847 9.17 9.56 9.91 10:00:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 10:10:01 PM 12 2835 9.96 10.54 10.34 10:20:01 PM 12 2949 9.01 9.21 9.74 10:30:01 PM 11 2829 9.99 9.79 9.73 10:40:01 PM 10 2845 9.13 9.29 9.52 10:50:01 PM 8 2825 9.54 9.11 9.27 11:00:01 PM 11 2852 8.87 9.09 9.18 11:10:01 PM 12 2849 9.19 9.28 9.23 11:20:01 PM 12 2828 10.28 9.62 9.38 11:30:01 PM 10 2840 11.23 10.27 9.75 11:40:01 PM 8 2855 10.60 10.82 10.29 11:50:01 PM 9 2844 9.62 9.88 10.03 Average: 7 2859 4.92 5.38 5.71 alert_log: IPC0 (ospid: 368909) has not called a wait for 63 secs. 2019-11-15T14:51:16.213026+04:00 IPC0 (ospid: 368909) is hung in an acceptable location ( 0x2.ffff). 2019-11-15T14:51:21.673764+04:00 IPC0 (ospid: 368909) has not called a wait for 78 secs. 2019-11-15T14:51:25.005513+04:00 CLMN: clean deferred state objects - failed 2019-11-15T14:51:46.745792+04:00 Errors in file /u01/app/oracle/diag/rdbms/cdbiamas/cdbiamas1/trace/cdbiamas1_lmhb_368993.trc (incident=6943370) (PDBNAME=CDB$ROOT): ORA-29770: global enqueue process IPC0 (OSID 368909) is hung for more than 70 seconds Incident details in: /u01/app/oracle/diag/rdbms/cdbiamas/cdbiamas1/incident/incdir_6943370/cdbiamas1_lmhb_368993_i6943370.trc 2019-11-15T14:51:57.035439+04:00 LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_CDBIAMAS by memno 1 2019-11-15T14:52:02.340709+04:00 ERROR: Some process(s) is not making progress. LMHB (ospid: 368993) is terminating the instance. Please check LMHB trace file for more details. Please also check the CPU load, I/O load and other system properties for anomalous behavior ERROR: Some process(s) is not making progress. LMHB (ospid: 368993): terminating the instance due to error 29770 2019-11-15T14:52:02.377773+04:00 IAMAS(3):opiodr aborting process unknown ospid (319293) as a result of ORA-1092 2019-11-15T14:52:02.395886+04:00 IAMAS(3):opiodr aborting process unknown ospid (345311) as a result of ORA-1092 2019-11-15T14:52:02.460201+04:00 System state dump requested by (instance=1, osid=368993 (LMHB)), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/cdbiamas/cdbiamas1/trace/cdbiamas1_diag_368925_20191115145202.trc 2019-11-15T14:52:04.698668+04:00 ORA-1092 : opitsk aborting process 2019-11-15T14:52:04.778808+04:00 ORA-1092 : opitsk aborting process 2019-11-15T14:52:05.774957+04:00 ORA-1092 : opitsk aborting process 2019-11-15T14:52:06.827478+04:00 ORA-1092 : opitsk aborting process 2019-11-15T14:52:16.532653+04:00 Instance termination failed to kill one or more processes 2019-11-15T14:52:25.652247+04:00 Instance termination failed to kill one or more processes 2019-11-15T14:52:25.666462+04:00 Instance terminated by LMHB, pid = 368993 2019-11-15T14:52:25.667823+04:00 Warning: 686 processes are still attach to shmid 130482192: (size: 2097152 bytes, creator pid: 368747, last attach/detach pid: 366515) 2019-11-15T14:52:45.976442+04:00 Starting ORACLE instance (normal) (OS id: 367022) 2019-11-15T14:52:45.986046+04:00 CLI notifier numLatches:131 maxDescs:12777 2019-11-15T14:52:45.988462+04:00 ERROR: Failed to get available system pages to allocate memory 2019-11-15T14:52:45.988532+04:00 ********************************************************************** 2019-11-15T14:52:45.988601+04:00 Dump of system resources acquired for SHARED GLOBAL AREA (SGA) 2019-11-15T14:52:45.988709+04:00 Per process system memlock (soft) limit = UNLIMITED 2019-11-15T14:52:45.988773+04:00 Expected per process system memlock (soft) limit to lock SHARED GLOBAL AREA (SGA) into memory: 630G 2019-11-15T14:52:45.988894+04:00 Available system pagesizes: 4K, 2048K 2019-11-15T14:52:45.989020+04:00 Supported system pagesize(s): 2019-11-15T14:52:45.989082+04:00 PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s) 2019-11-15T14:52:45.989197+04:00 2048K 311696 322453 311567 NONE 2019-11-15T14:52:45.989260+04:00 Reason for not supporting certain system pagesizes: 2019-11-15T14:52:45.989323+04:00 4K - Large pagesizes only 2019-11-15T14:52:45.989385+04:00 RECOMMENDATION: 2019-11-15T14:52:45.989448+04:00 1. Configure system with expected number of pages for every supported system pagesize prior to the next instance restart operation. 2019-11-15T14:52:45.999872+04:00 ********************************************************************** 2019-11-15T15:25:54.865760+04:00 Starting ORACLE instance (normal) (OS id: 89404) 2019-11-15T15:25:54.875488+04:00 CLI notifier numLatches:131 maxDescs:12777 2019-11-15T15:25:54.878652+04:00 ********************************************************************** 2019-11-15T15:25:54.878776+04:00 Dump of system resources acquired for SHARED GLOBAL AREA (SGA) 2019-11-15T15:25:54.878946+04:00 Per process system memlock (soft) limit = UNLIMITED 2019-11-15T15:25:54.879048+04:00 Expected per process system memlock (soft) limit to lock SHARED GLOBAL AREA (SGA) into memory: 630G 2019-11-15T15:25:54.879242+04:00 Available system pagesizes: 4K, 2048K 2019-11-15T15:25:54.879445+04:00 Supported system pagesize(s): 2019-11-15T15:25:54.879544+04:00 PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s) 2019-11-15T15:25:54.879714+04:00 2048K 326147 322453 322453 NONE 2019-11-15T15:25:54.879815+04:00 Reason for not supporting certain system pagesizes: 2019-11-15T15:25:54.879915+04:00 4K - Large pagesizes only 2019-11-15T15:25:54.880014+04:00 ********************************************************************** SGA Local memory support enabled LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Initial number of CPU is 96 i checked /var/log/messages log and there isn't any information error occured time(15.11.2019 14:52), so i checked zabbix log and there isn't any problem about this problem: Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:53:01 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:56 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:56 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:51 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:51 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:46 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:46 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:45 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:45 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:45 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:45 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:45 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:45 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:41 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:41 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:36 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:36 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:31 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:31 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:26 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:26 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:21 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:21 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:16 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:16 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:11 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:11 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:09 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:06 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:05 PM Stored PAM Session Closed exa-hddbadm01 11 Nov 15, 2019, 2:52:01 PM Auth Server Session Closed PAM Su User Impersonation exa-hddbadm01 11 Nov 15, 2019, 2:52:00 PM Privilege Access Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:00 PM Stored Linux login messages Message exa-hddbadm01 1 Nov 15, 2019, 2:52:00 PM Stored Opatch output: [oracle@exa-hddbadm01 OPatch]$ ./opatch lsinventory -detail Oracle Interim Patch Installer version 12.2.0.1.13 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.2.0.1/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.2.0.1/dbhome_1/oraInst.loc OPatch version : 12.2.0.1.13 OUI version : 12.2.0.1.4 Log file location : /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatch/opatch2019-11-20_11-13-01AM_1.log Lsinventory Output file location : /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2019-11-20_11-13-01AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: exa-hddbadm01.core.iamas.gov.az ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Database 12c 12.2.0.1.0 There are 1 products installed in this Oracle Home. Installed Products (128): Assistant Common Files 12.2.0.1.0 BLASLAPACK Component 12.2.0.1.0 Buildtools Common Files 12.2.0.1.0 Cluster Verification Utility Common Files 12.2.0.1.0 Database Configuration and Upgrade Assistants 12.2.0.1.0 Database Migration Assistant for Unicode 12.2.0.1.0 Database SQL Scripts 12.2.0.1.0 Database Workspace Manager 12.2.0.1.0 DB TOOLS Listener 12.2.0.1.0 Deinstallation Tool 12.2.0.1.0 Enterprise Edition Options 12.2.0.1.0 Expat libraries 2.0.1.0.3 Generic Connectivity Common Files 12.2.0.1.0 Hadoopcore Component 12.2.0.1.0 HAS Common Files 12.2.0.1.0 HAS Files for DB 12.2.0.1.0 Installation Common Files 12.2.0.1.0 Installation Plugin Files 12.2.0.1.0 Installer SDK Component 12.2.0.1.4 JAccelerator (COMPANION) 12.2.0.1.0 Java Development Kit 1.8.0.91.0 LDAP Required Support Files 12.2.0.1.0 OLAP SQL Scripts 12.2.0.1.0 Oracle Advanced Analytics 12.2.0.1.0 Oracle Advanced Security 12.2.0.1.0 Oracle Application Express 12.2.0.1.0 Oracle Bali Share 11.1.1.6.0 Oracle Call Interface (OCI) 12.2.0.1.0 Oracle Clusterware RDBMS Files 12.2.0.1.0 Oracle Configuration Manager 12.1.2.0.0 Oracle Configuration Manager Client 10.3.2.1.0 Oracle Configuration Manager Deconfiguration 10.3.1.0.0 Oracle Context Companion 12.2.0.1.0 Oracle Core Required Support Files 12.2.0.1.0 Oracle Core Required Support Files for Core DB 12.2.0.1.0 Oracle Database 12c 12.2.0.1.0 Oracle Database 12c 12.2.0.1.0 Oracle Database 12c Multimedia Files 12.2.0.1.0 Oracle Database Deconfiguration 12.2.0.1.0 Oracle Database Gateway for ODBC 12.2.0.1.0 Oracle Database Utilities 12.2.0.1.0 Oracle Database Vault option 12.2.0.1.0 Oracle DBCA Deconfiguration 12.2.0.1.0 Oracle Extended Windowing Toolkit 11.1.1.6.0 Oracle Globalization Support 12.2.0.1.0 Oracle Globalization Support 12.2.0.1.0 Oracle Globalization Support For Core 12.2.0.1.0 Oracle Help for Java 11.1.1.7.0 Oracle Help Share Library 11.1.1.7.0 Oracle Ice Browser 11.1.1.7.0 Oracle Internet Directory Client 12.2.0.1.0 Oracle Java Client 12.2.0.1.0 Oracle JDBC Server Support Package 12.2.0.1.0 Oracle JDBC/OCI Instant Client 12.2.0.1.0 Oracle JDBC/THIN Interfaces 12.2.0.1.0 Oracle JFC Extended Windowing Toolkit 11.1.1.6.0 Oracle JVM 12.2.0.1.0 Oracle JVM For Core 12.2.0.1.0 Oracle Label Security 12.2.0.1.0 Oracle LDAP administration 12.2.0.1.0 Oracle Locale Builder 12.2.0.1.0 Oracle Message Gateway Common Files 12.2.0.1.0 Oracle Multimedia 12.2.0.1.0 Oracle Multimedia Client Option 12.2.0.1.0 Oracle Multimedia Java Advanced Imaging 12.2.0.1.0 Oracle Multimedia Locator 12.2.0.1.0 Oracle Multimedia Locator Java Required Support Files 12.2.0.1.0 Oracle Multimedia Locator RDBMS Files 12.2.0.1.0 Oracle Net 12.2.0.1.0 Oracle Net Listener 12.2.0.1.0 Oracle Net Required Support Files 12.2.0.1.0 Oracle Net Services 12.2.0.1.0 Oracle Netca Client 12.2.0.1.0 Oracle Notification Service 12.2.0.1.0 Oracle Notification Service for Instant Client 12.2.0.1.0 Oracle ODBC Driver 12.2.0.1.0 Oracle ODBC Driverfor Instant Client 12.2.0.1.0 Oracle OLAP 12.2.0.1.0 Oracle OLAP API 12.2.0.1.0 Oracle OLAP RDBMS Files 12.2.0.1.0 Oracle One-Off Patch Installer 12.2.0.1.6 Oracle Partitioning 12.2.0.1.0 Oracle Programmer 12.2.0.1.0 Oracle R Enterprise Server Files 12.2.0.1.0 Oracle RAC Deconfiguration 12.2.0.1.0 Oracle RAC Required Support Files-HAS 12.2.0.1.0 Oracle Real Application Testing 12.2.0.1.0 Oracle Recovery Manager 12.2.0.1.0 Oracle Scheduler Agent 12.2.0.1.0 Oracle Security Developer Tools 12.2.0.1.0 Oracle Spatial and Graph 12.2.0.1.0 Oracle SQL Developer 12.2.0.1.0 Oracle Starter Database 12.2.0.1.0 Oracle Text 12.2.0.1.0 Oracle Text ATG Language Support Files 12.2.0.1.0 Oracle Text Required Support Files 12.2.0.1.0 Oracle Universal Connection Pool 12.2.0.1.0 Oracle Universal Installer 12.2.0.1.4 Oracle USM Deconfiguration 12.2.0.1.0 Oracle Wallet Manager 12.2.0.1.0 Oracle XML Development Kit 12.2.0.1.0 Oracle XML Query 12.2.0.1.0 oracle.swd.commonlogging 13.3.0.0.0 oracle.swd.opatchautodb 12.2.0.1.5 oracle.swd.oui.core.min 12.2.0.1.4 Parser Generator Required Support Files 12.2.0.1.0 Perl Interpreter 5.22.0.0.0 Perl Modules 5.22.0.0.0 PL/SQL 12.2.0.1.0 PL/SQL Embedded Gateway 12.2.0.1.0 Platform Required Support Files 12.2.0.1.0 Precompiler Common Files 12.2.0.1.0 Precompiler Common Files for Core 12.2.0.1.0 Precompiler Required Support Files 12.2.0.1.0 Precompilers 12.2.0.1.0 RDBMS Required Support Files 12.2.0.1.0 RDBMS Required Support Files for Instant Client 12.2.0.1.0 Required Support Files 12.2.0.1.0 Secure Socket Layer 12.2.0.1.0 SQL*Plus 12.2.0.1.0 SQL*Plus Files for Instant Client 12.2.0.1.0 SQL*Plus Required Support Files 12.2.0.1.0 SQLJ Runtime 12.2.0.1.0 SSL Required Support Files for InstantClient 12.2.0.1.0 Tracle File Analyzer 12.2.0.1.0 XDK Required Support Files 12.2.0.1.0 XML Parser for Java 12.2.0.1.0 XML Parser for Oracle JVM 12.2.0.1.0 There are 128 products installed in this Oracle Home. Interim patches (4) : Patch 30404909 : applied on Tue Oct 22 00:05:43 AZST 2019 Unique Patch ID: 23168510 Created on 17 Oct 2019, 22:04:51 hrs PST8PDT Bugs fixed: 30404909 This patch overlays patches: 28822515 This patch needs patches: 28822515 as prerequisites Files Touched: /ktma.o --> ORACLE_HOME/lib/libserver12.a ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ioracle Patch Location in Inventory: /u01/app/oracle/product/12.2.0.1/dbhome_1/inventory/oneoffs/30404909 Patch Location in Storage area: /u01/app/oracle/product/12.2.0.1/dbhome_1/.patch_storage/30404909_Oct_17_2019_22_04_51 Patch 28790651 : applied on Wed Feb 20 23:51:13 AZT 2019 Unique Patch ID: 22592706 Patch description: "OJVM RELEASE UPDATE: 12.2.0.1.190115 (28790651)" Created on 26 Nov 2018, 04:56:04 hrs PST8PDT Bugs fixed: 25811105, 25890046, 26023042, 26570134, 27000702, 27461740, 27952586 27642235, 27539876, 28502098, 28915870 Files Touched: java.security --> ORACLE_HOME/javavm/jdk/jdk8/lib/security/java.security cacerts --> ORACLE_HOME/javavm/jdk/jdk8/lib/security/cacerts sunjce_provider.jar --> ORACLE_HOME/javavm/jdk/jdk8/lib/sunjce_provider.jar jce.jar --> ORACLE_HOME/javavm/jdk/jdk8/lib/jce.jar classes.bin --> ORACLE_HOME/javavm/jdk/jdk8/admin/classes.bin lfclasses.bin --> ORACLE_HOME/javavm/jdk/jdk8/admin/lfclasses.bin jvmpsupii.sql --> ORACLE_HOME/javavm/install/jvmpsupii.sql jvmpsupi.sql --> ORACLE_HOME/javavm/install/jvmpsupi.sql jvmpsu.sql --> ORACLE_HOME/javavm/install/jvmpsu.sql jvmpsui.sql --> ORACLE_HOME/javavm/install/jvmpsui.sql jvmpsupdi.sql --> ORACLE_HOME/javavm/install/jvmpsupdi.sql jvmpsupdii.sql --> ORACLE_HOME/javavm/install/jvmpsupdii.sql libjavavm12.a --> ORACLE_HOME/javavm/jdk/jdk8/lib/libjavavm12.a 28790651_apply.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/28790651_apply.sql 28790651_rollback.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/28790651_rollback.sql jvmpsupdi.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/rollback_files/javavm/install/jvmpsupdi.sql jvmpsupdii.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/rollback_files/javavm/install/jvmpsupdii.sql jvmpsupii.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/rollback_files/javavm/install/jvmpsupii.sql jvmpsu.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/rollback_files/javavm/install/jvmpsu.sql jvmpsui.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/rollback_files/javavm/install/jvmpsui.sql jvmpsupi.sql --> ORACLE_HOME/sqlpatch/28790651/22592706/rollback_files/javavm/install/jvmpsupi.sql aurora.zip --> ORACLE_HOME/javavm/lib/aurora.zip jox.o --> ORACLE_HOME/rdbms/lib/jox.o ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/jox_on ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ioracle Patch Location in Inventory: /u01/app/oracle/product/12.2.0.1/dbhome_1/inventory/oneoffs/28790651 Patch Location in Storage area: /u01/app/oracle/product/12.2.0.1/dbhome_1/.patch_storage/28790651_Nov_26_2018_04_56_04 Patch 28870605 : applied on Wed Feb 20 23:15:53 AZT 2019 Unique Patch ID: 22543245 Patch description: "OCW JAN 2019 RELEASE UPDATE 12.2.0.1.190115 (28870605)" Created on 7 Jan 2019, 21:43:36 hrs PST8PDT Bugs fixed: 12816839, 13250991, 18701017, 20559126, 21477269, 21679331, 22322021 22734786, 22986384, 22999793, 23170674, 23297046, 23301767, 23340259 23346234, 23722215, 23760401, 23762756, 24330981, 24372941, 24396050 24403376, 24411578, 24422155, 24423011, 24445255, 24451580, 24462646 24464953, 24509984, 24546078, 24578464, 24583356, 24584419, 24600431 24657753, 24663993, 24664849, 24665035, 24670214, 24673586, 24696235 24732650, 24736511, 24792446, 24796183, 24797972, 24801915, 24808260 24820676, 24824518, 24831158, 24831217, 24902553, 24908629, 24932026 24953946, 25025157, 25026470, 25037011, 25037836, 25049033, 25066549 25089164, 25100966, 25131365, 25137752, 25164540, 25197395, 25197670 25216003, 25233797, 25245759, 25251382, 25269635, 25291358, 25301463 25304126, 25307145, 25325522, 25327670, 25329178, 25354530, 25360754 25361169, 25369170, 25371632, 25385515, 25397620, 25408895, 25409838 25448462, 25448476, 25485737, 25492219, 25493588, 25496699, 25499276 25505841, 25509979, 25518447, 25537905, 25540583, 25541343, 25556203 25564644, 25569634, 25576767, 25591658, 25633406, 25644425, 25646592 25655495, 25662028, 25667162, 25685286, 25697364, 25698636, 25715140 25715179, 25716769, 25717212, 25728787, 25729214, 25742471, 25742704 25774482, 25776294, 25784424, 25785073, 25789051, 25790699, 25795447 25801922, 25803563, 25810476, 25825732, 25825755, 25832312, 25832375 25851874, 25856024, 25865563, 25885834, 25890662, 25893570, 25915863 25927065, 25943805, 25947195, 25948946, 25951778, 25956724, 25987595 25996923, 25998093, 26002739, 26035888, 26052877, 26088770, 26111524 26115835, 26116932, 26121788, 26144044, 26185271, 26185431, 26198627 26230199, 26238801, 26248428, 26257547, 26264417, 26282865, 26301456 26307166, 26308703, 26313403, 26316238, 26326556, 26357098, 26372145 26379711, 26387283, 26391418, 26418606, 26419112, 26431982, 26433972 26433979, 26439462, 26439893, 26473232, 26474627, 26477705, 26493466 26494006, 26494536, 26527054, 26529142, 26546632, 26547521, 26552807 26567325, 26581118, 26586174, 26587652, 26588288, 26593581, 26624184 26633228, 26654615, 26664476, 26709107, 26710631, 26716037, 26747738 26754328, 26755613, 26791882, 26817081, 26817275, 26817995, 26818064 26818368, 26822082, 26827699, 26860285, 26875825, 26877592, 26884984 26943660, 26950548, 26958202, 26973611, 26979409, 26996813, 27006180 27012234, 27018734, 27023569, 27040560, 27047274, 27068526, 27070729 27080761, 27092991, 27094352, 27106915, 27114112, 27119820, 27130514 27133637, 27153755, 27154002, 27174938, 27182006, 27186837, 27190262 27213140, 27222423, 27222938, 27249544, 27252023, 27267992, 27274143 27297582, 27300007, 27309182, 27358232, 27370933, 27377219, 27378959 27379846, 27399762, 27403244, 27404599, 27426277, 27440094, 27475272 27507494, 27516078, 27526362, 27532009, 27534289, 27536738, 27542831 27554103, 27554304, 27555886, 27566153, 27577122, 27581484, 27593587 27595801, 27600706, 27624923, 27628221, 27664844, 27668379, 27682288 27691717, 27702244, 27703242, 27708711, 27712736, 27735534, 27739957 27747407, 27753602, 27757979, 27766679, 27769638, 27786669, 27811439 27850736, 27862636, 27899060, 27953102, 27970426, 27972271, 27984314 28030440, 28032758, 28050201, 28071462, 28084656, 28097951, 28105136 28127661, 28146584, 28161727, 28266751, 28285234, 28375138, 28383213 28393555, 28462215, 28477613, 28504730, 28535778, 28558622, 28578962 28593407, 28887933 Files Touched: srvm.jar --> ORACLE_HOME/oui/jlib/srvm.jar srvmasm.jar --> ORACLE_HOME/oui/jlib/srvmasm.jar libsrvm12.so --> ORACLE_HOME/oui/lib/linux/libsrvm12.so s_crsconfig_defs --> ORACLE_HOME/crs/install/s_crsconfig_defs clsrwrap.sbs --> ORACLE_HOME/crs/sbs/clsrwrap.sbs diagcollection.sh.sbs --> ORACLE_HOME/crs/sbs/diagcollection.sh.sbs localconfig.sbs --> ORACLE_HOME/crs/sbs/localconfig.sbs srvctl.sbs --> ORACLE_HOME/crs/sbs/srvctl.sbs catus.msg --> ORACLE_HOME/has/mesg/catus.msg clsdar.msb --> ORACLE_HOME/has/mesg/clsdar.msb clsdcs.msb --> ORACLE_HOME/has/mesg/clsdcs.msb clsdd.msb --> ORACLE_HOME/has/mesg/clsdd.msb clsddk.msb --> ORACLE_HOME/has/mesg/clsddk.msb clsde.msb --> ORACLE_HOME/has/mesg/clsde.msb clsdel.msb --> ORACLE_HOME/has/mesg/clsdel.msb clsdf.msb --> ORACLE_HOME/has/mesg/clsdf.msb clsdhu.msb --> ORACLE_HOME/has/mesg/clsdhu.msb clsdi.msb --> ORACLE_HOME/has/mesg/clsdi.msb clsdiw.msb --> ORACLE_HOME/has/mesg/clsdiw.msb clsdja.msb --> ORACLE_HOME/has/mesg/clsdja.msb clsdko.msb --> ORACLE_HOME/has/mesg/clsdko.msb clsdn.msb --> ORACLE_HOME/has/mesg/clsdn.msb clsdnl.msb --> ORACLE_HOME/has/mesg/clsdnl.msb clsdpl.msb --> ORACLE_HOME/has/mesg/clsdpl.msb clsdpt.msb --> ORACLE_HOME/has/mesg/clsdpt.msb clsdptb.msb --> ORACLE_HOME/has/mesg/clsdptb.msb clsdro.msb --> ORACLE_HOME/has/mesg/clsdro.msb clsdru.msb --> ORACLE_HOME/has/mesg/clsdru.msb clsds.msb --> ORACLE_HOME/has/mesg/clsds.msb clsdsf.msb --> ORACLE_HOME/has/mesg/clsdsf.msb clsdsk.msb --> ORACLE_HOME/has/mesg/clsdsk.msb clsdth.msb --> ORACLE_HOME/has/mesg/clsdth.msb clsdtr.msb --> ORACLE_HOME/has/mesg/clsdtr.msb clsdus.msb --> ORACLE_HOME/has/mesg/clsdus.msb clsdus.msg --> ORACLE_HOME/has/mesg/clsdus.msg clsdzhs.msb --> ORACLE_HOME/has/mesg/clsdzhs.msb clsdzht.msb --> ORACLE_HOME/has/mesg/clsdzht.msb clsgnar.msb --> ORACLE_HOME/has/mesg/clsgnar.msb clsgnca.msb --> ORACLE_HOME/has/mesg/clsgnca.msb clsgncs.msb --> ORACLE_HOME/has/mesg/clsgncs.msb clsgnd.msb --> ORACLE_HOME/has/mesg/clsgnd.msb clsgndk.msb --> ORACLE_HOME/has/mesg/clsgndk.msb clsgne.msb --> ORACLE_HOME/has/mesg/clsgne.msb clsgnel.msb --> ORACLE_HOME/has/mesg/clsgnel.msb clsgnesa.msb --> ORACLE_HOME/has/mesg/clsgnesa.msb clsgnf.msb --> ORACLE_HOME/has/mesg/clsgnf.msb clsgnhu.msb --> ORACLE_HOME/has/mesg/clsgnhu.msb clsgni.msb --> ORACLE_HOME/has/mesg/clsgni.msb clsgniw.msb --> ORACLE_HOME/has/mesg/clsgniw.msb clsgnja.msb --> ORACLE_HOME/has/mesg/clsgnja.msb clsgnko.msb --> ORACLE_HOME/has/mesg/clsgnko.msb clsgnn.msb --> ORACLE_HOME/has/mesg/clsgnn.msb clsgnnl.msb --> ORACLE_HOME/has/mesg/clsgnnl.msb clsgnpl.msb --> ORACLE_HOME/has/mesg/clsgnpl.msb clsgnpt.msb --> ORACLE_HOME/has/mesg/clsgnpt.msb clsgnptb.msb --> ORACLE_HOME/has/mesg/clsgnptb.msb clsgnro.msb --> ORACLE_HOME/has/mesg/clsgnro.msb clsgnru.msb --> ORACLE_HOME/has/mesg/clsgnru.msb clsgns.msb --> ORACLE_HOME/has/mesg/clsgns.msb clsgnsf.msb --> ORACLE_HOME/has/mesg/clsgnsf.msb clsgnsk.msb --> ORACLE_HOME/has/mesg/clsgnsk.msb clsgnth.msb --> ORACLE_HOME/has/mesg/clsgnth.msb clsgntr.msb --> ORACLE_HOME/has/mesg/clsgntr.msb clsgnus.msb --> ORACLE_HOME/has/mesg/clsgnus.msb clsgnus.msg --> ORACLE_HOME/has/mesg/clsgnus.msg clsgnzhs.msb --> ORACLE_HOME/has/mesg/clsgnzhs.msb clsgnzht.msb --> ORACLE_HOME/has/mesg/clsgnzht.msb clsnsar.msb --> ORACLE_HOME/has/mesg/clsnsar.msb clsnscs.msb --> ORACLE_HOME/has/mesg/clsnscs.msb clsnsd.msb --> ORACLE_HOME/has/mesg/clsnsd.msb clsnsdk.msb --> ORACLE_HOME/has/mesg/clsnsdk.msb clsnse.msb --> ORACLE_HOME/has/mesg/clsnse.msb clsnsel.msb --> ORACLE_HOME/has/mesg/clsnsel.msb clsnsf.msb --> ORACLE_HOME/has/mesg/clsnsf.msb clsnshu.msb --> ORACLE_HOME/has/mesg/clsnshu.msb clsnsi.msb --> ORACLE_HOME/has/mesg/clsnsi.msb clsnsiw.msb --> ORACLE_HOME/has/mesg/clsnsiw.msb clsnsja.msb --> ORACLE_HOME/has/mesg/clsnsja.msb clsnsko.msb --> ORACLE_HOME/has/mesg/clsnsko.msb clsnsn.msb --> ORACLE_HOME/has/mesg/clsnsn.msb clsnsnl.msb --> ORACLE_HOME/has/mesg/clsnsnl.msb clsnspl.msb --> ORACLE_HOME/has/mesg/clsnspl.msb clsnspt.msb --> ORACLE_HOME/has/mesg/clsnspt.msb clsnsptb.msb --> ORACLE_HOME/has/mesg/clsnsptb.msb clsnsro.msb --> ORACLE_HOME/has/mesg/clsnsro.msb clsnsru.msb --> ORACLE_HOME/has/mesg/clsnsru.msb clsnss.msb --> ORACLE_HOME/has/mesg/clsnss.msb clsnssf.msb --> ORACLE_HOME/has/mesg/clsnssf.msb clsnssk.msb --> ORACLE_HOME/has/mesg/clsnssk.msb clsnsth.msb --> ORACLE_HOME/has/mesg/clsnsth.msb clsnstr.msb --> ORACLE_HOME/has/mesg/clsnstr.msb clsnsus.msb --> ORACLE_HOME/has/mesg/clsnsus.msb clsnsus.msg --> ORACLE_HOME/has/mesg/clsnsus.msg clsnszhs.msb --> ORACLE_HOME/has/mesg/clsnszhs.msb clsnszht.msb --> ORACLE_HOME/has/mesg/clsnszht.msb clstar.msb --> ORACLE_HOME/has/mesg/clstar.msb clstcs.msb --> ORACLE_HOME/has/mesg/clstcs.msb clstd.msb --> ORACLE_HOME/has/mesg/clstd.msb clstdk.msb --> ORACLE_HOME/has/mesg/clstdk.msb clste.msb --> ORACLE_HOME/has/mesg/clste.msb clstel.msb --> ORACLE_HOME/has/mesg/clstel.msb clstf.msb --> ORACLE_HOME/has/mesg/clstf.msb clsthu.msb --> ORACLE_HOME/has/mesg/clsthu.msb clsti.msb --> ORACLE_HOME/has/mesg/clsti.msb clstiw.msb --> ORACLE_HOME/has/mesg/clstiw.msb clstja.msb --> ORACLE_HOME/has/mesg/clstja.msb clstko.msb --> ORACLE_HOME/has/mesg/clstko.msb clstn.msb --> ORACLE_HOME/has/mesg/clstn.msb clstnl.msb --> ORACLE_HOME/has/mesg/clstnl.msb clstpl.msb --> ORACLE_HOME/has/mesg/clstpl.msb clstpt.msb --> ORACLE_HOME/has/mesg/clstpt.msb clstptb.msb --> ORACLE_HOME/has/mesg/clstptb.msb clstro.msb --> ORACLE_HOME/has/mesg/clstro.msb clstru.msb --> ORACLE_HOME/has/mesg/clstru.msb clsts.msb --> ORACLE_HOME/has/mesg/clsts.msb clstsf.msb --> ORACLE_HOME/has/mesg/clstsf.msb clstsk.msb --> ORACLE_HOME/has/mesg/clstsk.msb clstth.msb --> ORACLE_HOME/has/mesg/clstth.msb clsttr.msb --> ORACLE_HOME/has/mesg/clsttr.msb clstus.msb --> ORACLE_HOME/has/mesg/clstus.msb clstus.msg --> ORACLE_HOME/has/mesg/clstus.msg clstzhs.msb --> ORACLE_HOME/has/mesg/clstzhs.msb clstzht.msb --> ORACLE_HOME/has/mesg/clstzht.msb clsuar.msb --> ORACLE_HOME/has/mesg/clsuar.msb clsuca.msb --> ORACLE_HOME/has/mesg/clsuca.msb clsucs.msb --> ORACLE_HOME/has/mesg/clsucs.msb clsud.msb --> ORACLE_HOME/has/mesg/clsud.msb clsudk.msb --> ORACLE_HOME/has/mesg/clsudk.msb clsue.msb --> ORACLE_HOME/has/mesg/clsue.msb clsuel.msb --> ORACLE_HOME/has/mesg/clsuel.msb clsuf.msb --> ORACLE_HOME/has/mesg/clsuf.msb clsuhu.msb --> ORACLE_HOME/has/mesg/clsuhu.msb clsui.msb --> ORACLE_HOME/has/mesg/clsui.msb clsuiw.msb --> ORACLE_HOME/has/mesg/clsuiw.msb clsuja.msb --> ORACLE_HOME/has/mesg/clsuja.msb clsuko.msb --> ORACLE_HOME/has/mesg/clsuko.msb clsun.msb --> ORACLE_HOME/has/mesg/clsun.msb clsunl.msb --> ORACLE_HOME/has/mesg/clsunl.msb clsupl.msb --> ORACLE_HOME/has/mesg/clsupl.msb clsupt.msb --> ORACLE_HOME/has/mesg/clsupt.msb clsuptb.msb --> ORACLE_HOME/has/mesg/clsuptb.msb clsuro.msb --> ORACLE_HOME/has/mesg/clsuro.msb clsuru.msb --> ORACLE_HOME/has/mesg/clsuru.msb clsusf.msb --> ORACLE_HOME/has/mesg/clsusf.msb clsusk.msb --> ORACLE_HOME/has/mesg/clsusk.msb clsuth.msb --> ORACLE_HOME/has/mesg/clsuth.msb clsutr.msb --> ORACLE_HOME/has/mesg/clsutr.msb clsuus.msb --> ORACLE_HOME/has/mesg/clsuus.msb clsuus.msg --> ORACLE_HOME/has/mesg/clsuus.msg clsuzhs.msb --> ORACLE_HOME/has/mesg/clsuzhs.msb clsuzht.msb --> ORACLE_HOME/has/mesg/clsuzht.msb clswar.msb --> ORACLE_HOME/has/mesg/clswar.msb clswcs.msb --> ORACLE_HOME/has/mesg/clswcs.msb clswd.msb --> ORACLE_HOME/has/mesg/clswd.msb clswdk.msb --> ORACLE_HOME/has/mesg/clswdk.msb clswe.msb --> ORACLE_HOME/has/mesg/clswe.msb clswel.msb --> ORACLE_HOME/has/mesg/clswel.msb clswf.msb --> ORACLE_HOME/has/mesg/clswf.msb clswhu.msb --> ORACLE_HOME/has/mesg/clswhu.msb clswi.msb --> ORACLE_HOME/has/mesg/clswi.msb clswiw.msb --> ORACLE_HOME/has/mesg/clswiw.msb clswja.msb --> ORACLE_HOME/has/mesg/clswja.msb clswko.msb --> ORACLE_HOME/has/mesg/clswko.msb clswn.msb --> ORACLE_HOME/has/mesg/clswn.msb clswnl.msb --> ORACLE_HOME/has/mesg/clswnl.msb clswpl.msb --> ORACLE_HOME/has/mesg/clswpl.msb clswpt.msb --> ORACLE_HOME/has/mesg/clswpt.msb clswptb.msb --> ORACLE_HOME/has/mesg/clswptb.msb clswro.msb --> ORACLE_HOME/has/mesg/clswro.msb clswru.msb --> ORACLE_HOME/has/mesg/clswru.msb clsws.msb --> ORACLE_HOME/has/mesg/clsws.msb clswsf.msb --> ORACLE_HOME/has/mesg/clswsf.msb clswsk.msb --> ORACLE_HOME/has/mesg/clswsk.msb clswth.msb --> ORACLE_HOME/has/mesg/clswth.msb clswtr.msb --> ORACLE_HOME/has/mesg/clswtr.msb clswus.msb --> ORACLE_HOME/has/mesg/clswus.msb clswus.msg --> ORACLE_HOME/has/mesg/clswus.msg clswzhs.msb --> ORACLE_HOME/has/mesg/clswzhs.msb clswzht.msb --> ORACLE_HOME/has/mesg/clswzht.msb cskmus.msg --> ORACLE_HOME/has/mesg/cskmus.msg gipcar.msb --> ORACLE_HOME/has/mesg/gipcar.msb gipcca.msb --> ORACLE_HOME/has/mesg/gipcca.msb gipccs.msb --> ORACLE_HOME/has/mesg/gipccs.msb gipcd.msb --> ORACLE_HOME/has/mesg/gipcd.msb gipcdk.msb --> ORACLE_HOME/has/mesg/gipcdk.msb gipce.msb --> ORACLE_HOME/has/mesg/gipce.msb gipcel.msb --> ORACLE_HOME/has/mesg/gipcel.msb gipcf.msb --> ORACLE_HOME/has/mesg/gipcf.msb gipchu.msb --> ORACLE_HOME/has/mesg/gipchu.msb gipci.msb --> ORACLE_HOME/has/mesg/gipci.msb gipciw.msb --> ORACLE_HOME/has/mesg/gipciw.msb gipcja.msb --> ORACLE_HOME/has/mesg/gipcja.msb gipcko.msb --> ORACLE_HOME/has/mesg/gipcko.msb gipcn.msb --> ORACLE_HOME/has/mesg/gipcn.msb gipcnl.msb --> ORACLE_HOME/has/mesg/gipcnl.msb gipcpl.msb --> ORACLE_HOME/has/mesg/gipcpl.msb gipcpt.msb --> ORACLE_HOME/has/mesg/gipcpt.msb gipcptb.msb --> ORACLE_HOME/has/mesg/gipcptb.msb gipcro.msb --> ORACLE_HOME/has/mesg/gipcro.msb gipcru.msb --> ORACLE_HOME/has/mesg/gipcru.msb gipcsf.msb --> ORACLE_HOME/has/mesg/gipcsf.msb gipcsk.msb --> ORACLE_HOME/has/mesg/gipcsk.msb gipcth.msb --> ORACLE_HOME/has/mesg/gipcth.msb gipctr.msb --> ORACLE_HOME/has/mesg/gipctr.msb gipcus.msb --> ORACLE_HOME/has/mesg/gipcus.msb gipcus.msg --> ORACLE_HOME/has/mesg/gipcus.msg gipczhs.msb --> ORACLE_HOME/has/mesg/gipczhs.msb gipczht.msb --> ORACLE_HOME/has/mesg/gipczht.msb odigus.msb --> ORACLE_HOME/has/mesg/odigus.msb odigus.msg --> ORACLE_HOME/has/mesg/odigus.msg rdear.msb --> ORACLE_HOME/has/mesg/rdear.msb rdeca.msb --> ORACLE_HOME/has/mesg/rdeca.msb rdecs.msb --> ORACLE_HOME/has/mesg/rdecs.msb rded.msb --> ORACLE_HOME/has/mesg/rded.msb rdedk.msb --> ORACLE_HOME/has/mesg/rdedk.msb rdee.msb --> ORACLE_HOME/has/mesg/rdee.msb rdeel.msb --> ORACLE_HOME/has/mesg/rdeel.msb rdef.msb --> ORACLE_HOME/has/mesg/rdef.msb rdehu.msb --> ORACLE_HOME/has/mesg/rdehu.msb rdei.msb --> ORACLE_HOME/has/mesg/rdei.msb rdeiw.msb --> ORACLE_HOME/has/mesg/rdeiw.msb rdeja.msb --> ORACLE_HOME/has/mesg/rdeja.msb rdeko.msb --> ORACLE_HOME/has/mesg/rdeko.msb rden.msb --> ORACLE_HOME/has/mesg/rden.msb rdenl.msb --> ORACLE_HOME/has/mesg/rdenl.msb rdepl.msb --> ORACLE_HOME/has/mesg/rdepl.msb rdept.msb --> ORACLE_HOME/has/mesg/rdept.msb rdeptb.msb --> ORACLE_HOME/has/mesg/rdeptb.msb rdero.msb --> ORACLE_HOME/has/mesg/rdero.msb rderu.msb --> ORACLE_HOME/has/mesg/rderu.msb rdes.msb --> ORACLE_HOME/has/mesg/rdes.msb rdesf.msb --> ORACLE_HOME/has/mesg/rdesf.msb rdesk.msb --> ORACLE_HOME/has/mesg/rdesk.msb rdeth.msb --> ORACLE_HOME/has/mesg/rdeth.msb rdetr.msb --> ORACLE_HOME/has/mesg/rdetr.msb rdeus.msb --> ORACLE_HOME/has/mesg/rdeus.msb rdeus.msg --> ORACLE_HOME/has/mesg/rdeus.msg rdezhs.msb --> ORACLE_HOME/has/mesg/rdezhs.msb rdezht.msb --> ORACLE_HOME/has/mesg/rdezht.msb s_oracle.has.common.acl --> ORACLE_HOME/install/acls/s_oracle.has.common.acl chaconfig.jar --> ORACLE_HOME/jlib/chaconfig.jar clsce.jar --> ORACLE_HOME/jlib/clsce.jar clscred.jar --> ORACLE_HOME/jlib/clscred.jar gns.jar --> ORACLE_HOME/jlib/gns.jar rd.jar --> ORACLE_HOME/jlib/rd.jar rdtool.jar --> ORACLE_HOME/jlib/rdtool.jar srvmasm.jar --> ORACLE_HOME/jlib/srvmasm.jar srvmhas.jar --> ORACLE_HOME/jlib/srvmhas.jar supercluster-common.jar --> ORACLE_HOME/jlib/supercluster-common.jar supercluster.jar --> ORACLE_HOME/jlib/supercluster.jar libclsce12.so --> ORACLE_HOME/lib/libclsce12.so libclscred12.so --> ORACLE_HOME/lib/libclscred12.so libclsnsjni12.so --> ORACLE_HOME/lib/libclsnsjni12.so libdbcfg12.so --> ORACLE_HOME/lib/libdbcfg12.so libeons.so --> ORACLE_HOME/lib/libeons.so libgns12.so --> ORACLE_HOME/lib/libgns12.so libgnsjni12.so --> ORACLE_HOME/lib/libgnsjni12.so liboevm.a --> ORACLE_HOME/lib/liboevm.a librdjni12.so --> ORACLE_HOME/lib/librdjni12.so libsrvmcred12.so --> ORACLE_HOME/lib/libsrvmcred12.so libsrvmhas12.so --> ORACLE_HOME/lib/libsrvmhas12.so libsrvmocr12.so --> ORACLE_HOME/lib/libsrvmocr12.so libuini12.so --> ORACLE_HOME/lib/libuini12.so s0clsrdmai.o --> ORACLE_HOME/racg/lib/s0clsrdmai.o s0clsreut.o --> ORACLE_HOME/racg/lib/s0clsreut.o s0clsrmain.o --> ORACLE_HOME/racg/lib/s0clsrmain.o s0clsrmdb.o --> ORACLE_HOME/racg/lib/s0clsrmdb.o clsrar.msb --> ORACLE_HOME/racg/mesg/clsrar.msb clsrcs.msb --> ORACLE_HOME/racg/mesg/clsrcs.msb clsrd.msb --> ORACLE_HOME/racg/mesg/clsrd.msb clsrdk.msb --> ORACLE_HOME/racg/mesg/clsrdk.msb clsre.msb --> ORACLE_HOME/racg/mesg/clsre.msb clsrel.msb --> ORACLE_HOME/racg/mesg/clsrel.msb clsrf.msb --> ORACLE_HOME/racg/mesg/clsrf.msb clsrhu.msb --> ORACLE_HOME/racg/mesg/clsrhu.msb clsri.msb --> ORACLE_HOME/racg/mesg/clsri.msb clsriw.msb --> ORACLE_HOME/racg/mesg/clsriw.msb clsrja.msb --> ORACLE_HOME/racg/mesg/clsrja.msb clsrko.msb --> ORACLE_HOME/racg/mesg/clsrko.msb clsrn.msb --> ORACLE_HOME/racg/mesg/clsrn.msb clsrnl.msb --> ORACLE_HOME/racg/mesg/clsrnl.msb clsrpl.msb --> ORACLE_HOME/racg/mesg/clsrpl.msb clsrpt.msb --> ORACLE_HOME/racg/mesg/clsrpt.msb clsrptb.msb --> ORACLE_HOME/racg/mesg/clsrptb.msb clsrro.msb --> ORACLE_HOME/racg/mesg/clsrro.msb clsrru.msb --> ORACLE_HOME/racg/mesg/clsrru.msb clsrs.msb --> ORACLE_HOME/racg/mesg/clsrs.msb clsrsf.msb --> ORACLE_HOME/racg/mesg/clsrsf.msb clsrsk.msb --> ORACLE_HOME/racg/mesg/clsrsk.msb clsrth.msb --> ORACLE_HOME/racg/mesg/clsrth.msb clsrtr.msb --> ORACLE_HOME/racg/mesg/clsrtr.msb clsrus.msb --> ORACLE_HOME/racg/mesg/clsrus.msb clsrus.msg --> ORACLE_HOME/racg/mesg/clsrus.msg clsrzhs.msb --> ORACLE_HOME/racg/mesg/clsrzhs.msb clsrzht.msb --> ORACLE_HOME/racg/mesg/clsrzht.msb ocrlog.ini --> ORACLE_HOME/srvm/admin/ocrlog.ini antlr-3.3-complete.jar --> ORACLE_HOME/srvm/jlib/antlr-3.3-complete.jar srvctl.jar --> ORACLE_HOME/srvm/jlib/srvctl.jar env_srvm.mk --> ORACLE_HOME/srvm/lib/env_srvm.mk ins_srvm.mk --> ORACLE_HOME/srvm/lib/ins_srvm.mk prgcrshm.o --> ORACLE_HOME/srvm/lib/prgcrshm.o sprgcrshm.o --> ORACLE_HOME/srvm/lib/sprgcrshm.o prcaus.msg --> ORACLE_HOME/srvm/mesg/prcaus.msg prccus.msg --> ORACLE_HOME/srvm/mesg/prccus.msg prcdus.msg --> ORACLE_HOME/srvm/mesg/prcdus.msg prceus.msg --> ORACLE_HOME/srvm/mesg/prceus.msg prcfus.msg --> ORACLE_HOME/srvm/mesg/prcfus.msg prchus.msg --> ORACLE_HOME/srvm/mesg/prchus.msg prcius.msg --> ORACLE_HOME/srvm/mesg/prcius.msg prclus.msg --> ORACLE_HOME/srvm/mesg/prclus.msg prcnus.msg --> ORACLE_HOME/srvm/mesg/prcnus.msg prcpus.msg --> ORACLE_HOME/srvm/mesg/prcpus.msg prcqus.msg --> ORACLE_HOME/srvm/mesg/prcqus.msg prcrus.msg --> ORACLE_HOME/srvm/mesg/prcrus.msg prcsus.msg --> ORACLE_HOME/srvm/mesg/prcsus.msg prctus.msg --> ORACLE_HOME/srvm/mesg/prctus.msg prcvus.msg --> ORACLE_HOME/srvm/mesg/prcvus.msg prcwus.msg --> ORACLE_HOME/srvm/mesg/prcwus.msg prczus.msg --> ORACLE_HOME/srvm/mesg/prczus.msg prghus.msg --> ORACLE_HOME/srvm/mesg/prghus.msg prgous.msg --> ORACLE_HOME/srvm/mesg/prgous.msg prgrus.msg --> ORACLE_HOME/srvm/mesg/prgrus.msg prgsus.msg --> ORACLE_HOME/srvm/mesg/prgsus.msg prifd.msb --> ORACLE_HOME/srvm/mesg/prifd.msb prife.msb --> ORACLE_HOME/srvm/mesg/prife.msb priff.msb --> ORACLE_HOME/srvm/mesg/priff.msb prifi.msb --> ORACLE_HOME/srvm/mesg/prifi.msb prifja.msb --> ORACLE_HOME/srvm/mesg/prifja.msb prifko.msb --> ORACLE_HOME/srvm/mesg/prifko.msb prifptb.msb --> ORACLE_HOME/srvm/mesg/prifptb.msb prifus.msb --> ORACLE_HOME/srvm/mesg/prifus.msb prifus.msg --> ORACLE_HOME/srvm/mesg/prifus.msg prkaus.msg --> ORACLE_HOME/srvm/mesg/prkaus.msg prkcus.msg --> ORACLE_HOME/srvm/mesg/prkcus.msg prkdus.msg --> ORACLE_HOME/srvm/mesg/prkdus.msg prkeus.msg --> ORACLE_HOME/srvm/mesg/prkeus.msg prkfus.msg --> ORACLE_HOME/srvm/mesg/prkfus.msg prkhus.msg --> ORACLE_HOME/srvm/mesg/prkhus.msg prknus.msg --> ORACLE_HOME/srvm/mesg/prknus.msg prkous.msg --> ORACLE_HOME/srvm/mesg/prkous.msg prkpus.msg --> ORACLE_HOME/srvm/mesg/prkpus.msg prkrus.msg --> ORACLE_HOME/srvm/mesg/prkrus.msg prksus.msg --> ORACLE_HOME/srvm/mesg/prksus.msg prkuus.msg --> ORACLE_HOME/srvm/mesg/prkuus.msg prkzus.msg --> ORACLE_HOME/srvm/mesg/prkzus.msg prveus.msg --> ORACLE_HOME/srvm/mesg/prveus.msg prvfus.msg --> ORACLE_HOME/srvm/mesg/prvfus.msg prvgus.msg --> ORACLE_HOME/srvm/mesg/prvgus.msg prvpus.msg --> ORACLE_HOME/srvm/mesg/prvpus.msg sclcus.msg --> ORACLE_HOME/srvm/mesg/sclcus.msg sclsus.msg --> ORACLE_HOME/srvm/mesg/sclsus.msg cluvfyrac.sh --> ORACLE_HOME/bin/cluvfyrac.sh crsdiag.pl --> ORACLE_HOME/bin/crsdiag.pl kfod --> ORACLE_HOME/bin/kfod lsnodes --> ORACLE_HOME/bin/lsnodes srvconfig --> ORACLE_HOME/bin/srvconfig oracle.has.db.acl --> ORACLE_HOME/install/acls/oracle.has.db.acl s_oracle.has.db.acl --> ORACLE_HOME/install/acls/s_oracle.has.db.acl libclsr12.a --> ORACLE_HOME/lib/libclsr12.a libskjcx12.so --> ORACLE_HOME/lib/libskjcx12.so libsrvm12.so --> ORACLE_HOME/oui/lib/libsrvm12.so ins_has.mk --> ORACLE_HOME/racg/lib/ins_has.mk clustdb.sql --> ORACLE_HOME/srvm/admin/clustdb.sql init.ora --> ORACLE_HOME/srvm/admin/init.ora libdbcfg12.a --> ORACLE_HOME/srvm/lib/libdbcfg12.a libocr12.a --> ORACLE_HOME/srvm/lib/libocr12.a libocrb12.a --> ORACLE_HOME/srvm/lib/libocrb12.a libocrutl12.a --> ORACLE_HOME/srvm/lib/libocrutl12.a libsrvm12.a --> ORACLE_HOME/srvm/lib/libsrvm12.a libsrvmhas12.a --> ORACLE_HOME/srvm/lib/libsrvmhas12.a libsrvmocr12.a --> ORACLE_HOME/srvm/lib/libsrvmocr12.a libuini12.a --> ORACLE_HOME/srvm/lib/libuini12.a pradmgrp.o --> ORACLE_HOME/srvm/lib/pradmgrp.o prifcg.o --> ORACLE_HOME/srvm/lib/prifcg.o prifcmn.o --> ORACLE_HOME/srvm/lib/prifcmn.o prifutl.o --> ORACLE_HOME/srvm/lib/prifutl.o prraw.o --> ORACLE_HOME/srvm/lib/prraw.o sprdbgrp.o --> ORACLE_HOME/srvm/lib/sprdbgrp.o sprifcg.o --> ORACLE_HOME/srvm/lib/sprifcg.o sprraw.o --> ORACLE_HOME/srvm/lib/sprraw.o env_has.mk --> ORACLE_HOME/crs/lib/env_has.mk crsar.msb --> ORACLE_HOME/crs/mesg/crsar.msb crscs.msb --> ORACLE_HOME/crs/mesg/crscs.msb crsd.msb --> ORACLE_HOME/crs/mesg/crsd.msb crsdk.msb --> ORACLE_HOME/crs/mesg/crsdk.msb crse.msb --> ORACLE_HOME/crs/mesg/crse.msb crsel.msb --> ORACLE_HOME/crs/mesg/crsel.msb crshu.msb --> ORACLE_HOME/crs/mesg/crshu.msb crsi.msb --> ORACLE_HOME/crs/mesg/crsi.msb crsiw.msb --> ORACLE_HOME/crs/mesg/crsiw.msb crsja.msb --> ORACLE_HOME/crs/mesg/crsja.msb crsko.msb --> ORACLE_HOME/crs/mesg/crsko.msb crsn.msb --> ORACLE_HOME/crs/mesg/crsn.msb crsnl.msb --> ORACLE_HOME/crs/mesg/crsnl.msb crspl.msb --> ORACLE_HOME/crs/mesg/crspl.msb crspt.msb --> ORACLE_HOME/crs/mesg/crspt.msb crsptb.msb --> ORACLE_HOME/crs/mesg/crsptb.msb crsro.msb --> ORACLE_HOME/crs/mesg/crsro.msb crsru.msb --> ORACLE_HOME/crs/mesg/crsru.msb crss.msb --> ORACLE_HOME/crs/mesg/crss.msb crssf.msb --> ORACLE_HOME/crs/mesg/crssf.msb crssk.msb --> ORACLE_HOME/crs/mesg/crssk.msb crsth.msb --> ORACLE_HOME/crs/mesg/crsth.msb crstr.msb --> ORACLE_HOME/crs/mesg/crstr.msb crsus.msb --> ORACLE_HOME/crs/mesg/crsus.msb crsus.msg --> ORACLE_HOME/crs/mesg/crsus.msg crszhs.msb --> ORACLE_HOME/crs/mesg/crszhs.msb crszht.msb --> ORACLE_HOME/crs/mesg/crszht.msb clssar.msb --> ORACLE_HOME/css/mesg/clssar.msb clsscs.msb --> ORACLE_HOME/css/mesg/clsscs.msb clssd.msb --> ORACLE_HOME/css/mesg/clssd.msb clssdk.msb --> ORACLE_HOME/css/mesg/clssdk.msb clsse.msb --> ORACLE_HOME/css/mesg/clsse.msb clssel.msb --> ORACLE_HOME/css/mesg/clssel.msb clsshu.msb --> ORACLE_HOME/css/mesg/clsshu.msb clssi.msb --> ORACLE_HOME/css/mesg/clssi.msb clssiw.msb --> ORACLE_HOME/css/mesg/clssiw.msb clssja.msb --> ORACLE_HOME/css/mesg/clssja.msb clssko.msb --> ORACLE_HOME/css/mesg/clssko.msb clssn.msb --> ORACLE_HOME/css/mesg/clssn.msb clssnl.msb --> ORACLE_HOME/css/mesg/clssnl.msb clsspl.msb --> ORACLE_HOME/css/mesg/clsspl.msb clsspt.msb --> ORACLE_HOME/css/mesg/clsspt.msb clssptb.msb --> ORACLE_HOME/css/mesg/clssptb.msb clssro.msb --> ORACLE_HOME/css/mesg/clssro.msb clssru.msb --> ORACLE_HOME/css/mesg/clssru.msb clsss.msb --> ORACLE_HOME/css/mesg/clsss.msb clsssf.msb --> ORACLE_HOME/css/mesg/clsssf.msb clsssk.msb --> ORACLE_HOME/css/mesg/clsssk.msb clssth.msb --> ORACLE_HOME/css/mesg/clssth.msb clsstr.msb --> ORACLE_HOME/css/mesg/clsstr.msb clssus.msb --> ORACLE_HOME/css/mesg/clssus.msb clssus.msg --> ORACLE_HOME/css/mesg/clssus.msg clsszhs.msb --> ORACLE_HOME/css/mesg/clsszhs.msb clsszht.msb --> ORACLE_HOME/css/mesg/clsszht.msb clsfeus.msb --> ORACLE_HOME/has/mesg/clsfeus.msb clsfeus.msg --> ORACLE_HOME/has/mesg/clsfeus.msg clswus.msb --> ORACLE_HOME/has/mesg/clswus.msb clswus.msg --> ORACLE_HOME/has/mesg/clswus.msg gipcar.msb --> ORACLE_HOME/has/mesg/gipcar.msb gipcca.msb --> ORACLE_HOME/has/mesg/gipcca.msb gipccs.msb --> ORACLE_HOME/has/mesg/gipccs.msb gipcd.msb --> ORACLE_HOME/has/mesg/gipcd.msb gipcdk.msb --> ORACLE_HOME/has/mesg/gipcdk.msb gipce.msb --> ORACLE_HOME/has/mesg/gipce.msb gipcel.msb --> ORACLE_HOME/has/mesg/gipcel.msb gipcf.msb --> ORACLE_HOME/has/mesg/gipcf.msb gipchu.msb --> ORACLE_HOME/has/mesg/gipchu.msb gipci.msb --> ORACLE_HOME/has/mesg/gipci.msb gipciw.msb --> ORACLE_HOME/has/mesg/gipciw.msb gipcja.msb --> ORACLE_HOME/has/mesg/gipcja.msb gipcko.msb --> ORACLE_HOME/has/mesg/gipcko.msb gipcn.msb --> ORACLE_HOME/has/mesg/gipcn.msb gipcnl.msb --> ORACLE_HOME/has/mesg/gipcnl.msb gipcpl.msb --> ORACLE_HOME/has/mesg/gipcpl.msb gipcpt.msb --> ORACLE_HOME/has/mesg/gipcpt.msb gipcptb.msb --> ORACLE_HOME/has/mesg/gipcptb.msb gipcro.msb --> ORACLE_HOME/has/mesg/gipcro.msb gipcru.msb --> ORACLE_HOME/has/mesg/gipcru.msb gipcsf.msb --> ORACLE_HOME/has/mesg/gipcsf.msb gipcsk.msb --> ORACLE_HOME/has/mesg/gipcsk.msb gipcth.msb --> ORACLE_HOME/has/mesg/gipcth.msb gipctr.msb --> ORACLE_HOME/has/mesg/gipctr.msb gipcus.msb --> ORACLE_HOME/has/mesg/gipcus.msb gipcus.msg --> ORACLE_HOME/has/mesg/gipcus.msg gipczhs.msb --> ORACLE_HOME/has/mesg/gipczhs.msb gipczht.msb --> ORACLE_HOME/has/mesg/gipczht.msb srvm.jar --> ORACLE_HOME/jlib/srvm.jar libagfw12.so --> ORACLE_HOME/lib/libagfw12.so libclsra12.so --> ORACLE_HOME/lib/libclsra12.so libhasgen12.so --> ORACLE_HOME/lib/libhasgen12.so libocr12.so --> ORACLE_HOME/lib/libocr12.so libocrb12.so --> ORACLE_HOME/lib/libocrb12.so libocrutl12.so --> ORACLE_HOME/lib/libocrutl12.so procd.msb --> ORACLE_HOME/srvm/mesg/procd.msb proce.msb --> ORACLE_HOME/srvm/mesg/proce.msb procf.msb --> ORACLE_HOME/srvm/mesg/procf.msb proci.msb --> ORACLE_HOME/srvm/mesg/proci.msb procja.msb --> ORACLE_HOME/srvm/mesg/procja.msb procko.msb --> ORACLE_HOME/srvm/mesg/procko.msb procld.msb --> ORACLE_HOME/srvm/mesg/procld.msb procle.msb --> ORACLE_HOME/srvm/mesg/procle.msb proclf.msb --> ORACLE_HOME/srvm/mesg/proclf.msb procli.msb --> ORACLE_HOME/srvm/mesg/procli.msb proclja.msb --> ORACLE_HOME/srvm/mesg/proclja.msb proclko.msb --> ORACLE_HOME/srvm/mesg/proclko.msb proclptb.msb --> ORACLE_HOME/srvm/mesg/proclptb.msb proclus.msb --> ORACLE_HOME/srvm/mesg/proclus.msb proclus.msg --> ORACLE_HOME/srvm/mesg/proclus.msg proclzhs.msb --> ORACLE_HOME/srvm/mesg/proclzhs.msb proclzht.msb --> ORACLE_HOME/srvm/mesg/proclzht.msb procptb.msb --> ORACLE_HOME/srvm/mesg/procptb.msb procus.msb --> ORACLE_HOME/srvm/mesg/procus.msb procus.msg --> ORACLE_HOME/srvm/mesg/procus.msg proczhs.msb --> ORACLE_HOME/srvm/mesg/proczhs.msb proczht.msb --> ORACLE_HOME/srvm/mesg/proczht.msb crsconvtoext.pm --> ORACLE_HOME/crs/install/crsconvtoext.pm crscpcfg.pm --> ORACLE_HOME/crs/install/crscpcfg.pm crsdeconfig.pm --> ORACLE_HOME/crs/install/crsdeconfig.pm crsdowngrade.pm --> ORACLE_HOME/crs/install/crsdowngrade.pm crsgpnp.pm --> ORACLE_HOME/crs/install/crsgpnp.pm crsinstall.pm --> ORACLE_HOME/crs/install/crsinstall.pm crska.pm --> ORACLE_HOME/crs/install/crska.pm crspatch.pm --> ORACLE_HOME/crs/install/crspatch.pm crstfa.pm --> ORACLE_HOME/crs/install/crstfa.pm crsupgrade.pm --> ORACLE_HOME/crs/install/crsupgrade.pm crsutils.pm --> ORACLE_HOME/crs/install/crsutils.pm dropdb.pl --> ORACLE_HOME/crs/install/dropdb.pl install.excl --> ORACLE_HOME/crs/install/install.excl installRemove.excl --> ORACLE_HOME/crs/install/installRemove.excl install_gi.excl --> ORACLE_HOME/crs/install/install_gi.excl oraClusterwareComp.pm --> ORACLE_HOME/crs/install/oraClusterwareComp.pm oraacfs.pm --> ORACLE_HOME/crs/install/oraacfs.pm oraafd.pm --> ORACLE_HOME/crs/install/oraafd.pm oraasm.pm --> ORACLE_HOME/crs/install/oraasm.pm orachm.pm --> ORACLE_HOME/crs/install/orachm.pm oracss.pm --> ORACLE_HOME/crs/install/oracss.pm oraios.pm --> ORACLE_HOME/crs/install/oraios.pm oraocr.pm --> ORACLE_HOME/crs/install/oraocr.pm oraohasd.pm --> ORACLE_HOME/crs/install/oraohasd.pm oraolr.pm --> ORACLE_HOME/crs/install/oraolr.pm oraons.pm --> ORACLE_HOME/crs/install/oraons.pm oraqosmserver.pm --> ORACLE_HOME/crs/install/oraqosmserver.pm orasrvm.pm --> ORACLE_HOME/crs/install/orasrvm.pm rootcrs.pl --> ORACLE_HOME/crs/install/rootcrs.pl roothas.pl --> ORACLE_HOME/crs/install/roothas.pl s_crsconfig_defs --> ORACLE_HOME/crs/install/s_crsconfig_defs s_crsutils.pm --> ORACLE_HOME/crs/install/s_crsutils.pm s_orachm.pm --> ORACLE_HOME/crs/install/s_orachm.pm s_oraocr.pm --> ORACLE_HOME/crs/install/s_oraocr.pm s_oraolr.pm --> ORACLE_HOME/crs/install/s_oraolr.pm cvu_config --> ORACLE_HOME/cv/admin/cvu_config admin.dtd --> ORACLE_HOME/cv/cvdata/admin.dtd admin.xml --> ORACLE_HOME/cv/cvdata/admin.xml constraints.dtd --> ORACLE_HOME/cv/cvdata/constraints.dtd constraints.xml --> ORACLE_HOME/cv/cvdata/constraints.xml cvu_prereq.xml --> ORACLE_HOME/cv/cvdata/cvu_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/ora_software_cfg.xml prereq.xsd --> ORACLE_HOME/cv/cvdata/prereq.xsd taskdefinition.xml --> ORACLE_HOME/cv/cvdata/taskdefinition.xml taskfactory.xsd --> ORACLE_HOME/cv/cvdata/taskfactory.xsd taskplacement.xml --> ORACLE_HOME/cv/cvdata/taskplacement.xml exectask --> ORACLE_HOME/cv/remenv/exectask exectask.sh --> ORACLE_HOME/cv/remenv/exectask.sh clsrscus.msb --> ORACLE_HOME/has/mesg/clsrscus.msb clsrscus.msg --> ORACLE_HOME/has/mesg/clsrscus.msg oracle.has.deconfig.acl --> ORACLE_HOME/install/acls/oracle.has.deconfig.acl s_oracle.has.deconfig.acl --> ORACLE_HOME/install/acls/s_oracle.has.deconfig.acl cvu.jar --> ORACLE_HOME/jlib/cvu.jar rhphelper122.jar --> ORACLE_HOME/jlib/rhphelper122.jar srvm.jar --> ORACLE_HOME/jlib/srvm.jar srvmasm.jar --> ORACLE_HOME/jlib/srvmasm.jar srvmhas.jar --> ORACLE_HOME/jlib/srvmhas.jar supercluster-common.jar --> ORACLE_HOME/jlib/supercluster-common.jar supercluster.jar --> ORACLE_HOME/jlib/supercluster.jar libdbcfg12.so --> ORACLE_HOME/lib/libdbcfg12.so libhasgen12.so --> ORACLE_HOME/lib/libhasgen12.so libocr12.so --> ORACLE_HOME/lib/libocr12.so libocrb12.so --> ORACLE_HOME/lib/libocrb12.so libocrutl12.so --> ORACLE_HOME/lib/libocrutl12.so libsrvmhas12.so --> ORACLE_HOME/lib/libsrvmhas12.so libsrvmocr12.so --> ORACLE_HOME/lib/libsrvmocr12.so mttrans --> ORACLE_HOME/srvm/admin/mttrans ractrans --> ORACLE_HOME/srvm/admin/ractrans rhpboot.incl --> ORACLE_HOME/srvm/admin/rhpboot.incl rhphelper --> ORACLE_HOME/srvm/admin/rhphelper rhpmovedb.pl --> ORACLE_HOME/srvm/admin/rhpmovedb.pl rhpmovegi.pl --> ORACLE_HOME/srvm/admin/rhpmovegi.pl rhpmovegi112.pl --> ORACLE_HOME/srvm/admin/rhpmovegi112.pl cvu.jar --> ORACLE_HOME/inventory/Scripts/ext/jlib/cvu.jar cvu_config --> ORACLE_HOME/cv/admin/cvu_config cvusys.sql --> ORACLE_HOME/cv/admin/cvusys.sql crsinst_prereq.xml --> ORACLE_HOME/cv/cvdata/101/crsinst_prereq.xml dbcfg_prereq.xml --> ORACLE_HOME/cv/cvdata/101/dbcfg_prereq.xml dbinst_prereq.xml --> ORACLE_HOME/cv/cvdata/101/dbinst_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/101/ora_software_cfg.xml sihainst_prereq.xml --> ORACLE_HOME/cv/cvdata/101/sihainst_prereq.xml crsinst_prereq.xml --> ORACLE_HOME/cv/cvdata/102/crsinst_prereq.xml dbcfg_prereq.xml --> ORACLE_HOME/cv/cvdata/102/dbcfg_prereq.xml dbinst_prereq.xml --> ORACLE_HOME/cv/cvdata/102/dbinst_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/102/ora_software_cfg.xml sihainst_prereq.xml --> ORACLE_HOME/cv/cvdata/102/sihainst_prereq.xml crsinst_prereq.xml --> ORACLE_HOME/cv/cvdata/111/crsinst_prereq.xml dbcfg_prereq.xml --> ORACLE_HOME/cv/cvdata/111/dbcfg_prereq.xml dbinst_prereq.xml --> ORACLE_HOME/cv/cvdata/111/dbinst_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/111/ora_software_cfg.xml sihainst_prereq.xml --> ORACLE_HOME/cv/cvdata/111/sihainst_prereq.xml crsinst_prereq.xml --> ORACLE_HOME/cv/cvdata/112/crsinst_prereq.xml dbcfg_prereq.xml --> ORACLE_HOME/cv/cvdata/112/dbcfg_prereq.xml dbinst_prereq.xml --> ORACLE_HOME/cv/cvdata/112/dbinst_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/112/ora_software_cfg.xml sihainst_prereq.xml --> ORACLE_HOME/cv/cvdata/112/sihainst_prereq.xml crsinst_prereq.xml --> ORACLE_HOME/cv/cvdata/121/crsinst_prereq.xml dbcfg_prereq.xml --> ORACLE_HOME/cv/cvdata/121/dbcfg_prereq.xml dbinst_prereq.xml --> ORACLE_HOME/cv/cvdata/121/dbinst_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/121/ora_software_cfg.xml sihainst_prereq.xml --> ORACLE_HOME/cv/cvdata/121/sihainst_prereq.xml crsinst_prereq.xml --> ORACLE_HOME/cv/cvdata/122/crsinst_prereq.xml dbcfg_prereq.xml --> ORACLE_HOME/cv/cvdata/122/dbcfg_prereq.xml dbinst_prereq.xml --> ORACLE_HOME/cv/cvdata/122/dbinst_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/122/ora_software_cfg.xml sihainst_prereq.xml --> ORACLE_HOME/cv/cvdata/122/sihainst_prereq.xml admin.dtd --> ORACLE_HOME/cv/cvdata/admin.dtd admin.xml --> ORACLE_HOME/cv/cvdata/admin.xml constraints.dtd --> ORACLE_HOME/cv/cvdata/constraints.dtd constraints.xml --> ORACLE_HOME/cv/cvdata/constraints.xml cvu_prereq.xml --> ORACLE_HOME/cv/cvdata/cvu_prereq.xml ora_software_cfg.xml --> ORACLE_HOME/cv/cvdata/ora_software_cfg.xml prereq.xsd --> ORACLE_HOME/cv/cvdata/prereq.xsd responsevars.xml --> ORACLE_HOME/cv/cvdata/responsevars.xml responsevars.xsd --> ORACLE_HOME/cv/cvdata/responsevars.xsd taskdefinition.xml --> ORACLE_HOME/cv/cvdata/taskdefinition.xml taskfactory.xsd --> ORACLE_HOME/cv/cvdata/taskfactory.xsd taskplacement.xml --> ORACLE_HOME/cv/cvdata/taskplacement.xml variables.xml --> ORACLE_HOME/cv/cvdata/variables.xml variables.xsd --> ORACLE_HOME/cv/cvdata/variables.xsd check_nodeadd.pl --> ORACLE_HOME/cv/cvutl/check_nodeadd.pl cvuhelper --> ORACLE_HOME/cv/remenv/cvuhelper cvuqdisk-1.0.10-1.rpm --> ORACLE_HOME/cv/remenv/cvuqdisk-1.0.10-1.rpm exectask --> ORACLE_HOME/cv/remenv/exectask exectask.sh --> ORACLE_HOME/cv/remenv/exectask.sh cvuhelper102.jar --> ORACLE_HOME/cv/remenv/jlib/cvuhelper102.jar cvuhelper111.jar --> ORACLE_HOME/cv/remenv/jlib/cvuhelper111.jar cvuhelper112.jar --> ORACLE_HOME/cv/remenv/jlib/cvuhelper112.jar cvuhelper121.jar --> ORACLE_HOME/cv/remenv/jlib/cvuhelper121.jar cvuhelper122.jar --> ORACLE_HOME/cv/remenv/jlib/cvuhelper122.jar orarun.sh --> ORACLE_HOME/cv/remenv/orarun.sh alert_log_file_size_analyzer.sh --> ORACLE_HOME/cv/remenv/pluggable/alert_log_file_size_analyzer.sh alert_log_file_size_analyzer2.sh --> ORACLE_HOME/cv/remenv/pluggable/alert_log_file_size_analyzer2.sh bdump_dest_trace_analyzer.sh --> ORACLE_HOME/cv/remenv/pluggable/bdump_dest_trace_analyzer.sh bdump_dest_trace_analyzer2.sh --> ORACLE_HOME/cv/remenv/pluggable/bdump_dest_trace_analyzer2.sh checkFSMountOptions.sh --> ORACLE_HOME/cv/remenv/pluggable/checkFSMountOptions.sh checkGSDResourceStatus.sh --> ORACLE_HOME/cv/remenv/pluggable/checkGSDResourceStatus.sh checkIOCPDeviceStatus.sh --> ORACLE_HOME/cv/remenv/pluggable/checkIOCPDeviceStatus.sh checkIPHostModel.sh --> ORACLE_HOME/cv/remenv/pluggable/checkIPHostModel.sh checkLogindIPCSettings.sh --> ORACLE_HOME/cv/remenv/pluggable/checkLogindIPCSettings.sh check_afd_drivers.sh --> ORACLE_HOME/cv/remenv/pluggable/check_afd_drivers.sh check_bpf_devices.sh --> ORACLE_HOME/cv/remenv/pluggable/check_bpf_devices.sh check_dax_access.sh --> ORACLE_HOME/cv/remenv/pluggable/check_dax_access.sh check_default_gateway.sh --> ORACLE_HOME/cv/remenv/pluggable/check_default_gateway.sh check_disk_asynch_io_linking.sh --> ORACLE_HOME/cv/remenv/pluggable/check_disk_asynch_io_linking.sh check_e1000.sh --> ORACLE_HOME/cv/remenv/pluggable/check_e1000.sh check_jumbo_frames.sh --> ORACLE_HOME/cv/remenv/pluggable/check_jumbo_frames.sh check_network_bonding.sh --> ORACLE_HOME/cv/remenv/pluggable/check_network_bonding.sh check_network_packet_reassembly.sh --> ORACLE_HOME/cv/remenv/pluggable/check_network_packet_reassembly.sh check_network_param.sh --> ORACLE_HOME/cv/remenv/pluggable/check_network_param.sh check_non_routable_network_interconnect.sh --> ORACLE_HOME/cv/remenv/pluggable/check_non_routable_network_interconnect.sh check_ofsctl.sh --> ORACLE_HOME/cv/remenv/pluggable/check_ofsctl.sh check_rp_filter.sh --> ORACLE_HOME/cv/remenv/pluggable/check_rp_filter.sh check_symvers.sh --> ORACLE_HOME/cv/remenv/pluggable/check_symvers.sh check_tcp_packet_retransmit.sh --> ORACLE_HOME/cv/remenv/pluggable/check_tcp_packet_retransmit.sh check_vip_restart_attempt.sh --> ORACLE_HOME/cv/remenv/pluggable/check_vip_restart_attempt.sh check_vmm.sh --> ORACLE_HOME/cv/remenv/pluggable/check_vmm.sh checkcorefile.sh --> ORACLE_HOME/cv/remenv/pluggable/checkcorefile.sh checkhugepage.sh --> ORACLE_HOME/cv/remenv/pluggable/checkhugepage.sh checkmemlock.sh --> ORACLE_HOME/cv/remenv/pluggable/checkmemlock.sh checkportavail.sh --> ORACLE_HOME/cv/remenv/pluggable/checkportavail.sh checksshd.sh --> ORACLE_HOME/cv/remenv/pluggable/checksshd.sh checksyslog.sh --> ORACLE_HOME/cv/remenv/pluggable/checksyslog.sh checktmpfs.sh --> ORACLE_HOME/cv/remenv/pluggable/checktmpfs.sh common_include.sh --> ORACLE_HOME/cv/remenv/pluggable/common_include.sh core_dump_dest_analyzer.sh --> ORACLE_HOME/cv/remenv/pluggable/core_dump_dest_analyzer.sh core_dump_dest_analyzer2.sh --> ORACLE_HOME/cv/remenv/pluggable/core_dump_dest_analyzer2.sh css_diagwait.sh --> ORACLE_HOME/cv/remenv/pluggable/css_diagwait.sh css_disk_timeout.sh --> ORACLE_HOME/cv/remenv/pluggable/css_disk_timeout.sh css_misscount.sh --> ORACLE_HOME/cv/remenv/pluggable/css_misscount.sh css_reboot_time.sh --> ORACLE_HOME/cv/remenv/pluggable/css_reboot_time.sh getNICSpeed.sh --> ORACLE_HOME/cv/remenv/pluggable/getNICSpeed.sh hangcheck_margin.sh --> ORACLE_HOME/cv/remenv/pluggable/hangcheck_margin.sh hangcheck_reboot.sh --> ORACLE_HOME/cv/remenv/pluggable/hangcheck_reboot.sh hangcheck_tick.sh --> ORACLE_HOME/cv/remenv/pluggable/hangcheck_tick.sh hangchecktimer.sh --> ORACLE_HOME/cv/remenv/pluggable/hangchecktimer.sh listener_naming_convention.sh --> ORACLE_HOME/cv/remenv/pluggable/listener_naming_convention.sh localenam_check.sh --> ORACLE_HOME/cv/remenv/pluggable/localenam_check.sh mus_check.sh --> ORACLE_HOME/cv/remenv/pluggable/mus_check.sh ora_00600_errors_analyzer.sh --> ORACLE_HOME/cv/remenv/pluggable/ora_00600_errors_analyzer.sh ora_00600_errors_analyzer2.sh --> ORACLE_HOME/cv/remenv/pluggable/ora_00600_errors_analyzer2.sh ora_07445_errors_analyzer.sh --> ORACLE_HOME/cv/remenv/pluggable/ora_07445_errors_analyzer.sh ora_07445_errors_analyzer2.sh --> ORACLE_HOME/cv/remenv/pluggable/ora_07445_errors_analyzer2.sh remove_cvuresource_baseline.sh --> ORACLE_HOME/cv/remenv/pluggable/remove_cvuresource_baseline.sh shutdown_hwclock_sync.sh --> ORACLE_HOME/cv/remenv/pluggable/shutdown_hwclock_sync.sh zeroconf.sh --> ORACLE_HOME/cv/remenv/pluggable/zeroconf.sh runfixup.sh --> ORACLE_HOME/cv/remenv/runfixup.sh cvuqdisk-1.0.10-1.rpm --> ORACLE_HOME/cv/rpm/cvuqdisk-1.0.10-1.rpm oracle.has.common.cvu.acl --> ORACLE_HOME/install/acls/oracle.has.common.cvu.acl s_oracle.has.common.cvu.acl --> ORACLE_HOME/install/acls/s_oracle.has.common.cvu.acl cvu.jar --> ORACLE_HOME/jlib/cvu.jar cvuhelper102.jar --> ORACLE_HOME/jlib/cvuhelper102.jar cvuhelper111.jar --> ORACLE_HOME/jlib/cvuhelper111.jar cvuhelper112.jar --> ORACLE_HOME/jlib/cvuhelper112.jar cvuhelper121.jar --> ORACLE_HOME/jlib/cvuhelper121.jar cvuhelper122.jar --> ORACLE_HOME/jlib/cvuhelper122.jar srvm.jar --> ORACLE_HOME/jlib/srvm.jar libhasgen12.so --> ORACLE_HOME/lib/libhasgen12.so libocr12.so --> ORACLE_HOME/lib/libocr12.so libocrb12.so --> ORACLE_HOME/lib/libocrb12.so libocrutl12.so --> ORACLE_HOME/lib/libocrutl12.so Patch Location in Inventory: /u01/app/oracle/product/12.2.0.1/dbhome_1/inventory/oneoffs/28870605 Patch Location in Storage area: /u01/app/oracle/product/12.2.0.1/dbhome_1/.patch_storage/28870605_Jan_7_2019_21_43_36 Patch 28822515 : applied on Wed Feb 20 23:15:19 AZT 2019 Unique Patch ID: 22636216 Patch description: "Database Jan 2019 Release Update : 12.2.0.1.190115 (28822515)" Created on 19 Dec 2018, 02:45:37 hrs PST8PDT Bugs fixed: 26285062, 8480838, 8932139, 13554903, 14221306, 14690846, 15931756 16002385, 16438495, 16727454, 16942578, 17027695, 17533661, 17947871 18308268, 18521691, 18594510, 18774543, 19072655, 19211433, 19285025 19327292, 19526548, 19614243, 19647894, 19649997, 19721304, 20003668 20087519, 20118035, 20120236, 20324049, 20436508, 20532077, 20591151 20620169, 20736227, 20756305, 20866970, 20976443, 21095391, 21143725 21147908, 21159907, 21178363, 21186167, 21197098, 21216226, 21320338 21433452, 21479706, 21520266, 21547051, 21744603, 21882528, 21981529 21985256, 22007324, 22070853, 22072543, 22087683, 22104866, 22179537 22347493, 22364044, 22367053, 22379010, 22446455, 22495673, 22503283 22503297, 22504793, 22530986, 22564336, 22568728, 22581771, 22594071 22599050, 22628825, 22645009, 22654475, 22700845, 22729345, 22826067 22843979, 22845846, 22864303, 22898198, 22950945, 22970869, 22981722 23019710, 23026585, 23035249, 23055900, 23061453, 23065002, 23066146 23080557, 23104033, 23105538, 23110523, 23125560, 23126545, 23127945 23151677, 23179662, 23184263, 23197730, 23234232, 23249829, 23271203 23300142, 23310101, 23312077, 23481673, 23491861, 23499004, 23499160 23521523, 23527363, 23533647, 23548817, 23567857, 23572982, 23581777 23588722, 23599216, 23600861, 23602213, 23645516, 23665623, 23709062 23715460, 23730961, 23733981, 23735292, 23741944, 23746128, 23749454 23761724, 24010030, 24289874, 24294174, 24303148, 24307571, 24308349 24326444, 24326846, 24332831, 24334708, 24336249, 24337882, 24341675 24343905, 24345420, 24346821, 24348685, 24350620, 24352981, 24368004 24371491, 24373756, 24374976, 24376875, 24376878, 24385983, 24401351 24403922, 24415926, 24421668, 24423416, 24425056, 24425998, 24435982 24437162, 24443539, 24457597, 24460392, 24461826, 24467122, 24468470 24470606, 24471473, 24473736, 24485034, 24485161, 24485174, 24486059 24486237, 24509056, 24534401, 24554533, 24555417, 24556967, 24560906 24563422, 24570214, 24570598, 24573817, 24578718, 24578797, 24589081 24589590, 24593740, 24595699, 24600330, 24609592, 24609996, 24616637 24617969, 24623975, 24624166, 24642495, 24654629, 24655717, 24664211 24668398, 24669189, 24674197, 24674955, 24676172, 24677696, 24680959 24689376, 24692973, 24693290, 24697323, 24699619, 24710696, 24713381 24714096, 24717183, 24717859, 24718260, 24719799, 24735430, 24737064 24737403, 24737581, 24744383, 24744686, 24757934, 24759556, 24760407 24764085, 24766309, 24786669, 24792678, 24793511, 24796092, 24797119 24800423, 24801152, 24802934, 24811725, 24812047, 24827228, 24827654 24831514, 24835919, 24843188, 24844549, 24845157, 24848746, 24848923 24850622, 24907917, 24908321, 24911709, 24912588, 24922704, 24923080 24923215, 24923338, 24923790, 24929210, 24938784, 24940060, 24942749 24953434, 24957555, 24960044, 24966594, 24966788, 24968162, 24976007 24978100, 25022574, 25027852, 25028996, 25029022, 25029423, 25034396 25036474, 25044977, 25045228, 25050160, 25051628, 25057811, 25058080 25062592, 25063971, 25065563, 25072986, 25078611, 25086233, 25087436 25093872, 25098160, 25099339, 25099497, 25099758, 25100063, 25100579 25103996, 25107662, 25110233, 25114561, 25120284, 25120668, 25120742 25121089, 25123585, 25124363, 25129925, 25140197, 25145163, 25145215 25150925, 25159176, 25162645, 25164293, 25166187, 25171084, 25175723 25176408, 25178032, 25178101, 25178179, 25179774, 25182817, 25184555 25186079, 25191872, 25192044, 25192729, 25199585, 25201454, 25202355 25203656, 25206864, 25207410, 25209912, 25210268, 25210499, 25211628 25223839, 25224242, 25225795, 25226665, 25227381, 25230870, 25230945 25237577, 25240188, 25240590, 25241448, 25241625, 25244807, 25248384 25251648, 25257085, 25259611, 25262869, 25263960, 25265499, 25283790 25287072, 25296876, 25299227, 25299807, 25300427, 25305405, 25307368 25309116, 25313154, 25313411, 25316758, 25317989, 25320555, 25323525 25328518, 25329664, 25335249, 25335360, 25335790, 25337332, 25337640 25348956, 25353983, 25356118, 25357142, 25362958, 25382812, 25383204 25384462, 25386748, 25388896, 25392535, 25395696, 25397936, 25405813 25410017, 25410180, 25410802, 25410877, 25411036, 25417050, 25417056 25417958, 25425451, 25425760, 25427662, 25429959, 25430120, 25433696 25435038, 25437699, 25440818, 25444961, 25445168, 25451531, 25455795 25457409, 25459958, 25462714, 25463844, 25472112, 25476149, 25478885 25479164, 25489342, 25489367, 25489607, 25492379, 25498930, 25498994 25516250, 25524955, 25528838, 25530080, 25530814, 25535668, 25536819 25537470, 25539063, 25540738, 25546580, 25546608, 25547901, 25551676 25553616, 25554787, 25555252, 25557886, 25558986, 25560487, 25561296 25569149, 25570929, 25575348, 25575628, 25579458, 25579761, 25594901 25597525, 25598473, 25599425, 25600342, 25600421, 25602488, 25603923 25606091, 25607726, 25612095, 25614866, 25616268, 25616359, 25616417 25616645, 25631933, 25633101, 25634317, 25634348, 25635149, 25638456 25639019, 25643818, 25643931, 25646373, 25647325, 25648731, 25653109 25654459, 25654936, 25655390, 25655966, 25659655, 25660847, 25661819 25662088, 25662101, 25662524, 25669791, 25670786, 25671354, 25672640 25674386, 25680221, 25685152, 25686739, 25687460, 25691904, 25694206 25695903, 25700654, 25710420, 25715167, 25717371, 25722055, 25722608 25722720, 25728085, 25729507, 25734963, 25736747, 25739065, 25754606 25757748, 25760195, 25762221, 25764020, 25766822, 25768681, 25772669 25774077, 25775213, 25780343, 25784002, 25785331, 25785441, 25788879 25789041, 25789277, 25789579, 25790353, 25797092, 25797124, 25797305 25800464, 25802510, 25803545, 25807997, 25810704, 25813931, 25818707 25822410, 25823754, 25825910, 25826740, 25830492, 25832935, 25834581 25838361, 25852885, 25856821, 25858672, 25861398, 25865785, 25870579 25871177, 25871639, 25871753, 25872127, 25872389, 25874050, 25874678 25882264, 25885148, 25888073, 25890056, 25890673, 25894239, 25895224 25897615, 25904273, 25904490, 25906117, 25911724, 25914276, 25919622 25932524, 25932728, 25933494, 25941836, 25943271, 25945130, 25947799 25953857, 25954022, 25954054, 25957038, 25963024, 25964954, 25967544 25967985, 25970731, 25973152, 25975723, 25977302, 25980605, 25980770 25981498, 25982666, 25990907, 25995938, 26006257, 26007010, 26019148 26024732, 26025681, 26029780, 26032573, 26034119, 26036748, 26037215 26038086, 26039623, 26040483, 26045732, 26078437, 26078493, 26080410 26083298, 26088426, 26088836, 26090767, 26091640, 26091786, 26095327 26095405, 26096382, 26108080, 26108337, 26110632, 26111842, 26112621 26115103, 26121990, 26124078, 26137367, 26138085, 26149904, 26153977 26169341, 26169345, 26170715, 26176002, 26187943, 26189861, 26198757 26198926, 26201113, 26203182, 26223039, 26237431, 26237773, 26238195 26242031, 26243698, 26244115, 26245237, 26248143, 26249718, 26256131 26259265, 26261327, 26263328, 26263721, 26269790, 26271001, 26277439 26285933, 26308650, 26309047, 26318627, 26323308, 26324769, 26327624 26330994, 26331743, 26333141, 26336977, 26338953, 26351334, 26353617 26358670, 26359091, 26362821, 26366517, 26367012, 26371725, 26374791 26375250, 26380097, 26385189, 26388538, 26396790, 26399626, 26399691 26406387, 26407408, 26412540, 26418088, 26420561, 26421667, 26422277 26423085, 26426526, 26426967, 26430737, 26434436, 26434999, 26435073 26436168, 26438612, 26440749, 26442308, 26444601, 26444887, 26446098 26452606, 26474703, 26475419, 26476244, 26478970, 26479173, 26486365 26492866, 26493289, 26498354, 26513709, 26521043, 26522439, 26523432 26526726, 26536320, 26537307, 26542135, 26544823, 26545688, 26546070 26546664, 26546754, 26548363, 26556014, 26569225, 26575788, 26580633 26582460, 26584641, 26597140, 26599395, 26608137, 26609942, 26615291 26615690, 26623652, 26626879, 26629381, 26633355, 26633558, 26635897 26637273, 26637824, 26639167, 26641610, 26650226, 26658759, 26659182 26680105, 26712331, 26714486, 26714910, 26717528, 26725687, 26727397 26729494, 26729611, 26740700, 26744595, 26745002, 26751106, 26751171 26755171, 26758193, 26764561, 26765212, 26775602, 26784509, 26794786 26797591, 26798516, 26802503, 26820076, 26822620, 26828994, 26840654 26844406, 26844870, 26849779, 26875822, 26883456, 26895149, 26896659 26898563, 26907327, 26908788, 26909100, 26909504, 26911000, 26939314 26944190, 26958896, 26963310, 26966916, 26967713, 26969321, 26970717 26981902, 26983259, 26986173, 26992964, 27006664, 27009164, 27013146 27028251, 27034890, 27037839, 27038986, 27039712, 27044297, 27052607 27058530, 27060167, 27060859, 27073314, 27079140, 27087426, 27090765 27093423, 27110878, 27117822, 27119621, 27122162, 27124624, 27125872 27133662, 27135647, 27135993, 27138325, 27142373, 27142529, 27151826 27153641, 27161071, 27162405, 27163928, 27165231, 27169796, 27170305 27181537, 27181897, 27199245, 27200959, 27207110, 27213224, 27223075 27229389, 27236722, 27244337, 27248917, 27250547, 27251690, 27255377 27256000, 27258578, 27259307, 27262945, 27266245, 27274456, 27274536 27276231, 27283960, 27285244, 27292213, 27293599, 27302711, 27302730 27304410, 27305039, 27314206, 27314390, 27320576, 27321179, 27329612 27333106, 27334316, 27338912, 27338946, 27339115, 27345231, 27346709 27348081, 27349393, 27351628, 27359178, 27367194, 27370965, 27375542 27381498, 27386467, 27394703, 27395416, 27396624, 27396813, 27397048 27400416, 27400598, 27404668, 27405645, 27416997, 27433870, 27434193 27439835, 27441326, 27442041, 27445727, 27466597, 27493674, 27501373 27501413, 27502420, 27504770, 27505229, 27508985, 27510959, 27525909 27534509, 27540613, 27544973, 27548131, 27555481, 27558861, 27560602 27567477, 27593501, 27595973, 27607563, 27611612, 27613080, 27613530 27617978, 27620808, 27634676, 27658186, 27687880, 27688036, 27688099 27688692, 27691920, 27691939, 27693713, 27695063, 27698953, 27700466 27709046, 27726780, 27740424, 27748954, 27757888, 27759077, 27793533 27799032, 27801337, 27835925, 27847259, 27855490, 27882176, 27898015 27931299, 27940876, 27945870, 27951817, 27959048, 27994325, 27995248 27997875, 27998003, 27999638, 28000269, 28022101, 28023081, 28023482 28033429, 28040776, 28067846, 28072567, 28074713, 28090453, 28099662 28111583, 28120951, 28140658, 28171079, 28174827, 28180464, 28184554 28188330, 28194173, 28218832, 28220398, 28226179, 28229360, 28282606 28290434, 28302049, 28305001, 28305362, 28320399, 28354603, 28361221 28365111, 28390273, 28396445, 28420042, 28420457, 28434028, 28437315 28454242, 28483184, 28489150, 28502343, 28508053, 28508557, 28522441 28528349, 28535272, 28571483, 28617959, 28621470, 28627255, 28636676 28691965, 28708023, 28728272, 28749289, 28960211, 28993590 Files Touched: /oracle/sysman/assistants/util/CommonUtils$ListenerInfo.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/CommonUtils$RACListener.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/CommonUtils.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/ha/HAService.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/hasi/HADatabaseUtils.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar prvtolsdd.plb --> ORACLE_HOME/rdbms/admin/prvtolsdd.plb libmql1.so --> ORACLE_HOME/lib/libmql1.so libipc1.so --> ORACLE_HOME/lib/libipc1.so ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iwrc ctxdef.sql --> ORACLE_HOME/ctx/admin/ctxdef.sql ctxposup.sql --> ORACLE_HOME/ctx/admin/ctxposup.sql ctxpreup.sql --> ORACLE_HOME/ctx/admin/ctxpreup.sql driparse.plb --> ORACLE_HOME/ctx/admin/driparse.plb drixmd.plb --> ORACLE_HOME/ctx/admin/drixmd.plb drvddl.plb --> ORACLE_HOME/ctx/admin/drvddl.plb drvxtab.plb --> ORACLE_HOME/ctx/admin/drvxtab.plb u1202000.sql --> ORACLE_HOME/ctx/admin/u1202000.sql ctxhx.o --> ORACLE_HOME/ctx/lib/ctxhx.o ctxhx --> ORACLE_HOME/ctx/bin/ctxhx adinit.dat --> ORACLE_HOME/ctx/lib/adinit.dat cmmap000.bin --> ORACLE_HOME/ctx/lib/cmmap000.bin compreg.bin --> ORACLE_HOME/ctx/lib/compreg.bin extract_archive --> ORACLE_HOME/ctx/lib/extract_archive libccbf.so --> ORACLE_HOME/ctx/lib/libccbf.so libccflex.so --> ORACLE_HOME/ctx/lib/libccflex.so libde_bmp.so --> ORACLE_HOME/ctx/lib/libde_bmp.so libde_hex.so --> ORACLE_HOME/ctx/lib/libde_hex.so libde_ss.so --> ORACLE_HOME/ctx/lib/libde_ss.so libde_tree.so --> ORACLE_HOME/ctx/lib/libde_tree.so libde_vect.so --> ORACLE_HOME/ctx/lib/libde_vect.so libde_wp.so --> ORACLE_HOME/ctx/lib/libde_wp.so libex_bf.so --> ORACLE_HOME/ctx/lib/libex_bf.so libex_catest.so --> ORACLE_HOME/ctx/lib/libex_catest.so libex_edrm.so --> ORACLE_HOME/ctx/lib/libex_edrm.so libex_gdsf.so --> ORACLE_HOME/ctx/lib/libex_gdsf.so libex_h5.so --> ORACLE_HOME/ctx/lib/libex_h5.so libex_html.so --> ORACLE_HOME/ctx/lib/libex_html.so libex_ihtml.so --> ORACLE_HOME/ctx/lib/libex_ihtml.so libex_img.so --> ORACLE_HOME/ctx/lib/libex_img.so libex_itext.so --> ORACLE_HOME/ctx/lib/libex_itext.so libex_ixml.so --> ORACLE_HOME/ctx/lib/libex_ixml.so libex_ml.so --> ORACLE_HOME/ctx/lib/libex_ml.so libex_pagelayout.so --> ORACLE_HOME/ctx/lib/libex_pagelayout.so libex_page.so --> ORACLE_HOME/ctx/lib/libex_page.so libexpatw.so --> ORACLE_HOME/ctx/lib/libexpatw.so libex_xml.so --> ORACLE_HOME/ctx/lib/libex_xml.so libfreetype.so --> ORACLE_HOME/ctx/lib/libfreetype.so libfreetype.so.6 --> ORACLE_HOME/ctx/lib/libfreetype.so.6 libfreetype.so.6.3.13 --> ORACLE_HOME/ctx/lib/libfreetype.so.6.3.13 libib_fpx2.so --> ORACLE_HOME/ctx/lib/libib_fpx2.so libib_gp42.so --> ORACLE_HOME/ctx/lib/libib_gp42.so libib_jpg2.so --> ORACLE_HOME/ctx/lib/libib_jpg2.so libib_pcd2.so --> ORACLE_HOME/ctx/lib/libib_pcd2.so libib_psd2.so --> ORACLE_HOME/ctx/lib/libib_psd2.so libib_xbm2.so --> ORACLE_HOME/ctx/lib/libib_xbm2.so libib_xpm2.so --> ORACLE_HOME/ctx/lib/libib_xpm2.so libib_xwd2.so --> ORACLE_HOME/ctx/lib/libib_xwd2.so libim_cdr2.so --> ORACLE_HOME/ctx/lib/libim_cdr2.so libim_cmx2.so --> ORACLE_HOME/ctx/lib/libim_cmx2.so libim_dsf2.so --> ORACLE_HOME/ctx/lib/libim_dsf2.so libim_fmv2.so --> ORACLE_HOME/ctx/lib/libim_fmv2.so libim_gdf2.so --> ORACLE_HOME/ctx/lib/libim_gdf2.so libim_gem2.so --> ORACLE_HOME/ctx/lib/libim_gem2.so libim_igs2.so --> ORACLE_HOME/ctx/lib/libim_igs2.so libim_met2.so --> ORACLE_HOME/ctx/lib/libim_met2.so libim_pif2.so --> ORACLE_HOME/ctx/lib/libim_pif2.so libim_ps_2.so --> ORACLE_HOME/ctx/lib/libim_ps_2.so libim_psi2.so --> ORACLE_HOME/ctx/lib/libim_psi2.so libim_psz2.so --> ORACLE_HOME/ctx/lib/libim_psz2.so libim_rnd2.so --> ORACLE_HOME/ctx/lib/libim_rnd2.so libim_wmf2.so --> ORACLE_HOME/ctx/lib/libim_wmf2.so libip_hgw2.so --> ORACLE_HOME/ctx/lib/libip_hgw2.so libis_unx2.so --> ORACLE_HOME/ctx/lib/libis_unx2.so liboc_dumper.so --> ORACLE_HOME/ctx/lib/liboc_dumper.so liboc_emul.so --> ORACLE_HOME/ctx/lib/liboc_emul.so liboicomponents.so --> ORACLE_HOME/ctx/lib/liboicomponents.so libos_gd.so --> ORACLE_HOME/ctx/lib/libos_gd.so libos_pdf.so --> ORACLE_HOME/ctx/lib/libos_pdf.so libos_webview.so --> ORACLE_HOME/ctx/lib/libos_webview.so libos_xwin.so --> ORACLE_HOME/ctx/lib/libos_xwin.so libsc_anno.so --> ORACLE_HOME/ctx/lib/libsc_anno.so libsc_ca.so --> ORACLE_HOME/ctx/lib/libsc_ca.so libsc_ch.so --> ORACLE_HOME/ctx/lib/libsc_ch.so libsc_da.so --> ORACLE_HOME/ctx/lib/libsc_da.so libsc_du.so --> ORACLE_HOME/ctx/lib/libsc_du.so libsc_em.so --> ORACLE_HOME/ctx/lib/libsc_em.so libsc_exind.so --> ORACLE_HOME/ctx/lib/libsc_exind.so libsc_ex.so --> ORACLE_HOME/ctx/lib/libsc_ex.so libsc_fa.so --> ORACLE_HOME/ctx/lib/libsc_fa.so libsc_fi.so --> ORACLE_HOME/ctx/lib/libsc_fi.so libsc_fmt.so --> ORACLE_HOME/ctx/lib/libsc_fmt.so libsc_fnt.so --> ORACLE_HOME/ctx/lib/libsc_fnt.so libsc_fut.so --> ORACLE_HOME/ctx/lib/libsc_fut.so libsc_img.so --> ORACLE_HOME/ctx/lib/libsc_img.so libsc_ind.so --> ORACLE_HOME/ctx/lib/libsc_ind.so libsc_lo.so --> ORACLE_HOME/ctx/lib/libsc_lo.so libsc_ra.so --> ORACLE_HOME/ctx/lib/libsc_ra.so libsc_sd.so --> ORACLE_HOME/ctx/lib/libsc_sd.so libsc_ta.so --> ORACLE_HOME/ctx/lib/libsc_ta.so libsc_ut.so --> ORACLE_HOME/ctx/lib/libsc_ut.so libsc_vw.so --> ORACLE_HOME/ctx/lib/libsc_vw.so libsc_xp.so --> ORACLE_HOME/ctx/lib/libsc_xp.so libsc_xt.so --> ORACLE_HOME/ctx/lib/libsc_xt.so libsdflex.so --> ORACLE_HOME/ctx/lib/libsdflex.so libvs_acad.so --> ORACLE_HOME/ctx/lib/libvs_acad.so libvs_acs.so --> ORACLE_HOME/ctx/lib/libvs_acs.so libvs_ami.so --> ORACLE_HOME/ctx/lib/libvs_ami.so libvs_arc.so --> ORACLE_HOME/ctx/lib/libvs_arc.so libvs_asf.so --> ORACLE_HOME/ctx/lib/libvs_asf.so libvs_atm.so --> ORACLE_HOME/ctx/lib/libvs_atm.so libvs_bdr.so --> ORACLE_HOME/ctx/lib/libvs_bdr.so libvs_bmp.so --> ORACLE_HOME/ctx/lib/libvs_bmp.so libvs_cdb.so --> ORACLE_HOME/ctx/lib/libvs_cdb.so libvs_cdrx.so --> ORACLE_HOME/ctx/lib/libvs_cdrx.so libvs_ceos.so --> ORACLE_HOME/ctx/lib/libvs_ceos.so libvs_cgm.so --> ORACLE_HOME/ctx/lib/libvs_cgm.so libvs_cwt.so --> ORACLE_HOME/ctx/lib/libvs_cwt.so libvs_dbs.so --> ORACLE_HOME/ctx/lib/libvs_dbs.so libvs_dez.so --> ORACLE_HOME/ctx/lib/libvs_dez.so libvs_dif.so --> ORACLE_HOME/ctx/lib/libvs_dif.so libvs_drw.so --> ORACLE_HOME/ctx/lib/libvs_drw.so libvs_dxf.so --> ORACLE_HOME/ctx/lib/libvs_dxf.so libvs_dxla.so --> ORACLE_HOME/ctx/lib/libvs_dxla.so libvs_dxlm.so --> ORACLE_HOME/ctx/lib/libvs_dxlm.so libvs_dx.so --> ORACLE_HOME/ctx/lib/libvs_dx.so libvs_emf.so --> ORACLE_HOME/ctx/lib/libvs_emf.so libvs_en4.so --> ORACLE_HOME/ctx/lib/libvs_en4.so libvs_ens.so --> ORACLE_HOME/ctx/lib/libvs_ens.so libvs_enw.so --> ORACLE_HOME/ctx/lib/libvs_enw.so libvs_eps.so --> ORACLE_HOME/ctx/lib/libvs_eps.so libvs_eshr.so --> ORACLE_HOME/ctx/lib/libvs_eshr.so libvs_exe2.so --> ORACLE_HOME/ctx/lib/libvs_exe2.so libvs_fax.so --> ORACLE_HOME/ctx/lib/libvs_fax.so libvs_fcd.so --> ORACLE_HOME/ctx/lib/libvs_fcd.so libvs_fcs.so --> ORACLE_HOME/ctx/lib/libvs_fcs.so libvs_fft.so --> ORACLE_HOME/ctx/lib/libvs_fft.so libvs_flex.so --> ORACLE_HOME/ctx/lib/libvs_flex.so libvs_flw.so --> ORACLE_HOME/ctx/lib/libvs_flw.so libvs_fwk.so --> ORACLE_HOME/ctx/lib/libvs_fwk.so libvs_gdsf.so --> ORACLE_HOME/ctx/lib/libvs_gdsf.so libvs_gif.so --> ORACLE_HOME/ctx/lib/libvs_gif.so libvs_gzip.so --> ORACLE_HOME/ctx/lib/libvs_gzip.so libvs_hgs.so --> ORACLE_HOME/ctx/lib/libvs_hgs.so libvs_html.so --> ORACLE_HOME/ctx/lib/libvs_html.so libvs_hwp2.so --> ORACLE_HOME/ctx/lib/libvs_hwp2.so libvs_hwp.so --> ORACLE_HOME/ctx/lib/libvs_hwp.so libvs_ich6.so --> ORACLE_HOME/ctx/lib/libvs_ich6.so libvs_ich.so --> ORACLE_HOME/ctx/lib/libvs_ich.so libvs_id3.so --> ORACLE_HOME/ctx/lib/libvs_id3.so libvs_img.so --> ORACLE_HOME/ctx/lib/libvs_img.so libvs_indd.so --> ORACLE_HOME/ctx/lib/libvs_indd.so libvs_inx.so --> ORACLE_HOME/ctx/lib/libvs_inx.so libvs_iwok13.so --> ORACLE_HOME/ctx/lib/libvs_iwok13.so libvs_iwok.so --> ORACLE_HOME/ctx/lib/libvs_iwok.so libvs_iwon.so --> ORACLE_HOME/ctx/lib/libvs_iwon.so libvs_iwop.so --> ORACLE_HOME/ctx/lib/libvs_iwop.so libvs_iwp.so --> ORACLE_HOME/ctx/lib/libvs_iwp.so libvs_jbg2.so --> ORACLE_HOME/ctx/lib/libvs_jbg2.so libvs_jp2.so --> ORACLE_HOME/ctx/lib/libvs_jp2.so libvs_jw.so --> ORACLE_HOME/ctx/lib/libvs_jw.so libvs_leg.so --> ORACLE_HOME/ctx/lib/libvs_leg.so libvs_lwp7.so --> ORACLE_HOME/ctx/lib/libvs_lwp7.so libvs_lzh.so --> ORACLE_HOME/ctx/lib/libvs_lzh.so libvs_m11.so --> ORACLE_HOME/ctx/lib/libvs_m11.so libvs_manu.so --> ORACLE_HOME/ctx/lib/libvs_manu.so libvs_mbox.so --> ORACLE_HOME/ctx/lib/libvs_mbox.so libvs_mcw.so --> ORACLE_HOME/ctx/lib/libvs_mcw.so libvs_mdb.so --> ORACLE_HOME/ctx/lib/libvs_mdb.so libvs_mif.so --> ORACLE_HOME/ctx/lib/libvs_mif.so libvs_mime.so --> ORACLE_HOME/ctx/lib/libvs_mime.so libvs_mm4.so --> ORACLE_HOME/ctx/lib/libvs_mm4.so libvs_mmfn.so --> ORACLE_HOME/ctx/lib/libvs_mmfn.so libvs_mm.so --> ORACLE_HOME/ctx/lib/libvs_mm.so libvs_mpp.so --> ORACLE_HOME/ctx/lib/libvs_mpp.so libvs_mp.so --> ORACLE_HOME/ctx/lib/libvs_mp.so libvs_msg.so --> ORACLE_HOME/ctx/lib/libvs_msg.so libvs_msw.so --> ORACLE_HOME/ctx/lib/libvs_msw.so libvs_mwkd.so --> ORACLE_HOME/ctx/lib/libvs_mwkd.so libvs_mwks.so --> ORACLE_HOME/ctx/lib/libvs_mwks.so libvs_mwp2.so --> ORACLE_HOME/ctx/lib/libvs_mwp2.so libvs_mwpf.so --> ORACLE_HOME/ctx/lib/libvs_mwpf.so libvs_mwrk.so --> ORACLE_HOME/ctx/lib/libvs_mwrk.so libvs_olm.so --> ORACLE_HOME/ctx/lib/libvs_olm.so libvs_one.so --> ORACLE_HOME/ctx/lib/libvs_one.so libvs_ow.so --> ORACLE_HOME/ctx/lib/libvs_ow.so libvs_pbm.so --> ORACLE_HOME/ctx/lib/libvs_pbm.so libvs_pcl.so --> ORACLE_HOME/ctx/lib/libvs_pcl.so libvs_pcx.so --> ORACLE_HOME/ctx/lib/libvs_pcx.so libvs_pdfi.so --> ORACLE_HOME/ctx/lib/libvs_pdfi.so libvs_pdf.so --> ORACLE_HOME/ctx/lib/libvs_pdf.so libvs_pdx.so --> ORACLE_HOME/ctx/lib/libvs_pdx.so libvs_pfs.so --> ORACLE_HOME/ctx/lib/libvs_pfs.so libvs_pgl.so --> ORACLE_HOME/ctx/lib/libvs_pgl.so libvs_pic.so --> ORACLE_HOME/ctx/lib/libvs_pic.so libvs_pict.so --> ORACLE_HOME/ctx/lib/libvs_pict.so libvs_png.so --> ORACLE_HOME/ctx/lib/libvs_png.so libvs_pntg.so --> ORACLE_HOME/ctx/lib/libvs_pntg.so libvs_pp12.so --> ORACLE_HOME/ctx/lib/libvs_pp12.so libvs_pp2.so --> ORACLE_HOME/ctx/lib/libvs_pp2.so libvs_pp7.so --> ORACLE_HOME/ctx/lib/libvs_pp7.so libvs_pp97.so --> ORACLE_HOME/ctx/lib/libvs_pp97.so libvs_ppl.so --> ORACLE_HOME/ctx/lib/libvs_ppl.so libvs_psd.so --> ORACLE_HOME/ctx/lib/libvs_psd.so libvs_pstf.so --> ORACLE_HOME/ctx/lib/libvs_pstf.so libvs_pst.so --> ORACLE_HOME/ctx/lib/libvs_pst.so libvs_qad.so --> ORACLE_HOME/ctx/lib/libvs_qad.so libvs_qa.so --> ORACLE_HOME/ctx/lib/libvs_qa.so libvs_qp6.so --> ORACLE_HOME/ctx/lib/libvs_qp6.so libvs_qp9.so --> ORACLE_HOME/ctx/lib/libvs_qp9.so libvs_qt.so --> ORACLE_HOME/ctx/lib/libvs_qt.so libvs_rar.so --> ORACLE_HOME/ctx/lib/libvs_rar.so libvs_ras.so --> ORACLE_HOME/ctx/lib/libvs_ras.so libvs_rbs.so --> ORACLE_HOME/ctx/lib/libvs_rbs.so libvs_rft.so --> ORACLE_HOME/ctx/lib/libvs_rft.so libvs_rfx.so --> ORACLE_HOME/ctx/lib/libvs_rfx.so libvs_riff.so --> ORACLE_HOME/ctx/lib/libvs_riff.so libvs_rpix.so --> ORACLE_HOME/ctx/lib/libvs_rpix.so libvs_rtf.so --> ORACLE_HOME/ctx/lib/libvs_rtf.so libvs_sam.so --> ORACLE_HOME/ctx/lib/libvs_sam.so libvs_sc5.so --> ORACLE_HOME/ctx/lib/libvs_sc5.so libvs_sdw.so --> ORACLE_HOME/ctx/lib/libvs_sdw.so libvs_shw3.so --> ORACLE_HOME/ctx/lib/libvs_shw3.so libvs_smd.so --> ORACLE_HOME/ctx/lib/libvs_smd.so libvs_sms.so --> ORACLE_HOME/ctx/lib/libvs_sms.so libvs_smt.so --> ORACLE_HOME/ctx/lib/libvs_smt.so libvs_snap.so --> ORACLE_HOME/ctx/lib/libvs_snap.so libvs_so6.so --> ORACLE_HOME/ctx/lib/libvs_so6.so libvs_soc6.so --> ORACLE_HOME/ctx/lib/libvs_soc6.so libvs_soc.so --> ORACLE_HOME/ctx/lib/libvs_soc.so libvs_soi6.so --> ORACLE_HOME/ctx/lib/libvs_soi6.so libvs_soi.so --> ORACLE_HOME/ctx/lib/libvs_soi.so libvs_sow.so --> ORACLE_HOME/ctx/lib/libvs_sow.so libvs_spt.so --> ORACLE_HOME/ctx/lib/libvs_spt.so libvs_ssml.so --> ORACLE_HOME/ctx/lib/libvs_ssml.so libvs_swf.so --> ORACLE_HOME/ctx/lib/libvs_swf.so libvs_taz.so --> ORACLE_HOME/ctx/lib/libvs_taz.so libvs_text.so --> ORACLE_HOME/ctx/lib/libvs_text.so libvs_tga.so --> ORACLE_HOME/ctx/lib/libvs_tga.so libvs_tif6.so --> ORACLE_HOME/ctx/lib/libvs_tif6.so libvs_tw.so --> ORACLE_HOME/ctx/lib/libvs_tw.so libvs_txt.so --> ORACLE_HOME/ctx/lib/libvs_txt.so libvs_vcrd.so --> ORACLE_HOME/ctx/lib/libvs_vcrd.so libvs_viso.so --> ORACLE_HOME/ctx/lib/libvs_viso.so libvs_vsdx.so --> ORACLE_HOME/ctx/lib/libvs_vsdx.so libvs_vw3.so --> ORACLE_HOME/ctx/lib/libvs_vw3.so libvs_w12.so --> ORACLE_HOME/ctx/lib/libvs_w12.so libvs_w6.so --> ORACLE_HOME/ctx/lib/libvs_w6.so libvs_w97.so --> ORACLE_HOME/ctx/lib/libvs_w97.so libvs_wbmp.so --> ORACLE_HOME/ctx/lib/libvs_wbmp.so libvs_wg2.so --> ORACLE_HOME/ctx/lib/libvs_wg2.so libvs_wk4.so --> ORACLE_HOME/ctx/lib/libvs_wk4.so libvs_wk6.so --> ORACLE_HOME/ctx/lib/libvs_wk6.so libvs_wks.so --> ORACLE_HOME/ctx/lib/libvs_wks.so libvs_wmf.so --> ORACLE_HOME/ctx/lib/libvs_wmf.so libvs_wml.so --> ORACLE_HOME/ctx/lib/libvs_wml.so libvs_wm.so --> ORACLE_HOME/ctx/lib/libvs_wm.so libvs_word.so --> ORACLE_HOME/ctx/lib/libvs_word.so libvs_work.so --> ORACLE_HOME/ctx/lib/libvs_work.so libvs_wp5.so --> ORACLE_HOME/ctx/lib/libvs_wp5.so libvs_wp6.so --> ORACLE_HOME/ctx/lib/libvs_wp6.so libvs_wpf.so --> ORACLE_HOME/ctx/lib/libvs_wpf.so libvs_wpg2.so --> ORACLE_HOME/ctx/lib/libvs_wpg2.so libvs_wpg.so --> ORACLE_HOME/ctx/lib/libvs_wpg.so libvs_wpl.so --> ORACLE_HOME/ctx/lib/libvs_wpl.so libvs_wpml.so --> ORACLE_HOME/ctx/lib/libvs_wpml.so libvs_wpw.so --> ORACLE_HOME/ctx/lib/libvs_wpw.so libvs_wrk9.so --> ORACLE_HOME/ctx/lib/libvs_wrk9.so libvs_ws2.so --> ORACLE_HOME/ctx/lib/libvs_ws2.so libvs_ws.so --> ORACLE_HOME/ctx/lib/libvs_ws.so libvs_xl12.so --> ORACLE_HOME/ctx/lib/libvs_xl12.so libvs_xl5.so --> ORACLE_HOME/ctx/lib/libvs_xl5.so libvs_xlsb.so --> ORACLE_HOME/ctx/lib/libvs_xlsb.so libvs_xml.so --> ORACLE_HOME/ctx/lib/libvs_xml.so libvs_xmp.so --> ORACLE_HOME/ctx/lib/libvs_xmp.so libvs_xps.so --> ORACLE_HOME/ctx/lib/libvs_xps.so libvs_xy.so --> ORACLE_HOME/ctx/lib/libvs_xy.so libvs_yim.so --> ORACLE_HOME/ctx/lib/libvs_yim.so libvs_zip.so --> ORACLE_HOME/ctx/lib/libvs_zip.so libwv_core.so --> ORACLE_HOME/ctx/lib/libwv_core.so libwv_gdlib.so --> ORACLE_HOME/ctx/lib/libwv_gdlib.so oit_font_metrics.db --> ORACLE_HOME/ctx/lib/oit_font_metrics.db textexport --> ORACLE_HOME/ctx/lib/textexport unxrc --> ORACLE_HOME/ctx/lib/unxrc ins_ctx.mk --> ORACLE_HOME/ctx/lib/ictxhx ins_ctx.mk --> ORACLE_HOME/ctx/lib/ictxload ins_ctx.mk --> ORACLE_HOME/ctx/lib/ictxlc ins_ctx.mk --> ORACLE_HOME/ctx/lib/ictxkbtc initjms.sql --> ORACLE_HOME/rdbms/admin/initjms.sql dgmus.msb --> ORACLE_HOME/rdbms/mesg/dgmus.msb dgmus.msg --> ORACLE_HOME/rdbms/mesg/dgmus.msg ins_net_client.mk --> ORACLE_HOME/network/lib/client_sharedlib ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iplshprof ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/isqlldr ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iextproc /nnfh.o --> ORACLE_HOME/lib/libnhost12.a /ntz.o --> ORACLE_HOME/lib/libntcps12.a /nsz.o --> ORACLE_HOME/lib/libn12.a /sntp.o --> ORACLE_HOME/lib/libnbeq12.a ins_net_client.mk --> ORACLE_HOME/network/lib/client_sharedlib diskmon --> ORACLE_HOME/bin/diskmon libnnz12.so --> ORACLE_HOME/lib/libnnz12.so lxinst --> ORACLE_HOME/bin/lxinst oci.h --> ORACLE_HOME/rdbms/public/oci.h schema --> ORACLE_HOME/bin/schema xml --> ORACLE_HOME/bin/xml xmldiff --> ORACLE_HOME/bin/xmldiff xmlpatch --> ORACLE_HOME/bin/xmlpatch xmlcg --> ORACLE_HOME/bin/xmlcg xsl --> ORACLE_HOME/bin/xsl xvm --> ORACLE_HOME/bin/xvm /kqp461.o --> ORACLE_HOME/lib/libserver12.a /kqp460.o --> ORACLE_HOME/lib/libserver12.a /kcb.o --> ORACLE_HOME/lib/libserver12.a /kcbb.o --> ORACLE_HOME/lib/libserver12.a /kcbflc.o --> ORACLE_HOME/lib/libserver12.a /kcbi.o --> ORACLE_HOME/lib/libserver12.a /kcbk.o --> ORACLE_HOME/lib/libserver12.a /kcbl.o --> ORACLE_HOME/lib/libserver12.a /kcbm.o --> ORACLE_HOME/lib/libserver12.a /kcbo.o --> ORACLE_HOME/lib/libserver12.a /kcbr.o --> ORACLE_HOME/lib/libserver12.a /kcbs.o --> ORACLE_HOME/lib/libserver12.a /kcbt.o --> ORACLE_HOME/lib/libserver12.a /kcbv.o --> ORACLE_HOME/lib/libserver12.a /kcbw.o --> ORACLE_HOME/lib/libserver12.a /kcbz.o --> ORACLE_HOME/lib/libserver12.a /kcl.o --> ORACLE_HOME/lib/libserver12.a /kjac.o --> ORACLE_HOME/lib/libserver12.a /kjxgg.o --> ORACLE_HOME/lib/libserver12.a /kjxgm.o --> ORACLE_HOME/lib/libserver12.a /kjxgna.o --> ORACLE_HOME/lib/libserver12.a /kjxgr.o --> ORACLE_HOME/lib/libserver12.a /kjzch.o --> ORACLE_HOME/lib/libserver12.a /kjzd.o --> ORACLE_HOME/lib/libserver12.a /kjzf.o --> ORACLE_HOME/lib/libserver12.a /kjzn.o --> ORACLE_HOME/lib/libserver12.a /kjznrmn.o --> ORACLE_HOME/lib/libserver12.a /kjb.o --> ORACLE_HOME/lib/libserver12.a /kjbdom.o --> ORACLE_HOME/lib/libserver12.a /kjbl.o --> ORACLE_HOME/lib/libserver12.a /kjbm.o --> ORACLE_HOME/lib/libserver12.a /kjbr.o --> ORACLE_HOME/lib/libserver12.a /kjdr.o --> ORACLE_HOME/lib/libserver12.a /kjfc.o --> ORACLE_HOME/lib/libserver12.a /kjfm.o --> ORACLE_HOME/lib/libserver12.a /kjgcr.o --> ORACLE_HOME/lib/libserver12.a /kji.o --> ORACLE_HOME/lib/libserver12.a /kjkm.o --> ORACLE_HOME/lib/libserver12.a /kjl.o --> ORACLE_HOME/lib/libserver12.a /kjm.o --> ORACLE_HOME/lib/libserver12.a /kjr.o --> ORACLE_HOME/lib/libserver12.a /kjs.o --> ORACLE_HOME/lib/libserver12.a /kju.o --> ORACLE_HOME/lib/libserver12.a /kjx.o --> ORACLE_HOME/lib/libserver12.a /dgl.o --> ORACLE_HOME/lib/libserver12.a /dglr.o --> ORACLE_HOME/lib/libserver12.a /kupd.o --> ORACLE_HOME/lib/libserver12.a /kupfio.o --> ORACLE_HOME/lib/libserver12.a /expcmo.o --> ORACLE_HOME/lib/libserver12.a /evaa2g.o --> ORACLE_HOME/lib/libserver12.a /evah2p.o --> ORACLE_HOME/lib/libserver12.a /evar2v.o --> ORACLE_HOME/lib/libserver12.a /odfdef.o --> ORACLE_HOME/lib/libserver12.a /sagesql.o --> ORACLE_HOME/lib/libserver12.a /kxto.o --> ORACLE_HOME/lib/libserver12.a /kkki.o --> ORACLE_HOME/lib/libserver12.a /kkdc.o --> ORACLE_HOME/lib/libserver12.a /kkdl.o --> ORACLE_HOME/lib/libserver12.a /kkm.o --> ORACLE_HOME/lib/libserver12.a /kkm2.o --> ORACLE_HOME/lib/libserver12.a /qsmkz.o --> ORACLE_HOME/lib/libserver12.a /kxs.o --> ORACLE_HOME/lib/libserver12.a /kxs1.o --> ORACLE_HOME/lib/libserver12.a /kql.o --> ORACLE_HOME/lib/libserver12.a /kqld.o --> ORACLE_HOME/lib/libserver12.a /kqlp.o --> ORACLE_HOME/lib/libserver12.a /kkpo.o --> ORACLE_HOME/lib/libserver12.a /kkpod.o --> ORACLE_HOME/lib/libserver12.a /kkxf.o --> ORACLE_HOME/lib/libserver12.a /kkxs.o --> ORACLE_HOME/lib/libserver12.a /kqd.o --> ORACLE_HOME/lib/libserver12.a /kqr.o --> ORACLE_HOME/lib/libserver12.a /kks.o --> ORACLE_HOME/lib/libserver12.a /kks1.o --> ORACLE_HOME/lib/libserver12.a /kksa.o --> ORACLE_HOME/lib/libserver12.a /akmdrv.o --> ORACLE_HOME/lib/libserver12.a /atb.o --> ORACLE_HOME/lib/libserver12.a /atbo.o --> ORACLE_HOME/lib/libserver12.a /ctc.o --> ORACLE_HOME/lib/libserver12.a /cvw.o --> ORACLE_HOME/lib/libserver12.a /kkae.o --> ORACLE_HOME/lib/libserver12.a /kkb.o --> ORACLE_HOME/lib/libserver12.a /kkcn.o --> ORACLE_HOME/lib/libserver12.a /kkeaf.o --> ORACLE_HOME/lib/libserver12.a /kkp.o --> ORACLE_HOME/lib/libserver12.a /kkt.o --> ORACLE_HOME/lib/libserver12.a /dmqx.o --> ORACLE_HOME/lib/libserver12.a /rfafo.o --> ORACLE_HOME/lib/libserver12.a /rfcfg.o --> ORACLE_HOME/lib/libserver12.a /rfipc.o --> ORACLE_HOME/lib/libserver12.a /rfmboot.o --> ORACLE_HOME/lib/libserver12.a /rfmedd.o --> ORACLE_HOME/lib/libserver12.a /rfmon.o --> ORACLE_HOME/lib/libserver12.a /rfmrac.o --> ORACLE_HOME/lib/libserver12.a /rfmrole.o --> ORACLE_HOME/lib/libserver12.a /rfmss.o --> ORACLE_HOME/lib/libserver12.a /rfmte.o --> ORACLE_HOME/lib/libserver12.a /rfmutil.o --> ORACLE_HOME/lib/libserver12.a /rfrdb.o --> ORACLE_HOME/lib/libserver12.a /rfrxpt.o --> ORACLE_HOME/lib/libserver12.a /rfsicd.o --> ORACLE_HOME/lib/libserver12.a /rfsx.o --> ORACLE_HOME/lib/libserver12.a /gwm.o --> ORACLE_HOME/lib/libserver12.a /gws.o --> ORACLE_HOME/lib/libserver12.a /jskq.o --> ORACLE_HOME/lib/libserver12.a /jskr.o --> ORACLE_HOME/lib/libserver12.a /jsks.o --> ORACLE_HOME/lib/libserver12.a /jscr.o --> ORACLE_HOME/lib/libserver12.a /krvrd.o --> ORACLE_HOME/lib/libserver12.a /krvrda.o --> ORACLE_HOME/lib/libserver12.a /krvx.o --> ORACLE_HOME/lib/libserver12.a /krvxb.o --> ORACLE_HOME/lib/libserver12.a /krvxp.o --> ORACLE_HOME/lib/libserver12.a /krvxr.o --> ORACLE_HOME/lib/libserver12.a /krvxs.o --> ORACLE_HOME/lib/libserver12.a /kmm.o --> ORACLE_HOME/lib/libserver12.a /kokl.o --> ORACLE_HOME/lib/libserver12.a /kokl2.o --> ORACLE_HOME/lib/libserver12.a /kokt.o --> ORACLE_HOME/lib/libserver12.a /qxcdt.o --> ORACLE_HOME/lib/libserver12.a /qxdid.o --> ORACLE_HOME/lib/libserver12.a /kokb2.o --> ORACLE_HOME/lib/libserver12.a /koke.o --> ORACLE_HOME/lib/libserver12.a /kokeg.o --> ORACLE_HOME/lib/libserver12.a /koklv.o --> ORACLE_HOME/lib/libserver12.a /kokq.o --> ORACLE_HOME/lib/libserver12.a /koks.o --> ORACLE_HOME/lib/libserver12.a /koks2.o --> ORACLE_HOME/lib/libserver12.a /koku.o --> ORACLE_HOME/lib/libserver12.a /kokv.o --> ORACLE_HOME/lib/libserver12.a /qxidm.o --> ORACLE_HOME/lib/libserver12.a /qxxq.o --> ORACLE_HOME/lib/libserver12.a /xscoz.o --> ORACLE_HOME/lib/libserver12.a /kwqa.o --> ORACLE_HOME/lib/libserver12.a /kwqbc.o --> ORACLE_HOME/lib/libserver12.a /kwqbe.o --> ORACLE_HOME/lib/libserver12.a /kwqbm.o --> ORACLE_HOME/lib/libserver12.a /kwqi.o --> ORACLE_HOME/lib/libserver12.a /kwqic.o --> ORACLE_HOME/lib/libserver12.a /kwqid.o --> ORACLE_HOME/lib/libserver12.a /kwqie.o --> ORACLE_HOME/lib/libserver12.a /kwqj.o --> ORACLE_HOME/lib/libserver12.a /kwqm.o --> ORACLE_HOME/lib/libserver12.a /kwqp.o --> ORACLE_HOME/lib/libserver12.a /kwqv.o --> ORACLE_HOME/lib/libserver12.a /kwqit.o --> ORACLE_HOME/lib/libserver12.a /kwqmn.o --> ORACLE_HOME/lib/libserver12.a /kwra.o --> ORACLE_HOME/lib/libserver12.a /kke.o --> ORACLE_HOME/lib/libserver12.a /kkecst.o --> ORACLE_HOME/lib/libserver12.a /kkespls.o --> ORACLE_HOME/lib/libserver12.a /kkest.o --> ORACLE_HOME/lib/libserver12.a /kkeutl.o --> ORACLE_HOME/lib/libserver12.a /kkex.o --> ORACLE_HOME/lib/libserver12.a /kkfi.o --> ORACLE_HOME/lib/libserver12.a /kko.o --> ORACLE_HOME/lib/libserver12.a /kkoap.o --> ORACLE_HOME/lib/libserver12.a /kkoar.o --> ORACLE_HOME/lib/libserver12.a /kkobm.o --> ORACLE_HOME/lib/libserver12.a /kkodp.o --> ORACLE_HOME/lib/libserver12.a /kkojm.o --> ORACLE_HOME/lib/libserver12.a /kkoop.o --> ORACLE_HOME/lib/libserver12.a /kkostr.o --> ORACLE_HOME/lib/libserver12.a /apai.o --> ORACLE_HOME/lib/libserver12.a /qosa.o --> ORACLE_HOME/lib/libserver12.a /qosd.o --> ORACLE_HOME/lib/libserver12.a /qosh.o --> ORACLE_HOME/lib/libserver12.a /qosp.o --> ORACLE_HOME/lib/libserver12.a /kkqanj.o --> ORACLE_HOME/lib/libserver12.a /kkqct.o --> ORACLE_HOME/lib/libserver12.a /kkqg.o --> ORACLE_HOME/lib/libserver12.a /kkqgbp.o --> ORACLE_HOME/lib/libserver12.a /kkqje.o --> ORACLE_HOME/lib/libserver12.a /kkqjf.o --> ORACLE_HOME/lib/libserver12.a /kkqjpd.o --> ORACLE_HOME/lib/libserver12.a /kkqore.o --> ORACLE_HOME/lib/libserver12.a /kkqs.o --> ORACLE_HOME/lib/libserver12.a /kkqs1.o --> ORACLE_HOME/lib/libserver12.a /kkqs2.o --> ORACLE_HOME/lib/libserver12.a /kkqst.o --> ORACLE_HOME/lib/libserver12.a /kkqsuf.o --> ORACLE_HOME/lib/libserver12.a /kkqte.o --> ORACLE_HOME/lib/libserver12.a /kkqtutl.o --> ORACLE_HOME/lib/libserver12.a /kkqu.o --> ORACLE_HOME/lib/libserver12.a /kkqvm.o --> ORACLE_HOME/lib/libserver12.a /kkqvt.o --> ORACLE_HOME/lib/libserver12.a /kkqzm.o --> ORACLE_HOME/lib/libserver12.a /nsoqbc.o --> ORACLE_HOME/lib/libserver12.a /sd.o --> ORACLE_HOME/lib/libserver12.a /skptch.o --> ORACLE_HOME/lib/libserver12.a /sqmuns.o --> ORACLE_HOME/lib/libserver12.a /kpdb.o --> ORACLE_HOME/lib/libserver12.a /kpdba.o --> ORACLE_HOME/lib/libserver12.a /kpdbc.o --> ORACLE_HOME/lib/libserver12.a /kpdbcv.o --> ORACLE_HOME/lib/libserver12.a /kpdbd.o --> ORACLE_HOME/lib/libserver12.a /kpdbe.o --> ORACLE_HOME/lib/libserver12.a /kpdbf.o --> ORACLE_HOME/lib/libserver12.a /kpdbfd.o --> ORACLE_HOME/lib/libserver12.a /kpdbicd.o --> ORACLE_HOME/lib/libserver12.a /kpdbldp.o --> ORACLE_HOME/lib/libserver12.a /kpdbutl.o --> ORACLE_HOME/lib/libserver12.a /kpdbv.o --> ORACLE_HOME/lib/libserver12.a /kpdbxmlt.o --> ORACLE_HOME/lib/libserver12.a /kpbf.o --> ORACLE_HOME/lib/libserver12.a /psdmsc.o --> ORACLE_HOME/lib/libserver12.a /psdpgi.o --> ORACLE_HOME/lib/libserver12.a /ddfnet.o --> ORACLE_HOME/lib/libserver12.a /kpki.o --> ORACLE_HOME/lib/libserver12.a /kpkvd.o --> ORACLE_HOME/lib/libserver12.a /kpn.o --> ORACLE_HOME/lib/libserver12.a /kpoa.o --> ORACLE_HOME/lib/libserver12.a /kpoal8.o --> ORACLE_HOME/lib/libserver12.a /kpoaq.o --> ORACLE_HOME/lib/libserver12.a /kpodp.o --> ORACLE_HOME/lib/libserver12.a /kpolon.o --> ORACLE_HOME/lib/libserver12.a /kpon.o --> ORACLE_HOME/lib/libserver12.a /kkxwtp.o --> ORACLE_HOME/lib/libserver12.a /opicui.o --> ORACLE_HOME/lib/libserver12.a /opiexe.o --> ORACLE_HOME/lib/libserver12.a /opiior.o --> ORACLE_HOME/lib/libserver12.a /opipls.o --> ORACLE_HOME/lib/libserver12.a /opix.o --> ORACLE_HOME/lib/libserver12.a /psdhngt.o --> ORACLE_HOME/lib/libserver12.a /psdicd.o --> ORACLE_HOME/lib/libserver12.a /plswarn.o --> ORACLE_HOME/lib/libserver12.a /kpsl.o --> ORACLE_HOME/lib/libserver12.a /kf.o --> ORACLE_HOME/lib/libserver12.a /kfa.o --> ORACLE_HOME/lib/libserver12.a /kfas.o --> ORACLE_HOME/lib/libserver12.a /kfbf.o --> ORACLE_HOME/lib/libserver12.a /kfc.o --> ORACLE_HOME/lib/libserver12.a /kfcb.o --> ORACLE_HOME/lib/libserver12.a /kfcl.o --> ORACLE_HOME/lib/libserver12.a /kfd.o --> ORACLE_HOME/lib/libserver12.a /kfda.o --> ORACLE_HOME/lib/libserver12.a /kfdh.o --> ORACLE_HOME/lib/libserver12.a /kfdp.o --> ORACLE_HOME/lib/libserver12.a /kfds.o --> ORACLE_HOME/lib/libserver12.a /kfdus.o --> ORACLE_HOME/lib/libserver12.a /kfdva.o --> ORACLE_HOME/lib/libserver12.a /kfdvav.o --> ORACLE_HOME/lib/libserver12.a /kfdvf.o --> ORACLE_HOME/lib/libserver12.a /kfdx.o --> ORACLE_HOME/lib/libserver12.a /kfe.o --> ORACLE_HOME/lib/libserver12.a /kff.o --> ORACLE_HOME/lib/libserver12.a /kffb.o --> ORACLE_HOME/lib/libserver12.a /kffd.o --> ORACLE_HOME/lib/libserver12.a /kffg.o --> ORACLE_HOME/lib/libserver12.a /kffm.o --> ORACLE_HOME/lib/libserver12.a /kffmap.o --> ORACLE_HOME/lib/libserver12.a /kffscrub.o --> ORACLE_HOME/lib/libserver12.a /kfg.o --> ORACLE_HOME/lib/libserver12.a /kfgb.o --> ORACLE_HOME/lib/libserver12.a /kfgrp.o --> ORACLE_HOME/lib/libserver12.a /kfhm.o --> ORACLE_HOME/lib/libserver12.a /kfi.o --> ORACLE_HOME/lib/libserver12.a /kfiac.o --> ORACLE_HOME/lib/libserver12.a /kfias.o --> ORACLE_HOME/lib/libserver12.a /kfio.o --> ORACLE_HOME/lib/libserver12.a /kfio0.o --> ORACLE_HOME/lib/libserver12.a /kfioer.o --> ORACLE_HOME/lib/libserver12.a /kfiof.o --> ORACLE_HOME/lib/libserver12.a /kfis.o --> ORACLE_HOME/lib/libserver12.a /kfkio.o --> ORACLE_HOME/lib/libserver12.a /kfks.o --> ORACLE_HOME/lib/libserver12.a /kfksubs.o --> ORACLE_HOME/lib/libserver12.a /kfmd.o --> ORACLE_HOME/lib/libserver12.a /kfn.o --> ORACLE_HOME/lib/libserver12.a /kfnb.o --> ORACLE_HOME/lib/libserver12.a /kfnc.o --> ORACLE_HOME/lib/libserver12.a /kfnm.o --> ORACLE_HOME/lib/libserver12.a /kfns.o --> ORACLE_HOME/lib/libserver12.a /kfpkg.o --> ORACLE_HOME/lib/libserver12.a /kfqg.o --> ORACLE_HOME/lib/libserver12.a /kfr.o --> ORACLE_HOME/lib/libserver12.a /kfrb.o --> ORACLE_HOME/lib/libserver12.a /kfrc.o --> ORACLE_HOME/lib/libserver12.a /kfrcv.o --> ORACLE_HOME/lib/libserver12.a /kfsp.o --> ORACLE_HOME/lib/libserver12.a /kftha.o --> ORACLE_HOME/lib/libserver12.a /kftm.o --> ORACLE_HOME/lib/libserver12.a /kfv.o --> ORACLE_HOME/lib/libserver12.a /kfvioc.o --> ORACLE_HOME/lib/libserver12.a /kfvsd.o --> ORACLE_HOME/lib/libserver12.a /kfvsu.o --> ORACLE_HOME/lib/libserver12.a /kfvx.o --> ORACLE_HOME/lib/libserver12.a /kfx.o --> ORACLE_HOME/lib/libserver12.a /kfz.o --> ORACLE_HOME/lib/libserver12.a /kfzp.o --> ORACLE_HOME/lib/libserver12.a /kkn.o --> ORACLE_HOME/lib/libserver12.a /kau.o --> ORACLE_HOME/lib/libserver12.a /kdbl.o --> ORACLE_HOME/lib/libserver12.a /kdblc.o --> ORACLE_HOME/lib/libserver12.a /kdblm.o --> ORACLE_HOME/lib/libserver12.a /kdd.o --> ORACLE_HOME/lib/libserver12.a /kdilm.o --> ORACLE_HOME/lib/libserver12.a /kdmfs.o --> ORACLE_HOME/lib/libserver12.a /kdmime.o --> ORACLE_HOME/lib/libserver12.a /kdmr.o --> ORACLE_HOME/lib/libserver12.a /kdmu.o --> ORACLE_HOME/lib/libserver12.a /kdo.o --> ORACLE_HOME/lib/libserver12.a /kds.o --> ORACLE_HOME/lib/libserver12.a /kdst.o --> ORACLE_HOME/lib/libserver12.a /kdstrc.o --> ORACLE_HOME/lib/libserver12.a /kdt.o --> ORACLE_HOME/lib/libserver12.a /kdu.o --> ORACLE_HOME/lib/libserver12.a /kdza.o --> ORACLE_HOME/lib/libserver12.a /kdzh.o --> ORACLE_HOME/lib/libserver12.a /kudbvs.o --> ORACLE_HOME/lib/libserver12.a /kdi.o --> ORACLE_HOME/lib/libserver12.a /kdic.o --> ORACLE_HOME/lib/libserver12.a /kdio.o --> ORACLE_HOME/lib/libserver12.a /kdir.o --> ORACLE_HOME/lib/libserver12.a /kdis.o --> ORACLE_HOME/lib/libserver12.a /kdiss.o --> ORACLE_HOME/lib/libserver12.a /kdlf.o --> ORACLE_HOME/lib/libserver12.a /kdli.o --> ORACLE_HOME/lib/libserver12.a /kdlw.o --> ORACLE_HOME/lib/libserver12.a /kdlxdup.o --> ORACLE_HOME/lib/libserver12.a /kkpap.o --> ORACLE_HOME/lib/libserver12.a /kxtt.o --> ORACLE_HOME/lib/libserver12.a /kdn.o --> ORACLE_HOME/lib/libserver12.a /kcc.o --> ORACLE_HOME/lib/libserver12.a /kccr.o --> ORACLE_HOME/lib/libserver12.a /kcf.o --> ORACLE_HOME/lib/libserver12.a /kcffo.o --> ORACLE_HOME/lib/libserver12.a /kcfis.o --> ORACLE_HOME/lib/libserver12.a /kcm.o --> ORACLE_HOME/lib/libserver12.a /kco.o --> ORACLE_HOME/lib/libserver12.a /kcp.o --> ORACLE_HOME/lib/libserver12.a /kcra.o --> ORACLE_HOME/lib/libserver12.a /kcrf.o --> ORACLE_HOME/lib/libserver12.a /kcrfr.o --> ORACLE_HOME/lib/libserver12.a /kcrfw.o --> ORACLE_HOME/lib/libserver12.a /kcrlc.o --> ORACLE_HOME/lib/libserver12.a /kcrm.o --> ORACLE_HOME/lib/libserver12.a /kcrp.o --> ORACLE_HOME/lib/libserver12.a /kcs.o --> ORACLE_HOME/lib/libserver12.a /kcv.o --> ORACLE_HOME/lib/libserver12.a /kcvfdb.o --> ORACLE_HOME/lib/libserver12.a /kcvs.o --> ORACLE_HOME/lib/libserver12.a /kr_lost_write.o --> ORACLE_HOME/lib/libserver12.a /krb.o --> ORACLE_HOME/lib/libserver12.a /krbabr.o --> ORACLE_HOME/lib/libserver12.a /krbb.o --> ORACLE_HOME/lib/libserver12.a /krbc.o --> ORACLE_HOME/lib/libserver12.a /krbd.o --> ORACLE_HOME/lib/libserver12.a /krbi.o --> ORACLE_HOME/lib/libserver12.a /krbm.o --> ORACLE_HOME/lib/libserver12.a /krbn.o --> ORACLE_HOME/lib/libserver12.a /krbo.o --> ORACLE_HOME/lib/libserver12.a /krbp.o --> ORACLE_HOME/lib/libserver12.a /krbr.o --> ORACLE_HOME/lib/libserver12.a /krbtdb.o --> ORACLE_HOME/lib/libserver12.a /krbx.o --> ORACLE_HOME/lib/libserver12.a /krby.o --> ORACLE_HOME/lib/libserver12.a /krbz.o --> ORACLE_HOME/lib/libserver12.a /krcc.o --> ORACLE_HOME/lib/libserver12.a /krcp.o --> ORACLE_HOME/lib/libserver12.a /krd.o --> ORACLE_HOME/lib/libserver12.a /krdm.o --> ORACLE_HOME/lib/libserver12.a /krf.o --> ORACLE_HOME/lib/libserver12.a /krfg.o --> ORACLE_HOME/lib/libserver12.a /krp.o --> ORACLE_HOME/lib/libserver12.a /krpm.o --> ORACLE_HOME/lib/libserver12.a /krr.o --> ORACLE_HOME/lib/libserver12.a /krsd.o --> ORACLE_HOME/lib/libserver12.a /krse.o --> ORACLE_HOME/lib/libserver12.a /krsf.o --> ORACLE_HOME/lib/libserver12.a /krsg.o --> ORACLE_HOME/lib/libserver12.a /krsm.o --> ORACLE_HOME/lib/libserver12.a /krsr.o --> ORACLE_HOME/lib/libserver12.a /krst.o --> ORACLE_HOME/lib/libserver12.a /krsw.o --> ORACLE_HOME/lib/libserver12.a /krt.o --> ORACLE_HOME/lib/libserver12.a /kra.o --> ORACLE_HOME/lib/libserver12.a /kras.o --> ORACLE_HOME/lib/libserver12.a /knac.o --> ORACLE_HOME/lib/libserver12.a /knahc.o --> ORACLE_HOME/lib/libserver12.a /knahs.o --> ORACLE_HOME/lib/libserver12.a /knal.o --> ORACLE_HOME/lib/libserver12.a /knaladm.o --> ORACLE_HOME/lib/libserver12.a /knalc.o --> ORACLE_HOME/lib/libserver12.a /knalf.o --> ORACLE_HOME/lib/libserver12.a /knalh.o --> ORACLE_HOME/lib/libserver12.a /knalkc.o --> ORACLE_HOME/lib/libserver12.a /knalkd.o --> ORACLE_HOME/lib/libserver12.a /knals.o --> ORACLE_HOME/lib/libserver12.a /knalse.o --> ORACLE_HOME/lib/libserver12.a /knalsp.o --> ORACLE_HOME/lib/libserver12.a /knanr.o --> ORACLE_HOME/lib/libserver12.a /knas.o --> ORACLE_HOME/lib/libserver12.a /knasc.o --> ORACLE_HOME/lib/libserver12.a /knasda.o --> ORACLE_HOME/lib/libserver12.a /knasn.o --> ORACLE_HOME/lib/libserver12.a /knasp.o --> ORACLE_HOME/lib/libserver12.a /knasx.o --> ORACLE_HOME/lib/libserver12.a /kkxa.o --> ORACLE_HOME/lib/libserver12.a /kndm.o --> ORACLE_HOME/lib/libserver12.a /knip.o --> ORACLE_HOME/lib/libserver12.a /knip2.o --> ORACLE_HOME/lib/libserver12.a /knipc.o --> ORACLE_HOME/lib/libserver12.a /knipg.o --> ORACLE_HOME/lib/libserver12.a /knipl.o --> ORACLE_HOME/lib/libserver12.a /knipr.o --> ORACLE_HOME/lib/libserver12.a /knipu.o --> ORACLE_HOME/lib/libserver12.a /knipu2.o --> ORACLE_HOME/lib/libserver12.a /knipx.o --> ORACLE_HOME/lib/libserver12.a /knla.o --> ORACLE_HOME/lib/libserver12.a /knlacdr.o --> ORACLE_HOME/lib/libserver12.a /knlb.o --> ORACLE_HOME/lib/libserver12.a /knll.o --> ORACLE_HOME/lib/libserver12.a /knlo.o --> ORACLE_HOME/lib/libserver12.a /knlogc.o --> ORACLE_HOME/lib/libserver12.a /knlp.o --> ORACLE_HOME/lib/libserver12.a /knlr.o --> ORACLE_HOME/lib/libserver12.a /knlu.o --> ORACLE_HOME/lib/libserver12.a /knlc.o --> ORACLE_HOME/lib/libserver12.a /knlcf.o --> ORACLE_HOME/lib/libserver12.a /knlcfc.o --> ORACLE_HOME/lib/libserver12.a /knlci.o --> ORACLE_HOME/lib/libserver12.a /knlcma.o --> ORACLE_HOME/lib/libserver12.a /knlcmsg.o --> ORACLE_HOME/lib/libserver12.a /knlcn.o --> ORACLE_HOME/lib/libserver12.a /knlco.o --> ORACLE_HOME/lib/libserver12.a /knlcr.o --> ORACLE_HOME/lib/libserver12.a /knlcs.o --> ORACLE_HOME/lib/libserver12.a /knlcsv.o --> ORACLE_HOME/lib/libserver12.a /knlcsx.o --> ORACLE_HOME/lib/libserver12.a /knlcu.o --> ORACLE_HOME/lib/libserver12.a /knld.o --> ORACLE_HOME/lib/libserver12.a /knlq.o --> ORACLE_HOME/lib/libserver12.a /knlsi.o --> ORACLE_HOME/lib/libserver12.a /kkz.o --> ORACLE_HOME/lib/libserver12.a /kkzd.o --> ORACLE_HOME/lib/libserver12.a /kkzf.o --> ORACLE_HOME/lib/libserver12.a /kkzg.o --> ORACLE_HOME/lib/libserver12.a /kkzi.o --> ORACLE_HOME/lib/libserver12.a /kkzj.o --> ORACLE_HOME/lib/libserver12.a /kkzl.o --> ORACLE_HOME/lib/libserver12.a /kkzoqc.o --> ORACLE_HOME/lib/libserver12.a /kkzp.o --> ORACLE_HOME/lib/libserver12.a /kkzr.o --> ORACLE_HOME/lib/libserver12.a /kkzt.o --> ORACLE_HOME/lib/libserver12.a /kkzu.o --> ORACLE_HOME/lib/libserver12.a /kni.o --> ORACLE_HOME/lib/libserver12.a /kntg.o --> ORACLE_HOME/lib/libserver12.a /knts.o --> ORACLE_HOME/lib/libserver12.a /kntx.o --> ORACLE_HOME/lib/libserver12.a /kbc0.o --> ORACLE_HOME/lib/libserver12.a /kbrs.o --> ORACLE_HOME/lib/libserver12.a /kxdam.o --> ORACLE_HOME/lib/libserver12.a /kxdbio.o --> ORACLE_HOME/lib/libserver12.a /kxdff0.o --> ORACLE_HOME/lib/libserver12.a /kxdffr.o --> ORACLE_HOME/lib/libserver12.a /kxdrs.o --> ORACLE_HOME/lib/libserver12.a /aud.o --> ORACLE_HOME/lib/libserver12.a /kza.o --> ORACLE_HOME/lib/libserver12.a /kzaac.o --> ORACLE_HOME/lib/libserver12.a /kzaf.o --> ORACLE_HOME/lib/libserver12.a /kzam.o --> ORACLE_HOME/lib/libserver12.a /kzan.o --> ORACLE_HOME/lib/libserver12.a /kzap.o --> ORACLE_HOME/lib/libserver12.a /kzax.o --> ORACLE_HOME/lib/libserver12.a /kzft.o --> ORACLE_HOME/lib/libserver12.a /kzctx.o --> ORACLE_HOME/lib/libserver12.a /kzd.o --> ORACLE_HOME/lib/libserver12.a /kzp.o --> ORACLE_HOME/lib/libserver12.a /kzra.o --> ORACLE_HOME/lib/libserver12.a /kzrt.o --> ORACLE_HOME/lib/libserver12.a /kzs.o --> ORACLE_HOME/lib/libserver12.a /kzsr.o --> ORACLE_HOME/lib/libserver12.a /kzu.o --> ORACLE_HOME/lib/libserver12.a /kzckm.o --> ORACLE_HOME/lib/libserver12.a /kzckmr.o --> ORACLE_HOME/lib/libserver12.a /kzekm.o --> ORACLE_HOME/lib/libserver12.a /kzeztsm.o --> ORACLE_HOME/lib/libserver12.a /kzvdve.o --> ORACLE_HOME/lib/libserver12.a /kzvradm.o --> ORACLE_HOME/lib/libserver12.a /kzld.o --> ORACLE_HOME/lib/libserver12.a /kzia.o --> ORACLE_HOME/lib/libserver12.a /zlllab.o --> ORACLE_HOME/lib/libserver12.a /kzst.o --> ORACLE_HOME/lib/libserver12.a /kta.o --> ORACLE_HOME/lib/libserver12.a /ktec.o --> ORACLE_HOME/lib/libserver12.a /ktfbn.o --> ORACLE_HOME/lib/libserver12.a /ktfs.o --> ORACLE_HOME/lib/libserver12.a /ktimcem.o --> ORACLE_HOME/lib/libserver12.a /ktm.o --> ORACLE_HOME/lib/libserver12.a /ktsap.o --> ORACLE_HOME/lib/libserver12.a /ktsima.o --> ORACLE_HOME/lib/libserver12.a /ktsimcs.o --> ORACLE_HOME/lib/libserver12.a /ktsj.o --> ORACLE_HOME/lib/libserver12.a /ktsk.o --> ORACLE_HOME/lib/libserver12.a /ktsla.o --> ORACLE_HOME/lib/libserver12.a /ktsld.o --> ORACLE_HOME/lib/libserver12.a /ktslj.o --> ORACLE_HOME/lib/libserver12.a /ktslm.o --> ORACLE_HOME/lib/libserver12.a /ktslr.o --> ORACLE_HOME/lib/libserver12.a /ktsp.o --> ORACLE_HOME/lib/libserver12.a /ktss.o --> ORACLE_HOME/lib/libserver12.a /ktsx.o --> ORACLE_HOME/lib/libserver12.a /ktt.o --> ORACLE_HOME/lib/libserver12.a /tbsdrv.o --> ORACLE_HOME/lib/libserver12.a /qeae.o --> ORACLE_HOME/lib/libserver12.a /qesa.o --> ORACLE_HOME/lib/libserver12.a /qesmm.o --> ORACLE_HOME/lib/libserver12.a /qespc.o --> ORACLE_HOME/lib/libserver12.a /qesrc.o --> ORACLE_HOME/lib/libserver12.a /qeszm.o --> ORACLE_HOME/lib/libserver12.a /kqfp.o --> ORACLE_HOME/lib/libserver12.a /kqf.o --> ORACLE_HOME/lib/libserver12.a /kqfd.o --> ORACLE_HOME/lib/libserver12.a /kkfd.o --> ORACLE_HOME/lib/libserver12.a /kxfp.o --> ORACLE_HOME/lib/libserver12.a /kxfx.o --> ORACLE_HOME/lib/libserver12.a /qerfx.o --> ORACLE_HOME/lib/libserver12.a /qergi.o --> ORACLE_HOME/lib/libserver12.a /qernc.o --> ORACLE_HOME/lib/libserver12.a /qerpx.o --> ORACLE_HOME/lib/libserver12.a /qersc.o --> ORACLE_HOME/lib/libserver12.a /qersq.o --> ORACLE_HOME/lib/libserver12.a /qertb.o --> ORACLE_HOME/lib/libserver12.a /qertqi.o --> ORACLE_HOME/lib/libserver12.a /qerwn.o --> ORACLE_HOME/lib/libserver12.a /qerxa.o --> ORACLE_HOME/lib/libserver12.a /qerxj.o --> ORACLE_HOME/lib/libserver12.a /qerxp.o --> ORACLE_HOME/lib/libserver12.a /qkabl.o --> ORACLE_HOME/lib/libserver12.a /qkadrv.o --> ORACLE_HOME/lib/libserver12.a /qkagby.o --> ORACLE_HOME/lib/libserver12.a /qkajoi.o --> ORACLE_HOME/lib/libserver12.a /qkaoby.o --> ORACLE_HOME/lib/libserver12.a /qkatab.o --> ORACLE_HOME/lib/libserver12.a /qkexr.o --> ORACLE_HOME/lib/libserver12.a /qkn.o --> ORACLE_HOME/lib/libserver12.a /qkne.o --> ORACLE_HOME/lib/libserver12.a /qkxr.o --> ORACLE_HOME/lib/libserver12.a /msqimb.o --> ORACLE_HOME/lib/libserver12.a /qkes.o --> ORACLE_HOME/lib/libserver12.a /qks3t.o --> ORACLE_HOME/lib/libserver12.a /qksbg.o --> ORACLE_HOME/lib/libserver12.a /qkscr.o --> ORACLE_HOME/lib/libserver12.a /qksctx.o --> ORACLE_HOME/lib/libserver12.a /qksfm.o --> ORACLE_HOME/lib/libserver12.a /qksfro.o --> ORACLE_HOME/lib/libserver12.a /qksmm.o --> ORACLE_HOME/lib/libserver12.a /qksop.o --> ORACLE_HOME/lib/libserver12.a /qksptf.o --> ORACLE_HOME/lib/libserver12.a /qksqb.o --> ORACLE_HOME/lib/libserver12.a /qkssage.o --> ORACLE_HOME/lib/libserver12.a /qksvc.o --> ORACLE_HOME/lib/libserver12.a /qkswc.o --> ORACLE_HOME/lib/libserver12.a /dbsdrv.o --> ORACLE_HOME/lib/libserver12.a /kpp3.o --> ORACLE_HOME/lib/libserver12.a /prsa.o --> ORACLE_HOME/lib/libserver12.a /prsc.o --> ORACLE_HOME/lib/libserver12.a /prssz.o --> ORACLE_HOME/lib/libserver12.a /qsmb.o --> ORACLE_HOME/lib/libserver12.a /qsmmix.o --> ORACLE_HOME/lib/libserver12.a /qsmp.o --> ORACLE_HOME/lib/libserver12.a /qsmqcsm.o --> ORACLE_HOME/lib/libserver12.a /qsmqutl.o --> ORACLE_HOME/lib/libserver12.a /kebm.o --> ORACLE_HOME/lib/libserver12.a /kelr.o --> ORACLE_HOME/lib/libserver12.a /kelt.o --> ORACLE_HOME/lib/libserver12.a /keltt.o --> ORACLE_HOME/lib/libserver12.a /ket.o --> ORACLE_HOME/lib/libserver12.a /keomn.o --> ORACLE_HOME/lib/libserver12.a /kehe.o --> ORACLE_HOME/lib/libserver12.a /kesai.o --> ORACLE_HOME/lib/libserver12.a /kesatm.o --> ORACLE_HOME/lib/libserver12.a /kesmc.o --> ORACLE_HOME/lib/libserver12.a /kesmdm.o --> ORACLE_HOME/lib/libserver12.a /kesmi.o --> ORACLE_HOME/lib/libserver12.a /kesms.o --> ORACLE_HOME/lib/libserver12.a /kespi.o --> ORACLE_HOME/lib/libserver12.a /kesqs.o --> ORACLE_HOME/lib/libserver12.a /kessi.o --> ORACLE_HOME/lib/libserver12.a /keswat.o --> ORACLE_HOME/lib/libserver12.a /keswpi.o --> ORACLE_HOME/lib/libserver12.a /keswspm.o --> ORACLE_HOME/lib/libserver12.a /keswsq.o --> ORACLE_HOME/lib/libserver12.a /keswts.o --> ORACLE_HOME/lib/libserver12.a /keswx.o --> ORACLE_HOME/lib/libserver12.a /kew.o --> ORACLE_HOME/lib/libserver12.a /kewa.o --> ORACLE_HOME/lib/libserver12.a /kewm.o --> ORACLE_HOME/lib/libserver12.a /kewmt.o --> ORACLE_HOME/lib/libserver12.a /kewr.o --> ORACLE_HOME/lib/libserver12.a /kews.o --> ORACLE_HOME/lib/libserver12.a /kewt.o --> ORACLE_HOME/lib/libserver12.a /kecp.o --> ORACLE_HOME/lib/libserver12.a /kecprm.o --> ORACLE_HOME/lib/libserver12.a /kla.o --> ORACLE_HOME/lib/libserver12.a /klm.o --> ORACLE_HOME/lib/libserver12.a /klu.o --> ORACLE_HOME/lib/libserver12.a /k2g.o --> ORACLE_HOME/lib/libserver12.a /k2q.o --> ORACLE_HOME/lib/libserver12.a /fbadrv.o --> ORACLE_HOME/lib/libserver12.a /ktc.o --> ORACLE_HOME/lib/libserver12.a /ktcn.o --> ORACLE_HOME/lib/libserver12.a /ktfa.o --> ORACLE_HOME/lib/libserver12.a /ktilm.o --> ORACLE_HOME/lib/libserver12.a /ktli.o --> ORACLE_HOME/lib/libserver12.a /ktma.o --> ORACLE_HOME/lib/libserver12.a /ktmb.o --> ORACLE_HOME/lib/libserver12.a /ktr.o --> ORACLE_HOME/lib/libserver12.a /ktu.o --> ORACLE_HOME/lib/libserver12.a /ktuq.o --> ORACLE_HOME/lib/libserver12.a /ktur.o --> ORACLE_HOME/lib/libserver12.a /ksb.o --> ORACLE_HOME/lib/libserver12.a /ksbt.o --> ORACLE_HOME/lib/libserver12.a /kscs.o --> ORACLE_HOME/lib/libserver12.a /ksct.o --> ORACLE_HOME/lib/libserver12.a /ksdxt.o --> ORACLE_HOME/lib/libserver12.a /ksfcd.o --> ORACLE_HOME/lib/libserver12.a /ksfd.o --> ORACLE_HOME/lib/libserver12.a /ksfdaf.o --> ORACLE_HOME/lib/libserver12.a /ksfdc.o --> ORACLE_HOME/lib/libserver12.a /ksfdss.o --> ORACLE_HOME/lib/libserver12.a /ksfq.o --> ORACLE_HOME/lib/libserver12.a /ksfsm.o --> ORACLE_HOME/lib/libserver12.a /ksipc.o --> ORACLE_HOME/lib/libserver12.a /ksir.o --> ORACLE_HOME/lib/libserver12.a /ksk.o --> ORACLE_HOME/lib/libserver12.a /kskq.o --> ORACLE_HOME/lib/libserver12.a /kskslv.o --> ORACLE_HOME/lib/libserver12.a /ksl.o --> ORACLE_HOME/lib/libserver12.a /ksl2.o --> ORACLE_HOME/lib/libserver12.a /kslt.o --> ORACLE_HOME/lib/libserver12.a /ksm.o --> ORACLE_HOME/lib/libserver12.a /ksmct.o --> ORACLE_HOME/lib/libserver12.a /ksmo.o --> ORACLE_HOME/lib/libserver12.a /ksmpdbs.o --> ORACLE_HOME/lib/libserver12.a /ksmst.o --> ORACLE_HOME/lib/libserver12.a /ksp.o --> ORACLE_HOME/lib/libserver12.a /kspt.o --> ORACLE_HOME/lib/libserver12.a /ksqtab.o --> ORACLE_HOME/lib/libserver12.a /ksrpc.o --> ORACLE_HOME/lib/libserver12.a /kss.o --> ORACLE_HOME/lib/libserver12.a /ksu.o --> ORACLE_HOME/lib/libserver12.a /ksucln.o --> ORACLE_HOME/lib/libserver12.a /ksut.o --> ORACLE_HOME/lib/libserver12.a /ksvt.o --> ORACLE_HOME/lib/libserver12.a /ksws.o --> ORACLE_HOME/lib/libserver12.a /ksxa.o --> ORACLE_HOME/lib/libserver12.a /ksxp.o --> ORACLE_HOME/lib/libserver12.a /ksz.o --> ORACLE_HOME/lib/libserver12.a /qmts.o --> ORACLE_HOME/lib/libserver12.a /qme.o --> ORACLE_HOME/lib/libserver12.a /qmtix.o --> ORACLE_HOME/lib/libserver12.a /qmtixc.o --> ORACLE_HOME/lib/libserver12.a /qmtixd.o --> ORACLE_HOME/lib/libserver12.a /qmtixm.o --> ORACLE_HOME/lib/libserver12.a /qmtixp.o --> ORACLE_HOME/lib/libserver12.a /qjsn.o --> ORACLE_HOME/lib/libserver12.a /qjsnrw.o --> ORACLE_HOME/lib/libserver12.a /qjsnt.o --> ORACLE_HOME/lib/libserver12.a /qmsqx.o --> ORACLE_HOME/lib/libserver12.a /qmxpt.o --> ORACLE_HOME/lib/libserver12.a /qmxqalg.o --> ORACLE_HOME/lib/libserver12.a /qmxqre.o --> ORACLE_HOME/lib/libserver12.a /qmxqrs.o --> ORACLE_HOME/lib/libserver12.a /qmxqrw.o --> ORACLE_HOME/lib/libserver12.a /qmxqrwft.o --> ORACLE_HOME/lib/libserver12.a /qmxstr.o --> ORACLE_HOME/lib/libserver12.a /qmxtab.o --> ORACLE_HOME/lib/libserver12.a /qmxtc.o --> ORACLE_HOME/lib/libserver12.a /qmxtcx.o --> ORACLE_HOME/lib/libserver12.a /qmxtr.o --> ORACLE_HOME/lib/libserver12.a /qmxtr2.o --> ORACLE_HOME/lib/libserver12.a /qmxtra.o --> ORACLE_HOME/lib/libserver12.a /qmxtrcb.o --> ORACLE_HOME/lib/libserver12.a /qmxtrt.o --> ORACLE_HOME/lib/libserver12.a /qm.o --> ORACLE_HOME/lib/libserver12.a /qmps.o --> ORACLE_HOME/lib/libserver12.a /kunid.o --> ORACLE_HOME/lib/libnid.a /ossdisk.o --> ORACLE_HOME/lib/libcell12.a /ossdscv.o --> ORACLE_HOME/lib/libcell12.a /osserr.o --> ORACLE_HOME/lib/libcell12.a /osslib.o --> ORACLE_HOME/lib/libcell12.a /ossnet.o --> ORACLE_HOME/lib/libcell12.a /ossnet_dfc.o --> ORACLE_HOME/lib/libcell12.a /osssec_lib.o --> ORACLE_HOME/lib/libcell12.a /ossmetrics.o --> ORACLE_HOME/lib/libcell12.a /c_apai.o --> ORACLE_HOME/rdbms/lib/libeva12.a /c_qeae.o --> ORACLE_HOME/rdbms/lib/libeva12.a /c_qesa.o --> ORACLE_HOME/rdbms/lib/libeva12.a /c_qmxtc.o --> ORACLE_HOME/rdbms/lib/libeva12.a /c_sagesql.o --> ORACLE_HOME/rdbms/lib/libeva12.a /kudmi_PERF.o --> ORACLE_HOME/rdbms/lib/libperfsrv12.a /kudmpp_PERF.o --> ORACLE_HOME/rdbms/lib/libperfsrv12.a /qcodfdef_PERF.o --> ORACLE_HOME/rdbms/lib/libperfsrv12.a /qmt_PERF.o --> ORACLE_HOME/rdbms/lib/libperfsrv12.a /qmxqc_PERF.o --> ORACLE_HOME/rdbms/lib/libperfsrv12.a /kqf4.o --> ORACLE_HOME/rdbms/lib/libdsga12.a /kgfasmcmd.o --> ORACLE_HOME/lib/libasmperl12.a bundlefcp_DBBP.xml --> ORACLE_HOME/rdbms/admin/bundlefcp_DBBP.xml catcdbviews.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/catcdbviews.sql dbmsjdev.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsjdev.sql bug28040776.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug28040776.sql dbmspool.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmspool.sql prvsash.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvsash.plb dbmsgwm.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwm.sql dbmsrman.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsrman.sql catlsby.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/catlsby.sql catstrc.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/catstrc.sql prvtsnap.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtsnap.plb prvtbapp.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtbapp.plb prvtblin.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtblin.plb prvtbxstr.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtbxstr.plb prvtlock.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtlock.plb prvtemx_cell.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtemx_cell.plb prvtpool.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtpool.plb prvtaqim.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtaqim.plb prvtaqds.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtaqds.plb prvtlmcb.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtlmcb.plb prvtwrr.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtwrr.plb prvtatsk.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtatsk.plb prvtadv.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtadv.plb prvtawrs.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtawrs.plb prvtash.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtash.plb prvtsqli.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtsqli.plb prvtstat.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtstat.plb prvtcmpr.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtcmpr.plb prvtilm.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtilm.plb catfusrg.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/catfusrg.sql prvtrmns.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtrmns.plb bug25410017.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug25410017.sql bug26088836.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug26088836.sql execemx.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/execemx.sql execsvrm.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/execsvrm.sql prvtolsdd.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtolsdd.plb catmacc.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/catmacc.sql driparse.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/ctx/admin/driparse.plb drixmd.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/ctx/admin/drixmd.plb drvddl.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/ctx/admin/drvddl.plb drvxtab.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/ctx/admin/drvxtab.plb bug24899895_rollback.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/ctx/admin/bug24899895_rollback.sql tsdpend.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/tsdpend.sql prvtgwm.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwm.sql bug26442308_rollback.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug26442308_rollback.sql bug25792605_rollback.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug25792605_rollback.sql dbmsoptim.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsoptim.sql bug25872389_rollback.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug25872389_rollback.sql bug25856821_rollback.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug25856821_rollback.sql dropjdev.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dropjdev.sql prvtoptim.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtoptim.plb rollback_25832935.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/rollback_25832935.sql bug25098160.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/bug25098160.sql dbmsgwmfix.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmfix.sql prvtgwmco.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmco.plb prvtgwmfx.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmfx.plb prvtgwmalt.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmalt.plb dbmsgwmut.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmut.sql dbmsgwmpl.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmpl.sql dbmsgwmdb.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmdb.sql prvtgwmcl.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmcl.plb prvtgwmpl.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/src/server/gwm/prvtgwmpl.sql dbmsgwmcl.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmcl.sql prvtgwmut.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmut.plb dbmsgwmfx.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmfx.sql prvtgwmdb.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmdb.plb dbmsrmansys.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsrmansys.sql dbmsgwmco.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmco.sql dbmsgwmalt.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/dbmsgwmalt.sql execsqlt.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/execsqlt.sql prvtgwmfix.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmfix.plb prvtgwmpl.plb --> ORACLE_HOME/sqlpatch/28822515/22636216/rollback_files/rdbms/admin/prvtgwmpl.plb 28822515_apply.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/28822515_apply.sql 28822515_rollback.sql --> ORACLE_HOME/sqlpatch/28822515/22636216/28822515_rollback.sql 28822515.xml --> ORACLE_HOME/sqlpatch/28822515/22636216/28822515.xml sqlpatch.pm --> ORACLE_HOME/sqlpatch/sqlpatch.pm dbmsjdev.sql --> ORACLE_HOME/rdbms/admin/dbmsjdev.sql dbmsslrt.sql --> ORACLE_HOME/rdbms/admin/dbmsslrt.sql asmcmdus.msb --> ORACLE_HOME/rdbms/mesg/asmcmdus.msb asmcmdus.msg --> ORACLE_HOME/rdbms/mesg/asmcmdus.msg kupus.msb --> ORACLE_HOME/rdbms/mesg/kupus.msb kupus.msg --> ORACLE_HOME/rdbms/mesg/kupus.msg asmcmdshare.pm --> ORACLE_HOME/lib/asmcmdshare.pm asmcommand.xml --> ORACLE_HOME/lib/asmcommand.xml asmcmdafd.pm --> ORACLE_HOME/lib/asmcmdafd.pm asmcmdambr.pm --> ORACLE_HOME/lib/asmcmdambr.pm asmcmdamdu.pm --> ORACLE_HOME/lib/asmcmdamdu.pm asmcmdbase.pm --> ORACLE_HOME/lib/asmcmdbase.pm asmcmdpasswd.pm --> ORACLE_HOME/lib/asmcmdpasswd.pm asmcmdsys.pm --> ORACLE_HOME/lib/asmcmdsys.pm kfod.o --> ORACLE_HOME/rdbms/lib/kfod.o asmcmdqg.pm --> ORACLE_HOME/lib/asmcmdqg.pm asmcmdcore --> ORACLE_HOME/bin/asmcmdcore sjssutrace.o --> ORACLE_HOME/rdbms/lib/sjssutrace.o prvtolsdd.plb --> ORACLE_HOME/rdbms/admin/prvtolsdd.plb prvtatsk.plb --> ORACLE_HOME/rdbms/admin/prvtatsk.plb prvtash.plb --> ORACLE_HOME/rdbms/admin/prvtash.plb diskmon.o --> ORACLE_HOME/rdbms/lib/diskmon.o emExadata.swf --> ORACLE_HOME/rdbms/xml/em/orarep/exadata/emExadata.swf exadataDiffHtml.xsl --> ORACLE_HOME/rdbms/xml/em/orarep/exadata/exadataDiffHtml.xsl exadataHtml.xsl --> ORACLE_HOME/rdbms/xml/em/orarep/exadata/exadataHtml.xsl kspapv33.var --> ORACLE_HOME/rdbms/admin/kspapv33.var kspasv62.var --> ORACLE_HOME/rdbms/admin/kspasv62.var kspasv2.var --> ORACLE_HOME/rdbms/admin/kspasv2.var kspapv32.var --> ORACLE_HOME/rdbms/admin/kspapv32.var kspapv1.var --> ORACLE_HOME/rdbms/admin/kspapv1.var kspapv35.var --> ORACLE_HOME/rdbms/admin/kspapv35.var kspasv1.var --> ORACLE_HOME/rdbms/admin/kspasv1.var kspapv3.var --> ORACLE_HOME/rdbms/admin/kspapv3.var kspapv30.var --> ORACLE_HOME/rdbms/admin/kspapv30.var kspasv61.var --> ORACLE_HOME/rdbms/admin/kspasv61.var kspasv64.var --> ORACLE_HOME/rdbms/admin/kspasv64.var kspasv3.var --> ORACLE_HOME/rdbms/admin/kspasv3.var kspapv2.var --> ORACLE_HOME/rdbms/admin/kspapv2.var bundledata_DBRU.xml --> ORACLE_HOME/rdbms/admin/bundledata_DBRU.xml libkubsagt12.a --> ORACLE_HOME/lib/libkubsagt12.a libkubsagt12.so --> ORACLE_HOME/lib/libkubsagt12.so ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/client_sharedlib ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/libasmperl12.ohso ins_plsql.mk --> ORACLE_HOME/plsql/lib/iwrap ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ioracle ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/idbv ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iorapwd ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/irenamedg ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/inid ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/idgmgrl ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iamdu ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ikfed ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ikfod ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ipc_relink /sqllob0.o --> ORACLE_HOME/lib/libsql12.a libcell12.so --> ORACLE_HOME/lib/libcell12.so ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/libasmclntsh12.ohso /skgxpg.o --> ORACLE_HOME/rdbms/lib/libskgxpg.a /ssskgxpg.o --> ORACLE_HOME/rdbms/lib/libskgxpg.a /rfobsv.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /rfscom.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /rfsd.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /rfsin.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /kudbv.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /kfed.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /kfodnm.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmb.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmc.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmdr.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmfm.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmi.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmk2.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmk3.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmk4.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmk5.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmk6.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmk7.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krml.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmm.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmp.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmpa.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmpp.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmq.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmr.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmr2.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krms.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmx.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmz.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /ulb.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /uldba.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /uldl.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /ulkmks.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /ull.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /uln.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /ulnd.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /ulutl.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmicdb.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /krmicds.o --> ORACLE_HOME/rdbms/lib/libdbtools12.a /p2c.o --> ORACLE_HOME/lib/libpsa12.a /psdhsh.o --> ORACLE_HOME/lib/libpsa12.a /psxgsc.o --> ORACLE_HOME/lib/libpsa12.a /psxini.o --> ORACLE_HOME/lib/libpsa12.a /psxlst.o --> ORACLE_HOME/lib/libpsa12.a /psxopt.o --> ORACLE_HOME/lib/libpsa12.a /psxpfx.o --> ORACLE_HOME/lib/libpsa12.a /psxser.o --> ORACLE_HOME/lib/libpsa12.a /psdhnm_pic.o --> ORACLE_HOME/lib/libplc12_pic.a /psuimc_pic.o --> ORACLE_HOME/lib/libplc12_pic.a /kfgp.o --> ORACLE_HOME/lib/libasmclnt12.a /kgf.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfdp.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfdputl.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfence.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfk.o --> ORACLE_HOME/lib/libasmclnt12.a /kgflk.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfm.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfn.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfo.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfp.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfspb.o --> ORACLE_HOME/lib/libasmclnt12.a /kgfz.o --> ORACLE_HOME/lib/libasmclnt12.a /psdhnm.o --> ORACLE_HOME/lib/libplc12.a /psuimc.o --> ORACLE_HOME/lib/libplc12.a /kpxs.o --> ORACLE_HOME/lib/libclient12.a /kpxt.o --> ORACLE_HOME/lib/libclient12.a /kubsx.o --> ORACLE_HOME/lib/libclient12.a /kudmcx.o --> ORACLE_HOME/lib/libclient12.a /kudmd.o --> ORACLE_HOME/lib/libclient12.a /kudml.o --> ORACLE_HOME/lib/libclient12.a /kudmlg.o --> ORACLE_HOME/lib/libclient12.a /kudmmal.o --> ORACLE_HOME/lib/libclient12.a /kudmp.o --> ORACLE_HOME/lib/libclient12.a /kupdc.o --> ORACLE_HOME/lib/libclient12.a /kupdces.o --> ORACLE_HOME/lib/libclient12.a /skudmcr.o --> ORACLE_HOME/lib/libclient12.a /kudmdot.o --> ORACLE_HOME/lib/libclient12.a /kudmdt.o --> ORACLE_HOME/lib/libclient12.a /kudmevc.o --> ORACLE_HOME/lib/libclient12.a /kudmi.o --> ORACLE_HOME/lib/libclient12.a /kudmpa.o --> ORACLE_HOME/lib/libclient12.a /kudmpp.o --> ORACLE_HOME/lib/libclient12.a /kudmr.o --> ORACLE_HOME/lib/libclient12.a /kudmx.o --> ORACLE_HOME/lib/libclient12.a /skudmi.o --> ORACLE_HOME/lib/libclient12.a /skudmr.o --> ORACLE_HOME/lib/libclient12.a /kupaf.o --> ORACLE_HOME/lib/libclient12.a /kupagm.o --> ORACLE_HOME/lib/libclient12.a /kupai.o --> ORACLE_HOME/lib/libclient12.a /kupax.o --> ORACLE_HOME/lib/libclient12.a /kpu.o --> ORACLE_HOME/lib/libclient12.a /kpuaq.o --> ORACLE_HOME/lib/libclient12.a /kpuatr.o --> ORACLE_HOME/lib/libclient12.a /kpucp.o --> ORACLE_HOME/lib/libclient12.a /kpudpp.o --> ORACLE_HOME/lib/libclient12.a /kpue.o --> ORACLE_HOME/lib/libclient12.a /kpuex.o --> ORACLE_HOME/lib/libclient12.a /kpuini.o --> ORACLE_HOME/lib/libclient12.a /kpunl.o --> ORACLE_HOME/lib/libclient12.a /kpuq.o --> ORACLE_HOME/lib/libclient12.a /kpuzln.o --> ORACLE_HOME/lib/libclient12.a /ocixad.o --> ORACLE_HOME/lib/libclient12.a /ocitrace.o --> ORACLE_HOME/lib/libclient12.a /upicdc.o --> ORACLE_HOME/lib/libclient12.a /knxi.o --> ORACLE_HOME/lib/libclient12.a /knxo.o --> ORACLE_HOME/lib/libclient12.a /kfodnm.o --> ORACLE_HOME/lib/libasmclntsh12.a /skfnbeq.o --> ORACLE_HOME/lib/libasmclntsh12.a /kfgp.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgf.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfdp.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfdputl.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfence.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfk.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgflk.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfm.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfn.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfo.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfp.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfspb.o --> ORACLE_HOME/lib/libasmclntsh12.a /kgfz.o --> ORACLE_HOME/lib/libasmclntsh12.a /sdbgrf.o --> ORACLE_HOME/lib/libgeneric12.a /sdbgrfu.o --> ORACLE_HOME/lib/libgeneric12.a /dbgtb.o --> ORACLE_HOME/lib/libgeneric12.a /qcdo.o --> ORACLE_HOME/lib/libgeneric12.a /kgi.o --> ORACLE_HOME/lib/libgeneric12.a /kgl.o --> ORACLE_HOME/lib/libgeneric12.a /kgl2.o --> ORACLE_HOME/lib/libgeneric12.a /kgl4.o --> ORACLE_HOME/lib/libgeneric12.a /kglid.o --> ORACLE_HOME/lib/libgeneric12.a /kgsc.o --> ORACLE_HOME/lib/libgeneric12.a /kgx.o --> ORACLE_HOME/lib/libgeneric12.a /qci.o --> ORACLE_HOME/lib/libgeneric12.a /nbio.o --> ORACLE_HOME/lib/libgeneric12.a /nhpn.o --> ORACLE_HOME/lib/libgeneric12.a /nhpp.o --> ORACLE_HOME/lib/libgeneric12.a /nhpq.o --> ORACLE_HOME/lib/libgeneric12.a /koc.o --> ORACLE_HOME/lib/libgeneric12.a /kola.o --> ORACLE_HOME/lib/libgeneric12.a /kolars.o --> ORACLE_HOME/lib/libgeneric12.a /kolf.o --> ORACLE_HOME/lib/libgeneric12.a /koll.o --> ORACLE_HOME/lib/libgeneric12.a /kolr.o --> ORACLE_HOME/lib/libgeneric12.a /kope2.o --> ORACLE_HOME/lib/libgeneric12.a /skgfr.o --> ORACLE_HOME/lib/libgeneric12.a /skgnfs.o --> ORACLE_HOME/lib/libgeneric12.a /skgnfs_rdma.o --> ORACLE_HOME/lib/libgeneric12.a /skgping.o --> ORACLE_HOME/lib/libgeneric12.a /slcpu.o --> ORACLE_HOME/lib/libgeneric12.a /kngl.o --> ORACLE_HOME/lib/libgeneric12.a /kngo.o --> ORACLE_HOME/lib/libgeneric12.a /kngu.o --> ORACLE_HOME/lib/libgeneric12.a /kxdvers.o --> ORACLE_HOME/lib/libgeneric12.a /qcpi3.o --> ORACLE_HOME/lib/libgeneric12.a /qcpi5.o --> ORACLE_HOME/lib/libgeneric12.a /qcpi7.o --> ORACLE_HOME/lib/libgeneric12.a /qcs.o --> ORACLE_HOME/lib/libgeneric12.a /qcs2.o --> ORACLE_HOME/lib/libgeneric12.a /qcs3.o --> ORACLE_HOME/lib/libgeneric12.a /qcscb.o --> ORACLE_HOME/lib/libgeneric12.a /qcso.o --> ORACLE_HOME/lib/libgeneric12.a /qcsp.o --> ORACLE_HOME/lib/libgeneric12.a /qcodfdef.o --> ORACLE_HOME/lib/libgeneric12.a /qct.o --> ORACLE_HOME/lib/libgeneric12.a /qctoj.o --> ORACLE_HOME/lib/libgeneric12.a /qctol.o --> ORACLE_HOME/lib/libgeneric12.a /kgcs.o --> ORACLE_HOME/lib/libgeneric12.a /kgh.o --> ORACLE_HOME/lib/libgeneric12.a /kgnfs.o --> ORACLE_HOME/lib/libgeneric12.a /kgnfsv3.o --> ORACLE_HOME/lib/libgeneric12.a /kgnfsv4.o --> ORACLE_HOME/lib/libgeneric12.a /kgodm.o --> ORACLE_HOME/lib/libgeneric12.a /kgsk.o --> ORACLE_HOME/lib/libgeneric12.a /qmt.o --> ORACLE_HOME/lib/libgeneric12.a /qmx.o --> ORACLE_HOME/lib/libgeneric12.a /qmxi.o --> ORACLE_HOME/lib/libgeneric12.a /qjsng.o --> ORACLE_HOME/lib/libgeneric12.a /qjsnpls.o --> ORACLE_HOME/lib/libgeneric12.a /qmxtgr2.o --> ORACLE_HOME/lib/libgeneric12.a /qmxtgx.o --> ORACLE_HOME/lib/libgeneric12.a /qmxtxa.o --> ORACLE_HOME/lib/libgeneric12.a /qmudx.o --> ORACLE_HOME/lib/libgeneric12.a /qmxqc.o --> ORACLE_HOME/lib/libgeneric12.a /keclrc.o --> ORACLE_HOME/rdbms/lib/libwrc12.a /keclrk.o --> ORACLE_HOME/rdbms/lib/libwrc12.a /keclrm.o --> ORACLE_HOME/rdbms/lib/libwrc12.a /homt.o --> ORACLE_HOME/lib/libagent12.a /hotk.o --> ORACLE_HOME/lib/libagent12.a /textio.o --> ORACLE_HOME/lib/libplp12.a /pdx8_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pdy7_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pdz2_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pef_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pei_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pen_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /penexc_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /per_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pet_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pevmchr_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pevmexe_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pfrd_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pfrexc_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pfrrun_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pfrsqc_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pfrstk_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pri_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pricbr_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /priknl_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /prilob_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /priold_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /prirep_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /plzsql_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pdttdl_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /ptd_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pbbicd_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /peibp_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pkc_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pkl_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pci_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pcmmsg_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /phd_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /phdd_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /phpkar_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /plscope_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /ppp_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /psf_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /psi_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /psstdo_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /spli_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /splsdbg_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /spssimb_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /ph2csql_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /ph2ima_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /ph2imt_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /ph2oa_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /ph1pp_pic.o --> ORACLE_HOME/lib/libpls12_pic.a /pdx8.o --> ORACLE_HOME/lib/libpls12.a /pdy7.o --> ORACLE_HOME/lib/libpls12.a /pdz2.o --> ORACLE_HOME/lib/libpls12.a /pef.o --> ORACLE_HOME/lib/libpls12.a /pei.o --> ORACLE_HOME/lib/libpls12.a /pen.o --> ORACLE_HOME/lib/libpls12.a /penexc.o --> ORACLE_HOME/lib/libpls12.a /per.o --> ORACLE_HOME/lib/libpls12.a /pet.o --> ORACLE_HOME/lib/libpls12.a /pevmchr.o --> ORACLE_HOME/lib/libpls12.a /pevmexe.o --> ORACLE_HOME/lib/libpls12.a /pevmjson.o --> ORACLE_HOME/lib/libpls12.a /pfrd.o --> ORACLE_HOME/lib/libpls12.a /pfrexc.o --> ORACLE_HOME/lib/libpls12.a /pfrrun.o --> ORACLE_HOME/lib/libpls12.a /pfrsqc.o --> ORACLE_HOME/lib/libpls12.a /pfrstk.o --> ORACLE_HOME/lib/libpls12.a /pri.o --> ORACLE_HOME/lib/libpls12.a /pricbr.o --> ORACLE_HOME/lib/libpls12.a /priknl.o --> ORACLE_HOME/lib/libpls12.a /prilob.o --> ORACLE_HOME/lib/libpls12.a /priold.o --> ORACLE_HOME/lib/libpls12.a /prirep.o --> ORACLE_HOME/lib/libpls12.a /plzsql.o --> ORACLE_HOME/lib/libpls12.a /pdttdl.o --> ORACLE_HOME/lib/libpls12.a /ptd.o --> ORACLE_HOME/lib/libpls12.a /pbbicd.o --> ORACLE_HOME/lib/libpls12.a /peibp.o --> ORACLE_HOME/lib/libpls12.a /pkc.o --> ORACLE_HOME/lib/libpls12.a /pkl.o --> ORACLE_HOME/lib/libpls12.a /pci.o --> ORACLE_HOME/lib/libpls12.a /pcmmsg.o --> ORACLE_HOME/lib/libpls12.a /phd.o --> ORACLE_HOME/lib/libpls12.a /phdd.o --> ORACLE_HOME/lib/libpls12.a /phpkar.o --> ORACLE_HOME/lib/libpls12.a /plscope.o --> ORACLE_HOME/lib/libpls12.a /ppp.o --> ORACLE_HOME/lib/libpls12.a /psf.o --> ORACLE_HOME/lib/libpls12.a /psi.o --> ORACLE_HOME/lib/libpls12.a /psstdo.o --> ORACLE_HOME/lib/libpls12.a /spli.o --> ORACLE_HOME/lib/libpls12.a /splsdbg.o --> ORACLE_HOME/lib/libpls12.a /spssimb.o --> ORACLE_HOME/lib/libpls12.a /ph2csql.o --> ORACLE_HOME/lib/libpls12.a /ph2ima.o --> ORACLE_HOME/lib/libpls12.a /ph2imt.o --> ORACLE_HOME/lib/libpls12.a /ph2oa.o --> ORACLE_HOME/lib/libpls12.a /ph1pp.o --> ORACLE_HOME/lib/libpls12.a /kdzdcol.o --> ORACLE_HOME/lib/libcommon12.a /kdzt.o --> ORACLE_HOME/lib/libcommon12.a /kdzk.o --> ORACLE_HOME/lib/libcommon12.a /kuddpm.o --> ORACLE_HOME/lib/libcommon12.a /si.o --> ORACLE_HOME/lib/libcommon12.a /skfnbeq.o --> ORACLE_HOME/lib/libcommon12.a /sllf.o --> ORACLE_HOME/lib/libcommon12.a /ttcx2y.o --> ORACLE_HOME/lib/libcommon12.a /knclp.o --> ORACLE_HOME/lib/libcommon12.a /sagecache.o --> ORACLE_HOME/lib/libcommon12.a /sagedata.o --> ORACLE_HOME/lib/libcommon12.a /sageet1.o --> ORACLE_HOME/lib/libcommon12.a /sageindex.o --> ORACLE_HOME/lib/libcommon12.a /sagetxn.o --> ORACLE_HOME/lib/libcommon12.a /storidx.o --> ORACLE_HOME/lib/libcommon12.a /kzsrcr.o --> ORACLE_HOME/lib/libcommon12.a /qesxlc.o --> ORACLE_HOME/lib/libcommon12.a /kudbvc.o --> ORACLE_HOME/lib/libcommon12.a /kutyxtt4.o --> ORACLE_HOME/lib/libcommon12.a /textio_pic.o --> ORACLE_HOME/lib/libplp12_pic.a /skgxpg.o --> ORACLE_HOME/rdbms/lib/libskgxpr.a /ssskgxpg.o --> ORACLE_HOME/rdbms/lib/libskgxpr.a env_rdbms.mk --> ORACLE_HOME/rdbms/lib/env_rdbms.mk libasmclntsh12.def --> ORACLE_HOME/rdbms/admin/libasmclntsh12.def oraus.msb --> ORACLE_HOME/rdbms/mesg/oraus.msb oraus.msg --> ORACLE_HOME/rdbms/mesg/oraus.msg pcmus.msb --> ORACLE_HOME/plsql/mesg/pcmus.msb pcmus.msg --> ORACLE_HOME/plsql/mesg/pcmus.msg smgus.msg --> ORACLE_HOME/rdbms/mesg/smgus.msg smgus.msb --> ORACLE_HOME/rdbms/mesg/smgus.msb qopiprep.bat --> ORACLE_HOME/QOpatch/qopiprep.bat rfsxc.o --> ORACLE_HOME/rdbms/lib/rfsxc.o libipc1.a --> ORACLE_HOME/lib/libipc1.a libmql1.a --> ORACLE_HOME/lib/libmql1.a libipc1.so --> ORACLE_HOME/lib/libipc1.so libskgxpg.so --> ORACLE_HOME/lib/libskgxpg.so libmql1.so --> ORACLE_HOME/lib/libmql1.so libskgxpr.so --> ORACLE_HOME/lib/libskgxpr.so ins_net_client.mk --> ORACLE_HOME/network/lib/client_sharedlib ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/iorion ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/libasmclntsh12.ohso ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ipc_relink catmacc.sql --> ORACLE_HOME/rdbms/admin/catmacc.sql /nzbe.o --> ORACLE_HOME/lib/libnnz12.a /nzos.o --> ORACLE_HOME/lib/libnnz12.a /nzosp.o --> ORACLE_HOME/lib/libnnz12.a /nzds.o --> ORACLE_HOME/lib/libnnz12.a /nztw.o --> ORACLE_HOME/lib/libnnz12.a /ztrsaadapter.o --> ORACLE_HOME/lib/libnnz12.a /gslcss.o --> ORACLE_HOME/lib/libldapclnt12.a /ext_nzbe.o --> ORACLE_HOME/lib/libnnzst12.a /ext_nzos.o --> ORACLE_HOME/lib/libnnzst12.a /ext_nzosp.o --> ORACLE_HOME/lib/libnnzst12.a /ext_nzds.o --> ORACLE_HOME/lib/libnnzst12.a /ext_nztw.o --> ORACLE_HOME/lib/libnnzst12.a /ext_ztrsaadapter.o --> ORACLE_HOME/lib/libnnzst12.a /jzn.o --> ORACLE_HOME/lib/libxml12.a /jzn0dom.o --> ORACLE_HOME/lib/libxml12.a /jznEng.o --> ORACLE_HOME/lib/libxml12.a /jznb.o --> ORACLE_HOME/lib/libxml12.a /jzncsv.o --> ORACLE_HOME/lib/libxml12.a /jzndom.o --> ORACLE_HOME/lib/libxml12.a /jznerr.o --> ORACLE_HOME/lib/libxml12.a /jznevq.o --> ORACLE_HOME/lib/libxml12.a /jznindex.o --> ORACLE_HOME/lib/libxml12.a /jznmerge.o --> ORACLE_HOME/lib/libxml12.a /jznoct.o --> ORACLE_HOME/lib/libxml12.a /jznp.o --> ORACLE_HOME/lib/libxml12.a /jznpatch.o --> ORACLE_HOME/lib/libxml12.a /jznprint.o --> ORACLE_HOME/lib/libxml12.a /jznselect.o --> ORACLE_HOME/lib/libxml12.a /jznt.o --> ORACLE_HOME/lib/libxml12.a /jzntok.o --> ORACLE_HOME/lib/libxml12.a /jznu.o --> ORACLE_HOME/lib/libxml12.a /jznxml.o --> ORACLE_HOME/lib/libxml12.a /lpxbuf.o --> ORACLE_HOME/lib/libxml12.a /lpxfsmbuf.o --> ORACLE_HOME/lib/libxml12.a jznus.msb --> ORACLE_HOME/xdk/mesg/jznus.msb jznus.msg --> ORACLE_HOME/xdk/mesg/jznus.msg libsc_fs.so --> ORACLE_HOME/ctx/lib/libsc_fs.so /drefc.o --> ORACLE_HOME/lib/libctx12.a /dreb.o --> ORACLE_HOME/lib/libctx12.a /dreby.o --> ORACLE_HOME/lib/libctx12.a /dreriy.o --> ORACLE_HOME/lib/libctx12.a /drhy.o --> ORACLE_HOME/lib/libctx12.a /drdm.o --> ORACLE_HOME/lib/libctx12.a /drefcr.o --> ORACLE_HOME/lib/libctx12.a /drek.o --> ORACLE_HOME/lib/libctx12.a /drep.o --> ORACLE_HOME/lib/libctx12.a /dres.o --> ORACLE_HOME/lib/libctx12.a /drebyc.o --> ORACLE_HOME/lib/libctx12.a /drery.o --> ORACLE_HOME/lib/libctx12.a /drhi.o --> ORACLE_HOME/lib/libctx12.a /drnp.o --> ORACLE_HOME/lib/libctx12.a /druo.o --> ORACLE_HOME/lib/libctx12.a /drya.o --> ORACLE_HOME/lib/libctx12.a /dryac.o --> ORACLE_HOME/lib/libctx12.a /dryds.o --> ORACLE_HOME/lib/libctx12.a /dryec.o --> ORACLE_HOME/lib/libctx12.a /dryex.o --> ORACLE_HOME/lib/libctx12.a /dryfc.o --> ORACLE_HOME/lib/libctx12.a /drym.o --> ORACLE_HOME/lib/libctx12.a /dryn.o --> ORACLE_HOME/lib/libctx12.a /dryn2.o --> ORACLE_HOME/lib/libctx12.a /drync.o --> ORACLE_HOME/lib/libctx12.a /drynh.o --> ORACLE_HOME/lib/libctx12.a /drynm.o --> ORACLE_HOME/lib/libctx12.a /dryo.o --> ORACLE_HOME/lib/libctx12.a /dryoc.o --> ORACLE_HOME/lib/libctx12.a /dryph.o --> ORACLE_HOME/lib/libctx12.a /dryphc.o --> ORACLE_HOME/lib/libctx12.a /dryprint.o --> ORACLE_HOME/lib/libctx12.a /drys.o --> ORACLE_HOME/lib/libctx12.a /drysc.o --> ORACLE_HOME/lib/libctx12.a /drysw.o --> ORACLE_HOME/lib/libctx12.a /dryt.o --> ORACLE_HOME/lib/libctx12.a /dryu.o --> ORACLE_HOME/lib/libctx12.a /dryun.o --> ORACLE_HOME/lib/libctx12.a /dryw.o --> ORACLE_HOME/lib/libctx12.a /drywe.o --> ORACLE_HOME/lib/libctx12.a /drywn.o --> ORACLE_HOME/lib/libctx12.a /drywnc.o --> ORACLE_HOME/lib/libctx12.a /dryxp.o --> ORACLE_HOME/lib/libctx12.a /drft.o --> ORACLE_HOME/lib/libctxc12.a /drexd.o --> ORACLE_HOME/lib/libctxc12.a /drexq.o --> ORACLE_HOME/lib/libctxc12.a /drexqc.o --> ORACLE_HOME/lib/libctxc12.a /drexqr.o --> ORACLE_HOME/lib/libctxc12.a /drexxp.o --> ORACLE_HOME/lib/libctxc12.a /drpf.o --> ORACLE_HOME/lib/libctxc12.a orachk.zip --> ORACLE_HOME/suptools/orachk.zip tfa_setup --> ORACLE_HOME/crs/install/tfa_setup ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/ipc_relink ons --> ORACLE_HOME/opmn/bin/ons bug24899895_rollback.sql --> ORACLE_HOME/ctx/admin/bug24899895_rollback.sql bug25098160.sql --> ORACLE_HOME/rdbms/admin/bug25098160.sql bug25410017.sql --> ORACLE_HOME/rdbms/admin/bug25410017.sql bug25792605_rollback.sql --> ORACLE_HOME/rdbms/admin/bug25792605_rollback.sql bug25856821_rollback.sql --> ORACLE_HOME/rdbms/admin/bug25856821_rollback.sql bug25872389_rollback.sql --> ORACLE_HOME/rdbms/admin/bug25872389_rollback.sql bug26088836.sql --> ORACLE_HOME/rdbms/admin/bug26088836.sql bug26442308_rollback.sql --> ORACLE_HOME/rdbms/admin/bug26442308_rollback.sql bug28040776.sql --> ORACLE_HOME/rdbms/admin/bug28040776.sql dbmsoptim.sql --> ORACLE_HOME/rdbms/admin/dbmsoptim.sql dropjdev.sql --> ORACLE_HOME/rdbms/admin/dropjdev.sql rollback_25832935.sql --> ORACLE_HOME/rdbms/admin/rollback_25832935.sql prvtoptim.plb --> ORACLE_HOME/rdbms/admin/prvtoptim.plb dbmsqopi.sql --> ORACLE_HOME/rdbms/admin/dbmsqopi.sql dbmssqlpatch.sql --> ORACLE_HOME/rdbms/admin/dbmssqlpatch.sql prvtqopi.plb --> ORACLE_HOME/rdbms/admin/prvtqopi.plb prvtsqlpatch.plb --> ORACLE_HOME/rdbms/admin/prvtsqlpatch.plb Zip.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip.pm Archive.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/Archive.pm BufferedFileHandle.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/BufferedFileHandle.pm DirectoryMember.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/DirectoryMember.pm FileMember.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/FileMember.pm Member.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/Member.pm MemberRead.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/MemberRead.pm MockFileHandle.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/MockFileHandle.pm NewFileMember.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/NewFileMember.pm StringMember.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/StringMember.pm Tree.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/Tree.pm ZipFileMember.pm --> ORACLE_HOME/sqlpatch/lib/Archive/Zip/ZipFileMember.pm Slurp.pm --> ORACLE_HOME/sqlpatch/lib/File/Slurp.pm String.pm --> ORACLE_HOME/sqlpatch/lib/IO/String.pm SigAction.pm --> ORACLE_HOME/sqlpatch/lib/Sys/SigAction.pm Simple.pm --> ORACLE_HOME/sqlpatch/lib/XML/Simple.pm sqlpatch --> ORACLE_HOME/sqlpatch/sqlpatch sqlpatch.bat --> ORACLE_HOME/sqlpatch/sqlpatch.bat sqlpatch.pl --> ORACLE_HOME/sqlpatch/sqlpatch.pl sqlpatch_bootstrap.sql --> ORACLE_HOME/sqlpatch/sqlpatch_bootstrap.sql bug24899895_apply.sql --> ORACLE_HOME/ctx/admin/bug24899895_apply.sql a1201000.sql --> ORACLE_HOME/rdbms/admin/a1201000.sql bug24714096_apply.sql --> ORACLE_HOME/rdbms/admin/bug24714096_apply.sql bug25872389_apply.sql --> ORACLE_HOME/rdbms/admin/bug25872389_apply.sql bug26442308_apply.sql --> ORACLE_HOME/rdbms/admin/bug26442308_apply.sql c1201000.sql --> ORACLE_HOME/rdbms/admin/c1201000.sql catcdbviews.sql --> ORACLE_HOME/rdbms/admin/catcdbviews.sql catcon.pm --> ORACLE_HOME/rdbms/admin/catcon.pm catctl.pl --> ORACLE_HOME/rdbms/admin/catctl.pl catctl.pm --> ORACLE_HOME/rdbms/admin/catctl.pm apply_25832935.sql --> ORACLE_HOME/rdbms/admin/apply_25832935.sql bug25792605_apply.sql --> ORACLE_HOME/rdbms/admin/bug25792605_apply.sql catfusrg.sql --> ORACLE_HOME/rdbms/admin/catfusrg.sql catilmtab.sql --> ORACLE_HOME/rdbms/admin/catilmtab.sql catlsby.sql --> ORACLE_HOME/rdbms/admin/catlsby.sql catpstrt.sql --> ORACLE_HOME/rdbms/admin/catpstrt.sql catsqlreg.sql --> ORACLE_HOME/rdbms/admin/catsqlreg.sql catstrc.sql --> ORACLE_HOME/rdbms/admin/catstrc.sql catupend.sql --> ORACLE_HOME/rdbms/admin/catupend.sql catuposb.sql --> ORACLE_HOME/rdbms/admin/catuposb.sql catuppst.sql --> ORACLE_HOME/rdbms/admin/catuppst.sql catupstr.sql --> ORACLE_HOME/rdbms/admin/catupstr.sql cdsec.sql --> ORACLE_HOME/rdbms/admin/cdsec.sql cdstrt.sql --> ORACLE_HOME/rdbms/admin/cdstrt.sql dbmsgwmco.sql --> ORACLE_HOME/rdbms/admin/dbmsgwmco.sql dbmsgwmut.sql --> ORACLE_HOME/rdbms/admin/dbmsgwmut.sql dbmspool.sql --> ORACLE_HOME/rdbms/admin/dbmspool.sql dbmsslrt.sql --> ORACLE_HOME/rdbms/admin/dbmsslrt.sql dplsql.bsq --> ORACLE_HOME/rdbms/admin/dplsql.bsq e1201000.sql --> ORACLE_HOME/rdbms/admin/e1201000.sql execsch.sql --> ORACLE_HOME/rdbms/admin/execsch.sql execsvrm.sql --> ORACLE_HOME/rdbms/admin/execsvrm.sql i1201000.sql --> ORACLE_HOME/rdbms/admin/i1201000.sql initcdc.sql --> ORACLE_HOME/rdbms/admin/initcdc.sql initqsma.sql --> ORACLE_HOME/rdbms/admin/initqsma.sql loc_to_common3.sql --> ORACLE_HOME/rdbms/admin/loc_to_common3.sql preupgrade_package.sql --> ORACLE_HOME/rdbms/admin/preupgrade_package.sql spctab.sql --> ORACLE_HOME/rdbms/admin/spctab.sql tsdpend.sql --> ORACLE_HOME/rdbms/admin/tsdpend.sql utlmmigtbls.sql --> ORACLE_HOME/rdbms/admin/utlmmigtbls.sql utluptabdata.sql --> ORACLE_HOME/rdbms/admin/utluptabdata.sql preupgrade_messages.properties --> ORACLE_HOME/rdbms/admin/preupgrade_messages.properties prvtrmns.plb --> ORACLE_HOME/rdbms/admin/prvtrmns.plb prvtlock.plb --> ORACLE_HOME/rdbms/admin/prvtlock.plb prvtpool.plb --> ORACLE_HOME/rdbms/admin/prvtpool.plb prvtlmcb.plb --> ORACLE_HOME/rdbms/admin/prvtlmcb.plb prvtaqim.plb --> ORACLE_HOME/rdbms/admin/prvtaqim.plb prvtgwmco.plb --> ORACLE_HOME/rdbms/admin/prvtgwmco.plb prvtgwmpl.plb --> ORACLE_HOME/rdbms/admin/prvtgwmpl.plb prvtaqds.plb --> ORACLE_HOME/rdbms/admin/prvtaqds.plb prvtstat.plb --> ORACLE_HOME/rdbms/admin/prvtstat.plb prvtcmpr.plb --> ORACLE_HOME/rdbms/admin/prvtcmpr.plb prvtilm.plb --> ORACLE_HOME/rdbms/admin/prvtilm.plb prvtbapp.plb --> ORACLE_HOME/rdbms/admin/prvtbapp.plb prvtblin.plb --> ORACLE_HOME/rdbms/admin/prvtblin.plb prvtbxstr.plb --> ORACLE_HOME/rdbms/admin/prvtbxstr.plb prvtsnap.plb --> ORACLE_HOME/rdbms/admin/prvtsnap.plb prvtadv.plb --> ORACLE_HOME/rdbms/admin/prvtadv.plb prvtsqli.plb --> ORACLE_HOME/rdbms/admin/prvtsqli.plb prvsash.plb --> ORACLE_HOME/rdbms/admin/prvsash.plb prvtash.plb --> ORACLE_HOME/rdbms/admin/prvtash.plb prvtawrs.plb --> ORACLE_HOME/rdbms/admin/prvtawrs.plb prvtwrr.plb --> ORACLE_HOME/rdbms/admin/prvtwrr.plb prvtemx_cell.plb --> ORACLE_HOME/rdbms/admin/prvtemx_cell.plb dbmsrman.sql --> ORACLE_HOME/rdbms/admin/dbmsrman.sql catconst.pm --> ORACLE_HOME/rdbms/admin/catconst.pm dbms_registry_basic.sql --> ORACLE_HOME/rdbms/admin/dbms_registry_basic.sql preupgrade.jar --> ORACLE_HOME/rdbms/admin/preupgrade.jar ins_precomp.mk --> ORACLE_HOME/precomp/lib/proc rmanus.msb --> ORACLE_HOME/rdbms/mesg/rmanus.msb rmanus.msg --> ORACLE_HOME/rdbms/mesg/rmanus.msg recover.bsq --> ORACLE_HOME/rdbms/admin/recover.bsq ins_rdbms.mk --> ORACLE_HOME/rdbms/lib/irman ins_precomp.mk --> ORACLE_HOME/precomp/lib/procob ldapmoddn --> ORACLE_HOME/bin/ldapmoddn ldapmodifymt --> ORACLE_HOME/bin/ldapmodifymt ldapdelete --> ORACLE_HOME/bin/ldapdelete ldapbind --> ORACLE_HOME/bin/ldapbind ldapaddmt --> ORACLE_HOME/bin/ldapaddmt dsml2ldif --> ORACLE_HOME/bin/dsml2ldif ldapmodify --> ORACLE_HOME/bin/ldapmodify ldapcompare --> ORACLE_HOME/bin/ldapcompare ldapadd --> ORACLE_HOME/bin/ldapadd ins_ldap.mk --> ORACLE_HOME/ldap/lib/ldapsearch /oracle/assistants/dbua/resource/cmdfmt_ext2raw_transformer_v12_2_0.xsl --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /oracle/assistants/dbua/resource/commandline_arguments.xml --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /oracle/assistants/dbua/bean/UpgradeDBSettings.class --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /oracle/assistants/dbua/bean/UpgradeSettings.class --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /oracle/assistants/dbua/util/loaders/UpgradeDBSettingsBeanLoader.class --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /oracle/assistants/dbua/backend/HAServicesStep.class --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /oracle/assistants/dbua/backend/PostMigration.class --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /oracle/assistants/dbua/util/helper/DBUADBServicesHelper.class --> ORACLE_HOME/assistants/dbua/jlib/dbua.jar /lfimkn.o --> ORACLE_HOME/lib/libcore12.a /slputcp.o --> ORACLE_HOME/lib/libcore12.a /lpu.o --> ORACLE_HOME/lib/libcore12.a /oracle/install/ivw/crs/action/MgmtDBAction.class --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/action/MgmtDBDeciderAction.class --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/driver/CRSSetupDriver$1.class --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/driver/CRSSetupDriver$BatchExecConfirmationPromptOption.class --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/driver/CRSSetupDriver.class --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/resource/crsinstall-flow-config.xml --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/resource/CRSDialogLabelResID.class --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/resource/responsefile/crsinstall_rspfmt_response_schema_v11_2_0.xsd --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/resource/responsefile/rspfmt_ext2raw_transformer_v11_2_0.xsl --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/install/ivw/crs/resource/responsefile/rspfmt_raw2ext_transformer_v11_2_0.xsl --> ORACLE_HOME/install/jlib/instcrs.jar /oracle/sysman/assistants/util/CommonUtils$ListenerInfo.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/CommonUtils$RACListener.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/CommonUtils.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/ha/HAService.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar /oracle/sysman/assistants/util/hasi/HADatabaseUtils.class --> ORACLE_HOME/assistants/jlib/assistantsCommon.jar log4j-api-2.9.1.jar --> ORACLE_HOME/md/jlib/log4j-api-2.9.1.jar log4j-core-2.9.1.jar --> ORACLE_HOME/md/jlib/log4j-core-2.9.1.jar log4j-slf4j-impl-2.9.1.jar --> ORACLE_HOME/md/jlib/log4j-slf4j-impl-2.9.1.jar slf4j-api-1.7.25.jar --> ORACLE_HOME/md/jlib/slf4j-api-1.7.25.jar ndmserver.ear.zip --> ORACLE_HOME/md/jlib/ndmserver.ear.zip ndm_tutorial.zip --> ORACLE_HOME/md/jlib/ndm_tutorial.zip groovy-2.4.5.jar --> ORACLE_HOME/md/property_graph/dal/groovy/groovy-2.4.5.jar jackson-annotations-2.9.3.jar --> ORACLE_HOME/md/property_graph/lib/jackson-annotations-2.9.3.jar jackson-core-2.9.3.jar --> ORACLE_HOME/md/property_graph/lib/jackson-core-2.9.3.jar jackson-core-asl-1.9.13.jar --> ORACLE_HOME/md/property_graph/lib/jackson-core-asl-1.9.13.jar jackson-databind-2.9.3.jar --> ORACLE_HOME/md/property_graph/lib/jackson-databind-2.9.3.jar jackson-dataformat-yaml-2.9.3.jar --> ORACLE_HOME/md/property_graph/lib/jackson-dataformat-yaml-2.9.3.jar jackson-mapper-asl-1.9.13.jar --> ORACLE_HOME/md/property_graph/lib/jackson-mapper-asl-1.9.13.jar log4j-1.2.17.jar --> ORACLE_HOME/md/jlib/log4j-1.2.17.jar slf4j-api-1.7.6.jar --> ORACLE_HOME/md/jlib/slf4j-api-1.7.6.jar slf4j-log4j12-1.7.6.jar --> ORACLE_HOME/md/jlib/slf4j-log4j12-1.7.6.jar Patch Location in Inventory: /u01/app/oracle/product/12.2.0.1/dbhome_1/inventory/oneoffs/28822515 Patch Location in Storage area: /u01/app/oracle/product/12.2.0.1/dbhome_1/.patch_storage/28822515_Dec_19_2018_02_45_37 -------------------------------------------------------------------------------- OPatch succeeded. |
Re: RAC instance crash
|
Administrator
|
Okay. I will try to help you..
*load averages seem acceptable. There are some peaks but not so sharp peaks. *You have an instance termination -> Errors in file /u01/app/oracle/diag/rdbms/cdbiamas/cdbiamas1/trace/cdbiamas1_lmhb_368993.trc (incident=6943370) (PDBNAME=CDB$ROOT): ORA-29770: global enqueue process IPC0 (OSID 368909) is hung for more than 70 seconds Incident details in: /u01/app/oracle/diag/rdbms/cdbiamas/cdbiamas1/incident/incdir_6943370/cdbiamas1_lmhb_368993_i6943370.trc 2019-11-15T14:51:57.035439+04:00 ERROR: Some process(s) is not making progress. LMHB (ospid: 368993) is terminating the instance. Please check LMHB trace file for more details. Please also check the CPU load, I/O load and other system properties for anomalous behavior *your syslog is clean. *your db version is 12.2.0.1. OKAY.. We need to see the LHMB trace as well.. There is a known issue about this-> LMHB (ospid: XXXX): terminating the instance due to error 29770 (Doc ID 2438319.1) Did you review the note above? |
|
|
--- LMHB (ospid: XXXX): terminating the instance due to error 29770 (Doc ID 2438319.1)
Yes i checked this note. _lm_enable_aff_benefit_stats parameter is also false our database. Now i load trace and incident files: TRACE_FILE: *** 2019-11-15T14:50:20.420113+04:00 (CDB$ROOT(1)) ============================== IPC0 (ospid: 368909) has not moved for 22 sec (1573815015.1573814993) : heartbeat check status 2 (acceptable) (threshold 70 sec) : heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10 : heartbeat state 0x2.ffff () pso-flag 0x0 : Not in wait; last wait ended 17 secs ago. : last wait_id 3106025 at 'rdbms ipc message'. ============================== Dumping PROCESS IPC0 (ospid: 368909) States ============================== ===[ Callstack ]=== *** 2019-11-15T14:50:20.420233+04:00 (CDB$ROOT(1)) Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909, pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431 ------------------------------------------------------------------------------- os thread scheduling delay history: (sampling every 1.000000 secs) 0.000000 secs at [ 14:50:19 ] NOTE: scheduling delay has not been sampled for 0.488611 secs 0.000000 secs from [ 14:50:15 - 14:50:20 ], 5 sec avg 0.000000 secs from [ 14:49:20 - 14:50:20 ], 1 min avg 0.000000 secs from [ 14:45:20 - 14:50:20 ], 5 min avg *** 2019-11-15T14:50:22.767007+04:00 (CDB$ROOT(1)) loadavg : 8.14 5.28 4.22 System user time: 0.07 sys time: 0.02 context switch: 61402 Memory (Avail / Total) = 2945.37M / 772256.64M Swap (Avail / Total) = 24000.20M / 24576.00M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:07:30 ora_ipc0_cdbiamas1 Name: ora_ipc0_cdbiam State: R (running) Tgid: 368909 Ngid: 0 Pid: 368909 PPid: 1 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 128 Groups: 1001 1002 NStgid: 368909 NSpid: 368909 NSpgid: 368909 NSsid: 368909 VmPeak: 668175752 kB VmSize: 668175748 kB VmLck: 0 kB VmPin: 562562088 kB VmHWM: 7413828 kB VmRSS: 7413828 kB VmData: 8388 kB VmStk: 156 kB VmExe: 343516 kB VmLib: 32424 kB VmPTE: 14924 kB VmPMD: 2172 kB VmSwap: 0 kB Threads: 1 SigQ: 1/3088249 SigPnd: 0000000000000800 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000016400207 SigCgt: 00000003c9887cf8 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff Seccomp: 0 Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff Cpus_allowed_list: 0-111 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 4588970 nonvoluntary_ctxt_switches: 11610 *** 2019-11-15T14:50:25.421552+04:00 (CDB$ROOT(1)) Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2653 ms ------------------------------------------------------------------------------- Process diagnostic dump actual duration=5.002000 sec (max dump time=5.000000 sec) *** 2019-11-15T14:50:25.421655+04:00 (CDB$ROOT(1)) *** 2019-11-15T14:50:30.879349+04:00 (CDB$ROOT(1)) kjfmGCR_HBCheckAll: elapse time between calls is 15 secs ---------------------------------------- SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1 SGA version=(1,0) (session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310 flags: (0x51) USR/- flags2: (0x409) -/-/INC flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/- DID: 0001-001F-000000040000-0000-00000000, short-term DID: txn branch: (nil) con_id/con_uid/con_name: 1/1/CDB$ROOT con_logonuid: 1 con_logonid: 1 con_scuid: 1 con_scid: 1 edition#: 0 user#/name: 0/SYS oct: 0, prv: 0, sql: (nil), psql: (nil) stats: 0x55bffcb060, PX stats: 0x110cb824 service name: SYS$BACKGROUND Current Wait Stack: Not in wait; last wait ended 0.000211 sec ago Wait State: fixed_waits=0 flags=0x21 boundary=(nil)/-1 Session Wait History: elapsed time of 0.000231 sec since last wait 0: waited for 'GCR CSS group update' =0x0, =0x0, =0x0 wait_id=14436246 seq_num=18705 snap_id=2 wait times: snap=0.000001 sec, exc=0.000002 sec, total=0.000333 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 1: waited for 'CSS operation: data update' function_id=0xd, =0x0, =0x0 wait_id=14436247 seq_num=18704 snap_id=1 wait times: snap=0.000331 sec, exc=0.000331 sec, total=0.000331 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 2: waited for 'GCR CSS group update' =0x0, =0x0, =0x0 wait_id=14436246 seq_num=18703 snap_id=1 wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000073 sec of elapsed time 3: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436245 seq_num=18702 snap_id=1 wait times: snap=0.150164 sec, exc=0.150164 sec, total=0.150164 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000077 sec of elapsed time 4: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436244 seq_num=18701 snap_id=1 wait times: snap=0.149855 sec, exc=0.149855 sec, total=0.149855 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000125 sec of elapsed time 5: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436243 seq_num=18700 snap_id=1 wait times: snap=0.149900 sec, exc=0.149900 sec, total=0.149900 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.005222 sec of elapsed time 6: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436242 seq_num=18699 snap_id=1 wait times: snap=5.001861 sec, exc=5.001861 sec, total=5.001861 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000097 sec of elapsed time 7: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436240 seq_num=18698 snap_id=2 wait times: snap=0.000139 sec, exc=2.347218 sec, total=5.001355 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 8: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436241 seq_num=18697 snap_id=1 wait times: snap=2.654137 sec, exc=2.654137 sec, total=2.654137 sec wait times: max=2.650000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 9: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436240 seq_num=18696 snap_id=1 wait times: snap=2.347079 sec, exc=2.347079 sec, total=2.347079 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000213 sec of elapsed time Sampled Session History of session 9560 serial 54991 --------------------------------------------------- The sampled session history is constructed by sampling the target session every 1 second. The sampling process captures at each sample if the session is in a non-idle wait, an idle wait, or not in a wait. If the session is in a non-idle wait then one interval is shown for all the samples the session was in the same non-idle wait. If the session is in an idle wait or not in a wait for consecutive samples then one interval is shown for all the consecutive samples. Though we display these consecutive samples in a single interval the session may NOT be continuously idle or not in a wait (the sampling process does not know). The history is displayed in reverse chronological order. sample interval: 1 sec, max history 120 sec --------------------------------------------------- [5 samples, 14:50:25 - 14:50:29] waited for 'ksdxexeotherwait', seq_num: 18699 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001861 sec (sample interval: 4 sec) [2 samples, 14:50:23 - 14:50:24] waited for 'ksdxexeotherwait', seq_num: 18697 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.654137 sec (sample interval: 1 sec) [3 samples, 14:50:20 - 14:50:22] waited for 'process diagnostic dump', seq_num: 18696 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.347079 sec (sample interval: 2 sec) [5 samples, 14:50:15 - 14:50:19] waited for 'ksdxexeotherwait', seq_num: 18695 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001145 sec (sample interval: 4 sec) [105 samples, 14:48:30 - 14:50:14] idle wait at each sample --------------------------------------------------- Sampled Session History Summary: longest_non_idle_wait: 'ksdxexeotherwait' [5 samples, 14:50:25 - 14:50:29] time_waited: 5.001861 sec (sample interval: 4 sec) --------------------------------------------------- ---------------------------------------- kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0 kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0 temporary object counter: 0 last pdbid to increment: 0, last function to increment: (null) BPS: empty ---------------------------------------- Virtual Thread: kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991 vc: (nil), proc: 0x566000f310, id: 9560 consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig: vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2 inwait: 0, short wait event: 0 posted_run: 0 lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)' lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)' location where insched last set: kgskthrrun location where insched last cleared: kgskthrrun1 location where inwait last set: NULL location where inwait last cleared: NULL is_assigned: 0, in_sched: 0 (0) qcls: (nil), qlink: FALSE vt_active: 0 (pending: 0) vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0) ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0) used quanta (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 exec start consumed time: 0 exec start elapsed time: 0 idle time: 0, active time: 0 (cg: 0) last updnumps: 0, active time (pq: 0 ps: 0) cpu yields: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu waits: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu wait time (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 ASL queued time outs: 0, time: 0 (cur 0, cg 0) PQQ queued time outs: 0, time: 0 (cur 0, cg 0) Queue timeout violation: 0 calls aborted: 0, num est exec limit hit: 0 undo current: 0k max: 0k KTU Session Commit Cache Dump for IDLs: KTU Session Commit Cache Dump for Non-IDLs: ---------------------------------------- KKS-UOL used : 0 locks(used=0, free=0) KGX Atomic Operation Log 0x973f6daae8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab38 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab88 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dabd8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 ---------------------------------------- KGL-UOL SO Cache(total=0, free=0) KGX Atomic Operation Log 0x971f689230 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689298 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689300 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689368 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6893d0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689438 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6894a0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689508 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689570 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 ---------------------------------------- SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1 SGA version=(1,0) (dummy) nxc=0, nlb=0 *** 2019-11-15T14:50:35.882962+04:00 (CDB$ROOT(1)) ============================== IPC0 (ospid: 368909) has not moved for 37 sec (1573815030.1573814993) : heartbeat check status 2 (acceptable) (threshold 70 sec) : heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10 : heartbeat state 0x2.ffff () pso-flag 0x0 : Not in wait; last wait ended 33 secs ago. : last wait_id 3106025 at 'rdbms ipc message'. ============================== Dumping PROCESS IPC0 (ospid: 368909) States ============================== ===[ Callstack ]=== *** 2019-11-15T14:50:35.883082+04:00 (CDB$ROOT(1)) Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909, pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431 ------------------------------------------------------------------------------- os thread scheduling delay history: (sampling every 1.000000 secs) 0.000000 secs at [ 14:50:34 ] NOTE: scheduling delay has not been sampled for 0.890380 secs 0.000000 secs from [ 14:50:30 - 14:50:35 ], 5 sec avg 0.000000 secs from [ 14:49:36 - 14:50:35 ], 1 min avg 0.000000 secs from [ 14:45:36 - 14:50:35 ], 5 min avg *** 2019-11-15T14:50:38.199797+04:00 (CDB$ROOT(1)) loadavg : 9.24 5.66 4.36 System user time: 0.05 sys time: 0.02 context switch: 160189 Memory (Avail / Total) = 6366.25M / 772256.64M Swap (Avail / Total) = 24000.20M / 24576.00M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:07:51 ora_ipc0_cdbiamas1 Name: ora_ipc0_cdbiam State: R (running) Tgid: 368909 Ngid: 0 Pid: 368909 PPid: 1 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 128 Groups: 1001 1002 NStgid: 368909 NSpid: 368909 NSpgid: 368909 NSsid: 368909 VmPeak: 668175752 kB VmSize: 668175748 kB VmLck: 0 kB VmPin: 562562088 kB VmHWM: 7413828 kB VmRSS: 7413828 kB VmData: 8388 kB VmStk: 156 kB VmExe: 343516 kB VmLib: 32424 kB VmPTE: 14924 kB VmPMD: 2172 kB VmSwap: 0 kB Threads: 1 SigQ: 2/3088249 SigPnd: 0000000000000800 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000016400207 SigCgt: 00000003c9887cf8 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff Seccomp: 0 Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff Cpus_allowed_list: 0-111 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 4589413 nonvoluntary_ctxt_switches: 14530 *** 2019-11-15T14:50:40.884556+04:00 (CDB$ROOT(1)) Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2683 ms ------------------------------------------------------------------------------- Process diagnostic dump actual duration=5.002000 sec (max dump time=5.000000 sec) *** 2019-11-15T14:50:40.884646+04:00 (CDB$ROOT(1)) *** 2019-11-15T14:50:45.885958+04:00 (CDB$ROOT(1)) kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat split1 check, id 3 failed *** 2019-11-15T14:50:45.890690+04:00 (CDB$ROOT(1)) kjgcr_DoPreAction: KJGCR_PREACTION? - id 36 *** 2019-11-15T14:50:45.890721+04:00 (CDB$ROOT(1)) kjgcr_DoAction: KJGCR_ACTION? - id 36 *** 2019-11-15T14:50:45.890738+04:00 (CDB$ROOT(1)) kjgcr_DoAction: KJGCR_ACTION - id 36 ran with status:0 clr:1 kjfmGCR_HBCheckAll: elapse time between calls is 15 secs ---------------------------------------- SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1 SGA version=(1,0) (session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310 flags: (0x51) USR/- flags2: (0x409) -/-/INC flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/- DID: 0001-001F-000000040000-0000-00000000, short-term DID: txn branch: (nil) con_id/con_uid/con_name: 1/1/CDB$ROOT con_logonuid: 1 con_logonid: 1 con_scuid: 1 con_scid: 1 edition#: 0 user#/name: 0/SYS oct: 0, prv: 0, sql: (nil), psql: (nil) stats: 0x55bffcb060, PX stats: 0x110cb824 service name: SYS$BACKGROUND Current Wait Stack: Not in wait; last wait ended 0.000190 sec ago Wait State: fixed_waits=0 flags=0x21 boundary=(nil)/-1 Session Wait History: elapsed time of 0.000222 sec since last wait 0: waited for 'GCR CSS group query' =0x0, =0x0, =0x0 wait_id=14436253 seq_num=18714 snap_id=2 wait times: snap=0.000002 sec, exc=0.000004 sec, total=0.000416 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 1: waited for 'CSS operation: data query' function_id=0x3, =0x0, =0x0 wait_id=14436254 seq_num=18713 snap_id=1 wait times: snap=0.000412 sec, exc=0.000412 sec, total=0.000412 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 2: waited for 'GCR CSS group query' =0x0, =0x0, =0x0 wait_id=14436253 seq_num=18712 snap_id=1 wait times: snap=0.000002 sec, exc=0.000002 sec, total=0.000002 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000202 sec of elapsed time 3: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436252 seq_num=18711 snap_id=1 wait times: snap=0.149943 sec, exc=0.149943 sec, total=0.149943 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.004943 sec of elapsed time 4: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436251 seq_num=18710 snap_id=1 wait times: snap=5.001112 sec, exc=5.001112 sec, total=5.001112 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000077 sec of elapsed time 5: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436249 seq_num=18709 snap_id=2 wait times: snap=0.000122 sec, exc=2.317113 sec, total=5.001503 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 6: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436250 seq_num=18708 snap_id=1 wait times: snap=2.684390 sec, exc=2.684390 sec, total=2.684390 sec wait times: max=2.680000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 7: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436249 seq_num=18707 snap_id=1 wait times: snap=2.316991 sec, exc=2.316991 sec, total=2.316991 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000200 sec of elapsed time 8: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436248 seq_num=18706 snap_id=1 wait times: snap=5.000485 sec, exc=5.000485 sec, total=5.000485 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.003099 sec of elapsed time 9: waited for 'GCR CSS group update' =0x0, =0x0, =0x0 wait_id=14436246 seq_num=18705 snap_id=2 wait times: snap=0.000001 sec, exc=0.000002 sec, total=0.000333 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time Sampled Session History of session 9560 serial 54991 --------------------------------------------------- The sampled session history is constructed by sampling the target session every 1 second. The sampling process captures at each sample if the session is in a non-idle wait, an idle wait, or not in a wait. If the session is in a non-idle wait then one interval is shown for all the samples the session was in the same non-idle wait. If the session is in an idle wait or not in a wait for consecutive samples then one interval is shown for all the consecutive samples. Though we display these consecutive samples in a single interval the session may NOT be continuously idle or not in a wait (the sampling process does not know). The history is displayed in reverse chronological order. sample interval: 1 sec, max history 120 sec --------------------------------------------------- [5 samples, 14:50:41 - 14:50:45] waited for 'ksdxexeotherwait', seq_num: 18710 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: >= 4 sec (still in wait) [3 samples, 14:50:38 - 14:50:40] waited for 'ksdxexeotherwait', seq_num: 18708 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.684390 sec (sample interval: 2 sec) [2 samples, 14:50:36 - 14:50:37] waited for 'process diagnostic dump', seq_num: 18707 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.316991 sec (sample interval: 1 sec) [5 samples, 14:50:31 - 14:50:35] waited for 'ksdxexeotherwait', seq_num: 18706 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.000485 sec (sample interval: 4 sec) [1 sample, 14:50:30] idle wait at each sample [5 samples, 14:50:25 - 14:50:29] waited for 'ksdxexeotherwait', seq_num: 18699 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001861 sec (sample interval: 4 sec) [2 samples, 14:50:23 - 14:50:24] waited for 'ksdxexeotherwait', seq_num: 18697 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.654137 sec (sample interval: 1 sec) [3 samples, 14:50:20 - 14:50:22] waited for 'process diagnostic dump', seq_num: 18696 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.347079 sec (sample interval: 2 sec) [5 samples, 14:50:15 - 14:50:19] waited for 'ksdxexeotherwait', seq_num: 18695 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001145 sec (sample interval: 4 sec) [90 samples, 14:48:46 - 14:50:14] idle wait at each sample --------------------------------------------------- Sampled Session History Summary: longest_non_idle_wait: 'ksdxexeotherwait' [5 samples, 14:50:25 - 14:50:29] time_waited: 5.001861 sec (sample interval: 4 sec) --------------------------------------------------- ---------------------------------------- kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0 kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0 temporary object counter: 0 last pdbid to increment: 0, last function to increment: (null) BPS: empty ---------------------------------------- Virtual Thread: kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991 vc: (nil), proc: 0x566000f310, id: 9560 consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig: vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2 inwait: 0, short wait event: 0 posted_run: 0 lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)' lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)' location where insched last set: kgskthrrun location where insched last cleared: kgskthrrun1 location where inwait last set: NULL location where inwait last cleared: NULL is_assigned: 0, in_sched: 0 (0) qcls: (nil), qlink: FALSE vt_active: 0 (pending: 0) vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0) ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0) used quanta (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 exec start consumed time: 0 exec start elapsed time: 0 idle time: 0, active time: 0 (cg: 0) last updnumps: 0, active time (pq: 0 ps: 0) cpu yields: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu waits: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu wait time (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 ASL queued time outs: 0, time: 0 (cur 0, cg 0) PQQ queued time outs: 0, time: 0 (cur 0, cg 0) Queue timeout violation: 0 calls aborted: 0, num est exec limit hit: 0 undo current: 0k max: 0k KTU Session Commit Cache Dump for IDLs: KTU Session Commit Cache Dump for Non-IDLs: ---------------------------------------- KKS-UOL used : 0 locks(used=0, free=0) KGX Atomic Operation Log 0x973f6daae8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab38 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab88 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dabd8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 ---------------------------------------- KGL-UOL SO Cache(total=0, free=0) KGX Atomic Operation Log 0x971f689230 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689298 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689300 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689368 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6893d0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689438 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6894a0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689508 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689570 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 ---------------------------------------- SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1 SGA version=(1,0) (dummy) nxc=0, nlb=0 *** 2019-11-15T14:50:51.045691+04:00 (CDB$ROOT(1)) ============================== IPC0 (ospid: 368909) has not moved for 52 sec (1573815045.1573814993) : heartbeat check status 2 (acceptable) (threshold 70 sec) : heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10 : heartbeat state 0x2.ffff () pso-flag 0x0 : Not in wait; last wait ended 48 secs ago. : last wait_id 3106025 at 'rdbms ipc message'. ============================== Dumping PROCESS IPC0 (ospid: 368909) States ============================== ===[ Callstack ]=== *** 2019-11-15T14:50:51.045811+04:00 (CDB$ROOT(1)) Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909, pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431 ------------------------------------------------------------------------------- os thread scheduling delay history: (sampling every 1.000000 secs) 0.000000 secs at [ 14:50:50 ] NOTE: scheduling delay has not been sampled for 0.993271 secs 0.000000 secs from [ 14:50:47 - 14:50:51 ], 5 sec avg 0.000000 secs from [ 14:49:51 - 14:50:51 ], 1 min avg 0.000000 secs from [ 14:45:51 - 14:50:51 ], 5 min avg *** 2019-11-15T14:50:53.367348+04:00 (CDB$ROOT(1)) loadavg : 195.27 48.34 18.43 System user time: 0.05 sys time: 0.02 context switch: 61840 Memory (Avail / Total) = 6364.98M / 772256.64M Swap (Avail / Total) = 24000.20M / 24576.00M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:04 ora_ipc0_cdbiamas1 Name: ora_ipc0_cdbiam State: R (running) Tgid: 368909 Ngid: 0 Pid: 368909 PPid: 1 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 128 Groups: 1001 1002 NStgid: 368909 NSpid: 368909 NSpgid: 368909 NSsid: 368909 VmPeak: 668175752 kB VmSize: 668175748 kB VmLck: 0 kB VmPin: 562562088 kB VmHWM: 7413828 kB VmRSS: 7413828 kB VmData: 8388 kB VmStk: 156 kB VmExe: 343516 kB VmLib: 32424 kB VmPTE: 14924 kB VmPMD: 2172 kB VmSwap: 0 kB Threads: 1 SigQ: 4/3088249 SigPnd: 0000000000000800 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000016400207 SigCgt: 00000003c9887cf8 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff Seccomp: 0 Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff Cpus_allowed_list: 0-111 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 4589413 nonvoluntary_ctxt_switches: 14559 *** 2019-11-15T14:50:56.047704+04:00 (CDB$ROOT(1)) Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2678 ms ------------------------------------------------------------------------------- Process diagnostic dump actual duration=5.002000 sec (max dump time=5.000000 sec) *** 2019-11-15T14:50:56.047804+04:00 (CDB$ROOT(1)) *** 2019-11-15T14:51:01.049333+04:00 (CDB$ROOT(1)) kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat split2 check, id 2 failed *** 2019-11-15T14:51:01.049469+04:00 (CDB$ROOT(1)) kjgcr_StatCheckCPU: Current load ratio is high (68672/1369) kjfmGCR_HBCheckAll: elapse time between calls is 15 secs ---------------------------------------- SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1 SGA version=(1,0) (session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310 flags: (0x51) USR/- flags2: (0x409) -/-/INC flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/- DID: 0001-001F-000000040000-0000-00000000, short-term DID: txn branch: (nil) con_id/con_uid/con_name: 1/1/CDB$ROOT con_logonuid: 1 con_logonid: 1 con_scuid: 1 con_scid: 1 edition#: 0 user#/name: 0/SYS oct: 0, prv: 0, sql: (nil), psql: (nil) stats: 0x55bffcb060, PX stats: 0x110cb824 service name: SYS$BACKGROUND Current Wait Stack: Not in wait; last wait ended 0.000191 sec ago Wait State: fixed_waits=0 flags=0x21 boundary=(nil)/-1 Session Wait History: elapsed time of 0.000212 sec since last wait 0: waited for 'GCR CSS group query' =0x0, =0x0, =0x0 wait_id=14436260 seq_num=18723 snap_id=2 wait times: snap=0.000001 sec, exc=0.000003 sec, total=0.000703 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 1: waited for 'CSS operation: data query' function_id=0x3, =0x0, =0x0 wait_id=14436261 seq_num=18722 snap_id=1 wait times: snap=0.000700 sec, exc=0.000700 sec, total=0.000700 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 2: waited for 'GCR CSS group query' =0x0, =0x0, =0x0 wait_id=14436260 seq_num=18721 snap_id=1 wait times: snap=0.000002 sec, exc=0.000002 sec, total=0.000002 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000205 sec of elapsed time 3: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436259 seq_num=18720 snap_id=1 wait times: snap=0.150108 sec, exc=0.150108 sec, total=0.150108 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.005407 sec of elapsed time 4: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436258 seq_num=18719 snap_id=1 wait times: snap=5.001340 sec, exc=5.001340 sec, total=5.001340 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000084 sec of elapsed time 5: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436256 seq_num=18718 snap_id=2 wait times: snap=0.000133 sec, exc=2.321943 sec, total=5.001918 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 6: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436257 seq_num=18717 snap_id=1 wait times: snap=2.679975 sec, exc=2.679975 sec, total=2.679975 sec wait times: max=2.670000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 7: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436256 seq_num=18716 snap_id=1 wait times: snap=2.321810 sec, exc=2.321810 sec, total=2.321810 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000211 sec of elapsed time 8: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436255 seq_num=18715 snap_id=1 wait times: snap=5.000770 sec, exc=5.000770 sec, total=5.000770 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.003562 sec of elapsed time 9: waited for 'GCR CSS group query' =0x0, =0x0, =0x0 wait_id=14436253 seq_num=18714 snap_id=2 wait times: snap=0.000002 sec, exc=0.000004 sec, total=0.000416 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time Sampled Session History of session 9560 serial 54991 --------------------------------------------------- The sampled session history is constructed by sampling the target session every 1 second. The sampling process captures at each sample if the session is in a non-idle wait, an idle wait, or not in a wait. If the session is in a non-idle wait then one interval is shown for all the samples the session was in the same non-idle wait. If the session is in an idle wait or not in a wait for consecutive samples then one interval is shown for all the consecutive samples. Though we display these consecutive samples in a single interval the session may NOT be continuously idle or not in a wait (the sampling process does not know). The history is displayed in reverse chronological order. sample interval: 1 sec, max history 120 sec --------------------------------------------------- [5 samples, 14:50:56 - 14:51:00] waited for 'ksdxexeotherwait', seq_num: 18719 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: >= 4 sec (still in wait) [3 samples, 14:50:53 - 14:50:55] waited for 'ksdxexeotherwait', seq_num: 18717 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.679975 sec (sample interval: 2 sec) [2 samples, 14:50:51 - 14:50:52] waited for 'process diagnostic dump', seq_num: 18716 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.321810 sec (sample interval: 1 sec) [5 samples, 14:50:46 - 14:50:50] waited for 'ksdxexeotherwait', seq_num: 18715 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.000770 sec (sample interval: 4 sec) [5 samples, 14:50:41 - 14:50:45] waited for 'ksdxexeotherwait', seq_num: 18710 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001112 sec (sample interval: 4 sec) [3 samples, 14:50:38 - 14:50:40] waited for 'ksdxexeotherwait', seq_num: 18708 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.684390 sec (sample interval: 2 sec) [2 samples, 14:50:36 - 14:50:37] waited for 'process diagnostic dump', seq_num: 18707 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.316991 sec (sample interval: 1 sec) [5 samples, 14:50:31 - 14:50:35] waited for 'ksdxexeotherwait', seq_num: 18706 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.000485 sec (sample interval: 4 sec) [1 sample, 14:50:30] idle wait at each sample [5 samples, 14:50:25 - 14:50:29] waited for 'ksdxexeotherwait', seq_num: 18699 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001861 sec (sample interval: 4 sec) [2 samples, 14:50:23 - 14:50:24] waited for 'ksdxexeotherwait', seq_num: 18697 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.654137 sec (sample interval: 1 sec) [3 samples, 14:50:20 - 14:50:22] waited for 'process diagnostic dump', seq_num: 18696 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.347079 sec (sample interval: 2 sec) [5 samples, 14:50:15 - 14:50:19] waited for 'ksdxexeotherwait', seq_num: 18695 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001145 sec (sample interval: 4 sec) [75 samples, 14:49:01 - 14:50:14] idle wait at each sample --------------------------------------------------- Sampled Session History Summary: longest_non_idle_wait: 'ksdxexeotherwait' [5 samples, 14:50:25 - 14:50:29] time_waited: 5.001861 sec (sample interval: 4 sec) --------------------------------------------------- ---------------------------------------- kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0 kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0 temporary object counter: 0 last pdbid to increment: 0, last function to increment: (null) BPS: empty ---------------------------------------- Virtual Thread: kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991 vc: (nil), proc: 0x566000f310, id: 9560 consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig: vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2 inwait: 0, short wait event: 0 posted_run: 0 lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)' lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)' location where insched last set: kgskthrrun location where insched last cleared: kgskthrrun1 location where inwait last set: NULL location where inwait last cleared: NULL is_assigned: 0, in_sched: 0 (0) qcls: (nil), qlink: FALSE vt_active: 0 (pending: 0) vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0) ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0) used quanta (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 exec start consumed time: 0 exec start elapsed time: 0 idle time: 0, active time: 0 (cg: 0) last updnumps: 0, active time (pq: 0 ps: 0) cpu yields: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu waits: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu wait time (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 ASL queued time outs: 0, time: 0 (cur 0, cg 0) PQQ queued time outs: 0, time: 0 (cur 0, cg 0) Queue timeout violation: 0 calls aborted: 0, num est exec limit hit: 0 undo current: 0k max: 0k KTU Session Commit Cache Dump for IDLs: KTU Session Commit Cache Dump for Non-IDLs: ---------------------------------------- KKS-UOL used : 0 locks(used=0, free=0) KGX Atomic Operation Log 0x973f6daae8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab38 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab88 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dabd8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 ---------------------------------------- KGL-UOL SO Cache(total=0, free=0) KGX Atomic Operation Log 0x971f689230 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689298 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689300 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689368 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6893d0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689438 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6894a0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689508 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689570 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 ---------------------------------------- SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1 SGA version=(1,0) (dummy) nxc=0, nlb=0 *** 2019-11-15T14:51:06.209982+04:00 (CDB$ROOT(1)) ============================== IPC0 (ospid: 368909) has not moved for 67 sec (1573815060.1573814993) : heartbeat check status 2 (acceptable) (threshold 70 sec) : heartbeat poke time 0x5dce82db req 0x3403b ack 0x3403a freq 10 : heartbeat state 0x2.ffff () pso-flag 0x0 : Not in wait; last wait ended 63 secs ago. : last wait_id 3106025 at 'rdbms ipc message'. ============================== Dumping PROCESS IPC0 (ospid: 368909) States ============================== ===[ Callstack ]=== *** 2019-11-15T14:51:06.210105+04:00 (CDB$ROOT(1)) Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909, pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431 ------------------------------------------------------------------------------- os thread scheduling delay history: (sampling every 1.000000 secs) 0.000000 secs at [ 14:51:06 ] NOTE: scheduling delay has not been sampled for 0.092426 secs 0.000000 secs from [ 14:51:02 - 14:51:07 ], 5 sec avg 0.000000 secs from [ 14:50:06 - 14:51:07 ], 1 min avg 0.000000 secs from [ 14:46:06 - 14:51:07 ], 5 min avg *** 2019-11-15T14:51:08.523175+04:00 (CDB$ROOT(1)) loadavg : 300.42 78.79 28.87 System user time: 0.05 sys time: 0.01 context switch: 54992 Memory (Avail / Total) = 6361.30M / 772256.64M Swap (Avail / Total) = 24000.20M / 24576.00M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:19 ora_ipc0_cdbiamas1 Name: ora_ipc0_cdbiam State: R (running) Tgid: 368909 Ngid: 0 Pid: 368909 PPid: 1 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 128 Groups: 1001 1002 NStgid: 368909 NSpid: 368909 NSpgid: 368909 NSsid: 368909 VmPeak: 668175752 kB VmSize: 668175748 kB VmLck: 0 kB VmPin: 562562088 kB VmHWM: 7413828 kB VmRSS: 7413828 kB VmData: 8388 kB VmStk: 156 kB VmExe: 343516 kB VmLib: 32424 kB VmPTE: 14924 kB VmPMD: 2172 kB VmSwap: 0 kB Threads: 1 SigQ: 6/3088249 SigPnd: 0000000000000800 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000016400207 SigCgt: 00000003c9887cf8 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff Seccomp: 0 Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff Cpus_allowed_list: 0-111 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 4589415 nonvoluntary_ctxt_switches: 14581 *** 2019-11-15T14:51:11.211384+04:00 (CDB$ROOT(1)) Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2687 ms ------------------------------------------------------------------------------- Process diagnostic dump actual duration=5.001000 sec (max dump time=5.000000 sec) *** 2019-11-15T14:51:11.211482+04:00 (CDB$ROOT(1)) *** 2019-11-15T14:51:16.213118+04:00 (CDB$ROOT(1)) kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat split3 check, id 1 failed *** 2019-11-15T14:51:16.213225+04:00 (CDB$ROOT(1)) kjgcr_StatCheckCPU: Current load ratio is high (91612/1369) *** 2019-11-15T14:51:16.668235+04:00 (CDB$ROOT(1)) kjfmGCR_HBCheckAll: elapse time between calls is 16 secs ---------------------------------------- SO: 0x566332e368, type: 4, owner: 0x566000f310, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=session, file=ksu.h LINE:15737, pg=1, conuid=1 SGA version=(1,0) (session) sid: 9560 ser: 54991 trans: (nil), creator: 0x566000f310 flags: (0x51) USR/- flags2: (0x409) -/-/INC flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/- DID: 0001-001F-000000040000-0000-00000000, short-term DID: txn branch: (nil) con_id/con_uid/con_name: 1/1/CDB$ROOT con_logonuid: 1 con_logonid: 1 con_scuid: 1 con_scid: 1 edition#: 0 user#/name: 0/SYS oct: 0, prv: 0, sql: (nil), psql: (nil) stats: 0x55bffcb060, PX stats: 0x110cb824 service name: SYS$BACKGROUND Current Wait Stack: Not in wait; last wait ended 0.000250 sec ago Wait State: fixed_waits=0 flags=0x21 boundary=(nil)/-1 Session Wait History: elapsed time of 0.000272 sec since last wait 0: waited for 'GCR CSS group update' =0x0, =0x0, =0x0 wait_id=14436269 seq_num=18734 snap_id=2 wait times: snap=0.000001 sec, exc=0.000002 sec, total=0.000452 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 1: waited for 'CSS operation: data update' function_id=0xd, =0x0, =0x0 wait_id=14436270 seq_num=18733 snap_id=1 wait times: snap=0.000450 sec, exc=0.000450 sec, total=0.000450 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 2: waited for 'GCR CSS group update' =0x0, =0x0, =0x0 wait_id=14436269 seq_num=18732 snap_id=1 wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec wait times: max=infinite wait counts: calls=0 os=0 occurred after 0.000068 sec of elapsed time 3: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436268 seq_num=18731 snap_id=1 wait times: snap=0.149957 sec, exc=0.149957 sec, total=0.149957 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000082 sec of elapsed time 4: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436267 seq_num=18730 snap_id=1 wait times: snap=0.149864 sec, exc=0.149864 sec, total=0.149864 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000110 sec of elapsed time 5: waited for 'GCR sleep' =0x0, =0x0, =0x0 wait_id=14436266 seq_num=18729 snap_id=1 wait times: snap=0.149669 sec, exc=0.149669 sec, total=0.149669 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.005414 sec of elapsed time 6: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436265 seq_num=18728 snap_id=1 wait times: snap=5.001068 sec, exc=5.001068 sec, total=5.001068 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000079 sec of elapsed time 7: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436263 seq_num=18727 snap_id=2 wait times: snap=0.000137 sec, exc=2.313499 sec, total=5.001312 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 8: waited for 'ksdxexeotherwait' =0x0, =0x0, =0x0 wait_id=14436264 seq_num=18726 snap_id=1 wait times: snap=2.687813 sec, exc=2.687813 sec, total=2.687813 sec wait times: max=2.680000 sec wait counts: calls=0 os=0 occurred after 0.000000 sec of elapsed time 9: waited for 'process diagnostic dump' =0x0, =0x0, =0x0 wait_id=14436263 seq_num=18725 snap_id=1 wait times: snap=2.313362 sec, exc=2.313362 sec, total=2.313362 sec wait times: max=5.000000 sec wait counts: calls=0 os=0 occurred after 0.000219 sec of elapsed time Sampled Session History of session 9560 serial 54991 --------------------------------------------------- The sampled session history is constructed by sampling the target session every 1 second. The sampling process captures at each sample if the session is in a non-idle wait, an idle wait, or not in a wait. If the session is in a non-idle wait then one interval is shown for all the samples the session was in the same non-idle wait. If the session is in an idle wait or not in a wait for consecutive samples then one interval is shown for all the consecutive samples. Though we display these consecutive samples in a single interval the session may NOT be continuously idle or not in a wait (the sampling process does not know). The history is displayed in reverse chronological order. sample interval: 1 sec, max history 120 sec --------------------------------------------------- [5 samples, 14:51:11 - 14:51:15] waited for 'ksdxexeotherwait', seq_num: 18728 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: >= 4 sec (still in wait) [3 samples, 14:51:08 - 14:51:10] waited for 'ksdxexeotherwait', seq_num: 18726 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.687813 sec (sample interval: 2 sec) [2 samples, 14:51:06 - 14:51:07] waited for 'process diagnostic dump', seq_num: 18725 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.313362 sec (sample interval: 1 sec) [5 samples, 14:51:01 - 14:51:05] waited for 'ksdxexeotherwait', seq_num: 18724 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.000459 sec (sample interval: 4 sec) [5 samples, 14:50:56 - 14:51:00] waited for 'ksdxexeotherwait', seq_num: 18719 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001340 sec (sample interval: 4 sec) [3 samples, 14:50:53 - 14:50:55] waited for 'ksdxexeotherwait', seq_num: 18717 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.679975 sec (sample interval: 2 sec) [2 samples, 14:50:51 - 14:50:52] waited for 'process diagnostic dump', seq_num: 18716 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.321810 sec (sample interval: 1 sec) [5 samples, 14:50:46 - 14:50:50] waited for 'ksdxexeotherwait', seq_num: 18715 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.000770 sec (sample interval: 4 sec) [5 samples, 14:50:41 - 14:50:45] waited for 'ksdxexeotherwait', seq_num: 18710 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001112 sec (sample interval: 4 sec) [3 samples, 14:50:38 - 14:50:40] waited for 'ksdxexeotherwait', seq_num: 18708 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.684390 sec (sample interval: 2 sec) [2 samples, 14:50:36 - 14:50:37] waited for 'process diagnostic dump', seq_num: 18707 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.316991 sec (sample interval: 1 sec) [5 samples, 14:50:31 - 14:50:35] waited for 'ksdxexeotherwait', seq_num: 18706 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.000485 sec (sample interval: 4 sec) [1 sample, 14:50:30] idle wait at each sample [5 samples, 14:50:25 - 14:50:29] waited for 'ksdxexeotherwait', seq_num: 18699 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001861 sec (sample interval: 4 sec) [2 samples, 14:50:23 - 14:50:24] waited for 'ksdxexeotherwait', seq_num: 18697 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.654137 sec (sample interval: 1 sec) [3 samples, 14:50:20 - 14:50:22] waited for 'process diagnostic dump', seq_num: 18696 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 2.347079 sec (sample interval: 2 sec) [5 samples, 14:50:15 - 14:50:19] waited for 'ksdxexeotherwait', seq_num: 18695 p1: ''=0x0 p2: ''=0x0 p3: ''=0x0 time_waited: 5.001145 sec (sample interval: 4 sec) [60 samples, 14:49:16 - 14:50:14] idle wait at each sample --------------------------------------------------- Sampled Session History Summary: longest_non_idle_wait: 'ksdxexeotherwait' [5 samples, 14:50:25 - 14:50:29] time_waited: 5.001861 sec (sample interval: 4 sec) --------------------------------------------------- ---------------------------------------- kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0 kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0 temporary object counter: 0 last pdbid to increment: 0, last function to increment: (null) BPS: empty ---------------------------------------- Virtual Thread: kgskvt: 0x9764fbecb8, sess: 0x566332e368/1 sid: 9560 ser: 54991 vc: (nil), proc: 0x566000f310, id: 9560 consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig: vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2 inwait: 0, short wait event: 0 posted_run: 0 lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)' lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)' location where insched last set: kgskthrrun location where insched last cleared: kgskthrrun1 location where inwait last set: NULL location where inwait last cleared: NULL is_assigned: 0, in_sched: 0 (0) qcls: (nil), qlink: FALSE vt_active: 0 (pending: 0) vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0) ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0) used quanta (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 exec start consumed time: 0 exec start elapsed time: 0 idle time: 0, active time: 0 (cg: 0) last updnumps: 0, active time (pq: 0 ps: 0) cpu yields: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu waits: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu wait time (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 ASL queued time outs: 0, time: 0 (cur 0, cg 0) PQQ queued time outs: 0, time: 0 (cur 0, cg 0) Queue timeout violation: 0 calls aborted: 0, num est exec limit hit: 0 undo current: 0k max: 0k KTU Session Commit Cache Dump for IDLs: KTU Session Commit Cache Dump for Non-IDLs: ---------------------------------------- KKS-UOL used : 0 locks(used=0, free=0) KGX Atomic Operation Log 0x973f6daae8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab38 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dab88 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x973f6dabd8 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 9560 efd 0 whr 0 slp 0 ---------------------------------------- KGL-UOL SO Cache(total=0, free=0) KGX Atomic Operation Log 0x971f689230 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689298 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689300 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689368 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6893d0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689438 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f6894a0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689508 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x971f689570 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 9560 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 ---------------------------------------- SO: 0x5592142818, type: 79, owner: 0x566332e368, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x566000f310, name=dummy, file=ktccts.h LINE:456, pg=1, conuid=1 SGA version=(1,0) (dummy) nxc=0, nlb=0 *** 2019-11-15T14:51:21.673454+04:00 (CDB$ROOT(1)) ============================== IPC0 (ospid: 368909) has not moved for 83 sec (1573815076.1573814993) : heartbeat check status 6 (no-heartbeat) (threshold 70 sec) *** 2019-11-15T14:51:26.675662+04:00 (CDB$ROOT(1)) ================================================== === IPC0 (ospid: 368909) Heartbeat Report ================================================== IPC0 (ospid: 368909) has no heartbeats for 93 sec. (threshold 70) : heartbeat state 0x2.ffff () pso-flag 0x0 : Not in wait; last wait ended 78 secs ago. : last wait_id 3106025 at 'rdbms ipc message'. ============================== Dumping PROCESS IPC0 (ospid: 368909) States ============================== ===[ System Load State ]=== CPU Total 96 Raw 96 Core 48 Socket 2 Load normal: Cur 104133 Highmark 122880 (406.76 480.00) ===[ Latch State ]=== Not in Latch Get ===[ Session State Object ]=== ---------------------------------------- SO: 0x18e1753030, type: 4, owner: 0x18e0002c50, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x18e0002c50, name=session, file=ksu.h LINE:15737, pg=0, conuid=1 SGA version=(1,0) (session) sid: 727 ser: 53431 trans: (nil), creator: 0x18e0002c50 flags: (0x51) USR/- flags2: (0x409) -/-/INC flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/- DID: 0000-0000-000000000000-0000-00000000, short-term DID: txn branch: (nil) con_id/con_uid/con_name: 1/1/CDB$ROOT con_logonuid: 1 con_logonid: 1 con_scuid: 1 con_scid: 1 edition#: 0 user#/name: 0/SYS oct: 0, prv: 0, sql: (nil), psql: (nil) stats: 0x183fffc798, PX stats: 0x110cb824 service name: SYS$BACKGROUND Current Wait Stack: Not in wait; last wait ended 1 min 28 sec ago Wait State: fixed_waits=0 flags=0x21 boundary=(nil)/-1 Session Wait History: elapsed time of 1 min 28 sec since last wait 0: waited for 'rdbms ipc message' timeout=0x12c, =0x0, =0x0 wait_id=3106025 seq_num=35086 snap_id=1 wait times: snap=3.003118 sec, exc=3.003118 sec, total=3.003118 sec wait times: max=3.000000 sec wait counts: calls=1 os=1 occurred after 0.000011 sec of elapsed time 1: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106024 seq_num=35085 snap_id=1 wait times: snap=0.020098 sec, exc=0.020098 sec, total=0.020098 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050228 sec of elapsed time 2: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106023 seq_num=35084 snap_id=1 wait times: snap=0.020110 sec, exc=0.020110 sec, total=0.020110 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000015 sec of elapsed time 3: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106022 seq_num=35083 snap_id=1 wait times: snap=0.020180 sec, exc=0.020180 sec, total=0.020180 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.049185 sec of elapsed time 4: waited for 'rdbms ipc message' timeout=0x69, =0x0, =0x0 wait_id=3106021 seq_num=35082 snap_id=1 wait times: snap=1.051154 sec, exc=1.051154 sec, total=1.051154 sec wait times: max=1.050000 sec wait counts: calls=1 os=1 occurred after 0.000011 sec of elapsed time 5: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106020 seq_num=35081 snap_id=1 wait times: snap=0.020146 sec, exc=0.020146 sec, total=0.020146 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050272 sec of elapsed time 6: waited for 'rdbms ipc message' timeout=0x12c, =0x0, =0x0 wait_id=3106019 seq_num=35080 snap_id=1 wait times: snap=1.878880 sec, exc=1.878880 sec, total=1.878880 sec wait times: max=3.000000 sec wait counts: calls=1 os=1 occurred after 0.000011 sec of elapsed time 7: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106018 seq_num=35079 snap_id=1 wait times: snap=0.020171 sec, exc=0.020171 sec, total=0.020171 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050231 sec of elapsed time 8: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106017 seq_num=35078 snap_id=1 wait times: snap=0.020184 sec, exc=0.020184 sec, total=0.020184 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000016 sec of elapsed time 9: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106016 seq_num=35077 snap_id=1 wait times: snap=0.020206 sec, exc=0.020206 sec, total=0.020206 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050239 sec of elapsed time Sampled Session History of session 727 serial 53431 --------------------------------------------------- The sampled session history is constructed by sampling the target session every 1 second. The sampling process captures at each sample if the session is in a non-idle wait, an idle wait, or not in a wait. If the session is in a non-idle wait then one interval is shown for all the samples the session was in the same non-idle wait. If the session is in an idle wait or not in a wait for consecutive samples then one interval is shown for all the consecutive samples. Though we display these consecutive samples in a single interval the session may NOT be continuously idle or not in a wait (the sampling process does not know). The history is displayed in reverse chronological order. sample interval: 1 sec, max history 120 sec --------------------------------------------------- [88 samples, 14:49:58 - 14:51:25] not in wait at each sample [9 samples, 14:49:49 - 14:49:57] idle wait at each sample [1 sample, 14:49:48] waited for 'RMA: IPC0 completion sync', seq_num: 35075 p1: 'send count'=0x0 p2: ''=0x0 p3: ''=0x23e time_waited: 0.020107 sec (sample interval: 0 sec) [4 samples, 14:49:44 - 14:49:47] idle wait at each sample [1 sample, 14:49:43] not in wait at each sample [13 samples, 14:49:30 - 14:49:42] idle wait at each sample [1 sample, 14:49:29] waited for 'RMA: IPC0 completion sync', seq_num: 35045 p1: 'send count'=0x0 p2: ''=0x0 p3: ''=0x23e time_waited: 0.020174 sec (sample interval: 0 sec) [4 samples, 14:49:26 - 14:49:28] idle wait at each sample --------------------------------------------------- Sampled Session History Summary: longest_non_idle_wait: 'RMA: IPC0 completion sync' [1 sample, 14:49:29 ] time_waited: 0.020174 sec (sample interval: 0 sec) --------------------------------------------------- ---------------------------------------- kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0 kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0 temporary object counter: 0 last pdbid to increment: 0, last function to increment: (null) BPS: empty ---------------------------------------- Virtual Thread: kgskvt: 0x976416cc20, sess: 0x18e1753030/1 sid: 727 ser: 53431 vc: (nil), proc: 0x18e0002c50, id: 727 consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig: vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2 inwait: 0, short wait event: 0 posted_run: 0 lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)' lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)' location where insched last set: kgskthrrun location where insched last cleared: kgskthrrun1 location where inwait last set: NULL location where inwait last cleared: NULL is_assigned: 0, in_sched: 0 (0) qcls: (nil), qlink: FALSE vt_active: 0 (pending: 0) vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0) ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0) used quanta (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 exec start consumed time: 0 exec start elapsed time: 0 idle time: 0, active time: 0 (cg: 0) last updnumps: 0, active time (pq: 0 ps: 0) cpu yields: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu waits: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu wait time (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 ASL queued time outs: 0, time: 0 (cur 0, cg 0) PQQ queued time outs: 0, time: 0 (cur 0, cg 0) Queue timeout violation: 0 calls aborted: 0, num est exec limit hit: 0 undo current: 0k max: 0k KTU Session Commit Cache Dump for IDLs: KTU Session Commit Cache Dump for Non-IDLs: ---------------------------------------- KKS-UOL used : 0 locks(used=0, free=0) KGX Atomic Operation Log 0x977f6f40a0 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x977f6f40f0 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x977f6f4140 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x977f6f4190 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 ---------------------------------------- KGL-UOL SO Cache(total=0, free=0) KGX Atomic Operation Log 0x975f73a2d8 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a340 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a3a8 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a410 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a478 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a4e0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a548 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a5b0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a618 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 ---------------------------------------- SO: 0x1812141ff8, type: 79, owner: 0x18e1753030, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x18e0002c50, name=dummy, file=ktccts.h LINE:456, pg=0, conuid=1 SGA version=(1,0) (dummy) nxc=0, nlb=0 ===[ Callstack ]=== *** 2019-11-15T14:51:26.683226+04:00 (CDB$ROOT(1)) Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909, pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431 ------------------------------------------------------------------------------- os thread scheduling delay history: (sampling every 1.000000 secs) 0.000000 secs at [ 14:51:26 ] NOTE: scheduling delay has not been sampled for 0.484605 secs 0.000000 secs from [ 14:51:22 - 14:51:27 ], 5 sec avg 0.000000 secs from [ 14:50:26 - 14:51:27 ], 1 min avg 0.000000 secs from [ 14:46:27 - 14:51:27 ], 5 min avg *** 2019-11-15T14:51:29.002962+04:00 (CDB$ROOT(1)) loadavg : 406.77 117.41 42.64 System user time: 0.07 sys time: 0.02 context switch: 66627 Memory (Avail / Total) = 6340.92M / 772256.64M Swap (Avail / Total) = 24000.20M / 24576.00M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:35 ora_ipc0_cdbiamas1 Name: ora_ipc0_cdbiam State: R (running) Tgid: 368909 Ngid: 0 Pid: 368909 PPid: 1 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 128 Groups: 1001 1002 NStgid: 368909 NSpid: 368909 NSpgid: 368909 NSsid: 368909 VmPeak: 668175752 kB VmSize: 668175748 kB VmLck: 0 kB VmPin: 562562088 kB VmHWM: 7413828 kB VmRSS: 7413828 kB VmData: 8388 kB VmStk: 156 kB VmExe: 343516 kB VmLib: 32424 kB VmPTE: 14924 kB VmPMD: 2172 kB VmSwap: 0 kB Threads: 1 SigQ: 11/3088249 SigPnd: 0000000000000800 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000016400207 SigCgt: 00000003c9887cf8 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff Seccomp: 0 Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff Cpus_allowed_list: 0-111 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 4589416 nonvoluntary_ctxt_switches: 14614 *** 2019-11-15T14:51:31.684041+04:00 (CDB$ROOT(1)) Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2680 ms ------------------------------------------------------------------------------- Process diagnostic dump actual duration=5.001000 sec (max dump time=5.000000 sec) *** 2019-11-15T14:51:31.684131+04:00 (CDB$ROOT(1)) ===[ Heartbeat State Object ]=== (lm heartbeat) state=0x38000000 active current heartbeat: time 0x5dce82d1 count 0x3403e where 0x1 last poke: time 0x5dce82db req 0x3403b ack 0x3403a freq 10 last wait: state 0x2 id 3106025 last record: time 0x5dce82dd count 0x3403e last check: time 0x5dce8324 misc: good dur 70 default 70 ckreq 10 tmupdcnt 0 hangcnt 1 Dump of memory from 0x000000979F1E9608 to 0x000000979F1E9818 979F1E9600 9F1D2450 00000097 [P$......] 979F1E9610 E9D93E40 00000096 9F1D2460 00000097 [@>......`$......] 979F1E9620 E9D93E50 00000096 9F1E9628 00000097 [P>......(.......] 979F1E9630 9F1E9628 00000097 E0002C50 00000018 [(.......P,......] 979F1E9640 6000A5A0 00000000 00000001 30435049 [...`........IPC0] 979F1E9650 00000000 38000000 00000000 00000000 [.......8........] 979F1E9660 00000046 00000046 00000000 0003403E [F...F.......>@..] 979F1E9670 5DCE82D1 00000001 00000000 0000000A [...]............] 979F1E9680 5DCE82DB 0003403B 0003403A 0000000A [...];@..:@......] 979F1E9690 0000000A 00000001 5DCE8324 5DCE8324 [........$..]$..]] 979F1E96A0 0003403E 5DCE82DD 5DCE8323 00000000 [>@.....]#..]....] 979F1E96B0 00000076 00000000 00000000 00000000 [v...............] 979F1E96C0 00000000 00000000 00000000 00000006 [................] 979F1E96D0 00000000 00000000 00000000 00000000 [................] 979F1E96E0 00000002 FFFFFFFF 00000000 00000000 [................] 979F1E96F0 00000000 00000000 002F64E9 00000000 [.........d/.....] 979F1E9700 04B4390A 00000000 00000007 00000000 [.9..............] 979F1E9710 00000000 00000000 00000000 00000000 [................] Repeat 4 times 979F1E9760 74C300A8 00000097 00000004 00000000 [...t............] 979F1E9770 00000000 00000000 00000000 370000FA [...............7] 979F1E9780 00000005 00000961 5DCE8324 00000000 [....a...$..]....] 979F1E9790 00000008 00000000 00000000 00000000 [................] 979F1E97A0 00000000 00000000 00000000 00000000 [................] Repeat 1 times 979F1E97C0 00000000 00000000 9F1E97C8 00000097 [................] 979F1E97D0 9F1E97C8 00000097 60048728 00000000 [........(..`....] 979F1E97E0 9F1D2620 00000097 00000000 00000000 [ &..............] 979F1E97F0 00000000 00000000 00000001 00000000 [................] 979F1E9800 00000000 00000000 00000000 00000000 [................] 979F1E9810 00000000 00000000 [........] ===[ Trace Buffer ]=== *** 2019-11-15T14:51:36.685443+04:00 (CDB$ROOT(1)) ===[ DLM IPC ]=== KJC Communication Dump: state 0x35 flags 0x0 mode 0x0 inst 1 inc 28 inst=1 inc=28 nrcv=9 nsp=9 nrcvbuf=1000 reg_msg: msgsz=640 mpools=2 cur=688 max=12300 init=17390 mp[0]: cur=685 out=(sga:3602.3530.0 initq:613) max=12297 ini=8695 pavl=0 pmax=0 mp[1]: cur=3 out=(sga:0.0.0 initq:3) max=13 ini=8695 pavl=0 pmax=0 big_msg: msgsz=8504 mpools=2 cur=86 max=947 init=17390 mp[0]: cur=66 out=(sga:0.0.0 initq:66) max=923 ini=8695 pavl=0 pmax=0 mp[1]: cur=46 out=(sga:0.0.0 initq:46) max=132 ini=8695 pavl=0 pmax=0 rsv_msg: msgsz=8504 mpools=1 cur=0 max=1 init=1000 mp[0]: cur=0 out=(sga:0.0.0 initq:0) max=1 tot=1000 pavl=0 pmax=0 rcvr: id=0 (orapid=26 LMD0) grp=1 ospid=368961 rcvr: id=8 (orapid=34 LMD2) grp=1 ospid=368989 rcvr: id=1 (orapid=28 LMS0) grp=2 ospid=368963 rcvr: id=5 (orapid=32 LMS4) grp=2 ospid=368977 rcvr: id=6 (orapid=27 LMS5) grp=2 ospid=368981 rcvr: id=4 (orapid=25 LMS3) grp=2 ospid=368972 rcvr: id=2 (orapid=23 LMS1) grp=2 ospid=368965 rcvr: id=7 (orapid=29 LMD1) grp=1 ospid=368983 rcvr: id=3 (orapid=30 LMS2) grp=2 ospid=368969 send proxy: id=0 orapid=26 ndst=1 (2:0 ) pso=0x18e000d1f0.(nil) state=x0 send proxy: id=8 orapid=34 ndst=1 (2:8 ) pso=0x18e0011430.(nil) state=x0 send proxy: id=1 orapid=28 ndst=1 (2:1 ) pso=0x18e000e280.(nil) state=x0 send proxy: id=5 orapid=32 ndst=1 (2:5 ) pso=0x18e00103a0.(nil) state=x0 send proxy: id=6 orapid=27 ndst=1 (2:6 ) pso=0x566000d1f0.(nil) state=x0 send proxy: id=4 orapid=25 ndst=1 (2:4 ) pso=0x566000c160.(nil) state=x0 send proxy: id=2 orapid=23 ndst=1 (2:2 ) pso=0x566000b0d0.(nil) state=x0 send proxy: id=7 orapid=29 ndst=1 (2:7 ) pso=0x566000e280.(nil) state=x0 send proxy: id=3 orapid=30 ndst=1 (2:3 ) pso=0x18e000f310.(nil) state=x0 GES resource limits: ges resources: cur=242473 max=503402 ini=241332 ges enqueues: cur=124210 max=157460 ini=363099 ges cresources: cur=124792 max=132896 gcs resources: cur=21223354 max=25452045 ini=77122167 gcs shadows: cur=18628493 max=18812915 ini=77122167 KJCTS state: seq-check=no timeout=yes waitticks=0x3 highload=no comm_interface=msgq-drm (rds) GES destination context: GES remote instance per receiver context: GES destination context: Dest=2 rcvr=0 inc=28 state=0x10041 tstate=0x0 batch-type=quick bmsg=(nil) tmout=0xd626881 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 81) sndq_flushs=44664975 last_flush_time=0x89a6388e8 flush_idx=4 sndq_flush_stats= [4]v:44664975|w:16|f:0|l:1|t:1000|s:1|m:1 [3]v:44664974|w:57|f:0|l:1|t:1000|s:1|m:1 [2]v:44664973|w:68|f:0|l:1|t:1000|s:1|m:1 [1]v:44664972|w:302|f:0|l:1|t:1000|s:1|m:1 [0]v:44664971|w:56|f:0|l:1|t:999|s:1|m:1 [9]v:44664970|w:188|f:0|l:1|t:1000|s:1|m:1 [8]v:44664969|w:81|f:0|l:1|t:1000|s:1|m:1 [7]v:44664968|w:25|f:0|l:1|t:999|s:1|m:1 [6]v:44664967|w:48|f:0|l:1|t:1000|s:1|m:1 [5]v:44664966|w:25|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=168981285 ack=0 snt=168981284 sync seq=0.168981284 inc=0 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.153767097,[PCM]0.0,[ALL]0.168981284 batch_snds=1683692 tot_msgs=3533432 max_sz=38 full_load=0 snd_seq=1683692 pbatch_snds=19271326 tot_msgs=19455116 sndq_msg_tot=46516315 tm=(0 5671) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 214 wqlen 29423807 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=1 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x13b91f9c msg_in_batch=0 tkt_total=1000 (min=1000) avl=748 sp_rsv=249 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 98) sndq_flushs=43452752 last_flush_time=0x89a6388f0 flush_idx=1 sndq_flush_stats= [1]v:43452752|w:16|f:0|l:1|t:999|s:1|m:1 [0]v:43452751|w:14|f:0|l:1|t:996|s:1|m:1 [9]v:43452750|w:69|f:0|l:1|t:999|s:1|m:1 [8]v:43452749|w:37|f:0|l:1|t:998|s:1|m:1 [7]v:43452748|w:149|f:0|l:1|t:998|s:1|m:1 [6]v:43452747|w:95|f:0|l:1|t:999|s:1|m:1 [5]v:43452746|w:13|f:0|l:1|t:999|s:1|m:1 [4]v:43452745|w:239|f:0|l:1|t:1000|s:1|m:1 [3]v:43452744|w:76|f:0|l:1|t:997|s:1|m:1 [2]v:43452743|w:36|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=132507620 ack=0 snt=132507619 sync seq=0.132507619 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85809507,[PCM]0.132507619,[ALL]0.132507619 batch_snds=1399886 tot_msgs=5033684 max_sz=96 full_load=642 snd_seq=1399886 pbatch_snds=21225728 tot_msgs=28383594 sndq_msg_tot=46911662 tm=(0 49774149) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41600432 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=2 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x109a8a06 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 97) sndq_flushs=42914870 last_flush_time=0x89a6388eb flush_idx=9 sndq_flush_stats= [9]v:42914870|w:26|f:0|l:1|t:999|s:1|m:1 [8]v:42914869|w:46|f:0|l:1|t:1000|s:1|m:1 [7]v:42914868|w:3|f:0|l:1|t:999|s:1|m:1 [6]v:42914867|w:1|f:0|l:1|t:1000|s:1|m:1 [5]v:42914866|w:0|f:0|l:1|t:1000|s:1|m:1 [4]v:42914865|w:13|f:0|l:1|t:999|s:1|m:1 [3]v:42914864|w:38|f:0|l:1|t:1000|s:1|m:1 [2]v:42914863|w:30|f:0|l:1|t:998|s:1|m:1 [1]v:42914862|w:21|f:0|l:1|t:999|s:1|m:1 [0]v:42914861|w:79|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=133268181 ack=0 snt=133268181 sync seq=0.133268181 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.89088728,[PCM]0.133268181,[ALL]0.133268181 batch_snds=1426683 tot_msgs=5103642 max_sz=77 full_load=569 snd_seq=1426683 pbatch_snds=20722409 tot_msgs=27863597 sndq_msg_tot=46412046 tm=(0 50945990) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 364 wqlen 41640453 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=3 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x113ff945 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 95) sndq_flushs=43539285 last_flush_time=0x89a6388b9 flush_idx=4 sndq_flush_stats= [4]v:43539285|w:30|f:0|l:1|t:999|s:1|m:1 [3]v:43539284|w:55|f:0|l:1|t:1000|s:1|m:1 [2]v:43539283|w:55|f:0|l:1|t:1000|s:1|m:1 [1]v:43539282|w:39|f:0|l:1|t:998|s:1|m:1 [0]v:43539281|w:27|f:0|l:1|t:1000|s:1|m:1 [9]v:43539280|w:18|f:0|l:1|t:997|s:1|m:1 [8]v:43539279|w:41|f:0|l:1|t:999|s:1|m:1 [7]v:43539278|w:97|f:0|l:1|t:1000|s:1|m:1 [6]v:43539277|w:43|f:0|l:1|t:999|s:1|m:1 [5]v:43539276|w:78|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=134352993 ack=0 snt=134352990 sync seq=0.134352990 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.86869831,[PCM]0.134352990,[ALL]0.134352990 batch_snds=1432672 tot_msgs=5118093 max_sz=91 full_load=605 snd_seq=1432672 pbatch_snds=21305579 tot_msgs=28448192 sndq_msg_tot=47043265 tm=(0 51174746) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 375 wqlen 41622154 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=4 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x1168f832 msg_in_batch=0 tkt_total=1000 (min=1000) avl=747 sp_rsv=249 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 103) sndq_flushs=43168383 last_flush_time=0x89a6388ef flush_idx=2 sndq_flush_stats= [2]v:43168383|w:16|f:0|l:1|t:998|s:1|m:1 [1]v:43168382|w:79|f:0|l:1|t:1000|s:1|m:1 [0]v:43168381|w:4|f:0|l:1|t:998|s:1|m:1 [9]v:43168380|w:12|f:0|l:1|t:999|s:1|m:1 [8]v:43168379|w:22|f:0|l:1|t:1000|s:1|m:1 [7]v:43168378|w:3|f:0|l:1|t:1000|s:1|m:1 [6]v:43168377|w:106|f:0|l:1|t:1000|s:1|m:1 [5]v:43168376|w:41|f:0|l:1|t:999|s:1|m:1 [4]v:43168375|w:15|f:0|l:1|t:996|s:1|m:1 [3]v:43168374|w:41|f:0|l:1|t:998|s:1|m:1 seq wrp=0 lst=143196222 ack=0 snt=143196221 sync seq=0.143196221 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.96323855,[PCM]0.143196221,[ALL]0.143196221 batch_snds=1427942 tot_msgs=5134969 max_sz=59 full_load=601 snd_seq=1427942 pbatch_snds=20915126 tot_msgs=28054314 sndq_msg_tot=46695919 tm=(0 50968179) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 346 wqlen 41912172 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=5 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x139c13ee msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 94) sndq_flushs=43574282 last_flush_time=0x89a6388b5 flush_idx=1 sndq_flush_stats= [1]v:43574282|w:15|f:0|l:1|t:997|s:1|m:1 [0]v:43574281|w:5|f:0|l:1|t:1000|s:1|m:1 [9]v:43574280|w:42|f:1|l:1|t:997|s:1|m:1 [8]v:43574279|w:85|f:0|l:1|t:999|s:1|m:1 [7]v:43574278|w:68|f:0|l:1|t:1000|s:1|m:1 [6]v:43574277|w:79|f:0|l:1|t:1000|s:1|m:1 [5]v:43574276|w:12|f:0|l:1|t:998|s:1|m:1 [4]v:43574275|w:33|f:0|l:1|t:999|s:1|m:1 [3]v:43574274|w:2|f:0|l:1|t:1000|s:1|m:1 [2]v:43574273|w:59|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=137469430 ack=0 snt=137469427 sync seq=0.137469427 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.93123683,[PCM]0.137469427,[ALL]0.137469427 batch_snds=1415019 tot_msgs=5079675 max_sz=74 full_load=587 snd_seq=1415019 pbatch_snds=21334518 tot_msgs=28460067 sndq_msg_tot=47061098 tm=(0 49472680) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41481937 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=6 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x14b8d66c msg_in_batch=0 tkt_total=1000 (min=1000) avl=748 sp_rsv=249 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 96) sndq_flushs=43191530 last_flush_time=0x89a6388ed flush_idx=9 sndq_flush_stats= [9]v:43191530|w:20|f:0|l:1|t:1000|s:1|m:1 [8]v:43191529|w:84|f:0|l:1|t:1000|s:1|m:1 [7]v:43191528|w:31|f:0|l:1|t:1000|s:1|m:1 [6]v:43191527|w:8|f:0|l:1|t:997|s:1|m:1 [5]v:43191526|w:24|f:0|l:1|t:998|s:1|m:1 [4]v:43191525|w:79|f:0|l:1|t:999|s:1|m:1 [3]v:43191524|w:149|f:0|l:1|t:977|s:1|m:1 [2]v:43191523|w:2|f:0|l:1|t:999|s:1|m:1 [1]v:43191522|w:1|f:0|l:1|t:1000|s:1|m:1 [0]v:43191521|w:21|f:0|l:1|t:993|s:1|m:1 seq wrp=0 lst=129460055 ack=0 snt=129460053 sync seq=0.129460053 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85305884,[PCM]0.129460053,[ALL]0.129460053 batch_snds=1414189 tot_msgs=5124380 max_sz=67 full_load=652 snd_seq=1414189 pbatch_snds=20885815 tot_msgs=28060372 sndq_msg_tot=46719857 tm=(0 50088934) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 359 wqlen 42071557 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=7 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0xdc9a426 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 54) sndq_flushs=46048278 last_flush_time=0x89a6388ea flush_idx=7 sndq_flush_stats= [7]v:46048278|w:2|f:0|l:1|t:1000|s:1|m:1 [6]v:46048277|w:0|f:0|l:1|t:1000|s:1|m:1 [5]v:46048276|w:48|f:0|l:1|t:999|s:1|m:1 [4]v:46048275|w:24|f:0|l:1|t:1000|s:1|m:1 [3]v:46048274|w:2|f:0|l:1|t:1000|s:1|m:1 [2]v:46048273|w:31|f:0|l:1|t:999|s:1|m:1 [1]v:46048272|w:37|f:0|l:1|t:1000|s:1|m:1 [0]v:46048271|w:85|f:0|l:1|t:1000|s:1|m:1 [9]v:46048270|w:45|f:0|l:1|t:999|s:1|m:1 [8]v:46048269|w:61|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=163062139 ack=0 snt=163062139 sync seq=0.163062139 inc=0 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.210,[PCM]0.0,[ALL]0.163062139 batch_snds=1938346 tot_msgs=3972927 max_sz=39 full_load=0 snd_seq=1938346 pbatch_snds=14296974 tot_msgs=14477098 sndq_msg_tot=48084777 tm=(0 5829) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 110 wqlen 35949749 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=8 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0xcf125a2 msg_in_batch=0 tkt_total=1000 (min=1000) avl=746 sp_rsv=248 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 55) sndq_flushs=38362337 last_flush_time=0x89a6388e8 flush_idx=6 sndq_flush_stats= [6]v:38362337|w:1|f:0|l:1|t:997|s:1|m:1 [5]v:38362336|w:69|f:0|l:1|t:998|s:1|m:1 [4]v:38362335|w:166|f:0|l:1|t:997|s:1|m:1 [3]v:38362334|w:68|f:0|l:1|t:1000|s:1|m:1 [2]v:38362333|w:42|f:0|l:1|t:1000|s:1|m:1 [1]v:38362332|w:73|f:0|l:1|t:999|s:1|m:1 [0]v:38362331|w:80|f:0|l:1|t:1000|s:1|m:1 [9]v:38362330|w:31|f:0|l:1|t:999|s:1|m:1 [8]v:38362329|w:43|f:0|l:1|t:997|s:1|m:1 [7]v:38362328|w:42|f:0|l:2|t:998|s:1|m:2 seq wrp=0 lst=161548233 ack=0 snt=161548231 sync seq=0.161548231 inc=0 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.218,[PCM]0.0,[ALL]0.161548231 batch_snds=1587433 tot_msgs=3357402 max_sz=45 full_load=0 snd_seq=1587433 pbatch_snds=15617700 tot_msgs=15800503 sndq_msg_tot=40133301 tm=(0 4194) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 218 wqlen 26651687 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 GES remote instance per receiver context: Rcvr=0 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.169438649 sync_rcv_seq=0.169438649 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1452889 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.166858040 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=8 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.160789037 sync_rcv_seq=0.160789031 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1208966 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.158381798 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=1 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.141250908 sync_rcv_seq=0.141250907 sync_ack_seq=0.129056901 inc=28 batch_rcv_seq=1845551 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.134058573 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=5 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.141357194 sync_rcv_seq=0.141357194 sync_ack_seq=0.131227202 inc=28 batch_rcv_seq=1858785 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.135396667 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=6 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.150276081 sync_rcv_seq=0.150276079 sync_ack_seq=0.137776658 inc=28 batch_rcv_seq=1893391 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.142779468 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=4 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.125328139 sync_rcv_seq=0.125328139 sync_ack_seq=0.114702593 inc=28 batch_rcv_seq=1863433 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.119202571 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=2 dest=2 inc=28 state=0x103 tstate=0x0 nak=2 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.124480280 sync_rcv_seq=0.124480280 sync_ack_seq=0.114194111 inc=28 batch_rcv_seq=1875123 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.118453984 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=7 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.165115287 sync_rcv_seq=0.165115287 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1989087 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.163643671 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=3 dest=2 inc=28 state=0x103 tstate=0x0 nak=5 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.120369356 sync_rcv_seq=0.120369356 sync_ack_seq=0.110107175 inc=28 batch_rcv_seq=1849776 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.114270602 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=0 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:111) msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0 rmpool: alloc=1334 (r:1311 s:23) avail=134 (r:111 s:23) max=(s:23) hist_last_rcv_tm=0x89a6388e9 count=224553208 idx=3 [3]t:x89a635f15|c:791|w:13|x:(668.123.0.0)99|m:552 [2]t:x89a62e9aa|c:2574|w:11|x:(2251.323.0.0)99|m:1817 [1]t:x89a62746e|c:19120|w:1|x:(18828.292.0.0)99|m:14268 [0]t:x89a61ff34|c:2824|w:10|x:(2526.298.0.0)99|m:2057 [9]t:x89a618a02|c:3096|w:9|x:(2785.311.0.0)99|m:2195 [8]t:x89a6114b8|c:3643|w:8|x:(3348.295.0.0)99|m:2806 [7]t:x89a609f72|c:3880|w:7|x:(3598.282.0.0)99|m:3044 [6]t:x89a602a2a|c:17602|w:1|x:(17318.284.0.0)99|m:15328 [5]t:x89a5fb4e7|c:3445|w:8|x:(3155.290.0.0)99|m:2819 [4]t:x89a5f3fa4|c:3861|w:7|x:(3566.295.0.0)98|m:3119 ibatch: 2[1372526] 3[53116] 4[16271] 5[4951] 6[2397] 7[1341] ibatch: 8[909] 9[761] 10[450] 11[173] 12[87] 13[63] ibatch: 14[28] 15[15] 16[11] 17[6] ibatch: sum=1453105 rcvs=1453215 tot_msgs=3042811 max_sz=35 avg_sz=2 pbatch: 2[7672] 3[415] 4[136] 5[61] 6[57] 7[17] pbatch: 8[4] 9[3] 10[4] 11[1] 12[0] 13[4] pbatch: 14[1] 15[5] 16[4] 17[1] pbatch: sum=8385 rcvs=20424692 tot_msgs=20579530 max_sz=103 avg_sz=2 Rcvr=8 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:157) msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0 rmpool: alloc=1380 (r:1357 s:23) avail=180 (r:157 s:23) max=(s:23) hist_last_rcv_tm=0x89a6388e8 count=217130446 idx=3 [3]t:x89a6364ac|c:752|w:12|x:(650.102.0.0)99|m:517 [2]t:x89a62ef5b|c:2631|w:11|x:(2307.324.0.0)99|m:1803 [1]t:x89a627a0c|c:16992|w:1|x:(16699.293.0.0)99|m:12695 [0]t:x89a6204bb|c:2630|w:11|x:(2297.333.0.0)99|m:1843 [9]t:x89a618f38|c:3198|w:9|x:(2887.311.0.0)99|m:2092 [8]t:x89a611a02|c:2654|w:11|x:(2321.333.0.0)99|m:1778 [7]t:x89a60a49e|c:2621|w:11|x:(2306.315.0.0)99|m:1828 [6]t:x89a602f6d|c:15008|w:2|x:(14713.295.0.0)99|m:13063 [5]t:x89a5fba3d|c:1997|w:15|x:(1652.345.0.0)99|m:1560 [4]t:x89a5f44ed|c:2134|w:14|x:(1795.339.0.0)99|m:1621 ibatch: 2[1113837] 3[61797] 4[20539] 5[6821] 6[3212] 7[1341] ibatch: 8[743] 9[391] 10[198] 11[116] 12[59] 13[35] ibatch: 14[21] 15[15] 16[7] 17[9] ibatch: sum=1209141 rcvs=1209227 tot_msgs=2573520 max_sz=38 avg_sz=2 pbatch: 2[5158] 3[535] 4[201] 5[91] 6[37] 7[31] pbatch: 8[20] 9[19] 10[13] 11[3] 12[6] 13[10] pbatch: 14[5] 15[6] 16[2] 17[9] pbatch: sum=6146 rcvs=19070248 tot_msgs=19222335 max_sz=103 avg_sz=2 Rcvr=1 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:364) msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0 rmpool: alloc=1633 (r:1564 s:69) avail=430 (r:364 s:66) max=(s:69) hist_last_rcv_tm=0x89a6388f0 count=330858922 idx=0 [0]t:x89a635b9b|c:2136|w:5|x:(2136.0.0.0)30|m:853 [9]t:x89a62e661|c:8054|w:3|x:(8054.0.0.0)30|m:3383 [8]t:x89a62712e|c:6558|w:4|x:(6558.0.0.0)30|m:2955 [7]t:x89a61fbfd|c:7561|w:3|x:(7561.0.0.0)30|m:3010 [6]t:x89a6186bf|c:5868|w:5|x:(5868.0.0.0)30|m:2357 [5]t:x89a61118a|c:5499|w:5|x:(5499.0.0.0)30|m:2345 [4]t:x89a609c55|c:6241|w:4|x:(6241.0.0.0)30|m:2705 [3]t:x89a60271c|c:7254|w:4|x:(7254.0.0.0)30|m:2809 [2]t:x89a5fb1e5|c:6797|w:4|x:(6797.0.0.0)30|m:3079 [1]t:x89a5f3cb2|c:6626|w:4|x:(6626.0.0.0)30|m:2850 ibatch: 2[1220363] 3[251612] 4[102831] 5[55882] 6[38196] 7[30964] ibatch: 8[23999] 9[20125] 10[16840] 11[15151] 12[13532] 13[12454] ibatch: 14[11550] 15[10833] 16[9778] 17[3805] ibatch: sum=1837915 rcvs=1849590 tot_msgs=6174214 max_sz=56 avg_sz=3 pbatch: 2[352286] 3[202796] 4[123721] 5[93438] 6[93402] 7[104490] pbatch: 8[27372] 9[18633] 10[17937] 11[15916] 12[15737] 13[13953] pbatch: 14[13953] 15[12735] 16[14167] 17[92343] pbatch: sum=1212879 rcvs=20362030 tot_msgs=32519217 max_sz=74 avg_sz=5 Rcvr=5 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341) msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0 rmpool: alloc=1610 (r:1541 s:69) avail=407 (r:341 s:66) max=(s:69) hist_last_rcv_tm=0x89a6388f5 count=328938604 idx=0 [0]t:x89a634fbf|c:2905|w:5|x:(2905.0.0.0)30|m:1253 [9]t:x89a62da8c|c:5846|w:5|x:(5846.0.0.0)30|m:2312 [8]t:x89a626543|c:5009|w:5|x:(5009.0.0.0)30|m:2451 [7]t:x89a61f00d|c:4569|w:6|x:(4569.0.0.0)30|m:1761 [6]t:x89a617ad2|c:3664|w:8|x:(3664.0.0.0)30|m:1473 [5]t:x89a610592|c:4264|w:7|x:(4262.2.0.0)31|m:1713 [4]t:x89a609049|c:4347|w:6|x:(4347.0.0.0)30|m:1854 [3]t:x89a601b18|c:4616|w:6|x:(4616.0.0.0)30|m:1556 [2]t:x89a5fa5e7|c:4752|w:6|x:(4752.0.0.0)30|m:2229 [1]t:x89a5f30ab|c:5274|w:5|x:(5274.0.0.0)30|m:2301 ibatch: 2[1225447] 3[254830] 4[103664] 5[56407] 6[38776] 7[30966] ibatch: 8[24303] 9[20643] 10[17098] 11[15454] 12[13628] 13[12816] ibatch: 14[11667] 15[10919] 16[9954] 17[4018] ibatch: sum=1850590 rcvs=1862824 tot_msgs=6243422 max_sz=52 avg_sz=3 pbatch: 2[350360] 3[200988] 4[125538] 5[93312] 6[94203] 7[104713] pbatch: 8[28340] 9[18640] 10[18044] 11[15789] 12[16167] 13[13836] pbatch: 14[13789] 15[12668] 16[13887] 17[92690] pbatch: sum=1212964 rcvs=19829482 tot_msgs=31998448 max_sz=74 avg_sz=5 Rcvr=6 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318) msgs_rdyq=0 watq=0 nsq=0 routq=1202 msgq=0 rmpool: alloc=1564 (r:1518 s:46) avail=362 (r:318 s:44) max=(s:46) hist_last_rcv_tm=0x89a6388ed count=347611873 idx=0 [0]t:x89a63463c|c:4359|w:3|x:(4359.0.0.0)30|m:1782 [9]t:x89a62d10c|c:7888|w:3|x:(7888.0.0.0)30|m:2981 [8]t:x89a625bd0|c:8434|w:3|x:(8434.0.0.0)30|m:3542 [7]t:x89a61e697|c:6469|w:4|x:(6469.0.0.0)30|m:2630 [6]t:x89a617150|c:5699|w:5|x:(5698.1.0.0)31|m:2280 [5]t:x89a60fc19|c:6267|w:4|x:(6267.0.0.0)30|m:2647 [4]t:x89a6086e8|c:5200|w:5|x:(5200.0.0.0)30|m:2163 [3]t:x89a6011b7|c:6625|w:4|x:(6625.0.0.0)30|m:2467 [2]t:x89a5f9c7f|c:5407|w:5|x:(5406.1.0.0)31|m:2495 [1]t:x89a5f273e|c:7966|w:3|x:(7966.0.0.0)30|m:3396 ibatch: 2[1251337] 3[258991] 4[106698] 5[56215] 6[39919] 7[29475] ibatch: 8[24834] 9[19768] 10[18088] 11[15360] 12[14221] 13[12597] ibatch: 14[12181] 15[11011] 16[10537] 17[4081] ibatch: sum=1885313 rcvs=1897647 tot_msgs=6349480 max_sz=54 avg_sz=3 pbatch: 2[357532] 3[210027] 4[123428] 5[95483] 6[91888] 7[107378] pbatch: 8[25641] 9[20011] 10[16849] 11[16988] 12[14832] 13[14752] pbatch: 14[13117] 15[13450] 16[13519] 17[93193] pbatch: sum=1228088 rcvs=19965179 tot_msgs=32152763 max_sz=74 avg_sz=5 Rcvr=4 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341) msgs_rdyq=0 watq=0 nsq=0 routq=1202 msgq=0 rmpool: alloc=1587 (r:1541 s:46) avail=385 (r:341 s:44) max=(s:46) hist_last_rcv_tm=0x89a6388ef count=291797547 idx=0 [0]t:x89a638682|c:64|w:10|x:(64.0.0.0)30|m:26 [9]t:x89a63114b|c:6950|w:4|x:(6950.0.0.0)30|m:3238 [8]t:x89a629c1a|c:10508|w:2|x:(10508.0.0.0)30|m:4547 [7]t:x89a6226e8|c:10742|w:2|x:(10742.0.0.0)30|m:3977 [6]t:x89a61b1ad|c:5043|w:5|x:(5043.0.0.0)30|m:1965 [5]t:x89a613c72|c:6451|w:4|x:(6451.0.0.0)30|m:2806 [4]t:x89a60c742|c:5526|w:5|x:(5526.0.0.0)30|m:2345 [3]t:x89a605211|c:7198|w:4|x:(7198.0.0.0)30|m:2769 [2]t:x89a5fdcd1|c:7269|w:4|x:(7269.0.0.0)30|m:3115 [1]t:x89a5f678c|c:5222|w:5|x:(5222.0.0.0)30|m:2593 ibatch: 2[1234999] 3[252810] 4[104488] 5[54128] 6[38980] 7[29147] ibatch: 8[24513] 9[19667] 10[17880] 11[15098] 12[13992] 13[12561] ibatch: 14[12222] 15[11097] 16[10306] 17[3819] ibatch: sum=1855707 rcvs=1867971 tot_msgs=6256370 max_sz=51 avg_sz=3 pbatch: 2[344218] 3[202531] 4[120299] 5[95235] 6[92709] 7[105848] pbatch: 8[26248] 9[19662] 10[17784] 11[17509] 12[14867] 13[14646] pbatch: 14[13139] 15[13514] 16[13571] 17[92942] pbatch: sum=1204722 rcvs=20477905 tot_msgs=32639611 max_sz=74 avg_sz=5 Rcvr=2 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318) msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0 rmpool: alloc=1587 (r:1518 s:69) avail=384 (r:318 s:66) max=(s:69) hist_last_rcv_tm=0x89a6388f5 count=278280354 idx=9 [9]t:x89a6315dd|c:6025|w:4|x:(6025.0.0.0)30|m:2729 [8]t:x89a62a09b|c:5382|w:5|x:(5382.0.0.0)30|m:2632 [7]t:x89a622b69|c:4661|w:6|x:(4661.0.0.0)30|m:2142 [6]t:x89a61b61e|c:3939|w:7|x:(3939.0.0.0)30|m:1740 [5]t:x89a6140e4|c:4523|w:6|x:(4521.2.0.0)31|m:1908 [4]t:x89a60cbaa|c:3731|w:8|x:(3731.0.0.0)30|m:1652 [3]t:x89a60567a|c:5259|w:5|x:(5259.0.0.0)30|m:1912 [2]t:x89a5fe148|c:4399|w:6|x:(4399.0.0.0)30|m:1996 [1]t:x89a5f6c02|c:3823|w:7|x:(3823.0.0.0)30|m:1899 [0]t:x89a5ef6bf|c:5992|w:5|x:(5992.0.0.0)30|m:2797 ibatch: 2[1228028] 3[255817] 4[107012] 5[56903] 6[40928] 7[31133] ibatch: 8[25761] 9[21059] 10[18571] 11[15973] 12[14775] 13[12869] ibatch: 14[12242] 15[11229] 16[10392] 17[4092] ibatch: sum=1866784 rcvs=1879395 tot_msgs=6367991 max_sz=54 avg_sz=3 pbatch: 2[353486] 3[206296] 4[122255] 5[94494] 6[91116] 7[105765] pbatch: 8[25985] 9[19548] 10[16768] 11[17365] 12[14744] 13[14894] pbatch: 14[13731] 15[13582] 16[13514] 17[92954] pbatch: sum=1216497 rcvs=19822588 tot_msgs=31993720 max_sz=74 avg_sz=5 Rcvr=7 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:180) msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0 rmpool: alloc=1403 (r:1380 s:23) avail=203 (r:180 s:23) max=(s:23) hist_last_rcv_tm=0x89a6388ea count=231318606 idx=2 [2]t:x89a6349c7|c:1286|w:12|x:(1100.186.0.0)99|m:820 [1]t:x89a62d45b|c:2683|w:11|x:(2377.306.0.0)99|m:1711 [0]t:x89a625f09|c:20508|w:1|x:(20227.281.0.0)99|m:14946 [9]t:x89a61e9d2|c:2751|w:10|x:(2419.332.0.0)99|m:1722 [8]t:x89a61745d|c:2983|w:10|x:(2667.316.0.0)99|m:1876 [7]t:x89a60ff24|c:3786|w:7|x:(3473.313.0.0)99|m:2292 [6]t:x89a6089eb|c:5113|w:5|x:(4776.337.0.0)99|m:3577 [5]t:x89a6014a3|c:16750|w:1|x:(16441.309.0.0)99|m:14276 [4]t:x89a5f9f6b|c:3434|w:8|x:(3095.339.0.0)99|m:2341 [3]t:x89a5f2a25|c:3668|w:8|x:(3348.320.0.0)98|m:2414 ibatch: 2[1870524] 3[105708] 4[10127] 5[1884] 6[732] 7[414] ibatch: 8[170] 9[81] 10[69] 11[30] 12[19] 13[26] ibatch: 14[4] 15[0] 16[1] 17[1] ibatch: sum=1989790 rcvs=1989854 tot_msgs=4119282 max_sz=23 avg_sz=2 pbatch: 2[6307] 3[414] 4[92] 5[29] 6[15] 7[5] pbatch: 8[4] 9[3] 10[5] 11[5] 12[3] 13[8] pbatch: 14[4] 15[4] 16[6] 17[8] pbatch: sum=6912 rcvs=16772268 tot_msgs=16925126 max_sz=103 avg_sz=2 Rcvr=3 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341) msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0 rmpool: alloc=1610 (r:1541 s:69) avail=407 (r:341 s:66) max=(s:69) hist_last_rcv_tm=0x89a6388f6 count=289399211 idx=9 [9]t:x89a631ef8|c:5444|w:4|x:(5444.0.0.0)30|m:2194 [8]t:x89a62a9be|c:5590|w:5|x:(5590.0.0.0)30|m:2162 [7]t:x89a623484|c:5274|w:5|x:(5274.0.0.0)30|m:2524 [6]t:x89a61bf46|c:5719|w:5|x:(5719.0.0.0)30|m:2388 [5]t:x89a614a01|c:4346|w:6|x:(4346.0.0.0)30|m:1769 [4]t:x89a60d4c5|c:5610|w:5|x:(5610.0.0.0)30|m:2220 [3]t:x89a605f8d|c:4747|w:6|x:(4747.0.0.0)30|m:1892 [2]t:x89a5fea57|c:4861|w:6|x:(4861.0.0.0)30|m:1844 [1]t:x89a5f7526|c:4966|w:6|x:(4966.0.0.0)30|m:2315 [0]t:x89a5effec|c:6927|w:4|x:(6927.0.0.0)30|m:2932 ibatch: 2[1221232] 3[251727] 4[102904] 5[56754] 6[39175] 7[30881] ibatch: 8[24073] 9[20703] 10[16862] 11[15386] 12[13402] 13[12711] ibatch: 14[11581] 15[10830] 16[9697] 17[3984] ibatch: sum=1841902 rcvs=1855190 tot_msgs=6259867 max_sz=48 avg_sz=3 pbatch: 2[360080] 3[204156] 4[124610] 5[92957] 6[94169] 7[105196] pbatch: 8[28209] 9[17678] 10[18665] 11[15940] 12[15763] 13[13774] pbatch: 14[13801] 15[12728] 16[13885] 17[93027] pbatch: sum=1224638 rcvs=20360708 tot_msgs=32542328 max_sz=74 avg_sz=5 GES PGA send message buffer usage: orapid=23 ospid=368965 pmp: alloc=69 avail=66 max=69 orapid=25 ospid=368972 pmp: alloc=46 avail=44 max=46 orapid=26 ospid=368961 pmp: alloc=23 avail=23 max=23 orapid=27 ospid=368981 pmp: alloc=46 avail=44 max=46 orapid=28 ospid=368963 pmp: alloc=69 avail=66 max=69 orapid=29 ospid=368983 pmp: alloc=23 avail=23 max=23 orapid=30 ospid=368969 pmp: alloc=69 avail=66 max=69 orapid=32 ospid=368977 pmp: alloc=69 avail=66 max=69 orapid=34 ospid=368989 pmp: alloc=23 avail=23 max=23 orapid=71 ospid=369094 pmp: alloc=0 avail=0 max=23 orapid=74 ospid=369090 pmp: alloc=0 avail=0 max=23 orapid=75 ospid=369121 pmp: alloc=0 avail=0 max=23 orapid=78 ospid=369098 pmp: alloc=0 avail=0 max=23 orapid=79 ospid=369123 pmp: alloc=0 avail=0 max=23 orapid=82 ospid=369108 pmp: alloc=0 avail=0 max=23 kjctseventdump-begin tail 220 heads 218 @ 0 220 @ -817966651 #0 id 439 exid 77713 dests 00 #1 id 439 exid 73326 dests 00 #2 id 439 exid 77713 dests 00 #3 id 439 exid 73326 dests 00 #4 id 439 exid 19919 dests 00 #5 id 439 exid 73326 dests 00 #6 id 439 exid 19919 dests 00 #7 id 439 exid 73326 dests 00 #8 id 439 exid 77713 dests 00 #9 id 439 exid 19919 dests 00 #10 id 439 exid 73326 dests 00 #11 id 439 exid 19919 dests 00 #12 id 439 exid 73326 dests 00 #13 id 439 exid 77713 dests 00 #14 id 439 exid 73326 dests 00 #15 id 439 exid 73326 dests 00 #16 id 439 exid 77713 dests 00 #17 id 439 exid 73326 dests 00 #18 id 439 exid 19919 dests 00 #19 id 439 exid 73326 dests 00 #20 id 439 exid 19919 dests 00 #21 id 439 exid 73326 dests 00 #22 id 439 exid 19919 dests 00 #23 id 439 exid 73326 dests 00 #24 id 439 exid 77713 dests 00 #25 id 439 exid 77713 dests 00 #26 id 439 exid 73326 dests 00 #27 id 439 exid 19919 dests 00 #28 id 439 exid 73326 dests 00 #29 id 439 exid 73326 dests 00 #30 id 439 exid 73326 dests 00 #31 id 439 exid 19919 dests 00 #32 id 439 exid 19919 dests 00 #33 id 439 exid 73326 dests 00 #34 id 439 exid 77713 dests 00 #35 id 439 exid 73326 dests 00 #36 id 439 exid 73326 dests 00 #37 id 439 exid 77713 dests 00 #38 id 439 exid 73326 dests 00 #39 id 439 exid 19919 dests 00 #40 id 439 exid 73326 dests 00 #41 id 439 exid 19919 dests 00 #42 id 439 exid 73326 dests 00 #43 id 439 exid 73326 dests 00 #44 id 439 exid 19919 dests 00 #45 id 439 exid 73326 dests 00 #46 id 439 exid 77713 dests 00 #47 id 439 exid 73326 dests 00 #48 id 439 exid 77713 dests 00 #49 id 439 exid 19919 dests 00 #50 id 439 exid 19919 dests 00 #51 id 439 exid 73326 dests 00 #52 id 439 exid 19919 dests 00 #53 id 439 exid 73326 dests 00 #54 id 439 exid 19919 dests 00 #55 id 439 exid 19919 dests 00 #56 id 439 exid 77713 dests 00 #57 id 439 exid 73326 dests 00 #58 id 439 exid 73326 dests 00 #59 id 439 exid 73326 dests 00 #60 id 439 exid 77713 dests 00 #61 id 439 exid 77713 dests 00 #62 id 439 exid 73326 dests 00 #63 id 439 exid 19919 dests 00 #64 id 439 exid 73326 dests 00 #65 id 439 exid 19919 dests 00 #66 id 439 exid 73326 dests 00 #67 id 439 exid 19919 dests 00 #68 id 439 exid 73326 dests 00 #69 id 439 exid 77713 dests 00 #70 id 439 exid 77713 dests 00 #71 id 439 exid 73326 dests 00 #72 id 439 exid 73326 dests 00 #73 id 439 exid 19919 dests 00 #74 id 439 exid 73326 dests 00 #75 id 439 exid 19919 dests 00 #76 id 439 exid 73326 dests 00 #77 id 439 exid 19919 dests 00 #78 id 439 exid 19919 dests 00 #79 id 439 exid 73326 dests 00 #80 id 439 exid 77713 dests 00 #81 id 439 exid 73326 dests 00 #82 id 439 exid 77713 dests 00 #83 id 439 exid 19919 dests 00 #84 id 439 exid 73326 dests 00 #85 id 439 exid 73326 dests 00 #86 id 439 exid 19919 dests 00 #87 id 439 exid 73326 dests 00 #88 id 439 exid 77713 dests 00 #89 id 439 exid 73326 dests 00 #90 id 439 exid 77713 dests 00 #91 id 439 exid 73326 dests 00 #92 id 439 exid 19919 dests 00 #93 id 439 exid 19919 dests 00 #94 id 439 exid 73326 dests 00 #95 id 439 exid 73326 dests 00 #96 id 439 exid 19919 dests 00 #97 id 439 exid 73326 dests 00 #98 id 439 exid 77713 dests 00 #99 id 439 exid 73326 dests 00 #100 id 439 exid 77713 dests 00 #101 id 439 exid 73326 dests 00 #102 id 439 exid 19919 dests 00 #103 id 439 exid 19919 dests 00 #104 id 439 exid 73326 dests 00 #105 id 439 exid 19919 dests 00 #106 id 439 exid 73326 dests 00 #107 id 439 exid 77713 dests 00 #108 id 439 exid 77713 dests 00 #109 id 439 exid 73326 dests 00 #110 id 439 exid 19919 dests 00 #111 id 439 exid 19919 dests 00 #112 id 439 exid 73326 dests 00 #113 id 439 exid 73326 dests 00 #114 id 439 exid 19919 dests 00 #115 id 439 exid 19919 dests 00 #116 id 439 exid 73326 dests 00 #117 id 439 exid 77713 dests 00 #118 id 439 exid 73326 dests 00 #119 id 439 exid 77713 dests 00 #120 id 439 exid 73326 dests 00 #121 id 439 exid 19919 dests 00 #122 id 439 exid 73326 dests 00 #123 id 439 exid 19919 dests 00 #124 id 439 exid 73326 dests 00 #125 id 439 exid 19919 dests 00 #126 id 439 exid 19919 dests 00 #127 id 439 exid 73326 dests 00 #128 id 439 exid 77713 dests 00 #129 id 439 exid 73326 dests 00 #130 id 439 exid 77713 dests 00 #131 id 439 exid 19919 dests 00 #132 id 439 exid 73326 dests 00 #133 id 439 exid 19919 dests 00 #134 id 439 exid 73326 dests 00 #135 id 439 exid 19919 dests 00 #136 id 439 exid 73326 dests 00 #137 id 439 exid 77713 dests 00 #138 id 439 exid 73326 dests 00 #139 id 439 exid 77713 dests 00 #140 id 439 exid 73326 dests 00 #141 id 439 exid 19919 dests 00 #142 id 439 exid 73326 dests 00 #143 id 439 exid 19919 dests 00 #144 id 439 exid 73326 dests 00 #145 id 439 exid 19919 dests 00 #146 id 439 exid 73326 dests 00 #147 id 439 exid 77713 dests 00 #148 id 439 exid 73326 dests 00 #149 id 439 exid 77713 dests 00 #150 id 439 exid 19919 dests 00 #151 id 439 exid 73326 dests 00 #152 id 439 exid 73326 dests 00 #153 id 439 exid 19919 dests 00 #154 id 439 exid 73326 dests 00 #155 id 439 exid 77713 dests 00 #156 id 439 exid 73326 dests 00 #157 id 439 exid 77713 dests 00 #158 id 439 exid 73326 dests 00 #159 id 439 exid 19919 dests 00 #160 id 439 exid 73326 dests 00 #161 id 439 exid 19919 dests 00 #162 id 439 exid 19919 dests 00 #163 id 439 exid 73326 dests 00 #164 id 439 exid 77713 dests 00 #165 id 439 exid 73326 dests 00 #166 id 439 exid 77713 dests 00 #167 id 439 exid 73326 dests 00 #168 id 439 exid 19919 dests 00 #169 id 439 exid 19919 dests 00 #170 id 439 exid 73326 dests 00 #171 id 439 exid 19919 dests 00 #172 id 439 exid 73326 dests 00 #173 id 439 exid 77713 dests 00 #174 id 439 exid 73326 dests 00 #175 id 439 exid 77713 dests 00 #176 id 439 exid 19919 dests 00 #177 id 439 exid 73326 dests 00 #178 id 439 exid 19919 dests 00 #179 id 439 exid 19919 dests 00 #180 id 439 exid 73326 dests 00 #181 id 439 exid 19919 dests 00 #182 id 439 exid 73326 dests 00 #183 id 439 exid 77713 dests 00 #184 id 439 exid 73326 dests 00 #185 id 439 exid 77713 dests 00 #186 id 439 exid 73326 dests 00 #187 id 439 exid 19919 dests 00 #188 id 439 exid 73326 dests 00 #189 id 439 exid 19919 dests 00 #190 id 439 exid 73326 dests 00 #191 id 439 exid 19919 dests 00 #192 id 439 exid 73326 dests 00 #193 id 439 exid 77713 dests 00 #194 id 439 exid 77713 dests 00 #195 id 439 exid 73326 dests 00 #196 id 439 exid 19919 dests 00 #197 id 439 exid 19919 dests 00 #198 id 439 exid 73326 dests 00 #199 id 439 exid 19919 dests 00 #200 id 439 exid 73326 dests 00 #201 id 439 exid 77713 dests 00 #202 id 439 exid 73326 dests 00 #203 id 439 exid 77713 dests 00 #204 id 439 exid 19919 dests 00 #205 id 439 exid 73326 dests 00 #206 id 439 exid 19919 dests 00 #207 id 439 exid 19919 dests 00 #208 id 439 exid 73326 dests 00 #209 id 439 exid 19919 dests 00 #210 id 439 exid 19919 dests 00 #211 id 439 exid 73326 dests 00 #212 id 439 exid 77713 dests 00 #213 id 439 exid 73326 dests 00 #214 id 439 exid 77713 dests 00 #215 id 439 exid 73326 dests 00 #216 id 439 exid 19919 dests 00 #217 id 439 exid 73326 dests 00 #218 id 439 exid 19919 dests 00 #219 id 439 exid 73326 dests 00 #220 id 439 exid 73326 dests 00 #221 id 439 exid 77713 dests 00 #222 id 439 exid 73326 dests 00 #223 id 439 exid 19919 dests 00 #224 id 439 exid 73326 dests 00 #225 id 439 exid 19919 dests 00 #226 id 439 exid 73326 dests 00 #227 id 439 exid 73326 dests 00 #228 id 439 exid 19919 dests 00 #229 id 439 exid 73326 dests 00 #230 id 439 exid 77713 dests 00 #231 id 439 exid 73326 dests 00 #232 id 439 exid 77713 dests 00 #233 id 439 exid 19919 dests 00 #234 id 439 exid 73326 dests 00 #235 id 439 exid 73326 dests 00 #236 id 439 exid 19919 dests 00 #237 id 439 exid 73326 dests 00 #238 id 439 exid 77713 dests 00 #239 id 439 exid 77713 dests 00 #240 id 439 exid 73326 dests 00 #241 id 439 exid 19919 dests 00 #242 id 439 exid 73326 dests 00 #243 id 439 exid 19919 dests 00 #244 id 439 exid 19919 dests 00 #245 id 439 exid 73326 dests 00 #246 id 439 exid 77713 dests 00 #247 id 439 exid 73326 dests 00 #248 id 439 exid 77713 dests 00 #249 id 439 exid 19919 dests 00 #250 id 439 exid 73326 dests 00 #251 id 439 exid 73326 dests 00 #252 id 439 exid 19919 dests 00 #253 id 439 exid 19919 dests 00 #254 id 439 exid 73326 dests 00 #255 id 439 exid 73326 dests 00 kjctseventdump-end tail 220 heads 218 @ 0 220 @ -817966651 End of KJC Communication Dump kjfmGCR_HBCheckAll: 1 process failed HBStatusCheck kjgcr_ServiceGCR: KJGCR_METRICS: Local metric heartbeat check, id 0 failed *** 2019-11-15T14:51:36.699115+04:00 (CDB$ROOT(1)) kjgcr_StatCheckCPU: Current load ratio is high (114711/1369) *** 2019-11-15T14:51:41.704742+04:00 (CDB$ROOT(1)) kjfmGCR_HBAnalyze: IPC0 has no heartbeat *** 2019-11-15T14:51:46.706154+04:00 (CDB$ROOT(1)) ============================== IPC0 (ospid: 368909) has not moved for 108 sec (1573815101.1573814993) 2019-11-15T14:51:46.746879+04:00 Incident 6943370 created, dump file: /u01/app/oracle/diag/rdbms/cdbiamas/cdbiamas1/incident/incdir_6943370/cdbiamas1_lmhb_368993_i6943370.trc ORA-29770: global enqueue process IPC0 (OSID 368909) is hung for more than 70 seconds ===[NO-HB PROCESS CALLSTACK]=== <Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 60 ms > ============================== ============================== Dumping PROCESS IPC0 (ospid: 368909) States ============================== ===[ System Load State ]=== CPU Total 96 Raw 96 Core 48 Socket 2 Load high: Cur 123801 Highmark 122880 (483.59 480.00) ===[ Latch State ]=== Not in Latch Get ===[ Session State Object ]=== ---------------------------------------- SO: 0x18e1753030, type: 4, owner: 0x18e0002c50, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x18e0002c50, name=session, file=ksu.h LINE:15737, pg=0, conuid=1 SGA version=(1,0) (session) sid: 727 ser: 53431 trans: (nil), creator: 0x18e0002c50 flags: (0x51) USR/- flags2: (0x409) -/-/INC flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/- DID: 0000-0000-000000000000-0000-00000000, short-term DID: txn branch: (nil) con_id/con_uid/con_name: 1/1/CDB$ROOT con_logonuid: 1 con_logonid: 1 con_scuid: 1 con_scid: 1 edition#: 0 user#/name: 0/SYS oct: 0, prv: 0, sql: (nil), psql: (nil) stats: 0x183fffc798, PX stats: 0x110cb824 service name: SYS$BACKGROUND Current Wait Stack: Not in wait; last wait ended 1 min 49 sec ago Wait State: fixed_waits=0 flags=0x21 boundary=(nil)/-1 Session Wait History: elapsed time of 1 min 49 sec since last wait 0: waited for 'rdbms ipc message' timeout=0x12c, =0x0, =0x0 wait_id=3106025 seq_num=35086 snap_id=1 wait times: snap=3.003118 sec, exc=3.003118 sec, total=3.003118 sec wait times: max=3.000000 sec wait counts: calls=1 os=1 occurred after 0.000011 sec of elapsed time 1: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106024 seq_num=35085 snap_id=1 wait times: snap=0.020098 sec, exc=0.020098 sec, total=0.020098 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050228 sec of elapsed time 2: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106023 seq_num=35084 snap_id=1 wait times: snap=0.020110 sec, exc=0.020110 sec, total=0.020110 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000015 sec of elapsed time 3: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106022 seq_num=35083 snap_id=1 wait times: snap=0.020180 sec, exc=0.020180 sec, total=0.020180 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.049185 sec of elapsed time 4: waited for 'rdbms ipc message' timeout=0x69, =0x0, =0x0 wait_id=3106021 seq_num=35082 snap_id=1 wait times: snap=1.051154 sec, exc=1.051154 sec, total=1.051154 sec wait times: max=1.050000 sec wait counts: calls=1 os=1 occurred after 0.000011 sec of elapsed time 5: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106020 seq_num=35081 snap_id=1 wait times: snap=0.020146 sec, exc=0.020146 sec, total=0.020146 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050272 sec of elapsed time 6: waited for 'rdbms ipc message' timeout=0x12c, =0x0, =0x0 wait_id=3106019 seq_num=35080 snap_id=1 wait times: snap=1.878880 sec, exc=1.878880 sec, total=1.878880 sec wait times: max=3.000000 sec wait counts: calls=1 os=1 occurred after 0.000011 sec of elapsed time 7: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106018 seq_num=35079 snap_id=1 wait times: snap=0.020171 sec, exc=0.020171 sec, total=0.020171 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050231 sec of elapsed time 8: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106017 seq_num=35078 snap_id=1 wait times: snap=0.020184 sec, exc=0.020184 sec, total=0.020184 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.000016 sec of elapsed time 9: waited for 'RMA: IPC0 completion sync' send count=0x0, =0x0, =0x23e wait_id=3106016 seq_num=35077 snap_id=1 wait times: snap=0.020206 sec, exc=0.020206 sec, total=0.020206 sec wait times: max=infinite wait counts: calls=1 os=1 occurred after 0.050239 sec of elapsed time Sampled Session History of session 727 serial 53431 --------------------------------------------------- The sampled session history is constructed by sampling the target session every 1 second. The sampling process captures at each sample if the session is in a non-idle wait, an idle wait, or not in a wait. If the session is in a non-idle wait then one interval is shown for all the samples the session was in the same non-idle wait. If the session is in an idle wait or not in a wait for consecutive samples then one interval is shown for all the consecutive samples. Though we display these consecutive samples in a single interval the session may NOT be continuously idle or not in a wait (the sampling process does not know). The history is displayed in reverse chronological order. sample interval: 1 sec, max history 120 sec --------------------------------------------------- [108 samples, 14:49:58 - 14:51:45] not in wait at each sample [9 samples, 14:49:49 - 14:49:57] idle wait at each sample [1 sample, 14:49:48] waited for 'RMA: IPC0 completion sync', seq_num: 35075 p1: 'send count'=0x0 p2: ''=0x0 p3: ''=0x23e time_waited: 0.020107 sec (sample interval: 0 sec) [3 samples, 14:49:46 - 14:49:47] idle wait at each sample --------------------------------------------------- Sampled Session History Summary: longest_non_idle_wait: 'RMA: IPC0 completion sync' [1 sample, 14:49:48 ] time_waited: 0.020107 sec (sample interval: 0 sec) --------------------------------------------------- ---------------------------------------- kpoxc: mode=0 static=0 inReq=0 pendErrInRep=0 callUseSCN=0 nonReq=0 kpoxc: trace=0 roundTrip=0 errCode=0 ctl=0 eocs=0 disErr=0 dscnEnabled=0 temporary object counter: 0 last pdbid to increment: 0, last function to increment: (null) BPS: empty ---------------------------------------- Virtual Thread: kgskvt: 0x976416cc20, sess: 0x18e1753030/1 sid: 727 ser: 53431 vc: (nil), proc: 0x18e0002c50, id: 727 consumer group cur: _ORACLE_BACKGROUND_GROUP_/1 (upd? 0), mapped: _ORACLE_BACKGROUND_GROUP_, orig: vt_state: 0x100, vt_flags: 0x30, blkrun: 0, numa: 2 inwait: 0, short wait event: 0 posted_run: 0 lastmodrngcnt: 0, lastmodrngcnt_loc: '(null)' lastmodrblcnt: 0, lastmodrblcnt_loc: '(null)' location where insched last set: kgskthrrun location where insched last cleared: kgskthrrun1 location where inwait last set: NULL location where inwait last cleared: NULL is_assigned: 0, in_sched: 0 (0) qcls: (nil), qlink: FALSE vt_active: 0 (pending: 0) vt_pq_active: 0, dop: 0, pq_servers (cur: 0 cg: 0) ps_allocs: 0, pxstmts (act: 0, done: 0 cg: 0) used quanta (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 exec start consumed time: 0 exec start elapsed time: 0 idle time: 0, active time: 0 (cg: 0) last updnumps: 0, active time (pq: 0 ps: 0) cpu yields: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu waits: stmt: 0, accum: 0, mapped: 0, tot: 0 cpu wait time (usecs): stmt: 0, accum: 0, mapped: 0, tot: 0 ASL queued time outs: 0, time: 0 (cur 0, cg 0) PQQ queued time outs: 0, time: 0 (cur 0, cg 0) Queue timeout violation: 0 calls aborted: 0, num est exec limit hit: 0 undo current: 0k max: 0k KTU Session Commit Cache Dump for IDLs: KTU Session Commit Cache Dump for Non-IDLs: ---------------------------------------- KKS-UOL used : 0 locks(used=0, free=0) KGX Atomic Operation Log 0x977f6f40a0 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x977f6f40f0 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x977f6f4140 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 KGX Atomic Operation Log 0x977f6f4190 Mutex (nil)(0, 0) idn 0 oper NONE(0) FSO mutex uid 727 efd 0 whr 0 slp 0 ---------------------------------------- KGL-UOL SO Cache(total=0, free=0) KGX Atomic Operation Log 0x975f73a2d8 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a340 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a3a8 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a410 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a478 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a4e0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a548 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a5b0 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 KGX Atomic Operation Log 0x975f73a618 Mutex (nil)(0, 0) idn 0 oper NONE(0) Library Cache uid 727 efd 0 whr 0 slp 0 oper=0 pt1=(nil) pt2=(nil) pt3=(nil) pt4=(nil) pt5=(nil) ub4=0 flg=0x0 msk=0000-0000-0000-0000-0000 ---------------------------------------- SO: 0x1812141ff8, type: 79, owner: 0x18e1753030, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3 proc=0x18e0002c50, name=dummy, file=ktccts.h LINE:456, pg=0, conuid=1 SGA version=(1,0) (dummy) nxc=0, nlb=0 ===[ Callstack ]=== *** 2019-11-15T14:51:46.868697+04:00 (CDB$ROOT(1)) Process diagnostic dump for oracle@exa-hddbadm01.core.iamas.gov.az (IPC0), OS id=368909, pid: 6, proc_ser: 1, sid: 727, sess_ser: 53431 ------------------------------------------------------------------------------- os thread scheduling delay history: (sampling every 1.000000 secs) 0.000000 secs at [ 14:51:46 ] NOTE: scheduling delay has not been sampled for 0.587919 secs 0.000000 secs from [ 14:51:42 - 14:51:47 ], 5 sec avg 0.000000 secs from [ 14:50:47 - 14:51:47 ], 1 min avg 0.000000 secs from [ 14:46:47 - 14:51:47 ], 5 min avg *** 2019-11-15T14:51:49.174275+04:00 (CDB$ROOT(1)) loadavg : 483.60 153.67 56.17 System user time: 0.06 sys time: 0.02 context switch: 54748 Memory (Avail / Total) = 6431.47M / 772256.64M Swap (Avail / Total) = 24000.20M / 24576.00M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 R oracle 368909 1 0 80 0 - 167043937 - Oct21 ? 00:08:54 ora_ipc0_cdbiamas1 Name: ora_ipc0_cdbiam State: R (running) Tgid: 368909 Ngid: 0 Pid: 368909 PPid: 1 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 128 Groups: 1001 1002 NStgid: 368909 NSpid: 368909 NSpgid: 368909 NSsid: 368909 VmPeak: 668175752 kB VmSize: 668175748 kB VmLck: 0 kB VmPin: 562562088 kB VmHWM: 7413828 kB VmRSS: 7413828 kB VmData: 8388 kB VmStk: 156 kB VmExe: 343516 kB VmLib: 32424 kB VmPTE: 14924 kB VmPMD: 2172 kB VmSwap: 0 kB Threads: 1 SigQ: 12/3088249 SigPnd: 0000000000000800 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000016400207 SigCgt: 00000003c9887cf8 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff Seccomp: 0 Cpus_allowed: ffff,ffffffff,ffffffff,ffffffff Cpus_allowed_list: 0-111 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 4589423 nonvoluntary_ctxt_switches: 14717 *** 2019-11-15T14:51:51.870518+04:00 (CDB$ROOT(1)) Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 2695 ms ------------------------------------------------------------------------------- Process diagnostic dump actual duration=5.002000 sec (max dump time=5.000000 sec) *** 2019-11-15T14:51:51.870615+04:00 (CDB$ROOT(1)) ===[ Heartbeat State Object ]=== (lm heartbeat) state=0x38000001 active noheartbeat current heartbeat: time 0x5dce82d1 count 0x3403e where 0x1 last poke: time 0x5dce82db req 0x3403b ack 0x3403a freq 10 last wait: state 0x2 id 3106025 last record: time 0x5dce82dd count 0x3403e last check: time 0x5dce833d misc: good dur 70 default 70 ckreq 10 tmupdcnt 0 hangcnt 2 Dump of memory from 0x000000979F1E9608 to 0x000000979F1E9818 979F1E9600 9F1D2450 00000097 [P$......] 979F1E9610 E9D93E40 00000096 E9D93E60 00000096 [@>......`>......] 979F1E9620 E9D93E60 00000096 9F1E9628 00000097 [`>......(.......] 979F1E9630 9F1E9628 00000097 E0002C50 00000018 [(.......P,......] 979F1E9640 6000A5A0 00000000 00000001 30435049 [...`........IPC0] 979F1E9650 00000000 38000001 00000000 00000000 [.......8........] 979F1E9660 00000046 00000046 00000000 0003403E [F...F.......>@..] 979F1E9670 5DCE82D1 00000001 00000000 0000000A [...]............] 979F1E9680 5DCE82DB 0003403B 0003403A 0000000A [...];@..:@......] 979F1E9690 0000000A 00000002 5DCE833D 5DCE833D [........=..]=..]] 979F1E96A0 0003403E 5DCE82DD 5DCE8338 00000000 [>@.....]8..]....] 979F1E96B0 0000009A 00000000 00000000 00000000 [................] 979F1E96C0 00000000 00000000 00000000 00000006 [................] 979F1E96D0 00000000 00000000 00000000 00000000 [................] 979F1E96E0 00000002 FFFFFFFF 00000000 00000000 [................] 979F1E96F0 00000000 00000000 002F64E9 00000000 [.........d/.....] 979F1E9700 06322FDE 00000000 00000007 00000000 [./2.............] 979F1E9710 00000000 00000000 00000000 00000000 [................] Repeat 4 times 979F1E9760 74C300A8 00000097 00000004 00000000 [...t............] 979F1E9770 00000000 00000000 00000000 370000FA [...............7] 979F1E9780 00000005 00000961 5DCE833D 00000000 [....a...=..]....] 979F1E9790 0000000A 00000000 00000000 00000000 [................] 979F1E97A0 00000000 00000000 00000000 00000000 [................] Repeat 1 times 979F1E97C0 00000000 00000000 9F1E97C8 00000097 [................] 979F1E97D0 9F1E97C8 00000097 60048728 00000000 [........(..`....] 979F1E97E0 9F1D2620 00000097 00000000 00000000 [ &..............] 979F1E97F0 00000000 00000000 00000001 00000000 [................] 979F1E9800 00000000 00000000 00000000 00000000 [................] 979F1E9810 00000000 00000000 [........] ===[ Trace Buffer ]=== *** 2019-11-15T14:51:56.871636+04:00 (CDB$ROOT(1)) =====[PROCESS SNAPSHOTS]===== SNAPSHOT 0 Time: 2019-11-15 14:51:21.673721 +04:00 Stack: <Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms > Wait Stack: not in wait Wait History: 0 : 'rdbms ipc message' (waitid 3106025) snap time=3.003118 sec total time=3.003118 sec time since last wait=0.000011 sec 1 : 'RMA: IPC0 completion sync' (waitid 3106024) snap time=0.020098 sec total time=0.020098 sec time since last wait=0.050228 sec 2 : 'RMA: IPC0 completion sync' (waitid 3106023) snap time=0.020110 sec total time=0.020110 sec time since last wait=0.000015 sec 3 : 'RMA: IPC0 completion sync' (waitid 3106022) snap time=0.020180 sec total time=0.020180 sec time since last wait=0.049185 sec 4 : 'rdbms ipc message' (waitid 3106021) snap time=1.051154 sec total time=1.051154 sec time since last wait=0.000011 sec SNAPSHOT 1 Time: 2019-11-15 14:51:11.211355 +04:00 Stack: <Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms > Wait Stack: not in wait Wait History: 0 : 'rdbms ipc message' (waitid 3106025) snap time=3.003118 sec total time=3.003118 sec time since last wait=0.000011 sec 1 : 'RMA: IPC0 completion sync' (waitid 3106024) snap time=0.020098 sec total time=0.020098 sec time since last wait=0.050228 sec 2 : 'RMA: IPC0 completion sync' (waitid 3106023) snap time=0.020110 sec total time=0.020110 sec time since last wait=0.000015 sec 3 : 'RMA: IPC0 completion sync' (waitid 3106022) snap time=0.020180 sec total time=0.020180 sec time since last wait=0.049185 sec 4 : 'rdbms ipc message' (waitid 3106021) snap time=1.051154 sec total time=1.051154 sec time since last wait=0.000011 sec SNAPSHOT 2 Time: 2019-11-15 14:50:56.047266 +04:00 Stack: <Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms > Wait Stack: not in wait Wait History: 0 : 'rdbms ipc message' (waitid 3106025) snap time=3.003118 sec total time=3.003118 sec time since last wait=0.000011 sec 1 : 'RMA: IPC0 completion sync' (waitid 3106024) snap time=0.020098 sec total time=0.020098 sec time since last wait=0.050228 sec 2 : 'RMA: IPC0 completion sync' (waitid 3106023) snap time=0.020110 sec total time=0.020110 sec time since last wait=0.000015 sec 3 : 'RMA: IPC0 completion sync' (waitid 3106022) snap time=0.020180 sec total time=0.020180 sec time since last wait=0.049185 sec 4 : 'rdbms ipc message' (waitid 3106021) snap time=1.051154 sec total time=1.051154 sec time since last wait=0.000011 sec SNAPSHOT 3 Time: 2019-11-15 14:50:40.883990 +04:00 Stack: <Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms > Wait Stack: not in wait Wait History: 0 : 'rdbms ipc message' (waitid 3106025) snap time=3.003118 sec total time=3.003118 sec time since last wait=0.000011 sec 1 : 'RMA: IPC0 completion sync' (waitid 3106024) snap time=0.020098 sec total time=0.020098 sec time since last wait=0.050228 sec 2 : 'RMA: IPC0 completion sync' (waitid 3106023) snap time=0.020110 sec total time=0.020110 sec time since last wait=0.000015 sec 3 : 'RMA: IPC0 completion sync' (waitid 3106022) snap time=0.020180 sec total time=0.020180 sec time since last wait=0.049185 sec 4 : 'rdbms ipc message' (waitid 3106021) snap time=1.051154 sec total time=1.051154 sec time since last wait=0.000011 sec SNAPSHOT 4 Time: 2019-11-15 14:50:25.421175 +04:00 Stack: <Error encountered - ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 368909, image: oracle@exa-hddbadm01.core.iamas.gov.az (IPC0)'; prior command execution time exceeds 5000 ms > Wait Stack: not in wait Wait History: 0 : 'rdbms ipc message' (waitid 3106025) snap time=3.003118 sec total time=3.003118 sec time since last wait=0.000011 sec 1 : 'RMA: IPC0 completion sync' (waitid 3106024) snap time=0.020098 sec total time=0.020098 sec time since last wait=0.050228 sec 2 : 'RMA: IPC0 completion sync' (waitid 3106023) snap time=0.020110 sec total time=0.020110 sec time since last wait=0.000015 sec 3 : 'RMA: IPC0 completion sync' (waitid 3106022) snap time=0.020180 sec total time=0.020180 sec time since last wait=0.049185 sec 4 : 'rdbms ipc message' (waitid 3106021) snap time=1.051154 sec total time=1.051154 sec time since last wait=0.000011 sec ============================== ===[ DLM IPC ]=== KJC Communication Dump: state 0x35 flags 0x0 mode 0x0 inst 1 inc 28 inst=1 inc=28 nrcv=9 nsp=9 nrcvbuf=1000 reg_msg: msgsz=640 mpools=2 cur=795 max=12300 init=17390 mp[0]: cur=792 out=(sga:3602.3530.0 initq:720) max=12297 ini=8695 pavl=0 pmax=0 mp[1]: cur=3 out=(sga:0.0.0 initq:3) max=13 ini=8695 pavl=0 pmax=0 big_msg: msgsz=8504 mpools=2 cur=115 max=947 init=17390 mp[0]: cur=88 out=(sga:0.0.0 initq:88) max=923 ini=8695 pavl=0 pmax=0 mp[1]: cur=53 out=(sga:0.0.0 initq:53) max=132 ini=8695 pavl=0 pmax=0 rsv_msg: msgsz=8504 mpools=1 cur=0 max=1 init=1000 mp[0]: cur=0 out=(sga:0.0.0 initq:0) max=1 tot=1000 pavl=0 pmax=0 rcvr: id=0 (orapid=26 LMD0) grp=1 ospid=368961 rcvr: id=8 (orapid=34 LMD2) grp=1 ospid=368989 rcvr: id=1 (orapid=28 LMS0) grp=2 ospid=368963 rcvr: id=5 (orapid=32 LMS4) grp=2 ospid=368977 rcvr: id=6 (orapid=27 LMS5) grp=2 ospid=368981 rcvr: id=4 (orapid=25 LMS3) grp=2 ospid=368972 rcvr: id=2 (orapid=23 LMS1) grp=2 ospid=368965 rcvr: id=7 (orapid=29 LMD1) grp=1 ospid=368983 rcvr: id=3 (orapid=30 LMS2) grp=2 ospid=368969 send proxy: id=0 orapid=26 ndst=1 (2:0 ) pso=0x18e000d1f0.(nil) state=x0 send proxy: id=8 orapid=34 ndst=1 (2:8 ) pso=0x18e0011430.(nil) state=x0 send proxy: id=1 orapid=28 ndst=1 (2:1 ) pso=0x18e000e280.(nil) state=x0 send proxy: id=5 orapid=32 ndst=1 (2:5 ) pso=0x18e00103a0.(nil) state=x0 send proxy: id=6 orapid=27 ndst=1 (2:6 ) pso=0x566000d1f0.(nil) state=x0 send proxy: id=4 orapid=25 ndst=1 (2:4 ) pso=0x566000c160.(nil) state=x0 send proxy: id=2 orapid=23 ndst=1 (2:2 ) pso=0x566000b0d0.(nil) state=x0 send proxy: id=7 orapid=29 ndst=1 (2:7 ) pso=0x566000e280.(nil) state=x0 send proxy: id=3 orapid=30 ndst=1 (2:3 ) pso=0x18e000f310.(nil) state=x0 GES resource limits: ges resources: cur=242473 max=503402 ini=241332 ges enqueues: cur=124210 max=157460 ini=363099 ges cresources: cur=124812 max=132896 gcs resources: cur=21223354 max=25452045 ini=77122167 gcs shadows: cur=18628493 max=18812915 ini=77122167 KJCTS state: seq-check=no timeout=yes waitticks=0x3 highload=yes comm_interface=msgq-drm (rds) GES destination context: GES remote instance per receiver context: GES destination context: Dest=2 rcvr=0 inc=28 state=0x10041 tstate=0x0 batch-type=quick bmsg=(nil) tmout=0xd626f14 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=249 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 81) sndq_flushs=44665413 last_flush_time=0x89a63d5cb flush_idx=2 sndq_flush_stats= [2]v:44665413|w:114|f:0|l:1|t:1000|s:1|m:1 [1]v:44665412|w:43|f:0|l:1|t:999|s:1|m:1 [0]v:44665411|w:143|f:0|l:1|t:1000|s:1|m:1 [9]v:44665410|w:0|f:0|l:1|t:1000|s:1|m:1 [8]v:44665409|w:4|f:0|l:1|t:1000|s:1|m:1 [7]v:44665408|w:4|f:0|l:1|t:998|s:1|m:1 [6]v:44665407|w:4|f:0|l:1|t:1000|s:1|m:1 [5]v:44665406|w:28|f:0|l:1|t:999|s:1|m:1 [4]v:44665405|w:8|f:0|l:1|t:1000|s:1|m:1 [3]v:44665404|w:36|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=168982587 ack=0 snt=168982587 sync seq=0.168982587 inc=0 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.153767097,[PCM]0.0,[ALL]0.168982587 batch_snds=1683713 tot_msgs=3533474 max_sz=38 full_load=0 snd_seq=1683713 pbatch_snds=19271455 tot_msgs=19455245 sndq_msg_tot=46516774 tm=(0 5671) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 214 wqlen 29424158 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=1 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x13b93631 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 98) sndq_flushs=43453450 last_flush_time=0x89a63d5cb flush_idx=8 sndq_flush_stats= [8]v:43453450|w:105|f:0|l:1|t:1000|s:1|m:1 [7]v:43453449|w:7|f:0|l:1|t:997|s:1|m:1 [6]v:43453448|w:28|f:0|l:1|t:998|s:1|m:1 [5]v:43453447|w:7|f:0|l:1|t:1000|s:1|m:1 [4]v:43453445|w:1|f:0|l:1|t:999|s:0|m:0 [3]v:43453444|w:0|f:0|l:3|t:999|s:0|m:0 [2]v:43453443|w:8|f:0|l:8|t:999|s:0|m:0 [1]v:43453442|w:11|f:0|l:1|t:1000|s:1|m:1 [0]v:43453441|w:18|f:0|l:1|t:1000|s:1|m:1 [9]v:43453440|w:9|f:0|l:1|t:999|s:1|m:1 seq wrp=0 lst=132509668 ack=0 snt=132509668 sync seq=0.132509668 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85809507,[PCM]0.132509668,[ALL]0.132509668 batch_snds=1399916 tot_msgs=5033794 max_sz=96 full_load=642 snd_seq=1399916 pbatch_snds=21225985 tot_msgs=28383853 sndq_msg_tot=46912409 tm=(0 49774893) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41601174 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=2 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x109a9ca1 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 97) sndq_flushs=42915584 last_flush_time=0x89a63d581 flush_idx=0 sndq_flush_stats= [0]v:42915584|w:74|f:0|l:1|t:1000|s:1|m:1 [9]v:42915582|w:14|f:0|l:5|t:1000|s:0|m:0 [8]v:42915581|w:8|f:0|l:1|t:1000|s:1|m:1 [7]v:42915580|w:15|f:0|l:1|t:999|s:1|m:1 [6]v:42915579|w:12|f:0|l:1|t:1000|s:1|m:1 [5]v:42915578|w:5|f:0|l:1|t:1000|s:1|m:1 [4]v:42915577|w:1|f:0|l:1|t:998|s:1|m:1 [3]v:42915576|w:7|f:0|l:1|t:999|s:1|m:1 [2]v:42915575|w:57|f:0|l:1|t:1000|s:1|m:1 [1]v:42915574|w:15|f:0|l:1|t:994|s:1|m:1 seq wrp=0 lst=133270032 ack=0 snt=133270028 sync seq=0.133270028 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.89088728,[PCM]0.133270028,[ALL]0.133270028 batch_snds=1426711 tot_msgs=5103727 max_sz=77 full_load=569 snd_seq=1426711 pbatch_snds=20722666 tot_msgs=27863854 sndq_msg_tot=46412781 tm=(0 50946416) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 364 wqlen 41640990 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=3 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x11400698 msg_in_batch=0 tkt_total=1000 (min=1000) avl=739 sp_rsv=246 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 95) sndq_flushs=43540002 last_flush_time=0x89a63d5eb flush_idx=9 sndq_flush_stats= [9]v:43540002|w:30|f:0|l:1|t:988|s:1|m:1 [8]v:43540001|w:6|f:0|l:1|t:997|s:1|m:1 [7]v:43540000|w:6|f:0|l:1|t:999|s:1|m:1 [6]v:43539999|w:29|f:0|l:1|t:1000|s:1|m:1 [5]v:43539998|w:35|f:0|l:1|t:998|s:1|m:1 [4]v:43539997|w:0|f:0|l:1|t:999|s:1|m:1 [3]v:43539996|w:30|f:0|l:1|t:1000|s:1|m:1 [2]v:43539995|w:23|f:0|l:1|t:998|s:1|m:1 [1]v:43539994|w:9|f:0|l:1|t:999|s:1|m:1 [0]v:43539993|w:16|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=134354755 ack=0 snt=134354753 sync seq=0.134354753 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.86869831,[PCM]0.134354753,[ALL]0.134354753 batch_snds=1432721 tot_msgs=5118238 max_sz=91 full_load=605 snd_seq=1432721 pbatch_snds=21305828 tot_msgs=28448445 sndq_msg_tot=47044023 tm=(0 51175426) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 375 wqlen 41622836 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=4 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x11690c6e msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 103) sndq_flushs=43169094 last_flush_time=0x89a63d5eb flush_idx=2 sndq_flush_stats= [2]v:43169094|w:41|f:0|l:1|t:991|s:1|m:1 [1]v:43169093|w:64|f:0|l:1|t:993|s:1|m:1 [0]v:43169092|w:70|f:0|l:1|t:1000|s:1|m:1 [9]v:43169091|w:18|f:0|l:1|t:989|s:1|m:1 [8]v:43169090|w:8|f:0|l:1|t:995|s:1|m:1 [7]v:43169089|w:16|f:0|l:1|t:999|s:1|m:1 [6]v:43169088|w:8|f:0|l:1|t:1000|s:1|m:1 [5]v:43169087|w:1|f:0|l:1|t:998|s:1|m:1 [4]v:43169086|w:4|f:0|l:1|t:999|s:1|m:1 [3]v:43169085|w:5|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=143198820 ack=0 snt=143198818 sync seq=0.143198818 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.96323855,[PCM]0.143198818,[ALL]0.143198818 batch_snds=1427975 tot_msgs=5135084 max_sz=59 full_load=601 snd_seq=1427975 pbatch_snds=20915386 tot_msgs=28054583 sndq_msg_tot=46696674 tm=(0 50968735) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 346 wqlen 41912789 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=5 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x139c2141 msg_in_batch=0 tkt_total=1000 (min=1000) avl=748 sp_rsv=247 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 94) sndq_flushs=43574952 last_flush_time=0x89a63d5f0 flush_idx=1 sndq_flush_stats= [1]v:43574952|w:4|f:0|l:1|t:996|s:1|m:1 [0]v:43574951|w:32|f:0|l:1|t:997|s:1|m:1 [9]v:43574950|w:7|f:0|l:1|t:999|s:1|m:1 [8]v:43574949|w:16|f:0|l:1|t:990|s:1|m:1 [7]v:43574948|w:1|f:0|l:1|t:994|s:1|m:1 [6]v:43574947|w:33|f:0|l:1|t:995|s:1|m:1 [5]v:43574946|w:23|f:0|l:1|t:997|s:1|m:1 [4]v:43574945|w:113|f:0|l:1|t:1000|s:1|m:1 [3]v:43574944|w:1|f:0|l:1|t:998|s:1|m:1 [2]v:43574943|w:8|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=137471407 ack=0 snt=137471407 sync seq=0.137471407 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.93123683,[PCM]0.137471407,[ALL]0.137471407 batch_snds=1415057 tot_msgs=5079786 max_sz=74 full_load=587 snd_seq=1415057 pbatch_snds=21334732 tot_msgs=28460294 sndq_msg_tot=47061791 tm=(0 49473126) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 323 wqlen 41482478 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=6 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0x14b8e6de msg_in_batch=0 tkt_total=1000 (min=1000) avl=749 sp_rsv=246 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 96) sndq_flushs=43192233 last_flush_time=0x89a63d5e8 flush_idx=0 sndq_flush_stats= [0]v:43192233|w:30|f:0|l:1|t:996|s:1|m:1 [9]v:43192232|w:5|f:1|l:1|t:998|s:1|m:1 [8]v:43192231|w:2|f:0|l:1|t:999|s:1|m:1 [7]v:43192230|w:145|f:0|l:1|t:1000|s:1|m:1 [6]v:43192229|w:18|f:0|l:1|t:1000|s:1|m:1 [5]v:43192228|w:7|f:0|l:1|t:993|s:1|m:1 [4]v:43192227|w:33|f:0|l:1|t:994|s:1|m:1 [3]v:43192226|w:46|f:0|l:1|t:1000|s:1|m:1 [2]v:43192225|w:12|f:0|l:1|t:1000|s:1|m:1 [1]v:43192224|w:21|f:0|l:1|t:992|s:1|m:1 seq wrp=0 lst=129461745 ack=0 snt=129461745 sync seq=0.129461745 inc=28 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.85305884,[PCM]0.129461745,[ALL]0.129461745 batch_snds=1414226 tot_msgs=5124499 max_sz=67 full_load=652 snd_seq=1414226 pbatch_snds=20886062 tot_msgs=28060625 sndq_msg_tot=46720607 tm=(0 50089865) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 359 wqlen 42072275 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=7 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=(nil) tmout=0xdc9ab30 msg_in_batch=0 tkt_total=1000 (min=1000) avl=750 sp_rsv=249 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 54) sndq_flushs=46048808 last_flush_time=0x89a63d5ef flush_idx=8 sndq_flush_stats= [8]v:46048808|w:28|f:0|l:1|t:1000|s:1|m:1 [7]v:46048807|w:7|f:0|l:1|t:1000|s:1|m:1 [6]v:46048806|w:58|f:0|l:1|t:1000|s:1|m:1 [5]v:46048805|w:56|f:0|l:1|t:1000|s:1|m:1 [4]v:46048804|w:44|f:0|l:1|t:1000|s:1|m:1 [3]v:46048803|w:54|f:0|l:1|t:1000|s:1|m:1 [2]v:46048802|w:11|f:0|l:1|t:1000|s:1|m:1 [1]v:46048800|w:36|f:0|l:2|t:1000|s:0|m:0 [0]v:46048799|w:29|f:0|l:1|t:1000|s:1|m:1 [9]v:46048798|w:58|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=163063310 ack=0 snt=163063310 sync seq=0.163063310 inc=0 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.210,[PCM]0.0,[ALL]0.163063310 batch_snds=1938366 tot_msgs=3972972 max_sz=39 full_load=0 snd_seq=1938366 pbatch_snds=14297054 tot_msgs=14477178 sndq_msg_tot=48085319 tm=(0 5829) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 110 wqlen 35950230 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 Dest=2 rcvr=8 inc=28 state=0x50041 tstate=0x0 batch-type=auto bmsg=0x7ff182378028 tmout=0xcf12c1c msg_in_batch=2 tkt_total=1000 (min=1000) avl=750 sp_rsv=250 max_sp_rsv=250 tkt_check(chks=0 state=0 fnd=0 inj=0 detect=0) tkt_inc=0 tkt_dec=0 tkt_rel_time 0 nullreq_sends=0 nullack_rcvs=(0 55) sndq_flushs=38362705 last_flush_time=0x89a63d537 flush_idx=6 sndq_flush_stats= [6]v:38362705|w:38|f:0|l:2|t:1000|s:0|m:0 [5]v:38362704|w:42|f:0|l:1|t:1000|s:1|m:1 [4]v:38362703|w:0|f:0|l:1|t:998|s:1|m:1 [3]v:38362702|w:92|f:0|l:1|t:999|s:1|m:1 [2]v:38362701|w:41|f:0|l:1|t:998|s:1|m:1 [1]v:38362700|w:1|f:0|l:1|t:998|s:1|m:1 [0]v:38362699|w:32|f:0|l:1|t:999|s:1|m:1 [9]v:38362698|w:78|f:0|l:1|t:1000|s:1|m:1 [8]v:38362697|w:33|f:0|l:1|t:999|s:1|m:1 [7]v:38362696|w:200|f:0|l:1|t:1000|s:1|m:1 seq wrp=0 lst=161549421 ack=0 snt=161549416 sync seq=0.161549421 inc=0 sndq_seq#{[mtype]wrap.sequence}=[RCFG/RM]0.218,[PCM]0.0,[ALL]0.161549421 batch_snds=1587448 tot_msgs=3357433 max_sz=45 full_load=0 snd_seq=1587448 pbatch_snds=15617786 tot_msgs=15800589 sndq_msg_tot=40133677 tm=(0 4194) sndq msg 0 (s:0 p:0) maxlmt 77122167 maxlen 218 wqlen 26651993 sndq msg 0 start_tm 0 end_tm 0 PSends-rcvr/prxy/tkts 0/0/0 GES remote instance per receiver context: Rcvr=0 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.169439835 sync_rcv_seq=0.169439835 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1452902 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.166858040 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=8 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.160790094 sync_rcv_seq=0.160790090 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1208972 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.158381798 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=1 dest=2 inc=28 state=0x103 tstate=0x0 nak=4 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.141253146 sync_rcv_seq=0.141253146 sync_ack_seq=0.129056901 inc=28 batch_rcv_seq=1845673 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.134058573 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=5 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.141358734 sync_rcv_seq=0.141358734 sync_ack_seq=0.131227202 inc=28 batch_rcv_seq=1858892 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.135396667 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=6 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.150277858 sync_rcv_seq=0.150277856 sync_ack_seq=0.137776658 inc=28 batch_rcv_seq=1893521 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.142779468 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=4 dest=2 inc=28 state=0x103 tstate=0x0 nak=1 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.125330474 sync_rcv_seq=0.125330474 sync_ack_seq=0.114702593 inc=28 batch_rcv_seq=1863564 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.119202571 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=2 dest=2 inc=28 state=0x103 tstate=0x0 nak=11 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.124482338 sync_rcv_seq=0.124482338 sync_ack_seq=0.114194111 inc=28 batch_rcv_seq=1875249 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.118453984 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=7 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.165116380 sync_rcv_seq=0.165116380 sync_ack_seq=0.0 inc=0 batch_rcv_seq=1989097 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.163643671 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=3 dest=2 inc=28 state=0x103 tstate=0x0 nak=0 cached=0 ua_bufs=0 tkt_thr 500 rcv_seq=0.120370819 sync_rcv_seq=0.120370818 sync_ack_seq=0.110107175 inc=28 batch_rcv_seq=1849905 work_q=no nullreq_q=no chgtkt_q=no tkts[min=1000 cur=1000 max_add 1000] tkt_ack_seq=0.114270602 WMovAvg=0 inc=0 dec=0 wt_tickets_remote=0 alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: alloc_change_reqs: unchanged alloc_history: unchanged cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: cached_stkts: no returned tickets Rcvr=0 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:111) msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0 rmpool: alloc=1334 (r:1311 s:23) avail=134 (r:111 s:23) max=(s:23) hist_last_rcv_tm=0x89a63d5cb count=224554961 idx=4 [4]t:x89a63d44b|c:39|w:11|x:(33.6.0.0)57|m:29 [3]t:x89a635f15|c:2505|w:11|x:(2193.312.0.0)99|m:1696 [2]t:x89a62e9aa|c:2574|w:11|x:(2251.323.0.0)99|m:1817 [1]t:x89a62746e|c:19120|w:1|x:(18828.292.0.0)99|m:14268 [0]t:x89a61ff34|c:2824|w:10|x:(2526.298.0.0)99|m:2057 [9]t:x89a618a02|c:3096|w:9|x:(2785.311.0.0)99|m:2195 [8]t:x89a6114b8|c:3643|w:8|x:(3348.295.0.0)99|m:2806 [7]t:x89a609f72|c:3880|w:7|x:(3598.282.0.0)99|m:3044 [6]t:x89a602a2a|c:17602|w:1|x:(17318.284.0.0)99|m:15328 [5]t:x89a5fb4e7|c:3445|w:8|x:(3155.290.0.0)99|m:2819 ibatch: 2[1372539] 3[53116] 4[16271] 5[4951] 6[2397] 7[1341] ibatch: 8[909] 9[761] 10[450] 11[173] 12[87] 13[63] ibatch: 14[28] 15[15] 16[11] 17[6] ibatch: sum=1453118 rcvs=1453228 tot_msgs=3042837 max_sz=35 avg_sz=2 pbatch: 2[7672] 3[415] 4[136] 5[61] 6[57] 7[17] pbatch: 8[4] 9[3] 10[4] 11[1] 12[0] 13[4] pbatch: 14[1] 15[5] 16[4] 17[1] pbatch: sum=8385 rcvs=20424780 tot_msgs=20579618 max_sz=103 avg_sz=2 Rcvr=8 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:157) msgs_rdyq=0 watq=0 nsq=0 routq=1201 msgq=0 rmpool: alloc=1380 (r:1357 s:23) avail=179 (r:157 s:22) max=(s:23) hist_last_rcv_tm=0x89a63d59a count=217132059 idx=3 [3]t:x89a6364ac|c:2365|w:12|x:(2057.308.0.0)99|m:1568 [2]t:x89a62ef5b|c:2631|w:11|x:(2307.324.0.0)99|m:1803 [1]t:x89a627a0c|c:16992|w:1|x:(16699.293.0.0)99|m:12695 [0]t:x89a6204bb|c:2630|w:11|x:(2297.333.0.0)99|m:1843 [9]t:x89a618f38|c:3198|w:9|x:(2887.311.0.0)99|m:2092 [8]t:x89a611a02|c:2654|w:11|x:(2321.333.0.0)99|m:1778 [7]t:x89a60a49e|c:2621|w:11|x:(2306.315.0.0)99|m:1828 [6]t:x89a602f6d|c:15008|w:2|x:(14713.295.0.0)99|m:13063 [5]t:x89a5fba3d|c:1997|w:15|x:(1652.345.0.0)99|m:1560 [4]t:x89a5f44ed|c:2134|w:14|x:(1795.339.0.0)99|m:1621 ibatch: 2[1113843] 3[61797] 4[20539] 5[6821] 6[3212] 7[1341] ibatch: 8[743] 9[391] 10[198] 11[116] 12[59] 13[35] ibatch: 14[21] 15[15] 16[7] 17[9] ibatch: sum=1209147 rcvs=1209233 tot_msgs=2573532 max_sz=38 avg_sz=2 pbatch: 2[5158] 3[535] 4[201] 5[91] 6[37] 7[31] pbatch: 8[20] 9[19] 10[13] 11[3] 12[6] 13[10] pbatch: 14[5] 15[6] 16[2] 17[9] pbatch: sum=6146 rcvs=19070301 tot_msgs=19222388 max_sz=103 avg_sz=2 Rcvr=1 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:364) msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0 rmpool: alloc=1633 (r:1564 s:69) avail=429 (r:364 s:65) max=(s:69) hist_last_rcv_tm=0x89a63d5f6 count=330864285 idx=1 [1]t:x89a63d0ce|c:397|w:3|x:(397.0.0.0)30|m:178 [0]t:x89a635b9b|c:7102|w:4|x:(7102.0.0.0)30|m:2729 [9]t:x89a62e661|c:8054|w:3|x:(8054.0.0.0)30|m:3383 [8]t:x89a62712e|c:6558|w:4|x:(6558.0.0.0)30|m:2955 [7]t:x89a61fbfd|c:7561|w:3|x:(7561.0.0.0)30|m:3010 [6]t:x89a6186bf|c:5868|w:5|x:(5868.0.0.0)30|m:2357 [5]t:x89a61118a|c:5499|w:5|x:(5499.0.0.0)30|m:2345 [4]t:x89a609c55|c:6241|w:4|x:(6241.0.0.0)30|m:2705 [3]t:x89a60271c|c:7254|w:4|x:(7254.0.0.0)30|m:2809 [2]t:x89a5fb1e5|c:6797|w:4|x:(6797.0.0.0)30|m:3079 ibatch: 2[1220451] 3[251625] 4[102841] 5[55889] 6[38198] 7[30966] ibatch: 8[23999] 9[20125] 10[16840] 11[15151] 12[13532] 13[12454] ibatch: 14[11550] 15[10833] 16[9778] 17[3805] ibatch: sum=1838037 rcvs=1849712 tot_msgs=6174530 max_sz=56 avg_sz=3 pbatch: 2[352287] 3[202797] 4[123721] 5[93438] 6[93402] 7[104490] pbatch: 8[27372] 9[18633] 10[17937] 11[15917] 12[15737] 13[13953] pbatch: 14[13953] 15[12735] 16[14167] 17[92343] pbatch: sum=1212882 rcvs=20362418 tot_msgs=32519618 max_sz=74 avg_sz=5 Rcvr=5 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341) msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0 rmpool: alloc=1610 (r:1541 s:69) avail=406 (r:341 s:65) max=(s:69) hist_last_rcv_tm=0x89a63d5f0 count=328941849 idx=1 [1]t:x89a63c4ef|c:936|w:4|x:(936.0.0.0)30|m:424 [0]t:x89a634fbf|c:5214|w:5|x:(5214.0.0.0)30|m:2193 [9]t:x89a62da8c|c:5846|w:5|x:(5846.0.0.0)30|m:2312 [8]t:x89a626543|c:5009|w:5|x:(5009.0.0.0)30|m:2451 [7]t:x89a61f00d|c:4569|w:6|x:(4569.0.0.0)30|m:1761 [6]t:x89a617ad2|c:3664|w:8|x:(3664.0.0.0)30|m:1473 [5]t:x89a610592|c:4264|w:7|x:(4262.2.0.0)31|m:1713 [4]t:x89a609049|c:4347|w:6|x:(4347.0.0.0)30|m:1854 [3]t:x89a601b18|c:4616|w:6|x:(4616.0.0.0)30|m:1556 [2]t:x89a5fa5e7|c:4752|w:6|x:(4752.0.0.0)30|m:2229 ibatch: 2[1225519] 3[254849] 4[103673] 5[56410] 6[38778] 7[30966] ibatch: 8[24303] 9[20643] 10[17098] 11[15454] 12[13628] 13[12816] ibatch: 14[11669] 15[10919] 16[9954] 17[4018] ibatch: sum=1850697 rcvs=1862931 tot_msgs=6243714 max_sz=52 avg_sz=3 pbatch: 2[350367] 3[200988] 4[125539] 5[93312] 6[94203] 7[104713] pbatch: 8[28340] 9[18640] 10[18044] 11[15789] 12[16168] 13[13836] pbatch: 14[13789] 15[12668] 16[13887] 17[92690] pbatch: sum=1212973 rcvs=19829883 tot_msgs=31998870 max_sz=74 avg_sz=5 Rcvr=6 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318) msgs_rdyq=0 watq=0 nsq=0 routq=1202 msgq=0 rmpool: alloc=1564 (r:1518 s:46) avail=362 (r:318 s:44) max=(s:46) hist_last_rcv_tm=0x89a63d5e8 count=347615844 idx=1 [1]t:x89a63bb73|c:1545|w:4|x:(1545.0.0.0)30|m:659 [0]t:x89a63463c|c:6785|w:4|x:(6785.0.0.0)30|m:2705 [9]t:x89a62d10c|c:7888|w:3|x:(7888.0.0.0)30|m:2981 [8]t:x89a625bd0|c:8434|w:3|x:(8434.0.0.0)30|m:3542 [7]t:x89a61e697|c:6469|w:4|x:(6469.0.0.0)30|m:2630 [6]t:x89a617150|c:5699|w:5|x:(5698.1.0.0)31|m:2280 [5]t:x89a60fc19|c:6267|w:4|x:(6267.0.0.0)30|m:2647 [4]t:x89a6086e8|c:5200|w:5|x:(5200.0.0.0)30|m:2163 [3]t:x89a6011b7|c:6625|w:4|x:(6625.0.0.0)30|m:2467 [2]t:x89a5f9c7f|c:5407|w:5|x:(5406.1.0.0)31|m:2495 ibatch: 2[1251432] 3[259009] 4[106705] 5[56219] 6[39923] 7[29475] ibatch: 8[24835] 9[19768] 10[18089] 11[15360] 12[14221] 13[12597] ibatch: 14[12181] 15[11011] 16[10537] 17[4081] ibatch: sum=1885443 rcvs=1897777 tot_msgs=6349814 max_sz=54 avg_sz=3 pbatch: 2[357537] 3[210028] 4[123428] 5[95485] 6[91888] 7[107378] pbatch: 8[25641] 9[20011] 10[16849] 11[16988] 12[14833] 13[14752] pbatch: 14[13117] 15[13450] 16[13519] 17[93193] pbatch: sum=1228097 rcvs=19965547 tot_msgs=32153157 max_sz=74 avg_sz=5 Rcvr=4 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341) msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0 rmpool: alloc=1587 (r:1541 s:46) avail=383 (r:341 s:42) max=(s:46) hist_last_rcv_tm=0x89a63d5ef count=291802152 idx=0 [0]t:x89a638682|c:4669|w:4|x:(4669.0.0.0)30|m:2161 [9]t:x89a63114b|c:6950|w:4|x:(6950.0.0.0)30|m:3238 [8]t:x89a629c1a|c:10508|w:2|x:(10508.0.0.0)30|m:4547 [7]t:x89a6226e8|c:10742|w:2|x:(10742.0.0.0)30|m:3977 [6]t:x89a61b1ad|c:5043|w:5|x:(5043.0.0.0)30|m:1965 [5]t:x89a613c72|c:6451|w:4|x:(6451.0.0.0)30|m:2806 [4]t:x89a60c742|c:5526|w:5|x:(5526.0.0.0)30|m:2345 [3]t:x89a605211|c:7198|w:4|x:(7198.0.0.0)30|m:2769 [2]t:x89a5fdcd1|c:7269|w:4|x:(7269.0.0.0)30|m:3115 [1]t:x89a5f678c|c:5222|w:5|x:(5222.0.0.0)30|m:2593 ibatch: 2[1235086] 3[252832] 4[104503] 5[54131] 6[38981] 7[29149] ibatch: 8[24513] 9[19668] 10[17880] 11[15098] 12[13992] 13[12561] ibatch: 14[12222] 15[11097] 16[10306] 17[3819] ibatch: sum=1855838 rcvs=1868102 tot_msgs=6256714 max_sz=51 avg_sz=3 pbatch: 2[344222] 3[202533] 4[120299] 5[95235] 6[92709] 7[105848] pbatch: 8[26248] 9[19663] 10[17784] 11[17509] 12[14867] 13[14646] pbatch: 14[13139] 15[13514] 16[13571] 17[92942] pbatch: sum=1204729 rcvs=20478276 tot_msgs=32639998 max_sz=74 avg_sz=5 Rcvr=2 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:318) msgs_rdyq=0 watq=0 nsq=0 routq=1203 msgq=0 rmpool: alloc=1587 (r:1518 s:69) avail=384 (r:318 s:66) max=(s:69) hist_last_rcv_tm=0x89a63d5ea count=278284752 idx=0 [0]t:x89a638b1b|c:4257|w:4|x:(4257.0.0.0)30|m:1778 [9]t:x89a6315dd|c:6166|w:4|x:(6166.0.0.0)30|m:2788 [8]t:x89a62a09b|c:5382|w:5|x:(5382.0.0.0)30|m:2632 [7]t:x89a622b69|c:4661|w:6|x:(4661.0.0.0)30|m:2142 [6]t:x89a61b61e|c:3939|w:7|x:(3939.0.0.0)30|m:1740 [5]t:x89a6140e4|c:4523|w:6|x:(4521.2.0.0)31|m:1908 [4]t:x89a60cbaa|c:3731|w:8|x:(3731.0.0.0)30|m:1652 [3]t:x89a60567a|c:5259|w:5|x:(5259.0.0.0)30|m:1912 [2]t:x89a5fe148|c:4399|w:6|x:(4399.0.0.0)30|m:1996 [1]t:x89a5f6c02|c:3823|w:7|x:(3823.0.0.0)30|m:1899 ibatch: 2[1228115] 3[255833] 4[107024] 5[56905] 6[40932] 7[31133] ibatch: 8[25764] 9[21059] 10[18571] 11[15973] 12[14775] 13[12869] ibatch: 14[12242] 15[11229] 16[10393] 17[4093] ibatch: sum=1866910 rcvs=1879521 tot_msgs=6368352 max_sz=54 avg_sz=3 pbatch: 2[353488] 3[206297] 4[122256] 5[94495] 6[91116] 7[105765] pbatch: 8[25985] 9[19548] 10[16768] 11[17365] 12[14744] 13[14894] pbatch: 14[13731] 15[13582] 16[13514] 17[92954] pbatch: sum=1216502 rcvs=19822972 tot_msgs=31994115 max_sz=74 avg_sz=5 Rcvr=7 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:180) msgs_rdyq=0 watq=0 nsq=0 routq=1200 msgq=0 rmpool: alloc=1403 (r:1380 s:23) avail=203 (r:180 s:23) max=(s:23) hist_last_rcv_tm=0x89a63d5ef count=231320391 idx=3 [3]t:x89a63bef7|c:640|w:9|x:(585.55.0.0)98|m:388 [2]t:x89a6349c7|c:2431|w:12|x:(2082.349.0.0)99|m:1514 [1]t:x89a62d45b|c:2683|w:11|x:(2377.306.0.0)99|m:1711 [0]t:x89a625f09|c:20508|w:1|x:(20227.281.0.0)99|m:14946 [9]t:x89a61e9d2|c:2751|w:10|x:(2419.332.0.0)99|m:1722 [8]t:x89a61745d|c:2983|w:10|x:(2667.316.0.0)99|m:1876 [7]t:x89a60ff24|c:3786|w:7|x:(3473.313.0.0)99|m:2292 [6]t:x89a6089eb|c:5113|w:5|x:(4776.337.0.0)99|m:3577 [5]t:x89a6014a3|c:16750|w:1|x:(16441.309.0.0)99|m:14276 [4]t:x89a5f9f6b|c:3434|w:8|x:(3095.339.0.0)99|m:2341 ibatch: 2[1870533] 3[105709] 4[10127] 5[1884] 6[732] 7[414] ibatch: 8[170] 9[81] 10[69] 11[30] 12[19] 13[26] ibatch: 14[4] 15[0] 16[1] 17[1] ibatch: sum=1989800 rcvs=1989864 tot_msgs=4119303 max_sz=23 avg_sz=2 pbatch: 2[6307] 3[414] 4[92] 5[29] 6[15] 7[5] pbatch: 8[4] 9[3] 10[5] 11[5] 12[3] 13[8] pbatch: 14[4] 15[4] 16[6] 17[8] pbatch: sum=6912 rcvs=16772328 tot_msgs=16925186 max_sz=103 avg_sz=2 Rcvr=3 state=0x12 rbufs_tot=1200 q_rcvs=1200 ubufs=0 free=(r:341) msgs_rdyq=0 watq=0 nsq=0 routq=1204 msgq=0 rmpool: alloc=1610 (r:1541 s:69) avail=406 (r:341 s:65) max=(s:69) hist_last_rcv_tm=0x89a63d5eb count=289402308 idx=0 [0]t:x89a63942a|c:2649|w:6|x:(2649.0.0.0)30|m:1088 [9]t:x89a631ef8|c:5892|w:5|x:(5892.0.0.0)30|m:2360 [8]t:x89a62a9be|c:5590|w:5|x:(5590.0.0.0)30|m:2162 [7]t:x89a623484|c:5274|w:5|x:(5274.0.0.0)30|m:2524 [6]t:x89a61bf46|c:5719|w:5|x:(5719.0.0.0)30|m:2388 [5]t:x89a614a01|c:4346|w:6|x:(4346.0.0.0)30|m:1769 [4]t:x89a60d4c5|c:5610|w:5|x:(5610.0.0.0)30|m:2220 [3]t:x89a605f8d|c:4747|w:6|x:(4747.0.0.0)30|m:1892 [2]t:x89a5fea57|c:4861|w:6|x:(4861.0.0.0)30|m:1844 [1]t:x89a5f7526|c:4966|w:6|x:(4966.0.0.0)30|m:2315 ibatch: 2[1221328] 3[251740] 4[102912] 5[56760] 6[39176] 7[30881] ibatch: 8[24076] 9[20703] 10[16863] 11[15387] 12[13402] 13[12711] ibatch: 14[11581] 15[10830] 16[9697] 17[3984] ibatch: sum=1842031 rcvs=1855319 tot_msgs=6260211 max_sz=48 avg_sz=3 pbatch: 2[360085] 3[204157] 4[124611] 5[92957] 6[94169] 7[105197] pbatch: 8[28209] 9[17678] 10[18665] 11[15940] 12[15763] 13[13774] pbatch: 14[13801] 15[12728] 16[13885] 17[93027] pbatch: sum=1224646 rcvs=20361103 tot_msgs=32542739 max_sz=74 avg_sz=5 GES PGA send message buffer usage: orapid=23 ospid=368965 pmp: alloc=69 avail=66 max=69 orapid=25 ospid=368972 pmp: alloc=46 avail=42 max=46 orapid=26 ospid=368961 pmp: alloc=23 avail=23 max=23 orapid=27 ospid=368981 pmp: alloc=46 avail=44 max=46 orapid=28 ospid=368963 pmp: alloc=69 avail=65 max=69 orapid=29 ospid=368983 pmp: alloc=23 avail=23 max=23 orapid=30 ospid=368969 pmp: alloc=69 avail=65 max=69 orapid=32 ospid=368977 pmp: alloc=69 avail=65 max=69 orapid=34 ospid=368989 pmp: alloc=23 avail=22 max=23 orapid=71 ospid=369094 pmp: alloc=0 avail=0 max=23 orapid=74 ospid=369090 pmp: alloc=0 avail=0 max=23 orapid=75 ospid=369121 pmp: alloc=0 avail=0 max=23 orapid=78 ospid=369098 pmp: alloc=0 avail=0 max=23 orapid=79 ospid=369123 pmp: alloc=0 avail=0 max=23 orapid=82 ospid=369108 pmp: alloc=0 avail=0 max=23 kjctseventdump-begin tail 231 heads 218 @ 0 231 @ -817936614 #0 id 439 exid 77713 dests 00 #1 id 439 exid 73326 dests 00 #2 id 439 exid 77713 dests 00 #3 id 439 exid 73326 dests 00 #4 id 439 exid 19919 dests 00 #5 id 439 exid 73326 dests 00 #6 id 439 exid 19919 dests 00 #7 id 439 exid 73326 dests 00 #8 id 439 exid 77713 dests 00 #9 id 439 exid 19919 dests 00 #10 id 439 exid 73326 dests 00 #11 id 439 exid 19919 dests 00 #12 id 439 exid 73326 dests 00 #13 id 439 exid 77713 dests 00 #14 id 439 exid 73326 dests 00 #15 id 439 exid 73326 dests 00 #16 id 439 exid 77713 dests 00 #17 id 439 exid 73326 dests 00 #18 id 439 exid 19919 dests 00 #19 id 439 exid 73326 dests 00 #20 id 439 exid 19919 dests 00 #21 id 439 exid 73326 dests 00 #22 id 439 exid 19919 dests 00 #23 id 439 exid 73326 dests 00 #24 id 439 exid 77713 dests 00 #25 id 439 exid 77713 dests 00 #26 id 439 exid 73326 dests 00 #27 id 439 exid 19919 dests 00 #28 id 439 exid 73326 dests 00 #29 id 439 exid 73326 dests 00 #30 id 439 exid 73326 dests 00 #31 id 439 exid 19919 dests 00 #32 id 439 exid 19919 dests 00 #33 id 439 exid 73326 dests 00 #34 id 439 exid 77713 dests 00 #35 id 439 exid 73326 dests 00 #36 id 439 exid 73326 dests 00 #37 id 439 exid 77713 dests 00 #38 id 439 exid 73326 dests 00 #39 id 439 exid 19919 dests 00 #40 id 439 exid 73326 dests 00 #41 id 439 exid 19919 dests 00 #42 id 439 exid 73326 dests 00 #43 id 439 exid 73326 dests 00 #44 id 439 exid 19919 dests 00 #45 id 439 exid 73326 dests 00 #46 id 439 exid 77713 dests 00 #47 id 439 exid 73326 dests 00 #48 id 439 exid 77713 dests 00 #49 id 439 exid 19919 dests 00 #50 id 439 exid 19919 dests 00 #51 id 439 exid 73326 dests 00 #52 id 439 exid 19919 dests 00 #53 id 439 exid 73326 dests 00 #54 id 439 exid 19919 dests 00 #55 id 439 exid 19919 dests 00 #56 id 439 exid 77713 dests 00 #57 id 439 exid 73326 dests 00 #58 id 439 exid 73326 dests 00 #59 id 439 exid 73326 dests 00 #60 id 439 exid 77713 dests 00 #61 id 439 exid 77713 dests 00 #62 id 439 exid 73326 dests 00 #63 id 439 exid 19919 dests 00 #64 id 439 exid 73326 dests 00 #65 id 439 exid 19919 dests 00 #66 id 439 exid 73326 dests 00 #67 id 439 exid 19919 dests 00 #68 id 439 exid 73326 dests 00 #69 id 439 exid 77713 dests 00 #70 id 439 exid 77713 dests 00 #71 id 439 exid 73326 dests 00 #72 id 439 exid 73326 dests 00 #73 id 439 exid 19919 dests 00 #74 id 439 exid 73326 dests 00 #75 id 439 exid 19919 dests 00 #76 id 439 exid 73326 dests 00 #77 id 439 exid 19919 dests 00 #78 id 439 exid 19919 dests 00 #79 id 439 exid 73326 dests 00 #80 id 439 exid 77713 dests 00 #81 id 439 exid 73326 dests 00 #82 id 439 exid 77713 dests 00 #83 id 439 exid 19919 dests 00 #84 id 439 exid 73326 dests 00 #85 id 439 exid 73326 dests 00 #86 id 439 exid 19919 dests 00 #87 id 439 exid 73326 dests 00 #88 id 439 exid 77713 dests 00 #89 id 439 exid 73326 dests 00 #90 id 439 exid 77713 dests 00 #91 id 439 exid 73326 dests 00 #92 id 439 exid 19919 dests 00 #93 id 439 exid 19919 dests 00 #94 id 439 exid 73326 dests 00 #95 id 439 exid 73326 dests 00 #96 id 439 exid 19919 dests 00 #97 id 439 exid 73326 dests 00 #98 id 439 exid 77713 dests 00 #99 id 439 exid 73326 dests 00 #100 id 439 exid 77713 dests 00 #101 id 439 exid 73326 dests 00 #102 id 439 exid 19919 dests 00 #103 id 439 exid 19919 dests 00 #104 id 439 exid 73326 dests 00 #105 id 439 exid 19919 dests 00 #106 id 439 exid 73326 dests 00 #107 id 439 exid 77713 dests 00 #108 id 439 exid 77713 dests 00 #109 id 439 exid 73326 dests 00 #110 id 439 exid 19919 dests 00 #111 id 439 exid 19919 dests 00 #112 id 439 exid 73326 dests 00 #113 id 439 exid 73326 dests 00 #114 id 439 exid 19919 dests 00 #115 id 439 exid 19919 dests 00 #116 id 439 exid 73326 dests 00 #117 id 439 exid 77713 dests 00 #118 id 439 exid 73326 dests 00 #119 id 439 exid 77713 dests 00 #120 id 439 exid 73326 dests 00 #121 id 439 exid 19919 dests 00 #122 id 439 exid 73326 dests 00 #123 id 439 exid 19919 dests 00 #124 id 439 exid 73326 dests 00 #125 id 439 exid 19919 dests 00 #126 id 439 exid 19919 dests 00 #127 id 439 exid 73326 dests 00 #128 id 439 exid 77713 dests 00 #129 id 439 exid 73326 dests 00 #130 id 439 exid 77713 dests 00 #131 id 439 exid 19919 dests 00 #132 id 439 exid 73326 dests 00 #133 id 439 exid 19919 dests 00 #134 id 439 exid 73326 dests 00 #135 id 439 exid 19919 dests 00 #136 id 439 exid 73326 dests 00 #137 id 439 exid 77713 dests 00 #138 id 439 exid 73326 dests 00 #139 id 439 exid 77713 dests 00 #140 id 439 exid 73326 dests 00 #141 id 439 exid 19919 dests 00 #142 id 439 exid 73326 dests 00 #143 id 439 exid 19919 dests 00 #144 id 439 exid 73326 dests 00 #145 id 439 exid 19919 dests 00 #146 id 439 exid 73326 dests 00 #147 id 439 exid 77713 dests 00 #148 id 439 exid 73326 dests 00 #149 id 439 exid 77713 dests 00 #150 id 439 exid 19919 dests 00 #151 id 439 exid 73326 dests 00 #152 id 439 exid 73326 dests 00 #153 id 439 exid 19919 dests 00 #154 id 439 exid 73326 dests 00 #155 id 439 exid 77713 dests 00 #156 id 439 exid 73326 dests 00 #157 id 439 exid 77713 dests 00 #158 id 439 exid 73326 dests 00 #159 id 439 exid 19919 dests 00 #160 id 439 exid 73326 dests 00 #161 id 439 exid 19919 dests 00 #162 id 439 exid 19919 dests 00 #163 id 439 exid 73326 dests 00 #164 id 439 exid 77713 dests 00 #165 id 439 exid 73326 dests 00 #166 id 439 exid 77713 dests 00 #167 id 439 exid 73326 dests 00 #168 id 439 exid 19919 dests 00 #169 id 439 exid 19919 dests 00 #170 id 439 exid 73326 dests 00 #171 id 439 exid 19919 dests 00 #172 id 439 exid 73326 dests 00 #173 id 439 exid 77713 dests 00 #174 id 439 exid 73326 dests 00 #175 id 439 exid 77713 dests 00 #176 id 439 exid 19919 dests 00 #177 id 439 exid 73326 dests 00 #178 id 439 exid 19919 dests 00 #179 id 439 exid 19919 dests 00 #180 id 439 exid 73326 dests 00 #181 id 439 exid 19919 dests 00 #182 id 439 exid 73326 dests 00 #183 id 439 exid 77713 dests 00 #184 id 439 exid 73326 dests 00 #185 id 439 exid 77713 dests 00 #186 id 439 exid 73326 dests 00 #187 id 439 exid 19919 dests 00 #188 id 439 exid 73326 dests 00 #189 id 439 exid 19919 dests 00 #190 id 439 exid 73326 dests 00 #191 id 439 exid 19919 dests 00 #192 id 439 exid 73326 dests 00 #193 id 439 exid 77713 dests 00 #194 id 439 exid 77713 dests 00 #195 id 439 exid 73326 dests 00 #196 id 439 exid 19919 dests 00 #197 id 439 exid 19919 dests 00 #198 id 439 exid 73326 dests 00 #199 id 439 exid 19919 dests 00 #200 id 439 exid 73326 dests 00 #201 id 439 exid 77713 dests 00 #202 id 439 exid 73326 dests 00 #203 id 439 exid 77713 dests 00 #204 id 439 exid 19919 dests 00 #205 id 439 exid 73326 dests 00 #206 id 439 exid 19919 dests 00 #207 id 439 exid 19919 dests 00 #208 id 439 exid 73326 dests 00 #209 id 439 exid 19919 dests 00 #210 id 439 exid 19919 dests 00 #211 id 439 exid 73326 dests 00 #212 id 439 exid 77713 dests 00 #213 id 439 exid 73326 dests 00 #214 id 439 exid 77713 dests 00 #215 id 439 exid 73326 dests 00 #216 id 439 exid 19919 dests 00 #217 id 439 exid 73326 dests 00 #218 id 439 exid 19919 dests 00 #219 id 439 exid 73326 dests 00 #220 id 439 exid 19919 dests 00 #221 id 439 exid 19919 dests 00 #222 id 439 exid 73326 dests 00 #223 id 439 exid 77713 dests 00 #224 id 439 exid 77713 dests 00 #225 id 439 exid 77713 dests 00 #226 id 439 exid 73326 dests 00 #227 id 439 exid 19919 dests 00 #228 id 439 exid 73326 dests 00 #229 id 439 exid 19919 dests 00 #230 id 439 exid 73326 dests 00 #231 id 439 exid 73326 dests 00 #232 id 439 exid 77713 dests 00 #233 id 439 exid 19919 dests 00 #234 id 439 exid 73326 dests 00 #235 id 439 exid 73326 dests 00 #236 id 439 exid 19919 dests 00 #237 id 439 exid 73326 dests 00 #238 id 439 exid 77713 dests 00 #239 id 439 exid 77713 dests 00 #240 id 439 exid 73326 dests 00 #241 id 439 exid 19919 dests 00 #242 id 439 exid 73326 dests 00 #243 id 439 exid 19919 dests 00 #244 id 439 exid 19919 dests 00 #245 id 439 exid 73326 dests 00 #246 id 439 exid 77713 dests 00 #247 id 439 exid 73326 dests 00 #248 id 439 exid 77713 dests 00 #249 id 439 exid 19919 dests 00 #250 id 439 exid 73326 dests 00 #251 id 439 exid 73326 dests 00 #252 id 439 exid 19919 dests 00 #253 id 439 exid 19919 dests 00 #254 id 439 exid 73326 dests 00 #255 id 439 exid 73326 dests 00 kjctseventdump-end tail 231 heads 218 @ 0 231 @ -817936614 End of KJC Communication Dump kjfmGCR_HBdisambig: action=Inst-kill *** 2019-11-15T14:51:56.885323+04:00 (CDB$ROOT(1)) kjgcr_DoPreAction: KJGCR_PREACTION? - id 1 *** 2019-11-15T14:51:57.034864+04:00 (CDB$ROOT(1)) GCR_SYS group GR+DB_CDBIAMAS LOCKED by memno 1 *** 2019-11-15T14:51:57.035188+04:00 (CDB$ROOT(1)) kjgcr_querycssginf: grp "GR+N_1" has no private grp data. *** 2019-11-15T14:51:57.035217+04:00 (CDB$ROOT(1)) kjgcr_pollcssg: grp "GR+DB_CDBIAMAS" mbr 1 LOCKED next state (3) grp1=0x0 grp2=0x7ffe97383098 act=1. *** 2019-11-15T14:51:57.035234+04:00 (CDB$ROOT(1)) LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_CDBIAMAS by memno 1 *** 2019-11-15T14:51:57.338024+04:00 (CDB$ROOT(1)) kjgcr_pollcssg: grp "GR+DB_+ASM" mbr 2 LOCKED next state (3) grp1=0x7ffe97382ff0 grp2=0x0 act=1. *** 2019-11-15T14:51:57.339033+04:00 (CDB$ROOT(1)) kjgcr_DoAction: KJGCR_ACTION? - id 1 *** 2019-11-15T14:51:57.339066+04:00 (CDB$ROOT(1)) kjgcr_RunCDCallback: Running action callback [kjgcr Dump OS Diagnostics] *** 2019-11-15T14:52:02.340436+04:00 (CDB$ROOT(1)) ============================== IPC0 (ospid: 368909) has not moved for 123 sec (1573815116.1573814993) *** 2019-11-15T14:52:02.351421+04:00 (CDB$ROOT(1)) Forcibly terminated 'oracle@exa-hddbadm01.core.iamas.gov.az (LGWR)' process Forcibly terminated 4 out of 4 LGWR worker processes *** 2019-11-15T14:52:02.351508+04:00 (CDB$ROOT(1)) LGWR LWN SCN: 0x000000004f84c33e, Issued SCN: 0x000000004f84c33e, On-disk SCN: 0x000000004f84c33eKSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS0)' KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS1)' KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS2)' KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS3)' KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS4)' KSZ: Terminated fatal process 'oracle@exa-hddbadm01.core.iamas.gov.az (LMS5)' 2019-11-15 14:52:02.351 :kjzduptcctx(): Notifying DIAG for crash event ----- Abridged Call Stack Trace ----- ksedsts()+346<-kjzduptcctx()+868<-kjzdicrshnfy()+1113<-ksuitm_opt()+1678<-kjgcr_KillInstance()+170<-kjgcr_RunCallback()+1242<-kjgcr_DoAction()+186<-kjgcr_servicecssg()+856<-kjgcr_Main()+418<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202 <-ssthrdmain()+417<-main()+262<-__libc_start_main()+253 ----- End of Abridged Call Stack Trace ----- *** 2019-11-15T14:52:02.375872+04:00 (CDB$ROOT(1)) LMHB (ospid: 368993): terminating the instance due to error 29770 ksuitm: waiting up to [5] seconds before killing DIAG(368925) *** 2019-11-15T14:52:16.532424+04:00 (CDB$ROOT(1)) Instance termination failed to kill one or more processes ksuitm_check: OS PID=360990 is still alive ksuitm_check: OS PID=360970 is still alive ksuitm_check: OS PID=360880 is still alive ksuitm_check: OS PID=360772 is still alive ksuitm_check: OS PID=360757 is still alive ksuitm_check: OS PID=360755 is still alive ksuitm_check: OS PID=360753 is still alive ksuitm_check: OS PID=360749 is still alive ksuitm_check: OS PID=360746 is still alive ksuitm_check: OS PID=360744 is still alive ksuitm_check: OS PID=360734 is still alive ksuitm_check: OS PID=360729 is still alive ksuitm_check: OS PID=360724 is still alive ksuitm_check: OS PID=360722 is still alive ksuitm_check: OS PID=360714 is still alive ksuitm_check: OS PID=360710 is still alive ksuitm_check: OS PID=360706 is still alive ksuitm_check: OS PID=360694 is still alive ksuitm_check: OS PID=360691 is still alive ksuitm_check: OS PID=360689 is still alive ksuitm_check: OS PID=360687 is still alive ksuitm_check: OS PID=360653 is still alive ksuitm_check: OS PID=360651 is still alive ksuitm_check: OS PID=360649 is still alive ksuitm_check: OS PID=360645 is still alive ksuitm_check: OS PID=360642 is still alive ksuitm_check: OS PID=360640 is still alive ksuitm_check: OS PID=360635 is still alive ksuitm_check: OS PID=360631 is still alive ksuitm_check: OS PID=360622 is still alive ksuitm_check: OS PID=360620 is still alive ksuitm_check: OS PID=360618 is still alive ksuitm_check: OS PID=360179 is still alive ksuitm_check: OS PID=360616 is still alive ksuitm_check: OS PID=360612 is still alive ksuitm_check: OS PID=360044 is still alive ksuitm_check: OS PID=360614 is still alive ksuitm_check: OS PID=360610 is still alive ksuitm_check: OS PID=360607 is still alive ksuitm_check: OS PID=360605 is still alive ksuitm_check: OS PID=358415 is still alive ksuitm_check: OS PID=360585 is still alive ksuitm_check: OS PID=358375 is still alive ksuitm_check: OS PID=360598 is still alive ksuitm_check: OS PID=358298 is still alive ksuitm_check: OS PID=358294 is still alive ksuitm_check: OS PID=358291 is still alive ksuitm_check: OS PID=358287 is still alive ksuitm_check: OS PID=358284 is still alive ksuitm_check: OS PID=358282 is still alive ksuitm_check: OS PID=358280 is still alive ksuitm_check: OS PID=358278 is still alive ksuitm_check: OS PID=358276 is still alive ksuitm_check: OS PID=358269 is still alive ksuitm_check: OS PID=358228 is still alive ksuitm_check: OS PID=358222 is still alive ksuitm_check: OS PID=358220 is still alive ksuitm_check: OS PID=358217 is still alive ksuitm_check: OS PID=358213 is still alive ksuitm_check: OS PID=358210 is still alive ksuitm_check: OS PID=358202 is still alive ksuitm_check: OS PID=358191 is still alive ksuitm_check: OS PID=358194 is still alive ksuitm_check: OS PID=358185 is still alive ksuitm_check: OS PID=358181 is still alive ksuitm_check: OS PID=358160 is still alive ksuitm_check: OS PID=358156 is still alive ksuitm_check: OS PID=358086 is still alive ksuitm_check: OS PID=358146 is still alive ksuitm_check: OS PID=358082 is still alive ksuitm_check: OS PID=358075 is still alive ksuitm_check: OS PID=358033 is still alive ksuitm_check: OS PID=358022 is still alive ksuitm_check: OS PID=358020 is still alive ksuitm_check: OS PID=357961 is still alive ksuitm_check: OS PID=357959 is still alive ksuitm_check: OS PID=357955 is still alive ksuitm_check: OS PID=357953 is still alive ksuitm_check: OS PID=357948 is still alive ksuitm_check: OS PID=357939 is still alive ksuitm_check: OS PID=357937 is still alive ksuitm_check: OS PID=357935 is still alive ksuitm_check: OS PID=357931 is still alive ksuitm_check: OS PID=357929 is still alive ksuitm_check: OS PID=357927 is still alive ksuitm_check: OS PID=357925 is still alive ksuitm_check: OS PID=357923 is still alive ksuitm_check: OS PID=357892 is still alive ksuitm_check: OS PID=357880 is still alive ksuitm_check: OS PID=357876 is still alive ksuitm_check: OS PID=357874 is still alive ksuitm_check: OS PID=357870 is still alive ksuitm_check: OS PID=358061 is still alive ksuitm_check: OS PID=357863 is still alive ksuitm_check: OS PID=357861 is still alive ksuitm_check: OS PID=357859 is still alive ksuitm_check: OS PID=357855 is still alive ksuitm_check: OS PID=357843 is still alive ksuitm_check: OS PID=357841 is still alive ksuitm_check: OS PID=357839 is still alive ksuitm_check: OS PID=357837 is still alive ksuitm_check: OS PID=357835 is still alive ksuitm_check: OS PID=357824 is still alive ksuitm_check: OS PID=357818 is still alive ksuitm_check: OS PID=357814 is still alive ksuitm_check: OS PID=357802 is still alive ksuitm_check: OS PID=357799 is still alive ksuitm_check: OS PID=357797 is still alive ksuitm_check: OS PID=357795 is still alive ksuitm_check: OS PID=357793 is still alive ksuitm_check: OS PID=357748 is still alive ksuitm_check: OS PID=357726 is still alive ksuitm_check: OS PID=357722 is still alive ksuitm_check: OS PID=357720 is still alive ksuitm_check: OS PID=357715 is still alive ksuitm_check: OS PID=357713 is still alive ksuitm_check: OS PID=357710 is still alive ksuitm_check: OS PID=357706 is still alive ksuitm_check: OS PID=357702 is still alive ksuitm_check: OS PID=357700 is still alive ksuitm_check: OS PID=357698 is still alive ksuitm_check: OS PID=357696 is still alive ksuitm_check: OS PID=357694 is still alive ksuitm_check: OS PID=357663 is still alive ksuitm_check: OS PID=357655 is still alive ksuitm_check: OS PID=357653 is still alive ksuitm_check: OS PID=357651 is still alive ksuitm_check: OS PID=357647 is still alive ksuitm_check: OS PID=357632 is still alive ksuitm_check: OS PID=357568 is still alive ksuitm_check: OS PID=357551 is still alive ksuitm_check: OS PID=357549 is still alive ksuitm_check: OS PID=357547 is still alive ksuitm_check: OS PID=357531 is still alive ksuitm_check: OS PID=357468 is still alive ksuitm_check: OS PID=357440 is still alive ksuitm_check: OS PID=357355 is still alive ksuitm_check: OS PID=357293 is still alive ksuitm_check: OS PID=357257 is still alive ksuitm_check: OS PID=357254 is still alive ksuitm_check: OS PID=357250 is still alive ksuitm_check: OS PID=357209 is still alive ksuitm_check: OS PID=357207 is still alive ksuitm_check: OS PID=357147 is still alive ksuitm_check: OS PID=357140 is still alive ksuitm_check: OS PID=357205 is still alive ksuitm_check: OS PID=357162 is still alive ksuitm_check: OS PID=357157 is still alive ksuitm_check: OS PID=356318 is still alive ksuitm_check: OS PID=356270 is still alive ksuitm_check: OS PID=354493 is still alive ksuitm_check: OS PID=354491 is still alive ksuitm_check: OS PID=354483 is still alive ksuitm_check: OS PID=354479 is still alive ksuitm_check: OS PID=354477 is still alive ksuitm_check: OS PID=354475 is still alive ksuitm_check: OS PID=354473 is still alive ksuitm_check: OS PID=354463 is still alive ksuitm_check: OS PID=358024 is still alive ksuitm_check: OS PID=354408 is still alive ksuitm_check: OS PID=354396 is still alive ksuitm_check: OS PID=354394 is still alive ksuitm_check: OS PID=354382 is still alive ksuitm_check: OS PID=354379 is still alive ksuitm_check: OS PID=354368 is still alive ksuitm_check: OS PID=354366 is still alive ksuitm_check: OS PID=354364 is still alive ksuitm_check: OS PID=354362 is still alive ksuitm_check: OS PID=354360 is still alive ksuitm_check: OS PID=354354 is still alive ksuitm_check: OS PID=354334 is still alive ksuitm_check: OS PID=354330 is still alive ksuitm_check: OS PID=354318 is still alive ksuitm_check: OS PID=354313 is still alive ksuitm_check: OS PID=354309 is still alive ksuitm_check: OS PID=354301 is still alive ksuitm_check: OS PID=354298 is still alive ksuitm_check: OS PID=354296 is still alive ksuitm_check: OS PID=354294 is still alive ksuitm_check: OS PID=354292 is still alive ksuitm_check: OS PID=354290 is still alive ksuitm_check: OS PID=354286 is still alive ksuitm_check: OS PID=354284 is still alive ksuitm_check: OS PID=354282 is still alive ksuitm_check: OS PID=354280 is still alive ksuitm_check: OS PID=354278 is still alive ksuitm_check: OS PID=354273 is still alive ksuitm_check: OS PID=354144 is still alive ksuitm_check: OS PID=354140 is still alive ksuitm_check: OS PID=354138 is still alive ksuitm_check: OS PID=354136 is still alive ksuitm_check: OS PID=354134 is still alive ksuitm_check: OS PID=354129 is still alive ksuitm_check: OS PID=354127 is still alive ksuitm_check: OS PID=354125 is still alive ksuitm_check: OS PID=354123 is still alive ksuitm_check: OS PID=354121 is still alive ksuitm_check: OS PID=354119 is still alive ksuitm_check: OS PID=354117 is still alive ksuitm_check: OS PID=354115 is still alive ksuitm_check: OS PID=354113 is still alive ksuitm_check: OS PID=354109 is still alive ksuitm_check: OS PID=354072 is still alive ksuitm_check: OS PID=354060 is still alive ksuitm_check: OS PID=354057 is still alive ksuitm_check: OS PID=354053 is still alive ksuitm_check: OS PID=354046 is still alive ksuitm_check: OS PID=354024 is still alive ksuitm_check: OS PID=354014 is still alive ksuitm_check: OS PID=354010 is still alive ksuitm_check: OS PID=354006 is still alive ksuitm_check: OS PID=354004 is still alive ksuitm_check: OS PID=354002 is still alive ksuitm_check: OS PID=354000 is still alive ksuitm_check: OS PID=353998 is still alive ksuitm_check: OS PID=353996 is still alive ksuitm_check: OS PID=353992 is still alive ksuitm_check: OS PID=353990 is still alive ksuitm_check: OS PID=353988 is still alive ksuitm_check: OS PID=353986 is still alive ksuitm_check: OS PID=353984 is still alive ksuitm_check: OS PID=353895 is still alive ksuitm_check: OS PID=353891 is still alive ksuitm_check: OS PID=353887 is still alive ksuitm_check: OS PID=353883 is still alive ksuitm_check: OS PID=353868 is still alive ksuitm_check: OS PID=353864 is still alive ksuitm_check: OS PID=353854 is still alive ksuitm_check: OS PID=353840 is still alive ksuitm_check: OS PID=353838 is still alive ksuitm_check: OS PID=353820 is still alive ksuitm_check: OS PID=353832 is still alive ksuitm_check: OS PID=353797 is still alive ksuitm_check: OS PID=353795 is still alive ksuitm_check: OS PID=353793 is still alive ksuitm_check: OS PID=353785 is still alive ksuitm_check: OS PID=353783 is still alive ksuitm_check: OS PID=353764 is still alive ksuitm_check: OS PID=353982 is still alive ksuitm_check: OS PID=353470 is still alive ksuitm_check: OS PID=353449 is still alive ksuitm_check: OS PID=353432 is still alive ksuitm_check: OS PID=353399 is still alive ksuitm_check: OS PID=353394 is still alive ksuitm_check: OS PID=353387 is still alive ksuitm_check: OS PID=353385 is still alive ksuitm_check: OS PID=353383 is still alive ksuitm_check: OS PID=353380 is still alive ksuitm_check: OS PID=353368 is still alive ksuitm_check: OS PID=353363 is still alive ksuitm_check: OS PID=353360 is still alive ksuitm_check: OS PID=353358 is still alive ksuitm_check: OS PID=353356 is still alive ksuitm_check: OS PID=353302 is still alive ksuitm_check: OS PID=353300 is still alive ksuitm_check: OS PID=353298 is still alive ksuitm_check: OS PID=353296 is still alive ksuitm_check: OS PID=353294 is still alive ksuitm_check: OS PID=353284 is still alive ksuitm_check: OS PID=353282 is still alive ksuitm_check: OS PID=353278 is still alive ksuitm_check: OS PID=353276 is still alive ksuitm_check: OS PID=353265 is still alive ksuitm_check: OS PID=353263 is still alive ksuitm_check: OS PID=353261 is still alive ksuitm_check: OS PID=353259 is still alive ksuitm_check: OS PID=353257 is still alive ksuitm_check: OS PID=353249 is still alive ksuitm_check: OS PID=353236 is still alive ksuitm_check: OS PID=353234 is still alive ksuitm_check: OS PID=353229 is still alive ksuitm_check: OS PID=353354 is still alive ksuitm_check: OS PID=353352 is still alive ksuitm_check: OS PID=353215 is still alive ksuitm_check: OS PID=360174 is still alive ksuitm_check: OS PID=353350 is still alive ksuitm_check: OS PID=353187 is still alive ksuitm_check: OS PID=353348 is still alive ksuitm_check: OS PID=353346 is still alive ksuitm_check: OS PID=353127 is still alive ksuitm_check: OS PID=353343 is still alive ksuitm_check: OS PID=353201 is still alive ksuitm_check: OS PID=360568 is still alive ksuitm_check: OS PID=353341 is still alive ksuitm_check: OS PID=352330 is still alive ksuitm_check: OS PID=352326 is still alive ksuitm_check: OS PID=352324 is still alive ksuitm_check: OS PID=352320 is still alive ksuitm_check: OS PID=352318 is still alive ksuitm_check: OS PID=352316 is still alive ksuitm_check: OS PID=352232 is still alive ksuitm_check: OS PID=352230 is still alive ksuitm_check: OS PID=352226 is still alive ksuitm_check: OS PID=352216 is still alive ksuitm_check: OS PID=352213 is still alive ksuitm_check: OS PID=352208 is still alive ksuitm_check: OS PID=352194 is still alive ksuitm_check: OS PID=352186 is still alive ksuitm_check: OS PID=352161 is still alive ksuitm_check: OS PID=352163 is still alive ksuitm_check: OS PID=352154 is still alive ksuitm_check: OS PID=352151 is still alive ksuitm_check: OS PID=352147 is still alive ksuitm_check: OS PID=352145 is still alive ksuitm_check: OS PID=352132 is still alive ksuitm_check: OS PID=352124 is still alive ksuitm_check: OS PID=352122 is still alive ksuitm_check: OS PID=352120 is still alive ksuitm_check: OS PID=352116 is still alive ksuitm_check: OS PID=352104 is still alive ksuitm_check: OS PID=352102 is still alive ksuitm_check: OS PID=352092 is still alive ksuitm_check: OS PID=352086 is still alive ksuitm_check: OS PID=352084 is still alive ksuitm_check: OS PID=352082 is still alive ksuitm_check: OS PID=352080 is still alive ksuitm_check: OS PID=352078 is still alive ksuitm_check: OS PID=352076 is still alive ksuitm_check: OS PID=352073 is still alive ksuitm_check: OS PID=352053 is still alive ksuitm_check: OS PID=352049 is still alive ksuitm_check: OS PID=352043 is still alive ksuitm_check: OS PID=352040 is still alive ksuitm_check: OS PID=352034 is still alive ksuitm_check: OS PID=352020 is still alive ksuitm_check: OS PID=352018 is still alive ksuitm_check: OS PID=352014 is still alive ksuitm_check: OS PID=352012 is still alive ksuitm_check: OS PID=352006 is still alive ksuitm_check: OS PID=352002 is still alive ksuitm_check: OS PID=351974 is still alive ksuitm_check: OS PID=351969 is still alive ksuitm_check: OS PID=351967 is still alive ksuitm_check: OS PID=351959 is still alive ksuitm_check: OS PID=351956 is still alive ksuitm_check: OS PID=351952 is still alive ksuitm_check: OS PID=351950 is still alive ksuitm_check: OS PID=351947 is still alive ksuitm_check: OS PID=351945 is still alive ksuitm_check: OS PID=351928 is still alive ksuitm_check: OS PID=351787 is still alive ksuitm_check: OS PID=351686 is still alive ksuitm_check: OS PID=351658 is still alive ksuitm_check: OS PID=351651 is still alive ksuitm_check: OS PID=351647 is still alive ksuitm_check: OS PID=351643 is still alive ksuitm_check: OS PID=351599 is still alive ksuitm_check: OS PID=351601 is still alive ksuitm_check: OS PID=351595 is still alive ksuitm_check: OS PID=351592 is still alive ksuitm_check: OS PID=351588 is still alive ksuitm_check: OS PID=351586 is still alive ksuitm_check: OS PID=351584 is still alive ksuitm_check: OS PID=351582 is still alive ksuitm_check: OS PID=351580 is still alive ksuitm_check: OS PID=351578 is still alive ksuitm_check: OS PID=351576 is still alive ksuitm_check: OS PID=351574 is still alive ksuitm_check: OS PID=351572 is still alive ksuitm_check: OS PID=351566 is still alive ksuitm_check: OS PID=351553 is still alive ksuitm_check: OS PID=351551 is still alive ksuitm_check: OS PID=351543 is still alive ksuitm_check: OS PID=351541 is still alive ksuitm_check: OS PID=351538 is still alive ksuitm_check: OS PID=351533 is still alive ksuitm_check: OS PID=351511 is still alive ksuitm_check: OS PID=351505 is still alive ksuitm_check: OS PID=351496 is still alive ksuitm_check: OS PID=351492 is still alive ksuitm_check: OS PID=351488 is still alive ksuitm_check: OS PID=351482 is still alive ksuitm_check: OS PID=351480 is still alive ksuitm_check: OS PID=351478 is still alive ksuitm_check: OS PID=351476 is still alive ksuitm_check: OS PID=351474 is still alive ksuitm_check: OS PID=351460 is still alive ksuitm_check: OS PID=351446 is still alive ksuitm_check: OS PID=351444 is still alive ksuitm_check: OS PID=351442 is still alive ksuitm_check: OS PID=351440 is still alive ksuitm_check: OS PID=351436 is still alive ksuitm_check: OS PID=351434 is still alive ksuitm_check: OS PID=351361 is still alive ksuitm_check: OS PID=351357 is still alive ksuitm_check: OS PID=351359 is still alive ksuitm_check: OS PID=351215 is still alive ksuitm_check: OS PID=351211 is still alive ksuitm_check: OS PID=351209 is still alive ksuitm_check: OS PID=351207 is still alive ksuitm_check: OS PID=351204 is still alive ksuitm_check: OS PID=351196 is still alive ksuitm_check: OS PID=351191 is still alive ksuitm_check: OS PID=351185 is still alive ksuitm_check: OS PID=351179 is still alive ksuitm_check: OS PID=351173 is still alive ksuitm_check: OS PID=351134 is still alive ksuitm_check: OS PID=351091 is still alive ksuitm_check: OS PID=351076 is still alive ksuitm_check: OS PID=351078 is still alive ksuitm_check: OS PID=351062 is still alive ksuitm_check: OS PID=351045 is still alive ksuitm_check: OS PID=350947 is still alive ksuitm_check: OS PID=350945 is still alive ksuitm_check: OS PID=350943 is still alive ksuitm_check: OS PID=350941 is still alive ksuitm_check: OS PID=350928 is still alive ksuitm_check: OS PID=351217 is still alive ksuitm_check: OS PID=350901 is still alive ksuitm_check: OS PID=350728 is still alive ksuitm_check: OS PID=350726 is still alive ksuitm_check: OS PID=350689 is still alive ksuitm_check: OS PID=350682 is still alive ksuitm_check: OS PID=350687 is still alive ksuitm_check: OS PID=350676 is still alive ksuitm_check: OS PID=350662 is still alive ksuitm_check: OS PID=350658 is still alive ksuitm_check: OS PID=350656 is still alive ksuitm_check: OS PID=350652 is still alive ksuitm_check: OS PID=350641 is still alive ksuitm_check: OS PID=350637 is still alive ksuitm_check: OS PID=350629 is still alive ksuitm_check: OS PID=350626 is still alive ksuitm_check: OS PID=350624 is still alive ksuitm_check: OS PID=350620 is still alive ksuitm_check: OS PID=350618 is still alive ksuitm_check: OS PID=350616 is still alive ksuitm_check: OS PID=350614 is still alive ksuitm_check: OS PID=350600 is still alive ksuitm_check: OS PID=350598 is still alive ksuitm_check: OS PID=350596 is still alive ksuitm_check: OS PID=350594 is still alive ksuitm_check: OS PID=350592 is still alive ksuitm_check: OS PID=350590 is still alive ksuitm_check: OS PID=350588 is still alive ksuitm_check: OS PID=350884 is still alive ksuitm_check: OS PID=350886 is still alive ksuitm_check: OS PID=350878 is still alive ksuitm_check: OS PID=350537 is still alive ksuitm_check: OS PID=350535 is still alive ksuitm_check: OS PID=350868 is still alive ksuitm_check: OS PID=350528 is still alive ksuitm_check: OS PID=350526 is still alive ksuitm_check: OS PID=350522 is still alive ksuitm_check: OS PID=350520 is still alive ksuitm_check: OS PID=350517 is still alive ksuitm_check: OS PID=350515 is still alive ksuitm_check: OS PID=350513 is still alive ksuitm_check: OS PID=350500 is still alive ksuitm_check: OS PID=350496 is still alive ksuitm_check: OS PID=350484 is still alive ksuitm_check: OS PID=350481 is still alive ksuitm_check: OS PID=350477 is still alive ksuitm_check: OS PID=350475 is still alive ksuitm_check: OS PID=350470 is still alive ksuitm_check: OS PID=350468 is still alive ksuitm_check: OS PID=350466 is still alive ksuitm_check: OS PID=350464 is still alive ksuitm_check: OS PID=350462 is still alive ksuitm_check: OS PID=350459 is still alive ksuitm_check: OS PID=350455 is still alive ksuitm_check: OS PID=350453 is still alive ksuitm_check: OS PID=350446 is still alive ksuitm_check: OS PID=350422 is still alive ksuitm_check: OS PID=350420 is still alive ksuitm_check: OS PID=350415 is still alive ksuitm_check: OS PID=350402 is still alive ksuitm_check: OS PID=350391 is still alive ksuitm_check: OS PID=350384 is still alive ksuitm_check: OS PID=350373 is still alive ksuitm_check: OS PID=350369 is still alive ksuitm_check: OS PID=350365 is still alive ksuitm_check: OS PID=350363 is still alive ksuitm_check: OS PID=350361 is still alive ksuitm_check: OS PID=350358 is still alive ksuitm_check: OS PID=350350 is still alive ksuitm_check: OS PID=350329 is still alive ksuitm_check: OS PID=350326 is still alive ksuitm_check: OS PID=350324 is still alive ksuitm_check: OS PID=350316 is still alive ksuitm_check: OS PID=350314 is still alive ksuitm_check: OS PID=350310 is still alive ksuitm_check: OS PID=350308 is still alive ksuitm_check: OS PID=350299 is still alive ksuitm_check: OS PID=350297 is still alive ksuitm_check: OS PID=350293 is still alive ksuitm_check: OS PID=350253 is still alive ksuitm_check: OS PID=350200 is still alive ksuitm_check: OS PID=350108 is still alive ksuitm_check: OS PID=350045 is still alive ksuitm_check: OS PID=349953 is still alive ksuitm_check: OS PID=349907 is still alive ksuitm_check: OS PID=349903 is still alive ksuitm_check: OS PID=349901 is still alive ksuitm_check: OS PID=349898 is still alive ksuitm_check: OS PID=349895 is still alive ksuitm_check: OS PID=349892 is still alive ksuitm_check: OS PID=349889 is still alive ksuitm_check: OS PID=349887 is still alive ksuitm_check: OS PID=349885 is still alive ksuitm_check: OS PID=349875 is still alive ksuitm_check: OS PID=349863 is still alive ksuitm_check: OS PID=349861 is still alive ksuitm_check: OS PID=347105 is still alive ksuitm_check: OS PID=346886 is still alive ksuitm_check: OS PID=346884 is still alive ksuitm_check: OS PID=346881 is still alive ksuitm_check: OS PID=346879 is still alive ksuitm_check: OS PID=346877 is still alive ksuitm_check: OS PID=346873 is still alive ksuitm_check: OS PID=346870 is still alive ksuitm_check: OS PID=346866 is still alive ksuitm_check: OS PID=346858 is still alive ksuitm_check: OS PID=346848 is still alive ksuitm_check: OS PID=346846 is still alive ksuitm_check: OS PID=346844 is still alive ksuitm_check: OS PID=346842 is still alive ksuitm_check: OS PID=346840 is still alive ksuitm_check: OS PID=346825 is still alive ksuitm_check: OS PID=346821 is still alive ksuitm_check: OS PID=346814 is still alive ksuitm_check: OS PID=346783 is still alive ksuitm_check: OS PID=346765 is still alive ksuitm_check: OS PID=346541 is still alive ksuitm_check: OS PID=346539 is still alive ksuitm_check: OS PID=346537 is still alive ksuitm_check: OS PID=346527 is still alive ksuitm_check: OS PID=346522 is still alive ksuitm_check: OS PID=346520 is still alive ksuitm_check: OS PID=346517 is still alive ksuitm_check: OS PID=346512 is still alive ksuitm_check: OS PID=346510 is still alive ksuitm_check: OS PID=346498 is still alive ksuitm_check: OS PID=346494 is still alive ksuitm_check: OS PID=346492 is still alive ksuitm_check: OS PID=346150 is still alive ksuitm_check: OS PID=346148 is still alive ksuitm_check: OS PID=346124 is still alive ksuitm_check: OS PID=346490 is still alive ksuitm_check: OS PID=346488 is still alive ksuitm_check: OS PID=346485 is still alive ksuitm_check: OS PID=346482 is still alive ksuitm_check: OS PID=346479 is still alive ksuitm_check: OS PID=346474 is still alive ksuitm_check: OS PID=346085 is still alive ksuitm_check: OS PID=346470 is still alive ksuitm_check: OS PID=346081 is still alive ksuitm_check: OS PID=346079 is still alive ksuitm_check: OS PID=346468 is still alive ksuitm_check: OS PID=346073 is still alive ksuitm_check: OS PID=346464 is still alive ksuitm_check: OS PID=346068 is still alive ksuitm_check: OS PID=346066 is still alive ksuitm_check: OS PID=346064 is still alive ksuitm_check: OS PID=346058 is still alive ksuitm_check: OS PID=346060 is still alive ksuitm_check: OS PID=346054 is still alive ksuitm_check: OS PID=346051 is still alive ksuitm_check: OS PID=346049 is still alive ksuitm_check: OS PID=346037 is still alive ksuitm_check: OS PID=346035 is still alive ksuitm_check: OS PID=346024 is still alive ksuitm_check: OS PID=346022 is still alive ksuitm_check: OS PID=346019 is still alive ksuitm_check: OS PID=346017 is still alive ksuitm_check: OS PID=346013 is still alive ksuitm_check: OS PID=346007 is still alive ksuitm_check: OS PID=346005 is still alive ksuitm_check: OS PID=346001 is still alive ksuitm_check: OS PID=345987 is still alive ksuitm_check: OS PID=345983 is still alive ksuitm_check: OS PID=345978 is still alive ksuitm_check: OS PID=345973 is still alive ksuitm_check: OS PID=345943 is still alive ksuitm_check: OS PID=345941 is still alive ksuitm_check: OS PID=345937 is still alive ksuitm_check: OS PID=345933 is still alive ksuitm_check: OS PID=345922 is still alive ksuitm_check: OS PID=345919 is still alive ksuitm_check: OS PID=345909 is still alive ksuitm_check: OS PID=345905 is still alive ksuitm_check: OS PID=345899 is still alive ksuitm_check: OS PID=345887 is still alive ksuitm_check: OS PID=345882 is still alive ksuitm_check: OS PID=345872 is still alive ksuitm_check: OS PID=345869 is still alive ksuitm_check: OS PID=345867 is still alive ksuitm_check: OS PID=345865 is still alive ksuitm_check: OS PID=345863 is still alive ksuitm_check: OS PID=345861 is still alive ksuitm_check: OS PID=345859 is still alive ksuitm_check: OS PID=345857 is still alive ksuitm_check: OS PID=345855 is still alive ksuitm_check: OS PID=345851 is still alive ksuitm_check: OS PID=345847 is still alive ksuitm_check: OS PID=345830 is still alive ksuitm_check: OS PID=345833 is still alive ksuitm_check: OS PID=345826 is still alive ksuitm_check: OS PID=345822 is still alive ksuitm_check: OS PID=345820 is still alive ksuitm_check: OS PID=333255 is still alive ksuitm_check: OS PID=344723 is still alive ksuitm_check: OS PID=352192 is still alive ksuitm_check: OS PID=332956 is still alive ksuitm_check: OS PID=332954 is still alive ksuitm_check: OS PID=332951 is still alive ksuitm_check: OS PID=332949 is still alive ksuitm_check: OS PID=351432 is still alive ksuitm_check: OS PID=344694 is still alive ksuitm_check: OS PID=332941 is still alive ksuitm_check: OS PID=349859 is still alive ksuitm_check: OS PID=332894 is still alive ksuitm_check: OS PID=315854 is still alive ksuitm_check: OS PID=332891 is still alive ksuitm_check: OS PID=332889 is still alive ksuitm_check: OS PID=315850 is still alive ksuitm_check: OS PID=332887 is still alive ksuitm_check: OS PID=315848 is still alive ksuitm_check: OS PID=332883 is still alive ksuitm_check: OS PID=315846 is still alive ksuitm_check: OS PID=275447 is still alive ksuitm_check: OS PID=345764 is still alive ksuitm_check: OS PID=320938 is still alive ksuitm_check: OS PID=319582 is still alive ksuitm_check: OS PID=344685 is still alive ksuitm_check: OS PID=332232 is still alive ksuitm_check: OS PID=144360 is still alive ksuitm_check: OS PID=345725 is still alive ksuitm_check: OS PID=338194 is still alive ksuitm_check: OS PID=352190 is still alive ksuitm_check: OS PID=350473 is still alive ksuitm_check: OS PID=332223 is still alive ksuitm_check: OS PID=345633 is still alive ksuitm_check: OS PID=339607 is still alive ksuitm_check: OS PID=345573 is still alive ksuitm_check: OS PID=315013 is still alive ksuitm_check: OS PID=339526 is still alive ksuitm_check: OS PID=339524 is still alive ksuitm_check: OS PID=332584 is still alive ksuitm_check: OS PID=332582 is still alive ksuitm_check: OS PID=345534 is still alive ksuitm_check: OS PID=302786 is still alive ksuitm_check: OS PID=332579 is still alive ksuitm_check: OS PID=302784 is still alive ksuitm_check: OS PID=349857 is still alive ksuitm_check: OS PID=224600 is still alive ksuitm_check: OS PID=251863 is still alive ksuitm_check: OS PID=332577 is still alive ksuitm_check: OS PID=332575 is still alive ksuitm_check: OS PID=332939 is still alive ksuitm_check: OS PID=332937 is still alive ksuitm_check: OS PID=332926 is still alive ksuitm_check: OS PID=332885 is still alive ksuitm_check: OS PID=332876 is still alive ksuitm_check: OS PID=332573 is still alive ksuitm_check: OS PID=344750 is still alive ksuitm_check: OS PID=332868 is still alive ksuitm_check: OS PID=251573 is still alive ksuitm_check: OS PID=301846 is still alive ksuitm_check: OS PID=314954 is still alive ksuitm_check: OS PID=345527 is still alive ksuitm_check: OS PID=333738 is still alive ksuitm_check: OS PID=314908 is still alive ksuitm_check: OS PID=346502 is still alive ksuitm_check: OS PID=333864 is still alive ksuitm_check: OS PID=275371 is still alive ksuitm_check: OS PID=332879 is still alive ksuitm_check: OS PID=339629 is still alive ksuitm_check: OS PID=344589 is still alive ksuitm_check: OS PID=344587 is still alive ksuitm_check: OS PID=354356 is still alive ksuitm_check: OS PID=338834 is still alive ksuitm_check: OS PID=349855 is still alive ksuitm_check: OS PID=349853 is still alive ksuitm_check: OS PID=349851 is still alive ksuitm_check: OS PID=332567 is still alive ksuitm_check: OS PID=71738 is still alive ksuitm_check: OS PID=333291 is still alive ksuitm_check: OS PID=337373 is still alive ksuitm_check: OS PID=307154 is still alive ksuitm_check: OS PID=345418 is still alive ksuitm_check: OS PID=113607 is still alive ksuitm_check: OS PID=238213 is still alive ksuitm_check: OS PID=332922 is still alive ksuitm_check: OS PID=332918 is still alive ksuitm_check: OS PID=332916 is still alive ksuitm_check: OS PID=332914 is still alive ksuitm_check: OS PID=122743 is still alive ksuitm_check: OS PID=346773 is still alive ksuitm_check: OS PID=334408 is still alive ksuitm_check: OS PID=233245 is still alive ksuitm_check: OS PID=332852 is still alive ksuitm_check: OS PID=71162 is still alive ksuitm_check: OS PID=333057 is still alive ksuitm_check: OS PID=269538 is still alive ksuitm_check: OS PID=332563 is still alive ksuitm_check: OS PID=349849 is still alive ksuitm_check: OS PID=267137 is still alive ksuitm_check: OS PID=276391 is still alive ksuitm_check: OS PID=332466 is still alive ksuitm_check: OS PID=333068 is still alive ksuitm_check: OS PID=353391 is still alive ksuitm_check: OS PID=349847 is still alive ksuitm_check: OS PID=353389 is still alive ksuitm_check: OS PID=345394 is still alive ksuitm_check: OS PID=353191 is still alive ksuitm_check: OS PID=345361 is still alive ksuitm_check: OS PID=338742 is still alive ksuitm_check: OS PID=123530 is still alive ksuitm_check: OS PID=350932 is still alive ksuitm_check: OS PID=296623 is still alive ksuitm_check: OS PID=338950 is still alive ksuitm_check: OS PID=338740 is still alive ksuitm_check: OS PID=338682 is still alive ksuitm_check: OS PID=338678 is still alive ksuitm_check: OS PID=338641 is still alive ksuitm_check: OS PID=359695 is still alive ksuitm_check: OS PID=338634 is still alive ksuitm_check: OS PID=388592 is still alive ksuitm_check: OS PID=338592 is still alive ksuitm_check: OS PID=388582 is still alive ksuitm_check: OS PID=151346 is still alive ksuitm_check: OS PID=137861 is still alive ksuitm_check: OS PID=307033 is still alive ksuitm_check: OS PID=388569 is still alive ksuitm_check: OS PID=239496 is still alive ksuitm_check: OS PID=334239 is still alive ksuitm_check: OS PID=354402 is still alive ksuitm_check: OS PID=334237 is still alive ksuitm_check: OS PID=354434 is still alive ksuitm_check: OS PID=388417 is still alive ksuitm_check: OS PID=339465 is still alive ksuitm_check: OS PID=288210 is still alive ksuitm_check: OS PID=353461 is still alive ksuitm_check: OS PID=354400 is still alive ksuitm_check: OS PID=344979 is still alive ksuitm_check: OS PID=320975 is still alive ksuitm_check: OS PID=354251 is still alive ksuitm_check: OS PID=42629 is still alive ksuitm_check: OS PID=332908 is still alive ksuitm_check: OS PID=354245 is still alive ksuitm_check: OS PID=353366 is still alive ksuitm_check: OS PID=287807 is still alive ksuitm_check: OS PID=360565 is still alive ksuitm_check: OS PID=354234 is still alive ksuitm_check: OS PID=354232 is still alive ksuitm_check: OS PID=269826 is still alive ksuitm_check: OS PID=354230 is still alive ksuitm_check: OS PID=319943 is still alive ksuitm_check: OS PID=357639 is still alive ksuitm_check: OS PID=354220 is still alive ksuitm_check: OS PID=354218 is still alive ksuitm_check: OS PID=320791 is still alive ksuitm_check: OS PID=354213 is still alive ksuitm_check: OS PID=42621 is still alive ksuitm_check: OS PID=354211 is still alive ksuitm_check: OS PID=354209 is still alive ksuitm_check: OS PID=354206 is still alive ksuitm_check: OS PID=358080 is still alive ksuitm_check: OS PID=354204 is still alive ksuitm_check: OS PID=354962 is still alive ksuitm_check: OS PID=354202 is still alive ksuitm_check: OS PID=332958 is still alive ksuitm_check: OS PID=347015 is still alive ksuitm_check: OS PID=354200 is still alive ksuitm_check: OS PID=354196 is still alive ksuitm_check: OS PID=349845 is still alive ksuitm_check: OS PID=350874 is still alive ksuitm_check: OS PID=346995 is still alive ksuitm_check: OS PID=341341 is still alive ksuitm_check: OS PID=312624 is still alive ksuitm_check: OS PID=8736 is still alive ksuitm_check: OS PID=282610 is still alive ksuitm_check: OS PID=72140 is still alive ksuitm_check: OS PID=346983 is still alive ksuitm_check: OS PID=361123 is still alive ksuitm_check: OS PID=177937 is still alive ksuitm_check: OS PID=320813 is still alive ksuitm_check: OS PID=387851 is still alive ksuitm_check: OS PID=339848 is still alive ksuitm_check: OS PID=339158 is still alive ksuitm_check: OS PID=346977 is still alive ksuitm_check: OS PID=346971 is still alive ksuitm_check: OS PID=345411 is still alive ksuitm_check: OS PID=286909 is still alive ksuitm_check: OS PID=346969 is still alive ksuitm_check: OS PID=353339 is still alive ksuitm_check: OS PID=346955 is still alive ksuitm_check: OS PID=346953 is still alive ksuitm_check: OS PID=314906 is still alive ksuitm_check: OS PID=346946 is still alive ksuitm_check: OS PID=350926 is still alive ksuitm_check: OS PID=346942 is still alive ksuitm_check: OS PID=346930 is still alive ksuitm_check: OS PID=346922 is still alive ksuitm_check: OS PID=298085 is still alive ksuitm_check: OS PID=346912 is still alive ksuitm_check: OS PID=350864 is still alive ksuitm_check: OS PID=346910 is still alive ksuitm_check: OS PID=294772 is still alive ksuitm_check: OS PID=346906 is still alive ksuitm_check: OS PID=339811 is still alive ksuitm_check: OS PID=361103 is still alive ksuitm_check: OS PID=42544 is still alive ksuitm_check: OS PID=344439 is still alive ksuitm_check: OS PID=353994 is still alive ksuitm_check: OS PID=345311 is still alive ksuitm_check: OS PID=346908 is still alive ksuitm_check: OS PID=338566 is still alive ksuitm_check: OS PID=162947 is still alive ksuitm_check: OS PID=327984 is still alive ksuitm_check: OS PID=338326 is still alive ksuitm_check: OS PID=185758 is still alive ksuitm_check: OS PID=370823 is still alive ksuitm_check: OS PID=370999 is still alive ksuitm_check: OS PID=346904 is still alive ksuitm_check: OS PID=346966 is still alive ksuitm_check: OS PID=185122 is still alive ksuitm_check: OS PID=155830 is still alive ksuitm_check: OS PID=374282 is still alive ksuitm_check: OS PID=339805 is still alive ksuitm_check: OS PID=161123 is still alive ksuitm_check: OS PID=364922 is still alive ksuitm_check: OS PID=345849 is still alive ksuitm_check: OS PID=287179 is still alive ksuitm_check: OS PID=334172 is still alive ksuitm_check: OS PID=393386 is still alive ksuitm_check: OS PID=353280 is still alive ksuitm_check: OS PID=352761 is still alive ksuitm_check: OS PID=322530 is still alive ksuitm_check: OS PID=58549 is still alive ksuitm_check: OS PID=375178 is still alive ksuitm_check: OS PID=375174 is still alive ksuitm_check: OS PID=375170 is still alive ksuitm_check: OS PID=375163 is still alive ksuitm_check: OS PID=375143 is still alive ksuitm_check: OS PID=345308 is still alive ksuitm_check: OS PID=319293 is still alive ksuitm_check: OS PID=264429 is still alive ksuitm_check: OS PID=393586 is still alive ksuitm_check: OS PID=375176 is still alive ksuitm_check: OS PID=375172 is still alive ksuitm_check: OS PID=375166 is still alive ksuitm_check: OS PID=375087 is still alive ksuitm_check: OS PID=375161 is still alive ksuitm_check: OS PID=375150 is still alive ksuitm_check: OS PID=338579 is still alive ksuitm_check: OS PID=375052 is still alive ksuitm_check: OS PID=375031 is still alive ksuitm_check: OS PID=374966 is still alive ksuitm_check: OS PID=374962 is still alive ksuitm_check: OS PID=374958 is still alive ksuitm_check: OS PID=374954 is still alive ksuitm_check: OS PID=350312 is still alive ksuitm_check: OS PID=374950 is still alive ksuitm_check: OS PID=374946 is still alive ksuitm_check: OS PID=375057 is still alive ksuitm_check: OS PID=374938 is still alive ksuitm_check: OS PID=375037 is still alive ksuitm_check: OS PID=374924 is still alive ksuitm_check: OS PID=374908 is still alive ksuitm_check: OS PID=374902 is still alive ksuitm_check: OS PID=374898 is still alive ksuitm_check: OS PID=374956 is still alive ksuitm_check: OS PID=374952 is still alive ksuitm_check: OS PID=374948 is still alive ksuitm_check: OS PID=157396 is still alive ksuitm_check: OS PID=374884 is still alive ksuitm_check: OS PID=374873 is still alive ksuitm_check: OS PID=374932 is still alive ksuitm_check: OS PID=374861 is still alive ksuitm_check: OS PID=374920 is still alive ksuitm_check: OS PID=374841 is still alive ksuitm_check: OS PID=374930 is still alive ksuitm_check: OS PID=374773 is still alive ksuitm_check: OS PID=374904 is still alive ksuitm_check: OS PID=374737 is still alive ksuitm_check: OS PID=374877 is still alive ksuitm_check: OS PID=374900 is still alive ksuitm_check: OS PID=350827 is still alive ksuitm_check: OS PID=374896 is still alive ksuitm_check: OS PID=374720 is still alive ksuitm_check: OS PID=374869 is still alive ksuitm_check: OS PID=374702 is still alive ksuitm_check: OS PID=374935 is still alive ksuitm_check: OS PID=374670 is still alive ksuitm_check: OS PID=374847 is still alive ksuitm_check: OS PID=374660 is still alive ksuitm_check: OS PID=374837 is still alive ksuitm_check: OS PID=374769 is still alive ksuitm_check: OS PID=374607 is still alive ksuitm_check: OS PID=374597 is still alive ksuitm_check: OS PID=374728 is still alive ksuitm_check: OS PID=354288 is still alive ksuitm_check: OS PID=374940 is still alive ksuitm_check: OS PID=374680 is still alive ksuitm_check: OS PID=374581 is still alive ksuitm_check: OS PID=374666 is still alive ksuitm_check: OS PID=374706 is still alive ksuitm_check: OS PID=360595 is still alive ksuitm_check: OS PID=374626 is still alive ksuitm_check: OS PID=374145 is still alive ksuitm_check: OS PID=374543 is still alive ksuitm_check: OS PID=374603 is still alive ksuitm_check: OS PID=374127 is still alive ksuitm_check: OS PID=373950 is still alive ksuitm_check: OS PID=374587 is still alive ksuitm_check: OS PID=345285 is still alive ksuitm_check: OS PID=374550 is still alive ksuitm_check: OS PID=373569 is still alive ksuitm_check: OS PID=374152 is still alive ksuitm_check: OS PID=319291 is still alive ksuitm_check: OS PID=374138 is still alive ksuitm_check: OS PID=373389 is still alive ksuitm_check: OS PID=374109 is still alive ksuitm_check: OS PID=314658 is still alive ksuitm_check: OS PID=166107 is still alive ksuitm_check: OS PID=150730 is still alive ksuitm_check: OS PID=328905 is still alive ksuitm_check: OS PID=373610 is still alive ksuitm_check: OS PID=334169 is still alive ksuitm_check: OS PID=373362 is still alive ksuitm_check: OS PID=325956 is still alive ksuitm_check: OS PID=238639 is still alive ksuitm_check: OS PID=373284 is still alive ksuitm_check: OS PID=373328 is still alive ksuitm_check: OS PID=373279 is still alive ksuitm_check: OS PID=373304 is still alive ksuitm_check: OS PID=373237 is still alive ksuitm_check: OS PID=320150 is still alive ksuitm_check: OS PID=373229 is still alive ksuitm_check: OS PID=373235 is still alive ksuitm_check: OS PID=373212 is still alive ksuitm_check: OS PID=373221 is still alive ksuitm_check: OS PID=151144 is still alive ksuitm_check: OS PID=373205 is still alive ksuitm_check: OS PID=373203 is still alive ksuitm_check: OS PID=373197 is still alive ksuitm_check: OS PID=310281 is still alive ksuitm_check: OS PID=349843 is still alive ksuitm_check: OS PID=373191 is still alive ksuitm_check: OS PID=373181 is still alive ksuitm_check: OS PID=373187 is still alive ksuitm_check: OS PID=373179 is still alive ksuitm_check: OS PID=373173 is still alive ksuitm_check: OS PID=373165 is still alive ksuitm_check: OS PID=373158 is still alive ksuitm_check: OS PID=373094 is still alive ksuitm_check: OS PID=373075 is still alive ksuitm_check: OS PID=373073 is still alive ksuitm_check: OS PID=372988 is still alive ksuitm_check: OS PID=373404 is still alive ksuitm_check: OS PID=360588 is still alive ksuitm_check: OS PID=372783 is still alive ksuitm_check: OS PID=372771 is still alive ksuitm_check: OS PID=372765 is still alive ksuitm_check: OS PID=372753 is still alive ksuitm_check: OS PID=364532 is still alive ksuitm_check: OS PID=372728 is still alive ksuitm_check: OS PID=372760 is still alive ksuitm_check: OS PID=364081 is still alive ksuitm_check: OS PID=372657 is still alive ksuitm_check: OS PID=334001 is still alive ksuitm_check: OS PID=372747 is still alive ksuitm_check: OS PID=373326 is still alive ksuitm_check: OS PID=372745 is still alive ksuitm_check: OS PID=372651 is still alive ksuitm_check: OS PID=39417 is still alive ksuitm_check: OS PID=372649 is still alive ksuitm_check: OS PID=372706 is still alive ksuitm_check: OS PID=372647 is still alive ksuitm_check: OS PID=162767 is still alive ksuitm_check: OS PID=372635 is still alive ksuitm_check: OS PID=372631 is still alive ksuitm_check: OS PID=372625 is still alive ksuitm_check: OS PID=157095 is still alive ksuitm_check: OS PID=372509 is still alive ksuitm_check: OS PID=374890 is still alive ksuitm_check: OS PID=371889 is still alive ksuitm_check: OS PID=353972 is still alive ksuitm_check: OS PID=302701 is still alive ksuitm_check: OS PID=369129 is still alive ksuitm_check: OS PID=369100 is still alive ksuitm_check: OS PID=369119 is still alive ksuitm_check: OS PID=369096 is still alive ksuitm_check: OS PID=369114 is still alive ksuitm_check: OS PID=369088 is still alive ksuitm_check: OS PID=369092 is still alive ksuitm_check: OS PID=369058 is still alive ksuitm_check: OS PID=369060 is still alive ksuitm_check: OS PID=328865 is still alive ksuitm_check: OS PID=369040 is still alive ksuitm_check: OS PID=369036 is still alive ksuitm_check: OS PID=369028 is still alive ksuitm_check: OS PID=368991 is still alive ksuitm_check: OS PID=368977 is still alive ksuitm_check: OS PID=368969 is still alive ksuitm_check: OS PID=368963 is still alive ksuitm_check: OS PID=368981 is still alive ksuitm_check: OS PID=368972 is still alive ksuitm_check: OS PID=368965 is still alive ksuitm_check: OS PID=368943 is still alive ksuitm_check: OS PID=368946 is still alive ksuitm_check: OS PID=368934 is still alive ksuitm_check: OS PID=339474 is still alive ksuitm_check: OS PID=368915 is still alive ksuitm_check: OS PID=368909 is still alive ksuitm_check: OS PID=368886 is still alive *** 2019-11-15T14:52:25.652020+04:00 (CDB$ROOT(1)) Instance termination failed to kill one or more processes ksuitm_check: OS PID=360990 is still alive ksuitm_check: OS PID=360970 is still alive ksuitm_check: OS PID=360880 is still alive ksuitm_check: OS PID=360772 is still alive ksuitm_check: OS PID=360757 is still alive ksuitm_check: OS PID=360755 is still alive ksuitm_check: OS PID=360753 is still alive ksuitm_check: OS PID=360749 is still alive ksuitm_check: OS PID=360746 is still alive ksuitm_check: OS PID=360744 is still alive ksuitm_check: OS PID=360734 is still alive ksuitm_check: OS PID=360729 is still alive ksuitm_check: OS PID=360724 is still alive ksuitm_check: OS PID=360722 is still alive ksuitm_check: OS PID=360714 is still alive ksuitm_check: OS PID=360710 is still alive ksuitm_check: OS PID=360706 is still alive ksuitm_check: OS PID=360694 is still alive ksuitm_check: OS PID=360691 is still alive ksuitm_check: OS PID=360689 is still alive ksuitm_check: OS PID=360687 is still alive ksuitm_check: OS PID=360653 is still alive ksuitm_check: OS PID=360651 is still alive ksuitm_check: OS PID=360649 is still alive ksuitm_check: OS PID=360645 is still alive ksuitm_check: OS PID=360642 is still alive ksuitm_check: OS PID=360640 is still alive ksuitm_check: OS PID=360635 is still alive ksuitm_check: OS PID=360631 is still alive ksuitm_check: OS PID=360622 is still alive ksuitm_check: OS PID=360620 is still alive ksuitm_check: OS PID=360618 is still alive ksuitm_check: OS PID=360179 is still alive ksuitm_check: OS PID=360616 is still alive ksuitm_check: OS PID=360612 is still alive ksuitm_check: OS PID=360044 is still alive ksuitm_check: OS PID=360614 is still alive ksuitm_check: OS PID=360610 is still alive ksuitm_check: OS PID=360607 is still alive ksuitm_check: OS PID=360605 is still alive ksuitm_check: OS PID=358415 is still alive ksuitm_check: OS PID=360585 is still alive ksuitm_check: OS PID=358375 is still alive ksuitm_check: OS PID=360598 is still alive ksuitm_check: OS PID=358298 is still alive ksuitm_check: OS PID=358294 is still alive ksuitm_check: OS PID=358291 is still alive ksuitm_check: OS PID=358287 is still alive ksuitm_check: OS PID=358284 is still alive ksuitm_check: OS PID=358282 is still alive ksuitm_check: OS PID=358280 is still alive ksuitm_check: OS PID=358278 is still alive ksuitm_check: OS PID=358276 is still alive ksuitm_check: OS PID=358269 is still alive ksuitm_check: OS PID=358228 is still alive ksuitm_check: OS PID=358222 is still alive ksuitm_check: OS PID=358220 is still alive ksuitm_check: OS PID=358217 is still alive ksuitm_check: OS PID=358213 is still alive ksuitm_check: OS PID=358210 is still alive ksuitm_check: OS PID=358202 is still alive ksuitm_check: OS PID=358191 is still alive ksuitm_check: OS PID=358194 is still alive ksuitm_check: OS PID=358185 is still alive ksuitm_check: OS PID=358181 is still alive ksuitm_check: OS PID=358160 is still alive ksuitm_check: OS PID=358156 is still alive ksuitm_check: OS PID=358086 is still alive ksuitm_check: OS PID=358146 is still alive ksuitm_check: OS PID=358082 is still alive ksuitm_check: OS PID=358075 is still alive ksuitm_check: OS PID=358033 is still alive ksuitm_check: OS PID=358022 is still alive ksuitm_check: OS PID=358020 is still alive ksuitm_check: OS PID=357961 is still alive ksuitm_check: OS PID=357959 is still alive ksuitm_check: OS PID=357955 is still alive ksuitm_check: OS PID=357953 is still alive ksuitm_check: OS PID=357948 is still alive ksuitm_check: OS PID=357939 is still alive ksuitm_check: OS PID=357937 is still alive ksuitm_check: OS PID=357935 is still alive ksuitm_check: OS PID=357931 is still alive ksuitm_check: OS PID=357929 is still alive ksuitm_check: OS PID=357927 is still alive ksuitm_check: OS PID=357925 is still alive ksuitm_check: OS PID=357923 is still alive ksuitm_check: OS PID=357892 is still alive ksuitm_check: OS PID=357880 is still alive ksuitm_check: OS PID=357876 is still alive ksuitm_check: OS PID=357874 is still alive ksuitm_check: OS PID=357870 is still alive ksuitm_check: OS PID=358061 is still alive ksuitm_check: OS PID=357863 is still alive ksuitm_check: OS PID=357861 is still alive ksuitm_check: OS PID=357859 is still alive ksuitm_check: OS PID=357855 is still alive ksuitm_check: OS PID=357843 is still alive ksuitm_check: OS PID=357841 is still alive ksuitm_check: OS PID=357839 is still alive ksuitm_check: OS PID=357837 is still alive ksuitm_check: OS PID=357835 is still alive ksuitm_check: OS PID=357824 is still alive ksuitm_check: OS PID=357818 is still alive ksuitm_check: OS PID=357814 is still alive ksuitm_check: OS PID=357802 is still alive ksuitm_check: OS PID=357799 is still alive ksuitm_check: OS PID=357797 is still alive ksuitm_check: OS PID=357795 is still alive ksuitm_check: OS PID=357793 is still alive ksuitm_check: OS PID=357748 is still alive ksuitm_check: OS PID=357726 is still alive ksuitm_check: OS PID=357722 is still alive ksuitm_check: OS PID=357720 is still alive ksuitm_check: OS PID=357715 is still alive ksuitm_check: OS PID=357713 is still alive ksuitm_check: OS PID=357710 is still alive ksuitm_check: OS PID=357706 is still alive ksuitm_check: OS PID=357702 is still alive ksuitm_check: OS PID=357700 is still alive ksuitm_check: OS PID=357698 is still alive ksuitm_check: OS PID=357696 is still alive ksuitm_check: OS PID=357694 is still alive ksuitm_check: OS PID=357663 is still alive ksuitm_check: OS PID=357655 is still alive ksuitm_check: OS PID=357653 is still alive ksuitm_check: OS PID=357651 is still alive ksuitm_check: OS PID=357647 is still alive ksuitm_check: OS PID=357632 is still alive ksuitm_check: OS PID=357568 is still alive ksuitm_check: OS PID=357551 is still alive ksuitm_check: OS PID=357549 is still alive ksuitm_check: OS PID=357547 is still alive ksuitm_check: OS PID=357531 is still alive ksuitm_check: OS PID=357468 is still alive ksuitm_check: OS PID=357440 is still alive ksuitm_check: OS PID=357355 is still alive ksuitm_check: OS PID=357293 is still alive ksuitm_check: OS PID=357257 is still alive ksuitm_check: OS PID=357254 is still alive ksuitm_check: OS PID=357250 is still alive ksuitm_check: OS PID=357209 is still alive ksuitm_check: OS PID=357207 is still alive ksuitm_check: OS PID=357147 is still alive ksuitm_check: OS PID=357140 is still alive ksuitm_check: OS PID=357205 is still alive ksuitm_check: OS PID=357162 is still alive ksuitm_check: OS PID=357157 is still alive ksuitm_check: OS PID=356318 is still alive ksuitm_check: OS PID=356270 is still alive ksuitm_check: OS PID=354493 is still alive ksuitm_check: OS PID=354491 is still alive ksuitm_check: OS PID=354483 is still alive ksuitm_check: OS PID=354479 is still alive ksuitm_check: OS PID=354477 is still alive ksuitm_check: OS PID=354475 is still alive ksuitm_check: OS PID=354473 is still alive ksuitm_check: OS PID=354463 is still alive ksuitm_check: OS PID=358024 is still alive ksuitm_check: OS PID=354408 is still alive ksuitm_check: OS PID=354396 is still alive ksuitm_check: OS PID=354394 is still alive ksuitm_check: OS PID=354382 is still alive ksuitm_check: OS PID=354379 is still alive ksuitm_check: OS PID=354368 is still alive ksuitm_check: OS PID=354366 is still alive ksuitm_check: OS PID=354364 is still alive ksuitm_check: OS PID=354362 is still alive ksuitm_check: OS PID=354360 is still alive ksuitm_check: OS PID=354354 is still alive ksuitm_check: OS PID=354334 is still alive ksuitm_check: OS PID=354330 is still alive ksuitm_check: OS PID=354318 is still alive ksuitm_check: OS PID=354313 is still alive ksuitm_check: OS PID=354309 is still alive ksuitm_check: OS PID=354301 is still alive ksuitm_check: OS PID=354298 is still alive ksuitm_check: OS PID=354296 is still alive ksuitm_check: OS PID=354294 is still alive ksuitm_check: OS PID=354292 is still alive ksuitm_check: OS PID=354290 is still alive ksuitm_check: OS PID=354286 is still alive ksuitm_check: OS PID=354284 is still alive ksuitm_check: OS PID=354282 is still alive ksuitm_check: OS PID=354280 is still alive ksuitm_check: OS PID=354278 is still alive ksuitm_check: OS PID=354273 is still alive ksuitm_check: OS PID=354144 is still alive ksuitm_check: OS PID=354140 is still alive ksuitm_check: OS PID=354138 is still alive ksuitm_check: OS PID=354136 is still alive ksuitm_check: OS PID=354134 is still alive ksuitm_check: OS PID=354129 is still alive ksuitm_check: OS PID=354127 is still alive ksuitm_check: OS PID=354125 is still alive ksuitm_check: OS PID=354123 is still alive ksuitm_check: OS PID=354121 is still alive ksuitm_check: OS PID=354119 is still alive ksuitm_check: OS PID=354117 is still alive ksuitm_check: OS PID=354115 is still alive ksuitm_check: OS PID=354113 is still alive ksuitm_check: OS PID=354109 is still alive ksuitm_check: OS PID=354072 is still alive ksuitm_check: OS PID=354060 is still alive ksuitm_check: OS PID=354057 is still alive ksuitm_check: OS PID=354053 is still alive ksuitm_check: OS PID=354046 is still alive ksuitm_check: OS PID=354024 is still alive ksuitm_check: OS PID=354014 is still alive ksuitm_check: OS PID=354010 is still alive ksuitm_check: OS PID=354006 is still alive ksuitm_check: OS PID=354004 is still alive ksuitm_check: OS PID=354002 is still alive ksuitm_check: OS PID=354000 is still alive ksuitm_check: OS PID=353998 is still alive ksuitm_check: OS PID=353996 is still alive ksuitm_check: OS PID=353992 is still alive ksuitm_check: OS PID=353990 is still alive ksuitm_check: OS PID=353988 is still alive ksuitm_check: OS PID=353986 is still alive ksuitm_check: OS PID=353984 is still alive ksuitm_check: OS PID=353895 is still alive ksuitm_check: OS PID=353891 is still alive ksuitm_check: OS PID=353887 is still alive ksuitm_check: OS PID=353883 is still alive ksuitm_check: OS PID=353868 is still alive ksuitm_check: OS PID=353864 is still alive ksuitm_check: OS PID=353854 is still alive ksuitm_check: OS PID=353840 is still alive ksuitm_check: OS PID=353838 is still alive ksuitm_check: OS PID=353820 is still alive ksuitm_check: OS PID=353832 is still alive ksuitm_check: OS PID=353797 is still alive ksuitm_check: OS PID=353795 is still alive ksuitm_check: OS PID=353793 is still alive ksuitm_check: OS PID=353785 is still alive ksuitm_check: OS PID=353783 is still alive ksuitm_check: OS PID=353764 is still alive ksuitm_check: OS PID=353982 is still alive ksuitm_check: OS PID=353470 is still alive ksuitm_check: OS PID=353449 is still alive ksuitm_check: OS PID=353432 is still alive ksuitm_check: OS PID=353399 is still alive ksuitm_check: OS PID=353394 is still alive ksuitm_check: OS PID=353387 is still alive ksuitm_check: OS PID=353385 is still alive ksuitm_check: OS PID=353383 is still alive ksuitm_check: OS PID=353380 is still alive ksuitm_check: OS PID=353368 is still alive ksuitm_check: OS PID=353363 is still alive ksuitm_check: OS PID=353360 is still alive ksuitm_check: OS PID=353358 is still alive ksuitm_check: OS PID=353356 is still alive ksuitm_check: OS PID=353302 is still alive ksuitm_check: OS PID=353300 is still alive ksuitm_check: OS PID=353298 is still alive ksuitm_check: OS PID=353296 is still alive ksuitm_check: OS PID=353294 is still alive ksuitm_check: OS PID=353284 is still alive ksuitm_check: OS PID=353282 is still alive ksuitm_check: OS PID=353278 is still alive ksuitm_check: OS PID=353276 is still alive ksuitm_check: OS PID=353265 is still alive ksuitm_check: OS PID=353263 is still alive ksuitm_check: OS PID=353261 is still alive ksuitm_check: OS PID=353259 is still alive ksuitm_check: OS PID=353257 is still alive ksuitm_check: OS PID=353249 is still alive ksuitm_check: OS PID=353236 is still alive ksuitm_check: OS PID=353234 is still alive ksuitm_check: OS PID=353229 is still alive ksuitm_check: OS PID=353354 is still alive ksuitm_check: OS PID=353352 is still alive ksuitm_check: OS PID=353215 is still alive ksuitm_check: OS PID=360174 is still alive ksuitm_check: OS PID=353350 is still alive ksuitm_check: OS PID=353187 is still alive ksuitm_check: OS PID=353348 is still alive ksuitm_check: OS PID=353346 is still alive ksuitm_check: OS PID=353127 is still alive ksuitm_check: OS PID=353343 is still alive ksuitm_check: OS PID=353201 is still alive ksuitm_check: OS PID=360568 is still alive ksuitm_check: OS PID=353341 is still alive ksuitm_check: OS PID=352330 is still alive ksuitm_check: OS PID=352326 is still alive ksuitm_check: OS PID=352324 is still alive ksuitm_check: OS PID=352320 is still alive ksuitm_check: OS PID=352318 is still alive ksuitm_check: OS PID=352316 is still alive ksuitm_check: OS PID=352232 is still alive ksuitm_check: OS PID=352230 is still alive ksuitm_check: OS PID=352226 is still alive ksuitm_check: OS PID=352216 is still alive ksuitm_check: OS PID=352213 is still alive ksuitm_check: OS PID=352208 is still alive ksuitm_check: OS PID=352194 is still alive ksuitm_check: OS PID=352186 is still alive ksuitm_check: OS PID=352161 is still alive ksuitm_check: OS PID=352163 is still alive ksuitm_check: OS PID=352154 is still alive ksuitm_check: OS PID=352151 is still alive ksuitm_check: OS PID=352147 is still alive ksuitm_check: OS PID=352145 is still alive ksuitm_check: OS PID=352132 is still alive ksuitm_check: OS PID=352124 is still alive ksuitm_check: OS PID=352122 is still alive ksuitm_check: OS PID=352120 is still alive ksuitm_check: OS PID=352116 is still alive ksuitm_check: OS PID=352104 is still alive ksuitm_check: OS PID=352102 is still alive ksuitm_check: OS PID=352092 is still alive ksuitm_check: OS PID=352086 is still alive ksuitm_check: OS PID=352084 is still alive ksuitm_check: OS PID=352082 is still alive ksuitm_check: OS PID=352080 is still alive ksuitm_check: OS PID=352078 is still alive ksuitm_check: OS PID=352076 is still alive ksuitm_check: OS PID=352073 is still alive ksuitm_check: OS PID=352053 is still alive ksuitm_check: OS PID=352049 is still alive ksuitm_check: OS PID=352043 is still alive ksuitm_check: OS PID=352040 is still alive ksuitm_check: OS PID=352034 is still alive ksuitm_check: OS PID=352020 is still alive ksuitm_check: OS PID=352018 is still alive ksuitm_check: OS PID=352014 is still alive ksuitm_check: OS PID=352012 is still alive ksuitm_check: OS PID=352006 is still alive ksuitm_check: OS PID=352002 is still alive ksuitm_check: OS PID=351974 is still alive ksuitm_check: OS PID=351969 is still alive ksuitm_check: OS PID=351967 is still alive ksuitm_check: OS PID=351959 is still alive ksuitm_check: OS PID=351956 is still alive ksuitm_check: OS PID=351952 is still alive ksuitm_check: OS PID=351950 is still alive ksuitm_check: OS PID=351947 is still alive ksuitm_check: OS PID=351945 is still alive ksuitm_check: OS PID=351928 is still alive ksuitm_check: OS PID=351787 is still alive ksuitm_check: OS PID=351686 is still alive ksuitm_check: OS PID=351658 is still alive ksuitm_check: OS PID=351651 is still alive ksuitm_check: OS PID=351647 is still alive ksuitm_check: OS PID=351643 is still alive ksuitm_check: OS PID=351599 is still alive ksuitm_check: OS PID=351601 is still alive ksuitm_check: OS PID=351595 is still alive ksuitm_check: OS PID=351592 is still alive ksuitm_check: OS PID=351588 is still alive ksuitm_check: OS PID=351586 is still alive ksuitm_check: OS PID=351584 is still alive ksuitm_check: OS PID=351582 is still alive ksuitm_check: OS PID=351580 is still alive ksuitm_check: OS PID=351578 is still alive ksuitm_check: OS PID=351576 is still alive ksuitm_check: OS PID=351574 is still alive ksuitm_check: OS PID=351572 is still alive ksuitm_check: OS PID=351566 is still alive ksuitm_check: OS PID=351553 is still alive ksuitm_check: OS PID=351551 is still alive ksuitm_check: OS PID=351543 is still alive ksuitm_check: OS PID=351541 is still alive ksuitm_check: OS PID=351538 is still alive ksuitm_check: OS PID=351533 is still alive ksuitm_check: OS PID=351511 is still alive ksuitm_check: OS PID=351505 is still alive ksuitm_check: OS PID=351496 is still alive ksuitm_check: OS PID=351492 is still alive ksuitm_check: OS PID=351488 is still alive ksuitm_check: OS PID=351482 is still alive ksuitm_check: OS PID=351480 is still alive ksuitm_check: OS PID=351478 is still alive ksuitm_check: OS PID=351476 is still alive ksuitm_check: OS PID=351474 is still alive ksuitm_check: OS PID=351460 is still alive ksuitm_check: OS PID=351446 is still alive ksuitm_check: OS PID=351444 is still alive ksuitm_check: OS PID=351442 is still alive ksuitm_check: OS PID=351440 is still alive ksuitm_check: OS PID=351436 is still alive ksuitm_check: OS PID=351434 is still alive ksuitm_check: OS PID=351361 is still alive ksuitm_check: OS PID=351357 is still alive ksuitm_check: OS PID=351359 is still alive ksuitm_check: OS PID=351215 is still alive ksuitm_check: OS PID=351211 is still alive ksuitm_check: OS PID=351209 is still alive ksuitm_check: OS PID=351207 is still alive ksuitm_check: OS PID=351204 is still alive ksuitm_check: OS PID=351196 is still alive ksuitm_check: OS PID=351191 is still alive ksuitm_check: OS PID=351185 is still alive ksuitm_check: OS PID=351179 is still alive ksuitm_check: OS PID=351173 is still alive ksuitm_check: OS PID=351134 is still alive ksuitm_check: OS PID=351091 is still alive ksuitm_check: OS PID=351076 is still alive ksuitm_check: OS PID=351078 is still alive ksuitm_check: OS PID=351062 is still alive ksuitm_check: OS PID=351045 is still alive ksuitm_check: OS PID=350947 is still alive ksuitm_check: OS PID=350945 is still alive ksuitm_check: OS PID=350943 is still alive ksuitm_check: OS PID=350941 is still alive ksuitm_check: OS PID=350928 is still alive ksuitm_check: OS PID=351217 is still alive ksuitm_check: OS PID=350901 is still alive ksuitm_check: OS PID=350728 is still alive ksuitm_check: OS PID=350726 is still alive ksuitm_check: OS PID=350689 is still alive ksuitm_check: OS PID=350682 is still alive ksuitm_check: OS PID=350687 is still alive ksuitm_check: OS PID=350676 is still alive ksuitm_check: OS PID=350662 is still alive ksuitm_check: OS PID=350658 is still alive ksuitm_check: OS PID=350656 is still alive ksuitm_check: OS PID=350652 is still alive ksuitm_check: OS PID=350641 is still alive ksuitm_check: OS PID=350637 is still alive ksuitm_check: OS PID=350629 is still alive ksuitm_check: OS PID=350626 is still alive ksuitm_check: OS PID=350624 is still alive ksuitm_check: OS PID=350620 is still alive ksuitm_check: OS PID=350618 is still alive ksuitm_check: OS PID=350616 is still alive ksuitm_check: OS PID=350614 is still alive ksuitm_check: OS PID=350600 is still alive ksuitm_check: OS PID=350598 is still alive ksuitm_check: OS PID=350596 is still alive ksuitm_check: OS PID=350594 is still alive ksuitm_check: OS PID=350592 is still alive ksuitm_check: OS PID=350590 is still alive ksuitm_check: OS PID=350588 is still alive ksuitm_check: OS PID=350884 is still alive ksuitm_check: OS PID=350886 is still alive ksuitm_check: OS PID=350878 is still alive ksuitm_check: OS PID=350537 is still alive ksuitm_check: OS PID=350535 is still alive ksuitm_check: OS PID=350868 is still alive ksuitm_check: OS PID=350528 is still alive ksuitm_check: OS PID=350526 is still alive ksuitm_check: OS PID=350522 is still alive ksuitm_check: OS PID=350520 is still alive ksuitm_check: OS PID=350517 is still alive ksuitm_check: OS PID=350515 is still alive ksuitm_check: OS PID=350513 is still alive ksuitm_check: OS PID=350500 is still alive ksuitm_check: OS PID=350496 is still alive ksuitm_check: OS PID=350484 is still alive ksuitm_check: OS PID=350481 is still alive ksuitm_check: OS PID=350477 is still alive ksuitm_check: OS PID=350475 is still alive ksuitm_check: OS PID=350470 is still alive ksuitm_check: OS PID=350468 is still alive ksuitm_check: OS PID=350466 is still alive ksuitm_check: OS PID=350464 is still alive ksuitm_check: OS PID=350462 is still alive ksuitm_check: OS PID=350459 is still alive ksuitm_check: OS PID=350455 is still alive ksuitm_check: OS PID=350453 is still alive ksuitm_check: OS PID=350446 is still alive ksuitm_check: OS PID=350422 is still alive ksuitm_check: OS PID=350420 is still alive ksuitm_check: OS PID=350415 is still alive ksuitm_check: OS PID=350402 is still alive ksuitm_check: OS PID=350391 is still alive ksuitm_check: OS PID=350384 is still alive ksuitm_check: OS PID=350373 is still alive ksuitm_check: OS PID=350369 is still alive ksuitm_check: OS PID=350365 is still alive ksuitm_check: OS PID=350363 is still alive ksuitm_check: OS PID=350361 is still alive ksuitm_check: OS PID=350358 is still alive ksuitm_check: OS PID=350350 is still alive ksuitm_check: OS PID=350329 is still alive ksuitm_check: OS PID=350326 is still alive ksuitm_check: OS PID=350324 is still alive ksuitm_check: OS PID=350316 is still alive ksuitm_check: OS PID=350314 is still alive ksuitm_check: OS PID=350310 is still alive ksuitm_check: OS PID=350308 is still alive ksuitm_check: OS PID=350299 is still alive ksuitm_check: OS PID=350297 is still alive ksuitm_check: OS PID=350293 is still alive ksuitm_check: OS PID=350253 is still alive ksuitm_check: OS PID=350200 is still alive ksuitm_check: OS PID=350108 is still alive ksuitm_check: OS PID=350045 is still alive ksuitm_check: OS PID=349953 is still alive ksuitm_check: OS PID=349907 is still alive ksuitm_check: OS PID=349903 is still alive ksuitm_check: OS PID=349901 is still alive ksuitm_check: OS PID=349898 is still alive ksuitm_check: OS PID=349895 is still alive ksuitm_check: OS PID=349892 is still alive ksuitm_check: OS PID=349889 is still alive ksuitm_check: OS PID=349887 is still alive ksuitm_check: OS PID=349885 is still alive ksuitm_check: OS PID=349875 is still alive ksuitm_check: OS PID=349863 is still alive ksuitm_check: OS PID=349861 is still alive ksuitm_check: OS PID=347105 is still alive ksuitm_check: OS PID=346886 is still alive ksuitm_check: OS PID=346884 is still alive ksuitm_check: OS PID=346881 is still alive ksuitm_check: OS PID=346879 is still alive ksuitm_check: OS PID=346877 is still alive ksuitm_check: OS PID=346873 is still alive ksuitm_check: OS PID=346870 is still alive ksuitm_check: OS PID=346866 is still alive ksuitm_check: OS PID=346858 is still alive ksuitm_check: OS PID=346848 is still alive ksuitm_check: OS PID=346846 is still alive ksuitm_check: OS PID=346844 is still alive ksuitm_check: OS PID=346842 is still alive ksuitm_check: OS PID=346840 is still alive ksuitm_check: OS PID=346825 is still alive ksuitm_check: OS PID=346821 is still alive ksuitm_check: OS PID=346814 is still alive ksuitm_check: OS PID=346783 is still alive ksuitm_check: OS PID=346765 is still alive ksuitm_check: OS PID=346541 is still alive ksuitm_check: OS PID=346539 is still alive ksuitm_check: OS PID=346537 is still alive ksuitm_check: OS PID=346527 is still alive ksuitm_check: OS PID=346522 is still alive ksuitm_check: OS PID=346520 is still alive ksuitm_check: OS PID=346517 is still alive ksuitm_check: OS PID=346512 is still alive ksuitm_check: OS PID=346510 is still alive ksuitm_check: OS PID=346498 is still alive ksuitm_check: OS PID=346494 is still alive ksuitm_check: OS PID=346492 is still alive ksuitm_check: OS PID=346150 is still alive ksuitm_check: OS PID=346148 is still alive ksuitm_check: OS PID=346124 is still alive ksuitm_check: OS PID=346490 is still alive ksuitm_check: OS PID=346488 is still alive ksuitm_check: OS PID=346485 is still alive ksuitm_check: OS PID=346482 is still alive ksuitm_check: OS PID=346479 is still alive ksuitm_check: OS PID=346474 is still alive ksuitm_check: OS PID=346085 is still alive ksuitm_check: OS PID=346470 is still alive ksuitm_check: OS PID=346081 is still alive ksuitm_check: OS PID=346079 is still alive ksuitm_check: OS PID=346468 is still alive ksuitm_check: OS PID=346073 is still alive ksuitm_check: OS PID=346464 is still alive ksuitm_check: OS PID=346068 is still alive ksuitm_check: OS PID=346066 is still alive ksuitm_check: OS PID=346064 is still alive ksuitm_check: OS PID=346058 is still alive ksuitm_check: OS PID=346060 is still alive ksuitm_check: OS PID=346054 is still alive ksuitm_check: OS PID=346051 is still alive ksuitm_check: OS PID=346049 is still alive ksuitm_check: OS PID=346037 is still alive ksuitm_check: OS PID=346035 is still alive ksuitm_check: OS PID=346024 is still alive ksuitm_check: OS PID=346022 is still alive ksuitm_check: OS PID=346019 is still alive ksuitm_check: OS PID=346017 is still alive ksuitm_check: OS PID=346013 is still alive ksuitm_check: OS PID=346007 is still alive ksuitm_check: OS PID=346005 is still alive ksuitm_check: OS PID=346001 is still alive ksuitm_check: OS PID=345987 is still alive ksuitm_check: OS PID=345983 is still alive ksuitm_check: OS PID=345978 is still alive ksuitm_check: OS PID=345973 is still alive ksuitm_check: OS PID=345943 is still alive ksuitm_check: OS PID=345941 is still alive ksuitm_check: OS PID=345937 is still alive ksuitm_check: OS PID=345933 is still alive ksuitm_check: OS PID=345922 is still alive ksuitm_check: OS PID=345919 is still alive ksuitm_check: OS PID=345909 is still alive ksuitm_check: OS PID=345905 is still alive ksuitm_check: OS PID=345899 is still alive ksuitm_check: OS PID=345887 is still alive ksuitm_check: OS PID=345882 is still alive ksuitm_check: OS PID=345872 is still alive ksuitm_check: OS PID=345869 is still alive ksuitm_check: OS PID=345867 is still alive ksuitm_check: OS PID=345865 is still alive ksuitm_check: OS PID=345863 is still alive ksuitm_check: OS PID=345861 is still alive ksuitm_check: OS PID=345859 is still alive ksuitm_check: OS PID=345857 is still alive ksuitm_check: OS PID=345855 is still alive ksuitm_check: OS PID=345851 is still alive ksuitm_check: OS PID=345847 is still alive ksuitm_check: OS PID=345830 is still alive ksuitm_check: OS PID=345833 is still alive ksuitm_check: OS PID=345826 is still alive ksuitm_check: OS PID=345822 is still alive ksuitm_check: OS PID=345820 is still alive ksuitm_check: OS PID=333255 is still alive ksuitm_check: OS PID=344723 is still alive ksuitm_check: OS PID=352192 is still alive ksuitm_check: OS PID=332956 is still alive ksuitm_check: OS PID=332954 is still alive ksuitm_check: OS PID=332951 is still alive ksuitm_check: OS PID=332949 is still alive ksuitm_check: OS PID=351432 is still alive ksuitm_check: OS PID=344694 is still alive ksuitm_check: OS PID=332941 is still alive ksuitm_check: OS PID=349859 is still alive ksuitm_check: OS PID=332894 is still alive ksuitm_check: OS PID=315854 is still alive ksuitm_check: OS PID=332891 is still alive ksuitm_check: OS PID=332889 is still alive ksuitm_check: OS PID=315850 is still alive ksuitm_check: OS PID=332887 is still alive ksuitm_check: OS PID=315848 is still alive ksuitm_check: OS PID=332883 is still alive ksuitm_check: OS PID=315846 is still alive ksuitm_check: OS PID=275447 is still alive ksuitm_check: OS PID=345764 is still alive ksuitm_check: OS PID=320938 is still alive ksuitm_check: OS PID=319582 is still alive ksuitm_check: OS PID=344685 is still alive ksuitm_check: OS PID=332232 is still alive ksuitm_check: OS PID=144360 is still alive ksuitm_check: OS PID=345725 is still alive ksuitm_check: OS PID=338194 is still alive ksuitm_check: OS PID=352190 is still alive ksuitm_check: OS PID=350473 is still alive ksuitm_check: OS PID=332223 is still alive ksuitm_check: OS PID=345633 is still alive ksuitm_check: OS PID=339607 is still alive ksuitm_check: OS PID=345573 is still alive ksuitm_check: OS PID=315013 is still alive ksuitm_check: OS PID=339526 is still alive ksuitm_check: OS PID=339524 is still alive ksuitm_check: OS PID=332584 is still alive ksuitm_check: OS PID=332582 is still alive ksuitm_check: OS PID=345534 is still alive ksuitm_check: OS PID=302786 is still alive ksuitm_check: OS PID=332579 is still alive ksuitm_check: OS PID=302784 is still alive ksuitm_check: OS PID=349857 is still alive ksuitm_check: OS PID=224600 is still alive ksuitm_check: OS PID=251863 is still alive ksuitm_check: OS PID=332577 is still alive ksuitm_check: OS PID=332575 is still alive ksuitm_check: OS PID=332939 is still alive ksuitm_check: OS PID=332937 is still alive ksuitm_check: OS PID=332926 is still alive ksuitm_check: OS PID=332885 is still alive ksuitm_check: OS PID=332876 is still alive ksuitm_check: OS PID=332573 is still alive ksuitm_check: OS PID=344750 is still alive ksuitm_check: OS PID=332868 is still alive ksuitm_check: OS PID=251573 is still alive ksuitm_check: OS PID=301846 is still alive ksuitm_check: OS PID=314954 is still alive ksuitm_check: OS PID=345527 is still alive ksuitm_check: OS PID=333738 is still alive ksuitm_check: OS PID=314908 is still alive ksuitm_check: OS PID=346502 is still alive ksuitm_check: OS PID=333864 is still alive ksuitm_check: OS PID=275371 is still alive ksuitm_check: OS PID=332879 is still alive ksuitm_check: OS PID=339629 is still alive ksuitm_check: OS PID=344589 is still alive ksuitm_check: OS PID=344587 is still alive ksuitm_check: OS PID=354356 is still alive ksuitm_check: OS PID=338834 is still alive ksuitm_check: OS PID=349855 is still alive ksuitm_check: OS PID=349853 is still alive ksuitm_check: OS PID=349851 is still alive ksuitm_check: OS PID=332567 is still alive ksuitm_check: OS PID=71738 is still alive ksuitm_check: OS PID=333291 is still alive ksuitm_check: OS PID=337373 is still alive ksuitm_check: OS PID=307154 is still alive ksuitm_check: OS PID=345418 is still alive ksuitm_check: OS PID=113607 is still alive ksuitm_check: OS PID=238213 is still alive ksuitm_check: OS PID=332922 is still alive ksuitm_check: OS PID=332918 is still alive ksuitm_check: OS PID=332916 is still alive ksuitm_check: OS PID=332914 is still alive ksuitm_check: OS PID=122743 is still alive ksuitm_check: OS PID=346773 is still alive ksuitm_check: OS PID=334408 is still alive ksuitm_check: OS PID=233245 is still alive ksuitm_check: OS PID=332852 is still alive ksuitm_check: OS PID=71162 is still alive ksuitm_check: OS PID=333057 is still alive ksuitm_check: OS PID=269538 is still alive ksuitm_check: OS PID=332563 is still alive ksuitm_check: OS PID=349849 is still alive ksuitm_check: OS PID=267137 is still alive ksuitm_check: OS PID=276391 is still alive ksuitm_check: OS PID=332466 is still alive ksuitm_check: OS PID=333068 is still alive ksuitm_check: OS PID=353391 is still alive ksuitm_check: OS PID=349847 is still alive ksuitm_check: OS PID=353389 is still alive ksuitm_check: OS PID=345394 is still alive ksuitm_check: OS PID=353191 is still alive ksuitm_check: OS PID=345361 is still alive ksuitm_check: OS PID=338742 is still alive ksuitm_check: OS PID=123530 is still alive ksuitm_check: OS PID=350932 is still alive ksuitm_check: OS PID=296623 is still alive ksuitm_check: OS PID=338950 is still alive ksuitm_check: OS PID=338740 is still alive ksuitm_check: OS PID=338682 is still alive ksuitm_check: OS PID=338678 is still alive ksuitm_check: OS PID=338641 is still alive ksuitm_check: OS PID=359695 is still alive ksuitm_check: OS PID=338634 is still alive ksuitm_check: OS PID=388592 is still alive ksuitm_check: OS PID=338592 is still alive ksuitm_check: OS PID=388582 is still alive ksuitm_check: OS PID=151346 is still alive ksuitm_check: OS PID=137861 is still alive ksuitm_check: OS PID=307033 is still alive ksuitm_check: OS PID=388569 is still alive ksuitm_check: OS PID=239496 is still alive ksuitm_check: OS PID=334239 is still alive ksuitm_check: OS PID=354402 is still alive ksuitm_check: OS PID=334237 is still alive ksuitm_check: OS PID=354434 is still alive ksuitm_check: OS PID=388417 is still alive ksuitm_check: OS PID=339465 is still alive ksuitm_check: OS PID=288210 is still alive ksuitm_check: OS PID=353461 is still alive ksuitm_check: OS PID=354400 is still alive ksuitm_check: OS PID=344979 is still alive ksuitm_check: OS PID=320975 is still alive ksuitm_check: OS PID=354251 is still alive ksuitm_check: OS PID=42629 is still alive ksuitm_check: OS PID=332908 is still alive ksuitm_check: OS PID=354245 is still alive ksuitm_check: OS PID=353366 is still alive ksuitm_check: OS PID=287807 is still alive ksuitm_check: OS PID=360565 is still alive ksuitm_check: OS PID=354234 is still alive ksuitm_check: OS PID=354232 is still alive ksuitm_check: OS PID=269826 is still alive ksuitm_check: OS PID=354230 is still alive ksuitm_check: OS PID=319943 is still alive ksuitm_check: OS PID=357639 is still alive ksuitm_check: OS PID=354220 is still alive ksuitm_check: OS PID=354218 is still alive ksuitm_check: OS PID=320791 is still alive ksuitm_check: OS PID=354213 is still alive ksuitm_check: OS PID=42621 is still alive ksuitm_check: OS PID=354211 is still alive ksuitm_check: OS PID=354209 is still alive ksuitm_check: OS PID=354206 is still alive ksuitm_check: OS PID=358080 is still alive ksuitm_check: OS PID=354204 is still alive ksuitm_check: OS PID=354962 is still alive ksuitm_check: OS PID=354202 is still alive ksuitm_check: OS PID=332958 is still alive ksuitm_check: OS PID=347015 is still alive ksuitm_check: OS PID=354200 is still alive ksuitm_check: OS PID=354196 is still alive ksuitm_check: OS PID=349845 is still alive ksuitm_check: OS PID=350874 is still alive ksuitm_check: OS PID=346995 is still alive ksuitm_check: OS PID=341341 is still alive ksuitm_check: OS PID=312624 is still alive ksuitm_check: OS PID=8736 is still alive ksuitm_check: OS PID=282610 is still alive ksuitm_check: OS PID=72140 is still alive ksuitm_check: OS PID=346983 is still alive ksuitm_check: OS PID=361123 is still alive ksuitm_check: OS PID=177937 is still alive ksuitm_check: OS PID=320813 is still alive ksuitm_check: OS PID=387851 is still alive ksuitm_check: OS PID=339848 is still alive ksuitm_check: OS PID=339158 is still alive ksuitm_check: OS PID=346977 is still alive ksuitm_check: OS PID=346971 is still alive ksuitm_check: OS PID=345411 is still alive ksuitm_check: OS PID=286909 is still alive ksuitm_check: OS PID=346969 is still alive ksuitm_check: OS PID=353339 is still alive ksuitm_check: OS PID=346955 is still alive ksuitm_check: OS PID=346953 is still alive ksuitm_check: OS PID=314906 is still alive ksuitm_check: OS PID=346946 is still alive ksuitm_check: OS PID=350926 is still alive ksuitm_check: OS PID=346942 is still alive ksuitm_check: OS PID=346930 is still alive ksuitm_check: OS PID=346922 is still alive ksuitm_check: OS PID=298085 is still alive ksuitm_check: OS PID=346912 is still alive ksuitm_check: OS PID=350864 is still alive ksuitm_check: OS PID=346910 is still alive ksuitm_check: OS PID=294772 is still alive ksuitm_check: OS PID=346906 is still alive ksuitm_check: OS PID=339811 is still alive ksuitm_check: OS PID=361103 is still alive ksuitm_check: OS PID=42544 is still alive ksuitm_check: OS PID=344439 is still alive ksuitm_check: OS PID=353994 is still alive ksuitm_check: OS PID=345311 is still alive ksuitm_check: OS PID=346908 is still alive ksuitm_check: OS PID=338566 is still alive ksuitm_check: OS PID=162947 is still alive ksuitm_check: OS PID=327984 is still alive ksuitm_check: OS PID=338326 is still alive ksuitm_check: OS PID=185758 is still alive ksuitm_check: OS PID=370823 is still alive ksuitm_check: OS PID=370999 is still alive ksuitm_check: OS PID=346904 is still alive ksuitm_check: OS PID=346966 is still alive ksuitm_check: OS PID=185122 is still alive ksuitm_check: OS PID=155830 is still alive ksuitm_check: OS PID=374282 is still alive ksuitm_check: OS PID=339805 is still alive ksuitm_check: OS PID=161123 is still alive ksuitm_check: OS PID=364922 is still alive ksuitm_check: OS PID=345849 is still alive ksuitm_check: OS PID=287179 is still alive ksuitm_check: OS PID=334172 is still alive ksuitm_check: OS PID=393386 is still alive ksuitm_check: OS PID=353280 is still alive ksuitm_check: OS PID=352761 is still alive ksuitm_check: OS PID=322530 is still alive ksuitm_check: OS PID=58549 is still alive ksuitm_check: OS PID=375178 is still alive ksuitm_check: OS PID=375174 is still alive ksuitm_check: OS PID=375170 is still alive ksuitm_check: OS PID=375163 is still alive ksuitm_check: OS PID=375143 is still alive ksuitm_check: OS PID=345308 is still alive ksuitm_check: OS PID=319293 is still alive ksuitm_check: OS PID=264429 is still alive ksuitm_check: OS PID=393586 is still alive ksuitm_check: OS PID=375176 is still alive ksuitm_check: OS PID=375172 is still alive ksuitm_check: OS PID=375166 is still alive ksuitm_check: OS PID=375087 is still alive ksuitm_check: OS PID=375161 is still alive ksuitm_check: OS PID=375150 is still alive ksuitm_check: OS PID=338579 is still alive ksuitm_check: OS PID=375052 is still alive ksuitm_check: OS PID=375031 is still alive ksuitm_check: OS PID=374966 is still alive ksuitm_check: OS PID=374962 is still alive ksuitm_check: OS PID=374958 is still alive ksuitm_check: OS PID=374954 is still alive ksuitm_check: OS PID=350312 is still alive ksuitm_check: OS PID=374950 is still alive ksuitm_check: OS PID=374946 is still alive ksuitm_check: OS PID=375057 is still alive ksuitm_check: OS PID=374938 is still alive ksuitm_check: OS PID=375037 is still alive ksuitm_check: OS PID=374924 is still alive ksuitm_check: OS PID=374908 is still alive ksuitm_check: OS PID=374902 is still alive ksuitm_check: OS PID=374898 is still alive ksuitm_check: OS PID=374956 is still alive ksuitm_check: OS PID=374952 is still alive ksuitm_check: OS PID=374948 is still alive ksuitm_check: OS PID=157396 is still alive ksuitm_check: OS PID=374884 is still alive ksuitm_check: OS PID=374873 is still alive ksuitm_check: OS PID=374932 is still alive ksuitm_check: OS PID=374861 is still alive ksuitm_check: OS PID=374920 is still alive ksuitm_check: OS PID=374841 is still alive ksuitm_check: OS PID=374930 is still alive ksuitm_check: OS PID=374773 is still alive ksuitm_check: OS PID=374904 is still alive ksuitm_check: OS PID=374737 is still alive ksuitm_check: OS PID=374877 is still alive ksuitm_check: OS PID=374900 is still alive ksuitm_check: OS PID=350827 is still alive ksuitm_check: OS PID=374896 is still alive ksuitm_check: OS PID=374720 is still alive ksuitm_check: OS PID=374869 is still alive ksuitm_check: OS PID=374702 is still alive ksuitm_check: OS PID=374935 is still alive ksuitm_check: OS PID=374670 is still alive ksuitm_check: OS PID=374847 is still alive ksuitm_check: OS PID=374660 is still alive ksuitm_check: OS PID=374837 is still alive ksuitm_check: OS PID=374769 is still alive ksuitm_check: OS PID=374607 is still alive ksuitm_check: OS PID=374597 is still alive ksuitm_check: OS PID=374728 is still alive ksuitm_check: OS PID=354288 is still alive ksuitm_check: OS PID=374940 is still alive ksuitm_check: OS PID=374680 is still alive ksuitm_check: OS PID=374581 is still alive ksuitm_check: OS PID=374666 is still alive ksuitm_check: OS PID=374706 is still alive ksuitm_check: OS PID=360595 is still alive ksuitm_check: OS PID=374626 is still alive ksuitm_check: OS PID=374145 is still alive ksuitm_check: OS PID=374543 is still alive ksuitm_check: OS PID=374603 is still alive ksuitm_check: OS PID=374127 is still alive ksuitm_check: OS PID=373950 is still alive ksuitm_check: OS PID=374587 is still alive ksuitm_check: OS PID=345285 is still alive ksuitm_check: OS PID=374550 is still alive ksuitm_check: OS PID=373569 is still alive ksuitm_check: OS PID=374152 is still alive ksuitm_check: OS PID=319291 is still alive ksuitm_check: OS PID=374138 is still alive ksuitm_check: OS PID=373389 is still alive ksuitm_check: OS PID=374109 is still alive ksuitm_check: OS PID=314658 is still alive ksuitm_check: OS PID=166107 is still alive ksuitm_check: OS PID=150730 is still alive ksuitm_check: OS PID=328905 is still alive ksuitm_check: OS PID=373610 is still alive ksuitm_check: OS PID=334169 is still alive ksuitm_check: OS PID=373362 is still alive ksuitm_check: OS PID=325956 is still alive ksuitm_check: OS PID=238639 is still alive ksuitm_check: OS PID=373284 is still alive ksuitm_check: OS PID=373328 is still alive ksuitm_check: OS PID=373279 is still alive ksuitm_check: OS PID=373304 is still alive ksuitm_check: OS PID=373237 is still alive ksuitm_check: OS PID=320150 is still alive ksuitm_check: OS PID=373229 is still alive ksuitm_check: OS PID=373235 is still alive ksuitm_check: OS PID=373212 is still alive ksuitm_check: OS PID=373221 is still alive ksuitm_check: OS PID=151144 is still alive ksuitm_check: OS PID=373205 is still alive ksuitm_check: OS PID=373203 is still alive ksuitm_check: OS PID=373197 is still alive ksuitm_check: OS PID=310281 is still alive ksuitm_check: OS PID=349843 is still alive ksuitm_check: OS PID=373191 is still alive ksuitm_check: OS PID=373181 is still alive ksuitm_check: OS PID=373187 is still alive ksuitm_check: OS PID=373179 is still alive ksuitm_check: OS PID=373173 is still alive ksuitm_check: OS PID=373165 is still alive ksuitm_check: OS PID=373158 is still alive ksuitm_check: OS PID=373094 is still alive ksuitm_check: OS PID=373075 is still alive ksuitm_check: OS PID=373073 is still alive ksuitm_check: OS PID=372988 is still alive ksuitm_check: OS PID=373404 is still alive ksuitm_check: OS PID=360588 is still alive ksuitm_check: OS PID=372783 is still alive ksuitm_check: OS PID=372771 is still alive ksuitm_check: OS PID=372765 is still alive ksuitm_check: OS PID=372753 is still alive ksuitm_check: OS PID=364532 is still alive ksuitm_check: OS PID=372728 is still alive ksuitm_check: OS PID=372760 is still alive ksuitm_check: OS PID=364081 is still alive ksuitm_check: OS PID=372657 is still alive ksuitm_check: OS PID=334001 is still alive ksuitm_check: OS PID=372747 is still alive ksuitm_check: OS PID=373326 is still alive ksuitm_check: OS PID=372745 is still alive ksuitm_check: OS PID=372651 is still alive ksuitm_check: OS PID=39417 is still alive ksuitm_check: OS PID=372649 is still alive ksuitm_check: OS PID=372706 is still alive ksuitm_check: OS PID=372647 is still alive ksuitm_check: OS PID=162767 is still alive ksuitm_check: OS PID=372635 is still alive ksuitm_check: OS PID=372631 is still alive ksuitm_check: OS PID=372625 is still alive ksuitm_check: OS PID=157095 is still alive ksuitm_check: OS PID=372509 is still alive ksuitm_check: OS PID=374890 is still alive ksuitm_check: OS PID=371889 is still alive ksuitm_check: OS PID=353972 is still alive ksuitm_check: OS PID=302701 is still alive ksuitm_check: OS PID=369129 is still alive ksuitm_check: OS PID=369100 is still alive ksuitm_check: OS PID=369119 is still alive ksuitm_check: OS PID=369096 is still alive ksuitm_check: OS PID=369114 is still alive ksuitm_check: OS PID=369088 is still alive ksuitm_check: OS PID=369092 is still alive ksuitm_check: OS PID=369058 is still alive ksuitm_check: OS PID=369060 is still alive ksuitm_check: OS PID=328865 is still alive ksuitm_check: OS PID=369040 is still alive ksuitm_check: OS PID=369036 is still alive ksuitm_check: OS PID=369028 is still alive ksuitm_check: OS PID=368991 is still alive ksuitm_check: OS PID=368977 is still alive ksuitm_check: OS PID=368969 is still alive ksuitm_check: OS PID=368963 is still alive ksuitm_check: OS PID=368981 is still alive ksuitm_check: OS PID=368972 is still alive ksuitm_check: OS PID=368965 is still alive ksuitm_check: OS PID=368943 is still alive ksuitm_check: OS PID=368946 is still alive ksuitm_check: OS PID=368934 is still alive ksuitm_check: OS PID=339474 is still alive ksuitm_check: OS PID=368915 is still alive ksuitm_check: OS PID=368909 is still alive ksuitm_check: OS PID=368886 is still alive Warning: 686 processes are still attach to shmid 130482192: *** 2019-11-15T14:52:25.668067+04:00 (size: 2097152 bytes, creator pid: 368747, last attach/detach pid: 366515) *** 2019-11-15T14:52:25.668095+04:00 *********************** Dumping ipcs output ******************** *** 2019-11-15T14:52:25.675549+04:00 ------ Shared Memory Segments -------- *** 2019-11-15T14:52:25.675589+04:00 key shmid owner perms bytes nattch status *** 2019-11-15T14:52:25.675606+04:00 0x00000000 0 root 644 64 2 dest *** 2019-11-15T14:52:25.675620+04:00 0x00000000 32769 root 644 16384 2 dest *** 2019-11-15T14:52:25.675633+04:00 0x00000000 65538 root 644 280 2 dest *** 2019-11-15T14:52:25.675650+04:00 0x00000000 98307 root 644 80 2 *** 2019-11-15T14:52:25.675663+04:00 0x00000000 131076 root 644 16384 2 *** 2019-11-15T14:52:25.675677+04:00 0x00000000 163845 root 644 280 2 *** 2019-11-15T14:52:25.675690+04:00 0x00000000 1545764870 oracle 600 10485760 47 *** 2019-11-15T14:52:25.675704+04:00 0x00000000 1545797639 oracle 600 3204448256 47 *** 2019-11-15T14:52:25.675717+04:00 0xec5fc108 1545830408 oracle 600 16384 47 *** 2019-11-15T14:52:25.675731+04:00 0x00000000 1545863177 oracle 600 4096 0 *** 2019-11-15T14:52:25.675747+04:00 0x00000000 1545895946 oracle 600 4096 0 *** 2019-11-15T14:52:25.675760+04:00 0xef74da1c 1545928715 oracle 600 24576 31 *** 2019-11-15T14:52:25.675774+04:00 0x00000000 130351116 oracle 600 31457280 686 dest *** 2019-11-15T14:52:25.675787+04:00 0x00000000 130383885 oracle 600 85899345920 1372 dest *** 2019-11-15T14:52:25.675801+04:00 0x00000000 130416654 oracle 600 590021132288 3437 dest *** 2019-11-15T14:52:25.675814+04:00 0x00000000 130449423 oracle 600 278921216 686 dest *** 2019-11-15T14:52:25.675827+04:00 0x00000000 130482192 oracle 600 2097152 686 dest *** 2019-11-15T14:52:25.675844+04:00 ------ Semaphore Arrays -------- *** 2019-11-15T14:52:25.675857+04:00 key semid owner perms nsems *** 2019-11-15T14:52:25.675871+04:00 0x61001b17 98306 root 666 1 *** 2019-11-15T14:52:25.675884+04:00 0x00000000 163844 root 666 3 *** 2019-11-15T14:52:25.675897+04:00 0x00000000 1769477 root 666 3 *** 2019-11-15T14:52:25.675911+04:00 0xcfe88130 284229638 oracle 600 514 *** 2019-11-15T14:52:25.675924+04:00 0xcfe88131 284262407 oracle 600 514 *** 2019-11-15T14:52:25.675940+04:00 0xcfe88132 284295176 oracle 600 514 *** 2019-11-15T14:52:25.675954+04:00 0x670d69c8 284426249 oracle 600 124 *** 2019-11-15T14:52:25.675967+04:00 0x00000000 4096019 root 666 3 *** 2019-11-15T14:52:25.675980+04:00 0x00000000 5701652 root 666 3 *** 2019-11-15T14:52:25.675993+04:00 0x00000000 7307285 root 666 3 *** 2019-11-15T14:52:25.676007+04:00 0x00000000 8912918 root 666 3 *** 2019-11-15T14:52:25.676020+04:00 0x00000000 10518551 root 666 3 *** 2019-11-15T14:52:25.676036+04:00 0x00000000 12124184 root 666 3 *** 2019-11-15T14:52:25.676050+04:00 0x00000000 13729817 root 666 3 *** 2019-11-15T14:52:25.676063+04:00 0x00000000 15335450 root 666 3 *** 2019-11-15T14:52:25.676076+04:00 0x00000000 16941083 root 666 3 *** 2019-11-15T14:52:25.676090+04:00 0x00000000 18546716 root 666 3 *** 2019-11-15T14:52:25.676103+04:00 0x00000000 20152349 root 666 3 *** 2019-11-15T14:52:25.676121+04:00 0x00000000 21757982 root 666 3 *** 2019-11-15T14:52:25.676137+04:00 0x00000000 23363615 root 666 3 *** 2019-11-15T14:52:25.676151+04:00 0x00000000 24969248 root 666 3 *** 2019-11-15T14:52:25.676164+04:00 0x00000000 26574881 root 666 3 *** 2019-11-15T14:52:25.676177+04:00 0x00000000 31326242 root 666 3 *** 2019-11-15T14:52:25.676190+04:00 0x00000000 32931875 root 666 3 *** 2019-11-15T14:52:25.676204+04:00 0x00000000 34537508 root 666 3 *** 2019-11-15T14:52:25.676217+04:00 0x00000000 36143141 root 666 3 *** 2019-11-15T14:52:25.676233+04:00 0x00000000 37748774 root 666 3 *** 2019-11-15T14:52:25.676246+04:00 0x00000000 39354407 root 666 3 *** 2019-11-15T14:52:25.676259+04:00 0x00000000 41353256 root 666 3 *** 2019-11-15T14:52:25.676273+04:00 0x00000000 42958889 root 666 3 *** 2019-11-15T14:52:25.676286+04:00 0x00000000 44564522 root 666 3 *** 2019-11-15T14:52:25.676299+04:00 0x00000000 46170155 root 666 3 *** 2019-11-15T14:52:25.676312+04:00 0x00000000 47775788 root 666 3 *** 2019-11-15T14:52:25.676328+04:00 0x00000000 49381421 root 666 3 *** 2019-11-15T14:52:25.676342+04:00 0x00000000 50987054 root 666 3 *** 2019-11-15T14:52:25.676355+04:00 0x00000000 52592687 root 666 3 *** 2019-11-15T14:52:25.676368+04:00 0x00000000 54198320 root 666 3 *** 2019-11-15T14:52:25.676381+04:00 0x00000000 55803953 root 666 3 *** 2019-11-15T14:52:25.676395+04:00 0x00000000 57409586 root 666 3 *** 2019-11-15T14:52:25.676408+04:00 0x00000000 59015219 root 666 3 *** 2019-11-15T14:52:25.676424+04:00 0x00000000 60620852 root 666 3 *** 2019-11-15T14:52:25.676437+04:00 0x00000000 62226485 root 666 3 *** 2019-11-15T14:52:25.676451+04:00 0x00000000 63832118 root 666 3 *** 2019-11-15T14:52:25.676464+04:00 0x00000000 65830967 root 666 3 *** 2019-11-15T14:52:25.676477+04:00 0x00000000 67436600 root 666 3 *** 2019-11-15T14:52:25.676493+04:00 0x00000000 69042233 root 666 3 *** 2019-11-15T14:52:25.676506+04:00 0x00000000 70647866 root 666 3 *** 2019-11-15T14:52:25.676522+04:00 0x00000000 72253499 root 666 3 *** 2019-11-15T14:52:25.676535+04:00 0x00000000 73859132 root 666 3 *** 2019-11-15T14:52:25.676549+04:00 0x00000000 75464765 root 666 3 *** 2019-11-15T14:52:25.676562+04:00 0x00000000 77070398 root 666 3 *** 2019-11-15T14:52:25.676575+04:00 0x00000000 78676031 root 666 3 *** 2019-11-15T14:52:25.676593+04:00 0x00000000 80281664 root 666 3 *** 2019-11-15T14:52:25.676607+04:00 0x00000000 81887297 root 666 3 *** 2019-11-15T14:52:25.676622+04:00 0x00000000 83492930 root 666 3 *** 2019-11-15T14:52:25.676636+04:00 0x00000000 85098563 root 666 3 *** 2019-11-15T14:52:25.676649+04:00 0x00000000 87097412 root 666 3 *** 2019-11-15T14:52:25.676662+04:00 0x00000000 88703045 root 666 3 *** 2019-11-15T14:52:25.676675+04:00 0x00000000 90308678 root 666 3 *** 2019-11-15T14:52:25.676689+04:00 0x00000000 92307527 root 666 3 *** 2019-11-15T14:52:25.676702+04:00 0x00000000 93913160 root 666 3 *** 2019-11-15T14:52:25.676718+04:00 0x00000000 95518793 root 666 3 *** 2019-11-15T14:52:25.676734+04:00 0x00000000 97124426 root 666 3 *** 2019-11-15T14:52:25.676748+04:00 0x00000000 98762827 root 666 3 *** 2019-11-15T14:52:25.676761+04:00 0x00000000 100368460 root 666 3 *** 2019-11-15T14:52:25.676774+04:00 0x00000000 101974093 root 666 3 *** 2019-11-15T14:52:25.676788+04:00 0x00000000 103579726 root 666 3 *** 2019-11-15T14:52:25.676801+04:00 0x00000000 105185359 root 666 3 *** 2019-11-15T14:52:25.676817+04:00 0x00000000 106790992 root 666 3 *** 2019-11-15T14:52:25.676830+04:00 0x00000000 108396625 root 666 3 *** 2019-11-15T14:52:25.676844+04:00 0x00000000 110002258 root 666 3 *** 2019-11-15T14:52:25.676857+04:00 0x00000000 111607891 root 666 3 *** 2019-11-15T14:52:25.676870+04:00 0x00000000 113213524 root 666 3 *** 2019-11-15T14:52:25.676884+04:00 0x00000000 114819157 root 666 3 *** 2019-11-15T14:52:25.676897+04:00 0x00000000 116424790 root 666 3 *** 2019-11-15T14:52:25.676913+04:00 0x00000000 118030423 root 666 3 *** 2019-11-15T14:52:25.676926+04:00 0x00000000 119636056 root 666 3 *** 2019-11-15T14:52:25.676939+04:00 0x00000000 121241689 root 666 3 *** 2019-11-15T14:52:25.676953+04:00 0x00000000 122847322 root 666 3 *** 2019-11-15T14:52:25.676966+04:00 0x00000000 124452955 root 666 3 *** 2019-11-15T14:52:25.676979+04:00 0x00000000 126058588 root 666 3 *** 2019-11-15T14:52:25.676993+04:00 0x00000000 127664221 root 666 3 *** 2019-11-15T14:52:25.677009+04:00 0x00000000 129269854 root 666 3 *** 2019-11-15T14:52:25.677022+04:00 0x00000000 130875487 root 666 3 *** 2019-11-15T14:52:25.677035+04:00 0x00000000 132481120 root 666 3 *** 2019-11-15T14:52:25.677049+04:00 0x00000000 134086753 root 666 3 *** 2019-11-15T14:52:25.677062+04:00 0x00000000 135692386 root 666 3 *** 2019-11-15T14:52:25.677075+04:00 0x00000000 137298019 root 666 3 *** 2019-11-15T14:52:25.677089+04:00 0x00000000 138903652 root 666 3 *** 2019-11-15T14:52:25.677105+04:00 0x00000000 140509285 root 666 3 *** 2019-11-15T14:52:25.677118+04:00 0x00000000 142114918 root 666 3 *** 2019-11-15T14:52:25.677131+04:00 0x00000000 143720551 root 666 3 *** 2019-11-15T14:52:25.677144+04:00 0x00000000 145326184 root 666 3 *** 2019-11-15T14:52:25.677158+04:00 0x00000000 146931817 root 666 3 *** 2019-11-15T14:52:25.677171+04:00 0x00000000 148537450 root 666 3 *** 2019-11-15T14:52:25.677184+04:00 0x00000000 150143083 root 666 3 *** 2019-11-15T14:52:25.677200+04:00 0x00000000 151748716 root 666 3 *** 2019-11-15T14:52:25.677214+04:00 0x00000000 153354349 root 666 3 *** 2019-11-15T14:52:25.677227+04:00 0x00000000 154959982 root 666 3 *** 2019-11-15T14:52:25.677240+04:00 0x00000000 156565615 root 666 3 *** 2019-11-15T14:52:25.677254+04:00 0x00000000 158171248 root 666 3 *** 2019-11-15T14:52:25.677267+04:00 0x00000000 159776881 root 666 3 *** 2019-11-15T14:52:25.677280+04:00 0x00000000 161382514 root 666 3 *** 2019-11-15T14:52:25.677296+04:00 0x00000000 162988147 root 666 3 *** 2019-11-15T14:52:25.677310+04:00 0x00000000 164593780 root 666 3 *** 2019-11-15T14:52:25.677323+04:00 0x00000000 166199413 root 666 3 *** 2019-11-15T14:52:25.677339+04:00 0x00000000 167805046 root 666 3 *** 2019-11-15T14:52:25.677353+04:00 0x00000000 169410679 root 666 3 *** 2019-11-15T14:52:25.677366+04:00 0x00000000 171737208 root 666 3 *** 2019-11-15T14:52:25.677379+04:00 0x00000000 173342841 root 666 3 *** 2019-11-15T14:52:25.677395+04:00 0x00000000 174948474 root 666 3 *** 2019-11-15T14:52:25.677408+04:00 0x00000000 176554107 root 666 3 *** 2019-11-15T14:52:25.677422+04:00 0x00000000 178159740 root 666 3 *** 2019-11-15T14:52:25.677435+04:00 0x00000000 179765373 root 666 3 *** 2019-11-15T14:52:25.677448+04:00 0x00000000 181371006 root 666 3 *** 2019-11-15T14:52:25.677462+04:00 0x00000000 182976639 root 666 3 *** 2019-11-15T14:52:25.677475+04:00 0x00000000 184582272 root 666 3 *** 2019-11-15T14:52:25.677491+04:00 0x00000000 186187905 root 666 3 *** 2019-11-15T14:52:25.677504+04:00 0x00000000 187793538 root 666 3 *** 2019-11-15T14:52:25.677517+04:00 0x00000000 189399171 root 666 3 *** 2019-11-15T14:52:25.677533+04:00 0x00000000 191004804 root 666 3 *** 2019-11-15T14:52:25.677546+04:00 0x00000000 192610437 root 666 3 *** 2019-11-15T14:52:25.677559+04:00 0x00000000 194216070 root 666 3 *** 2019-11-15T14:52:25.677573+04:00 0x00000000 195821703 root 666 3 *** 2019-11-15T14:52:25.677592+04:00 0x00000000 197427336 root 666 3 *** 2019-11-15T14:52:25.677606+04:00 0x00000000 199032969 root 666 3 *** 2019-11-15T14:52:25.677619+04:00 0x00000000 200638602 root 666 3 *** 2019-11-15T14:52:25.677632+04:00 0x00000000 202244235 root 666 3 *** 2019-11-15T14:52:25.677645+04:00 0x00000000 203849868 root 666 3 *** 2019-11-15T14:52:25.677659+04:00 0x00000000 205455501 root 666 3 *** 2019-11-15T14:52:25.677672+04:00 0x00000000 207061134 root 666 3 *** 2019-11-15T14:52:25.677688+04:00 0x00000000 208666767 root 666 3 *** 2019-11-15T14:52:25.677701+04:00 0x00000000 210272400 root 666 3 *** 2019-11-15T14:52:25.677715+04:00 0x00000000 211878033 root 666 3 *** 2019-11-15T14:52:25.677728+04:00 0x00000000 213483666 root 666 3 *** 2019-11-15T14:52:25.677741+04:00 0x00000000 215089299 root 666 3 *** 2019-11-15T14:52:25.677755+04:00 0x00000000 216694932 root 666 3 *** 2019-11-15T14:52:25.677768+04:00 0x00000000 218300565 root 666 3 *** 2019-11-15T14:52:25.677784+04:00 0x00000000 219906198 root 666 3 *** 2019-11-15T14:52:25.677797+04:00 0x00000000 221511831 root 666 3 *** 2019-11-15T14:52:25.677810+04:00 0x00000000 223117464 root 666 3 *** 2019-11-15T14:52:25.677824+04:00 0x00000000 224723097 root 666 3 *** 2019-11-15T14:52:25.677837+04:00 0x00000000 226328730 root 666 3 *** 2019-11-15T14:52:25.677850+04:00 0x00000000 227934363 root 666 3 *** 2019-11-15T14:52:25.677864+04:00 0x00000000 229539996 root 666 3 *** 2019-11-15T14:52:25.677880+04:00 0x00000000 231145629 root 666 3 *** 2019-11-15T14:52:25.677893+04:00 0x00000000 232751262 root 666 3 *** 2019-11-15T14:52:25.677906+04:00 0x00000000 234356895 root 666 3 *** 2019-11-15T14:52:25.677920+04:00 0x00000000 235962528 root 666 3 *** 2019-11-15T14:52:25.677933+04:00 0x00000000 237568161 root 666 3 *** 2019-11-15T14:52:25.677950+04:00 0x00000000 239173794 root 666 3 *** 2019-11-15T14:52:25.677964+04:00 0x00000000 240779427 root 666 3 *** 2019-11-15T14:52:25.677980+04:00 0x00000000 242385060 root 666 3 *** 2019-11-15T14:52:25.677993+04:00 0x00000000 243990693 root 666 3 *** 2019-11-15T14:52:25.678007+04:00 0x00000000 245596326 root 666 3 *** 2019-11-15T14:52:25.678020+04:00 0x00000000 247201959 root 666 3 *** 2019-11-15T14:52:25.678033+04:00 0x00000000 248807592 root 666 3 *** 2019-11-15T14:52:25.678047+04:00 0x00000000 250413225 root 666 3 *** 2019-11-15T14:52:25.678061+04:00 0x00000000 252018858 root 666 3 *** 2019-11-15T14:52:25.678077+04:00 0x00000000 253624491 root 666 3 *** 2019-11-15T14:52:25.678090+04:00 0x00000000 255230124 root 666 3 *** 2019-11-15T14:52:25.678104+04:00 0x00000000 256835757 root 666 3 *** 2019-11-15T14:52:25.678117+04:00 0x00000000 258441390 root 666 3 *** 2019-11-15T14:52:25.678130+04:00 0x00000000 260047023 root 666 3 *** 2019-11-15T14:52:25.678144+04:00 0x00000000 261652656 root 666 3 *** 2019-11-15T14:52:25.678157+04:00 0x00000000 263258289 root 666 3 *** 2019-11-15T14:52:25.678173+04:00 0x00000000 264863922 root 666 3 *** 2019-11-15T14:52:25.678186+04:00 0x00000000 266469555 root 666 3 *** 2019-11-15T14:52:25.678199+04:00 0x00000000 268075188 root 666 3 *** 2019-11-15T14:52:25.678213+04:00 0x00000000 269680821 root 666 3 *** 2019-11-15T14:52:25.678226+04:00 0x00000000 271286454 root 666 3 *** 2019-11-15T14:52:25.678240+04:00 0x00000000 272892087 root 666 3 *** 2019-11-15T14:52:25.678253+04:00 0x00000000 274497720 root 666 3 *** 2019-11-15T14:52:25.678269+04:00 0x00000000 276103353 root 666 3 *** 2019-11-15T14:52:25.678282+04:00 0x00000000 277708986 root 666 3 *** 2019-11-15T14:52:25.678295+04:00 0x00000000 279314619 root 666 3 *** 2019-11-15T14:52:25.678309+04:00 0x00000000 280920252 root 666 3 *** 2019-11-15T14:52:25.678322+04:00 0x00000000 282525885 root 666 3 *** 2019-11-15T14:52:25.678335+04:00 0x00000000 286032062 root 666 3 *** 2019-11-15T14:52:25.678349+04:00 0x00000000 287637695 root 666 3 *** 2019-11-15T14:52:25.678365+04:00 0x00000000 289243328 root 666 3 *** 2019-11-15T14:52:25.678378+04:00 0x00000000 290848961 root 666 3 *** 2019-11-15T14:52:25.678391+04:00 0x00000000 292454594 root 666 3 *** 2019-11-15T14:52:25.678405+04:00 0x00000000 294060227 root 666 3 *** 2019-11-15T14:52:25.678418+04:00 0x00000000 295665860 root 666 3 *** 2019-11-15T14:52:25.678431+04:00 0x00000000 297271493 root 666 3 *** 2019-11-15T14:52:25.678444+04:00 0x00000000 298877126 root 666 3 *** 2019-11-15T14:52:25.678460+04:00 0x00000000 300482759 root 666 3 *** 2019-11-15T14:52:25.678474+04:00 0x00000000 302088392 root 666 3 *** 2019-11-15T14:52:25.678487+04:00 0x00000000 303694025 root 666 3 *** 2019-11-15T14:52:25.678500+04:00 0x00000000 305299658 root 666 3 *** 2019-11-15T14:52:25.678513+04:00 0x00000000 306905291 root 666 3 *** 2019-11-15T14:52:25.678527+04:00 0x00000000 308510924 root 666 3 *** 2019-11-15T14:52:25.678540+04:00 0x00000000 310116557 root 666 3 *** 2019-11-15T14:52:25.678560+04:00 0x00000000 311722190 root 666 3 *** 2019-11-15T14:52:25.678574+04:00 0x00000000 313327823 root 666 3 *** 2019-11-15T14:52:25.678590+04:00 0x00000000 314933456 root 666 3 *** 2019-11-15T14:52:25.678604+04:00 0x00000000 316539089 root 666 3 *** 2019-11-15T14:52:25.678617+04:00 0x00000000 318144722 root 666 3 *** 2019-11-15T14:52:25.678631+04:00 0x00000000 319750355 root 666 3 *** 2019-11-15T14:52:25.678644+04:00 0x00000000 321355988 root 666 3 *** 2019-11-15T14:52:25.678660+04:00 0x00000000 322961621 root 666 3 *** 2019-11-15T14:52:25.678673+04:00 0x00000000 324567254 root 666 3 *** 2019-11-15T14:52:25.678687+04:00 0x00000000 326172887 root 666 3 *** 2019-11-15T14:52:25.678700+04:00 0x00000000 327778520 root 666 3 *** 2019-11-15T14:52:25.678713+04:00 0x00000000 329384153 root 666 3 *** 2019-11-15T14:52:25.678727+04:00 0x00000000 330989786 root 666 3 *** 2019-11-15T14:52:25.678740+04:00 0x00000000 332595419 root 666 3 *** 2019-11-15T14:52:25.678756+04:00 0x00000000 334201052 root 666 3 *** 2019-11-15T14:52:25.678769+04:00 0x00000000 335806685 root 666 3 *** 2019-11-15T14:52:25.678783+04:00 0x00000000 337412318 root 666 3 *** 2019-11-15T14:52:25.678796+04:00 0x00000000 339017951 root 666 3 *** 2019-11-15T14:52:25.678809+04:00 0x00000000 340623584 root 666 3 *** 2019-11-15T14:52:25.678823+04:00 0x00000000 342229217 root 666 3 *** 2019-11-15T14:52:25.678836+04:00 0x00000000 343834850 root 666 3 *** 2019-11-15T14:52:25.678852+04:00 0x00000000 345440483 root 666 3 *** 2019-11-15T14:52:25.678865+04:00 0x00000000 347046116 root 666 3 *** 2019-11-15T14:52:25.678879+04:00 0x00000000 348651749 root 666 3 *** 2019-11-15T14:52:25.678892+04:00 0x00000000 350257382 root 666 3 *** 2019-11-15T14:52:25.678905+04:00 0x00000000 352256231 root 666 3 *** 2019-11-15T14:52:25.678919+04:00 0x00000000 353861864 root 666 3 *** 2019-11-15T14:52:25.678932+04:00 0x00000000 355467497 root 666 3 *** 2019-11-15T14:52:25.678948+04:00 0x00000000 357073130 root 666 3 *** 2019-11-15T14:52:25.678961+04:00 0x00000000 358678763 root 666 3 *** 2019-11-15T14:52:25.678975+04:00 0x00000000 360284396 root 666 3 *** 2019-11-15T14:52:25.678988+04:00 0x00000000 361890029 root 666 3 *** 2019-11-15T14:52:25.679001+04:00 0x00000000 363495662 root 666 3 *** 2019-11-15T14:52:25.679015+04:00 0x00000000 365101295 root 666 3 *** 2019-11-15T14:52:25.679028+04:00 0x00000000 366706928 root 666 3 *** 2019-11-15T14:52:25.679044+04:00 0x00000000 368312561 root 666 3 *** 2019-11-15T14:52:25.679057+04:00 0x00000000 369918194 root 666 3 *** 2019-11-15T14:52:25.679070+04:00 0x00000000 371523827 root 666 3 *** 2019-11-15T14:52:25.679084+04:00 0x00000000 373129460 root 666 3 *** 2019-11-15T14:52:25.679097+04:00 0x00000000 374735093 root 666 3 *** 2019-11-15T14:52:25.679110+04:00 0x00000000 376340726 root 666 3 *** 2019-11-15T14:52:25.679124+04:00 0x00000000 377946359 root 666 3 *** 2019-11-15T14:52:25.679140+04:00 0x00000000 379551992 root 666 3 *** 2019-11-15T14:52:25.679153+04:00 0x00000000 381157625 root 666 3 *** 2019-11-15T14:52:25.679169+04:00 0x00000000 382763258 root 666 3 *** 2019-11-15T14:52:25.679182+04:00 0x00000000 384368891 root 666 3 *** 2019-11-15T14:52:25.679196+04:00 0x00000000 385974524 root 666 3 *** 2019-11-15T14:52:25.679209+04:00 0x00000000 387580157 root 666 3 *** 2019-11-15T14:52:25.679222+04:00 0x00000000 389185790 root 666 3 *** 2019-11-15T14:52:25.679238+04:00 0x00000000 389480703 root 666 3 *** 2019-11-15T14:52:25.679252+04:00 ------ Message Queues -------- *** 2019-11-15T14:52:25.679265+04:00 key msqid owner perms used-bytes messages *** 2019-11-15T14:52:25.679305+04:00 *********************** End of ipcs command dump ************** *** 2019-11-15T14:52:25.679319+04:00 |
Re: RAC instance crash /ORA-29770: global enqueue process
|
Administrator
|
----* NEXT TIME, please upload your trace files.. Don't copy/paste them...--
LHMB is the Global Cache/Enqueue Service Heartbeat Monitor process. It monitors LMON, LMD, and LMSn processes to ensure they are running normally without blocking or spinning. It seems your IPC0 process is in hung status .. "IPC0 (ospid: 368909) has not moved for 108 sec (1573815101.1573814993)" "IPC0 (ospid: 368909) has not moved for 123 sec (1573815116.1573814993)" This is where it breaks.. LGWR is killed at this point. "Forcibly terminated 'oracle@exa-hddbadm01.core.iamas.gov.az (LGWR)' process" Look athe call stack for the kill-> "Abridged Call Stack Trace ----- ksedsts()+346<-kjzduptcctx()+868<-kjzdicrshnfy()+1113<-ksuitm_opt()+1678<-kjgcr_KillInstance()+170<-kjgcr_RunCallback()+1242<-kjgcr_DoAction()+186<-kjgcr_servicecssg()+856<-kjgcr_Main()+418<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202 <-ssthrdmain()+417<-main()+262<-__libc_start_main()+253" After the LGWR is killed, instance is terminated by LMHB. "LMHB (ospid: 368993): terminating the instance due to error 29770" That's why your instance gets killed -> kjfmGCR_HBdisambig: action=Inst-kill Look your LOAD AVERAGE is very high here -> ===[ System Load State ]=== CPU Total 96 Raw 96 Core 48 Socket 2 Load high: Cur 123801 Highmark 122880 (483.59 480.00) Note that, probably Oracle gets the load information from "cpu_sup - A CPU Load and CPU Utilization Supervisor Process".. That 's why those Cur and HighMark values are divided to 256.. *But why is that? Probably we can't get cpu cycles.. I mean the background processes.. Did you check what is causing that load? It is a sudden high load.. It is a peak.. Look -> loadavg : 406.77 117.41 42.64 """At this point, I strongy recommend analyzing your environment for this load.. Currently I think that this is caused by your high load.""" Take a look at this note as well: ORA-29770: global enqueue process LGWR (ospid: xxxxxx) is hung for more than 70 seconds (Doc ID 2457765.1) The patches provided in the above note are not directly addressing your issue, but you may still consider applying them. |
«
Return to EXADATA
|
1 view|%1 views
| Free forum by Nabble | Edit this page |