X5-2 ODA issues
Locked
 
|
Hello Erman,
We are encountering a couple issues with our VM-ODA_BASE ODA X5-2 ODA, We are running OVM. We initially were patching from DOM0, and the patch ended up trashing our second ODA_BASE node. We were able to get some of it restored but are still encountering some issues with OAKCLI commands. I have replaced the IPS in the third comment to something generic. 1. oakcli commands don't seem to work on the oda_base node we restored: [root@wdcoda01cn01 ~]# oakcli show repo OAKERR:6000 Exception encountered in the module getMasterNode OAKERR:6002 Master node was not found 2. Our storage topology seems to be off, this may be because of Firmware versions?: [root@wdcoda01cn01 ~]# oakcli validate -c StorageTopology It may take a while. Please wait... INFO : ODA Topology Verification INFO : Running on Node1 INFO : Check hardware type SUCCESS : Type of hardware found : X5-2 INFO : Check for Environment(Bare Metal or Virtual Machine) SUCCESS : Type of environment found : Virtual Machine(ODA BASE) SUCCESS : Number of External SCSI controllers found : 2 INFO : Check for Controllers correct PCIe slot address SUCCESS : External LSI SAS controller 0 : 00:04.0 SUCCESS : External LSI SAS controller 1 : 00:05.0 INFO : Check if JBOD powered on SUCCESS : 1JBOD : Powered-on INFO : Check for correct number of EBODS(2 or 4) SUCCESS : EBOD found : 2 INFO : Check for External Controller 0 FAILURE : Controller connected to correct EBOD number FAILURE : Controller port connected to correct EBOD port FAILURE : Overall Cable check for controller 0 INFO : Check for External Controller 1 FAILURE : Controller connected to correct EBOD number FAILURE : Controller port connected to correct EBOD port FAILURE : Overall Cable check for Controller 1 INFO : Check for overall status of cable validation on Node1 FAILURE : Overall Cable Validation on Node1 INFO : Check Node Identification status FAILURE : Node Identification ERROR : Display wrong connection found on the Node CTRL0, PORT0 <==> JBOD0, EBOD0, PORT1 CTRL1, PORT1 <==> JBOD0, EBOD1, PORT1 INFO : Correct cable connections with 1 JBOD NODE 0: CTRL0, PORT0 <==> JBOD0, EBOD1, PORT0 : [DARK BLUE CABLE] CTRL0, PORT1 <==> FREE CTRL1, PORT0 <==> FREE CTRL1, PORT1 <==> JBOD0, EBOD0, PORT0 : [LIGHT BLUE CABLE] NODE 1: CTRL0, PORT0 <==> FREE CTRL0, PORT1 <==> JBOD0, EBOD1, PORT1 : [DARK RED CABLE] CTRL1, PORT0 <==> JBOD0, EBOD0, PORT1 : [LIGHT RED CABLE] CTRL1, PORT1 <==> FREE INFO : Convention Used CTRL0 : Controller 0 CTRL1 : Controller 1 <==> : Physical cable connection between controller port and EBOD port INFO : The details for Storage Topology Validation can also be found in the log file=/opt/oracle/oak/log/wdcoda01cn01/storagetopology/StorageTopology-2018-03-06-09:20:21_32048_14930.log 3. Here are the versioning details. [root@wdcoda01cn01 ~]# oakcli show version -detail Reading the metadata. It takes a while... ERROR: 2018-03-06 09:16:11: Failed to connect to oakVmAgent INFO: 2018-03-06 09:16:11: RPC::XML::Client::send_request: HTTP server error: Can't connect to 123.456.72.32:2523 (connect: Connection refused) ERROR: 2018-03-06 09:16:11: Failed to connect to oakVmAgent INFO: 2018-03-06 09:16:11: RPC::XML::Client::send_request: HTTP server error: Can't connect to 123.456.72.32:2523 (connect: Connection refused) ERROR: 2018-03-06 09:16:11: Failed to connect to oakVmAgent INFO: 2018-03-06 09:16:11: RPC::XML::Client::send_request: HTTP server error: Can't connect to 123.456.72.32:2523 (connect: Connection refused) ERROR: 2018-03-06 09:16:11: Failed to connect to oakVmAgent INFO: 2018-03-06 09:16:11: RPC::XML::Client::send_request: HTTP server error: Can't connect to 123.456.72.32:2523 (connect: Connection refused) ERROR: 2018-03-06 09:16:11: Failed to connect to oakVmAgent INFO: 2018-03-06 09:16:11: RPC::XML::Client::send_request: HTTP server error: Can't connect to 123.456.72.32:2523 (connect: Connection refused) ERROR: 2018-03-06 09:17:04: Failed to connect to oakVmAgent INFO: 2018-03-06 09:17:04: RPC::XML::Client::send_request: HTTP server error: Can't connect to 123.456.72.32:2523 (connect: Connection refused) System Version Component Name Installed Version Supported Version -------------- --------------- ------------------ ----------------- 12.1.2.10.0 ERROR: 2018-03-06 09:17:04: Failed to connect to oakVmAgent INFO: 2018-03-06 09:17:04: RPC::XML::Client::send_request: HTTP server error: Can't connect to 123.456.72.32:2523 (connect: Connection refused) Controller_EXT 13.00.00.00 10.00.00.00 Expander 0018 Up-to-date SSD_SHARED { [ c1d20,c1d21,c1d22, A29A Up-to-date c1d23 ] [ c1d16,c1d17,c1d18, A29A Up-to-date c1d19 ] } HDD_SHARED PAG1 Up-to-date ILOM unknown No-update BIOS unknown No-update IPMI 1.8.12.4 Up-to-date HMP 2.3.5.2.8 Up-to-date OAK 12.1.2.10.0 Up-to-date OL 6.8 Up-to-date OVM 3.2.11 GI_HOME 12.1.0.2.170117(2473 Up-to-date 2082,24828633) DB_HOME 12.1.0.2.170117(2473 Up-to-date 2082,24828633) |
Administrator
|
Hello Colby,
1)You may have a cabling issue + a network configuration issue (on the relevant ODA node/nodes) Did you do anything with cabling? Is your cabling correct? 2)Is node1 properly working? If yes, take it as your reference and check the ifconfig output of it.. compare it with node's ifconfig output. Also check node 1 and node 2 by comparing the modprobe.conf and bridge interfaces. (brctl show) Do you see any difference? 3)Which Image version are you using current on your nodes? Which Image version were you trying to update to? 4)Did you check "Installed ODA 12.1.2.3 patch and oda_base is not starting (Community Discussion ID 3726035)" ? |
|
|
Erman,
1)You may have a cabling issue + a network configuration issue (on the relevant ODA node/nodes) Did you do anything with cabling? Is your cabling correct? We never touched the cabling as it's in a different location, we did try patching it which makes me think something may have been messed up with the configuration whether its because entries are missing or incorrect or the drivers have been updated/not updated and that may be causing issues? 2)Is node1 properly working? If yes, take it as your reference and check the ifconfig output of it.. compare it with node's ifconfig output. Also check node 1 and node 2 by comparing the modprobe.conf and bridge interfaces. (brctl show) Do you see any difference? ODA_BASE00 never crashed like ODA_BASE01 which was being patched, I will check into those files. 3)Which Image version are you using current on your nodes? Which Image version were you trying to update to? We are currently on the 12.1.2.10.0 and trying to update to 12.1.2.12.0, now that i am checking it appears our ODA_BASE 00 was upated to 12.1.2.12.0 [root@wdcoda01cn00 ~]# oakcli show version -detail Reading the metadata. It takes a while... System Version Component Name Installed Version Supported Version -------------- --------------- ------------------ ----------------- 12.1.2.12.0 Controller_INT 4.230.40-3739 4.650.00-7176 Controller_EXT fwversion_13.00.00.0 fwversion_13.00.00.0 0, efi_15.00.00.00 0, efi_15.00.02.00 Expander 0018 No-update SSD_SHARED { [ c1d20,c1d21,c1d22, A29A No-update c1d23 ] [ c1d16,c1d17,c1d18, A29A No-update c1d19 ] } HDD_LOCAL A7E0 No-update HDD_SHARED PAG1 No-update ILOM unknown No-update BIOS unknown No-update IPMI 1.8.12.4 No-update HMP 2.3.5.2.8 No-update OAK 12.1.2.10.0 No-update OL 6.8 No-update OVM 3.2.11 No-update GI_HOME 12.1.0.2.170117(2473 No-update 2082,24828633) DB_HOME 12.1.0.2.170117(2473 No-update 2082,24828633) ASR 5.5.1 No-update [root@wdcoda01cn01 adapters]# oakcli show version Version ------- 12.1.2.10.0 [root@wdcoda01cn01 adapters]# oakcli show version -detail Reading the metadata. It takes a while... ERROR: 2018-03-06 13:00:23: Failed to connect to oakVmAgent INFO: 2018-03-06 13:00:23: RPC::XML::Client::send_request: HTTP server error: Can't connect to 192.168.16.25:7882 (connect: Connection refused) ERROR: 2018-03-06 13:00:23: Failed to connect to oakVmAgent INFO: 2018-03-06 13:00:23: RPC::XML::Client::send_request: HTTP server error: Can't connect to 192.168.16.25:7882 (connect: Connection refused) ERROR: 2018-03-06 13:00:23: Failed to connect to oakVmAgent INFO: 2018-03-06 13:00:23: RPC::XML::Client::send_request: HTTP server error: Can't connect to 192.168.16.25:7882 (connect: Connection refused) ERROR: 2018-03-06 13:00:23: Failed to connect to oakVmAgent INFO: 2018-03-06 13:00:23: RPC::XML::Client::send_request: HTTP server error: Can't connect to 192.168.16.25:7882 (connect: Connection refused) ERROR: 2018-03-06 13:00:23: Failed to connect to oakVmAgent INFO: 2018-03-06 13:00:23: RPC::XML::Client::send_request: HTTP server error: Can't connect to 192.168.16.25:7882 (connect: Connection refused) ERROR: 2018-03-06 13:01:14: Failed to connect to oakVmAgent INFO: 2018-03-06 13:01:14: RPC::XML::Client::send_request: HTTP server error: Can't connect to 192.168.16.25:7882 (connect: Connection refused) System Version Component Name Installed Version Supported Version -------------- --------------- ------------------ ----------------- 12.1.2.10.0 ERROR: 2018-03-06 13:01:14: Failed to connect to oakVmAgent INFO: 2018-03-06 13:01:14: RPC::XML::Client::send_request: HTTP server error: Can't connect to 192.168.16.25:7882 (connect: Connection refused) Controller_EXT 13.00.00.00 10.00.00.00 Expander 0018 Up-to-date SSD_SHARED { [ c1d20,c1d21,c1d22, A29A Up-to-date c1d23 ] [ c1d16,c1d17,c1d18, A29A Up-to-date c1d19 ] } HDD_SHARED PAG1 Up-to-date ILOM unknown No-update BIOS unknown No-update IPMI 1.8.12.4 Up-to-date HMP 2.3.5.2.8 Up-to-date OAK 12.1.2.10.0 Up-to-date OL 6.8 Up-to-date OVM 3.2.11 GI_HOME 12.1.0.2.170117(2473 Up-to-date 2082,24828633) DB_HOME 12.1.0.2.170117(2473 Up-to-date 2082,24828633) 4)Did you check "Installed ODA 12.1.2.3 patch and oda_base is not starting (Community Discussion ID 3726035)" ? Our ODA_BASE 01 starts just is having some issues with storage/oakcli commands, I will check into this doc more. |
|
|
I had a similar issue on an X5-2HA. Tried to patch and the system ended up getting trashed. In addition before this, we were running into issues where occasionally a running VM would become hung - couldn't login it -- the only solution was to revert to a windows 'fix' (recycle the power). The consideration to patch was hoping that it would fix the hanging VM problem.
Anyway after attempting to patch, it all went to sh*t. Here's what we were seeing on the server: root@tpin-oda1 ~]# oakcli show repo NAME TYPE NODENUM FREE SPACE STATE SIZE repo shared 0 N/A UNKNOWN N/A repo shared 1 99.96% ONLINE 5120000.0M odarepo1 local 0 N/A N/A N/A odarepo2 local 1 N/A N/A N/A shared shared 0 N/A UNKNOWN N/A shared shared 1 32.37% ONLINE 204800.0M And.. [oracle@oda0 ~]$ sqlplus / as sysdba SQL*Plus: Release 12.1.0.2.0 Production on Wed Feb 14 10:03:33 2018 Copyright (c) 1982, 2014, Oracle. All rights reserved. ERROR: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925 ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925 But when I ran the df -h command, there was plenty of disk space.... The fix ==== Turns out the system had exhausted the available iNodes: root@oda0 ~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvdb1 6111232 611232 0 100% /u01 This could be quickly cleaned up by running: cd /u01/app/12.1.0.2/grid/rdbms/audit/ find . -mtime +30 -exec ls -alR|wc -l\; But the 'ultimate' solution was to run the purgeODALog.sh script and finally to place a call to the script into cron: [root@oda0 ~]# crontab -l 00 02 * * * /opt/oracle/oak/bin/purgeODALog.sh -orcl 20 -tfa 10 -osw 10 -oak 10 Since then, rock solid. Never a problem with hanging VMs and I'm thinking that I might even be wanting to patch the system just to stay on the current release. |
|
|
Alex,
We have not encountered VM's getting hung as of yet... This is the command run from ODA_BAS 00 as it does not currently work from ODA_BASE 01 [root@wdcoda01cn00 ~]# oakcli show repo NAME TYPE NODENUM FREE SPACE STATE SIZE odarepo1 local 0 N/A N/A N/A odarepo2 local 1 N/A N/A N/A templates shared 0 16.21% ONLINE 256000.0M templates shared 1 N/A OFFLINE N/A vmprod shared 0 5.00% ONLINE 1689600.0M vmprod shared 1 N/A OFFLINE N/A It's funny that you mention the second bit about ORA-09925: Unable to create audit trail file... once we were able to get ODA_BASE 01 up we used TSM to restore back all of the old ORACLE_HOMES.. in doing so i noticed that /adump directory was missing from each individual oracle home in /u01/app/oracle/diag/rdbms. Once I created new /adump dirs & also restored my WALLET I was able to startup the databases.. all except for one.. which was complaining about the following Fri Mar 02 12:09:23 2018 PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s) Fri Mar 02 12:09:23 2018 2048K 0 2050 0 NONE Fri Mar 02 12:09:23 2018 Reason for not supporting certain system pagesizes: Fri Mar 02 12:09:23 2018 4K - Large pagesizes only Fri Mar 02 12:09:23 2018 RECOMMENDATION: Fri Mar 02 12:09:23 2018 1. Configure system with expected number of pages for every supported system pagesize prior to the next instance restart operation. I then found that hugepages were not working correctly and had to add an entry to /etc/sysctl.conf bounced everything and seemed to get the database started along with the hugepages working. # Added by ODA Development. Do not remove fix_control=19265077 vm.nr_hugepages=25821 I am seeing the following when checking for iNode usage.. [root@wdcoda01cn01 ~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda2 3637248 51172 3586076 2% / tmpfs 12366284 625 12365659 1% /dev/shm /dev/xvda1 121920 39 121881 1% /boot /dev/xvdb1 12214272 470641 11743631 4% /u01 /dev/asm/datastore-72 71303168 59336960 11966208 84% /u01/app/oracle/oradata/datastore /dev/asm/flashdata-409 585105408 76940584 508164824 14% /u02/app/oracle/oradata/flashdata /dev/asm/acfsvol-38 104857600 19355080 85502520 19% /cloudfs /dev/asm/nfsifa_ad01-231 104857600 49433696 55423904 48% /nfsifinapp_ad01 /dev/asm/nfsifa_au01-231 104857600 5302000 99555600 6% /nfsifinapp_au01 /dev/asm/nfsefa_c01-231 104857600 966080 103891520 1% /nfsefinapp_c01 /dev/asm/nfsefa_au02-231 209715200 572192 209143008 1% /nfsefinapp_au02 /dev/asm/nfsifa_au03-231 209715200 572192 209143008 1% /nfsifinapp_au03 /dev/asm/nfsifa_c01-231 629145600 228323784 400821816 37% /nfsifinapp_c01 /dev/asm/nfsifa_au02-231 209715200 96213448 113501752 46% /nfsifinapp_au02 /dev/asm/templates-231 524288000 439291888 84996112 84% /u01/app/sharedrepo/templates /dev/asm/nfsefa_ad02-231 838860800 692813808 146046992 83% /nfsefinapp_ad02 /dev/asm/nfsifa_ad02-231 1258291200 622119104 636172096 50% /nfsifinapp_ad02 /dev/asm/c01-231 2097152000 1232540064 864611936 59% /c01 /dev/asm/datastore-231 24735907840 13605734504 11130173336 56% /u02/app/oracle/oradata/datastore /dev/asm/vmprod-231 3460300800 3287124080 173176720 95% /u01/app/sharedrepo/vmprod /dev/asm/datastore-38 4062183424 493625160 3568558264 13% /u01/app/oracle/fast_recovery_area/datastore |
|
Administrator
|
Hi Colby,
1)You restored Oracle Homes, but what about the oda-related binaries? As you know, by applying a patch bundle, we patch; SERVER Operating System, ILOM,BIOS, Hardware Management Pack, IPMI, OAK, ASR and local disks. For virtualized environment, it also includes the Dom0 updates. Oracle Grid Infrastructure (GI) Stack including ASM, ACFS and TFA. STORAGE Shared storage including HDD, SSD, Controller, and Expander DATABASE RDBMS stack 2)In which step did you failed while applying ODA Patch bundle to node 2? Send me the read me and the failure point.. 3)Also, you are in a situation where one of your nodes is in a higher version and one of your nodes is in lower version.. This is not good at all. Are you planning to upgrade node 2 again? 4)Did you check "Installed ODA 12.1.2.3 patch and oda_base is not starting (Community Discussion ID 3726035)" ? Could you compare the ifconfig outputs of the nodes. Also check node 1 and node 2 by comparing the modprobe.conf and bridge interfaces. (brctl show) Do you see any difference? |
|
|
Hi All,
I have an update to provided, we were able to fix this with Oracle supports help. Let me gather the notes.. My colleague did and i will post them. Cheers, Colby |
|
Administrator
|
Thank you Colby..
Waiting for you to share the notes :) |
|
|
Hi All,
This will be a long post with all of the info included, my colleague provided me with 3 of his ssh session logs after he met with a tech from oracle. Apologies for the length. SESSION #1 [2018-03-06 22:34:39] [root@wdcoda01cn01 ~]# oakcli show ismaster [2018-03-06 22:34:44] OAKD is in Slave Mode [2018-03-06 22:34:44] [root@wdcoda01cn01 ~]# oakcli show vm [2018-03-06 22:34:49] OAKERR:6000 Exception encountered in the module getMasterNode [2018-03-06 22:34:49] OAKERR:6002 Master node was not found [2018-03-06 22:34:49] [root@wdcoda01cn01 ~]# oakcli ps -ef | grep oak [2018-03-06 22:35:59] root 2587 1 0 14:43 ? 00:00:00 /bin/sh /etc/init.d/init.oak run [2018-03-06 22:35:59] root 5054 1 0 14:44 ? 00:03:26 /bin/sh ./OSWatcher.sh 10 504 gzip /opt/oracle/oak/oswbb/archive [2018-03-06 22:35:59] root 5549 1 0 14:44 ? 00:00:02 /usr/bin/python /opt/oracle/oak/adapters/odaBaseAgent.py [2018-03-06 22:35:59] root 5551 1 0 14:44 ? 00:00:03 /usr/bin/python /opt/oracle/oak/bin/dcliagent.py [2018-03-06 22:35:59] root 6735 5054 0 14:44 ? 00:00:01 /bin/sh ./OSWatcherFM.sh 504 /opt/oracle/oak/oswbb/archive [2018-03-06 22:35:59] root 16016 2587 0 14:45 ? 00:02:44 /opt/oracle/oak/bin/oakd foreground [2018-03-06 22:35:59] root 50283 40206 0 22:33 pts/0 00:00:00 grep oak [2018-03-06 22:35:59] [root@wdcoda01cn01 ~]# /u01/app/12.1.0.2/grid/bin/srvctl status havip [2018-03-06 22:36:31] HAVIP ora.hanfsvip_id1.havip is enabled [2018-03-06 22:36:32] HAVIP ora.hanfsvip_id1.havip is running on nodes wdcoda01cn01 [2018-03-06 22:36:32] HAVIP ora.havip_3.havip is enabled [2018-03-06 22:36:32] HAVIP ora.havip_3.havip is running on nodes wdcoda01cn00 [2018-03-06 22:36:32] HAVIP ora.havip_4.havip is enabled [2018-03-06 22:36:32] HAVIP ora.havip_4.havip is not running [2018-03-06 22:36:32] [root@wdcoda01cn01 ~]# /u01/app/12.1.0.2/grid/bin/srvctl status havipconfig [2018-03-06 22:36:43] HAVIP exists: /hanfsvip_id1/12.34.567.89, network number 1 [2018-03-06 22:36:44] Description: hanfs wdc oda [2018-03-06 22:36:44] Home Node: [2018-03-06 22:36:44] HAVIP is enabled. [2018-03-06 22:36:44] HAVIP is individually enabled on nodes: [2018-03-06 22:36:44] HAVIP is individually disabled on nodes: [2018-03-06 22:36:44] HAVIP exists: /havip_3/123.456.78.91, network number 3 [2018-03-06 22:36:44] Description: HAVIP_NODE1 [2018-03-06 22:36:44] Home Node: wdcoda01cn00 [2018-03-06 22:36:44] HAVIP is enabled. [2018-03-06 22:36:44] HAVIP is individually enabled on nodes: [2018-03-06 22:36:45] HAVIP is individually disabled on nodes: [2018-03-06 22:36:45] HAVIP exists: /havip_4/123.456.78.91, network number 4 [2018-03-06 22:36:45] Description: HAVIP_NODE2 [2018-03-06 22:36:45] Home Node: wdcoda01cn01 [2018-03-06 22:36:45] HAVIP is enabled. [2018-03-06 22:36:45] HAVIP is individually enabled on nodes: [2018-03-06 22:36:45] HAVIP is individually disabled on nodes: [2018-03-06 22:36:45] [root@wdcoda01cn01 ~]# /u01/app/12.1.0.2/grid/bin/srvctl config havipstatus havip rt havip -id havip_4 [2018-03-06 22:37:30] PRCR-1079 : Failed to start resource ora.havip_4.havip [2018-03-06 22:37:31] CRS-2805: Unable to start 'ora.havip_4.havip' because it has a 'hard' dependency on resource type 'ora.havip_4.export.type' and no resource of that type can satisfy the dependency [2018-03-06 22:37:31] [root@wdcoda01cn01 ~]# ifconfig [2018-03-06 22:37:39] eth0 Link encap:Ethernet HWaddr 00:16:3E:1D:B1:9E [2018-03-06 22:37:39] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:37:39] RX packets:471 errors:0 dropped:471 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:1000 [2018-03-06 22:37:39] RX bytes:108801 (106.2 KiB) TX bytes:0 (0.0 b) [2018-03-06 22:37:39] [2018-03-06 22:37:39] eth1 Link encap:Ethernet HWaddr 00:16:3E:0E:23:C0 [2018-03-06 22:37:39] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:37:39] RX packets:76028 errors:0 dropped:471 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:1000 [2018-03-06 22:37:39] RX bytes:7664501 (7.3 MiB) TX bytes:0 (0.0 b) [2018-03-06 22:37:39] [2018-03-06 22:37:39] eth3 Link encap:Ethernet HWaddr 00:16:3E:44:E9:FE [2018-03-06 22:37:39] inet addr:10.70.155.9 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:37:39] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:37:39] RX packets:129610 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:45089 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:1000 [2018-03-06 22:37:39] RX bytes:15123631 (14.4 MiB) TX bytes:18162549 (17.3 MiB) [2018-03-06 22:37:39] [2018-03-06 22:37:39] eth3:1 Link encap:Ethernet HWaddr 00:16:3E:44:E9:FE [2018-03-06 22:37:39] inet addr:10.70.155.11 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:37:39] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:37:39] [2018-03-06 22:37:39] eth3:2 Link encap:Ethernet HWaddr 00:16:3E:44:E9:FE [2018-03-06 22:37:39] inet addr:10.70.155.12 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:37:39] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:37:39] [2018-03-06 22:37:39] eth3:3 Link encap:Ethernet HWaddr 00:16:3E:44:E9:FE [2018-03-06 22:37:39] inet addr:12.34.567.89 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:37:39] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:37:39] [2018-03-06 22:37:39] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:37:39] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:37:39] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:37:39] ibbond0 Link encap:InfiniBand HWaddr 80:00:05:4C:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:37:39] inet addr:123.456.78.91 Bcast:123.456.78.255 Mask:255.255.255.0 [2018-03-06 22:37:39] UP BROADCAST RUNNING MASTER MULTICAST MTU:7000 Metric:1 [2018-03-06 22:37:39] RX packets:6606548 errors:0 dropped:2 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:7408622 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:0 [2018-03-06 22:37:39] RX bytes:6171659569 (5.7 GiB) TX bytes:9269190701 (8.6 GiB) [2018-03-06 22:37:39] [2018-03-06 22:37:39] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:37:39] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:37:39] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:37:39] ibbond0:1 Link encap:InfiniBand HWaddr 80:00:05:4C:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:37:39] inet addr:169.254.221.228 Bcast:169.254.255.255 Mask:255.255.0.0 [2018-03-06 22:37:39] UP BROADCAST RUNNING MASTER MULTICAST MTU:7000 Metric:1 [2018-03-06 22:37:39] [2018-03-06 22:37:39] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:37:39] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:37:39] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:37:39] ib0 Link encap:InfiniBand HWaddr 80:00:05:4C:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:37:39] UP BROADCAST RUNNING SLAVE MULTICAST MTU:7000 Metric:1 [2018-03-06 22:37:39] RX packets:6606548 errors:0 dropped:2 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:7408622 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:4096 [2018-03-06 22:37:39] RX bytes:6171659569 (5.7 GiB) TX bytes:9269190701 (8.6 GiB) [2018-03-06 22:37:39] [2018-03-06 22:37:39] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:37:39] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:37:39] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:37:39] ib1 Link encap:InfiniBand HWaddr 80:00:05:4D:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:37:39] UP BROADCAST RUNNING SLAVE MULTICAST MTU:7000 Metric:1 [2018-03-06 22:37:39] RX packets:0 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:4096 [2018-03-06 22:37:39] RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) [2018-03-06 22:37:39] [2018-03-06 22:37:39] lo Link encap:Local Loopback [2018-03-06 22:37:39] inet addr:127.0.0.1 Mask:255.0.0.0 [2018-03-06 22:37:39] UP LOOPBACK RUNNING MTU:16436 Metric:1 [2018-03-06 22:37:39] RX packets:785079 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:785079 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:0 [2018-03-06 22:37:39] RX bytes:582689050 (555.6 MiB) TX bytes:582689050 (555.6 MiB) [2018-03-06 22:37:39] [2018-03-06 22:37:39] veth1 Link encap:Ethernet HWaddr 00:16:3E:EA:1C:8C [2018-03-06 22:37:39] inet addr:123.456.78.20 Bcast:123.456.78.255 Mask:255.255.255.0 [2018-03-06 22:37:39] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:37:39] RX packets:23652 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:37:39] TX packets:23725 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:37:39] collisions:0 txqueuelen:1000 [2018-03-06 22:37:39] RX bytes:1361720 (1.2 MiB) TX bytes:1271396 (1.2 MiB) [2018-03-06 22:37:39] [2018-03-06 22:37:39] [root@wdcoda01cn01 ~]# cd /opt/oracle/oak/log/wdcoda01cn01odab /odabaseagent/ [2018-03-06 22:38:51] [root@wdcoda01cn01 odabaseagent]# vi odabaseagent.log [2018-03-06 22:38:53] "odabaseagent.log" 216L, 28020C2018-02-28 06:47:17,875 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [69] Starting odabaseagent [2018-03-06 22:38:53] 2018-02-28 06:47:17,875 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [98] In runServer [2018-03-06 22:38:53] 2018-02-28 06:47:17,875 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [99] About to initialize the agent server and register the services... [2018-03-06 22:38:53] 2018-02-28 06:47:17,875 [Cmd_EnvId] [MainThread] [odabaseutils] [INFO] [46] The ODABase Agent utility module is initialized [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [sharedrepoactions] [INFO] [54] sharedrepoactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [odaactions] [INFO] [32] reporesourceactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [vdiskactions] [INFO] [30] vdiskactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [snapactions] [INFO] [36] snapactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [infinibandactions] [INFO] [16] infiniband actions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [38] common_agentutils initialized [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [120] getodabaseagentnodes invoked [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [95] isValidaEnvForIBFix invoked [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [41] get_env invoked [2018-03-06 22:38:53] 2018-02-28 06:47:17,925 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [44] vm detected [2018-03-06 22:38:53] 2018-02-28 06:47:17,939 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [102] returning False [2018-03-06 22:38:53] 2018-02-28 06:47:17,940 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [125] returning odaBase agent nodes: {0: {'IP': '123.456.78.27', 'port': '9000'}, 1: {'IP': '123.456.78.91', 'port': '9000'}} [2018-03-06 22:38:53] 2018-02-28 06:47:17,942 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [81] ro root=LABEL=rootfs tsc=reliable nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe numa=offf console=tty0 console=ttyS0,115200n8 selinux=0 nohz=off crashkernel=256M@64M loglevel=3 panic=60 ipv6.disable=1 transparent_hugepage=never NODENUM=1 PRODUCT=ORACLE_SERVER_X55-2 TYPE=V4 pci=noaer,nocrs [2018-03-06 22:38:53] 2018-02-28 06:47:17,942 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [85] The Node number is 1 [2018-03-06 22:38:53] 2018-02-28 06:47:17,943 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [212] Registered the functions, running the xmlrpc server [2018-03-06 22:38:53] 2018-02-28 06:55:03,390 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [69] Starting odabaseagent [2018-03-06 22:38:53] 2018-02-28 06:55:03,396 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [98] In runServer [2018-03-06 22:38:53] 2018-02-28 06:55:03,396 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [99] About to initialize the agent server and register the services... [2018-03-06 22:38:53] 2018-02-28 06:55:03,396 [Cmd_EnvId] [MainThread] [odabaseutils] [INFO] [46] The ODABase Agent utility module is initialized [2018-03-06 22:38:53] 2018-02-28 06:55:03,445 [Cmd_EnvId] [MainThread] [sharedrepoactions] [INFO] [54] sharedrepoactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:55:03,445 [Cmd_EnvId] [MainThread] [odaactions] [INFO] [32] reporesourceactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:55:03,445 [Cmd_EnvId] [MainThread] [vdiskactions] [INFO] [30] vdiskactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:55:03,445 [Cmd_EnvId] [MainThread] [snapactions] [INFO] [36] snapactions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:55:03,445 [Cmd_EnvId] [MainThread] [infinibandactions] [INFO] [16] infiniband actions module is initialized [2018-03-06 22:38:53] 2018-02-28 06:55:03,445 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [38] common_agentutils initialized [2018-03-06 22:38:53] 2018-02-28 06:55:03,446 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [120] getodabaseagentnodes invoked [2018-03-06 22:38:53] 2018-02-28 06:55:03,446 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [95] isValidaEnvForIBFix invoked [2018-03-06 22:38:53] 2018-02-28 06:55:03,446 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [41] get_env invoked [2018-03-06 22:38:53] 2018-02-28 06:55:03,446 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [44] vm detected [2018-03-06 22:38:53] 2018-02-28 06:55:03,459 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [102] returning False [2018-03-06 22:38:53] 2018-02-28 06:55:03,459 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [125] returning odaBase agent nodes: {0: {'IP': '123.456.78.27', 'port': '9000'}, 1: {'IP': '123.456.78.91', 'port': '9000'}} [2018-03-06 22:38:53] 2018-02-28 06:55:03,461 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [81] ro root=LABEL=rootfs tsc=reliable nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe numa=offf console=tty0 console=ttyS0,115200n8 selinux=0 nohz=off crashkernel=256M@64M loglevel=3 panic=60 ipv6.disable=1 transparent_hugepage=never NODENUM=1 PRODUCT=ORACLE_SERVER_X55-2 TYPE=V4 pci=noaer,nocrs [2018-03-06 22:38:53] 2018-02-28 06:55:03,461 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [85] The Node number is 1 [2018-03-06 22:38:53] 2018-02-28 06:55:03,462 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [212] Registered the functions, running the xmlrpc server [2018-03-06 22:38:53] 2018-02-28 07:50:28,010 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [69] Starting odabaseagent [2018-03-06 22:38:53] 2018-02-28 07:50:28,012 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [98] In runServer [2018-03-06 22:38:53] 2018-02-28 07:50:28,012 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [99] About to initialize the agent server and register the services... [2018-03-06 22:38:53] 2018-02-28 07:50:28,012 [Cmd_EnvId] [MainThread] [odabaseutils] [INFO] [46] The ODABase Agent utility module is initialized [2018-03-06 22:38:53] 2018-02-28 07:50:28,061 [Cmd_EnvId] [MainThread] [sharedrepoactions] [INFO] [54] sharedrepoactions module is initialized [2018-03-06 22:38:53] 2018-02-28 07:50:28,061 [Cmd_EnvId] [MainThread] [odaactions] [INFO] [32] reporesourceactions module is initialized [2018-03-06 22:38:53] 2018-02-28 07:50:28,061 [Cmd_EnvId] [MainThread] [vdiskactions] [INFO] [30] vdiskactions module is initialized [2018-03-06 22:38:53] 2018-02-28 07:50:28,061 [Cmd_EnvId] [MainThread] [snapactions] [INFO] [36] snapactions module is initialized [2018-03-06 22:38:53] 2018-02-28 07:50:28,061 [Cmd_EnvId] [MainThread] [infinibandactions] [INFO] [16] infiniband actions module is initialized [2018-03-06 22:38:53] 2018-02-28 07:50:28,061 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [38] common_agentutils initialized [2018-03-06 22:38:53] 2018-02-28 07:50:28,061 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [120] getodabaseagentnodes invoked [2018-03-06 22:38:53] 2018-02-28 07:50:28,062 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [95] isValidaEnvForIBFix invoked [2018-03-06 22:38:53] 2018-02-28 07:50:28,062 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [41] get_env invoked [2018-03-06 22:38:53] 2018-02-28 07:50:28,062 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [44] vm detected2018-03-05 12:55:03,707 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [95] isValidaEnvForIBFix invoked [2018-03-06 22:38:55] 2018-03-05 12:55:03,708 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [41] get_env invoked [2018-03-06 22:38:55] 2018-03-05 12:55:03,708 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [44] vm detected [2018-03-06 22:38:55] 2018-03-05 12:55:03,724 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [239] isIBFixEnabled invoked [2018-03-06 22:38:55] 2018-03-05 12:55:03,724 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [102] returning False [2018-03-06 22:38:55] 2018-03-05 12:55:03,725 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [125] returning odaBase agent nodes: {0: {'IP': '123.456.78.27', 'port': '9000'}, 1: {'IP': '123.456.78.91', 'port': '9000'}} [2018-03-06 22:38:55] 2018-03-05 12:55:03,726 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [81] ro root=LABEL=rootfs tsc=reliable nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe numa=offf console=tty0 console=ttyS0,115200n8 selinux=0 nohz=off crashkernel=256M@64M loglevel=3 panic=60 ipv6.disable=1 transparent_hugepage=never NODENUM=1 PRODUCT=ORACLE_SERVER_X55-2 TYPE=V4 pci=noaer,nocrs [2018-03-06 22:38:55] 2018-03-05 12:55:03,727 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [85] The Node number is 1 [2018-03-06 22:38:55] 2018-03-05 12:55:03,727 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [212] Registered the functions, running the xmlrpc server [2018-03-06 22:38:55] 2018-03-06 08:28:56,560 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [69] Starting odabaseagent [2018-03-06 22:38:55] 2018-03-06 08:28:56,560 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [98] In runServer [2018-03-06 22:38:55] 2018-03-06 08:28:56,561 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [99] About to initialize the agent server and register the services... [2018-03-06 22:38:55] 2018-03-06 08:28:56,561 [Cmd_EnvId] [MainThread] [odabaseutils] [INFO] [46] The ODABase Agent utility module is initialized [2018-03-06 22:38:55] 2018-03-06 08:28:56,613 [Cmd_EnvId] [MainThread] [sharedrepoactions] [INFO] [54] sharedrepoactions module is initialized [2018-03-06 22:38:55] 2018-03-06 08:28:56,613 [Cmd_EnvId] [MainThread] [odaactions] [INFO] [32] reporesourceactions module is initialized [2018-03-06 22:38:55] 2018-03-06 08:28:56,614 [Cmd_EnvId] [MainThread] [vdiskactions] [INFO] [30] vdiskactions module is initialized [2018-03-06 22:38:55] 2018-03-06 08:28:56,614 [Cmd_EnvId] [MainThread] [snapactions] [INFO] [36] snapactions module is initialized [2018-03-06 22:38:55] 2018-03-06 08:28:56,614 [Cmd_EnvId] [MainThread] [infinibandactions] [INFO] [16] infiniband actions module is initialized [2018-03-06 22:38:55] 2018-03-06 08:28:56,614 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [38] common_agentutils initialized [2018-03-06 22:38:55] 2018-03-06 08:28:56,614 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [120] getodabaseagentnodes invoked [2018-03-06 22:38:55] 2018-03-06 08:28:56,614 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [95] isValidaEnvForIBFix invoked [2018-03-06 22:38:55] 2018-03-06 08:28:56,614 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [41] get_env invoked [2018-03-06 22:38:55] 2018-03-06 08:28:56,615 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [44] vm detected [2018-03-06 22:38:55] 2018-03-06 08:28:56,629 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [239] isIBFixEnabled invoked [2018-03-06 22:38:55] 2018-03-06 08:28:56,629 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [102] returning False [2018-03-06 22:38:55] 2018-03-06 08:28:56,629 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [125] returning odaBase agent nodes: {0: {'IP': '123.456.78.27', 'port': '9000'}, 1: {'IP': '123.456.78.91', 'port': '9000'}} [2018-03-06 22:38:55] 2018-03-06 08:28:56,631 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [81] ro root=LABEL=rootfs tsc=reliable nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe numa=offf console=tty0 console=ttyS0,115200n8 selinux=0 nohz=off crashkernel=256M@64M loglevel=3 panic=60 ipv6.disable=1 transparent_hugepage=never NODENUM=1 PRODUCT=ORACLE_SERVER_X55-2 TYPE=V4 pci=noaer,nocrs [2018-03-06 22:38:55] 2018-03-06 08:28:56,631 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [85] The Node number is 1 [2018-03-06 22:38:55] 2018-03-06 08:28:56,632 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [212] Registered the functions, running the xmlrpc server [2018-03-06 22:38:55] 2018-03-06 14:45:40,749 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [69] Starting odabaseagent [2018-03-06 22:38:55] 2018-03-06 14:45:40,766 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [98] In runServer [2018-03-06 22:38:55] 2018-03-06 14:45:40,766 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [99] About to initialize the agent server and register the services... [2018-03-06 22:38:55] 2018-03-06 14:45:40,766 [Cmd_EnvId] [MainThread] [odabaseutils] [INFO] [46] The ODABase Agent utility module is initialized [2018-03-06 22:38:55] 2018-03-06 14:45:40,820 [Cmd_EnvId] [MainThread] [sharedrepoactions] [INFO] [54] sharedrepoactions module is initialized [2018-03-06 22:38:55] 2018-03-06 14:45:40,820 [Cmd_EnvId] [MainThread] [odaactions] [INFO] [32] reporesourceactions module is initialized [2018-03-06 22:38:55] 2018-03-06 14:45:40,820 [Cmd_EnvId] [MainThread] [vdiskactions] [INFO] [30] vdiskactions module is initialized [2018-03-06 22:38:55] 2018-03-06 14:45:40,821 [Cmd_EnvId] [MainThread] [snapactions] [INFO] [36] snapactions module is initialized [2018-03-06 22:38:55] 2018-03-06 14:45:40,821 [Cmd_EnvId] [MainThread] [infinibandactions] [INFO] [16] infiniband actions module is initialized [2018-03-06 22:38:55] 2018-03-06 14:45:40,821 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [38] common_agentutils initialized [2018-03-06 22:38:55] 2018-03-06 14:45:40,821 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [120] getodabaseagentnodes invoked [2018-03-06 22:38:55] 2018-03-06 14:45:40,821 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [95] isValidaEnvForIBFix invoked [2018-03-06 22:38:55] 2018-03-06 14:45:40,821 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [41] get_env invoked [2018-03-06 22:38:55] 2018-03-06 14:45:40,821 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [44] vm detected [2018-03-06 22:38:55] 2018-03-06 14:45:40,836 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [239] isIBFixEnabled invoked [2018-03-06 22:38:55] 2018-03-06 14:45:40,836 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [102] returning False [2018-03-06 22:38:55] 2018-03-06 14:45:40,836 [Cmd_EnvId] [MainThread] [common_agentutils] [INFO] [125] returning odaBase agent nodes: {0: {'IP': '123.456.78.27', 'port': '9000'}, 1: {'IP': '123.456.78.91', 'port': '9000'}} [2018-03-06 22:38:55] 2018-03-06 14:45:40,838 [Cmd_EnvId] [MainThread] [odaBaseAgent] [DEBUG] [81] ro root=LABEL=rootfs tsc=reliable nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe numa=offf console=tty0 console=ttyS0,115200n8 selinux=0 nohz=off crashkernel=256M@64M loglevel=3 panic=60 ipv6.disable=1 transparent_hugepage=never NODENUM=1 PRODUCT=ORACLE_SERVER_X55-2 TYPE=V4 pci=noaer,nocrs [2018-03-06 22:38:55] 2018-03-06 14:45:40,838 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [85] The Node number is 1 [2018-03-06 22:38:55] 2018-03-06 14:45:40,839 [Cmd_EnvId] [MainThread] [odaBaseAgent] [INFO] [212] Registered the functions, running the xmlrpc server [2018-03-06 22:39:24] :q [root@wdcoda01cn01 odabaseagent]# [2018-03-06 22:39:28] [root@wdcoda01cn01 odabaseagent]# [2018-03-06 22:39:28] [root@wdcoda01cn01 odabaseagent]# cd /opt/oracle/oak/conf [2018-03-06 22:39:35] [root@wdcoda01cn01 conf]# ls -al [2018-03-06 22:39:37] total 180 [2018-03-06 22:39:37] drwxr-xr-x 3 root root 4096 Mar 6 14:46 . [2018-03-06 22:39:37] drwxr-xr-x 14 root root 4096 Feb 28 16:46 .. [2018-03-06 22:39:37] -rw-r--r-- 1 root root 0 Feb 28 10:50 .clustered [2018-03-06 22:39:37] -rw-r--r-- 1 root root 0 Feb 28 10:50 .configlock [2018-03-06 22:39:37] -rw-rw-r-- 1 root root 0 Feb 15 2017 .odaib [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 1019 Feb 15 2017 70-persistent-nvme.rules [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 787 Feb 15 2017 99-oda_multipath.rules [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 3182 Feb 15 2017 99-oda_singleuser_multipath.rules [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 3332 Feb 15 2017 agentnodeConstants.py [2018-03-06 22:39:37] -rw-r--r-- 1 root root 2676 Feb 28 16:46 agentnodeConstants.pyc [2018-03-06 22:39:37] drwxrwxrwx 2 root root 4096 Feb 28 10:42 backupConfig [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 239 Feb 15 2017 dcliagent.conf [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 572 Feb 15 2017 discoverdummy.xml [2018-03-06 22:39:37] -rw-r--r-- 1 root root 3258 Mar 6 22:27 discoverjbod.xml [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 245 Feb 15 2017 initoak.conf [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 456 Feb 15 2017 logConfig.conf [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 61532 Feb 28 10:49 oak_conf.xml [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 31605 Feb 15 2017 oak_lite_conf.xml [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 258 Feb 15 2017 odaBaseAgent.conf [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 2160 Feb 15 2017 odalite_validation_props.xml [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 9812 Feb 15 2017 stm_defs.xml [2018-03-06 22:39:37] -rwxr-xr-x 1 root root 6859 Feb 15 2017 validation_props.xml I believe the .odaib file was so large they just touched a 1 to it then removed it below. [2018-03-06 22:39:51] [root@wdcoda01cn01 conf]# cat .odaib [2018-03-06 22:39:55] 1 [2018-03-06 22:39:55] [root@wdcoda01cn01 conf]# cat .odaib [2018-03-06 22:40:00] 1 [2018-03-06 22:40:00] [root@wdcoda01cn01 conf]# rm .orad ib daib [2018-03-06 22:41:07] rm: remove regular file `.odaib'? y [2018-03-06 22:41:10] [root@wdcoda01cn01 conf]# touch .odaib [2018-03-06 22:41:13] [root@wdcoda01cn01 conf]# l s-al [2018-03-06 22:41:16] -bash: l: command not found [2018-03-06 22:41:16] [root@wdcoda01cn01 conf]# ls -al [2018-03-06 22:41:18] total 180 [2018-03-06 22:41:18] drwxr-xr-x 3 root root 4096 Mar 6 22:38 . [2018-03-06 22:41:18] drwxr-xr-x 14 root root 4096 Feb 28 16:46 .. [2018-03-06 22:41:18] -rw-r--r-- 1 root root 0 Feb 28 10:50 .clustered [2018-03-06 22:41:18] -rw-r--r-- 1 root root 0 Feb 28 10:50 .configlock [2018-03-06 22:41:18] -rw-r--r-- 1 root root 0 Mar 6 22:38 .odaib [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 1019 Feb 15 2017 70-persistent-nvme.rules [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 787 Feb 15 2017 99-oda_multipath.rules [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 3182 Feb 15 2017 99-oda_singleuser_multipath.rules [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 3332 Feb 15 2017 agentnodeConstants.py [2018-03-06 22:41:18] -rw-r--r-- 1 root root 2676 Feb 28 16:46 agentnodeConstants.pyc [2018-03-06 22:41:18] drwxrwxrwx 2 root root 4096 Feb 28 10:42 backupConfig [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 239 Feb 15 2017 dcliagent.conf [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 572 Feb 15 2017 discoverdummy.xml [2018-03-06 22:41:18] -rw-r--r-- 1 root root 3258 Mar 6 22:37 discoverjbod.xml [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 245 Feb 15 2017 initoak.conf [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 456 Feb 15 2017 logConfig.conf [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 61532 Feb 28 10:49 oak_conf.xml [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 31605 Feb 15 2017 oak_lite_conf.xml [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 258 Feb 15 2017 odaBaseAgent.conf [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 2160 Feb 15 2017 odalite_validation_props.xml [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 9812 Feb 15 2017 stm_defs.xml [2018-03-06 22:41:18] -rwxr-xr-x 1 root root 6859 Feb 15 2017 validation_props.xml [2018-03-06 22:41:18] [root@wdcoda01cn01 conf]# echo 1 > ib_fix [2018-03-06 22:41:35] [root@wdcoda01cn01 conf]# ls -al [2018-03-06 22:41:37] total 184 [2018-03-06 22:41:37] drwxr-xr-x 3 root root 4096 Mar 6 22:38 . [2018-03-06 22:41:37] drwxr-xr-x 14 root root 4096 Feb 28 16:46 .. [2018-03-06 22:41:37] -rw-r--r-- 1 root root 0 Feb 28 10:50 .clustered [2018-03-06 22:41:37] -rw-r--r-- 1 root root 0 Feb 28 10:50 .configlock [2018-03-06 22:41:37] -rw-r--r-- 1 root root 0 Mar 6 22:38 .odaib [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 1019 Feb 15 2017 70-persistent-nvme.rules [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 787 Feb 15 2017 99-oda_multipath.rules [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 3182 Feb 15 2017 99-oda_singleuser_multipath.rules [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 3332 Feb 15 2017 agentnodeConstants.py [2018-03-06 22:41:37] -rw-r--r-- 1 root root 2676 Feb 28 16:46 agentnodeConstants.pyc [2018-03-06 22:41:37] drwxrwxrwx 2 root root 4096 Feb 28 10:42 backupConfig [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 239 Feb 15 2017 dcliagent.conf [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 572 Feb 15 2017 discoverdummy.xml [2018-03-06 22:41:37] -rw-r--r-- 1 root root 3258 Mar 6 22:37 discoverjbod.xml [2018-03-06 22:41:37] -rw-r--r-- 1 root root 2 Mar 6 22:38 ib_fix [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 245 Feb 15 2017 initoak.conf [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 456 Feb 15 2017 logConfig.conf [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 61532 Feb 28 10:49 oak_conf.xml [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 31605 Feb 15 2017 oak_lite_conf.xml [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 258 Feb 15 2017 odaBaseAgent.conf [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 2160 Feb 15 2017 odalite_validation_props.xml [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 9812 Feb 15 2017 stm_defs.xml [2018-03-06 22:41:37] -rwxr-xr-x 1 root root 6859 Feb 15 2017 validation_props.xml [2018-03-06 22:41:37] [root@wdcoda01cn01 conf]# cat ib_fix [2018-03-06 22:41:41] 1 [2018-03-06 22:41:41] [root@wdcoda01cn01 conf]# cd ../install/ [2018-03-06 22:42:12] [root@wdcoda01cn01 install]# ls -al [2018-03-06 22:42:14] total 52 [2018-03-06 22:42:14] drwxr-xr-x 2 root root 4096 Mar 6 09:51 . [2018-03-06 22:42:14] drwxr-xr-x 14 root root 4096 Feb 28 16:46 .. [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 14698 Feb 15 2017 init.oak [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 66 Feb 15 2017 inittab [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 82 Feb 15 2017 inittab.baseagent [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 74 Feb 15 2017 inittab.dcli [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 2613 Feb 15 2017 oak2init.sh [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 6 Mar 6 09:56 oakdrun [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 1789 Feb 15 2017 oakpatch2init.sh [2018-03-06 22:42:14] -rwxr-xr-x 1 root root 1283 Feb 15 2017 patch.oak [2018-03-06 22:42:14] [root@wdcoda01cn01 install]# vi oakdrun [2018-03-06 22:42:18] "oakdrun" 1L, 6Cstart [2018-03-06 22:42:18] ~ :q [root@wdcoda01cn01 install]# [2018-03-06 22:47:14] [root@wdcoda01cn01 ~]# ifconfig [2018-03-06 22:47:18] eth0 Link encap:Ethernet HWaddr 00:16:3E:57:64:1B [2018-03-06 22:47:18] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:47:18] RX packets:1 errors:0 dropped:1 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:1000 [2018-03-06 22:47:18] RX bytes:231 (231.0 b) TX bytes:0 (0.0 b) [2018-03-06 22:47:18] [2018-03-06 22:47:18] eth1 Link encap:Ethernet HWaddr 00:16:3E:04:F2:8C [2018-03-06 22:47:18] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:47:18] RX packets:176 errors:0 dropped:1 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:1000 [2018-03-06 22:47:18] RX bytes:17731 (17.3 KiB) TX bytes:0 (0.0 b) [2018-03-06 22:47:18] [2018-03-06 22:47:18] eth3 Link encap:Ethernet HWaddr 00:16:3E:0A:29:36 [2018-03-06 22:47:18] inet addr:10.70.155.9 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:47:18] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:47:18] RX packets:298 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:168 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:1000 [2018-03-06 22:47:18] RX bytes:35344 (34.5 KiB) TX bytes:28897 (28.2 KiB) [2018-03-06 22:47:18] [2018-03-06 22:47:18] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:47:18] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:47:18] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:47:18] ibbond0 Link encap:InfiniBand HWaddr 80:00:08:51:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:47:18] inet addr:123.456.78.91 Bcast:123.456.78.255 Mask:255.255.255.0 [2018-03-06 22:47:18] UP BROADCAST RUNNING MASTER MULTICAST MTU:7000 Metric:1 [2018-03-06 22:47:18] RX packets:371 errors:0 dropped:12 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:95 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:0 [2018-03-06 22:47:18] RX bytes:259853 (253.7 KiB) TX bytes:21838 (21.3 KiB) [2018-03-06 22:47:18] [2018-03-06 22:47:18] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:47:18] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:47:18] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:47:18] ib0 Link encap:InfiniBand HWaddr 80:00:08:51:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:47:18] UP BROADCAST RUNNING SLAVE MULTICAST MTU:7000 Metric:1 [2018-03-06 22:47:18] RX packets:371 errors:0 dropped:12 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:95 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:4096 [2018-03-06 22:47:18] RX bytes:259853 (253.7 KiB) TX bytes:21838 (21.3 KiB) [2018-03-06 22:47:18] [2018-03-06 22:47:18] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:47:18] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:47:18] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:47:18] ib1 Link encap:InfiniBand HWaddr 80:00:08:52:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:47:18] UP BROADCAST RUNNING SLAVE MULTICAST MTU:7000 Metric:1 [2018-03-06 22:47:18] RX packets:0 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:4096 [2018-03-06 22:47:18] RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) [2018-03-06 22:47:18] [2018-03-06 22:47:18] lo Link encap:Local Loopback [2018-03-06 22:47:18] inet addr:127.0.0.1 Mask:255.0.0.0 [2018-03-06 22:47:18] UP LOOPBACK RUNNING MTU:16436 Metric:1 [2018-03-06 22:47:18] RX packets:150 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:150 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:0 [2018-03-06 22:47:18] RX bytes:20384 (19.9 KiB) TX bytes:20384 (19.9 KiB) [2018-03-06 22:47:18] [2018-03-06 22:47:18] veth1 Link encap:Ethernet HWaddr 00:16:3E:EA:1C:8C [2018-03-06 22:47:18] inet addr:123.456.78.20 Bcast:123.456.78.255 Mask:255.255.255.0 [2018-03-06 22:47:18] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:47:18] RX packets:62 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:47:18] TX packets:64 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:47:18] collisions:0 txqueuelen:1000 [2018-03-06 22:47:18] RX bytes:2448 (2.3 KiB) TX bytes:8105 (7.9 KiB) [2018-03-06 22:47:18] [2018-03-06 22:47:24] [root@wdcoda01cn01 ~]# [2018-03-06 22:47:25] [root@wdcoda01cn01 ~]# clear [2018-03-06 22:47:30] [root@wdcoda01cn01 ~]# oakcli show ismaster [2018-03-06 22:47:34] Unable to connect to oakd. [2018-03-06 22:47:35] [root@wdcoda01cn01 ~]# oakcli show ismaster [2018-03-06 22:48:50] OAKD is in Slave Mode [2018-03-06 22:48:50] [root@wdcoda01cn01 ~]# [2018-03-06 22:48:53] [root@wdcoda01cn01 ~]# [2018-03-06 22:48:54] [root@wdcoda01cn01 ~]# oakcli show vm [2018-03-06 22:48:57] [2018-03-06 22:48:58] NAME NODENUM MEMORY VCPU STATE REPOSITORY [2018-03-06 22:48:58] [2018-03-06 22:48:58] wdc1fin2utl01 0 2048M 2 ONLINE vmprod [2018-03-06 22:48:58] wdc2fin2app01 1 65536M 2 OFFLINE vmprod [2018-03-06 22:48:58] wdc2fin2app02 0 65536M 2 ONLINE vmprod [2018-03-06 22:48:58] wdc2fin2app03 0 16384M 2 ONLINE vmprod [2018-03-06 22:48:58] wdc2fin2app03_junk 0 2048M 2 OFFLINE templates [2018-03-06 22:48:58] wdc2fin2app04 0 16384M 2 ONLINE vmprod [2018-03-06 22:48:58] wdc2fin2app05 1 2048M 2 OFFLINE vmprod [2018-03-06 22:48:58] wdc2fin2app11 1 16384M 2 OFFLINE vmprod [2018-03-06 22:48:58] wdc2fin2app12 0 16384M 2 OFFLINE vmprod [2018-03-06 22:48:58] [2018-03-06 22:48:58] [2018-03-06 22:48:58] [root@wdcoda01cn01 ~]# ifconfig [2018-03-06 22:50:58] eth0 Link encap:Ethernet HWaddr 00:16:3E:57:64:1B [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] RX packets:5 errors:0 dropped:5 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:1000 [2018-03-06 22:50:58] RX bytes:1155 (1.1 KiB) TX bytes:0 (0.0 b) [2018-03-06 22:50:58] [2018-03-06 22:50:58] eth1 Link encap:Ethernet HWaddr 00:16:3E:04:F2:8C [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] RX packets:768 errors:0 dropped:5 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:1000 [2018-03-06 22:50:58] RX bytes:77455 (75.6 KiB) TX bytes:0 (0.0 b) [2018-03-06 22:50:58] [2018-03-06 22:50:58] eth3 Link encap:Ethernet HWaddr 00:16:3E:0A:29:36 [2018-03-06 22:50:58] inet addr:10.70.155.9 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] RX packets:1770 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:1085 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:1000 [2018-03-06 22:50:58] RX bytes:231709 (226.2 KiB) TX bytes:260281 (254.1 KiB) [2018-03-06 22:50:58] [2018-03-06 22:50:58] eth3:1 Link encap:Ethernet HWaddr 00:16:3E:0A:29:36 [2018-03-06 22:50:58] inet addr:10.70.155.11 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] [2018-03-06 22:50:58] eth3:2 Link encap:Ethernet HWaddr 00:16:3E:0A:29:36 [2018-03-06 22:50:58] inet addr:10.70.155.12 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] [2018-03-06 22:50:58] eth3:3 Link encap:Ethernet HWaddr 00:16:3E:0A:29:36 [2018-03-06 22:50:58] inet addr:12.34.567.89 Bcast:10.70.155.63 Mask:255.255.255.192 [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] [2018-03-06 22:50:58] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:50:58] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:50:58] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:50:58] ibbond0 Link encap:InfiniBand HWaddr 80:00:08:51:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:50:58] inet addr:123.456.78.91 Bcast:123.456.78.255 Mask:255.255.255.0 [2018-03-06 22:50:58] UP BROADCAST RUNNING MASTER MULTICAST MTU:7000 Metric:1 [2018-03-06 22:50:58] RX packets:1129914 errors:0 dropped:12 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:370453 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:0 [2018-03-06 22:50:58] RX bytes:3298605796 (3.0 GiB) TX bytes:304169248 (290.0 MiB) [2018-03-06 22:50:58] [2018-03-06 22:50:58] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:50:58] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:50:58] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:50:58] ibbond0:1 Link encap:InfiniBand HWaddr 80:00:08:51:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:50:58] inet addr:169.254.221.228 Bcast:169.254.255.255 Mask:255.255.0.0 [2018-03-06 22:50:58] UP BROADCAST RUNNING MASTER MULTICAST MTU:7000 Metric:1 [2018-03-06 22:50:58] [2018-03-06 22:50:58] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:50:58] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:50:58] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:50:58] ib0 Link encap:InfiniBand HWaddr 80:00:08:51:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:50:58] UP BROADCAST RUNNING SLAVE MULTICAST MTU:7000 Metric:1 [2018-03-06 22:50:58] RX packets:1129914 errors:0 dropped:12 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:370453 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:4096 [2018-03-06 22:50:58] RX bytes:3298605796 (3.0 GiB) TX bytes:304169248 (290.0 MiB) [2018-03-06 22:50:58] [2018-03-06 22:50:58] Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. [2018-03-06 22:50:58] Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. [2018-03-06 22:50:58] Ifconfig is obsolete! For replacement check ip. [2018-03-06 22:50:58] ib1 Link encap:InfiniBand HWaddr 80:00:08:52:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 [2018-03-06 22:50:58] UP BROADCAST RUNNING SLAVE MULTICAST MTU:7000 Metric:1 [2018-03-06 22:50:58] RX packets:0 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:4096 [2018-03-06 22:50:58] RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) [2018-03-06 22:50:58] [2018-03-06 22:50:58] lo Link encap:Local Loopback [2018-03-06 22:50:58] inet addr:127.0.0.1 Mask:255.0.0.0 [2018-03-06 22:50:58] UP LOOPBACK RUNNING MTU:16436 Metric:1 [2018-03-06 22:50:58] RX packets:65565 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:65565 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:0 [2018-03-06 22:50:58] RX bytes:30745584 (29.3 MiB) TX bytes:30745584 (29.3 MiB) [2018-03-06 22:50:58] [2018-03-06 22:50:58] veth1 Link encap:Ethernet HWaddr 00:16:3E:EA:1C:8C [2018-03-06 22:50:58] inet addr:123.456.78.20 Bcast:123.456.78.255 Mask:255.255.255.0 [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] RX packets:2904 errors:0 dropped:0 overruns:0 frame:0 [2018-03-06 22:50:58] TX packets:2732 errors:0 dropped:0 overruns:0 carrier:0 [2018-03-06 22:50:58] collisions:0 txqueuelen:1000 [2018-03-06 22:50:58] RX bytes:391213 (382.0 KiB) TX bytes:483948 (472.6 KiB) [2018-03-06 22:50:58] [2018-03-06 22:50:58] veth1:1 Link encap:Ethernet HWaddr 00:16:3E:EA:1C:8C [2018-03-06 22:50:58] inet addr:123.456.78.91 Bcast:123.456.78.255 Mask:255.255.255.0 [2018-03-06 22:50:58] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 [2018-03-06 22:50:58] [2018-03-06 22:50:58] [root@wdcoda01cn01 ~]# oakcli show vm [2018-03-06 22:51:16] [2018-03-06 22:51:16] NAME NODENUM MEMORY VCPU STATE REPOSITORY [2018-03-06 22:51:16] [2018-03-06 22:51:16] wdc1fin2utl01 0 2048M 2 ONLINE vmprod [2018-03-06 22:51:16] wdc2fin2app01 1 65536M 2 OFFLINE vmprod [2018-03-06 22:51:16] wdc2fin2app02 0 65536M 2 ONLINE vmprod [2018-03-06 22:51:16] wdc2fin2app03 0 16384M 2 ONLINE vmprod [2018-03-06 22:51:16] wdc2fin2app03_junk 0 2048M 2 OFFLINE templates [2018-03-06 22:51:16] wdc2fin2app04 0 16384M 2 ONLINE vmprod [2018-03-06 22:51:16] wdc2fin2app05 1 2048M 2 OFFLINE vmprod [2018-03-06 22:51:16] wdc2fin2app11 1 16384M 2 OFFLINE vmprod [2018-03-06 22:51:16] wdc2fin2app12 0 16384M 2 OFFLINE vmprod [2018-03-06 22:51:16] [2018-03-06 22:51:16] [2018-03-06 22:51:16] [root@wdcoda01cn01 ~]# oakcli start vm wdc2fin2app01 [2018-03-06 22:51:30] [2018-03-06 22:51:36] Started VM : wdc2fin2app01 on Node Number : 1 [2018-03-06 22:51:36] [2018-03-06 22:51:36] [2018-03-06 22:51:36] [root@wdcoda01cn01 ~]# oakcli start vm wdc2fin2app01 3 [2018-03-06 22:51:50] [2018-03-06 22:51:50] OAKERR : Resource wdc2fin2app03 is already ONLINE [2018-03-06 22:51:50] [2018-03-06 22:51:50] [2018-03-06 22:51:50] [root@wdcoda01cn01 ~]# oakcli start vm wdc2fin2app03 5 [2018-03-06 22:52:01] [2018-03-06 22:52:07] Started VM : wdc2fin2app05 on Node Number : 1 [2018-03-06 22:52:07] [2018-03-06 22:52:07] [2018-03-06 22:52:07] [root@wdcoda01cn01 ~]# oakcli start vm wdc2fin2app0531how vm [2018-03-06 22:52:29] [2018-03-06 22:52:29] NAME NODENUM MEMORY VCPU STATE REPOSITORY [2018-03-06 22:52:29] [2018-03-06 22:52:29] wdc1fin2utl01 0 2048M 2 ONLINE vmprod [2018-03-06 22:52:29] wdc2fin2app01 1 65536M 2 ONLINE vmprod [2018-03-06 22:52:29] wdc2fin2app02 0 65536M 2 ONLINE vmprod [2018-03-06 22:52:29] wdc2fin2app03 0 16384M 2 ONLINE vmprod [2018-03-06 22:52:29] wdc2fin2app03_junk 0 2048M 2 OFFLINE templates [2018-03-06 22:52:29] wdc2fin2app04 0 16384M 2 ONLINE vmprod [2018-03-06 22:52:29] wdc2fin2app05 1 2048M 2 ONLINE vmprod [2018-03-06 22:52:29] wdc2fin2app11 1 16384M 2 OFFLINE vmprod [2018-03-06 22:52:29] wdc2fin2app12 0 16384M 2 OFFLINE vmprod [2018-03-06 22:52:29] [2018-03-06 22:52:29] [2018-03-06 22:52:29] [root@wdcoda01cn01 ~]# [root@wdcoda01cn01 ~]# su - grid [2018-03-06 22:53:55] [grid@wdcoda01cn01 ~]$ oifcfg getif [2018-03-06 22:53:58] ibbond0 123.456.78.0 global cluster_interconnect,asm [2018-03-06 22:53:59] eth3 10.70.155.0 global public [2018-03-06 22:53:59] [grid@wdcoda01cn01 ~]$ exit [2018-03-06 22:54:17] logout [2018-03-06 22:54:17] [root@wdcoda01cn01 ~]# su - gridoakcli show vm [2018-03-06 22:55:17] [2018-03-06 22:55:17] NAME NODENUM MEMORY VCPU STATE REPOSITORY [2018-03-06 22:55:17] [2018-03-06 22:55:17] wdc1fin2utl01 0 2048M 2 ONLINE vmprod [2018-03-06 22:55:17] wdc2fin2app01 1 65536M 2 ONLINE vmprod [2018-03-06 22:55:17] wdc2fin2app02 0 65536M 2 ONLINE vmprod [2018-03-06 22:55:17] wdc2fin2app03 0 16384M 2 ONLINE vmprod [2018-03-06 22:55:17] wdc2fin2app03_junk 0 2048M 2 OFFLINE templates [2018-03-06 22:55:17] wdc2fin2app04 0 16384M 2 ONLINE vmprod [2018-03-06 22:55:17] wdc2fin2app05 1 2048M 2 ONLINE vmprod [2018-03-06 22:55:17] wdc2fin2app11 1 16384M 2 OFFLINE vmprod [2018-03-06 22:55:17] wdc2fin2app12 0 16384M 2 OFFLINE vmprod [2018-03-06 22:55:17] [2018-03-06 22:55:17] [2018-03-06 22:55:17] [root@wdcoda01cn01 ~]# oakcli show vmsu - gridoakcli show vmtart vm wdc2fin2app05 11 [2018-03-06 22:55:31] [2018-03-06 22:55:37] Started VM : wdc2fin2app11 on Node Number : 1 [2018-03-06 22:55:37] [2018-03-06 22:55:37] [2018-03-06 22:55:37] [2018-03-06 22:55:37] [root@wdcoda01cn01 ~]# oakcli show version -detail [2018-03-06 22:56:11] Reading the metadata. It takes a while... [2018-03-06 22:56:12] System Version Component Name Installed Version Supported Version [2018-03-06 22:57:25] -------------- --------------- ------------------ ----------------- [2018-03-06 22:57:25] 12.1.2.10.0 [2018-03-06 22:57:25] Controller_INT 4.650.00-7176 4.230.40-3739 [2018-03-06 22:57:32] Controller_EXT 4.650.00-7176 4.230.40-3739 [2018-03-06 22:57:38] Expander 0018 Up-to-date [2018-03-06 22:57:38] SSD_SHARED { [2018-03-06 22:57:38] [ c1d20,c1d21,c1d22, A29A Up-to-date [2018-03-06 22:57:38] c1d23 ] [2018-03-06 22:57:38] [ c1d16,c1d17,c1d18, A29A Up-to-date [2018-03-06 22:57:38] c1d19 ] [2018-03-06 22:57:38] } [2018-03-06 22:57:38] HDD_LOCAL A7E0 A72A [2018-03-06 22:57:38] HDD_SHARED PAG1 Up-to-date [2018-03-06 22:57:38] ILOM 3.2.9.23 r116695 3.2.8.24 r114580 [2018-03-06 22:57:38] BIOS 30110000 30100400 [2018-03-06 22:57:38] IPMI 1.8.12.4 Up-to-date [2018-03-06 22:57:38] HMP 2.3.5.2.8 Up-to-date [2018-03-06 22:57:38] OAK 12.1.2.10.0 Up-to-date [2018-03-06 22:57:38] OL 6.8 Up-to-date [2018-03-06 22:57:38] OVM 3.2.11 Up-to-date [2018-03-06 22:57:38] GI_HOME 12.1.0.2.170117(2473 Up-to-date [2018-03-06 22:57:38] 2082,24828633) [2018-03-06 22:57:38] DB_HOME 12.1.0.2.170117(2473 Up-to-date SESSION #2 [root@wdcoda01cn00 ~]# oakcli show ismaster [2018-03-06 22:34:59] OAKD is in Master Mode [2018-03-06 22:34:59] [root@wdcoda01cn00 ~]# ps -ef | grep oak [2018-03-06 22:35:18] root 2586 1 0 14:32 ? 00:00:00 /bin/sh /etc/init.d/init.oak run [2018-03-06 22:35:18] root 5805 1 0 14:33 ? 00:03:48 /bin/sh ./OSWatcher.sh 10 504 gzip /opt/oracle/oak/oswbb/archive [2018-03-06 22:35:18] root 6193 1 1 14:33 ? 00:05:17 /usr/bin/python /opt/oracle/oak/adapters/odaBaseAgent.py [2018-03-06 22:35:18] root 6195 1 0 14:33 ? 00:00:03 /usr/bin/python /opt/oracle/oak/bin/dcliagent.py [2018-03-06 22:35:18] root 6944 5805 0 14:34 ? 00:00:01 /bin/sh ./OSWatcherFM.sh 504 /opt/oracle/oak/oswbb/archive [2018-03-06 22:35:18] root 13277 2586 1 14:35 ? 00:09:28 /opt/oracle/oak/bin/oakd foreground [2018-03-06 22:35:18] root 84513 13277 0 22:35 ? 00:00:00 /bin/sh /opt/oracle/oak/adapters/SharedRepoAdapter.scr start [2018-03-06 22:35:18] root 84531 84513 3 22:35 ? 00:00:00 /usr/bin/python /opt/oracle/oak/adapters/oakVmClient.py -a startSharedRepo [2018-03-06 22:35:18] root 84728 13277 0 22:35 ? 00:00:00 /bin/sh /opt/oracle/oak/adapters/SharedRepoAdapter.scr check [2018-03-06 22:35:18] root 84734 84728 6 22:35 ? 00:00:00 /usr/bin/python /opt/oracle/oak/adapters/oakVmClient.py -a checkSharedRepo [2018-03-06 22:35:18] root 84932 51925 0 22:35 pts/0 00:00:00 grep oak [2018-03-06 22:35:18] [root@wdcoda01cn00 ~]# oakcli show repo [2018-03-06 22:35:33] [2018-03-06 22:35:33] NAME TYPE NODENUM FREE SPACESTATE SIZE [2018-03-06 22:35:33] [2018-03-06 22:35:33] [2018-03-06 22:35:33] odarepo1 local 0 N/AN/A N/A [2018-03-06 22:35:33] [2018-03-06 22:35:33] odarepo2 local 1 N/AN/A N/A [2018-03-06 22:35:33] [2018-03-06 22:35:33] templates shared 0 16.21%ONLINE 256000.0M [2018-03-06 22:35:33] [2018-03-06 22:35:33] templates shared 1 N/AOFFLINE N/A [2018-03-06 22:35:33] [2018-03-06 22:35:33] vmprod shared 0 5.00%ONLINE 1689600.0M [2018-03-06 22:35:33] [2018-03-06 22:35:33] vmprod shared 1 N/AOFFLINE N/A [2018-03-06 22:35:33] [2018-03-06 22:35:33] [2018-03-06 22:35:33] [root@wdcoda01cn00 ~]# [root@wdcoda01cn00 ~]# su - grid [2018-03-06 22:54:06] [grid@wdcoda01cn00 ~]$ oifcfg getif [2018-03-06 22:54:09] ibbond0 123.456.78.9 global cluster_interconnect,asm [2018-03-06 22:54:09] eth3 12.34.567.0 global public [2018-03-06 22:54:09] [grid@wdcoda01cn00 ~]$ exit [2018-03-06 22:54:14] logout SESSION #3 0[2018-03-06 22:34:33] [root@wdcoda01dom01 ~]# ps -ef | grep oak [2018-03-06 22:37:51] root 16870 15597 0 14:43 ? 00:00:32 /usr/lib64/xen/bin/qemu-dm -d 1 -domain-name oakDom1 -videoram 4 -vnc 0.0.0.0:0 -vncunused -vcpus 24 -vcpu_avail 0xffffff -boot c -serial pty -acpi -net none -M xenfv [2018-03-06 22:37:51] root 16898 1 0 14:43 ? 00:00:51 /usr/bin/python /opt/oracle/oak/bin/oakVmAgent.py run [2018-03-06 22:37:51] root 68874 68277 0 22:34 pts/2 00:00:00 grep oak [2018-03-06 22:37:51] [root@wdcoda01dom01 ~]# cd /opt/oracle/oak/log/ [2018-03-06 22:38:03] fishwrap/ oak2/ ovs-95-cb-b0/ wdcoda01dom01/ [2018-03-06 22:38:03] [root@wdcoda01dom01 ~]# cd /opt/oracle/oak/log/wdcoda01dom01/ [2018-03-06 22:38:09] [root@wdcoda01dom01 wdcoda01dom01]# ls -lrth [2018-03-06 22:38:12] total 16K [2018-03-06 22:38:12] drwxr-xr-x 2 root root 4.0K Feb 28 06:11 common_agentutils [2018-03-06 22:38:12] drwxr-xr-x 2 root root 4.0K Mar 1 18:27 client [2018-03-06 22:38:12] drwxr-xr-x 2 root root 4.0K Mar 6 11:16 oakvmagent [2018-03-06 22:38:12] drwxr-xr-x 2 root root 4.0K Mar 6 14:43 patch [2018-03-06 22:38:12] [root@wdcoda01dom01 wdcoda01dom01]# cd oakvmagent/ [2018-03-06 22:38:15] [root@wdcoda01dom01 oakvmagent]# ls -lrth [2018-03-06 22:38:16] total 78M [2018-03-06 22:38:16] -rw-r--r-- 1 root root 10M Mar 1 04:59 oakvmagent.log.7 [2018-03-06 22:38:16] -rw-r--r-- 1 root root 10M Mar 2 03:45 oakvmagent.log.6 [2018-03-06 22:38:16] -rw-r--r-- 1 root root 10M Mar 2 22:19 oakvmagent.log.5 [2018-03-06 22:38:16] -rw-r--r-- 1 root root 10M Mar 3 17:52 oakvmagent.log.4 [2018-03-06 22:38:16] -rw-r--r-- 1 root root 10M Mar 4 16:33 oakvmagent.log.3 [2018-03-06 22:38:16] -rw-r--r-- 1 root root 10M Mar 5 12:26 oakvmagent.log.2 [2018-03-06 22:38:16] -rw-r--r-- 1 root root 10M Mar 6 08:07 oakvmagent.log.1 [2018-03-06 22:38:16] -rw-r--r-- 1 root root 7.3M Mar 6 22:35 oakvmagent.log [2018-03-06 22:38:16] [root@wdcoda01dom01 oakvmagent]# vi oakvmagent.log [2018-03-06 22:38:22] "oakvmagent.log" "oakvmagent.log" 73869L, 7551198C2018-03-06 08:07:37,941 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:37,944 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:37,992 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:37,995 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:40,417 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:40,419 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:40,508 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:40,511 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:40,954 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:40,957 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:43,530 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:43,533 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:43,701 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:43,704 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:44,278 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:44,281 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:46,100 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:46,103 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:46,660 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:46,663 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:49,015 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:49,018 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:49,112 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:49,115 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:51,468 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:51,471 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:51,935 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:51,938 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:54,239 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:54,242 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:54,446 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:54,449 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:56,849 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:56,852 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:56,986 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:56,989 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:57,173 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:57,176 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:57,376 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:57,378 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:59,820 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:59,824 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:59,986 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:07:59,989 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:00,763 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:00,766 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:00,961 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:00,964 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:02,586 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:02,589 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:03,203 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:03,205 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:05,609 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:05,613 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:05,714 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:05,717 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:08,107 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:22] 2018-03-06 08:08:08,110 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos2018-03-06 22:34:46,458 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:46,461 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:49,434 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:49,437 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:49,526 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:49,530 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:50,987 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:50,990 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:52,302 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:52,305 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:52,425 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:52,429 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:54,709 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:54,713 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:54,846 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:54,849 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:57,581 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:57,584 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:57,681 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:57,684 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:59,876 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:59,880 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:59,969 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:34:59,972 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:02,751 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:02,754 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:02,845 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:02,849 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:05,062 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:05,065 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:05,155 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:05,159 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:08,005 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:08,008 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:08,096 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:08,099 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:08,189 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:08,192 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:10,684 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:10,687 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:10,805 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:10,808 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:13,587 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:13,590 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:13,680 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:13,684 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:15,944 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:15,947 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:16,037 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:16,040 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:18,875 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:18,878 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:18,967 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:18,970 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:21,168 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:21,171 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:21,262 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:25] 2018-03-06 22:35:21,265 [Cmd_EnvId] [MainThread] [repoactions] [INFO] [160] Checking for shared repos [2018-03-06 22:38:27] :q [root@wdcoda01dom01 oakvmagent]# cd /opt/oracle/oak/conf/ [2018-03-06 22:43:45] [root@wdcoda01dom01 conf]# ls -al [2018-03-06 22:43:46] total 76 [2018-03-06 22:43:46] drwxr-xr-x 2 root root 4096 Feb 28 09:20 . [2018-03-06 22:43:46] drwxr-xr-x 7 root root 4096 Feb 28 04:33 .. [2018-03-06 22:43:46] -rwxr-xr-x 1 root root 3332 Feb 15 2017 agentnodeConstants.py [2018-03-06 22:43:46] -rw-r--r-- 1 root root 2908 Feb 28 04:41 agentnodeConstants.pyc [2018-03-06 22:43:46] -rw-r--r-- 1 root root 566 Feb 28 09:17 dom0.xml [2018-03-06 22:43:46] -rw-r--r-- 1 root root 429 Feb 28 09:13 dom0.xml.orig [2018-03-06 22:43:46] -rw-r--r-- 1 root root 2 Feb 28 04:41 ib_fix [2018-03-06 22:43:46] -rwxr-xr-x 1 root root 456 Feb 15 2017 logConfig.conf [2018-03-06 22:43:46] -rw-r--r-- 1 root root 10 Feb 28 04:41 node_num.conf [2018-03-06 22:43:46] -rw-r--r-- 1 root root 12288 Mar 6 11:18 oakcpu.db [2018-03-06 22:43:46] -rw-r--r-- 1 root root 943 Mar 6 14:44 oakvlanNetwork.xml [2018-03-06 22:43:46] -rw-r--r-- 1 root root 24576 Mar 6 11:18 oakvmagentIp.db [2018-03-06 22:43:46] [root@wdcoda01dom01 conf]# cat ib_fix [2018-03-06 22:43:50] 1 [2018-03-06 22:43:50] [root@wdcoda01dom01 conf]# ls -al /etc/rc3.d/ [2018-03-06 22:44:29] total 8 [2018-03-06 22:44:29] drwxr-xr-x 2 root root 4096 Mar 3 08:53 . [2018-03-06 22:44:29] drwxr-xr-x 10 root root 4096 Feb 28 04:05 .. [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 22 Feb 28 04:33 K01ovm-consoled -> ../init.d/ovm-consoled [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 16 Feb 28 04:33 K01smartd -> ../init.d/smartd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:33 K05anacron -> ../init.d/anacron [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:33 K10hwmgmtd -> ../init.d/hwmgmtd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:33 K18ocfs2 -> ../init.d/ocfs2 [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 14 Feb 28 04:33 K19o2cb -> ../init.d/o2cb [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 13 Feb 28 04:14 K20nfs -> ../init.d/nfs [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 14 Feb 28 04:33 K30exim -> ../init.d/exim [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:33 K44rawdevices -> ../init.d/rawdevices [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 26 Feb 28 04:33 K50hmp-interconnect -> ../init.d/hmp-interconnect [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 18 Feb 28 04:33 K50hmp-ipmi -> ../init.d/hmp-ipmi [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:05 K50netconsole -> ../init.d/netconsole [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:04 K50snmpd -> ../init.d/snmpd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 19 Feb 28 04:04 K50snmptrapd -> ../init.d/snmptrapd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:14 K69rpcsvcgssd -> ../init.d/rpcsvcgssd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 14 Feb 28 04:01 K74nscd -> ../init.d/nscd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 14 Feb 28 04:02 K74ntpd -> ../init.d/ntpd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 18 Feb 28 03:58 K85ebtables -> ../init.d/ebtables [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:30 K85mdmpd -> ../init.d/mdmpd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 19 Feb 28 04:33 K85rpcidmapd -> ../init.d/rpcidmapd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 18 Feb 28 04:33 K87mcstrans -> ../init.d/mcstrans [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:33 K87multipathd -> ../init.d/multipathd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 21 Feb 28 04:33 K87restorecond -> ../init.d/restorecond [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:33 K89iscsi -> ../init.d/iscsi [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 16 Feb 28 04:33 K89iscsid -> ../init.d/iscsid [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 18 Feb 28 04:01 K89netplugd -> ../init.d/netplugd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 03:58 K89rdisc -> ../init.d/rdisc [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:09 K90xsigo -> ../init.d/xsigo [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 21 Feb 28 04:14 K91openvswitch -> ../init.d/openvswitch [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 19 Feb 28 03:58 K92ip6tables -> ../init.d/ip6tables [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 18 Feb 28 03:58 K92iptables -> ../init.d/iptables [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:04 K99oscap-scan -> ../init.d/oscap-scan [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 22 Feb 28 04:01 S02lvm2-monitor -> ../init.d/lvm2-monitor [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:33 S05openibd -> ../init.d/openibd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:05 S10network -> ../init.d/network [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 16 Feb 28 04:06 S12syslog -> ../init.d/syslog [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:07 S13irqbalance -> ../init.d/irqbalance [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:06 S13portmap -> ../init.d/portmap [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 13 Feb 28 04:01 S14mpp -> ../init.d/mpp [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:14 S14nfslock -> ../init.d/nfslock [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 19 Feb 28 04:30 S15mdmonitor -> ../init.d/mdmonitor [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:33 S15opensmd -> ../init.d/opensmd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 04:14 S19rpcgssd -> ../init.d/rpcgssd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:32 S20kdump -> ../init.d/kdump [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:04 S22messagebus -> ../init.d/messagebus [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:05 S25netfs -> ../init.d/netfs [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:06 S26acpid -> ../init.d/acpid [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 19 Feb 28 04:14 S26haldaemon -> ../init.d/haldaemon [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 14 Feb 28 04:33 S26ipmi -> ../init.d/ipmi [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 03:58 S26lm_sensors -> ../init.d/lm_sensors [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 17 Feb 28 03:58 S50mcelogd -> ../init.d/mcelogd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 14 Feb 28 04:07 S55sshd -> ../init.d/sshd [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:30 S70xencommons -> ../init.d/xencommons [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 15 Feb 28 04:06 S90crond -> ../init.d/crond [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 34 Mar 3 08:53 S98enablepci -> /opt/oracle/oak/tools/enablepci.py [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 14 Feb 28 04:30 S98xend -> ../init.d/xend [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 11 Feb 28 04:05 S99local -> ../rc.local [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 18 Feb 28 04:07 S99ovmwatch -> ../init.d/ovmwatch [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 19 Feb 28 04:04 S99ovs-agent -> ../init.d/ovs-agent [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:07 S99ovs-devmon -> ../init.d/ovs-devmon [2018-03-06 22:44:29] lrwxrwxrwx 1 root root 20 Feb 28 04:30 S99xendomains -> ../init.d/xendomains [2018-03-06 22:44:29] [root@wdcoda01dom01 conf]# ls -al /etc/xen/auto/oakDom1vm.cfg [2018-03-06 22:44:52] lrwxrwxrwx 1 root root 60 Mar 5 11:06 /etc/xen/auto/oakDom1vm.cfg -> /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/vm.cfg [2018-03-06 22:44:52] [root@wdcoda01dom01 conf]# oakcli restart oda_base [2018-03-06 22:45:08] INFO: Stopping ODA base domain... [2018-03-06 22:45:25] INFO: Stopping all the shared repos [2018-03-06 22:45:25] INFO: Starting ODA base domain... [2018-03-06 22:45:34] INFO: Started ODA base domain [2018-03-06 22:45:41] [root@wdcoda01dom01 conf]# vi /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/vm.cfg [2018-03-06 22:46:01] "/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/vm.cfg" "/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/vm.cfg" 23L, 1140Ckernel = '/usr/lib/xen/boot/hvmloader' [2018-03-06 22:46:01] vif = ['type=netfront,bridge=net1', 'type=netfront,bridge=net2', 'type=netfront,bridge=vbr1,mac=00:16:3e:ea:1c:8c', 'type=netfront,bridge=vlan2629', 'type=netfront,bridge=vllan2629', 'type=netfront,bridge=vlan2630', 'type=netfront,bridge=vlan2630', 'type=netfront,bridge=vlan2628', 'type=netfront,bridge=vlan2628'] [2018-03-06 22:46:01] vnclisten = '0.0.0.0' [2018-03-06 22:46:01] cpus = '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23' [2018-03-06 22:46:01] ib_pfs = ['90:00.0'] [2018-03-06 22:46:01] serial = 'pty' [2018-03-06 22:46:01] disk = ['file:/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/System.img,xvda,w', 'file:/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/u01.img,xvdb,w', 'file:/OOVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/swap.img,xvdc,w'] [2018-03-06 22:46:01] vncunused = 1 [2018-03-06 22:46:01] uuid = '683c55ca-a6a3-4dd7-bb39-10c6e76cfefb' [2018-03-06 22:46:01] on_reboot = 'restart' [2018-03-06 22:46:01] pci = ['03:00.0@15', '13:00.0@16'] [2018-03-06 22:46:01] pae = 1 [2018-03-06 22:46:01] memory = 98304 [2018-03-06 22:46:01] acpi = 1 [2018-03-06 22:46:01] timer_mode = 0 [2018-03-06 22:46:01] vnc = 1 [2018-03-06 22:46:01] on_crash = 'coredump-restart' [2018-03-06 22:46:01] apic = 1 [2018-03-06 22:46:01] ib_pkeys = [{'pkey': ['0xffff'], 'pf': '90:00.0', 'port': '1'}, {'pkey': ['0xffff'], 'pf': '90:00.0', 'port': '2'}] [2018-03-06 22:46:01] name = 'oakDom1' [2018-03-06 22:46:01] device_model = '/usr/lib64/xen/bin/qemu-dm' [2018-03-06 22:46:01] builder = 'hvm' [2018-03-06 22:46:01] vcpus = 24 [2018-03-06 22:46:01] ~ :q [root@wdcoda01dom01 conf]# vi /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/vm.cfg ls -al [2018-03-06 22:46:15] total 16 [2018-03-06 22:46:15] drwxr-xr-x 4 root root 4096 Mar 1 13:36 . [2018-03-06 22:46:15] drwxr-xr-x 3 root root 4096 Feb 28 06:44 .. [2018-03-06 22:46:15] lrwxrwxrwx 1 root root 8 Feb 28 10:26 683c55ca-a6a3-4dd7-bb39-10c6e76cfefb -> oakDom1/ [2018-03-06 22:46:15] drwxr-xr-x 2 root root 4096 Mar 1 20:28 bjf [2018-03-06 22:46:15] drwxr-xr-x 2 root root 4096 Mar 6 22:43 oakDom1 [2018-03-06 22:46:15] [root@wdcoda01dom01 conf]# [2018-03-06 22:46:18] [root@wdcoda01dom01 conf]# [2018-03-06 22:46:18] [root@wdcoda01dom01 conf]# clear [2018-03-06 22:46:19] [root@wdcoda01dom01 conf]# ping 123.456.78.91 [2018-03-06 22:46:23] PING 123.456.78.91 (123.456.78.91) 56(84) bytes of data. [2018-03-06 22:46:23] From 123.456.78.91 icmp_seq=2 Destination Host Unreachable [2018-03-06 22:46:26] From 123.456.78.91 icmp_seq=3 Destination Host Unreachable [2018-03-06 22:46:26] From 123.456.78.91 icmp_seq=4 Destination Host Unreachable [2018-03-06 22:46:26] From 123.456.78.91 icmp_seq=6 Destination Host Unreachable [2018-03-06 22:46:31] From 123.456.78.91 icmp_seq=7 Destination Host Unreachable [2018-03-06 22:46:31] From 123.456.78.91 icmp_seq=8 Destination Host Unreachable [2018-03-06 22:46:31] From 123.456.78.91 icmp_seq=10 Destination Host Unreachable [2018-03-06 22:46:35] From 123.456.78.91 icmp_seq=11 Destination Host Unreachable [2018-03-06 22:46:35] From 123.456.78.91 icmp_seq=12 Destination Host Unreachable [2018-03-06 22:46:35] From 123.456.78.91 icmp_seq=14 Destination Host Unreachable [2018-03-06 22:46:39] From 123.456.78.91 icmp_seq=15 Destination Host Unreachable [2018-03-06 22:46:39] From 123.456.78.91 icmp_seq=16 Destination Host Unreachable [2018-03-06 22:46:39] From 123.456.78.91 icmp_seq=18 Destination Host Unreachable [2018-03-06 22:46:43] From 123.456.78.91 icmp_seq=19 Destination Host Unreachable [2018-03-06 22:46:43] From 123.456.78.91 icmp_seq=20 Destination Host Unreachable [2018-03-06 22:46:43] From 123.456.78.91 icmp_seq=22 Destination Host Unreachable [2018-03-06 22:46:47] From 123.456.78.91 icmp_seq=23 Destination Host Unreachable [2018-03-06 22:46:47] From 123.456.78.91 icmp_seq=24 Destination Host Unreachable [2018-03-06 22:46:47] 64 bytes from 123.456.78.91: icmp_seq=25 ttl=64 time=1355 ms [2018-03-06 22:46:49] 64 bytes from 123.456.78.91: icmp_seq=26 ttl=64 time=354 ms [2018-03-06 22:46:49] 64 bytes from 123.456.78.91: icmp_seq=27 ttl=64 time=0.219 ms [2018-03-06 22:46:50] 64 bytes from 123.456.78.91: icmp_seq=28 ttl=64 time=0.245 ms [2018-03-06 22:46:51] 64 bytes from 123.456.78.91: icmp_seq=29 ttl=64 time=0.230 ms [2018-03-06 22:46:52] 64 bytes from 123.456.78.91: icmp_seq=30 ttl=64 time=0.254 ms [2018-03-06 22:46:53] 64 bytes from 123.456.78.91: icmp_seq=31 ttl=64 time=0.182 ms [2018-03-06 22:46:54] 64 bytes from 123.456.78.91: icmp_seq=32 ttl=64 time=0.245 ms [2018-03-06 22:46:55] 64 bytes from 123.456.78.91: icmp_seq=33 ttl=64 time=0.252 ms [2018-03-06 22:46:56] 64 bytes from 123.456.78.91: icmp_seq=34 ttl=64 time=0.117 ms [2018-03-06 22:46:57] 64 bytes from 123.456.78.91: icmp_seq=35 ttl=64 time=0.183 ms [2018-03-06 22:46:58] 64 bytes from 123.456.78.91: icmp_seq=36 ttl=64 time=0.275 ms [2018-03-06 22:46:59] 64 bytes from 123.456.78.91: icmp_seq=37 ttl=64 time=0.241 ms [2018-03-06 22:47:00] 64 bytes from 123.456.78.91: icmp_seq=38 ttl=64 time=0.162 ms [2018-03-06 22:47:01] ^C [2018-03-06 22:47:01] --- 123.456.78.91 ping statistics --- [2018-03-06 22:47:01] 38 packets transmitted, 14 received, +18 errors, 63% packet loss, time 37010ms [2018-03-06 22:47:01] rtt min/avg/max/mdev = 0.117/122.350/1355.627/353.952 ms, pipe 3 [2018-03-06 22:47:01] [root@wdcoda01dom01 conf]# [2018-03-06 22:47:03] [root@wdcoda01dom01 conf]# [2018-03-06 22:47:03] [root@wdcoda01dom01 conf]# mount [2018-03-06 22:49:38] /dev/sda3 on / type ext3 (rw) [2018-03-06 22:49:38] proc on /proc type proc (rw) [2018-03-06 22:49:38] sysfs on /sys type sysfs (rw) [2018-03-06 22:49:38] devpts on /dev/pts type devpts (rw,gid=5,mode=620) [2018-03-06 22:49:38] /dev/sda2 on /OVS type ext3 (rw) [2018-03-06 22:49:38] /dev/sda1 on /boot type ext3 (rw) [2018-03-06 22:49:38] tmpfs on /dev/shm type tmpfs (rw) [2018-03-06 22:49:38] debugfs on /sys/kernel/debug type debugfs (rw) [2018-03-06 22:49:38] xenfs on /proc/xen type xenfs (rw) [2018-03-06 22:49:38] none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) [2018-03-06 22:49:38] sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) [2018-03-06 22:49:38] none on /var/lib/xenstored type tmpfs (rw) [2018-03-06 22:49:38] 123.456.78.91:/u01/app/sharedrepo/templates on /OVS/Repositories/templates type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=123.456.78.91) [2018-03-06 22:49:38] 123.456.78.91:/u01/app/sharedrepo/vmprod on /OVS/Repositories/vmprod type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=123.456.78.91) [2018-03-06 22:49:38] [root@wdcoda01dom01 conf]# [root@wdcoda01dom01 conf]# |
|
Administrator
|
Hi Colby,
Thanks for the SSH terminal outputs. It is hard to see everything there, but If I m not seeing wrong, the problem solved after restarting the oda_base, which may require a downtime. So your oakd was running but it was not functional. Just like recommended in the following blog post: http://ermanarslan.blogspot.com.tr/2016/02/oda-virtualized-odabase-oakd-oakcli.html There is one exception in your case, as your oakd was already running.. But, I also see those .odaib file-related actions in your command outputs. On the other hand, oda base was restarted just after these actions.. So I don't think .odaib file-related actions were the solution for this. So I believe, your issue solved by restarting oda_base. What do you say? |
|
|
Erman,
I would agree with it fixing the issue after starting the oda_base... however we tried that several times... I know the ssh logs aren't that clear I apologize.. I was even trying to track down what exactly happened, but all i have to go off of is the logs supplied. |
«
Return to Oracle Database Appliance(ODA) & VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |