Erman Arslan's Oracle Forum - Re: X5-2 ODA issues

Erman Arslan's Oracle Forum › Oracle Database Appliance(ODA) & VM

Re: X5-2 ODA issues

Posted by alexb on Mar 06, 2018; 10:15pm
URL: http://erman-arslan-s-oracle-forum.124.s1.nabble.com/X5-2-ODA-issues-tp5322p5330.html

I had a similar issue on an X5-2HA. Tried to patch and the system ended up getting trashed. In addition before this, we were running into issues where occasionally a running VM would become hung - couldn't login it -- the only solution was to revert to a windows 'fix' (recycle the power). The consideration to patch was hoping that it would fix the hanging VM problem.

Anyway after attempting to patch, it all went to sh*t.

Here's what we were seeing on the server:
root@tpin-oda1 ~]# oakcli show repo

NAME TYPE NODENUM FREE SPACE STATE SIZE

repo shared 0 N/A UNKNOWN N/A

repo shared 1 99.96% ONLINE 5120000.0M

odarepo1 local 0 N/A N/A N/A

odarepo2 local 1 N/A N/A N/A

shared shared 0 N/A UNKNOWN N/A

shared shared 1 32.37% ONLINE 204800.0M

And..
[oracle@oda0 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Feb 14 10:03:33 2018

Copyright (c) 1982, 2014, Oracle. All rights reserved.

ERROR:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925

But when I ran the df -h command, there was plenty of disk space....

The fix
====
Turns out the system had exhausted the available iNodes:
root@oda0 ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on

/dev/xvdb1 6111232 611232 0 100% /u01

This could be quickly cleaned up by running:
cd /u01/app/12.1.0.2/grid/rdbms/audit/
find . -mtime +30 -exec ls -alR|wc -l\;

But the 'ultimate' solution was to run the purgeODALog.sh script and finally to place a call to the script into cron:

[root@oda0 ~]# crontab -l
00 02 * * * /opt/oracle/oak/bin/purgeODALog.sh -orcl 20 -tfa 10 -osw 10 -oak 10

Since then, rock solid. Never a problem with hanging VMs and I'm thinking that I might even be wanting to patch the system just to stay on the current release.