Disk addition to diskgroup

[satish]

Hi erman,

We are following note id How To Add a New Disk(s) to An Existing Diskgroup on RAC Cluster or Standalone ASM Configuration (Best Practices). (Doc ID 557348.1) to add disks to existing DATA diskgroup.
We are using ASMLIB.Are below steps mandatory to perfrom below step 2 and step 3

2) Confirm the disks are visible (I/O) at OS level (as oracle or grid OS user) from each node:

$> id
$> ls -l <disk name> 
$> dd if=<raw/character or block partition device in question> of=/dev/null count=100 bs=8192

Example:

$> dd if=/dev/raw/raw1 of=/dev/null count=100 bs=8192 # raw device
$> dd if=/dev/oracleasm/disks/PRODA07 of=/dev/null count=100 bs=8192 # ASMLIB device
$> dd if=/dev/rdsk/c1t3d0s6 of=/dev/null count=100 bs=8192 # raw device

3) Verify that kfed can read on the candidate disk(s) at OS level (as oracle or grid OS user) from each node:

$> kfed read <raw/character or block partition device in question> 
Example:
$> kfed read /dev/raw/raw1
$> kfed read /dev/oracleasm/disks/PRODA07
$> kfed read /dev/rdsk/c1t3d0s6
Note: If the disk is not a FORMER disk, then kfed on new disks will return: kfbh.type: 0 ; 0x002: KFBTYP_INVALID and that is expected.


SQL> select GROUP_NUMBER,DISK_NUMBER,STATE,name,path,MOUNT_STATUS,HEADER_STATUS from v$asm_disk;

GROUP_NUMBER DISK_NUMBER STATE NAME PATH MOUNT_S HEADER_STATU
------------ ----------- -------- ------------------------------ ----------------------------------- ------- ------------
2 0 NORMAL FRADISK1 ORCL:ARCHDISK1 CACHED MEMBER
2 1 NORMAL FRADISK2 ORCL:ARCHDISK2 CACHED MEMBER
2 2 NORMAL FRADISK3 ORCL:ARCHDISK3 CACHED MEMBER
2 3 NORMAL FRADISK4 ORCL:ARCHDISK4 CACHED MEMBER
2 4 NORMAL FRADISK5 ORCL:ARCHDISK5 CACHED MEMBER
1 0 NORMAL DATADISK1 ORCL:DATADISK1 CACHED MEMBER
1 1 NORMAL DATADISK2 ORCL:DATADISK2 CACHED MEMBER
1 2 NORMAL DATADISK3 ORCL:DATADISK3 CACHED MEMBER
1 3 NORMAL DATADISK4 ORCL:DATADISK4 CACHED MEMBER
1 4 NORMAL DATADISK5 ORCL:DATADISK5 CACHED MEMBER
1 5 NORMAL DATADISK6 ORCL:DATADISK6 CACHED MEMBER
1 6 NORMAL DATADISK7 ORCL:DATADISK7 CACHED MEMBER
1 7 NORMAL DATADISK8 ORCL:DATADISK8 CACHED MEMBER
3 0 NORMAL OCRVOTEDISK1 ORCL:OCRVOTEDISK1 CACHED MEMBER
3 1 NORMAL OCRVOTEDISK2 ORCL:OCRVOTEDISK2 CACHED MEMBER
3 2 NORMAL OCRVOTEDISK3 ORCL:OCRVOTEDISK3 CACHED MEMBER
3 3 NORMAL OCRVOTEDISK4 ORCL:OCRVOTEDISK4 CACHED MEMBER
3 4 NORMAL OCRVOTEDISK5 ORCL:OCRVOTEDISK5 CACHED MEMBER
3 5 NORMAL OCRVOTEDISK6 ORCL:OCRVOTEDISK6 CACHED MEMBER
4 0 NORMAL REDO1ADISK1 ORCL:REDO1DISK1 CACHED MEMBER
5 0 NORMAL REDO1DISK10 ORCL:REDO1DISK10 CACHED MEMBER
4 1 NORMAL REDO1ADISK2 ORCL:REDO1DISK2 CACHED MEMBER
4 2 NORMAL REDO1ADISK3 ORCL:REDO1DISK3 CACHED MEMBER
4 3 NORMAL REDO1ADISK4 ORCL:REDO1DISK4 CACHED MEMBER
4 4 NORMAL REDO1ADISK5 ORCL:REDO1DISK5 CACHED MEMBER
5 1 NORMAL REDO1DISK6 ORCL:REDO1DISK6 CACHED MEMBER
5 2 NORMAL REDO1DISK7 ORCL:REDO1DISK7 CACHED MEMBER
5 3 NORMAL REDO1DISK8 ORCL:REDO1DISK8 CACHED MEMBER
5 4 NORMAL REDO1DISK9 ORCL:REDO1DISK9 CACHED MEMBER
6 0 NORMAL REDO2DISK1 ORCL:REDO2DISK1 CACHED MEMBER
7 0 NORMAL REDO2DISK10 ORCL:REDO2DISK10 CACHED MEMBER
6 1 NORMAL REDO2DISK2 ORCL:REDO2DISK2 CACHED MEMBER
6 2 NORMAL REDO2DISK3 ORCL:REDO2DISK3 CACHED MEMBER
6 3 NORMAL REDO2DISK4 ORCL:REDO2DISK4 CACHED MEMBER
6 4 NORMAL REDO2DISK5 ORCL:REDO2DISK5 CACHED MEMBER
7 1 NORMAL REDO2DISK6 ORCL:REDO2DISK6 CACHED MEMBER
7 2 NORMAL REDO2DISK7 ORCL:REDO2DISK7 CACHED MEMBER
7 3 NORMAL REDO2DISK8 ORCL:REDO2DISK8 CACHED MEMBER
7 4 NORMAL REDO2DISK9 ORCL:REDO2DISK9 CACHED MEMBER
0 16 NORMAL ORCL:DATADISK11 CLOSED PROVISIONED
0 15 NORMAL ORCL:DATADISK19 CLOSED PROVISIONED
0 14 NORMAL ORCL:DATADISK15 CLOSED PROVISIONED
0 13 NORMAL ORCL:DATADISK24 CLOSED PROVISIONED
0 12 NORMAL ORCL:DATADISK20 CLOSED PROVISIONED
0 11 NORMAL ORCL:DATADISK14 CLOSED PROVISIONED
0 10 NORMAL ORCL:DATADISK12 CLOSED PROVISIONED
0 9 NORMAL ORCL:DATADISK18 CLOSED PROVISIONED
0 8 NORMAL ORCL:DATADISK22 CLOSED PROVISIONED
0 7 NORMAL ORCL:DATADISK9 CLOSED PROVISIONED
0 6 NORMAL ORCL:DATADISK16 CLOSED PROVISIONED
0 5 NORMAL ORCL:DATADISK10 CLOSED PROVISIONED
0 4 NORMAL ORCL:DATADISK23 CLOSED PROVISIONED
0 3 NORMAL ORCL:DATADISK21 CLOSED PROVISIONED
0 2 NORMAL ORCL:DATADISK17 CLOSED PROVISIONED
0 1 NORMAL ORCL:DATADISK25 CLOSED PROVISIONED
0 0 NORMAL ORCL:DATADISK13 CLOSED PROVISIONED

56 rows selected.

SQL>


On Node 1:
=========

[root@erpproddb1 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@erpproddb1 ~]# id
uid=0(root) gid=0(root) groups=0(root)
[root@erpproddb1 ~]# /etc/init.d/oracleasm listdisks
ARCHDISK1
ARCHDISK2
ARCHDISK3
ARCHDISK4
ARCHDISK5
DATADISK1
DATADISK10
DATADISK11
DATADISK12
DATADISK13
DATADISK14
DATADISK15
DATADISK16
DATADISK17
DATADISK18
DATADISK19
DATADISK2
DATADISK20
DATADISK21
DATADISK22
DATADISK23
DATADISK24
DATADISK25
DATADISK3
DATADISK4
DATADISK5
DATADISK6
DATADISK7
DATADISK8
DATADISK9
OCRVOTEDISK1
OCRVOTEDISK2
OCRVOTEDISK3
OCRVOTEDISK4
OCRVOTEDISK5
OCRVOTEDISK6
REDO1DISK1
REDO1DISK10
REDO1DISK2
REDO1DISK3
REDO1DISK4
REDO1DISK5
REDO1DISK6
REDO1DISK7
REDO1DISK8
REDO1DISK9
REDO2DISK1
REDO2DISK10
REDO2DISK2
REDO2DISK3
REDO2DISK4
REDO2DISK5
REDO2DISK6
REDO2DISK7
REDO2DISK8
REDO2DISK9
[root@erpproddb1 ~]#

[root@erpproddb1 ~]# /etc/init.d/oracleasm querydisk DATADISK9
Disk "DATADISK9" is a valid ASM disk




On Node 2:
==========
[root@erpproddb2 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@erpproddb2 ~]# id
uid=0(root) gid=0(root) groups=0(root)
[root@erpproddb2 ~]# /etc/init.d/oracleasm listdisks
ARCHDISK1
ARCHDISK2
ARCHDISK3
ARCHDISK4
ARCHDISK5
DATADISK1
DATADISK10
DATADISK11
DATADISK12
DATADISK13
DATADISK14
DATADISK15
DATADISK16
DATADISK17
DATADISK18
DATADISK19
DATADISK2
DATADISK20
DATADISK21
DATADISK22
DATADISK23
DATADISK24
DATADISK25
DATADISK3
DATADISK4
DATADISK5
DATADISK6
DATADISK7
DATADISK8
DATADISK9
OCRVOTEDISK1
OCRVOTEDISK2
OCRVOTEDISK3
OCRVOTEDISK4
OCRVOTEDISK5
OCRVOTEDISK6
REDO1DISK1
REDO1DISK10
REDO1DISK2
REDO1DISK3
REDO1DISK4
REDO1DISK5
REDO1DISK6
REDO1DISK7
REDO1DISK8
REDO1DISK9
REDO2DISK1
REDO2DISK10
REDO2DISK2
REDO2DISK3
REDO2DISK4
REDO2DISK5
REDO2DISK6
REDO2DISK7
REDO2DISK8
REDO2DISK9
[root@erpproddb2 ~]# /etc/init.d/oracleasm querydisk DATADISK9
Disk "DATADISK9" is a valid ASM disk
[root@erpproddb2 ~]#


Thanks

[satish]

CREATE DISKGROUP TEST123 EXTERNAL REDUNDANCY DISK 'ORCL:DATADISK9';
CREATE DISKGROUP TEST123 EXTERNAL REDUNDANCY DISK 'ORCL:DATADISK9'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification 'ORCL:DATADISK9' matches no disks


SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[griduat@erpuatdb1 ~]$ cd /dev/oracleasm/disks/
[griduat@erpuatdb1 disks]$ ls
ARCH01 ARCH05 DATA04 DATA08 DATA12 DATA16 DATA20 DATA24 OCRVOTEDISK3 REDO1DISK1 REDO1DISK4 REDO1DISK8 REDO2DISK2 REDO2DISK6
ARCH02 DATA01 DATA05 DATA09 DATA13 DATA17 DATA21 DATA25 OCRVOTEDISK4 REDO1DISK10 REDO1DISK5 REDO1DISK9 REDO2DISK3 REDO2DISK7
ARCH03 DATA02 DATA06 DATA10 DATA14 DATA18 DATA22 OCRVOTEDISK1 OCRVOTEDISK5 REDO1DISK2 REDO1DISK6 REDO2DISK1 REDO2DISK4 REDO2DISK8
ARCH04 DATA03 DATA07 DATA11 DATA15 DATA19 DATA23 OCRVOTEDISK2 OCRVOTEDISK6 REDO1DISK3 REDO1DISK7 REDO2DISK10 REDO2DISK5 REDO2DISK9
[griduat@erpuatdb1 disks]$ sqlplus "/as sysasm";

SQL*Plus: Release 12.1.0.2.0 Production on Tue Nov 9 11:50:42 2021

Copyright (c) 1982, 2014, Oracle. All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL>
SQL> CREATE DISKGROUP TEST123 EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/DATA09';
CREATE DISKGROUP TEST123 EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/DATA09'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/disks/DATA09' matches no disks
ORA-15014: path '/dev/oracleasm/disks/DATA09' is not in the discovery set


SQL>
SQL> sho parameter dis

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string OCRVOTE1, REDO1ADG, REDO1BDG,
FRADG, REDO2ADG, REDO2BDG, DAT
ADG
asm_diskstring string ORCL:*
SQL>

[ErmanArslansOracleBlog]

Hi,

did you check the following MOS note?

Disk Is not Discovered in ASM, Diskgroup Creation Fails with Ora-15018 Ora-15031 Ora-15014 (Doc ID 431013.1)

[satish]

Hi erman,

We are on 12.1.0.2,Can we ignore this error and move back..

[gridprod@erpproddb2 ~]$ kfed read /dev/oracleasm/disks/DATADISK9
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 168233017 ; 0x00c: 0x0a070839
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
000000000 00000000 00000000 00000000 0A070839 [............9...]
000000010 00000000 00000000 00000000 00000000 [................]
000000020 4C43524F 4B534944 41544144 4B534944 [ORCLDISKDATADISK]
000000030 00000039 00000000 00000000 00000000 [9...............]
000000040 00000000 00000000 00000000 00000000 [................]
Repeat 251 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[gridprod@erpproddb2 ~]$

[ErmanArslansOracleBlog]

What do you mean by "ignore and move back"?

I already sent you an update on this one.

Seems like there is some data on that disk remaining from previous usage.

Did you take the necessary actions as given in the MOS note named -> Disk Is not Discovered in ASM, Diskgroup Creation Fails with Ora-15018 Ora-15031 Ora-15014 (Doc ID-431013.1)

Did you check the following? ->

KFED Is Reporting KFED-00322 [Invalid OSM block type] On Healthy ASM Member Disks. (Doc ID 1624223.1)