read-only file system error

[Dan]

Hi Erman

I have a question about my linux disk which gives many times ready-only file system error.
Meaning many times is after fixing it with command umount -l /ax4 and "mount /dev/mapper/VolGrpYedek-yedekvol   /ax4 -t ext3"  it occurs again.

Is there a physical damage on my disk or Can I fix it with Linux cd or something like that ?

[ErmanArslansOracleBlog]

Hi,

Please send me the OS logs representing the issue, as well as dmesg output.

[CG]

here is the dmesg output and there are log files in the attachment

CIFS VFS: No response for cmd 50 mid 28761
end_request: I/O error, dev sdq, sector 0
printk: 79 messages suppressed.
Buffer I/O error on device sdq, logical block 0
Buffer I/O error on device sdq, logical block 1varlogmessages.rar
Buffer I/O error on device sdq, logical block 2
Buffer I/O error on device sdq, logical block 3
end_request: I/O error, dev sdq, sector 0
Buffer I/O error on device sdq, logical block 0
end_request: I/O error, dev sdq, sector 2097024
end_request: I/O error, dev sdq, sector 2097136
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdq, sector 8
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdr, sector 0
Buffer I/O error on device sdr, logical block 0
Buffer I/O error on device sdr, logical block 1
Buffer I/O error on device sdr, logical block 2
Buffer I/O error on device sdr, logical block 3
end_request: I/O error, dev sdr, sector 0
Buffer I/O error on device sdr, logical block 0
end_request: I/O error, dev sdr, sector 2097024
end_request: I/O error, dev sdr, sector 2097136
end_request: I/O error, dev sdr, sector 0
end_request: I/O error, dev sdr, sector 8
end_request: I/O error, dev sdr, sector 0
cciss: cmd f7600250 has CHECK CONDITION sense key = 0x4
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdr, sector 0
cciss: cmd f7600000 has CHECK CONDITION sense key = 0x4
usb 4-1: USB disconnect, address 2
usb 4-1: new low speed USB device using uhci_hcd and address 3
usb 4-1: configuration #1 chosen from 1 choice
input: shengxiangzuo LANBE_KVM47 as /class/input/input5
input: USB HID v1.11 Keyboard [shengxiangzuo LANBE_KVM47] on usb-0000:00:1d.2-1
input: shengxiangzuo LANBE_KVM47 as /class/input/input6
input: USB HID v1.11 Mouse [shengxiangzuo LANBE_KVM47] on usb-0000:00:1d.2-1
ext3_abort called.
EXT3-fs error (device dm-20): ext3_put_super: Couldn't clean up the journal
kjournald starting.  Commit interval 5 seconds
EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS on dm-20, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
cciss: cmd f7600000 has CHECK CONDITION sense key = 0x4
Buffer I/O error on device cciss/c0d1, logical block 0
Buffer I/O error on device cciss/c0d1, logical block 1
Buffer I/O error on device cciss/c0d1, logical block 2
Buffer I/O error on device cciss/c0d1, logical block 3
cciss: cmd f7600000 has CHECK CONDITION sense key = 0x4
Buffer I/O error on device cciss/c0d1, logical block 0
cciss: cmd f7600000 has CHECK CONDITION sense key = 0x4
Buffer I/O error on device cciss/c0d1, logical block 143341669
cciss: cmd f7600000 has CHECK CONDITION sense key = 0x4
Buffer I/O error on device cciss/c0d1, logical block 143341669
cciss: cmd f7600250 has CHECK CONDITION sense key = 0x4
Buffer I/O error on device cciss/c0d1, logical block 0
Buffer I/O error on device cciss/c0d1, logical block 1
Buffer I/O error on device cciss/c0d1, logical block 2
cciss: cmd f7600250 has CHECK CONDITION sense key = 0x4
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdq, sector 2097144
end_request: I/O error, dev sdq, sector 2097144
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdq, sector 0
end_request: I/O error, dev sdr, sector 0
end_request: I/O error, dev sdr, sector 0
end_request: I/O error, dev sdr, sector 0
end_request: I/O error, dev sdr, sector 0
end_request: I/O error, dev sdr, sector 2097144
end_request: I/O error, dev sdr, sector 2097144
end_request: I/O error, dev sdr, sector 0
end_request: I/O error, dev sdr, sector 0

ErmanArslansOracleBlog wrote

Hi,

Please send me the OS logs representing the issue, as well as dmesg output.

[ErmanArslansOracleBlog]

Send me the output of the following commands

run them with root:

scsi_id -g -u /dev/sdp
scsi_id -g -u /dev/sdq

[CG]

I don't have these disks.

[ErmanArslansOracleBlog]

what is the output of:

multipath -ll
multipath -v2

[CG]

[root@oracledb ~]# multipath -ll

sdq: checker msg is "emc_clariion_checker: Logical Unit is unbound or LUNZ"
sdr: checker msg is "emc_clariion_checker: Logical Unit is unbound or LUNZ"
mpath2 (3600508b40008d0f30000800000050000) dm-2 HP,HSV300
[size=20G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 2:0:2:2 sdf 8:80  [active][ready]
 \_ 3:0:2:2 sdn 8:208 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 2:0:1:2 sdc 8:32  [active][ready]
 \_ 3:0:1:2 sdk 8:160 [active][ready]
mpath0 (3600508b40008d0f30000300002750000) dm-1 HP,HSV300
[size=1000G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 2:0:2:1 sde 8:64  [active][ready]
 \_ 3:0:2:1 sdm 8:192 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 2:0:1:1 sdb 8:16  [active][ready]
 \_ 3:0:1:1 sdj 8:144 [active][ready]
mpath9 (36006016029b0380029dbd37cf734e411) dm-4 DGC,VRAID
[size=1.5T][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=1][active]
 \_ 3:0:3:0 sdp 8:240 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:5:0 sdh 8:112 [active][ready]
mpath8 (3600508b40008d0f300009000000a0000) dm-3 HP,HSV300
[size=1.4T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 2:0:2:3 sdg 8:96  [active][ready]
 \_ 3:0:2:3 sdo 8:224 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 2:0:1:3 sdd 8:48  [active][ready]
 \_ 3:0:1:3 sdl 8:176 [active][ready]
mpath3 (3600601608f902c00faf59e4f5ba2e011) dm-0 DGC,RAID 5
[size=600G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=1][active]
 \_ 2:0:0:0 sda 8:0   [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 3:0:0:0 sdi 8:128 [active][ready]




[root@oracledb ~]# multipath -v2
sdq: checker msg is "emc_clariion_checker: Logical Unit is unbound or LUNZ"
sdr: checker msg is "emc_clariion_checker: Logical Unit is unbound or LUNZ"

[ErmanArslansOracleBlog]

the problem is obvious:

sdq: checker msg is "emc_clariion_checker: Logical Unit is unbound or LUNZ"
sdr: checker msg is "emc_clariion_checker: Logical Unit is unbound or LUNZ"

[ErmanArslansOracleBlog]

What is the output of command ->  vgs

[CG]

[root@oracledb ~]# vgs
  /dev/sdq: read failed after 0 of 4096 at 0: Input/output error
  /dev/VolGroup04/LogVol03: read failed after 0 of 4096 at 0: Input/output error
  /dev/VolGroup04/LogVol04: read failed after 0 of 4096 at 0: Input/output error
  /dev/VolGroup04/LogVol05: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdr: read failed after 0 of 4096 at 0: Input/output error
  /dev/cciss/c0d1: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdh: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdi: read failed after 0 of 4096 at 0: Input/output error
  VG          #PV #LV #SN Attr   VSize   VFree
  VolGroup03    1   3   0 wz--n- 300.07G      0
  VolGroup04    1   5   0 wz--n- 699.92G      0
  VolGroup05    1   1   0 wz--n- 600.00G      0
  VolGrp01      1   5   0 wz--n-   1.46T 200.00G
  VolGrpSwap    1   1   0 wz--n-  20.00G      0
  VolGrpYedek   1   1   0 wz--n-   1.37T 400.00G

[CG]

sorry , sdq and sdr disks are for another problem..

My problem is "my disk being readonly randomly"

here is the relevant log lines

Apr 14 14:55:50 oracledb multipathd: sdg: tur checker reports path is down
Apr 14 14:55:50 oracledb multipathd: sdm: tur checker reports path is down
Apr 14 14:55:50 oracledb multipathd: sdn: tur checker reports path is down
Apr 14 14:55:50 oracledb multipathd: sdo: tur checker reports path is down
Apr 14 14:55:55 oracledb multipathd: sdb: tur checker reports path is down
Apr 14 14:55:55 oracledb multipathd: sdc: tur checker reports path is down
Apr 14 14:55:55 oracledb multipathd: sdd: tur checker reports path is down
Apr 14 14:55:55 oracledb multipathd: sdj: tur checker reports path is down
Apr 14 14:55:55 oracledb multipathd: sdk: tur checker reports path is down
Apr 14 14:55:55 oracledb multipathd: sdl: tur checker reports path is down

Apr 14 14:56:00 oracledb multipathd: sde: tur checker reports path is down
Apr 14 14:56:00 oracledb multipathd: sdf: tur checker reports path is down
Apr 14 14:56:00 oracledb multipathd: sdg: tur checker reports path is down
Apr 14 14:56:00 oracledb multipathd: sdm: tur checker reports path is down
Apr 14 14:56:00 oracledb multipathd: sdn: tur checker reports path is down
Apr 14 14:56:00 oracledb multipathd: sdo: tur checker reports path is down
Apr 14 14:56:01 oracledb kernel: ext3_abort called.
Apr 14 14:56:01 oracledb kernel: EXT3-fs error (device dm-20): ext3_journal_start_sb: Detected aborted journal
Apr 14 14:56:01 oracledb kernel: Remounting filesystem read-only