南方报业物流传媒集团RAC故障

1. 数据库ASM服务被teminated，但磁盘空间正常。
2. rac02自动重启。
3. rac01群集服务在rac02重启之后也被搞挂了。

Mar 13 07:51:39 rac01 multipathd: SAS_LUN11: load table [0 209715200 multipath 3 pg_init_retries 50 queue_if_no_path 1 rdac 2 1 round-robin 0 1 1 66:208 1 round-robin 0 1 1 8:144 1]
Mar 13 07:51:39 rac01 kernel: sd 1:0:0:9: rdac: array DELL_MD3820f, ctlr 0, queueing MODE_SELECT command
Mar 13 07:51:39 rac01 kernel: sd 1:0:0:9: rdac: array DELL_MD3820f, ctlr 0, MODE_SELECT returned with sense 06/94/01
Mar 13 07:51:39 rac01 kernel: sd 1:0:0:9: rdac: array DELL_MD3820f, ctlr 0, retrying MODE_SELECT command
Mar 13 07:51:40 rac01 kernel: sd 1:0:0:9: rdac: array DELL_MD3820f, ctlr 0, MODE_SELECT completed
Mar 13 07:51:40 rac01 multipathd: SAS_LUN11: load table [0 209715200 multipath 3 pg_init_retries 50 queue_if_no_path 1 rdac 2 1 round-robin 0 1 1 66:208 1 round-robin 0 1 1 8:144 1]
Mar 13 07:52:00 rac01 kernel: device-mapper: table: 253:17: multipath: error getting device
Mar 13 07:52:00 rac01 kernel: device-mapper: ioctl: error adding target to table
Mar 13 07:52:00 rac01 multipathd: SAS_LUN11: failed to update map : Device or resource busy

Troubleshooting

dmsetup info -C
grep -v ^$ /etc/multipath.conf

his error is most often caused by multipath attempting to map a device which is already in use, such as if its mounted or mapped by an active LVM logical volume. If the blacklist in /etc/multipath.conf does not restrict multipath from mapping the device, then when it gets run during boot it will attempt to map it. Because the device is in use, this mapping will fail.

中信海直

1. 数据库挂了。
2. 只有node1 / 磁盘空间满了