Contents
明确chrac数据库存储迁移需求
将现有的EMC存储替换为PowerStore X1000存储，采用asm rebalance技术进行在线迁移。
前期准备工作
执行备份，建议手动再备份asm磁盘组的归档日志
SQL> alter system switch logfile
* ASM copy archivelog FROM ASM to LOCAL
mkdir /backup/$(date +%F)
chown grid:oinstall /backup/$(date +%F)
for i in $(asmcmd ls RECO/CHRAC/ARCHIVELOG/); do asmcmd cp +RECO/CHRAC/ARCHIVELOG/$i /backup/2023-03-18 done
查看ASM磁盘组信息
- login in as grid user
$ asmcmd lsdg
[grid@chrac1 ~]$ asmcmd lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 1454066 492658 0 492658 0 N DATA/ MOUNTED EXTERN N 512 4096 1048576 102398 90859 0 90859 0 N RECO/ MOUNTED NORMAL N 512 4096 1048576 15344 14418 5115 4651 0 Y SYSTEM/
查看ASM 磁盘空间使用情况
- login in as grid user
$ asmcmd lsdsk -k
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
307196 104032 307196 DATA1 DATA1 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA1 UNKNOWN ORCL:DATA1
102397 34675 102397 DATA2 DATA2 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA2 UNKNOWN ORCL:DATA2
102398 34678 102398 DATA3 DATA3 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA3 UNKNOWN ORCL:DATA3
122879 41643 122879 DATA4 DATA4 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA4 UNKNOWN ORCL:DATA4
204797 69353 204797 DATA5 DATA5 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA5 UNKNOWN ORCL:DATA5
614399 208277 614399 DATA6 DATA6 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA6 UNKNOWN ORCL:DATA6
5115 4808 5115 OCR1 OCR1 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) OCR1 UNKNOWN ORCL:OCR1
5114 4804 5114 OCR2 OCR2 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) OCR2 UNKNOWN ORCL:OCR2
5115 4806 5115 OCR3 OCR3 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) OCR3 UNKNOWN ORCL:OCR3
102398 90859 102398 RECO1 RECO1 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) RECO1 UNKNOWN ORCL:RECO1$ sqlplus / as sysasm
SQL> set linesize 300 column path format a50 column NAME format a15; select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk; GROUP_NUMBER DISK_NUMBER HEADER_STATUS MODE_STATUS MOUNT_STATUS TOTAL_MB FREE_MB NAME PATH ------------ ----------- ------------------------ -------------- -------------- ---------- ---------- ---------------- -------------------- 2 0 MEMBER ONLINE CACHED 307196 104032 DATA1 ORCL:DATA1 1 0 MEMBER ONLINE CACHED 5115 4808 OCR1 ORCL:OCR1 1 1 MEMBER ONLINE CACHED 5114 4804 OCR2 ORCL:OCR2 1 2 MEMBER ONLINE CACHED 5115 4806 OCR3 ORCL:OCR3 3 0 MEMBER ONLINE CACHED 10239890859 RECO1 ORCL:RECO1 2 1 MEMBER ONLINE CACHED 10239734675 DATA2 ORCL:DATA2 2 2 MEMBER ONLINE CACHED 10239834678 DATA3 ORCL:DATA3 2 3 MEMBER ONLINE CACHED 12287941643 DATA4 ORCL:DATA4 2 4 MEMBER ONLINE CACHED 20479769353 DATA5 ORCL:DATA5 2 5 MEMBER ONLINE CACHED 614399 208277 DATA6 ORCL:DATA6 10 rows selected.
[grid@chrac1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 4145560fdd2f4f14bffd18c1de6e5f5e (ORCL:OCR1) [SYSTEM] 2. ONLINE 21dd7ea749de4f99bfd44393b3de21f9 (ORCL:OCR2) [SYSTEM] 3. ONLINE 9b71588a3ab74f84bfc7d06676f3f513 (ORCL:OCR3) [SYSTEM]
/etc/init.d/oracleasm querydisk -d OCR1
/etc/init.d/oracleasm querydisk -d OCR2
/etc/init.d/oracleasm querydisk -d OCR3
ls -l /dev/ | grep 8 | grep 65
给chrac数据库虚拟化分配共享vmdk磁盘
参考VMware KB 1034165
chrac数据库存储迁移步骤
查看并确认新挂载的磁盘状态
$ lsblk
[root@chrac1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 3.6G 0 loop /mnt sr0 11:0 1 55.9M 0 rom sda 8:0 0 300G 0 disk |-sda1 8:1 0 500M 0 part /boot `-sda2 8:2 0 299.5G 0 part |-vg_chrac1-lv_root (dm-0) 253:0 0 291.7G 0 lvm / `-vg_chrac1-lv_swap (dm-1) 253:1 0 7.9G 0 lvm [SWAP] sdb 8:16 0 5G 0 disk `-sdb1 8:17 0 5G 0 part sdc 8:32 0 5G 0 disk `-sdc1 8:33 0 5G 0 part sde 8:64 0 300G 0 disk `-sde1 8:65 0 300G 0 part sdf 8:80 0 100G 0 disk `-sdf1 8:81 0 100G 0 part sdd 8:48 0 5G 0 disk `-sdd1 8:49 0 5G 0 part sdg 8:96 0 100G 0 disk `-sdg1 8:97 0 100G 0 part sdh 8:112 0 100G 0 disk `-sdh1 8:113 0 100G 0 part sdi 8:128 0 120G 0 disk `-sdi1 8:129 0 120G 0 part sdj 8:144 0 200G 0 disk `-sdj1 8:145 0 200G 0 part sdk 8:160 0 600G 0 disk `-sdk1 8:161 0 600G 0 part
* 结合dmesg命令输出确认新增加的磁盘信息
分区对齐
分区LUN时，建议分区对齐1M边界。使用 fdisk 或 parted 创建分区。但是，只有 parted 可以创建大于 2 TB 的分区。 本项目中将使用 parted 创建分区
在创建分区之前，将设备标记为 GPT。然后，指定分区偏移量为 2048 扇区（1M）。以下命令创建一个占用整个 LUN 的分区。创建分区后，使用分区文件 /dev/mapper/ora-asm-data-001p1 创建 ASMLib 卷。
parted /dev/sdl mklabel gpt mkpart primary 2048s 100% parted /dev/sdm mklabel gpt mkpart primary 2048s 100% parted /dev/sdn mklabel gpt mkpart primary 2048s 100% parted /dev/sdo mklabel gpt mkpart primary 2048s 100% parted /dev/sdp mklabel gpt mkpart primary 2048s 100% parted /dev/sdq mklabel gpt mkpart primary 2048s 100% parted /dev/sdr mklabel gpt mkpart primary 2048s 100% parted /dev/sds mklabel gpt mkpart primary 2048s 100% parted /dev/sdt mklabel gpt mkpart primary 2048s 100% parted /dev/sdu mklabel gpt mkpart primary 2048s 100% parted /dev/sdv mklabel gpt mkpart primary 2048s 100% parted /dev/sdw mklabel gpt mkpart primary 2048s 100% parted /dev/sdy mklabel gpt mkpart primary 2048s 100%
创建oracleasm磁盘
SYSTEM
oracleasm createdisk NOCR1 /dev/sdl1 oracleasm createdisk NOCR2 /dev/sdm1 oracleasm createdisk NOCR3 /dev/sdn1
DATA
oracleasm createdisk NDATA1 /dev/sdo1 oracleasm createdisk NDATA2 /dev/sdp1 oracleasm createdisk NDATA3 /dev/sdq1 oracleasm createdisk NDATA4 /dev/sdr1 oracleasm createdisk NDATA5 /dev/sds1 oracleasm createdisk NDATA6 /dev/sdt1 oracleasm createdisk NDATA7 /dev/sdu1 oracleasm createdisk NDATA8 /dev/sdv1
RECO
oracleasm createdisk NRECO1 /dev/sdw1 oracleasm createdisk NRECO2 /dev/sdy1
确认asm能够识别到生效后的磁盘
Add the device mappings to the device mapper disks.
* FOR VM partprobe /dev/sdx
* FOR Physical kpartx -a /dev/mapper/NDATA5-TEST
- login in as grid user
$ oracleasm scandisks
$ oracleasm listdisks
$ sqlplus / as sysasm
SQL> select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;
$ tail -f .../diag/asm/+asm/+ASM/trace/alert.log
正式加入前创建磁盘组测试
为新oracleasm磁盘创建测试磁盘组以确认磁盘是可以正常使用
测试完成之后, 删除该测试磁盘组, 再执行加入现有磁盘组的操作
将磁盘NOCR1-NOCR3加入到SYSTEM磁盘组
alter diskgroup SYSTEM add disk 'ORCL:NOCR1','ORCL:NOCR2','ORCL:NOCR3' rebalance power 6;
或者通过以下命令依次添加(建议一起添加).
alter diskgroup SYSTEM add disk 'ORCL:NOCR1' rebalance power 6; alter diskgroup SYSTEM add disk 'ORCL:NOCR2' rebalance power 6; alter diskgroup SYSTEM add disk 'ORCL:NOCR3' rebalance power 6;
SQL> select * from gv$asm_operation;
然后在DATA中加入DATA磁盘
alter diskgroup DATA add disk 'ORCL:NDATA1','ORCL:NDATA2','ORCL:NDATA3','ORCL:NDATA4','ORCL:NDATA5','ORCL:NDATA6','ORCL:NDATA7','ORCL:NDATA8' rebalance power 8;
然后在RECO中加入NRECO磁盘
alter diskgroup RECO add disk 'ORCL:NRECO1','ORCL:NRECO2' rebalance power 6;
查看rebalance情况与日志对比
$ asmcmd -p lsop
SQL> select * from gv$asm_operation;
$ tail -f alert.log
备注：确认所有磁盘rebalance完成后，再进行下一步删盘操作；
逐次剔除旧存储对应的asm disk
sqlplus -s '/ as sysasm' << "EOF" alter diskgroup SYSTEM drop disk OCR1 rebalance power 6; alter diskgroup SYSTEM drop disk OCR2 rebalance power 6; alter diskgroup SYSTEM drop disk OCR3 rebalance power 6; EOF
sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATA drop disk DATA1 rebalance power 9; alter diskgroup DATA drop disk DATA2 rebalance power 9; alter diskgroup DATA drop disk DATA3 rebalance power 9; alter diskgroup DATA drop disk DATA4 rebalance power 9; alter diskgroup DATA drop disk DATA5 rebalance power 9; alter diskgroup DATA drop disk DATA6 rebalance power 9; alter diskgroup RECO drop disk RECO1 rebalance power 6; EOF
- select * from gv$asm_operation;
查看rebalance情况与asm日志
$ asmcmd -p lsop
$ tail -f alert.log
依次回收VMware上的磁盘
先验证一个磁盘回收, 如无异常, 回收所有旧存储的磁盘.
最后建议是重启验证一下, 如果可能的话.