明确htrac数据库存储迁移需求

将现有的EMC存储替换为PowerStore X1000存储，采用asm rebalance技术进行在线迁移。

前期准备工作

执行备份，建议手动再备份asm磁盘组的归档日志

SQL> alter system switch logfile

* ASM copy archivelog FROM ASM to LOCAL

mkdir /backup/$(date +%F)

chown grid:oinstall /backup/$(date +%F)

for i in $(asmcmd ls RECO/HTRAC/ARCHIVELOG/); do
  asmcmd cp +RECO/HTRAC/ARCHIVELOG/$i /backup/2023-03-18
done

查看ASM磁盘组信息

• login in as grid user

$ asmcmd lsdg

[grid@htrac1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    409592    69647                0           69647              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576    112637   111635                0          111635              0             N  RECO/
MOUNTED  NORMAL  N         512   4096  1048576      9206     8280             3069            2605              0             Y  SYSTEM

查看ASM 磁盘空间使用情况

• login in as grid user

$ asmcmd lsdsk -k

Total_MB  Free_MB   OS_MB  Name   Failgroup  Failgroup_Type  Library                                               Label  UDID  Product  Redund   Path
   51199     8704   51199  DATA1  DATA1      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA1                 UNKNOWN  ORCL:DATA1
  153597    26118  153597  DATA2  DATA2      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA2                 UNKNOWN  ORCL:DATA2
  102398    17413  102398  DATA3  DATA3      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA3                 UNKNOWN  ORCL:DATA3
  102398    17412  102398  DATA4  DATA4      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA4                 UNKNOWN  ORCL:DATA4
    3068     2761    3068  OCR1   OCR1       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR1                  UNKNOWN  ORCL:OCR1
    3069     2760    3069  OCR2   OCR2       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR2                  UNKNOWN  ORCL:OCR2
    3069     2759    3069  OCR3   OCR3       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR3                  UNKNOWN  ORCL:OCR3
   30719    30444   30719  RECO1  RECO1      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  RECO1                 UNKNOWN  ORCL:RECO1
   30719    30440   30719  RECO2  RECO2      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  RECO2                 UNKNOWN  ORCL:RECO2
   51199    50751   51199  RECO3  RECO3      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  RECO3                 UNKNOWN  ORCL:RECO3

$ sqlplus / as sysasm

SQL> set linesize 300
column path format a50
column NAME format a15;
select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;

GROUP_NUMBER DISK_NUMBER HEADER_STATUS            MODE_STATUS     MOUNT_STATUS  TOTAL_MB    FREE_MB         NAME       PATH
------------ ----------- ------------------------ -------------- -------------- ---------- ---------- ---------------- --------------------
   2       0 MEMBER       ONLINE  CACHED   51199       8704 DATA1     ORCL:DATA1
   1       0 MEMBER       ONLINE  CACHED    3068       2761 OCR1     ORCL:OCR1
   1       1 MEMBER       ONLINE  CACHED    3069       2760 OCR2     ORCL:OCR2
   1       2 MEMBER       ONLINE  CACHED    3069       2759 OCR3     ORCL:OCR3
   3       0 MEMBER       ONLINE  CACHED   30719      30444 RECO1     ORCL:RECO1
   2       2 MEMBER       ONLINE  CACHED  153597      26118 DATA2     ORCL:DATA2
   3       1 MEMBER       ONLINE  CACHED   30719      30440 RECO2     ORCL:RECO2
   2       3 MEMBER       ONLINE  CACHED  102398      17413 DATA3     ORCL:DATA3
   3       2 MEMBER       ONLINE  CACHED   51199      50751 RECO3     ORCL:RECO3
   2       4 MEMBER       ONLINE  CACHED  102398      17412 DATA4     ORCL:DATA4

10 rows selected.

[grid@htrac1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7436bd33ce064f6cbf17609ed9a95ed6 (ORCL:OCR1) [SYSTEM]
 2. ONLINE   095cf9e23d9e4fedbfd7aae019809f38 (ORCL:OCR2) [SYSTEM]
 3. ONLINE   7783356222294f42bf8a0de169b65d28 (ORCL:OCR3) [SYSTEM]

/etc/init.d/oracleasm querydisk -d OCR1

/etc/init.d/oracleasm querydisk -d OCR2

/etc/init.d/oracleasm querydisk -d OCR3

ls -l /dev/ | grep 8 | grep 65

给htrac数据库虚拟化分配共享vmdk磁盘

参考VMware KB 1034165

htrac数据库存储迁移步骤

查看并确认新挂载的磁盘状态

$ lsblk

[root@htrac1 ~]# lsblk
NAME                          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
NAME                          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0                            11:0    1  1024M  0 rom
sda                             8:0    0   300G  0 disk
|-sda1                          8:1    0   500M  0 part /boot
`-sda2                          8:2    0 299.5G  0 part
  |-vg_cwdb1-lv_root (dm-0)   253:0    0    50G  0 lvm  /
  |-vg_cwdb1-lv_swap (dm-1)   253:1    0   7.9G  0 lvm  [SWAP]
  |-vg_cwdb1-lv_backup (dm-2) 253:2    0   100G  0 lvm  /backup
  `-vg_cwdb1-lv_home (dm-3)   253:3    0 141.7G  0 lvm  /home
sdc                             8:32   0     3G  0 disk
`-sdc1                          8:33   0     3G  0 part
sdf                             8:80   0    30G  0 disk
`-sdf1                          8:81   0    30G  0 part
sdg                             8:96   0    80G  0 disk
sde                             8:64   0    50G  0 disk
`-sde1                          8:65   0    50G  0 part
sdb                             8:16   0     3G  0 disk
`-sdb1                          8:17   0     3G  0 part
sdd                             8:48   0     3G  0 disk
`-sdd1                          8:49   0     3G  0 part
sdh                             8:112  0   150G  0 disk
`-sdh1                          8:113  0   150G  0 part
sdi                             8:128  0    30G  0 disk
`-sdi1                          8:129  0    30G  0 part
sdj                             8:144  0   100G  0 disk
`-sdj1                          8:145  0   100G  0 part
sdk                             8:160  0    50G  0 disk
`-sdk1                          8:161  0    50G  0 part
sdl                             8:176  0   100G  0 disk
`-sdl1                          8:177  0   100G  0 part

* 结合dmesg命令输出确认新增加的磁盘信息

分区对齐

分区LUN时，建议分区对齐1M边界。使用 fdisk 或 parted 创建分区。但是，只有 parted 可以创建大于 2 TB 的分区。 本项目中将使用 parted 创建分区

在创建分区之前，将设备标记为 GPT。然后，指定分区偏移量为 2048 扇区（1M）。以下命令创建一个占用整个 LUN 的分区。创建分区后，使用分区文件 /dev/mapper/ora-asm-data-001p1 创建 ASMLib 卷。

parted /dev/sdj mklabel gpt mkpart primary 2048s 100%

创建oracleasm磁盘

SYSTEM

oracleasm createdisk OCR01 /dev/sdj1

oracleasm createdisk OCR02 /dev/sdk1

oracleasm createdisk OCR03 /dev/sdl1

DATA

oracleasm createdisk DATA01 /dev/sdm1

oracleasm createdisk DATA02 /dev/sdn1

oracleasm createdisk DATA03 /dev/sdo1

RECO

oracleasm createdisk REDO1 /dev/sdp1

确认asm能够识别到生效后的磁盘

Add the device mappings to the device mapper disks.

kpartx -a /dev/mapper/DATA05-TEST

• login in as grid user

$ oracleasm scandisks

$ oracleasm listdisks

$ sqlplus / as sysasm

SQL> select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;

$ tail -f .../diag/asm/+asm/+ASM/trace/alert.log

正式加入前创建磁盘组测试

为新oracleasm磁盘创建测试磁盘组以确认磁盘是可以正常使用

测试完成之后, 删除该测试磁盘组, 再执行加入现有磁盘组的操作

将磁盘OCR01-OCR03加入到SYSTEM磁盘组

alter diskgroup SYSTEM add disk 'ORCL:CRS01','ORCL:CRS02','ORCL:CRS03' rebalance power 6;

或者通过以下命令依次添加(建议一起添加).

alter diskgroup SYSTEM add disk 'ORCL:CRS01' rebalance power 6;
alter diskgroup SYSTEM add disk 'ORCL:CRS02' rebalance power 6;
alter diskgroup SYSTEM add disk 'ORCL:CRS03' rebalance power 6;

SQL> select * from gv$asm_operation;

然后在DATA中加入DATA磁盘

alter diskgroup DATA add disk 'ORCL:DATA01','ORCL:DATA02','ORCL:DATA03' rebalance power 6;

然后在RECO中加入REDO磁盘

alter diskgroup RECO add disk 'ORCL:REDO1' rebalance power 6;

查看rebalance情况与日志对比

$ asmcmd -p lsop

$ tail -f alert.log

备注：确认所有磁盘rebalance完成后，再进行下一步删盘操作；

逐次剔除旧存储对应的asm disk

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup SYSTEM drop disk OCR1 rebalance power 6;
alter diskgroup SYSTEM drop disk OCR2 rebalance power 6;
alter diskgroup SYSTEM drop disk OCR3 rebalance power 6;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATA drop disk DATA1 rebalance power 6;
alter diskgroup DATA drop disk DATA2 rebalance power 6;
alter diskgroup DATA drop disk DATA3 rebalance power 6;
alter diskgroup DATA drop disk DATA4 rebalance power 6;
alter diskgroup DATA drop disk RECO1 rebalance power 6;
alter diskgroup DATA drop disk RECO2 rebalance power 6;
alter diskgroup DATA drop disk RECO3 rebalance power 6;
EOF

• select * from gv$asm_operation;

查看rebalance情况与asm日志

$ asmcmd -p lsop

$ tail -f alert.log

依次回收VMware上的磁盘

先验证一个磁盘回收, 如无异常, 回收所有旧存储的磁盘.

最后建议是重启验证一下, 如果可能的话.

References

https://www.dell.com/support/kbdoc/en-sg/000105896/powerstore-supported-host-os-for-non-disruptive-migration-of-storage-resources