Differences between revisions 13 and 14
⇤ ← Revision 13 as of 2023-02-24 15:59:24 →
Size: 9754
Editor: localhost
Comment:
← Revision 14 as of 2023-02-24 16:00:29 → ⇥
Size: 9786
Editor: localhost
Comment:
Deletions are marked like this. Additions are marked like this.
Line 237: Line 237:
先验证一个磁盘回收,

最好是重启验证一下, 如果可能的话.		 先验证一个磁盘回收, 如无异常, 回收所有旧存储的磁盘.

最后建议是重启验证一下.

Describe 首頁/work/QH-htrac-VM-replace-disk here.

Contents

  1. 明确htrac数据库存储迁移需求
  2. 前期准备工作
    1. 查看ASM磁盘组信息
    2. 查看ASM 磁盘空间使用情况
    3. 给htrac数据库虚拟化分配共享vmdk磁盘
  3. htrac数据库存储迁移步骤
    1. 查看并确认新挂载的磁盘状态
    2. 分区对齐
    3. 创建oracleasm磁盘
      1. SYSTEM
      2. DATA
      3. RECO
    4. 正式加入前创建磁盘组测试
    5. 将磁盘OCR01-OCR03加入到SYSTEM磁盘组
      1. 确认asm能够识别到生效后的磁盘
      2. 然后在RECO中加入REDO磁盘
    6. 查看rebalance情况与日志对比
    7. 逐次剔除旧存储对应的asm disk
    8. 查看rebalance情况与asm日志
    9. 依次回收VMware上的磁盘
  4. References

明确htrac数据库存储迁移需求

将现有的EMC存储替换为PowerStore X1000存储，采用asm rebalance技术进行在线迁移。

前期准备工作

查看ASM磁盘组信息

  • login in as grid user

$ asmcmd lsdg 

[grid@htrac1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    624628   183534                0          183534              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576    102398    27343                0           27343              0             N  RECO/
MOUNTED  NORMAL  N         512   4096  1048576     15345    14419             5115            4652              0             Y  SYSTEM/

查看ASM 磁盘空间使用情况

  • login in as grid user

$ asmcmd lsdsk -k 

Total_MB  Free_MB   OS_MB  Name   Failgroup  Failgroup_Type  Library                                               Label  UDID  Product  Redund   Path
  307196    90267  307196  DATA1  DATA1      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA1                 UNKNOWN  ORCL:DATA1
  112635    33092  112635  DATA2  DATA2      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA2                 UNKNOWN  ORCL:DATA2
  204797    60175  204797  DATA3  DATA3      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA3                 UNKNOWN  ORCL:DATA3
    5115     4807    5115  OCR1   OCR1       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR1                  UNKNOWN  ORCL:OCR1
    5115     4805    5115  OCR2   OCR2       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR2                  UNKNOWN  ORCL:OCR2
    5115     4807    5115  OCR3   OCR3       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR3                  UNKNOWN  ORCL:OCR3
  102398    27343  102398  RECO1  RECO1      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  RECO1                 UNKNOWN  ORCL:RECO1

$ sqlplus / as sysasm 

SQL> set linesize 300
column path format a50
column NAME format a15;
select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;

GROUP_NUMBER DISK_NUMBER HEADER_STATUS            MODE_STATUS     MOUNT_STATUS  TOTAL_MB    FREE_MB         NAME       PATH
------------ ----------- ------------------------ -------------- -------------- ---------- ---------- ---------------- --------------------
   2       0 MEMBER   ONLINE CACHED     30719690267 DATA1       ORCL:DATA1
   1       0 MEMBER   ONLINE CACHED       5115 4807 OCR1       ORCL:OCR1
   1       1 MEMBER   ONLINE CACHED       5115 4805 OCR2       ORCL:OCR2
   1       2 MEMBER   ONLINE CACHED       5115 4807 OCR3       ORCL:OCR3
   3       0 MEMBER   ONLINE CACHED     10239827343 RECO1       ORCL:RECO1
   2       1 MEMBER   ONLINE CACHED     11263533092 DATA2       ORCL:DATA2
   2       2 MEMBER   ONLINE CACHED     20479760175 DATA3       ORCL:DATA3

7 rows selected.

[grid@htrac1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6a7669a7d58c4f22bf06341bbacde673 (ORCL:OCR1) [SYSTEM]
 2. ONLINE   076dc3fef5374f2cbf8af0958ffc5a9b (ORCL:OCR2) [SYSTEM]
 3. ONLINE   9f6d65dc01364feebf566749760ced41 (ORCL:OCR3) [SYSTEM]

/etc/init.d/oracleasm querydisk -d OCR1

/etc/init.d/oracleasm querydisk -d OCR2

/etc/init.d/oracleasm querydisk -d OCR3

ls -l /dev/ | grep 8 | grep 65

给htrac数据库虚拟化分配共享vmdk磁盘

参考VMware KB 1034165

htrac数据库存储迁移步骤

查看并确认新挂载的磁盘状态

$ lsblk 

[root@htrac1 ~]# lsblk
NAME                          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                           7:0    0   3.6G  0 loop /mnt
sr0                            11:0    1   3.6G  0 rom  
sdb                             8:16   0     5G  0 disk 
`-sdb1                          8:17   0     5G  0 part 
sdc                             8:32   0     5G  0 disk 
`-sdc1                          8:33   0     5G  0 part 
sde                             8:64   0   300G  0 disk 
`-sde1                          8:65   0   300G  0 part 
sda                             8:0    0   300G  0 disk 
|-sda1                          8:1    0   500M  0 part /boot
`-sda2                          8:2    0 299.5G  0 part 
  |-vg_htdb1-lv_root (dm-0)   253:0    0  83.5G  0 lvm  /
  |-vg_htdb1-lv_swap (dm-1)   253:1    0    16G  0 lvm  [SWAP]
  `-vg_htdb1-lv_backup (dm-2) 253:2    0   300G  0 lvm  /backup
sdd                             8:48   0     5G  0 disk 
`-sdd1                          8:49   0     5G  0 part 
sdf                             8:80   0   100G  0 disk 
`-sdf1                          8:81   0   100G  0 part 
sdg                             8:96   0   100G  0 disk 
`-vg_htdb1-lv_backup (dm-2)   253:2    0   300G  0 lvm  /backup
sdh                             8:112  0   110G  0 disk 
`-sdh1                          8:113  0   110G  0 part 
sdi                             8:128  0   200G  0 disk 
`-sdi1                          8:129  0   200G  0 part

* 结合dmesg命令输出确认新增加的磁盘信息

分区对齐

分区LUN时，建议分区对齐1M边界。使用 fdisk 或 parted 创建分区。但是，只有 parted 可以创建大于 2 TB 的分区。 本项目中将使用 parted 创建分区

在创建分区之前，将设备标记为 GPT。然后，指定分区偏移量为 2048 扇区（1M）。以下命令创建一个占用整个 LUN 的分区。创建分区后，使用分区文件 /dev/mapper/ora-asm-data-001p1 创建 ASMLib 卷。 

parted /dev/sdj mklabel gpt mkpart primary 2048s 100%

创建oracleasm磁盘

SYSTEM

oracleasm createdisk OCR01 /dev/sdj1

oracleasm createdisk OCR02 /dev/sdk1

oracleasm createdisk OCR03 /dev/sdl1

DATA

oracleasm createdisk DATA01 /dev/sdm1

oracleasm createdisk DATA02 /dev/sdn1

oracleasm createdisk DATA03 /dev/sdo1

RECO

oracleasm createdisk REDO1 /dev/sdp1

正式加入前创建磁盘组测试

为新oracleasm磁盘创建测试磁盘组以确认磁盘是可以正常使用

测试完成之后, 删除该测试磁盘组, 再执行加入现有磁盘组的操作

将磁盘OCR01-OCR03加入到SYSTEM磁盘组

alter diskgroup SYSTEM add disk 'ORCL:CRS01','ORCL:CRS02','ORCL:CRS03' rebalance power 6;

或者通过以下命令依次添加(建议一起添加). 

alter diskgroup SYSTEM add disk 'ORCL:CRS01' rebalance power 6;
alter diskgroup SYSTEM add disk 'ORCL:CRS02' rebalance power 6;
alter diskgroup SYSTEM add disk 'ORCL:CRS03' rebalance power 6;

SQL> select * from gv$asm_operation;

确认asm能够识别到生效后的磁盘

  • login in as grid user

$ oracleasm scandisks

$ oracleasm listdisks

$ sqlplus / as sysasm 

SQL> select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;

$ tail -f /u01/app/grid/diag/asm/+asm/+ASM/trace/alert.log

备注：尽量在业务不繁忙的情况下操作;

然后在RECO中加入REDO磁盘

alter diskgroup RECO add disk 'ORCL:REDO1' rebalance power 6;

SQL> select * from gv$asm_operation;

$ tail -f alert_+ASM.log

查看rebalance情况与日志对比

  • login in as grid user

使用以下命令查看观察:当没有输出时则表明reblance完成。 

select * from gv$asm_operation;
  * 以下为加第一块时的输出，看EST_MINUTES大约需要29分钟。
SQL> select * from gv$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           2 REBAL RUN           8          8      33691     830701      26839          29

$ asmcmd lsdsk -k 

[grid@MCPOS-HOST-01 ~]$ asmcmd lsdsk -k

$ asmcmd -p lsop
Group_Name  Dsk_Num  State  Power  EST_WORK  EST_RATE  EST_TIME
DATA      REBAL    RUN    8      830701    0         0

$ tail -f alert.log

备注：确认所有磁盘rebalance完成后，再进行下一步删盘操作； 尽量在业务不繁忙的情况下操作;

逐次剔除旧存储对应的asm disk

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup SYSTEM drop disk OCR1 rebalance power 6;
alter diskgroup SYSTEM drop disk OCR2 rebalance power 6;
alter diskgroup SYSTEM drop disk OCR3 rebalance power 6;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATA drop disk DATA1 rebalance power 6;
alter diskgroup DATA drop disk DATA2 rebalance power 6;
alter diskgroup DATA drop disk DATA3 rebalance power 6;
alter diskgroup DATA drop disk RECO1 rebalance power 6;
EOF
  • select * from gv$asm_operation;

查看rebalance情况与asm日志

$ asmcmd -p lsop
$ tail -f alert.log

依次回收VMware上的磁盘

先验证一个磁盘回收, 如无异常, 回收所有旧存储的磁盘.

最后建议是重启验证一下.

References

https://www.dell.com/support/kbdoc/en-sg/000105896/powerstore-supported-host-os-for-non-disruptive-migration-of-storage-resources

首頁/work/QH-htrac-VM-replace-disk (last edited 2023-02-25 02:21:41 by localhost)