Differences between revisions 2 and 3

Describe 首頁/work/QH-htrac-VM-replace-disk here.

Contents

明确htrac数据库存储迁移需求
前期准备工作
DATABASE-NAME数据库存储迁移步骤
References

明确htrac数据库存储迁移需求

将现有的EMC存储替换为PowerStore X1000存储，采用asm rebalance技术进行在线迁移。

前期准备工作

查看ASM磁盘组信息

$ asmcmd lsdg

[grid@htrac1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    624628   183534                0          183534              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576    102398    27343                0           27343              0             N  RECO/
MOUNTED  NORMAL  N         512   4096  1048576     15345    14419             5115            4652              0             Y  SYSTEM/

查看ASM 磁盘空间使用情况

$ asmcmd lsdsk -k

Total_MB  Free_MB   OS_MB  Name   Failgroup  Failgroup_Type  Library                                               Label  UDID  Product  Redund   Path
  307196    90267  307196  DATA1  DATA1      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA1                 UNKNOWN  ORCL:DATA1
  112635    33092  112635  DATA2  DATA2      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA2                 UNKNOWN  ORCL:DATA2
  204797    60175  204797  DATA3  DATA3      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  DATA3                 UNKNOWN  ORCL:DATA3
    5115     4807    5115  OCR1   OCR1       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR1                  UNKNOWN  ORCL:OCR1
    5115     4805    5115  OCR2   OCR2       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR2                  UNKNOWN  ORCL:OCR2
    5115     4807    5115  OCR3   OCR3       REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  OCR3                  UNKNOWN  ORCL:OCR3
  102398    27343  102398  RECO1  RECO1      REGULAR         ASM Library - Generic Linux, version 2.0.4 (KABI_V2)  RECO1                 UNKNOWN  ORCL:RECO1

$ sqlplus / as sysasm

SQL> set linesize 300
column path format a50
column NAME format a15;
select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;

GROUP_NUMBER DISK_NUMBER HEADER_STATUS            MODE_STATUS     MOUNT_STATUS  TOTAL_MB    FREE_MB         NAME       PATH
------------ ----------- ------------------------ -------------- -------------- ---------- ---------- ---------------- --------------------
   2       0 MEMBER   ONLINE CACHED     30719690267 DATA1       ORCL:DATA1
   1       0 MEMBER   ONLINE CACHED       5115 4807 OCR1       ORCL:OCR1
   1       1 MEMBER   ONLINE CACHED       5115 4805 OCR2       ORCL:OCR2
   1       2 MEMBER   ONLINE CACHED       5115 4807 OCR3       ORCL:OCR3
   3       0 MEMBER   ONLINE CACHED     10239827343 RECO1       ORCL:RECO1
   2       1 MEMBER   ONLINE CACHED     11263533092 DATA2       ORCL:DATA2
   2       2 MEMBER   ONLINE CACHED     20479760175 DATA3       ORCL:DATA3

7 rows selected.

给htrac数据库虚拟化分配共享vmdk磁盘

参考VMware KB 1034165

DATABASE-NAME数据库存储迁移步骤

查看并确认新挂载的磁盘状态

$ lsblk

* 结合dmesg命令输出确认新增加的磁盘信息

分区对齐

分区LUN时，建议分区对齐1M边界。使用 fdisk 或 parted 创建分区。但是，只有 parted 可以创建大于 2 TB 的分区。本项目中将使用 parted 创建分区

在创建分区之前，将设备标记为 GPT。然后，指定分区偏移量为 2048 扇区（1M）。以下命令创建一个占用整个 LUN 的分区。创建分区后，使用分区文件 /dev/mapper/ora-asm-data-001p1 创建 ASMLib 卷。

将磁盘X1000-DATA01p1、X1000-DATA02p1、X1000-DATA03p1、X1000-DATA04p1加入到 DATAVG

确认asm能够识别到udev规则生效后的磁盘

$ sqlplus / as sysasm

SQL> select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;

首先在DATAVG中加入X1000存储的DATA LUN

alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA01p1' rebalance power 8;

alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA02p1' rebalance power 12;
SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           2 REBAL RUN          11         11      11870     851620      37158          22

alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA03p1' rebalance power 12;

alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA04p1' rebalance power 12;

SQL> alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA05p1','/dev/oracleasm/disks/X1000-DATA06p1' rebalance power 12;
Diskgroup altered.

$ tail -f /u01/app/grid/diag/asm/+asm/+ASM/trace/alert.log
SQL> alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA05p1','/dev/oracleasm/disks/X1000-DATA06p1' rebalance power 12

备注：尽量在业务不繁忙的情况下操作;

然后在REDOVG中加入X1000存储的REDO LUN

alter diskgroup REDOVG add disk '/dev/oracleasm/disks/X1000-REDO01p1' rebalance power 8;

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           2 REBAL RUN          11         11    1082386    1082386          0           0
           3 REBAL WAIT          8

$ tail -f alert_+ASM.log

alter diskgroup REDOVG add disk '/dev/oracleasm/disks/X1000-REDO02p1' rebalance power 5;

查看rebalance情况与日志对比

使用以下命令查看观察:当没有输出时则表明reblance完成。

select * from v$asm_operation;
  * 以下为加第一块时的输出，看EST_MINUTES大约需要29分钟。
SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           2 REBAL RUN           8          8      33691     830701      26839          29

$ asmcmd lsdsk -k

[grid@MCPOS-HOST-01 ~]$ asmcmd lsdsk -k

$ asmcmd -p lsop
Group_Name  Dsk_Num  State  Power  EST_WORK  EST_RATE  EST_TIME
DATAVG      REBAL    RUN    8      830701    0         0

$ tail -f alert.log

备注：确认所有磁盘rebalance完成后，再进行下一步删盘操作；尽量在业务不繁忙的情况下操作;

逐次剔除旧存储AFF8040对应的asm disk

DROP AFF8040 REDOVG MEMBERs

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup REDOVG drop disk REDOVG_0002 rebalance power 11;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup REDOVG drop disk REDOVG_0003 rebalance power 11;
EOF

DROP AFF8040 DATAVG MEMBERs

RETVAL=`sqlplus -s '/ as sysasm' << "EOF"
select * from v$asm_operation;
EXIT;
EOF`

if [ -z "$RETVAL" ]; then
  echo "No rows returned from database"
  exit 0
else
  echo $RETVAL
  echo "DROP FAILED"
fi

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATAVG drop disk DATAVG_0003 rebalance power 11;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATAVG drop disk DATAVG_0004 rebalance power 11;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATAVG drop disk DATAVG_0005 rebalance power 11;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATAVG drop disk DATAVG_0006 rebalance power 11;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATAVG drop disk DATAVG_0007 rebalance power 11;
EOF

sqlplus -s '/ as sysasm' << "EOF"
alter diskgroup DATAVG drop disk DATAVG_0008 rebalance power 11;
EOF

select * from v$asm_operation;

查看rebalance情况与asm日志对比

$ asmcmd -p lsop
$ tail -f alert.log

删除ARCHVG磁盘组

sqlplus / as sysasm
SQL> alter diskgroup ARCHVG dismount;
SQL> drop diskgroup ARCHVG force including contents;

References

https://www.dell.com/support/kbdoc/en-sg/000105896/powerstore-supported-host-os-for-non-disruptive-migration-of-storage-resources

首頁/work/QH-htrac-VM-replace-disk (last edited 2023-02-25 02:21:41 by localhost)

-  ⇤ ← Revision 2 as of 2023-02-24 10:39:19 → 
  Size: 9969
  Editor: localhost
  Comment:
+   ← Revision 3 as of 2023-02-24 10:55:59 → ⇥
  Size: 9093
  Editor: localhost
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 5:
-数据库存储迁移流程

= 明确DATABASE-NAME数据库存储迁移需求 =
+= 明确htrac数据库存储迁移需求 =
-Line 56:
+Line 54:
-== 检查多路径状态 ==
multipath –ll
+== 给htrac数据库虚拟化分配共享vmdk磁盘 ==
参考VMware KB 1034165
-Line 61:
+Line 59:
-== 查看磁盘的分区和挂载情况 ==
lsblk

== 确认新挂载的磁盘状态及链路正常 ==
multipath -ll | grep -A 7 'X1000'
{{{
X1000-DATA02 (3600a098000f63f48000001c75c7db889) dm-29 NETAPP,INF-01-00
size=1.0T features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 7:0:9:3  sdy  65:128  active ready running
| `- 8:0:8:3  sdah 66:16   active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:8:3  sdp  8:240   active ready running
  `- 8:0:9:3  sdap 66:144  active ready running
}}}

== 配置多路径 ==
  * /etc/multipath.conf
{{{
# 2023-02-25 added For X1000
multipath {
        alias X1000-DATA01
        wwid 3600a098000e3c7e6000002635c7db855
}
}}}

  * /etc/multipath/bindings
{{{
X1000-DATA01 3600A098000E3C7E6000002635C7DB855
}}}

== 多路径配置生效 ==
service multipathd reload

== udev 磁盘绑定 ==
cat /etc/udev/rules.d/99-oracle-asmdevices.rules
{{{
# 2023-02-25 added for X1000
KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-3600a098000f63f48000001cb5c7db88e",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME+="oracleasm/disks/X1000-DATA06p1"
}}}

== udev 规则生效 ==

===  优先采用使用Redhat udev 规则生效推荐的方法操作 ===
{{{
udevadm control --reload-rules
udevadm trigger --type=devices --action=change

...
..
.
}}}
+== 查看并确认新挂载的磁盘状态 ==
$ lsblk

* 结合dmesg命令输出确认新增加的磁盘信息
== 分区对齐 ==

分区LUN时，建议分区对齐1M边界。使用 fdisk 或 parted 创建分区。但是，只有 parted 可以创建大于 2 TB 的分区。
本项目中将使用 parted 创建分区

在创建分区之前，将设备标记为 GPT。然后，指定分区偏移量为 2048 扇区（1M）。以下命令创建一个占用整个 LUN 的分区。创建分区后，使用分区文件 /dev/mapper/ora-asm-data-001p1 创建 ASMLib 卷。

Diff for "首頁/work/QH-htrac-VM-replace-disk"