|
Size: 9969
Comment:
|
← Revision 22 as of 2023-02-25 02:21:41 ⇥
Size: 9366
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 5: | Line 5: |
| 数据库存储迁移流程 = 明确DATABASE-NAME数据库存储迁移需求 = |
= 明确htrac数据库存储迁移需求 = |
| Line 11: | Line 9: |
| 执行备份，建议手动再备份asm磁盘组的归档日志 SQL> alter system switch logfile ASM copy archivelog FROM ASM to LOCAL {{{ for i in $(asmcmd ls fra/CARPOLY/ARCHIVELOG/2019_12_29); do asmcmd cp +fra/CARPOLY/ARCHIVELOG/2019_12_29/$i /mnt/backup/cloud1 done }}} |
|
| Line 56: | Line 70: |
| == 检查多路径状态 == multipath –ll = DATABASE-NAME数据库存储迁移步骤 = == 查看磁盘的分区和挂载情况 == lsblk == 确认新挂载的磁盘状态及链路正常 == multipath -ll | grep -A 7 'X1000' {{{ X1000-DATA02 (3600a098000f63f48000001c75c7db889) dm-29 NETAPP,INF-01-00 size=1.0T features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 7:0:9:3 sdy 65:128 active ready running | `- 8:0:8:3 sdah 66:16 active ready running `-+- policy='round-robin 0' prio=10 status=enabled |- 7:0:8:3 sdp 8:240 active ready running `- 8:0:9:3 sdap 66:144 active ready running }}} == 配置多路径 == * /etc/multipath.conf {{{ # 2023-02-25 added For X1000 multipath { alias X1000-DATA01 wwid 3600a098000e3c7e6000002635c7db855 } }}} * /etc/multipath/bindings {{{ X1000-DATA01 3600A098000E3C7E6000002635C7DB855 }}} == 多路径配置生效 == service multipathd reload == udev 磁盘绑定 == cat /etc/udev/rules.d/99-oracle-asmdevices.rules {{{ # 2023-02-25 added for X1000 KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-3600a098000f63f48000001cb5c7db88e",OWNER="grid",GROUP="asmadmin",MODE="0660",NAME+="oracleasm/disks/X1000-DATA06p1" }}} == udev 规则生效 == === 优先采用使用Redhat udev 规则生效推荐的方法操作 === {{{ udevadm control --reload-rules udevadm trigger --type=devices --action=change ... .. . }}} == 将磁盘X1000-DATA01p1、X1000-DATA02p1、X1000-DATA03p1、X1000-DATA04p1加入到 DATAVG == === 确认asm能够识别到udev规则生效后的磁盘 === |
{{{ [grid@htrac1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6a7669a7d58c4f22bf06341bbacde673 (ORCL:OCR1) [SYSTEM] 2. ONLINE 076dc3fef5374f2cbf8af0958ffc5a9b (ORCL:OCR2) [SYSTEM] 3. ONLINE 9f6d65dc01364feebf566749760ced41 (ORCL:OCR3) [SYSTEM] }}} /etc/init.d/oracleasm querydisk -d OCR1 /etc/init.d/oracleasm querydisk -d OCR2 /etc/init.d/oracleasm querydisk -d OCR3 ls -l /dev/ | grep 8 | grep 65 == 给htrac数据库虚拟化分配共享vmdk磁盘 == 参考VMware KB 1034165 = htrac数据库存储迁移步骤 = == 查看并确认新挂载的磁盘状态 == $ lsblk {{{ [root@htrac1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 3.6G 0 loop /mnt sr0 11:0 1 3.6G 0 rom sdb 8:16 0 5G 0 disk `-sdb1 8:17 0 5G 0 part sdc 8:32 0 5G 0 disk `-sdc1 8:33 0 5G 0 part sde 8:64 0 300G 0 disk `-sde1 8:65 0 300G 0 part sda 8:0 0 300G 0 disk |-sda1 8:1 0 500M 0 part /boot `-sda2 8:2 0 299.5G 0 part |-vg_htdb1-lv_root (dm-0) 253:0 0 83.5G 0 lvm / |-vg_htdb1-lv_swap (dm-1) 253:1 0 16G 0 lvm [SWAP] `-vg_htdb1-lv_backup (dm-2) 253:2 0 300G 0 lvm /backup sdd 8:48 0 5G 0 disk `-sdd1 8:49 0 5G 0 part sdf 8:80 0 100G 0 disk `-sdf1 8:81 0 100G 0 part sdg 8:96 0 100G 0 disk `-vg_htdb1-lv_backup (dm-2) 253:2 0 300G 0 lvm /backup sdh 8:112 0 110G 0 disk `-sdh1 8:113 0 110G 0 part sdi 8:128 0 200G 0 disk `-sdi1 8:129 0 200G 0 part }}} * 结合dmesg命令输出确认新增加的磁盘信息 == 分区对齐 == 分区LUN时，建议分区对齐1M边界。使用 fdisk 或 parted 创建分区。但是，只有 parted 可以创建大于 2 TB 的分区。 本项目中将使用 parted 创建分区 在创建分区之前，将设备标记为 GPT。然后，指定分区偏移量为 2048 扇区（1M）。以下命令创建一个占用整个 LUN 的分区。创建分区后，使用分区文件 /dev/mapper/ora-asm-data-001p1 创建 ASMLib 卷。 {{{ parted /dev/sdj mklabel gpt mkpart primary 2048s 100% }}} == 创建oracleasm磁盘 == === SYSTEM === oracleasm createdisk OCR01 /dev/sdj1 oracleasm createdisk OCR02 /dev/sdk1 oracleasm createdisk OCR03 /dev/sdl1 === DATA === oracleasm createdisk DATA01 /dev/sdm1 oracleasm createdisk DATA02 /dev/sdn1 oracleasm createdisk DATA03 /dev/sdo1 === RECO === oracleasm createdisk REDO1 /dev/sdp1 === 确认asm能够识别到生效后的磁盘 === Add the device mappings to the device mapper disks. kpartx -a /dev/mapper/DATA05-TEST |
| Line 117: | Line 161: |
| $ oracleasm scandisks $ oracleasm listdisks |
|
| Line 122: | Line 170: |
| === 首先在DATAVG中加入X1000存储的DATA LUN === {{{ alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA01p1' rebalance power 8; }}} {{{ alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA02p1' rebalance power 12; SQL> select * from v$asm_operation; GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- -------------------------------------------- 2 REBAL RUN 11 11 11870 851620 37158 22 }}} {{{ alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA03p1' rebalance power 12; }}} {{{ alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA04p1' rebalance power 12; }}} {{{ SQL> alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA05p1','/dev/oracleasm/disks/X1000-DATA06p1' rebalance power 12; Diskgroup altered. $ tail -f /u01/app/grid/diag/asm/+asm/+ASM/trace/alert.log SQL> alter diskgroup DATAVG add disk '/dev/oracleasm/disks/X1000-DATA05p1','/dev/oracleasm/disks/X1000-DATA06p1' rebalance power 12 }}} 备注：尽量在业务不繁忙的情况下操作; === 然后在REDOVG中加入X1000存储的REDO LUN === {{{ alter diskgroup REDOVG add disk '/dev/oracleasm/disks/X1000-REDO01p1' rebalance power 8; SQL> select * from v$asm_operation; GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- -------------------------------------------- 2 REBAL RUN 11 11 1082386 1082386 0 0 3 REBAL WAIT 8 $ tail -f alert_+ASM.log }}} {{{ alter diskgroup REDOVG add disk '/dev/oracleasm/disks/X1000-REDO02p1' rebalance power 5; }}} |
$ tail -f .../diag/asm/+asm/+ASM/trace/alert.log == 正式加入前创建磁盘组测试 == 为新oracleasm磁盘创建测试磁盘组以确认磁盘是可以正常使用 测试完成之后, 删除该测试磁盘组, 再执行加入现有磁盘组的操作 == 将磁盘OCR01-OCR03加入到SYSTEM磁盘组 == {{{ alter diskgroup SYSTEM add disk 'ORCL:CRS01','ORCL:CRS02','ORCL:CRS03' rebalance power 6; }}} 或者通过以下命令依次添加(建议一起添加). {{{ alter diskgroup SYSTEM add disk 'ORCL:CRS01' rebalance power 6; alter diskgroup SYSTEM add disk 'ORCL:CRS02' rebalance power 6; alter diskgroup SYSTEM add disk 'ORCL:CRS03' rebalance power 6; }}} SQL> select * from gv$asm_operation; === 然后在DATA中加入DATA磁盘 === {{{ alter diskgroup DATA add disk 'ORCL:DATA01','ORCL:DATA02','ORCL:DATA03' rebalance power 6; }}} === 然后在RECO中加入REDO磁盘 === {{{ alter diskgroup RECO add disk 'ORCL:REDO1' rebalance power 6; }}} |
| Line 175: | Line 203: |
| * login in as grid user 使用以下命令查看观察:当没有输出时则表明reblance完成。 {{{ select * from v$asm_operation; * 以下为加第一块时的输出，看EST_MINUTES大约需要29分钟。 SQL> select * from v$asm_operation; GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- -------------------------------------------- 2 REBAL RUN 8 8 33691 830701 26839 29 }}} $ asmcmd lsdsk -k {{{ [grid@MCPOS-HOST-01 ~]$ asmcmd lsdsk -k }}} {{{ |
|
| Line 195: | Line 204: |
| Group_Name Dsk_Num State Power EST_WORK EST_RATE EST_TIME DATAVG REBAL RUN 8 830701 0 0 |
|
| Line 199: | Line 206: |
| }}} | |
| Line 201: | Line 208: |
| 尽量在业务不繁忙的情况下操作; == 逐次剔除旧存储AFF8040对应的asm disk == === DROP AFF8040 REDOVG MEMBERs === |
== 逐次剔除旧存储对应的asm disk == |
| Line 207: | Line 212: |
| alter diskgroup REDOVG drop disk REDOVG_0002 rebalance power 11; | alter diskgroup SYSTEM drop disk OCR1 rebalance power 6; alter diskgroup SYSTEM drop disk OCR2 rebalance power 6; alter diskgroup SYSTEM drop disk OCR3 rebalance power 6; |
| Line 210: | Line 217: |
| Line 212: | Line 220: |
| alter diskgroup REDOVG drop disk REDOVG_0003 rebalance power 11; | alter diskgroup DATA drop disk DATA1 rebalance power 6; alter diskgroup DATA drop disk DATA2 rebalance power 6; alter diskgroup DATA drop disk DATA3 rebalance power 6; alter diskgroup DATA drop disk RECO1 rebalance power 6; |
| Line 216: | Line 227: |
| === DROP AFF8040 DATAVG MEMBERs === {{{ #! /bin/bash RETVAL=`sqlplus -s '/ as sysasm' << "EOF" select * from v$asm_operation; EXIT; EOF` if [ -z "$RETVAL" ]; then echo "No rows returned from database" exit 0 else echo $RETVAL echo "DROP FAILED" fi }}} {{{ sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATAVG drop disk DATAVG_0003 rebalance power 11; EOF }}} {{{ sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATAVG drop disk DATAVG_0004 rebalance power 11; EOF }}} {{{ sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATAVG drop disk DATAVG_0005 rebalance power 11; EOF }}} {{{ sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATAVG drop disk DATAVG_0006 rebalance power 11; EOF }}} {{{ sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATAVG drop disk DATAVG_0007 rebalance power 11; EOF }}} {{{ sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATAVG drop disk DATAVG_0008 rebalance power 11; EOF }}} * select * from v$asm_operation; == 查看rebalance情况与asm日志对比 == {{{ |
* select * from gv$asm_operation; == 查看rebalance情况与asm日志 == |
| Line 274: | Line 231: |
| Line 275: | Line 233: |
| }}} == 删除ARCHVG磁盘组 == {{{ sqlplus / as sysasm SQL> alter diskgroup ARCHVG dismount; SQL> drop diskgroup ARCHVG force including contents; }}} |
== 依次回收VMware上的磁盘 == 先验证一个磁盘回收, 如无异常, 回收所有旧存储的磁盘. 最后建议是重启验证一下, 如果可能的话. |
Describe 首頁/work/QH-htrac-VM-replace-disk here.
Contents
明确htrac数据库存储迁移需求
将现有的EMC存储替换为PowerStore X1000存储，采用asm rebalance技术进行在线迁移。
前期准备工作
执行备份，建议手动再备份asm磁盘组的归档日志
SQL> alter system switch logfile
ASM copy archivelog FROM ASM to LOCAL
for i in $(asmcmd ls fra/CARPOLY/ARCHIVELOG/2019_12_29); do asmcmd cp +fra/CARPOLY/ARCHIVELOG/2019_12_29/$i /mnt/backup/cloud1 done
查看ASM磁盘组信息
- login in as grid user
$ asmcmd lsdg
[grid@htrac1 ~]$ asmcmd lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 624628 183534 0 183534 0 N DATA/ MOUNTED EXTERN N 512 4096 1048576 102398 27343 0 27343 0 N RECO/ MOUNTED NORMAL N 512 4096 1048576 15345 14419 5115 4652 0 Y SYSTEM/
查看ASM 磁盘空间使用情况
- login in as grid user
$ asmcmd lsdsk -k
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path
307196 90267 307196 DATA1 DATA1 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA1 UNKNOWN ORCL:DATA1
112635 33092 112635 DATA2 DATA2 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA2 UNKNOWN ORCL:DATA2
204797 60175 204797 DATA3 DATA3 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) DATA3 UNKNOWN ORCL:DATA3
5115 4807 5115 OCR1 OCR1 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) OCR1 UNKNOWN ORCL:OCR1
5115 4805 5115 OCR2 OCR2 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) OCR2 UNKNOWN ORCL:OCR2
5115 4807 5115 OCR3 OCR3 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) OCR3 UNKNOWN ORCL:OCR3
102398 27343 102398 RECO1 RECO1 REGULAR ASM Library - Generic Linux, version 2.0.4 (KABI_V2) RECO1 UNKNOWN ORCL:RECO1$ sqlplus / as sysasm
SQL> set linesize 300 column path format a50 column NAME format a15; select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk; GROUP_NUMBER DISK_NUMBER HEADER_STATUS MODE_STATUS MOUNT_STATUS TOTAL_MB FREE_MB NAME PATH ------------ ----------- ------------------------ -------------- -------------- ---------- ---------- ---------------- -------------------- 2 0 MEMBER ONLINE CACHED 30719690267 DATA1 ORCL:DATA1 1 0 MEMBER ONLINE CACHED 5115 4807 OCR1 ORCL:OCR1 1 1 MEMBER ONLINE CACHED 5115 4805 OCR2 ORCL:OCR2 1 2 MEMBER ONLINE CACHED 5115 4807 OCR3 ORCL:OCR3 3 0 MEMBER ONLINE CACHED 10239827343 RECO1 ORCL:RECO1 2 1 MEMBER ONLINE CACHED 11263533092 DATA2 ORCL:DATA2 2 2 MEMBER ONLINE CACHED 20479760175 DATA3 ORCL:DATA3 7 rows selected.
[grid@htrac1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6a7669a7d58c4f22bf06341bbacde673 (ORCL:OCR1) [SYSTEM] 2. ONLINE 076dc3fef5374f2cbf8af0958ffc5a9b (ORCL:OCR2) [SYSTEM] 3. ONLINE 9f6d65dc01364feebf566749760ced41 (ORCL:OCR3) [SYSTEM]
/etc/init.d/oracleasm querydisk -d OCR1
/etc/init.d/oracleasm querydisk -d OCR2
/etc/init.d/oracleasm querydisk -d OCR3
ls -l /dev/ | grep 8 | grep 65
给htrac数据库虚拟化分配共享vmdk磁盘
参考VMware KB 1034165
htrac数据库存储迁移步骤
查看并确认新挂载的磁盘状态
$ lsblk
[root@htrac1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 3.6G 0 loop /mnt sr0 11:0 1 3.6G 0 rom sdb 8:16 0 5G 0 disk `-sdb1 8:17 0 5G 0 part sdc 8:32 0 5G 0 disk `-sdc1 8:33 0 5G 0 part sde 8:64 0 300G 0 disk `-sde1 8:65 0 300G 0 part sda 8:0 0 300G 0 disk |-sda1 8:1 0 500M 0 part /boot `-sda2 8:2 0 299.5G 0 part |-vg_htdb1-lv_root (dm-0) 253:0 0 83.5G 0 lvm / |-vg_htdb1-lv_swap (dm-1) 253:1 0 16G 0 lvm [SWAP] `-vg_htdb1-lv_backup (dm-2) 253:2 0 300G 0 lvm /backup sdd 8:48 0 5G 0 disk `-sdd1 8:49 0 5G 0 part sdf 8:80 0 100G 0 disk `-sdf1 8:81 0 100G 0 part sdg 8:96 0 100G 0 disk `-vg_htdb1-lv_backup (dm-2) 253:2 0 300G 0 lvm /backup sdh 8:112 0 110G 0 disk `-sdh1 8:113 0 110G 0 part sdi 8:128 0 200G 0 disk `-sdi1 8:129 0 200G 0 part
* 结合dmesg命令输出确认新增加的磁盘信息
分区对齐
分区LUN时，建议分区对齐1M边界。使用 fdisk 或 parted 创建分区。但是，只有 parted 可以创建大于 2 TB 的分区。 本项目中将使用 parted 创建分区
在创建分区之前，将设备标记为 GPT。然后，指定分区偏移量为 2048 扇区（1M）。以下命令创建一个占用整个 LUN 的分区。创建分区后，使用分区文件 /dev/mapper/ora-asm-data-001p1 创建 ASMLib 卷。
parted /dev/sdj mklabel gpt mkpart primary 2048s 100%
创建oracleasm磁盘
SYSTEM
oracleasm createdisk OCR01 /dev/sdj1
oracleasm createdisk OCR02 /dev/sdk1
oracleasm createdisk OCR03 /dev/sdl1
DATA
oracleasm createdisk DATA01 /dev/sdm1
oracleasm createdisk DATA02 /dev/sdn1
oracleasm createdisk DATA03 /dev/sdo1
RECO
oracleasm createdisk REDO1 /dev/sdp1
确认asm能够识别到生效后的磁盘
Add the device mappings to the device mapper disks.
kpartx -a /dev/mapper/DATA05-TEST
- login in as grid user
$ oracleasm scandisks
$ oracleasm listdisks
$ sqlplus / as sysasm
SQL> select GROUP_NUMBER,DISK_NUMBER,HEADER_STATUS,MODE_STATUS,MOUNT_STATUS,STATE,total_mb,free_mb,NAME,PATH from v$asm_disk;
$ tail -f .../diag/asm/+asm/+ASM/trace/alert.log
正式加入前创建磁盘组测试
为新oracleasm磁盘创建测试磁盘组以确认磁盘是可以正常使用
测试完成之后, 删除该测试磁盘组, 再执行加入现有磁盘组的操作
将磁盘OCR01-OCR03加入到SYSTEM磁盘组
alter diskgroup SYSTEM add disk 'ORCL:CRS01','ORCL:CRS02','ORCL:CRS03' rebalance power 6;
或者通过以下命令依次添加(建议一起添加).
alter diskgroup SYSTEM add disk 'ORCL:CRS01' rebalance power 6; alter diskgroup SYSTEM add disk 'ORCL:CRS02' rebalance power 6; alter diskgroup SYSTEM add disk 'ORCL:CRS03' rebalance power 6;
SQL> select * from gv$asm_operation;
然后在DATA中加入DATA磁盘
alter diskgroup DATA add disk 'ORCL:DATA01','ORCL:DATA02','ORCL:DATA03' rebalance power 6;
然后在RECO中加入REDO磁盘
alter diskgroup RECO add disk 'ORCL:REDO1' rebalance power 6;
查看rebalance情况与日志对比
$ asmcmd -p lsop
$ tail -f alert.log
备注：确认所有磁盘rebalance完成后，再进行下一步删盘操作；
逐次剔除旧存储对应的asm disk
sqlplus -s '/ as sysasm' << "EOF" alter diskgroup SYSTEM drop disk OCR1 rebalance power 6; alter diskgroup SYSTEM drop disk OCR2 rebalance power 6; alter diskgroup SYSTEM drop disk OCR3 rebalance power 6; EOF
sqlplus -s '/ as sysasm' << "EOF" alter diskgroup DATA drop disk DATA1 rebalance power 6; alter diskgroup DATA drop disk DATA2 rebalance power 6; alter diskgroup DATA drop disk DATA3 rebalance power 6; alter diskgroup DATA drop disk RECO1 rebalance power 6; EOF
- select * from gv$asm_operation;
查看rebalance情况与asm日志
$ asmcmd -p lsop
$ tail -f alert.log
依次回收VMware上的磁盘
先验证一个磁盘回收, 如无异常, 回收所有旧存储的磁盘.
最后建议是重启验证一下, 如果可能的话.