Exadata中的Quorum Disk
前言
Oracle数据库从11g开始，就支持在ASM磁盘组中存放OCR和Votingdisk，但Votingdisk要求Normal冗余的磁盘组至少有3组failgroup，High冗余的磁盘组至少有5组failgroup。
由于Exadata环境，一个存储节点才是一组failgroup，所以对于一个小型的Exadata环境（2个计算+3个存储的架构），存放OCR和Votingdisk的ASM磁盘组就只能是Normal冗余，由于Normal冗余只有两个副本，所以数据的安全性不如High冗余高。为了解决这个问题，就引入了Quorum Disk特性。
Quorum Disk特性，就是在每个计算节点上各划出一个磁盘空间，通过iSCS映射共享出来，这样，计算节点iSCS映射出来的两块盘，再加上3个存储节点，构成了5个failgroup，此时的磁盘组也可以创建成 High冗余,用来存放OCR和Votingdisk。计算节点上划出的这个Quorum Disk比较特殊，它不会存放用户数据，只是作为表决盘的作用。
Quorum Disk概念
Exadata uses quorum disks to maintain redundancy and high availability for critical Oracle ASM metadata and clusterware voting files on small Exadata systems.
A quorum failure group is a special type of failure group that does not contain user data. A quorum failure group contains only a quorum disk, which may be used to store copies of the PST and clusterware voting files. Quorum failure groups (and quorum disks) are only required on Exadata systems that do not contain enough storage servers to provide the required minimum number of failure groups. The most common requirement for quorum failure groups (and quorum disks) is on Exadata systems with high redundancy ASM disk groups and fewer than 5 Exadata storage servers.
You can create quorum failure groups (and quorum disks) with Oracle Exadata Deployment Assistant (OEDA) as part of the Exadata deployment process or create and manage them later using the Quorum Disk Manager Utility.
Quorum disks are implemented on Exadata database servers using iSCSI devices. The iSCSI quorum disk implementation leverages high performance and high availability by using the Exadata RDMA Network Fabric. As illustrated in the following diagram, the quorum disks leverages a multipath device where each path corresponds to a separate iSCSI device for each RDMA Network Fabric port on the database node (one for ib0 or re0 and the other for ib1 or re1).
The Quorum Disk Manager utility enables you to create and manage quorum disks on the Exadata database servers. The Quorum Disk Manager utility (quorumdiskmgr) can create and manage all the necessary components for implementing quorum disks, including the iSCSI configuration, the iSCSI targets, the iSCSI LUNs, and the iSCSI devices.
Quorum Disk相关的一些操作命令：
创建基于IB交换机的Quorum Disk配置文件：
quorumdiskmgr --create --config --owner=oracle --group=dba --network-iface-list="ib0, ib1"
创建基于RoCE交换机的Quorum Disk配置文件：
quorumdiskmgr --create --config --owner=oracle --group=dba --network-iface-list="re0, re1"
创建target：
quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.45, 192.168.10.46" --host-name=db01
创建device：
quorumdiskmgr --create --device --target-ip-list="192.168.10.45, 192.168.10.46"
查看Quorum Disk配置文件：
quorumdiskmgr --list --config
查看target：
quorumdiskmgr --list --target --asm-disk-group=datac1
查看device：
quorumdiskmgr --list --device --asm-disk-group datac1
删除Quorum Disk配置文件：
quorumdiskmgr --delete --config
删除target：
quorumdiskmgr --delete --target --asm-disk-group=datac1
删除device：
quorumdiskmgr --delete --device --host-name=db01
示例：
| GROUP_NUMBER DISK_NUMBER NAME PATH FAILGROUP TOTAL_MB FREE_MB ------------ ----------- ------------------------------ ------------------------------------------------ ---------- ---------- ---------- 1 0 DATA_EX01_CD_00_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_00_ex01cel01 EX01CEL01 1978368 451276 1 1 DATA_EX01_CD_01_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_01_ex01cel01 EX01CEL01 1978368 451784 1 2 DATA_EX01_CD_02_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_02_ex01cel01 EX01CEL01 1978368 451492 1 3 DATA_EX01_CD_03_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_03_ex01cel01 EX01CEL01 1978368 451616 1 4 DATA_EX01_CD_04_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_04_ex01cel01 EX01CEL01 1978368 452152 1 5 DATA_EX01_CD_05_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_05_ex01cel01 EX01CEL01 1978368 452352 1 6 DATA_EX01_CD_06_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_06_ex01cel01 EX01CEL01 1978368 452160 1 7 DATA_EX01_CD_07_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_07_ex01cel01 EX01CEL01 1978368 451164 1 8 DATA_EX01_CD_08_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_08_ex01cel01 EX01CEL01 1978368 451872 1 9 DATA_EX01_CD_09_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_09_ex01cel01 EX01CEL01 1978368 451692 1 10 DATA_EX01_CD_10_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_10_ex01cel01 EX01CEL01 1978368 452312 1 11 DATA_EX01_CD_11_EX01CEL01 o/192.168.10.3/DATA_EX01_CD_11_ex01cel01 EX01CEL01 1978368 451492 1 12 DATA_EX01_CD_00_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_00_ex01cel02 EX01CEL02 1978368 451632 1 13 DATA_EX01_CD_01_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_01_ex01cel02 EX01CEL02 1978368 451536 1 14 DATA_EX01_CD_02_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_02_ex01cel02 EX01CEL02 1978368 451576 1 15 DATA_EX01_CD_03_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_03_ex01cel02 EX01CEL02 1978368 451916 1 16 DATA_EX01_CD_04_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_04_ex01cel02 EX01CEL02 1978368 452068 1 17 DATA_EX01_CD_05_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_05_ex01cel02 EX01CEL02 1978368 452148 1 18 DATA_EX01_CD_06_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_06_ex01cel02 EX01CEL02 1978368 451604 1 19 DATA_EX01_CD_07_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_07_ex01cel02 EX01CEL02 1978368 451556 1 20 DATA_EX01_CD_08_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_08_ex01cel02 EX01CEL02 1978368 451828 1 21 DATA_EX01_CD_09_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_09_ex01cel02 EX01CEL02 1978368 451516 1 22 DATA_EX01_CD_10_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_10_ex01cel02 EX01CEL02 1978368 452352 1 23 DATA_EX01_CD_11_EX01CEL02 o/192.168.10.4/DATA_EX01_CD_11_ex01cel02 EX01CEL02 1978368 451588 1 24 DATA_EX01_CD_00_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_00_ex01cel03 EX01CEL03 1978368 452032 1 25 DATA_EX01_CD_01_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_01_ex01cel03 EX01CEL03 1978368 451456 1 26 DATA_EX01_CD_02_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_02_ex01cel03 EX01CEL03 1978368 451324 1 27 DATA_EX01_CD_03_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_03_ex01cel03 EX01CEL03 1978368 451692 1 28 DATA_EX01_CD_04_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_04_ex01cel03 EX01CEL03 1978368 452400 1 29 DATA_EX01_CD_05_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_05_ex01cel03 EX01CEL03 1978368 452368 1 30 DATA_EX01_CD_06_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_06_ex01cel03 EX01CEL03 1978368 451884 1 31 DATA_EX01_CD_07_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_07_ex01cel03 EX01CEL03 1978368 451260 1 32 DATA_EX01_CD_08_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_08_ex01cel03 EX01CEL03 1978368 451856 1 33 DATA_EX01_CD_09_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_09_ex01cel03 EX01CEL03 1978368 451068 1 34 DATA_EX01_CD_10_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_10_ex01cel03 EX01CEL03 1978368 451820 1 35 DATA_EX01_CD_11_EX01CEL03 o/192.168.10.5/DATA_EX01_CD_11_ex01cel03 EX01CEL03 1978368 452268 1 36 QD_DATA_EX01_EX01DB01 /dev/exadata_quorum/QD_DATA_EX01_EX01DB01 EX01DB01 128 116 1 37 QD_DATA_EX01_EX01DB02 /dev/exadata_quorum/QD_DATA_EX01_EX01DB02 EX01DB02 128 116 2 6 DBFS_DG_CD_02_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_02_ex01cel01 EX01CEL01 34496 34420 2 25 DBFS_DG_CD_03_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_03_ex01cel01 EX01CEL01 34496 34416 2 8 DBFS_DG_CD_04_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_04_ex01cel01 EX01CEL01 34496 34436 2 30 DBFS_DG_CD_05_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_05_ex01cel01 EX01CEL01 34496 34432 2 11 DBFS_DG_CD_06_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_06_ex01cel01 EX01CEL01 34496 34436 2 4 DBFS_DG_CD_07_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_07_ex01cel01 EX01CEL01 34496 34424 2 14 DBFS_DG_CD_08_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_08_ex01cel01 EX01CEL01 34496 34440 2 2 DBFS_DG_CD_09_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_09_ex01cel01 EX01CEL01 34496 34432 2 10 DBFS_DG_CD_10_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_10_ex01cel01 EX01CEL01 34496 34396 2 27 DBFS_DG_CD_11_EX01CEL01 o/192.168.10.3/DBFS_DG_CD_11_ex01cel01 EX01CEL01 34496 34424 2 28 DBFS_DG_CD_02_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_02_ex01cel02 EX01CEL02 34496 34420 2 18 DBFS_DG_CD_03_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_03_ex01cel02 EX01CEL02 34496 34424 2 5 DBFS_DG_CD_04_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_04_ex01cel02 EX01CEL02 34496 34420 2 16 DBFS_DG_CD_05_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_05_ex01cel02 EX01CEL02 34496 34444 2 0 DBFS_DG_CD_06_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_06_ex01cel02 EX01CEL02 34496 34388 2 26 DBFS_DG_CD_07_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_07_ex01cel02 EX01CEL02 34496 34416 2 9 DBFS_DG_CD_08_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_08_ex01cel02 EX01CEL02 34496 34444 2 21 DBFS_DG_CD_09_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_09_ex01cel02 EX01CEL02 34496 34428 2 3 DBFS_DG_CD_10_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_10_ex01cel02 EX01CEL02 34496 34428 2 19 DBFS_DG_CD_11_EX01CEL02 o/192.168.10.4/DBFS_DG_CD_11_ex01cel02 EX01CEL02 34496 34444 2 17 DBFS_DG_CD_02_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_02_ex01cel03 EX01CEL03 34496 34440 2 12 DBFS_DG_CD_03_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_03_ex01cel03 EX01CEL03 34496 34420 2 31 DBFS_DG_CD_04_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_04_ex01cel03 EX01CEL03 34496 34432 2 7 DBFS_DG_CD_05_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_05_ex01cel03 EX01CEL03 34496 34440 2 23 DBFS_DG_CD_06_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_06_ex01cel03 EX01CEL03 34496 34424 2 20 DBFS_DG_CD_07_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_07_ex01cel03 EX01CEL03 34496 34424 2 1 DBFS_DG_CD_08_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_08_ex01cel03 EX01CEL03 34496 34376 2 15 DBFS_DG_CD_09_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_09_ex01cel03 EX01CEL03 34496 34444 2 22 DBFS_DG_CD_10_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_10_ex01cel03 EX01CEL03 34496 34432 2 13 DBFS_DG_CD_11_EX01CEL03 o/192.168.10.5/DBFS_DG_CD_11_ex01cel03 EX01CEL03 34496 34424 2 29 QD_DBFS_DG_EX01DB01 /dev/exadata_quorum/QD_DBFS_DG_EX01DB01 EX01DB01 128 84 2 24 QD_DBFS_DG_EX01DB02 /dev/exadata_quorum/QD_DBFS_DG_EX01DB02 EX01DB02 128 84 3 0 RECO_EX01_CD_00_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_00_ex01cel01 EX01CEL01 848096 633500 3 1 RECO_EX01_CD_01_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_01_ex01cel01 EX01CEL01 848096 633644 3 2 RECO_EX01_CD_02_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_02_ex01cel01 EX01CEL01 848096 633292 3 3 RECO_EX01_CD_03_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_03_ex01cel01 EX01CEL01 848096 633308 3 4 RECO_EX01_CD_04_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_04_ex01cel01 EX01CEL01 848096 633572 3 5 RECO_EX01_CD_05_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_05_ex01cel01 EX01CEL01 848096 633824 3 6 RECO_EX01_CD_06_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_06_ex01cel01 EX01CEL01 848096 633200 3 7 RECO_EX01_CD_07_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_07_ex01cel01 EX01CEL01 848096 633112 3 8 RECO_EX01_CD_08_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_08_ex01cel01 EX01CEL01 848096 633644 3 9 RECO_EX01_CD_09_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_09_ex01cel01 EX01CEL01 848096 633744 3 10 RECO_EX01_CD_10_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_10_ex01cel01 EX01CEL01 848096 633688 3 11 RECO_EX01_CD_11_EX01CEL01 o/192.168.10.3/RECO_EX01_CD_11_ex01cel01 EX01CEL01 848096 633272 3 12 RECO_EX01_CD_00_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_00_ex01cel02 EX01CEL02 848096 633612 3 13 RECO_EX01_CD_01_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_01_ex01cel02 EX01CEL02 848096 633636 3 14 RECO_EX01_CD_02_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_02_ex01cel02 EX01CEL02 848096 634072 3 15 RECO_EX01_CD_03_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_03_ex01cel02 EX01CEL02 848096 633464 3 16 RECO_EX01_CD_04_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_04_ex01cel02 EX01CEL02 848096 633356 3 17 RECO_EX01_CD_05_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_05_ex01cel02 EX01CEL02 848096 632788 3 18 RECO_EX01_CD_06_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_06_ex01cel02 EX01CEL02 848096 633204 3 19 RECO_EX01_CD_07_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_07_ex01cel02 EX01CEL02 848096 633316 3 20 RECO_EX01_CD_08_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_08_ex01cel02 EX01CEL02 848096 633588 3 21 RECO_EX01_CD_09_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_09_ex01cel02 EX01CEL02 848096 633372 3 22 RECO_EX01_CD_10_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_10_ex01cel02 EX01CEL02 848096 633632 3 23 RECO_EX01_CD_11_EX01CEL02 o/192.168.10.4/RECO_EX01_CD_11_ex01cel02 EX01CEL02 848096 633760 3 24 RECO_EX01_CD_00_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_00_ex01cel03 EX01CEL03 848096 633288 3 25 RECO_EX01_CD_01_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_01_ex01cel03 EX01CEL03 848096 633724 3 26 RECO_EX01_CD_02_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_02_ex01cel03 EX01CEL03 848096 633652 3 27 RECO_EX01_CD_03_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_03_ex01cel03 EX01CEL03 848096 634020 3 28 RECO_EX01_CD_04_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_04_ex01cel03 EX01CEL03 848096 633320 3 29 RECO_EX01_CD_05_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_05_ex01cel03 EX01CEL03 848096 633620 3 30 RECO_EX01_CD_06_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_06_ex01cel03 EX01CEL03 848096 632888 3 31 RECO_EX01_CD_07_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_07_ex01cel03 EX01CEL03 848096 633712 3 32 RECO_EX01_CD_08_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_08_ex01cel03 EX01CEL03 848096 633172 3 33 RECO_EX01_CD_09_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_09_ex01cel03 EX01CEL03 848096 633644 3 34 RECO_EX01_CD_10_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_10_ex01cel03 EX01CEL03 848096 633384 3 35 RECO_EX01_CD_11_EX01CEL03 o/192.168.10.5/RECO_EX01_CD_11_ex01cel03 EX01CEL03 848096 633376 3 36 QD_RECO_EX01_EX01DB01 /dev/exadata_quorum/QD_RECO_EX01_EX01DB01 EX01DB01 128 116 3 37 QD_RECO_EX01_EX01DB02 /dev/exadata_quorum/QD_RECO_EX01_EX01DB02 EX01DB02 128 116 108 rows selected. |
从这个示例可以看出，v$ASM_DISK中那些/dev/exadata_quora开头的磁盘就是quorum disk，大小为128M。
