Oracle集群OCR磁盘组掉盘问题处理
问题描述
填写问题的基础信息。
| 系统名称 | - |
| IP地址 | - |
| 操作系统 | HP-UNIX |
| 数据库 | Oracle 11.2.0.4 两节点RAC |
症状表现
问题的症状表现如下
集群的OCR磁盘组掉了一块盘(/dev/rdisk/disk52):
查询集群仲裁盘发现只有两块（原来是有三块）：
$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------1. ONLINE 08338bbafc7a4fd5bffe759528ad58a0 (/dev/rdisk/disk51) [OCR_VOTE]2. ONLINE 311f8eb857bd4f18bf862055a7f22a0c (/dev/rdisk/disk53) [OCR_VOTE]
Located 2 voting disk(s).
$ ocrcheck
Status of Oracle Cluster Registry is as follows :Version : 3Total space (kbytes) : 262120Used space (kbytes) : 2952Available space (kbytes) : 259168ID : 112220174Device/File Name : +OCR_VOTEDevice/File integrity check succeededDevice/File not configuredDevice/File not configuredDevice/File not configuredDevice/File not configuredCluster registry integrity check succeededLogical corruption check bypassed due to non-privileged user$ 查看ASM告警日志发现/dev/rdisk/disk52已经被offline drop了：
处理过程
处理过程推荐按照时间以列表形式，将处理过程时间点，处理内容。
现状分析：
既然已经被offline_drop了，日常lsdg也不会看到对应磁盘组有offline disk计数信息：
在这样的情况下，/dev/rhsidk/disk52虽然被踢出ocr磁盘组，但是该磁盘头信息还有关于ocr_vote磁盘组的元数据信息，如果这个时候再把/dev/rhdisk/disk52添加到OCR_VOTE磁盘组，会报错如下：
SQL> alter diskgroup OCR_VOTE add disk '/dev/rhsidk/disk52';
alter diskgroup OCR_VOTE add disk '/dev/rhsidk/disk52'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk '/dev/rhsidk/disk52' belongs to diskgroup "OCR_VOTE"如果尝试强制添加该磁盘到OCR_VOTE磁盘组，可能会导致实例终止（来自网络上的测试）：
ocr磁盘组掉盘故障处理 – 提供7*24专业数据库(Oracle,SQL Server,MySQL,PostgreSQL等)恢复和技术支持@Tel:17813235971 - 惜分飞
解决思路：
1、将/dev/rdisk/hdisk52的磁盘头信息清除之后再加入到OCR_VOTE磁盘组，保证集群仲裁三副本冗余。
2、存储上再划分一小块大小相同的磁盘，添加进OCR_VOTE磁盘组中，保证集群仲裁三副本冗余。
问题原因
问题原因如下
由于磁盘或者操作系统IO故障导致/dev/rdisk/hdisk52被踢出OCR_VOTE磁盘组，目前集群仲裁盘不能保证三副本冗余。
问题解决
问题解决如下
建议选择方案2，服务器上尽量不要做清理磁盘头的操作（不怕万一，就怕一万）。等待新的磁盘加入到OCR_VOTE磁盘组后，磁盘组内部会自动进行refresh，新磁盘会被自动同步集群仲裁信息，以下是测试环境中的测试：
测试验证
先找一个测试环境验证一下流程
测试环境：Oracle 11.2.0.4 RAC Centos7
准备一个正常的环境：
[grid@centos73 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------1. ONLINE 2654cf6472314f07bfefab8fc83311e4 (/dev/raw/raw1) [OCR]2. ONLINE 9d3d26e93ac14fb2bf7c8b2e2df979bc (/dev/raw/raw2) [OCR]3. ONLINE 0ad3fa5c46874fc4bf5e3e70df47cd3a (/dev/raw/raw3) [OCR]
Located 3 voting disk(s).
[grid@centos73 ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Tue Jun 10 16:53:48 2025Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management optionsSQL> set lines 900
col path for a65
select g.name,d.header_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU STATE PATH
------------------------------ ------------ ----------- -----------------------------------------------------------------
DATA MEMBER MOUNTED /dev/raw/raw4
OCR MEMBER MOUNTED /dev/raw/raw2
OCR MEMBER MOUNTED /dev/raw/raw3
OCR MEMBER MOUNTED /dev/raw/raw1模拟OCR磁盘组掉盘
#正常情况下，磁盘组内磁盘第0个au和第510个au互为镜像，asm实例主要通过第一个块来判断那块盘属于哪个磁盘组
[grid@centos73 ~]$ kfed read /dev/raw/raw2 blknum=510 |egrep 'ausize|dsknum|dskname|grpname|fgname|grpnum|dsksize|blksize'
kfdhdb.dsknum: 1 ; 0x024: 0x0001
kfdhdb.dskname: OCR_0001 ; 0x028: length=8
kfdhdb.grpname: OCR ; 0x048: length=3
kfdhdb.fgname: OCR_0001 ; 0x068: length=8
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize: 2048 ; 0x0c4: 0x00000800
[grid@centos73 ~]$ kfed read /dev/raw/raw2 blknum=0 |egrep 'ausize|dsknum|dskname|grpname|fgname|grpnum|dsksize|blksize'
kfdhdb.dsknum: 1 ; 0x024: 0x0001
kfdhdb.dskname: OCR_0001 ; 0x028: length=8
kfdhdb.grpname: OCR ; 0x048: length=3
kfdhdb.fgname: OCR_0001 ; 0x068: length=8
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize: 2048 ; 0x0c4: 0x00000800先将磁盘的前50个au进行备份：
dd if=/dev/raw/raw2 of=/tmp/raw2.dump bs=1048576 count=50然后破坏一下该磁盘（记得使用conv=notrunc用法）：
dd if=/dev/zero of=/dev/raw/raw2 bs=1048576 count=50 conv=notrunc问题复现：
重启集群之后，ASM实例报告OCR掉了一块盘(/dev/raw/raw2):
OTE: Diskgroup used for Voting files is:OCR
Diskgroup with spfile:OCR
Diskgroup used for OCR is:OCR
NOTE: cache registered group DATA number=1 incarn=0x8e3f07c5
NOTE: cache began mount (first) of group DATA number=1 incarn=0x8e3f07c5
NOTE: cache registered group OCR number=2 incarn=0x8e3f07c6
NOTE: cache began mount (first) of group OCR number=2 incarn=0x8e3f07c6
NOTE: Assigning number (1,0) to disk (/dev/raw/raw4)
NOTE: Assigning number (2,0) to disk (/dev/raw/raw1)
NOTE: Assigning number (2,2) to disk (/dev/raw/raw3)
WARNING: GMON has insufficient disks to maintain consensus. Minimum required is 2: updating 2 PST copies from a total of 3.
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 4 for pid 23, osid 9601
NOTE: cache opening disk 0 of grp 1: DATA_0000 path:/dev/raw/raw4
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache mounting (first) external redundancy group 1/0x8E3F07C5 (DATA)
* allocate domain 1, invalid = TRUE
NOTE: attached to recovery domain 1
NOTE: cache recovered group 1 to fcn 0.29494
NOTE: redo buffer size is 256 blocks (1053184 bytes)
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATA)
Process LGWR (pid 9581) is running at high priority QoS for Exadata I/O
NOTE: LGWR found thread 1 closed at ABA 25.4191
NOTE: LGWR mounted thread 1 for diskgroup 1 (DATA)
NOTE: LGWR opening thread 1 at fcn 0.29494 ABA 26.4192
NOTE: cache mounting group 1/0x8E3F07C5 (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x8e3f07c5
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: GMON heartbeating for grp 2
GMON querying group 2 at 6 for pid 23, osid 9601
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: Assigning number (2,1) to disk ()
GMON querying group 2 at 7 for pid 23, osid 9601
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: cache dismounting (clean) group 2/0x8E3F07C6 (OCR)
NOTE: messaging CKPT to quiesce pins Unix process pid: 9601, image: oracle@centos73 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 2/0x8E3F07C6 (OCR)
NOTE: cache ending mount (fail) of group OCR number=2 incarn=0x8e3f07c6
NOTE: cache deleting context for group OCR 2/0x8e3f07c6
GMON dismounting group 2 at 8 for pid 23, osid 9601
NOTE: Disk OCR_0000 in mode 0x7f marked for de-assignment
NOTE: Disk in mode 0x7f marked for de-assignment
NOTE: Disk OCR_0002 in mode 0x7f marked for de-assignment
ERROR: diskgroup OCR was not mounted
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup DATA was mounted
WARNING: Disk Group OCR containing spfile for this instance is not mounted
WARNING: Disk Group OCR containing configured OCR is not mounted
WARNING: Disk Group OCR containing voting files is not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "2"
这个时候集群是启动失败的，卡在了ocr启动失败：
[grid@centos73 ~]$ crsctl status res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[grid@centos73 ~]$ crsctl status res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm1 ONLINE INTERMEDIATE centos73 OCR not started
ora.cluster_interconnect.haip1 ONLINE ONLINE centos73
ora.crf1 ONLINE ONLINE centos73
ora.crsd1 ONLINE OFFLINE
ora.cssd1 ONLINE ONLINE centos73
ora.cssdmonitor1 ONLINE ONLINE centos73
ora.ctssd1 ONLINE ONLINE centos73 ACTIVE:0
ora.diskmon1 OFFLINE OFFLINE
ora.evmd1 ONLINE INTERMEDIATE centos73
ora.gipcd1 ONLINE ONLINE centos73
ora.gpnpd1 ONLINE ONLINE centos73
ora.mdnsd1 ONLINE ONLINE centos73 这个时候需要强制mount一下ocr磁盘组：
强制mount之后，集群会以两个仲裁盘形式运行，同样的ocr会掉一块盘：
[grid@centos73 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------1. ONLINE 2654cf6472314f07bfefab8fc83311e4 (/dev/raw/raw1) [OCR]2. ONLINE 0ad3fa5c46874fc4bf5e3e70df47cd3a (/dev/raw/raw3) [OCR]
Located 2 voting disk(s).
[grid@centos73 ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Tue Jun 10 17:29:32 2025Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management optionsSQL> set lines 900
col path for a65
select g.name,d.header_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU STATE PATH
------------------------------ ------------ ----------- -----------------------------------------------------------------
DATA MEMBER MOUNTED /dev/raw/raw4
OCR MEMBER MOUNTED /dev/raw/raw3
OCR MEMBER MOUNTED /dev/raw/raw1
asm告警日志也会显示先前故障磁盘被强制drop了
SUCCESS: diskgroup OCR was mounted
SUCCESS: alter diskgroup ocr mount force
Tue Jun 10 17:26:28 2025
ASM Health Checker found 1 new failures
Tue Jun 10 17:26:29 2025
WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
ERROR: failed to update diskgroup resource ora.OCR.dg
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
Tue Jun 10 17:26:45 2025
WARNING: PST-initiated drop of 1 disk(s) in group 2(.1618937807))
SQL> alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=2
Tue Jun 10 17:26:48 2025
GMON updating for reconfiguration, group 2 at 23 for pid 26, osid 10010
NOTE: cache closing disk 1 of grp 2: (not open) OCR_0001
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group 2 PST updated.
Tue Jun 10 17:26:48 2025
NOTE: membership refresh pending for group 2/0x607f07cf (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
GMON querying group 2 at 24 for pid 18, osid 9587
NOTE: cache closing disk 1 of grp 2: (not open) _DROPPED_0001_OCR
SUCCESS: refreshed membership for 2/0x607f07cf (OCR)
NOTE: starting rebalance of group 2/0x607f07cf (OCR) at power 1
SUCCESS: alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */
SUCCESS: PST-initiated drop disk in group 2(1618937807))
Starting background process ARB0
Tue Jun 10 17:26:51 2025
ARB0 started with pid=27, OS id=10023
NOTE: assigning ARB0 to group 2/0x607f07cf (OCR) with 1 parallel I/O
cellip.ora not found.
NOTE: F1X0 copy 2 relocating from 1:2 to 2:2 for diskgroup 2 (OCR)
NOTE: F1X0 copy 3 relocating from 2:2 to 65534:4294967294 for diskgroup 2 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
Tue Jun 10 17:26:54 2025
NOTE: [crsd.bin@centos73 (TNS V1-V3) 10028] opening OCR file
Starting background process ASMB
Tue Jun 10 17:26:54 2025
ASMB started with pid=29, OS id=10042
NOTE: Successful voting file relocation on diskgroup OCR
Tue Jun 10 17:26:54 2025
NOTE: client +ASM1:+ASM registered, osid 10044, mbr 0x0
Tue Jun 10 17:26:58 2025
NOTE: Rebalance has restored redundancy for any existing control file or redo log in disk group OCR
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 2/0x607f07cf (OCR)
Tue Jun 10 17:27:00 2025
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=2
Tue Jun 10 17:27:03 2025
GMON updating for reconfiguration, group 2 at 25 for pid 27, osid 10142
NOTE: cache closing disk 1 of grp 2: (not open) _DROPPED_0001_OCR
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group 2 PST updated.
SUCCESS: grp 2 disk _DROPPED_0001_OCR going offline
GMON updating for reconfiguration, group 2 at 26 for pid 27, osid 10142
NOTE: cache closing disk 1 of grp 2: (not open) _DROPPED_0001_OCR
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group 2 PST updated.
NOTE: membership refresh pending for group 2/0x607f07cf (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
GMON querying group 2 at 27 for pid 18, osid 9587
GMON querying group 2 at 28 for pid 18, osid 9587
NOTE: Disk _DROPPED_0001_OCR in mode 0x0 marked for de-assignment
SUCCESS: refreshed membership for 2/0x607f07cf (OCR)
Tue Jun 10 17:27:09 2025
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
Tue Jun 10 17:28:01 2025
ALTER SYSTEM SET local_listener=' (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.3.113)(PORT=1521))))' SCOPE=MEMORY SID='+ASM1';
Tue Jun 10 17:31:32 2025
这个时候我们再看一下v$asm_disk中的磁盘信息发现/dev/raw/raw2变成了CANDIDATE，这是因为我们先前是把/dev/raw/raw2磁盘头信息都清除了。
SQL> set lines 900
col path for a65
select g.name,d.header_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number(+)=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU STATE PATH
------------------------------ ------------ ----------- -----------------------------------------------------------------
DATA MEMBER MOUNTED /dev/raw/raw4
OCR MEMBER MOUNTED /dev/raw/raw1
OCR MEMBER MOUNTED /dev/raw/raw3CANDIDATE /dev/raw/raw2
尝试将/dev/raw/raw2的磁盘头信息恢复试试，使用下面的方法恢复（一定记得用conv=notrunc）：
dd if=/tmp/raw2.dump of=/dev/raw/raw2 bs=1048576 count=50 conv=notrunc再次查看/dev/raw/raw2对应的header_status变成member了，但是现在它已经不是ocr磁盘组的一部分了
SQL> set lines 900
col path for a65
select g.name,d.header_status,d.mount_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number(+)=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU MOUNT_S STATE PATH
------------------------------ ------------ ------- ----------- -----------------------------------------------------------------
DATA MEMBER CACHED MOUNTED /dev/raw/raw4
OCR MEMBER CACHED MOUNTED /dev/raw/raw1
OCR MEMBER CACHED MOUNTED /dev/raw/raw3MEMBER CLOSED /dev/raw/raw2
尝试添加/dev/raw/raw2到ocr磁盘组：
SQL> alter diskgroup ocr add disk '/dev/raw/raw2';
alter diskgroup ocr add disk '/dev/raw/raw2'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk '/dev/raw/raw2' belongs to diskgroup "OCR"直接报错，因为该磁盘头信息显示该盘属于ocr磁盘组。
解决方案：
1、将/dev/raw/raw2磁盘头信息清除再尝试加入，这个操作要格外小心，不能清除错盘了。
2、使用add disk force命令强制添加，原理和1差不多，但是网上有强制添加导致实例crash的案例。
3、存储再重新划分一块磁盘添加进去，回收之前的磁盘（比较推荐）。
最终解决效果：
最终选择add disk force的方式将/dev/raw/raw2加入到ocr磁盘组，生产环境有条件最好还是选择方案3。
添加完成之后，ocr自动做refresh，将新加入的盘上同步仲裁相关信息：
Wed Jun 11 09:58:50 2025
SQL> alter diskgroup ocr add disk '/dev/raw/raw2' force
NOTE: GroupBlock outside rolling migration privileged region
NOTE: Assigning number (2,1) to disk (/dev/raw/raw2)
NOTE: requesting all-instance membership refresh for group=2
NOTE: initializing header on grp 2 disk OCR_0001
NOTE: requesting all-instance disk validation for group=2
Wed Jun 11 09:58:51 2025
NOTE: skipping rediscovery for group 2/0x8c294763 (OCR) on local instance.
NOTE: requesting all-instance disk validation for group=2
NOTE: skipping rediscovery for group 2/0x8c294763 (OCR) on local instance.
Wed Jun 11 09:58:51 2025
GMON updating for reconfiguration, group 2 at 19 for pid 31, osid 2491
NOTE: group 2 PST updated.
NOTE: initiating PST update: grp = 2
GMON updating group 2 at 20 for pid 31, osid 2491
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group OCR: updated PST location: disk 0001 (PST copy 2)
NOTE: PST update grp = 2 completed successfully
NOTE: membership refresh pending for group 2/0x8c294763 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Failed voting file relocation on diskgroup OCR
GMON querying group 2 at 21 for pid 18, osid 1742
NOTE: cache opening disk 1 of grp 2: OCR_0001 path:/dev/raw/raw2
GMON querying group 2 at 22 for pid 18, osid 1742
SUCCESS: refreshed membership for 2/0x8c294763 (OCR)
NOTE: starting rebalance of group 2/0x8c294763 (OCR) at power 1
SUCCESS: alter diskgroup ocr add disk '/dev/raw/raw2' force
Starting background process ARB0
Wed Jun 11 09:58:54 2025
ARB0 started with pid=28, OS id=4645
NOTE: assigning ARB0 to group 2/0x8c294763 (OCR) with 1 parallel I/O
cellip.ora not found.
NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 1:2 for diskgroup 2 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
Wed Jun 11 09:59:01 2025
NOTE: voting file allocation on grp 2 disk OCR_0001
NOTE: Successful voting file relocation on diskgroup OCR
Wed Jun 11 09:59:06 2025
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=2
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 2/0x8c294763 (OCR)
NOTE: membership refresh pending for group 2/0x8c294763 (OCR)
Wed Jun 11 09:59:12 2025
GMON querying group 2 at 23 for pid 18, osid 1742
Wed Jun 11 09:59:12 2025
SUCCESS: refreshed membership for 2/0x8c294763 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 3 voting file(s).
集群仲裁信息恢复三副本：
