标签归档：RAC

0racle 11.2.0.3 RAC在线迁移OCR&VOTE ASM磁盘组

1条回复

最近一个土豪客户要更换存储（原存储IBM DS8300，新存储EMC VMAX 20K，这都是高端存储），因此涉及到系统和数据库的迁移，这个项目很大，据说三年前就已经开始策划。其中要迁移的核心数据库是一套DB2，数据量达30T+，还有一套11.2.0.2 Oracle RAC，数据量500G+，DB2迁移暂先不说，因为不是专门搞那个的。由于要负责这套Oracle RAC的迁移，采用的迁移方案是，先将OCR&VOTE ASM磁盘组在线从原存储IBM DS8300迁移到新存储EMC VMAX 20K的ASM磁盘组上，然后将集群全部关闭，通过存储间的拷贝方式将原IBM DS8300存储上的DATA和ARCH ASM磁盘组中的数据拷贝到新存储EMC VMAX 20K上，拷贝完成后，扫描磁盘然后启动集群。这里我们不讨论这个迁移方案是否合理，因为这其中夹杂着很多政治因素，我们只关心迁移方案中的第一步，即OCR&VOTE ASM磁盘组在线迁移问题。其实，这里我们并不用非常关心OCR&VOTE ASM磁盘组到底是从IBM DS8300迁移到EMC VMAX 20K上还是从EMC VMAX 20K迁移到IBM DS8300上，因为从本质上讲只是将OCR&VOTE换一个位置而已。下面是我在自己虚拟机上进行的一个测试，特记录一下，留作备用。

一、测试环境

----软件版本
Oracle VirtualBox 4.3 + Oracle Linux 5.8 + UDEV + 11.2.0.3 Grid + 11.2.0.3 DB

----IP地址
# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
# Public IP
192.168.56.111 11grac1.localdomain 11grac1
192.168.56.112 11grac2.localdomain 11grac2
# Private IP
192.168.10.1 11grac1-priv.localdomain 11grac1-priv
192.168.10.2 11grac2-priv.localdomain 11grac2-priv
# Virtual IP
192.168.56.113 11grac1-vip.localdomain 11grac1-vip
192.168.56.114 11grac2-vip.localdomain 11grac2-vip
# SCAN IP
192.168.56.115 SCAN

----共享存储磁盘
# ls -l /dev/sd*
brw-rw---- 1 root root 8, 16 Dec 25  2013 sdb
brw-rw---- 1 root root 8, 32 Dec 25  2013 sdc
brw-rw---- 1 root root 8, 48 Dec 25  2013 sdd

----UDEV绑定ASM磁盘
# ls -l /dev/asm*
brw-rw---- 1 grid asmadmin 8, 16 Dec 25 05:22 /dev/asm-diskb
brw-rw---- 1 grid asmadmin 8, 32 Dec 25 03:45 /dev/asm-diskc
brw-rw---- 1 grid asmadmin 8, 48 Dec 25 05:22 /dev/asm-diskd

----ASM实例
SQL> select instance_name,status from gv$instance;
  
INSTANCE_NAME    STATUS
---------------- ------------
+ASM1            STARTED
+ASM2            STARTED

----ASM实例spfile
SYS@+ASM1> show parameter spfile;
  
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +OCRVOTE/rac11g-cluster/asmpar
                                                 ameterfile/registry.253.835086289
SYS@+ASM2> show parameter spfile;
  
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +OCRVOTE/rac11g-cluster/asmpar
                                                 ameterfile/registry.253.835086289

----ASM DISKGROUP
SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME            STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ --------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA            MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           3 OCRVOTE         MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME            STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ --------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA            MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           3 OCRVOTE         MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0

----ASM DISK
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME            PATH            FAILGROUP
------------ ----------- -------- ------- ---------- ---------- --------------- --------------- ----------------
           0           0 NORMAL   UNKNOWN          0          0                 /dev/asm-diskd
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000       /dev/asm-diskc  DATA_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000    /dev/asm-diskb  OCRVOTE_0000
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;  
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME              PATH            FAILGROUP
------------ ----------- -------- ------- ---------- ---------- --------------- --------------- -----------------
           0           0 NORMAL   UNKNOWN          0          0                 /dev/asm-diskd
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000       /dev/asm-diskc  DATA_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000    /dev/asm-diskb  OCRVOTE_0000

----DB实例
SQL> select instance_name,status from gv$instance;
  
INSTANCE_NAME    STATUS
---------------- ------------
RAC11g2          OPEN
RAC11g1          OPEN

----DB实例spfile
SQL> show parameter spfile;
  
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/rac11g/spfilerac11g.ora

---OCR&VOTE信息
[root@11grac1 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2960
         Available space (kbytes) :     259160
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5cc943d1e0544f49bf7a1f698de670c9 (/dev/asm-diskb) [OCRVOTE]
Located 1 voting disk(s).
[root@11grac2 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2960
         Available space (kbytes) :     259160
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5cc943d1e0544f49bf7a1f698de670c9 (/dev/asm-diskb) [OCRVOTE]
Located 1 voting disk(s).

二、测试目标
目前该RAC的OCR&VOTE存放在OCRVOTE ASM磁盘组，准备在线迁移到OCRVOTE_NEW ASM磁盘组上。 三、详细过程

1.查看OCR和VOTE状态与位置（在两个节点root执行）
[root@11grac1 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2960
         Available space (kbytes) :     259160
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5cc943d1e0544f49bf7a1f698de670c9 (/dev/asm-diskb) [OCRVOTE]
[root@11grac2 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2960
         Available space (kbytes) :     259160
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5cc943d1e0544f49bf7a1f698de670c9 (/dev/asm-diskb) [OCRVOTE]
Located 1 voting disk(s).

2.在两节点查看OCR备份情况（在两节点root执行）
[root@11grac1 ~]# ocrconfig -showbackup 
11grac2     2013/12/25 07:46:20     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup00.ocr 
11grac2     2013/12/25 07:46:20     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/day.ocr 
11grac2     2013/12/25 07:46:20     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/week.ocr 
11grac2     2013/12/24 20:54:01     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_205401.ocr 
11grac2     2013/12/24 20:53:22     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_205322.ocr 
11grac1     2013/12/24 20:21:31     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_202131.ocr 
11grac1     2013/12/24 20:16:55     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_201655.ocr 
11grac1     2013/12/24 18:09:31     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_180931.ocr
[root@11grac2 ~]# ocrconfig -showbackup  
11grac2     2013/12/25 07:46:20     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup00.ocr  
11grac2     2013/12/25 07:46:20     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/day.ocr  
11grac2     2013/12/25 07:46:20     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/week.ocr 
11grac2     2013/12/24 20:54:01     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_205401.ocr  
11grac2     2013/12/24 20:53:22     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_205322.ocr  
11grac1     2013/12/24 20:21:31     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_202131.ocr  
11grac1     2013/12/24 20:16:55     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_201655.ocr  
11grac1     2013/12/24 18:09:31     /grid/product/11.2.0/grid_1/cdata/rac11g-cluster/backup_20131224_180931.ocr

3.手动执行物理备份---可选操作（在一个节点或两个节点root执行）
# ocrconfig -manualbackup

4.再次查看OCR备份情况---可选操作（在两个节点root执行）
# ocrconfig -showbackup

5.创建将要迁移OCR&VOTE到新的ASM磁盘组OCRVOTE_NEW（在一个节点grid用户登录ASM实例执行）
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME            PATH            FAILGROUP
------------ ----------- -------- ------- ---------- ---------- --------------- --------------- ----------------
           0           0 NORMAL   UNKNOWN          0          0                 /dev/asm-diskd
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000       /dev/asm-diskc  DATA_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000    /dev/asm-diskb  OCRVOTE_0000
  
SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME            STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ --------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA            MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           3 OCRVOTE         MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME              PATH            FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ----------------- --------------- -----------------
           0           0 NORMAL   UNKNOWN          0          0                   /dev/asm-diskd
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000         /dev/asm-diskc  DATA_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000      /dev/asm-diskb  OCRVOTE_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME              STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ----------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA              MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           3 OCRVOTE           MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
  
SYS@+ASM1> create diskgroup OCRVOTE_NEW EXTERNAL REDUNDANCY DISK '/dev/asm-diskd' attribute 'compatible.asm'='11.2';
  
Diskgroup created.
----创建OCRVOTE_NEW ASM磁盘组的节点ASM实例会自动mount该磁盘组，但是另一个节点不会mount上该磁盘组

6.查看新创建的ASM磁盘组（在两个节点grid用户登录ASM实例执行）
SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME            STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ --------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA            MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           3 OCRVOTE         MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
           2 OCRVOTE_NEW     MOUNTED     EXTERN       2048       1996 N 11.2.0.0.0
  
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME               PATH            FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ------------------ --------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000          /dev/asm-diskc  DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1996 OCRVOTE_NEW_0000   /dev/asm-diskd  OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000       /dev/asm-diskb  OCRVOTE_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME              STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ----------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA              MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           0 OCRVOTE_NEW       DISMOUNTED                  0          0 N 0.0.0.0.0
           3 OCRVOTE           MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME              PATH            FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ----------------- --------------- -----------------
           0           0 NORMAL   UNKNOWN          0          0                   /dev/asm-diskd
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000         /dev/asm-diskc  DATA_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000      /dev/asm-diskb  OCRVOTE_0000
----节点2 ASM实例没有mount上OCRVOTE_NEW磁盘组

7.挂载新创建的ASM磁盘组（在在没有mount上新创建的ASM磁盘组节点上以grid用户登录ASM实例执行）
SYS@+ASM2> alter diskgroup OCRVOTE_NEW mount;
  
Diskgroup altered.
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME              PATH            FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ----------------- --------------- -----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000         /dev/asm-diskc  DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1953 OCRVOTE_NEW_0000  /dev/asm-diskd  OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000      /dev/asm-diskb  OCRVOTE_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME              STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ----------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA              MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW       MOUNTED     EXTERN       2048       1953 N 11.2.0.0.0
           3 OCRVOTE           MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0

8.将新创建的ASM磁盘组OCRVOTE_NEW添加到OCR中（在一个节点root执行）
[root@11grac1 ~]# ocrconfig -add +OCRVOTE_NEW

9.查看OCR和VOTE状态与位置（在两个节点root执行）
[root@11grac1 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5cc943d1e0544f49bf7a1f698de670c9 (/dev/asm-diskb) [OCRVOTE]
Located 1 voting disk(s).
[root@11grac2 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   5cc943d1e0544f49bf7a1f698de670c9 (/dev/asm-diskb) [OCRVOTE]
Located 1 voting disk(s).

SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1685 N 11.2.0.0.0
           3 OCRVOTE          MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
  
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1685 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000     /dev/asm-diskb   OCRVOTE_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1685 N 11.2.0.0.0
           3 OCRVOTE          MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1685 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_0000     /dev/asm-diskb   OCRVOTE_0000
----此时会看见OCR的位置出现在两处，其中+OCRVOTE_NEW为OCR的mirror

10.迁移VOTE DISK到新创建的ASM磁盘组OCRVOTE_NEW上（在一个节点root执行）
[root@11grac1 ~]# crsctl replace votedisk +OCRVOTE_NEW
Successful addition of voting disk 1bd1f2b28abc4ff6bf1a4113f2ce1067.
Successful deletion of voting disk 5cc943d1e0544f49bf7a1f698de670c9.
Successfully replaced voting disk group with +OCRVOTE_NEW.
CRS-4266: Voting file(s) successfully replaced

11.查看此时OCR&VOTE ASM磁盘组状态（在两个节点grid用户登录ASM实例执行）
SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME               STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ------------------ ----------- ------ ---------- ---------- - ---------------
           1 DATA               MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           3 OCRVOTE            MOUNTED     EXTERN       2048       1684 N 11.2.0.0.0
           2 OCRVOTE_NEW        MOUNTED     EXTERN       2048       1653 Y 11.2.0.0.0
----可以看见执行crsctl replace votedisk +OCRVOTE_NEW操作的节点上v$asm_diskgroup中已显示将voting file存放在新建的ASM磁盘组OCRVOTE_NEW上
----这里其实存在疑问，我测试过几次，得出的结果不同，1节点执行完crsctl replace votedisk +OCRVOTE_NEW操作后，v$asm_diskgroup中也会显示voting file同时存放在ASM磁盘组OCRVOTE和OCRVOTE_NEW上
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME              STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ----------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA              MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW       MOUNTED     EXTERN       2048       1653 N 11.2.0.0.0
           3 OCRVOTE           MOUNTED     EXTERN       2048       1684 Y 11.2.0.0.0
----未执行crsctl replace votedisk +OCRVOTE_NEW操作的节点上v$asm_diskgroup中显示voting file仍然存放在ASM磁盘组上
----这里其实也存在疑问，我测试过几次，得出的结果不同，未执行crsctl replace votedisk +OCRVOTE_NEW操作的节点上v$asm_diskgroup中显示voting file既没有存放在OCRVOTE上，也没有存放在OCRVOTE_NEW上

12.查看OCR和VOTE状态与位置（在两个节点root执行）
[root@11grac1 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1bd1f2b28abc4ff6bf1a4113f2ce1067 (/dev/asm-diskd) [OCRVOTE_NEW]
[root@11grac2 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1bd1f2b28abc4ff6bf1a4113f2ce1067 (/dev/asm-diskd) [OCRVOTE_NEW]
Located 1 voting disk(s).

13.创建ASM实例spfile到新创建的OCRVOTE_NEW ASM磁盘组上（在一个节点grid用户登录ASM实例执行）
SYS@+ASM1> create pfile='/home/grid/asmpfile.ora' from spfile;
  
File created.
  
SYS@+ASM1> create spfile='+OCRVOTE_NEW' from pfile='/home/grid/asmpfile.ora';
  
File created.

14.删除OCR原位置ASM磁盘组OCRVOTE（在一个节点root执行）
[root@11grac1 ~]# ocrconfig -delete +OCRVOTE

15.查看OCR和VOTE新状态与位置（在两个节点root执行）
[root@11grac1 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1bd1f2b28abc4ff6bf1a4113f2ce1067 (/dev/asm-diskd) [OCRVOTE_NEW]
Located 1 voting disk(s).
[root@11grac2 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1bd1f2b28abc4ff6bf1a4113f2ce1067 (/dev/asm-diskd) [OCRVOTE_NEW]
Located 1 voting disk(s).

SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
           3 OCRVOTE          MOUNTED     EXTERN       2048       1684 N 11.2.0.0.0
  
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1684 OCRVOTE_0000     /dev/asm-diskb   OCRVOTE_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 N 11.2.0.0.0
           3 OCRVOTE          MOUNTED     EXTERN       2048       1684 Y 11.2.0.0.0
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1684 OCRVOTE_0000     /dev/asm-diskb   OCRVOTE_0000

16.重启集群（在两个节点root执行）
[root@11grac1 ~]# crsctl stop crs && crsctl start crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '11grac1'
CRS-2673: Attempting to stop 'ora.crsd' on '11grac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '11grac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on '11grac1'
CRS-2673: Attempting to stop 'ora.cvu' on '11grac1'
CRS-2673: Attempting to stop 'ora.oc4j' on '11grac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '11grac1'
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on '11grac1'
CRS-2673: Attempting to stop 'ora.OCRVOTE_NEW.dg' on '11grac1'
CRS-2673: Attempting to stop 'ora.rac11g.db' on '11grac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on '11grac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.11grac1.vip' on '11grac1'
CRS-2677: Stop of 'ora.scan1.vip' on '11grac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on '11grac2'
CRS-2677: Stop of 'ora.11grac1.vip' on '11grac1' succeeded
CRS-2672: Attempting to start 'ora.11grac1.vip' on '11grac2'
CRS-2676: Start of 'ora.scan1.vip' on '11grac2' succeeded
CRS-2676: Start of 'ora.11grac1.vip' on '11grac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on '11grac2'
CRS-2677: Stop of 'ora.cvu' on '11grac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on '11grac2' succeeded
CRS-2677: Stop of 'ora.rac11g.db' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on '11grac1'
CRS-2677: Stop of 'ora.DATA.dg' on '11grac1' succeeded
CRS-2677: Stop of 'ora.oc4j' on '11grac1' succeeded
CRS-2677: Stop of 'ora.OCRVOTE.dg' on '11grac1' succeeded
CRS-2677: Stop of 'ora.OCRVOTE_NEW.dg' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '11grac1'
CRS-2677: Stop of 'ora.asm' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '11grac1'
CRS-2677: Stop of 'ora.ons' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '11grac1'
CRS-2677: Stop of 'ora.net1.network' on '11grac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '11grac1' has completed
CRS-2677: Stop of 'ora.crsd' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on '11grac1'
CRS-2673: Attempting to stop 'ora.ctssd' on '11grac1'
CRS-2673: Attempting to stop 'ora.evmd' on '11grac1'
CRS-2673: Attempting to stop 'ora.asm' on '11grac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on '11grac1'
CRS-2677: Stop of 'ora.crf' on '11grac1' succeeded
CRS-2677: Stop of 'ora.evmd' on '11grac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on '11grac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on '11grac1' succeeded
CRS-2677: Stop of 'ora.asm' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '11grac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '11grac1'
CRS-2677: Stop of 'ora.cssd' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on '11grac1'
CRS-2677: Stop of 'ora.gipcd' on '11grac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on '11grac1'
CRS-2677: Stop of 'ora.gpnpd' on '11grac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '11grac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
[root@11grac2 ~]# crsctl stop crs && crsctl start crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '11grac2'
CRS-2673: Attempting to stop 'ora.crsd' on '11grac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '11grac2'
CRS-2673: Attempting to stop 'ora.11grac2.vip' on '11grac2'
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on '11grac2'
CRS-2673: Attempting to stop 'ora.OCRVOTE_NEW.dg' on '11grac2'
CRS-2673: Attempting to stop 'ora.rac11g.db' on '11grac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '11grac2'
CRS-2677: Stop of 'ora.11grac2.vip' on '11grac2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.11grac1.vip' on '11grac2'
CRS-2677: Stop of 'ora.11grac1.vip' on '11grac2' succeeded
CRS-2677: Stop of 'ora.rac11g.db' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on '11grac2'
CRS-2677: Stop of 'ora.DATA.dg' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on '11grac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on '11grac2'
CRS-2677: Stop of 'ora.scan1.vip' on '11grac2' succeeded
CRS-2677: Stop of 'ora.OCRVOTE.dg' on '11grac2' succeeded
CRS-2677: Stop of 'ora.OCRVOTE_NEW.dg' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on '11grac2'
CRS-2677: Stop of 'ora.asm' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on '11grac2'
CRS-2677: Stop of 'ora.ons' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '11grac2'
CRS-2677: Stop of 'ora.net1.network' on '11grac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '11grac2' has completed
CRS-2677: Stop of 'ora.crsd' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on '11grac2'
CRS-2673: Attempting to stop 'ora.ctssd' on '11grac2'
CRS-2673: Attempting to stop 'ora.evmd' on '11grac2'
CRS-2673: Attempting to stop 'ora.asm' on '11grac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on '11grac2'
CRS-2677: Stop of 'ora.crf' on '11grac2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on '11grac2' succeeded
CRS-2677: Stop of 'ora.evmd' on '11grac2' succeeded
CRS-2677: Stop of 'ora.ctssd' on '11grac2' succeeded
CRS-2677: Stop of 'ora.asm' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '11grac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on '11grac2'
CRS-2677: Stop of 'ora.cssd' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on '11grac2'
CRS-2677: Stop of 'ora.gipcd' on '11grac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on '11grac2'
CRS-2677: Stop of 'ora.gpnpd' on '11grac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '11grac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

17.查看OCR&VOTE位置及ASM实例spfile位置（在两个节点grid用户登录ASM实例执行）
[root@11grac1 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   943d41e7dd624fd9bf8becbf18b434f5 (/dev/asm-diskd) [OCRVOTE_NEW]
Located 1 voting disk(s).
[root@11grac2 ~]# ocrcheck && crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2976
         Available space (kbytes) :     259144
         ID                       :  331841301
         Device/File Name         : +OCRVOTE_NEW
                                    Device/File integrity check succeeded
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
                                    Device/File not configured
  
         Cluster registry integrity check succeeded
  
         Logical corruption check succeeded
  
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   943d41e7dd624fd9bf8becbf18b434f5 (/dev/asm-diskd) [OCRVOTE_NEW]
Located 1 voting disk(s).

SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
           3 OCRVOTE          MOUNTED     EXTERN       2048       1684 N 11.2.0.0.0
  
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1684 OCRVOTE_0000     /dev/asm-diskb   OCRVOTE_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
           3 OCRVOTE          MOUNTED     EXTERN       2048       1684 N 11.2.0.0.0
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1684 OCRVOTE_0000     /dev/asm-diskb   OCRVOTE_0000           
  
SYS@+ASM1> show parameter spfile;
  
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +OCRVOTE_NEW/rac11g-cluster/as
                                                 mparameterfile/registry.253.83
                                                 5094929
  
SYS@+ASM2> show parameter spfile;
  
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +OCRVOTE_NEW/rac11g-cluster/as
                                                 mparameterfile/registry.253.83
                                                 5094929

18.将OCR原位置ASM磁盘组OCRVOTE卸载掉（在一个节点grid用户登录ASM实例执行）
SYS@+ASM1> alter diskgroup OCRVOTE dismount;
  
Diskgroup altered.
  
SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
           0 OCRVOTE          DISMOUNTED                  0          0 N 0.0.0.0.0
  
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           0           0 NORMAL   UNKNOWN          0          0                  /dev/asm-diskb
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
           3 OCRVOTE          MOUNTED     EXTERN       2048       1684 N 11.2.0.0.0
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
           3           0 NORMAL   UNKNOWN       2048       1684 OCRVOTE_0000     /dev/asm-diskb   OCRVOTE_0000

19.删除OCR原位置ASM磁盘组OCRVOTE（在另一个节点grid用户登录ASM实例执行）
SYS@+ASM2> drop diskgroup OCRVOTE including contents;
  
Diskgroup dropped.
  
SYS@+ASM1> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
  
SYS@+ASM1> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           0           0 NORMAL   UNKNOWN          0          0                  /dev/asm-diskb
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000
  
SYS@+ASM2> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
  
GROUP_NUMBER NAME             STATE       TYPE     TOTAL_MB    FREE_MB V COMPATIBILITY
------------ ---------------- ----------- ------ ---------- ---------- - ---------------
           1 DATA             MOUNTED     EXTERN       5120       2792 N 11.2.0.0.0
           2 OCRVOTE_NEW      MOUNTED     EXTERN       2048       1652 Y 11.2.0.0.0
  
SYS@+ASM2> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
  
GROUP_NUMBER DISK_NUMBER STATE    REDUNDA   TOTAL_MB    FREE_MB NAME             PATH             FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ---------------- ---------------- ----------------
           0           0 NORMAL   UNKNOWN          0          0                  /dev/asm-diskb
           1           0 NORMAL   UNKNOWN       5120       2792 DATA_0000        /dev/asm-diskc   DATA_0000
           2           0 NORMAL   UNKNOWN       2048       1652 OCRVOTE_NEW_0000 /dev/asm-diskd   OCRVOTE_NEW_0000 
----这里不执行19步无法删除该磁盘组

20.整个迁移过程结束

以上为Oracle 11gR2在线迁移OCR&VOTE ASM磁盘组过程，其中不涉及存储中所使用多路径软件问题，如果涉及到存储磁盘路径改变，则需要去修改ASM实例的disk_strings参数，具体操作可以参考Maclea的这篇博客在11gR2 RAC中修改ASM DISK Path磁盘路径，论坛贴链接[讨论]11G RAC更换存储，如有问题，欢迎讨论。

Oracle RAC导致实例逐出的五大问题

这篇文档在MOS上看到被翻译成了中文，觉得比较实用，可适用于10.2.0.1至11.2.0.3版本的RAC，因此贴出来方便别人查阅，后期会尽量多贴一些MOS上的中文技术资料，希望可以帮到别人，同时也算是自己学习积累的过程。

问题 1：警报日志显示 ora-29740 是实例崩溃/驱逐的原因
症状：
实例崩溃，警报日志显示“ORA-29740:evicted by member …（被成员…驱逐）”错误。
可能的原因：
一个实例将另一个实例从 RAC 数据库驱逐时，出现了 ORA-29740 错误。被驱逐的实例会在警报日志中报告 ora-29740 错误。
此问题的部分原因是集群中的通信错误、向控制文件发送“心跳”失败以及其它原因。
检查所有实例的 lmon 跟踪文件，这对确定实例驱逐的原因代码而言非常重要。查找包含“kjxgrrcfgchk:Initiating reconfig”的行。
这将提供一个原因代码，如“kjxgrrcfgchk:Initiating reconfig, reason 3”。实例驱逐时发生的大多数 ora-29740 错误是由于原因 3（“通信故障”） 造成的。
Document 219361.1 (Troubleshooting ORA-29740 in a RAC Environment) 介绍了以下几种可能造成原因 3的 ora-29740 错误原因：
a) 网络问题。
b) 资源耗尽（CPU、I/O 等）
c) 严重的数据库争用。
d) Oracle bug。
解决方案：
1) 检查网络，确保无网络错误，如 UDP 错误或 IP 数据包丢失或故障错误。
2) 检查网络配置，确保所有节点上的所有网络配置均设置正确。
   例如，所有节点上 MTU 的大小必须相同，并且如果使用巨帧，交换机也能够支持大小为 9000 的 MTU。
3) 检查服务器是否存在 CPU 负载问题或可用内存不足。
4) 检查数据库在实例驱逐之前是否正处于挂起状态或存在严重的性能问题。
5) 检查 CHM (Cluster Health Monitor) 输出，以查看服务器是否存在 CPU 或内存负载问题、网络问题或者 lmd 或 lms 进程出现死循环。CHM 输出只能在特定平台和版本中使用，因此请参阅 CHM 常见问题 Document 1328466.1
6) 如果 OSWatcher 尚未设置，请按照 Document 301137.1 中的说明进行设置以运行 OSWatcher。
CHM 输出不可用时，使用 OSWatcher 输出将有所帮助。

问题 2：警报日志在实例崩溃或驱逐前显示“ipc send timeout”错误
症状：
实例驱逐时，警报日志显示许多“IPC send timeout”错误。此消息通常伴随数据库性能问题。
可能的原因：
在 RAC 中，数据库进程，例如 lmon、lmd 和 lms 会不断地和其他实例的进程通信。lmd0 进程负责管理 enqueue，而 lms 进程负责管理数据块资源并传输数据块以支持 Cache Fusion。如果这些进程中的一个或多个受阻、死循环或异常繁忙，则可能导致“IPC send timeout（IPC 发送超时）”错误。
lmon、lms 和 lmd 进程报告“IPC send timeout”错误的另一个原因是网络问题或服务器资源（CPU 和内存）问题。这些进程可能无法获得 CPU 运行调度或这些进程发送的网络数据包丢失。
涉及 lmon、lmd 和 lms 进程的通信问题导致实例驱逐。被驱逐实例的警报日志显示的信息类似于如下示例
IPC Send timeout detected.Sender: ospid 1519
Receiver: inst 8 binc 997466802 ospid 23309
如果某实例被驱逐，警报日志中的“IPC Send timeout detected（检测到 IPC 发送超时）”通常伴随着其它问题，如 ora-29740 和“Waiting for clusterware split-brain resolution（等待集群件“脑裂”解决方案）”
解决方案：
此处的解决方案与问题 1 相似。
1) 检查网络，确保无网络错误，如 UDP 错误或 IP 数据包丢失或故障错误。
2) 检查网络配置，确保所有节点上的所有网络配置均设置正确。
   例如，所有节点上 MTU 的大小必须相同，并且如果使用巨帧，交换机也能够支持大小为 9000 的 MTU。
3) 检查服务器是否存在 CPU 负载问题或可用内存不足。
4) 检查数据库在实例驱逐之前是否正处于挂起状态或存在严重的性能问题。
5) 检查 CHM (Cluster Health Monitor) 输出，以查看服务器是否存在 CPU 或内存负载问题、网络问题或者 lmd 或 lms 进程出现死循环。CHM 输出只能在特定平台和版本中使用，因此请参阅 CHM 常见问题 Document 1328466.1
6) 如果 OSWatcher 尚未设置，请按照 Document 301137.1 中的说明进行设置以运行 OSWatcher。
CHM 输出不可用时，使用 OSWatcher 输出将有所帮助。

问题 3：在实例崩溃或驱逐前，问题实例处于挂起状态
症状：
在实例崩溃/驱逐前，该实例或数据库正处于挂起状态。当然，也可能是节点挂起。
可能的原因：
由于 lmon、lmd 和 lms 等不同进程与其它实例上对应的进程通信，因此当实例和数据库挂起时，这些进程可能正在等待某个资源，如 latch、enqueue 或数据块。这些等待中的进程得不到网络响应，或无法通过网络向远程实例发送任何通信。因此，其它实例将驱逐问题实例。
在执行驱逐其他实例动作的实例警报日志中，您可能会看到与以下消息类似的消息：
Remote instance kill is issued [112:1]:8
或者
Evicting instance 2 from cluster
解决方案：
1) 查找数据库或实例挂起的原因。对数据库或实例挂起问题进行故障排除时，获取全局 systemstate 转储和全局hang analyze 转储是关键。如果无法获取全局 systemstate 转储，则应获取在大致相同时间所有实例的本地 systemstate 转储。
2) 检查 CHM (Cluster Health Monitor) 输出，以查看服务器是否存在 CPU 或内存负载问题、网络问题或者 lmd 或 lms 进程出现死循环。CHM 输出只能在某些平台和版本中使用，因此请参阅 CHM 常见问题 Document 1328466.1
3) 如果 OSWatcher 尚未设置，请按照 Document 301137.1 中的说明进行设置以运行 OSWatcher。
CHM 输出不可用时，使用 OSWatcher 输出将有所帮助。

问题 4：在一个或多个实例崩溃或驱逐前，警报日志显示“Waiting for clusterware split-brain resolution（等待集群“脑裂”解决方案）”
症状：
在一个或多个实例崩溃之前，警报日志显示“Waiting for clusterware split-brain resolution（等待集群件“脑裂”解决方案）”。这通常伴随着“Evicting instance n from cluster（从集群驱逐实例 n）”，其中 n 是指被驱逐的实例编号。
可能的原因：
lmon 进程向远程实例发送一个网络 ping，如果远程实例上的 lmon 进程不响应，则出现实例级别的“脑裂”。因此，查找 lmon 不能相互通信的原因对解决此问题而言非常重要。
常见原因有：
1) 实例级别的“脑裂”通常由网络问题导致，因此检查网络设置和连接非常重要。但是，因为如果网络已关闭，集群件 (CRS) 就会出现故障，所以只要 CRS 和数据库使用同一网络，则网络不太可能会关闭。  
2) 服务器非常繁忙和/或可用内存量低（频繁的交换和内存扫描），将阻止 lmon 进程被调度。
3) 数据库或实例正处于挂起状态，并且 lmon 进程受阻。
4) Oracle bug
以上原因与问题 1的原因相似（警报日志显示 ora-29740 是实例崩溃/驱逐的原因）。
解决方案：
此处的解决方案与问题 1 相似。
1) 检查网络，确保无网络错误，如 UDP 错误或 IP 数据包丢失或故障错误。
2) 检查网络配置，确保所有节点上的所有网络配置均设置正确。
   例如，所有节点上 MTU 的大小必须相同，并且如果使用巨帧，交换机也能够支持大小为 9000 的 MTU。
3) 检查服务器是否存在 CPU 负载问题或可用内存不足。
4) 检查数据库在实例驱逐之前是否正处于挂起状态或存在严重的性能问题。
5) 检查 CHM (Cluster Health Monitor) 输出，以查看服务器是否存在 CPU 或内存负载问题、网络问题或者 lmd 或 lms 进程出现死循环。CHM 输出只能在特定平台和版本中使用，因此请参阅 CHM 常见问题 Document 1328466.1
6) 如果 OSWatcher 尚未设置，请按照 Document 301137.1 中的说明进行设置以运行 OSWatcher。
CHM 输出不可用时，使用 OSWatcher 输出将有所帮助。

问题 5：另一个实例尝试驱逐问题实例，但由于一些原因未能成功驱逐，最终CRS会终止该问题实例。
症状：
一个实例驱逐其他实例时，在问题实例自己关闭之前，所有实例都处于等待状态，但是如果问题实例因为某些原因不能终止自己，发起驱逐的实例将发出 Member Kill 请求。Member Kill 请求会要求 CRS 终止问题实例。此功能适用于 11.1 及更高版本。
可能的原因：
要求 CRS 终止问题实例的实例警报日志显示
Remote instance kill is issued [112:1]:8
例如，以上消息表示终止实例 8 的 Member Kill 请求已发送至 CRS。
问题实例由于某种原因正处于挂起状态且无响应。这可能是由于节点存在 CPU 和内存问题，并且问题实例的进程无法获得 CPU 运行调度。
第二个常见原因是数据库资源争用严重，导致问题实例无法完成远程实例驱逐该实例的请求。
另一个原因可能是由于实例尝试中止自己时，一个或多个进程“幸存”了下来。除非实例的所有进程全部终止，否则 CRS 不认为该实例已终止，而且不会通知其它实例该问题实例已经被终止。这种情况下的一个常见问题是一个或多个进程变成僵尸进程且未终止。
并导致CRS通过节点重启或 rebootless restart（ CRS 重新启动但节点不重启）进行重新启动。这种情况下，问题实例的警报日志显示
Instance termination failed to kill one or more processes
Instance terminated by LMON, pid = 23305
（实例终止未能终止一个或多个进程
实例被 LMON, pid = 23305 终止）
解决方案：
此问题的解决方案与问题 3 相似
1) 查找数据库或实例挂起的原因。对数据库或实例挂起问题进行故障排除时，获取全局 systemstate 转储和全局hang analyze 转储是关键。如果无法获取全局 systemstate 转储，则应获取在大致相同时间所有实例的本地 systemstate 转储。
2) 检查 CHM (Cluster Health Monitor) 输出，以查看服务器是否存在 CPU 或内存负载问题、网络问题或者 lmd 或 lms 进程出现死循环。CHM 输出只能在某些平台和版本中使用，因此请参阅 CHM 常见问题 Document 1328466.1
3) 如果 OSWatcher 尚未设置，请按照 Document 301137.1 中的说明进行设置以运行 OSWatcher。
CHM 输出不可用时，使用 OSWatcher 输出将有所帮助.