分类目录归档:Oracle

闰秒对Oracle RAC的影响

1.什么是闰秒
闰秒是在协调世界时(UTC中增加或减少一秒,使它与平太阳时贴近所做调整。UTC,是透过广播作为民用时的官方时间基础,它使用非常精确的原子钟来维护。要保持UTC与平太阳时的一致性,偶尔需要调整,也就是“跳个”1秒来做调整,就是所谓添加闰秒。闰秒时间现在是由国际地球自转和参考作坐标系统服务(IERS)来确认,而在1988年1月1日之前是由国际时间局(BIH)承担这项职责。
当要增加正闰秒时,这一秒是增加在第二天的00:00:00之前,效果是延缓UTC第二天的开始。当天23:59:59的下一秒被记为23:59:60,然后才是第二天的00:00:00。如果是负闰秒的话,23:59:58的下一秒就是第二天的00:00:00了,但目前还没有负闰秒调整的需求。需要时的日长度必须低于1750-1892年的平均日长度,才会累积足够调整1秒所需要的时间。除了每天4毫秒的波动外,日长度自1700年以来都保持一样[1]。然而,从历史上的日食观测则显示,自公元前700年以来,每个世纪的日长度大约增加1.7毫秒[2]。(来自维基百科)

2.闰秒的出现时间

3.闰秒的影响
从当前所了解的参考资料信息来看,闰秒主要会对Linux和Solaris平台系统存在影响,其主要影响是因为使用NTP的缘故,可能会导致OS CPU使用率100%。对于Oracle Database影响,则主要可能会导致RAC节点重启,EM OMS或Agent所导致的OS CPU使用100%,以及因为执行与时间戳数据类型相关的SQL而导致的ORA-01852错误。
3.1对于Linux影响
内核版本大于2.6.22的都受影响,详见:
http://www-01.ibm.com/support/docview.wss?uid=swg21602521
主要是可能出现CPU使用100%的情况,详见Leap Second Hang – CPU Can Be Seen at 100% (文档 ID 1472421.1),根据该文档描述,主要影响是Linux 4.4 – Linux 6.2的中间版本。该情况所涉及的OS环境版本范围:
Oracle VM – Version 2.1.1 to 3.1.1 [Release OVM211 to OVM31]
Linux OS – Version Oracle Linux 4.4 to Oracle Linux 6.2 with Unbreakable Enterprise Kernel [2.6.39] [Release OL4U4 to OL6U2]
Information in this document applies to any platform.
当ntpd调用adjtimex(2)告知kernel插入a leap second所导致livelock而引发的bug
 解决办法:
# /etc/init.d/ntpd stop
#  date -s “`date`”    (reset the system clock)
# /etc/init.d/ntpd start
Or Reboot the Server.
3.2对于Solaris影响
影响Solaris 8、9、10等几个版本,最新的Solaris 11不受影响,解决办法与3.1类似,重启NTP或系统。
3.3闰秒对Oracle的影响
1)RAC集群节点重启
该情况所涉及的Oracle环境版本范围:
Oracle Server – Enterprise Edition – Version 10.1.0.2 to 11.1.0.7 [Release 10.1 to 11.1]
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
Sun Solaris SPARC (64-bit)Sun Solaris x86-64 (64-bit)
Oracle Clusterware and patchsets 10.2.0.1 – 11.1.0.7
Sun Solaris 5.8 – 5.10 adjusting time through NTP daemon (xntpd)
 解决办法:
(1)对于通用平台OS下NTP都应该使用NTP daemon with -x 参数选项,对于Solaris还要进行如下操作来避免集群node reboot:
Configure xntpd Solaris daemon running on the Oracle Clusterware cluster node to 
disable PLL mode and enable slewing

adding to the /etc/inet/ntp.conf file the two lines:  
                   slewalways yes  
                   disable pll  
To restart xntpd the commands are:  
Solaris 10:      svcadm restart ntp  
Solaris 8 and 9: /etc/init.d/xntpd stop ; /etc/init.d/xntpd start
这里还需要注意一点(针对Sun Solaris),如果无法保证NTP server和client时间同步,在重启集群或系统之前一定要先修改ntp.conf文件重启xntpd服务。
2)打Oracle Clusterware patch bundles or a recent MLR (i.e. MLR # 9 or higher) in order to resolve bugs 5015469 and 6022204补丁。针对Sun CR#6595936问题,可以修改cluster oprocd daemon进程的timeout margin>5s解决。
(2)ORA-01852错误
由于闰秒事件导致一分钟>60秒,此时如果恰好在数据库中执行插入带有时间戳的SQL语句时会报ORA-01852错误。
该情况所涉及的Oracle Database环境版本范围:
Oracle Database – Enterprise Edition – Version 8.1.7.4 and later
Oracle Database – Standard Edition – Version 8.1.7.4 and later
Information in this document applies to any platform.
出现该报错的原因是因为Oracle数据库无法存储大于59秒的日期时间戳数据
 解决办法:
创建varchar2数据类型表去存储闰秒的时间戳记录:

SQL> select * from v$version;
 
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production
 
SQL> create table t_630(a number,b timestamp(8));
 
Table created.
 
SQL> insert into t_630 values(1,to_timestamp('2015-06-10 12:10:10.1','yyyy-mm-dd hh24:mi:ss.ff')); 
 
1 row created.
 
         A B
---------- ---------------------------------------------------------------------------
         1 10-JUN-15 12.10.10.10000000 PM
 
SQL> insert into t_630 values(2,to_timestamp('2015-06-30 23:59:60','yyyy-mm-dd hh24:mi:ss'));
insert into t_630 values(2,to_timestamp('2015-06-30 23:59:60','yyyy-mm-dd hh24:mi:ss'))
                                        *
ERROR at line 1:
ORA-01852: seconds must be between 0 and 59
对于此问题,Oracle建议使用varchar2数据类型来替代timestamp
SQL> drop table t_630 purge;
 
Table dropped.
 
SQL> create table t_630(a number,b varchar2(30));
 
Table created.
 
SQL> insert into t_630 values(1,'2015-06-10 12:10:10');
 
1 row created.
 
SQL> insert into t_630 values(2,'2015-06-30 23:59:60');
 
1 row created.
 
SQL> commit;
 
Commit complete.
 
SQL> select * from t_630;
 
         A B
---------- ------------------------------
         1 2015-06-10 12:10:10
         2 2015-06-30 23:59:60

3)Oracle EM agent或OMS过度Linux OS CPU
该情况所涉及的Oracle EM环境版本范围:
Enterprise Manager Base Platform – Version 10.2.0.5 to 10.2.0.5 [Release 10.2]
Enterprise Manager Base Platform – Version 11.1.0.1 to 11.1.0.1 [Release 11.1]
Enterprise Manager Base Platform – Version 12.1.0.1.0 and later
Linux x86
Linux x86-64
该情况在系统层面表现可能如下:
Check top on the system:
$ top – 09:38:24 up 354 days,  5:48,  4 users,  load average: 6.49, 6.34, 6.44
Tasks: 296 total,   4 running, 292 sleeping,   0 stopped,   0 zombie
Cpu(s): 97.2%us,  1.8%sy,  0.0%ni,  0.7%id,  0.1%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:     15991M total,    15937M used,       53M free,      107M buffers
Swap:     8110M total,       72M used,     8038M free,    13614M cached
PID USER      PR    NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
22564 oracle    16   0 1400m 421m 109m S  353  2.6   2225:11 java      <<<<<<< Note: Java PID consuming 353%  CPU
7294 oracle     17   0 3096m 108m 104m S   22  0.7   0:02.61 oracle
Check for  the following message from dmesg:
$ dmesg | grep -i leap
[10703552.860274] Clock: inserting leap second 23:59:60 UTC
 解决办法:
该情况的解决办法与3.1类似,重启NTP或是系统。

4.总结
除了按照上述Oracle建义来应对闰秒事件所造成的结果,同时为了尽可能避免闰秒事件可能导致的影响,最好是在闰秒日调整之前半小时,先将数据库环境下NTP Server和Client时间同步,然后停止服务器NTP,当闰秒调整结束后再开启服务器NTP继续同步。

5.参考文档
Leap seconds (extra second in a year) and impact on the Oracle database. (文档 ID 730795.1)
Enterprise Manager Management Agent or OMS CPU Use Is Excessive near Leap Second Additions on Linux (文档 ID 1472651.1)
Leap Second on Oracle SuperCluster (文档 ID 1991954.1)
NTP leap second event causing Oracle Clusterware node reboot (文档 ID 759143.1)
leap seconds in Oracle Linux (文档 ID 2012659.1)
Insert leap seconds into a timestamp column fails with ORA-01852 (文档 ID 1553906.1)
Leap Second Time Adjustment (e.g. on June 30, 2015 at 23:59:59 UTC) and Its Impact on Exadata Database Machine (文档 ID 1986986.1)
ODA (Oracle Database Appliance): Leap Second adjustment impact (文档 ID 2015972.1)
Related questions with leap second on mysql (文档 ID 1450441.1)
http://www-01.ibm.com/support/docview.wss?uid=swg21602521
 

11.2.0.4.0 RAC升级PSU 11.2.0.4.6

自11.2.0.2版本之后的PSU补丁,由于Oracle官方推荐使用OCM响应文件以apply auto方式打补丁,因此每次发布的PSU补丁,对于DB虽有单独的补丁包,但是GI的PSU中也是包含DB的PSU补丁的,这样做的好处在于,使用OCM响应文件以apply auto方式打补丁方式,相比以前的manual方式就变得很简单,特别是对于RAC环境,每个节点只各执行一条命令即可完成,本文档就以官方推荐的方式使用OCM响应文件以apply auto方式,将11.2.0.4.0 RAC数据库更新到11.2.0.4.6

--- 环境描述
OEL: 5.10
GI : 11.2.0.4.0
DB : 11.2.0.4.0
PSU: p20485808_112040_Linux-x86-64.zip (该PSU补丁为GI 112.0.4.6其中包含DB 11.2.0.4.6)

--- 集群资源服务状态
[grid@11grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    11grac1     
ora....C1.lsnr application    ONLINE    ONLINE    11grac1     
ora....ac1.gsd application    OFFLINE   OFFLINE               
ora....ac1.ons application    ONLINE    ONLINE    11grac1     
ora....ac1.vip ora....t1.type ONLINE    ONLINE    11grac1     
ora....SM2.asm application    ONLINE    ONLINE    11grac2     
ora....C2.lsnr application    ONLINE    ONLINE    11grac2     
ora....ac2.gsd application    OFFLINE   OFFLINE               
ora....ac2.ons application    ONLINE    ONLINE    11grac2     
ora....ac2.vip ora....t1.type ONLINE    ONLINE    11grac2     
ora.DATA.dg    ora....up.type ONLINE    ONLINE    11grac1     
ora....ER.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora....N1.lsnr ora....er.type ONLINE    ONLINE    11grac2     
ora....N2.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora....N3.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora.OVDF.dg    ora....up.type ONLINE    ONLINE    11grac1     
ora.RFA.dg     ora....up.type ONLINE    ONLINE    11grac1     
ora.asm        ora.asm.type   ONLINE    ONLINE    11grac1     
ora.cvu        ora.cvu.type   ONLINE    ONLINE    11grac2     
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    11grac1     
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    11grac2     
ora.ons        ora.ons.type   ONLINE    ONLINE    11grac1     
ora.rac11g.db  ora....se.type ONLINE    ONLINE    11grac1     
ora....srv.svc ora....ce.type ONLINE    ONLINE    11grac1     
ora....ect.svc ora....ce.type ONLINE    ONLINE    11grac2     
ora....ry.acfs ora....fs.type ONLINE    ONLINE    11grac1     
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    11grac2     
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    11grac1     
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    11grac1     
[grid@11grac1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.OVDF.dg
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.RFA.dg
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.asm
               ONLINE  ONLINE       11grac1                  Started             
               ONLINE  ONLINE       11grac2                  Started             
ora.gsd
               OFFLINE OFFLINE      11grac1                                      
               OFFLINE OFFLINE      11grac2                                      
ora.net1.network
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.ons
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.registry.acfs
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.11grac1.vip
      1        ONLINE  ONLINE       11grac1                                      
ora.11grac2.vip
      1        ONLINE  ONLINE       11grac2                                      
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       11grac2                                      
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       11grac1                                      
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       11grac1                                      
ora.cvu
      1        ONLINE  ONLINE       11grac2                                      
ora.oc4j
      1        ONLINE  ONLINE       11grac2                                      
ora.rac11g.db
      1        ONLINE  ONLINE       11grac1                  Open                
      2        ONLINE  ONLINE       11grac2                  Open                
ora.rac11g.rac11gsrv.svc
      1        ONLINE  ONLINE       11grac1                                      
ora.rac11g.rac11gsrv_preconnect.svc
      1        ONLINE  ONLINE       11grac2                                      
ora.scan1.vip
      1        ONLINE  ONLINE       11grac2                                      
ora.scan2.vip
      1        ONLINE  ONLINE       11grac1                                      
ora.scan3.vip
      1        ONLINE  ONLINE       11grac1

1.检查OPatch及数据库版本信息
1)grid的OPatch版本
[grid@11grac1 ~]$ cd $GI_HOME/OPatch
[grid@11grac1 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2015-05-19_06-37-20AM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_06-37-20AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = 11grac1
  Remote node = 11grac2

--------------------------------------------------------------------------------

OPatch succeeded.

[grid@11grac2 ~]$ cd $GI_HOME/OPatch 
[grid@11grac2 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2015-05-19_06-46-43AM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_06-46-43AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = 11grac2
  Remote node = 11grac1

--------------------------------------------------------------------------------

OPatch succeeded.

2)oracle的OPatch版本
[oracle@11grac1 ~]$ cd $ORACLE_HOME/OPatch
[oracle@11grac1 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2015-05-19_06-37-55AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_06-37-55AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = 11grac1
  Remote node = 11grac2

--------------------------------------------------------------------------------

OPatch succeeded.

[oracle@11grac2 ~]$ cd $ORACLE_HOME/OPatch
[oracle@11grac2 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2015-05-19_06-47-39AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_06-47-39AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = 11grac2
  Remote node = 11grac1

--------------------------------------------------------------------------------

OPatch succeeded.

3)数据库版本
SQL> select * from DBA_REGISTRY_HISTORY

ACTION_TIME		       ACTION	       NAMESPACE  VERSION	   ID BUNDLE_SERIES   COMMENTS
------------------------------ --------------- ---------- ---------- -------- --------------- -------------------------
19-MAY-15 12.26.56.586864 AM   VIEW INVALIDATE			      8289601		      view invalidation
19-MAY-15 12.30.29.942568 AM   UPGRADE	       SERVER	  11.2.0.4.0			      Upgraded from 11.1.0.7.0
19-MAY-15 12.57.34.299763 AM   APPLY	       SERVER	  11.2.0.4	    0 PSU	      Patchset 11.2.0.2.0

2.升级前准备
根据11.2.0.4.6 Patch包readme文档说明,打该PSU需要OPatch版本最低为11.2.0.3.6及以上,因此需要更新grid和oracle软件安装目录的OPatch版本。
1)更新grid和oracle的OPatch版本
[grid@11grac1 ~]$ cd /tmp/psu/
[grid@11grac1 psu]$ unzip p6880880_112000_Linux-x86-64.zip -d $GI_HOME
[oracle@11grac1 ~]$ cd /tmp/psu/
[oracle@11grac1 psu]$ unzip p6880880_112000_Linux-x86-64.zip -d $ORACLE_HOME

[grid@11grac1 ~]$ $GI_HOME/OPatch/opatch version
OPatch Version: 11.2.0.3.6

OPatch succeeded.
[oracle@11grac1 ~]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 11.2.0.3.6

OPatch succeeded.

[grid@11grac2 ~]$ cd /tmp/psu/
[grid@11grac2 psu]$ unzip p6880880_112000_Linux-x86-64.zip -d $GI_HOME
[oracle@11grac2 ~]$ cd /tmp/psu/
[oracle@11grac2 psu]$ unzip p6880880_112000_Linux-x86-64.zip -d $ORACLE_HOME

[grid@11grac2 psu]$ $GI_HOME/OPatch/opatch version
OPatch Version: 11.2.0.3.6

OPatch succeeded.

[oracle@11grac2 ~]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 11.2.0.3.6

OPatch succeeded.

2)配置OCM响应文件
--- grid ocm响应文件
[grid@11grac1 ~]$ $GI_HOME/OPatch/ocm/bin/emocmrsp  -no_banner -output /tmp/grid.ocm
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/tmp/grid.ocm) was successfully created.

[grid@11grac2 ~]$ $GI_HOME/OPatch/ocm/bin/emocmrsp  -no_banner -output /tmp/grid.ocm
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/tmp/grid.ocm) was successfully created.

--- oracle ocm响应文件
[oracle@11grac1 ~]$ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp  -no_banner -output /tmp/oracle.com
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/tmp/oracle.com) was successfully created.

[oracle@11grac2 ~]$ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp  -no_banner -output /tmp/oracle.com
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/tmp/oracle.com) was successfully created.

3)校验Oracle Inventory
[grid@11grac1 ~]$ $GI_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/11.2.0/grid
[grid@11grac2 ~]$ $GI_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/11.2.0/grid
[oracle@11grac1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/oracle/product/11.2.0/dbhome_1
[oracle@11grac2 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/oracle/product/11.2.0/dbhome_1

4)下载解压补丁包
[grid@11grac1 ~]$ cd /tmp/psu/
[grid@11grac1 psu]$ unzip p20485808_112040_Linux-x86-64.zip
[grid@11grac2 ~]$ cd /tmp/psu/
[grid@11grac2 psu]$ unzip p20485808_112040_Linux-x86-64.zip

5)停EM(若未安装可以忽略此步骤)
[oracle@11grac1 ~]$ emctl stop dbconsole
[oracle@11grac2 ~]$ emctl stop dbconsole

6)检测One-off Patch冲突
根据Patch Recommendations and Patch Plans features on the Patches & Updates tab去检查

3.安装PSU补丁
As root user, execute the following command on each node of the cluster.

1)节点1安装11.2.0.4.6 PSU
[root@11grac1 ~]# $GI_HOME/OPatch/opatch auto /tmp/psu/20485808 -ocmrf /tmp/psu/grid.ocm
Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /tmp/psu -patchn 20485808 -ocmrf /tmp/psu/grid.ocm -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2015-05-19_08-26-20.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.0/grid/cfgtoollogs/opatchauto2015-05-19_08-26-20.report.log

2015-05-19 08:26:20: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

patch /tmp/psu/20485808/20299013  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1 
patch /tmp/psu/20485808/20420937/custom/server/20420937  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1 

Stopping CRS...
Stopped CRS successfully

patch /tmp/psu/20485808/20299013  apply successful for home  /u01/app/11.2.0/grid 
patch /tmp/psu/20485808/20420937  apply successful for home  /u01/app/11.2.0/grid 
patch /tmp/psu/20485808/20299019  apply successful for home  /u01/app/11.2.0/grid 

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Started RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

opatch auto succeeded.

--- 检查1节点PSU安装情况
[grid@11grac1 ~]$ $GI_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2015-05-19_10-19-18AM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_10-19-18AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.


Interim patches (3) :

Patch  20299019     : applied on Tue May 19 09:04:19 CST 2015
Unique Patch ID:  18573450
Patch description:  "ACFS Patch set update : 11.2.0.4.6 (20299019)"
   Created on 27 Mar 2015, 15:26:30 hrs 
   Bugs fixed:
     17510275, 17172303, 16318126, 19690653, 17203009, 17376318, 17503605
     20140148, 17611362, 17721778, 17164243, 19053182, 17696547, 17699423
     17488768, 18915417, 18155334, 18321597, 19919907, 18168684, 17363999
     17636008, 18143006, 17428148, 17070158, 17475946

Patch  20420937     : applied on Tue May 19 08:58:18 CST 2015
Unique Patch ID:  18573450
Patch description:  "OCW Patch set update : 11.2.0.4.6 (20420937)"
   Created on 27 Mar 2015, 15:19:31 hrs PST8PDT
   Bugs fixed:
     18328800, 19270660, 18691572, 20365005, 17750548, 17387214, 17617807
     14497275, 17733927, 18180541, 18962892, 17292250, 17378618, 16759171
     17065496, 13991403, 17273020, 17155238, 18053580, 18261183, 17039197
     16317771, 17947785, 10052729, 20340620, 18199185, 18399991, 18024089
     20246071, 18414137, 17001914, 17927970, 14378120, 16346413, 15986647
     18068871, 18143836, 16206997, 19168690, 20235511, 18343490, 16613232
     19276791, 17722664, 12928658, 16249829, 18520351, 18226143, 18265482
     18229842, 17172091, 17818075, 18231837, 14373486, 17483479, 18120545
     18729166, 13843841, 17405302, 18709496, 18330979, 18187697, 14525998
     20531190, 14385860, 19479503, 18348155, 18370031, 17516024, 17764053
     17551223, 14671408, 14207615, 18272135, 17500165, 18875012, 19558324
     18464784, 18848125, 19241857, 17955615, 14693336, 20315294, 16284825
     20014326, 17352230, 17238586, 17089344, 17405605, 17531342, 17640316
     17159489, 16543190, 18346135, 16281493, 17481314, 15986311, 17208793
     18700935, 18999857, 14076173, 18428146, 17435488, 18352845, 18352846
     17387779, 17391726, 20141091, 15832129, 17305100, 19885321, 16901346
     17985714, 18536826, 17780903, 18752378, 18946768, 16875342, 16876500
     19955755, 16429265, 18336452, 17273003, 17059927, 17046460, 18053631
     16867761, 20235486, 15869775, 19642566, 17447588, 15920201

Patch  20299013     : applied on Tue May 19 08:53:08 CST 2015
Unique Patch ID:  18573940
Patch description:  "Database Patch Set Update : 11.2.0.4.6 (20299013)"
   Created on 4 Mar 2015, 02:27:44 hrs PST8PDT
Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17288409, 17798953, 18273830, 18607546, 17811429, 17205719, 20506699
     17816865, 19972566, 17922254, 17754782, 16384983, 17726838, 13364795
     16934803, 17311728, 17284817, 17441661, 17360606, 13645875, 18199537
     16992075, 16542886, 17446237, 14015842, 17889549, 14565184, 19972569
     17071721, 20299015, 17610798, 17375354, 17449815, 17397545, 19463897
     18230522, 13866822, 17235750, 17982555, 16360112, 18317531, 17478514
     19769489, 12905058, 14338435, 18235390, 13944971, 18641451, 20142975
     17811789, 16929165, 18704244, 12747740, 18430495, 20506706, 17546973
     14054676, 17088068, 17346091, 18264060, 17016369, 17042658, 17343514
     14602788, 19972568, 19680952, 18471685, 19788842, 18508861, 14657740
     17332800, 19211724, 13837378, 13951456, 16315398, 17186905, 18744139
     19972564, 16850630, 18315328, 17437634, 19049453, 18673304, 17883081
     19006849, 19915271, 19013183, 18641419, 17296856, 18674024, 18262334
     17006183, 18277454, 16833527, 17232014, 16855292, 10136473, 17762296
     14692762, 17705023, 18051556, 17865671, 17852463, 18554871, 17853498
     19121551, 18334586, 19854503, 17551709, 19309466, 17588480, 19827973
     17344412, 17842825, 18828868, 18681862, 18554763, 17390160, 18456514
     16306373, 17025461, 13955826, 18139690, 11883252, 13609098, 17501491
     17239687, 17752121, 17299889, 17602269, 19197175, 17889583, 18316692
     17313525, 18673325, 12611721, 19544839, 18293054, 17242746, 18964939
     17600719, 18191164, 19393542, 17571306, 18482502, 19466309, 17951233
     17649265, 18094246, 19615136, 17040527, 17011832, 17165204, 18098207
     16785708, 16870214, 17465741, 16180763, 17174582, 17477958, 12982566
     16777840, 18522509, 20631274, 16091637, 17323222, 19463893, 16595641
     16875449, 12816846, 16524926, 17237521, 18228645, 18282562, 17596908
     19358317, 17811438, 17811447, 17945983, 18762750, 17156148, 18031668
     16912439, 17184721, 16494615, 18061914, 17282229, 17545847, 18331850
     18202441, 17082359, 18723434, 19554106, 17614134, 13558557, 17341326
     14034426, 17891946, 18339044, 17716305, 19458377, 17752995, 16392068
     19271443, 17891943, 18092127, 17258090, 17767676, 16668584, 18384391
     17614227, 17040764, 16903536, 17381384, 14106803, 15913355, 18973907
     18356166, 18673342, 17389192, 14084247, 16194160, 17612828, 17006570
     20506715, 17721717, 13853126, 17390431, 18203837, 17570240, 14245531
     16043574, 16863422, 17848897, 17877323, 18325460, 19727057, 17468141
     17786518, 17912217, 16422541, 19972570, 17267114, 17037130, 18244962
     18765602, 18203838, 18155762, 16956380, 16198143, 17246576, 17478145
     17394950, 14829250, 18189036, 18641461, 18619917, 17835627, 17027426
     16268425, 18247991, 19584068, 14458214, 18436307, 17265217, 17634921
     13498382, 16692232, 17786278, 17227277, 16042673, 16314254, 17443671
     18000422, 16228604, 16837842, 17571039, 17393683, 16344544, 17787259
     18009564, 20074391, 14354737, 15861775, 18135678, 18614015, 16399083
     18362222, 18018515, 16472716, 17835048, 17050888, 17936109, 14010183
     17325413, 18747196, 17080436, 16613964, 17036973, 17761775, 16579084
     16721594, 17082983, 18384537, 18280813, 20296213, 17302277, 16901385
     18084625, 15979965, 15990359, 18203835, 17297939, 17811456, 16731148
     13829543, 14133975, 17215560, 17694209, 18091059, 17385178, 8322815
     17586955, 18441944, 17201159, 16450169, 9756271, 17655634, 19730508
     17892268, 18868646, 17648596, 16220077, 16069901, 11733603, 16285691
     17587063, 18180390, 16538760, 18193833, 17348614, 17393915, 17957017
     17274537, 18096714, 17308789, 17238511, 18436647, 17824637, 14285317
     19289642, 14764829, 17622427, 18328509, 16571443, 16943711, 14368995
     18306996, 17346671, 14852021, 18996843, 17783588, 16618694, 17853456
     18674047, 17672719, 18856999, 12364061, 18783224, 17851160, 17546761



Rac system comprising of multiple nodes
  Local node = 11grac1
  Remote node = 11grac2

--------------------------------------------------------------------------------

OPatch succeeded.

[oracle@11grac1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2015-05-19_10-19-55AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_10-19-55AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.


Interim patches (2) :

Patch  20420937     : applied on Tue May 19 08:44:58 CST 2015
Unique Patch ID:  18573450
Patch description:  "OCW Patch set update : 11.2.0.4.6 (20420937)"
   Created on 27 Mar 2015, 15:19:23 hrs PST8PDT
   Bugs fixed:
     18328800, 19270660, 18691572, 20365005, 17750548, 17387214, 17617807
     14497275, 17733927, 18180541, 18962892, 17292250, 17378618, 16759171
     17065496, 13991403, 17273020, 17155238, 18053580, 18261183, 17039197
     16317771, 17947785, 10052729, 20340620, 18199185, 18399991, 18024089
     20246071, 18414137, 17001914, 17927970, 14378120, 16346413, 15986647
     18068871, 18143836, 16206997, 19168690, 20235511, 18343490, 16613232
     19276791, 17722664, 12928658, 16249829, 18520351, 18226143, 18265482
     18229842, 17172091, 17818075, 18231837, 14373486, 17483479, 18120545
     18729166, 13843841, 17405302, 18709496, 18330979, 18187697, 14525998
     20531190, 14385860, 19479503, 18348155, 18370031, 17516024, 17764053
     17551223, 14671408, 14207615, 18272135, 17500165, 18875012, 19558324
     18464784, 18848125, 19241857, 17955615, 14693336, 20315294, 16284825
     20014326, 17352230, 17238586, 17089344, 17405605, 17531342, 17640316
     17159489, 16543190, 18346135, 16281493, 17481314, 15986311, 17208793
     18700935, 18999857, 14076173, 18428146, 17435488, 18352845, 18352846
     17387779, 17391726, 20141091, 15832129, 17305100, 19885321, 16901346
     17985714, 18536826, 17780903, 18752378, 18946768, 16875342, 16876500
     19955755, 16429265, 18336452, 17273003, 17059927, 17046460, 18053631
     16867761, 20235486, 15869775, 19642566, 17447588, 15920201

Patch  20299013     : applied on Tue May 19 08:36:51 CST 2015
Unique Patch ID:  18573940
Patch description:  "Database Patch Set Update : 11.2.0.4.6 (20299013)"
   Created on 4 Mar 2015, 02:27:44 hrs PST8PDT
Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17288409, 17798953, 18273830, 18607546, 17811429, 17205719, 20506699
     17816865, 19972566, 17922254, 17754782, 16384983, 17726838, 13364795
     16934803, 17311728, 17284817, 17441661, 17360606, 13645875, 18199537
     16992075, 16542886, 17446237, 14015842, 17889549, 14565184, 19972569
     17071721, 20299015, 17610798, 17375354, 17449815, 17397545, 19463897
     18230522, 13866822, 17235750, 17982555, 16360112, 18317531, 17478514
     19769489, 12905058, 14338435, 18235390, 13944971, 18641451, 20142975
     17811789, 16929165, 18704244, 12747740, 18430495, 20506706, 17546973
     14054676, 17088068, 17346091, 18264060, 17016369, 17042658, 17343514
     14602788, 19972568, 19680952, 18471685, 19788842, 18508861, 14657740
     17332800, 19211724, 13837378, 13951456, 16315398, 17186905, 18744139
     19972564, 16850630, 18315328, 17437634, 19049453, 18673304, 17883081
     19006849, 19915271, 19013183, 18641419, 17296856, 18674024, 18262334
     17006183, 18277454, 16833527, 17232014, 16855292, 10136473, 17762296
     14692762, 17705023, 18051556, 17865671, 17852463, 18554871, 17853498
     19121551, 18334586, 19854503, 17551709, 19309466, 17588480, 19827973
     17344412, 17842825, 18828868, 18681862, 18554763, 17390160, 18456514
     16306373, 17025461, 13955826, 18139690, 11883252, 13609098, 17501491
     17239687, 17752121, 17299889, 17602269, 19197175, 17889583, 18316692
     17313525, 18673325, 12611721, 19544839, 18293054, 17242746, 18964939
     17600719, 18191164, 19393542, 17571306, 18482502, 19466309, 17951233
     17649265, 18094246, 19615136, 17040527, 17011832, 17165204, 18098207
     16785708, 16870214, 17465741, 16180763, 17174582, 17477958, 12982566
     16777840, 18522509, 20631274, 16091637, 17323222, 19463893, 16595641
     16875449, 12816846, 16524926, 17237521, 18228645, 18282562, 17596908
     19358317, 17811438, 17811447, 17945983, 18762750, 17156148, 18031668
     16912439, 17184721, 16494615, 18061914, 17282229, 17545847, 18331850
     18202441, 17082359, 18723434, 19554106, 17614134, 13558557, 17341326
     14034426, 17891946, 18339044, 17716305, 19458377, 17752995, 16392068
     19271443, 17891943, 18092127, 17258090, 17767676, 16668584, 18384391
     17614227, 17040764, 16903536, 17381384, 14106803, 15913355, 18973907
     18356166, 18673342, 17389192, 14084247, 16194160, 17612828, 17006570
     20506715, 17721717, 13853126, 17390431, 18203837, 17570240, 14245531
     16043574, 16863422, 17848897, 17877323, 18325460, 19727057, 17468141
     17786518, 17912217, 16422541, 19972570, 17267114, 17037130, 18244962
     18765602, 18203838, 18155762, 16956380, 16198143, 17246576, 17478145
     17394950, 14829250, 18189036, 18641461, 18619917, 17835627, 17027426
     16268425, 18247991, 19584068, 14458214, 18436307, 17265217, 17634921
     13498382, 16692232, 17786278, 17227277, 16042673, 16314254, 17443671
     18000422, 16228604, 16837842, 17571039, 17393683, 16344544, 17787259
     18009564, 20074391, 14354737, 15861775, 18135678, 18614015, 16399083
     18362222, 18018515, 16472716, 17835048, 17050888, 17936109, 14010183
     17325413, 18747196, 17080436, 16613964, 17036973, 17761775, 16579084
     16721594, 17082983, 18384537, 18280813, 20296213, 17302277, 16901385
     18084625, 15979965, 15990359, 18203835, 17297939, 17811456, 16731148
     13829543, 14133975, 17215560, 17694209, 18091059, 17385178, 8322815
     17586955, 18441944, 17201159, 16450169, 9756271, 17655634, 19730508
     17892268, 18868646, 17648596, 16220077, 16069901, 11733603, 16285691
     17587063, 18180390, 16538760, 18193833, 17348614, 17393915, 17957017
     17274537, 18096714, 17308789, 17238511, 18436647, 17824637, 14285317
     19289642, 14764829, 17622427, 18328509, 16571443, 16943711, 14368995
     18306996, 17346671, 14852021, 18996843, 17783588, 16618694, 17853456
     18674047, 17672719, 18856999, 12364061, 18783224, 17851160, 17546761



Rac system comprising of multiple nodes
  Local node = 11grac1
  Remote node = 11grac2

--------------------------------------------------------------------------------

OPatch succeeded.

2)节点2安装11.2.0.4.6 PSU
[root@11grac2 ~]# $GI_HOME/OPatch/opatch auto /tmp/psu/20485808 -ocmrf /tmp/psu/grid.ocm
Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /tmp/psu -patchn 20485808 -ocmrf /tmp/psu/grid.ocm -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2015-05-19_10-40-50.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.0/grid/cfgtoollogs/opatchauto2015-05-19_10-40-50.report.log

2015-05-19 10:40:50: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

patch /tmp/psu/20485808/20299013  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1 
patch /tmp/psu/20485808/20420937/custom/server/20420937  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1 

Stopping CRS...
Stopped CRS successfully

patch /tmp/psu/20485808/20299013  apply successful for home  /u01/app/11.2.0/grid 
patch /tmp/psu/20485808/20420937  apply successful for home  /u01/app/11.2.0/grid 
patch /tmp/psu/20485808/20299019  apply successful for home  /u01/app/11.2.0/grid 

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Started RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully

opatch auto succeeded.

--- 检查2节点PSU安装情况
[grid@11grac2 ~]$ $GI_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2015-05-19_11-41-10AM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_11-41-10AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.


Interim patches (3) :

Patch  20299019     : applied on Tue May 19 11:31:45 CST 2015
Unique Patch ID:  18573450
Patch description:  "ACFS Patch set update : 11.2.0.4.6 (20299019)"
   Created on 27 Mar 2015, 15:26:30 hrs 
   Bugs fixed:
     17510275, 17172303, 16318126, 19690653, 17203009, 17376318, 17503605
     20140148, 17611362, 17721778, 17164243, 19053182, 17696547, 17699423
     17488768, 18915417, 18155334, 18321597, 19919907, 18168684, 17363999
     17636008, 18143006, 17428148, 17070158, 17475946

Patch  20420937     : applied on Tue May 19 11:25:58 CST 2015
Unique Patch ID:  18573450
Patch description:  "OCW Patch set update : 11.2.0.4.6 (20420937)"
   Created on 27 Mar 2015, 15:19:31 hrs PST8PDT
   Bugs fixed:
     18328800, 19270660, 18691572, 20365005, 17750548, 17387214, 17617807
     14497275, 17733927, 18180541, 18962892, 17292250, 17378618, 16759171
     17065496, 13991403, 17273020, 17155238, 18053580, 18261183, 17039197
     16317771, 17947785, 10052729, 20340620, 18199185, 18399991, 18024089
     20246071, 18414137, 17001914, 17927970, 14378120, 16346413, 15986647
     18068871, 18143836, 16206997, 19168690, 20235511, 18343490, 16613232
     19276791, 17722664, 12928658, 16249829, 18520351, 18226143, 18265482
     18229842, 17172091, 17818075, 18231837, 14373486, 17483479, 18120545
     18729166, 13843841, 17405302, 18709496, 18330979, 18187697, 14525998
     20531190, 14385860, 19479503, 18348155, 18370031, 17516024, 17764053
     17551223, 14671408, 14207615, 18272135, 17500165, 18875012, 19558324
     18464784, 18848125, 19241857, 17955615, 14693336, 20315294, 16284825
     20014326, 17352230, 17238586, 17089344, 17405605, 17531342, 17640316
     17159489, 16543190, 18346135, 16281493, 17481314, 15986311, 17208793
     18700935, 18999857, 14076173, 18428146, 17435488, 18352845, 18352846
     17387779, 17391726, 20141091, 15832129, 17305100, 19885321, 16901346
     17985714, 18536826, 17780903, 18752378, 18946768, 16875342, 16876500
     19955755, 16429265, 18336452, 17273003, 17059927, 17046460, 18053631
     16867761, 20235486, 15869775, 19642566, 17447588, 15920201

Patch  20299013     : applied on Tue May 19 11:19:35 CST 2015
Unique Patch ID:  18573940
Patch description:  "Database Patch Set Update : 11.2.0.4.6 (20299013)"
   Created on 4 Mar 2015, 02:27:44 hrs PST8PDT
Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17288409, 17798953, 18273830, 18607546, 17811429, 17205719, 20506699
     17816865, 19972566, 17922254, 17754782, 16384983, 17726838, 13364795
     16934803, 17311728, 17284817, 17441661, 17360606, 13645875, 18199537
     16992075, 16542886, 17446237, 14015842, 17889549, 14565184, 19972569
     17071721, 20299015, 17610798, 17375354, 17449815, 17397545, 19463897
     18230522, 13866822, 17235750, 17982555, 16360112, 18317531, 17478514
     19769489, 12905058, 14338435, 18235390, 13944971, 18641451, 20142975
     17811789, 16929165, 18704244, 12747740, 18430495, 20506706, 17546973
     14054676, 17088068, 17346091, 18264060, 17016369, 17042658, 17343514
     14602788, 19972568, 19680952, 18471685, 19788842, 18508861, 14657740
     17332800, 19211724, 13837378, 13951456, 16315398, 17186905, 18744139
     19972564, 16850630, 18315328, 17437634, 19049453, 18673304, 17883081
     19006849, 19915271, 19013183, 18641419, 17296856, 18674024, 18262334
     17006183, 18277454, 16833527, 17232014, 16855292, 10136473, 17762296
     14692762, 17705023, 18051556, 17865671, 17852463, 18554871, 17853498
     19121551, 18334586, 19854503, 17551709, 19309466, 17588480, 19827973
     17344412, 17842825, 18828868, 18681862, 18554763, 17390160, 18456514
     16306373, 17025461, 13955826, 18139690, 11883252, 13609098, 17501491
     17239687, 17752121, 17299889, 17602269, 19197175, 17889583, 18316692
     17313525, 18673325, 12611721, 19544839, 18293054, 17242746, 18964939
     17600719, 18191164, 19393542, 17571306, 18482502, 19466309, 17951233
     17649265, 18094246, 19615136, 17040527, 17011832, 17165204, 18098207
     16785708, 16870214, 17465741, 16180763, 17174582, 17477958, 12982566
     16777840, 18522509, 20631274, 16091637, 17323222, 19463893, 16595641
     16875449, 12816846, 16524926, 17237521, 18228645, 18282562, 17596908
     19358317, 17811438, 17811447, 17945983, 18762750, 17156148, 18031668
     16912439, 17184721, 16494615, 18061914, 17282229, 17545847, 18331850
     18202441, 17082359, 18723434, 19554106, 17614134, 13558557, 17341326
     14034426, 17891946, 18339044, 17716305, 19458377, 17752995, 16392068
     19271443, 17891943, 18092127, 17258090, 17767676, 16668584, 18384391
     17614227, 17040764, 16903536, 17381384, 14106803, 15913355, 18973907
     18356166, 18673342, 17389192, 14084247, 16194160, 17612828, 17006570
     20506715, 17721717, 13853126, 17390431, 18203837, 17570240, 14245531
     16043574, 16863422, 17848897, 17877323, 18325460, 19727057, 17468141
     17786518, 17912217, 16422541, 19972570, 17267114, 17037130, 18244962
     18765602, 18203838, 18155762, 16956380, 16198143, 17246576, 17478145
     17394950, 14829250, 18189036, 18641461, 18619917, 17835627, 17027426
     16268425, 18247991, 19584068, 14458214, 18436307, 17265217, 17634921
     13498382, 16692232, 17786278, 17227277, 16042673, 16314254, 17443671
     18000422, 16228604, 16837842, 17571039, 17393683, 16344544, 17787259
     18009564, 20074391, 14354737, 15861775, 18135678, 18614015, 16399083
     18362222, 18018515, 16472716, 17835048, 17050888, 17936109, 14010183
     17325413, 18747196, 17080436, 16613964, 17036973, 17761775, 16579084
     16721594, 17082983, 18384537, 18280813, 20296213, 17302277, 16901385
     18084625, 15979965, 15990359, 18203835, 17297939, 17811456, 16731148
     13829543, 14133975, 17215560, 17694209, 18091059, 17385178, 8322815
     17586955, 18441944, 17201159, 16450169, 9756271, 17655634, 19730508
     17892268, 18868646, 17648596, 16220077, 16069901, 11733603, 16285691
     17587063, 18180390, 16538760, 18193833, 17348614, 17393915, 17957017
     17274537, 18096714, 17308789, 17238511, 18436647, 17824637, 14285317
     19289642, 14764829, 17622427, 18328509, 16571443, 16943711, 14368995
     18306996, 17346671, 14852021, 18996843, 17783588, 16618694, 17853456
     18674047, 17672719, 18856999, 12364061, 18783224, 17851160, 17546761



Rac system comprising of multiple nodes
  Local node = 11grac2
  Remote node = 11grac1

--------------------------------------------------------------------------------

OPatch succeeded.

[oracle@11grac2 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2015-05-19_11-41-47AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2015-05-19_11-41-47AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.


Interim patches (2) :

Patch  20420937     : applied on Tue May 19 11:07:20 CST 2015
Unique Patch ID:  18573450
Patch description:  "OCW Patch set update : 11.2.0.4.6 (20420937)"
   Created on 27 Mar 2015, 15:19:23 hrs PST8PDT
   Bugs fixed:
     18328800, 19270660, 18691572, 20365005, 17750548, 17387214, 17617807
     14497275, 17733927, 18180541, 18962892, 17292250, 17378618, 16759171
     17065496, 13991403, 17273020, 17155238, 18053580, 18261183, 17039197
     16317771, 17947785, 10052729, 20340620, 18199185, 18399991, 18024089
     20246071, 18414137, 17001914, 17927970, 14378120, 16346413, 15986647
     18068871, 18143836, 16206997, 19168690, 20235511, 18343490, 16613232
     19276791, 17722664, 12928658, 16249829, 18520351, 18226143, 18265482
     18229842, 17172091, 17818075, 18231837, 14373486, 17483479, 18120545
     18729166, 13843841, 17405302, 18709496, 18330979, 18187697, 14525998
     20531190, 14385860, 19479503, 18348155, 18370031, 17516024, 17764053
     17551223, 14671408, 14207615, 18272135, 17500165, 18875012, 19558324
     18464784, 18848125, 19241857, 17955615, 14693336, 20315294, 16284825
     20014326, 17352230, 17238586, 17089344, 17405605, 17531342, 17640316
     17159489, 16543190, 18346135, 16281493, 17481314, 15986311, 17208793
     18700935, 18999857, 14076173, 18428146, 17435488, 18352845, 18352846
     17387779, 17391726, 20141091, 15832129, 17305100, 19885321, 16901346
     17985714, 18536826, 17780903, 18752378, 18946768, 16875342, 16876500
     19955755, 16429265, 18336452, 17273003, 17059927, 17046460, 18053631
     16867761, 20235486, 15869775, 19642566, 17447588, 15920201

Patch  20299013     : applied on Tue May 19 10:52:28 CST 2015
Unique Patch ID:  18573940
Patch description:  "Database Patch Set Update : 11.2.0.4.6 (20299013)"
   Created on 4 Mar 2015, 02:27:44 hrs PST8PDT
Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17288409, 17798953, 18273830, 18607546, 17811429, 17205719, 20506699
     17816865, 19972566, 17922254, 17754782, 16384983, 17726838, 13364795
     16934803, 17311728, 17284817, 17441661, 17360606, 13645875, 18199537
     16992075, 16542886, 17446237, 14015842, 17889549, 14565184, 19972569
     17071721, 20299015, 17610798, 17375354, 17449815, 17397545, 19463897
     18230522, 13866822, 17235750, 17982555, 16360112, 18317531, 17478514
     19769489, 12905058, 14338435, 18235390, 13944971, 18641451, 20142975
     17811789, 16929165, 18704244, 12747740, 18430495, 20506706, 17546973
     14054676, 17088068, 17346091, 18264060, 17016369, 17042658, 17343514
     14602788, 19972568, 19680952, 18471685, 19788842, 18508861, 14657740
     17332800, 19211724, 13837378, 13951456, 16315398, 17186905, 18744139
     19972564, 16850630, 18315328, 17437634, 19049453, 18673304, 17883081
     19006849, 19915271, 19013183, 18641419, 17296856, 18674024, 18262334
     17006183, 18277454, 16833527, 17232014, 16855292, 10136473, 17762296
     14692762, 17705023, 18051556, 17865671, 17852463, 18554871, 17853498
     19121551, 18334586, 19854503, 17551709, 19309466, 17588480, 19827973
     17344412, 17842825, 18828868, 18681862, 18554763, 17390160, 18456514
     16306373, 17025461, 13955826, 18139690, 11883252, 13609098, 17501491
     17239687, 17752121, 17299889, 17602269, 19197175, 17889583, 18316692
     17313525, 18673325, 12611721, 19544839, 18293054, 17242746, 18964939
     17600719, 18191164, 19393542, 17571306, 18482502, 19466309, 17951233
     17649265, 18094246, 19615136, 17040527, 17011832, 17165204, 18098207
     16785708, 16870214, 17465741, 16180763, 17174582, 17477958, 12982566
     16777840, 18522509, 20631274, 16091637, 17323222, 19463893, 16595641
     16875449, 12816846, 16524926, 17237521, 18228645, 18282562, 17596908
     19358317, 17811438, 17811447, 17945983, 18762750, 17156148, 18031668
     16912439, 17184721, 16494615, 18061914, 17282229, 17545847, 18331850
     18202441, 17082359, 18723434, 19554106, 17614134, 13558557, 17341326
     14034426, 17891946, 18339044, 17716305, 19458377, 17752995, 16392068
     19271443, 17891943, 18092127, 17258090, 17767676, 16668584, 18384391
     17614227, 17040764, 16903536, 17381384, 14106803, 15913355, 18973907
     18356166, 18673342, 17389192, 14084247, 16194160, 17612828, 17006570
     20506715, 17721717, 13853126, 17390431, 18203837, 17570240, 14245531
     16043574, 16863422, 17848897, 17877323, 18325460, 19727057, 17468141
     17786518, 17912217, 16422541, 19972570, 17267114, 17037130, 18244962
     18765602, 18203838, 18155762, 16956380, 16198143, 17246576, 17478145
     17394950, 14829250, 18189036, 18641461, 18619917, 17835627, 17027426
     16268425, 18247991, 19584068, 14458214, 18436307, 17265217, 17634921
     13498382, 16692232, 17786278, 17227277, 16042673, 16314254, 17443671
     18000422, 16228604, 16837842, 17571039, 17393683, 16344544, 17787259
     18009564, 20074391, 14354737, 15861775, 18135678, 18614015, 16399083
     18362222, 18018515, 16472716, 17835048, 17050888, 17936109, 14010183
     17325413, 18747196, 17080436, 16613964, 17036973, 17761775, 16579084
     16721594, 17082983, 18384537, 18280813, 20296213, 17302277, 16901385
     18084625, 15979965, 15990359, 18203835, 17297939, 17811456, 16731148
     13829543, 14133975, 17215560, 17694209, 18091059, 17385178, 8322815
     17586955, 18441944, 17201159, 16450169, 9756271, 17655634, 19730508
     17892268, 18868646, 17648596, 16220077, 16069901, 11733603, 16285691
     17587063, 18180390, 16538760, 18193833, 17348614, 17393915, 17957017
     17274537, 18096714, 17308789, 17238511, 18436647, 17824637, 14285317
     19289642, 14764829, 17622427, 18328509, 16571443, 16943711, 14368995
     18306996, 17346671, 14852021, 18996843, 17783588, 16618694, 17853456
     18674047, 17672719, 18856999, 12364061, 18783224, 17851160, 17546761



Rac system comprising of multiple nodes
  Local node = 11grac2
  Remote node = 11grac1

--------------------------------------------------------------------------------

OPatch succeeded.

3)应用one-off conflict resolution patches

4)Loading Modified SQL Files into the Database(运行psu apply脚本)
For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows

--- 节点1实例
[oracle@11grac1 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@11grac1 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue May 19 11:47:34 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @catbundle.sql psu apply

--- 节点2实例
[oracle@11grac2 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@11grac2 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue May 19 11:47:34 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @catbundle.sql psu apply

--- 运行完该脚本后请检查日志输出文件看是否存在错误:
1节点:/u01/app/cfgtoollogs/catbundle/catbundle_PSU_RAC11G_APPLY_2015May19_11_48_06.log
2节点:/u01/app/cfgtoollogs/catbundle/catbundle_PSU_RAC11G_APPLY_2015May19_11_52_45.log

--- 根据实际情况选做执行以下脚本
If the OJVM PSU was applied for a previous GI PSU patch, you may see invalid Java classes after execution of the catbundle.sql script in the previous step. If this is the case, run utlrp.sql to re-validate these Java classes.
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql

Upgrade Oracle Recovery Manager Catalog
If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it:
$ rman catalog username/password@alias
RMAN> UPGRADE CATALOG;

5)集群资源服务状态
[grid@11grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    11grac1     
ora....C1.lsnr application    ONLINE    ONLINE    11grac1     
ora....ac1.gsd application    OFFLINE   OFFLINE               
ora....ac1.ons application    ONLINE    ONLINE    11grac1     
ora....ac1.vip ora....t1.type ONLINE    ONLINE    11grac1     
ora....SM2.asm application    ONLINE    ONLINE    11grac2     
ora....C2.lsnr application    ONLINE    ONLINE    11grac2     
ora....ac2.gsd application    OFFLINE   OFFLINE               
ora....ac2.ons application    ONLINE    ONLINE    11grac2     
ora....ac2.vip ora....t1.type ONLINE    ONLINE    11grac2     
ora.DATA.dg    ora....up.type ONLINE    ONLINE    11grac1     
ora....ER.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora....N1.lsnr ora....er.type ONLINE    ONLINE    11grac2     
ora....N2.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora....N3.lsnr ora....er.type ONLINE    ONLINE    11grac1     
ora.OVDF.dg    ora....up.type ONLINE    ONLINE    11grac1     
ora.RFA.dg     ora....up.type ONLINE    ONLINE    11grac1     
ora.asm        ora.asm.type   ONLINE    ONLINE    11grac1     
ora.cvu        ora.cvu.type   ONLINE    ONLINE    11grac1     
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    11grac1     
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    11grac1     
ora.ons        ora.ons.type   ONLINE    ONLINE    11grac1     
ora.rac11g.db  ora....se.type ONLINE    ONLINE    11grac1     
ora....srv.svc ora....ce.type ONLINE    ONLINE    11grac1     
ora....ect.svc ora....ce.type ONLINE    ONLINE    11grac2     
ora....ry.acfs ora....fs.type ONLINE    ONLINE    11grac1     
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    11grac2     
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    11grac1     
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    11grac1
[grid@11grac1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.OVDF.dg
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.RFA.dg
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.asm
               ONLINE  ONLINE       11grac1                  Started             
               ONLINE  ONLINE       11grac2                  Started             
ora.gsd
               OFFLINE OFFLINE      11grac1                                      
               OFFLINE OFFLINE      11grac2                                      
ora.net1.network
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.ons
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
ora.registry.acfs
               ONLINE  ONLINE       11grac1                                      
               ONLINE  ONLINE       11grac2                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.11grac1.vip
      1        ONLINE  ONLINE       11grac1                                      
ora.11grac2.vip
      1        ONLINE  ONLINE       11grac2                                      
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       11grac2                                      
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       11grac1                                      
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       11grac1                                      
ora.cvu
      1        ONLINE  ONLINE       11grac1                                      
ora.oc4j
      1        ONLINE  ONLINE       11grac1                                      
ora.rac11g.db
      1        ONLINE  ONLINE       11grac1                  Open                
      2        ONLINE  ONLINE       11grac2                  Open                
ora.rac11g.rac11gsrv.svc
      1        ONLINE  ONLINE       11grac1                                      
ora.rac11g.rac11gsrv_preconnect.svc
      1        ONLINE  ONLINE       11grac2                                      
ora.scan1.vip
      1        ONLINE  ONLINE       11grac2                                      
ora.scan2.vip
      1        ONLINE  ONLINE       11grac1                                      
ora.scan3.vip
      1        ONLINE  ONLINE       11grac1                                      

4.检查数据库PSU版本
SQL> select * from dba_registry_history

ACTION_TIME                      ACTION          NAMESPACE  VERSION              ID BUNDLE_SERIES   COMMENTS
-------------------------------- --------------- ---------- ------------ ---------- --------------- -------------------------
19-MAY-15 12.26.56.586864 AM     VIEW INVALIDATE                            8289601                 view invalidation
19-MAY-15 12.30.29.942568 AM     UPGRADE         SERVER     11.2.0.4.0                              Upgraded from 11.1.0.7.0
19-MAY-15 12.57.34.299763 AM     APPLY           SERVER     11.2.0.4              0 PSU             Patchset 11.2.0.2.0
19-MAY-15 11.51.52.913326 AM     APPLY           SERVER     11.2.0.4              6 PSU             PSU 11.2.0.4.6
19-MAY-15 11.53.18.266198 AM     APPLY           SERVER     11.2.0.4              6 PSU             PSU 11.2.0.4.6