| In this Document | Physical Layout of the Private Interconnect |
| Why Do We Need a Private Interconnect ? |
| Interconnect High Availability |
| Private Interconnect for Oracle Clusterware |
| Private Interconnect for RAC |
| Identification of the Private Interconnect for RAC |
| Private Interconnect for ASM |
| Different Interconnects for Clusterware and RAC |
| CLUSTER_INTERCONNECTS Parameter |
| Changing the Private Interconnect |
Applies to: Oracle Database - Enterprise Edition - Version 10.1.0.2 and later
Oracle Database Cloud Schema Service - Version N/A and later
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.
Oracle Server Enterprise Edition - Version: 10.1.0.2 to 11.1.0.7
Last Updated: 20-Feb-2012
Purpose The purpose of this note is to make DBAs and System Administrators familiar with the concept of the cluster or private interconnect and its usage in Oracle Clusterware and Oracle Real Application Clusters The main audience is DBAs and System Administrators. This note applies to Oracle Clusterware and Oracle Real Application Clusters releases 10.1, 10.2 and 11.1. Details The term 'Private Interconnect' or 'Cluster Interconnect' is the private communication link between cluster nodes.
For Oracle Real Application Clusters and Oracle Clusterware we can differentiate between
1. Physical interface (including NIC, cables and switches)
2. Private Interconnect used by Oracle Clusterware
3. Private Interconnect used by Oracle Real Application Clusters (RAC) Physical Layout of the Private Interconnect The basic requirements are described in the Installation Guide for each platform. Additional information about certification can be found on Metalink Certify.
The interconnect as identified by both subnet number and interface name must be configured on all clustered nodes.
A switch between the clustered nodes is an absolute requirement. Why Do We Need a Private Interconnect ? Clusterware uses the interconnect for cluster synchronization (network heartbeat) and daemon communication between the the clustered nodes. This communication is based on the TCP protocol.
RAC uses the interconnect for cache fusion (UDP) and inter-process communication (TCP). Cache Fusion is the remote memory mapping of Oracle buffers, shared between the caches of participating nodes in the cluster. The volume and traffic patterns of this type of data, shared between nodes can vary greatly depending on the applications.
There are some vendor specific protocol exceptions in Oracle 10g. Interconnect Failure The private interconnect is the critical communication link between nodes and instances. Network errors will negatively impact Oracle Clusterware communication as well as RAC communication and performance.
Private interconnect failures are recognized by Oracle Clusterware and result in what is known as a 'split-brain' or subdivided cluster.
A subdivided cluster can result in data corruption, consequently immediate action is taken to resolve this condition. Interconnect failures, therefore, result in a node or subset of nodes in the cluster shutting down. In the case of two equally sized sub clusters, it is basically random which sub cluster will survive, and a customer's architecture and design should be based on this. At this time the Oracle Clusterware happens to use the node numbers to resolve this, but this could change in the future.
RAC instances will wait for the end of cluster reconfiguration to start their own reconfiguration. Interconnect High Availability It is an Oracle Best Practice to make the private interconnect highly available. Depending on the operating system and vendor you can use OS network drivers such as bonding, teaming, IPMP, Etherchannel, APA, MultiPrivNIC. For that purpose the interconnect should be configured across two NICs as well as two switches for complete redundancy so that the cluster can survive a single point of failure.
The setup of the highly available interconnect is transparent to Oracle Clusterware and RAC. This means that an underlying failure is handled by the OS or networking drivers that manage the interfaces. Oracle software does not recognize the change underneath, because the failover is handled by the operating system transparently to Oracle. Private Interconnect for Oracle Clusterware The private node name determines the interface being used for Oracle Clusterware and is defined during installation of the Clusterware.
With 3rd party Vendor Clusterware in place Oracle Clusterware should be configured to use the same interconnect (often referred to as the heartbeat) as the underlying vendor cluster software.
There exist three ways to identify the private node name after installation: - olsnodes -n -p can be used to identify the private node name.
[oracle@racnode1 ~]$ olsnodes -n -p
racnode1 1 racnode1-priv
racnode2 2 racnode2-priv
[oracle@racnode1 ~]$ - You may also check the private node name in the ocrdump output.
[SYSTEM.css.node_numbers.node1.privatename]
ORATEXT : racnode1-priv - ocssd.log has a line with clssnmClusterListener
[ CSSD]2009-02-23 03:09:06.945 [3086] >TRACE: clssnmClusterListener: Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=racnode1-priv)(PORT=49895))
Although the Oracle Universal Installer allows IP addresses when prompted for the private node name you should always use the host names defined in the hosts file or DNS. This enables you to change the IP addresses when you need to move the server to a different IP range. You would have to reinstall Oracle Clusterware when you used IP-addresses instead of host names and you are changing the server's IP adresses. Private Interconnect for RAC The database uses the private interconnect for communication and cache fusion. The private interconnect is also referred to as "The Cluster Interconnect". Oracle highly recommends that the database and Oracle Clusterware share the same interconnect.
The instances get the private interconnect definition from - spfile or init.ora when the parameter CLUSTER_INTERCONNECTS was set
- Oracle Cluster Regsitry (OCR) defined during the installation (default)
- automatically from platform defaults if not defined in the spfile/init.ora and OCR.
A value in the spfile or init.ora overrides a definition in the OCR. Identification of the Private Interconnect for RAC The value of the private interconnect for an instance can be identified using - The views V$CLUSTER_INTERCONNECTS and V$CONFIGURED_INTERCONNECTS
V$CLUSTER_INTERCONNECTS displays one or more interconnects that are being used for cluster communication.
V$CONFIGURED_INTERCONNECTS displays all the interconnects that Oracle is aware of. This view aims to answer the question on where Oracle found the information about a specific interconnect.
These views are not available in Oracle 10g 10.1. - The alert.log
- CLUSTER_INTERCONNECTS parameter is set
- Value read from Cluster Registry
Interface type 1 Database Team 10.x.x.0 configured from OCR for use as a cluster interconnect - Interconnect finally used
Cluster communication is configured to use the following interface(s) for this instance
10.x.x.1
Note that there might be a difference to the interconnect defined in the OCR because the value was overridden by setting the CLUSTER_INTERCONNECTS parameter or when Oracle selected another interconnect because the one in the OCR was not available. - Protocol
Unix and Linux cluster interconnect IPC version:Oracle UDP/IP (generic)
Windows: cluster interconnect IPC version:Oracle 9i Winsock2 TCP/IP IPC
Private Interconnect for ASM Oracle ASM must use the same interconnect as Oracle Clusterware. Different Interconnects for Clusterware and RAC Oracle Clusterware does not monitor interfaces other than the one specified via the private host name during installation. If RAC instances run with an interconnect that is different from the Oracle Clusterware interconnect any failure will remain undetected by Oracle Clusterware. The instances will detect this failure themselves and clear the situation by a mechanism called Instance Membership Recovery (IMR). Instance evictions due to IMR happen after 10 minutes.
This configuration should be avoided. CLUSTER_INTERCONNECTS Parameter This parameter overrides the value configured in the OCR. You may specify multiple IP addresses in CLUSTER_INTERCONNECTS which will cause all defined interfaces to be used. Keep in mind that a failure of one of those interfaces will cause the instance to fail. The failed instance can only be restarted when all the interfaces are fixed or you start all the instances without the faulty interface.
Changing the Private Interconnect - Clusterware
- Change of the IP Address: change the IP address in the hosts file and/or DNS and make sure that ASM and the database also use the same interconnect
- Change of the private node name used by Oracle Clusterware: requires a reinstall of Oracle Clusterware
- Real Application Clusters
- OCR: Use Note 283684.1 How to Change Interconnect/Public Interface IP or Subnet in Oracle Clusterware
- CLUSTER_INTERCONNECTS: shutdown all instances, change the IP address and restart the instances
Database - RAC/Scalability Community
To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Database - RAC/Scalability Community References
NOTE:283684.1 - How to Modify Private Network Information in Oracle Clusterware
NOTE:341788.1 - Recommendation for the Real Application Cluster Interconnect and Jumbo Frames | 