安装一套RAC11G的测试环境，操作系统为Solaris 10，共享磁盘管理使用Volumn Cluster Manager，Oracle版本为1106。

这篇描述Oracle Clusterware的安装。

Solaris10下安装Oracle1106RAC环境（一）：http://yangtingkun.itpub.net/post/468/467067

首先利用Oracle Cluster包中自带的检测工具，检查是否所有的配置准备工作已经完成：

bash-3.00$ ./runcluvfy.sh stage -pre crsinst -n newtrade1,newtrade2 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "newtrade1"
Destination Node Reachable?
------------------------------------ ------------------------
newtrade2 yes
newtrade1 yes
Result: Node reachability check passed from node "newtrade1".

Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
newtrade2 passed
newtrade1 passed
Result: User equivalence check passed for user "oracle".

Checking administrative privileges...

Check: Existence of user "oracle"
Node Name User Exists Comment
------------ ------------------------ ------------------------
newtrade2 yes passed
newtrade1 yes passed
Result: User existence check passed for "oracle".

Check: Existence of group "oinstall"
Node Name Status Group ID
------------ ------------------------ ------------------------
newtrade2 exists 100
newtrade1 exists 100
Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
---------------- ------------ ------------ ------------ ------------ ------------
newtrade2 yes yes yes yes passed
newtrade1 yes yes yes yes passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed.

Administrative privileges check passed.

Checking node connectivity...

Interface information for node "newtrade2"
Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address
---------------- ------------ ------------ ------------ ------------ ------------
ce0 172.0.2.55 172.0.2.0 172.0.2.55 UNKNOWN 00:14:4f:47:8d:12
ce1 10.0.1.4 10.0.1.0 10.0.1.4 UNKNOWN 00:14:4f:47:8d:12

Interface information for node "newtrade1"
Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address
---------------- ------------ ------------ ------------ ------------ ------------
ce0 172.0.2.54 172.0.2.0 172.0.2.54 UNKNOWN 00:14:4f:ad:47:cc
ce1 10.0.1.3 10.0.1.0 10.0.1.3 UNKNOWN 00:14:4f:ad:47:cc

Check: Node connectivity of subnet "172.0.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
newtrade2:ce0 newtrade1:ce0 yes
Result: Node connectivity check passed for subnet "172.0.2.0" with node(s) newtrade2,newtrade1.

Check: Node connectivity of subnet "10.0.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
newtrade2:ce1 newtrade1:ce1 yes
Result: Node connectivity check passed for subnet "10.0.1.0" with node(s) newtrade2,newtrade1.

Interfaces found on subnet "172.0.2.0" that are likely candidates for VIP:
newtrade2 ce0:172.0.2.55
newtrade1 ce0:172.0.2.54

Interfaces found on subnet "10.0.1.0" that are likely candidates for VIP:
newtrade2 ce1:10.0.1.4
newtrade1 ce1:10.0.1.3

WARNING:
Could not find a suitable set of interfaces for the private interconnect.

Result: Node connectivity check passed.

Checking system requirements for 'crs'...

Check: Total memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
newtrade2 32GB (33554432KB) 1GB (1048576KB) passed
newtrade1 32GB (33554432KB) 1GB (1048576KB) passed
Result: Total memory check passed.

Check: Free disk space in "/tmp" dir
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
newtrade2 57.62GB (60423736KB) 400MB (409600KB) passed
newtrade1 55.95GB (58670696KB) 400MB (409600KB) passed
Result: Free disk space check passed.

Check: Swap space
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
newtrade2 31.26GB (32776888KB) 1.5GB (1572864KB) passed
newtrade1 30.28GB (31749112KB) 1.5GB (1572864KB) passed
Result: Swap space check passed.

Check: System architecture
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
newtrade2 64-bit sparcv9 kernel modules 64-bit passed
newtrade1 64-bit sparcv9 kernel modules 64-bit passed
Result: System architecture check passed.

Check: Kernel version
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
newtrade2 10,REV=2006.11.14.15.02 10,REV=2005.01.22.19.14 passed
newtrade1 10,REV=2007.08.16.13.29 10,REV=2005.01.22.19.14 passed
Result: Kernel version check passed.

Check: Package existence for "SUNWarc"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWarc:11.10.0 passed
newtrade1 SUNWarc:11.10.0 passed
Result: Package existence check passed for "SUNWarc".

Check: Package existence for "SUNWbtool"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWbtool:11.10.0 passed
newtrade1 SUNWbtool:11.10.0 passed
Result: Package existence check passed for "SUNWbtool".

Check: Package existence for "SUNWhea"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWhea:11.10.0 passed
newtrade1 SUNWhea:11.10.0 passed
Result: Package existence check passed for "SUNWhea".

Check: Package existence for "SUNWlibm"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWlibm:5.10 passed
newtrade1 SUNWlibm:5.10 passed
Result: Package existence check passed for "SUNWlibm".

Check: Package existence for "SUNWlibms"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWlibms:5.10 passed
newtrade1 SUNWlibms:5.10 passed
Result: Package existence check passed for "SUNWlibms".

Check: Package existence for "SUNWsprot"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWsprot:5.10 passed
newtrade1 SUNWsprot:5.10 passed
Result: Package existence check passed for "SUNWsprot".

Check: Package existence for "SUNWtoo"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWtoo:11.10.0 passed
newtrade1 SUNWtoo:11.10.0 passed
Result: Package existence check passed for "SUNWtoo".

Check: Package existence for "SUNWi1of"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWi1of:6.6.2.7400 passed
newtrade1 SUNWi1of:6.6.2.7400 passed
Result: Package existence check passed for "SUNWi1of".

Check: Package existence for "SUNWi1cs"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 ERROR: information for "SUNWi1cs" was not found passed
newtrade1 SUNWi1cs:2.0 passed
Result: Package existence check passed for "SUNWi1cs".

Check: Package existence for "SUNWi15cs"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 ERROR: information for "SUNWi15cs" was not found passed
newtrade1 SUNWi15cs:2.0 passed
Result: Package existence check passed for "SUNWi15cs".

Check: Package existence for "SUNWxwfnt"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWxwfnt:6.6.2.7400 passed
newtrade1 SUNWxwfnt:6.6.2.7400 passed
Result: Package existence check passed for "SUNWxwfnt".

Check: Package existence for "SUNWlibC"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
newtrade2 SUNWlibC:5.10 passed
newtrade1 SUNWlibC:5.10 passed
Result: Package existence check passed for "SUNWlibC".

Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
newtrade2 exists passed
newtrade1 exists passed
Result: Group existence check passed for "dba".

Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
newtrade2 exists passed
newtrade1 exists passed
Result: Group existence check passed for "oinstall".

Check: User existence for "nobody"
Node Name Status Comment
------------ ------------------------ ------------------------
newtrade2 exists passed
newtrade1 exists passed
Result: User existence check passed for "nobody".

System requirement passed for 'crs'

Pre-check for cluster services setup was successful.

可以看到，除了节点2上个别系统包没有完全安装之外，所有的检查已经顺利通过。由于安装是在节点1上进行，所以这个问题可以忽略。

利用VOLUMN CLUSTER MANAGER划分共享存储，建立CLUSTER安装所需的VOTE和OCR裸设备：

root@newtrade1 # vxassist -g datavg make ocr 300m user=oracle group=oinstall mode=660
root@newtrade1 # vxassist -g datavg make vot 300m user=oracle group=oinstall mode=660

下面就可以安装CLUSTER：

root@newtrade1 # xhost +
access control disabled, clients can connect from any host
root@newtrade1 # su - oracle
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
$ bash
bash-3.00$ cd /data/clusterware/
bash-3.00$ ./runInstaller

需要注意几点，在安装CLUSTER的时候Oracle默认选择路径/data/oracle/product/11.1/database，这里将其改为/data/oracle/product/11.1/crs。在添加节点后，Oracle认为172.0.2.0和10.0.1.0两个网段都是PUBLIC的候选，这里需要手工编辑将其改为PRIVATE。

由于是测试环境，选择外部冗余，在OCR处输入前面创建的：/dev/vx/rdsk/datavg/ocr。

在VOT磁盘处同样选择外部冗余，输入：/dev/vx/rdsk/datavg/vot。

最后安装成功后，需要分别在节点1、节点2执行两个shell脚本：

root@newtrade1 # . /data/oracle/oraInventory/orainstRoot.sh
Changing permissions of /data/oracle/oraInventory to 770.
Changing groupname of /data/oracle/oraInventory to oinstall.
The execution of the script is complete

节点1上执行第二个脚本：

root@newtrade1 # . /data/oracle/product/11.1/crs/root.sh
WARNING: directory '/data/oracle/product/11.1' is not owned by root
WARNING: directory '/data/oracle/product' is not owned by root
WARNING: directory '/data/oracle' is not owned by root
WARNING: directory '/data' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/data/oracle/product/11.1' is not owned by root. Changing owner to root
The directory '/data/oracle/product' is not owned by root. Changing owner to root
The directory '/data/oracle' is not owned by root. Changing owner to root
The directory '/data' is not owned by root. Changing owner to root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: newtrade1 newtrade1-priv newtrade1
node 2: newtrade2 newtrade2-priv newtrade2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/vx/rdsk/datavg/vot
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
newtrade1
Cluster Synchronization Services is inactive on these nodes.
newtrade2
Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.

节点2上执行第二个SQL：

root@newtrade2 # . /data/oracle/product/11.1/crs/root.sh
WARNING: directory '/data/oracle/product/11.1' is not owned by root
WARNING: directory '/data/oracle/product' is not owned by root
WARNING: directory '/data/oracle' is not owned by root
WARNING: directory '/data' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/data/oracle/product/11.1' is not owned by root. Changing owner to root
The directory '/data/oracle/product' is not owned by root. Changing owner to root
The directory '/data/oracle' is not owned by root. Changing owner to root
The directory '/data' is not owned by root. Changing owner to root
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: newtrade1 newtrade1-priv newtrade1
node 2: newtrade2 newtrade2-priv newtrade2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
newtrade1
newtrade2
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps

Creating VIP application resource on (2) nodes...
Creating GSD application resource on (2) nodes...
Creating ONS application resource on (2) nodes...
Starting VIP application resource on (2) nodes...
Starting GSD application resource on (2) nodes...
Starting ONS application resource on (2) nodes...

Done.

最后Oracle会自动设置3个和CLUSTER相关的工具，至此CLUSTER安装完成。