Oracle 11gR2 RAC 添加节点 / 憋错料

1. 概述

生产，测试数据库添加节点。

2. 安装前准备

1、首先，物理链路的准备。这过程包括对db3进行存储映射、心跳互联等物理环境的准备；

2、根据db1、db2的操作系统配置，安装、配置db3的操作系统；注意此处需要配置的操作系统内容较多。大致包括确认RAC需要的系统安装包、系统核心参数配置、ASMLIB的配置、/etc/hosts配置等等。详细可参考官方的安装指导手册。

3、根据db1、db2的操作系统组、用户的信息，在db3上创建相应的组、用户；创建对于的目录信息；注意：创建的组、用户，其ID要与db1、db2上的一致！

4、确保每个节点均配置ssh互信；

5、采用CVU，验证db3与db1、db2的连通性等。注意：以下操作指令在db1或者db2上执行：

2.1 创建用户和组

#/usr/sbin/groupadd -g 501 oinstall

#/usr/sbin/groupadd -g 502 dba

#/usr/sbin/groupadd -g 504 asmadmin

#/usr/sbin/groupadd -g 506 asmdba

#/usr/sbin/groupadd -g 507 asmoper

#/usr/sbin/useradd -u 501 -g oinstall -G asmadmin,asmdba,asmoper grid

#/usr/sbin/useradd -u 502 -g oinstall -G dba,asmdba oracle

Changing password for user oracle.

New UNIX password: password

retype new UNIX password: password

passwd: all authentication tokens updated successfully.

passwd grid

Changing password for user oracle.

New UNIX password: password

retype new UNIX password: password

passwd: all authentication tokens updated successfully.

备注：Oracle，grid 用户创建完成后，对照其他节点检查是否一样的。

2.2 配置时间服务器

节点1编辑ntp服务配置时间服务

service xinetd start

service ntpd start

chkconfig time on

chkconfig ntpd on

chkconfig xinetd on

其他节点操作，建立一个每分钟的同步

crontab -e

0-59/1 * * * * /usr/bin/rdate -s 192.168.8.177 >/dev/null 2>&1

0-59/1 * * * * /usr/sbin/ntpdate 192.168.8.177 >/dev/null 2>&1

2.3 网络配置

vi /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1 localhost

--host ip

192.168.150.33 node1-11gr2

192.168.150.34 node2-11gr2

192.168.150.35 node3-11gr2

192.168.150.36 node4-11gr2

--host vip

192.168.150.133 node1-11gr2-vip

192.168.150.134 node2-11gr2-vip

192.168.150.136 node3-11gr2-vip

192.168.150.137 node4-11gr2-vip

--host priv ip

10.1.1.10 node1-11gr2-priv

10.1.1.11 node2-11gr2-priv

10.1.1.12 node3-11gr2-priv

10.1.1.13 node4-11gr2-priv

--cluster vip

192.168.150.135 scan-cluster

备注：根据其他节点参照配置

2.4 修改内核参数

vi /etc/sysctl.conf

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 512 x processes (for example 6815744 for 13312 processes)

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

#/sbin/sysctl -p

备注：根据其他节点修改

2.5 Oracle用户配置

vi /etc/security/limits.conf

grid soft nproc 2047

grid hard nproc 16384

grid soft nofile 1024

grid hard nofile 65536

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

vi /etc/pam.d/login

session required pam_limits.so

vi /etc/profile

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then

if [ $SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

umask 022

2.6 创建文件目录

--Create the Oracle Inventory Directory

To create the Oracle Inventory directory, enter the following commands as the root user:

# mkdir -p /u01/app/oraInventory

# chown -R grid:oinstall /u01/app/oraInventory

# chmod -R 775 /u01/app/oraInventory

--Creating the Oracle Grid Infrastructure Home Directory

To create the Grid Infrastructure home directory, enter the following commands as the root user:

# mkdir -p /u01/11.2.0/grid

# chown -R grid:oinstall /u01/11.2.0/grid

# chmod -R 775 /u01/11.2.0/grid

--Creating the Oracle Base Directory

To create the Oracle Base directory, enter the following commands as the root user:

# mkdir -p /u01/app/oracle

# mkdir /u01/app/oracle/cfgtoollogs

# chown -R oracle:oinstall /u01/app/oracle

# chmod -R 775 /u01/app/oracle

--Creating the Oracle RDBMS Home Directory

To create the Oracle RDBMS Home directory, enter the following commands as the root user:

# mkdir -p /u01/app/oracle/product/11.2.0/db_1

# chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/db_1

# chmod -R 775 /u01/app/oracle/product/11.2.0/db_1

2.7 用户环境配置

--grid 用户

export EDITOR=vi

export ORACLE_SID=+ASM1

export ORACLE_BASE=/u01/app/oracle

export ORACLE_HOME=/u01/11.2.0/grid

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

export PATH=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin

export NLS_LANG="AMERICAN_AMERICA.ZHS16GBK"

umask 022

stty erase ^h

ulimit -s 32768

ulimit -n 65536

--Oracle用户

export EDITOR=vi

export ORACLE_SID=prod1

export ORACLE_BASE=/u01/app/oracle

export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

export PATH=$ORACLE_HOME/bin:/u01/11.2.0/grid/bin:/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin:/u01/11.2.0/grid/bin

export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK

umask 022

stty erase ^h

ulimit -s 32768

ulimit -n 65536

2.8 互信配置

--新加入节点node2 grid,oracle 用户配置

su - grid

$ mkdir ~/.ssh

$ chmod 700 ~/.ssh

$ /usr/bin/ssh-keygen -t rsa

$ /usr/bin/ssh-keygen -t dsa

$ touch ~/.ssh/authorized_keys

$ cd ~/.ssh

$ ls

su - oracle

$ mkdir ~/.ssh

$ chmod 700 ~/.ssh

$ /usr/bin/ssh-keygen -t rsa

$ /usr/bin/ssh-keygen -t dsa

$ touch ~/.ssh/authorized_keys

$ cd ~/.ssh

$ ls

--node2,oracle 用户(其中一个节点执行)

[[email protected] .ssh]$cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[[email protected] .ssh]$cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[[email protected] .ssh$ ssh node1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[email protected] password:

[[email protected] .ssh$ ssh node1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[email protected] password:

--node1,oracle 远程拷贝

[[email protected] .ssh]scp ~/.ssh/authorized_keys node1:~/.ssh/authorized_keys

--node1,node2

$ chmod 600 ~/.ssh/authorized_keys

ssh node1-11gr2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

ssh node3-11gr2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

ssh node4-11gr2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

ssh node1-11gr2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

ssh node3-11gr2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

ssh node4-11gr2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys node1-11gr2:~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys node3-11gr2:~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys node4-11gr2:~/.ssh/authorized_keys

--测试

ssh rac1 date

ssh rac2 date

ssh rac1-priv date

ssh rac2-priv date

ssh scan-cluster date

--执行脚本测试

ssh node1-11gr2-priv date;ssh node2-11gr2-priv date;ssh node3-11gr2-priv date;ssh node4-11gr2-priv date;ssh scan-cluster date

ssh node1-11gr2 date;ssh node2-11gr2 date;ssh node3-11gr2 date;ssh node4-11gr2 date;

2.9 OracleRPM包检查

32位系统rpm 包

libXp-1.0.0-8.i386.rpm

openmotif22-2.2.3-18.i386.rpm

compat-db-4.2.52-5.1.i386.rpm

compat-db42.i686 0:4.2.52-15.el6

compat-db43.i686 0:4.3.29-15.el6

kernel-headers-2.6.18-92.el5.i386.rpm

glibc-headers-2.5-24.i386.rpm

compat-gcc-34-3.4.6-4.i386.rpm

compat-gcc-34-c++-3.4.6-4.i386.rpm

compat-libstdc++-33-3.2.3-61.i386.rpm

libaio-0.3.106-3.2.i386.rpm

libgomp-4.1.2-42.el5.i386.rpm

gcc-4.1.2-42.el5.i386.rpm

binutils-2.15.92.0.2-22

64位系统 rpm 包

libXp-1.0.0-8.i386.rpm
openmotif22-2.2.3-18.i386.rpm
binutils-2.15.92.0.2-22 (x86_64)
compat-db-4.1.25-9 (i386)
compat-db-4.1.25-9 (x86_64)
compat-libstdc++-33-3.2.3-47.3.i386.rpm
control-center-2.8.0-12.rhel4.5 (x86_64)
kernel-headers-2.6.18-194.el5.x86_64.rpm
glibc-headers-2.5-49.x86_64.rpm
glibc-common-2.3.4-2.36 (x86_64)
glibc-devel-2.3.4-2.36 (x86_64)
glibc-devel-2.3.4-2.36 (i386)
glibc-2.3.4-2.36 (i686)
glibc-2.3.4-2.36 (x86_64)
libstdc++-3.4.6-8 (i386)
libstdc++-3.4.6-8 (x86_64)
libstdc++-devel-4.1.2-48.el5.x86_64.rpm
make-3.81-3.el5.x86_64.rpm
pdksh-5.2.14-36.el5.x86_64.rpm
sysstat-7.0.2-3.el5.x86_64.rpm
libaio-0.3.105-2 (i386)
libgomp-4.1.2-42.el5.i386.rpm
gcc-4.1.2-48.el5 (x86_64)
gcc-c++-4.1.2-48.el5 (x86_64)
elfutils-libelf-devel-0.137-3.el5.x86_64
elfutils-libelf-devel-0.137-3.el5.i386
libaio-devel-0.3.106-5.i386
libaio-devel-0.3.106-5.x86_64
unixODBC.i386 0:2.2.11-10.el5
unixODBC-devel

yum安装

yum install binutils -y

yum install compat-libcap1 -y

yum install compat-libstdc++-33 -y

yum install compat-libstdc++-33.i686 -y

yum install gcc -y

yum install gcc-c++ -y

yum install glibc -y

yum install glibc.i686 -y

yum install glibc-devel -y

yum install glibc-devel.i686 -y

yum install ksh -y

yum install libgcc -y

yum install libgcc.i686 -y

yum install libstdc++ -y

yum install libstdc++.i686 -y

yum install libstdc++-devel -y

yum install libstdc++-devel.i686 -y

yum install libaio -y

yum install libaio.i686 -y

yum install libaio-devel -y

yum install libaio-devel.i686 -y

yum install libXext -y

yum install libXext.i686 -y

yum install libXtst -y

yum install libXtst.i686 -y

yum install libX11 -y

yum install libX11.i686 -y

yum install libXau -y

yum install libXau.i686 -y

yum install libxcb -y

yum install libxcb.i686 -y

yum install libXi -y

yum install libXi.i686 -y

yum install make -y

yum install sysstat -y

yum install unixODBC -y

yum install unixODBC-devel -y

安装脚本检查

rpm -q --qf ‘%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n‘ binutils \

binutils \

compat-libcap1 \

compat-libstdc++-33 \

gcc \

gcc-c++ \

glibc \

glibc-common \

glibc-devel \

glibc-headers \

ksh \

libgcc \

libaio \

libaio-devel \

libstdc++ \

libstdc++-devel \

libXext \

libXtst \

libX11 \

libXau \

libxcb \

libXi \

make \

sysstat \

elfutils-libelf \

elfutils-libelf-devel \

unixODBC \

unixODBC-devel

--本地yun源配置

虚拟机挂着 ISO光盘

mount /dev/cdrom /mnt

修改yum源文件

[[email protected] yum.repos.d]# cat my.repo

[c6-media]

name=CentOS-$releasever - Media

baseurl=file:///mnt/Server

gpgcheck=0

enabled=1

2.10 UDEV共享磁盘配置

[[email protected] rules.d]# cat 99-oracle-asmdevices.rules

ACTION=="add", KERNEL=="/dev/sdb1",RUN+="/bin/raw /dev/raw/raw1 %N"

ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="17",RUN+="/bin/raw /dev/raw/raw1 %M %m"

ACTION=="add", KERNEL=="/dev/sdc1",RUN+="/bin/raw /dev/raw/raw2 %N"

ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="33",RUN+="/bin/raw /dev/raw/raw2 %M %m"

ACTION=="add", KERNEL=="/dev/sdd1",RUN+="/bin/raw /dev/raw/raw3 %N"

ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="49",RUN+="/bin/raw /dev/raw/raw3 %M %m"

ACTION=="add", KERNEL=="/dev/sde1",RUN+="/bin/raw /dev/raw/raw4 %N"

ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="65",RUN+="/bin/raw /dev/raw/raw4 %M %m"

ACTION=="add", KERNEL=="/dev/sdf1",RUN+="/bin/raw /dev/raw/raw5 %N"

ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="81",RUN+="/bin/raw /dev/raw/raw5 %M %m"

KERNEL=="raw[1-5]", OWNER="grid", GROUP="asmadmin", MODE="660"

在所有节点上启动udev服务，或者重启服务器即可

[[email protected] rules.d]# /sbin/udevcontrol reload_rules

[[email protected] rules.d]# /sbin/start_udev

Starting udev: [ OK ]

检查设备是否到位

[[email protected] rules.d]# raw -qa

/dev/raw/raw1: bound to major 8, minor 17

/dev/raw/raw2: bound to major 8, minor 33

/dev/raw/raw3: bound to major 8, minor 49

/dev/raw/raw4: bound to major 8, minor 65

/dev/raw/raw5: bound to major 8, minor 81

[[email protected] raw]# ls -lhtr

total 0

crw-rw---- 1 grid asmadmin 162, 1 Feb 16 14:07 raw1

crw-rw---- 1 grid asmadmin 162, 3 Feb 16 14:07 raw3

crw-rw---- 1 grid asmadmin 162, 4 Feb 16 14:07 raw4

crw-rw---- 1 grid asmadmin 162, 2 Feb 16 14:07 raw2

crw-rw---- 1 grid asmadmin 162, 5 Feb 16 14:07 raw5

[[email protected] rules.d]# cd /dev

[[email protected] dev]# ls -l ocr*

brw-rw---- 1 grid asmadmin 8, 32 Jul 10 17:31 ocr1

brw-rw---- 1 grid asmadmin 8, 48 Jul 10 17:31 ocr2

[[email protected] dev]# ls -l asm-disk*

brw-rw---- 1 grid asmadmin 8, 64 Jul 10 17:31 asm-disk1

brw-rw---- 1 grid asmadmin 8, 208 Jul 10 17:31 asm-disk10

brw-rw---- 1 grid asmadmin 8, 224 Jul 10 17:31 asm-disk11

brw-rw---- 1 grid asmadmin 8, 240 Jul 10 17:31 asm-disk12

brw-rw---- 1 grid asmadmin 8, 80 Jul 10 17:31 asm-disk2

brw-rw---- 1 grid asmadmin 8, 96 Jul 10 17:31 asm-disk3

brw-rw---- 1 grid asmadmin 8, 112 Jul 10 17:31 asm-disk4

brw-rw---- 1 grid asmadmin 8, 128 Jul 10 17:31 asm-disk5

brw-rw---- 1 grid asmadmin 8, 144 Jul 10 17:31 asm-disk6

brw-rw---- 1 grid asmadmin 8, 160 Jul 10 17:31 asm-disk7

brw-rw---- 1 grid asmadmin 8, 176 Jul 10 17:31 asm-disk8

brw-rw---- 1 grid asmadmin 8, 192 Jul 10 17:31 asm-disk9

2.11 安装检查

--node1 grid 用户执行每个节点执行

[[email protected] ~]$ cluvfy stage -post hwos -n node2-11gr2 -verbose

--备注 node3 需要安装以下工具

[[email protected] grid]# rpm -ivh cvuqdisk-1.0.9-1.rpm

Preparing... ########################################### [100%]

Using default group oinstall to install package

1:cvuqdisk ########################################### [100%]

当然，还可以通过以下命令将db3与db1、db2的系统配置进行比较。一致的，显示为“matched”、不一致的显示为“mismatched”

--安装检查脚本

cluvfy comp peer -refnode node1-11gr2 -n node2-11gr2 -orainv oinstall -osdba asmdba -verbose

cluvfy stage -pre nodeadd -n node2-11gr2 -fixup -verbose

cluvfy stage -post hwos -n node2-11gr2 -verbose

检查执行结果，以节点1 检查结果为例

[[email protected] ~]$ cluvfy stage -post hwos -n node2-11gr2 -verbose

Performing post-checks for hardware and operating system setup

Checking node reachability...

Check: Node reachability from node "node1-11gr2"

Destination Node Reachable?

------------------------------------ ------------------------

node2-11gr2 yes

Result: Node reachability check passed from node "node1-11gr2"

Checking user equivalence...

Check: User equivalence for user "grid"

Node Name Status

------------------------------------ ------------------------

node2-11gr2 passed

Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...

Node Name Status

------------------------------------ ------------------------

node2-11gr2 passed

Verification of the hosts config file successful

Interface information for node "node2-11gr2"

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

------ --------------- --------------- --------------- --------------- ----------------- ------

eth0 192.168.150.34 192.168.150.0 0.0.0.0 192.168.150.254 00:0C:29:05:36:7F 1500

eth1 10.1.1.11 10.0.0.0 0.0.0.0 192.168.150.254 00:0C:29:05:36:89 1500

Check: Node connectivity for interface "eth0"

Result: Node connectivity passed for interface "eth0"

Check: TCP connectivity of subnet "192.168.150.0"

Source Destination Connected?

------------------------------ ------------------------------ ----------------

node1-11gr2:192.168.150.33 node2-11gr2:192.168.150.34 passed

Result: TCP connectivity check passed for subnet "192.168.150.0"

Check: Node connectivity for interface "eth1"

Result: Node connectivity passed for interface "eth1"

Check: TCP connectivity of subnet "10.0.0.0"

Source Destination Connected?

------------------------------ ------------------------------ ----------------

node1-11gr2:192.168.150.33 node2-11gr2:10.1.1.11 passed

Result: TCP connectivity check passed for subnet "10.0.0.0"

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.150.0" for multicast communication with multicast group "230.0.1.0"...

Check of subnet "192.168.150.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0"...

Check of subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking for multiple users with UID value 0

Result: Check for multiple users with UID value 0 passed

Check: Time zone consistency

Result: Time zone consistency check passed

Checking shared storage accessibility...

Disk Sharing Nodes (1 in count)

------------------------------------ ------------------------

/dev/sda node2-11gr2

/dev/sdb node2-11gr2

/dev/sdc node2-11gr2

/dev/sde node2-11gr2

/dev/sdd node2-11gr2

/dev/sdf node2-11gr2

/dev/sdg node2-11gr2

/dev/sdh node2-11gr2

Shared storage check was successful on nodes "node2-11gr2"

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...

Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...

Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined

More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file

Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed

Post-check for hardware and operating system setup was successful.

[[email protected] ~]$ cluvfy comp peer -refnode node1-11gr2 -n node2-11gr2 -orainv oinstall -osdba asmdba -verbose

Verifying peer compatibility

Checking peer compatibility...

Compatibility check: Physical memory [reference node: node1-11gr2]