Storm-1.0.1+ZooKeeper-3.4.8+Netty-4.1.3 HA集群安装
下载Storm-1.0.1
http://mirrors.tuna.tsinghua.edu.cn/apache/storm/apache-storm-1.0.1/apache-storm-1.0.1.tar.gz
下载ZooKeeper-3.4.8
http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
下载Netty-4.1.3.Final.tar.bz2
http://akamai.bintray.com/e7/e769383d99e244b7173b496f90f9b4a00574f8a33cdec2fe40b54f957a373e01?__gda__=exp=1469008981~hmac=7f3864e58af529209d0c0460cc3d5a73f267d7b550ad82bf14c0f645f8608e74&response-content-disposition=attachment%3Bfilename%3D%22netty-4.1.3.Final.tar.bz2%22&response-content-type=application%2Fx-bzip2
软件包如下：
apache-storm-1.0.1.tar.gz
jdk-7u65-linux-x64.tar.gz
netty-4.1.3.Final.tar.bz2
zookeeper-3.4.8.tar.gz
集群规划
主机 IP Soft Process
nimbus01 192.168.14.162 Netty、Storm nimbus、core、logviewer
nimbus02 192.168.14.156 Netty、Storm nimbus、logviewer
supervisor01 192.168.14.59 Netty、ZooKeeper、Storm supervisor、logviewer
supervisor02 192.168.14.119 Netty、ZooKeeper、Storm supervisor、logviewer
supervisor03 192.168.14.139 Netty、ZooKeeper、Storm supervisor、logviewer
一、配置免密码登录
二、安装Jdk
在所有节点上安装jdk
cd /usr/local/soft/
tar -zxvf jdk-7u65-linux-x64.tar.gz
cd jdk1.7.0_65
vim /etc/profile
export JAVA_HOME=/usr/local/soft/jdk1.7.0_65
export PATH=.:$JAVA_HOME/bin:$PATH
source /etc/profile
三、安装ZooKeeper-3.4.8
在supervisor01节点上配置
cd /usr/local/soft/
tar -zxvf zookeeper-3.4.8.tar.gz
cd zookeeper-3.4.8
mv conf/zoo_sample.cfg conf/zoo.cfg
vim conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/soft/zookeeper-3.4.8/data
clientPort=2181
server.1=suspervisor01:2888:3888
server.2=suspervisor02:2888:3888
server.3=suspervisor03:2888:3888
mkdir data && echo "1">data/myid
复制supervisor01节点上的ZooKeeper到supervisor02上
scp -r /usr/local/soft/zookeeper-3.4.8 [email protected]:/usr/local/soft/
echo "2" >/usr/local/soft/zookeeper-3.4.8/data/myid
复制supervisor01节点上的ZooKeeper到supervisor03上
scp -r /usr/local/soft/zookeeper-3.4.8 [email protected]:/usr/local/soft/
echo "3" >/usr/local/soft/zookeeper-3.4.8/data/myid
四、安装Storm-1.0.1
在nimbus01节点上配置
cd /usr/local/soft/
tar -zxvf apache-storm-1.0.1.tar.gz
cd apache-storm-1.0.1
vim conf/storm.yaml
# #### Storm dependents The ZooKeeper Cluster ####
storm.zookeeper.servers:
- "supervisor01"
- "supervisor02"
- "supervisor03"
# #### Storm Nimbus Nodes HA ####
nimbus.seeds: ["nimbus01", "nimbus02"]
# #### Storm Local Storage ####
storm.local.dir: "/usr/local/soft/apache-storm-1.0.1/localdir"
# #### Storm Supervisor Nodes Worker Process ####
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
# #### Not Selector ZeroMQ, Storm Use Netty Transport Protocol ####
storm.messaging.transport: "org.apache.storm.messaging.netty.Context"
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880
storm.messaging.netty.max_retries: 300
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100
mkdir localdir
复制nimbus01节点上的Storm到nimbus02、supervisor01、supervisor02、supervisor03上
scp -r /usr/local/soft/apache-storm-1.0.1 [email protected]:/usr/local/soft/
scp -r /usr/local/soft/apache-storm-1.0.1 [email protected]:/usr/local/soft/
scp -r /usr/local/soft/apache-storm-1.0.1 [email protected]:/usr/local/soft/
scp -r /usr/local/soft/apache-storm-1.0.1 [email protected]:/usr/local/soft/
五、启动集群
1、ZooKeeper
启动集群
for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; /usr/local/soft/zookeeper-3.4.8/bin/zkServer.sh start"; done
关闭集群
for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; /usr/local/soft/zookeeper-3.4.8/bin/zkServer.sh stop"; done
查看ZooKeeper进程状态
for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; /usr/local/soft/zookeeper-3.4.8/bin/zkServer.sh status"; done
查看进程
for i in supervisor01 supervisor02 supervisor03; do echo $i; ssh $i "source /etc/profile; jps"; done
2、启动Storm（需要确保ZooKeeper集群已经启动）
启动nimbus（storm.yaml中nimbus.seeds: []的节点都需要启动nimbus进程）
在nimbus01上启动如下进程
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm nimbus &
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm ui &
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &
在nimbus02上启动如下进程
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm nimbus &
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &
在supervisor01上启动如下进程
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm supervisor &
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &
在supervisor02上启动如下进程
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm supervisor &
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &
在supervisor03上启动如下进程
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm supervisor &
nohup /usr/local/soft/apache-storm-1.0.1/bin/storm logviewer &
时间： 2025-01-08 15:25:54