storm部署实践

1.下载storm并解压

2.修改文件 storm.yaml

我这里填的是域名，填ip也可以

[email protected]:~/storm-0.9.2/conf$ vim storm.yaml

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

########### These MUST be filled in for a storm configuration

# storm.zookeeper.servers:

#     - "server1"

#     - "server2"

#

# nimbus.host: "nimbus"

#

#

# ##### These may optionally be filled in:

"storm.yaml" 48L, 1613C                                                                                    1,1           Top

nimbus.host: "namenode

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

########### These MUST be filled in for a storm configuration

storm.zookeeper.servers:

- "namenode"

- "datanode1"

- "datanode1"

#

nimbus.host: "namenode"

storm.local.dir: "/home/hadoop/storm-0.9.2/data"

supervisor.slots.ports:

- 6700

- 6701

- 6702

- 6703

#

#

# ##### These may optionally be filled in:

#

## List of custom serializations

# topology.kryo.register:

#     - org.mycompany.MyType

#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer

#

## List of custom kryo decorators

# topology.kryo.decorators:

#     - org.mycompany.MyDecorator

#

## Locations of the drpc servers

# drpc.servers:

#     - "server1"

3.创建临时文件目录

[email protected]:~/storm-0.9.2$ mkdir data
[email protected]:~/storm-0.9.2$ ls
bin  CHANGELOG.md  conf  data  DISCLAIMER  examples  external  lib  LICENSE  logback  NOTICE  public  README.markdown  RELEASE  SECURITY.md                   

4.将storm复制到各个节点

[email protected]:~$ scp -r storm-0.9.2/ [email protected]:~/

[email protected]:~$ scp -r storm-0.9.2/ [email protected]:~/

5.配置stom环境变量

export STORM_HOME=/home/hadoop/storm-0.9.2

export PATH=$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$MAHOUT_HOME/bin:$HBASE_HOME/bin:$STORM_HOME/bin:$PATH

6.开启storm

[email protected]:~$ storm nimbus &
[1] 8341

[email protected]:~$ Running: /usr/programs/jdk1.7.0_65/bin/java -server -Dstorm.options= -Dstorm.home=/home/hadoop/storm-0.9.2 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/hadoop/storm-0.9.2/lib/commons-io-2.4.jar:/home/hadoop/storm-0.9.2/lib/chill-java-0.3.5.jar:/home/hadoop/storm-0.9.2/lib/commons-lang-2.5.jar:/home/hadoop/storm-0.9.2/lib/kryo-2.21.jar:/home/hadoop/storm-0.9.2/lib/hiccup-0.3.6.jar:/home/hadoop/storm-0.9.2/lib/reflectasm-1.07-shaded.jar:/home/hadoop/storm-0.9.2/lib/zookeeper-3.4.5.jar:/home/hadoop/storm-0.9.2/lib/clojure-1.5.1.jar:/home/hadoop/storm-0.9.2/lib/json-simple-1.1.jar:/home/hadoop/storm-0.9.2/lib/ring-devel-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/math.numeric-tower-0.0.1.jar:/home/hadoop/storm-0.9.2/lib/disruptor-2.10.1.jar:/home/hadoop/storm-0.9.2/lib/commons-logging-1.1.3.jar:/home/hadoop/storm-0.9.2/lib/carbonite-1.4.0.jar:/home/hadoop/storm-0.9.2/lib/netty-3.6.3.Final.jar:/home/hadoop/storm-0.9.2/lib/minlog-1.2.jar:/home/hadoop/storm-0.9.2/lib/jline-2.11.jar:/home/hadoop/storm-0.9.2/lib/clj-stacktrace-0.2.4.jar:/home/hadoop/storm-0.9.2/lib/httpclient-4.3.3.jar:/home/hadoop/storm-0.9.2/lib/netty-3.2.2.Final.jar:/home/hadoop/storm-0.9.2/lib/logback-core-1.0.6.jar:/home/hadoop/storm-0.9.2/lib/jetty-6.1.26.jar:/home/hadoop/storm-0.9.2/lib/commons-exec-1.1.jar:/home/hadoop/storm-0.9.2/lib/compojure-1.1.3.jar:/home/hadoop/storm-0.9.2/lib/tools.macro-0.1.0.jar:/home/hadoop/storm-0.9.2/lib/servlet-api-2.5-20081211.jar:/home/hadoop/storm-0.9.2/lib/tools.logging-0.2.3.jar:/home/hadoop/storm-0.9.2/lib/httpcore-4.3.2.jar:/home/hadoop/storm-0.9.2/lib/guava-13.0.jar:/home/hadoop/storm-0.9.2/lib/jetty-util-6.1.26.jar:/home/hadoop/storm-0.9.2/lib/commons-fileupload-1.2.1.jar:/home/hadoop/storm-0.9.2/lib/curator-client-2.4.0.jar:/home/hadoop/storm-0.9.2/lib/curator-framework-2.4.0.jar:/home/hadoop/storm-0.9.2/lib/ring-servlet-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/joda-time-2.0.jar:/home/hadoop/storm-0.9.2/lib/clout-1.0.1.jar:/home/hadoop/storm-0.9.2/lib/slf4j-api-1.6.5.jar:/home/hadoop/storm-0.9.2/lib/commons-codec-1.6.jar:/home/hadoop/storm-0.9.2/lib/servlet-api-2.5.jar:/home/hadoop/storm-0.9.2/lib/logback-classic-1.0.6.jar:/home/hadoop/storm-0.9.2/lib/core.incubator-0.1.0.jar:/home/hadoop/storm-0.9.2/lib/tools.cli-0.2.4.jar:/home/hadoop/storm-0.9.2/lib/ring-core-1.1.5.jar:/home/hadoop/storm-0.9.2/lib/ring-jetty-adapter-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/jgrapht-core-0.9.0.jar:/home/hadoop/storm-0.9.2/lib/clj-time-0.4.1.jar:/home/hadoop/storm-0.9.2/lib/snakeyaml-1.11.jar:/home/hadoop/storm-0.9.2/lib/storm-core-0.9.2-incubating.jar:/home/hadoop/storm-0.9.2/lib/log4j-over-slf4j-1.6.6.jar:/home/hadoop/storm-0.9.2/lib/objenesis-1.2.jar:/home/hadoop/storm-0.9.2/lib/asm-4.0.jar:/home/hadoop/storm-0.9.2/conf
-Xmx1024m -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/home/hadoop/storm-0.9.2/logback/cluster.xml backtype.storm.daemon.nimbus

[email protected]:~$ storm ui &
[2] 8391

[email protected]:~$ Running: /usr/programs/jdk1.7.0_65/bin/java -server -Dstorm.options= -Dstorm.home=/home/hadoop/storm-0.9.2 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/hadoop/storm-0.9.2/lib/commons-io-2.4.jar:/home/hadoop/storm-0.9.2/lib/chill-java-0.3.5.jar:/home/hadoop/storm-0.9.2/lib/commons-lang-2.5.jar:/home/hadoop/storm-0.9.2/lib/kryo-2.21.jar:/home/hadoop/storm-0.9.2/lib/hiccup-0.3.6.jar:/home/hadoop/storm-0.9.2/lib/reflectasm-1.07-shaded.jar:/home/hadoop/storm-0.9.2/lib/zookeeper-3.4.5.jar:/home/hadoop/storm-0.9.2/lib/clojure-1.5.1.jar:/home/hadoop/storm-0.9.2/lib/json-simple-1.1.jar:/home/hadoop/storm-0.9.2/lib/ring-devel-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/math.numeric-tower-0.0.1.jar:/home/hadoop/storm-0.9.2/lib/disruptor-2.10.1.jar:/home/hadoop/storm-0.9.2/lib/commons-logging-1.1.3.jar:/home/hadoop/storm-0.9.2/lib/carbonite-1.4.0.jar:/home/hadoop/storm-0.9.2/lib/netty-3.6.3.Final.jar:/home/hadoop/storm-0.9.2/lib/minlog-1.2.jar:/home/hadoop/storm-0.9.2/lib/jline-2.11.jar:/home/hadoop/storm-0.9.2/lib/clj-stacktrace-0.2.4.jar:/home/hadoop/storm-0.9.2/lib/httpclient-4.3.3.jar:/home/hadoop/storm-0.9.2/lib/netty-3.2.2.Final.jar:/home/hadoop/storm-0.9.2/lib/logback-core-1.0.6.jar:/home/hadoop/storm-0.9.2/lib/jetty-6.1.26.jar:/home/hadoop/storm-0.9.2/lib/commons-exec-1.1.jar:/home/hadoop/storm-0.9.2/lib/compojure-1.1.3.jar:/home/hadoop/storm-0.9.2/lib/tools.macro-0.1.0.jar:/home/hadoop/storm-0.9.2/lib/servlet-api-2.5-20081211.jar:/home/hadoop/storm-0.9.2/lib/tools.logging-0.2.3.jar:/home/hadoop/storm-0.9.2/lib/httpcore-4.3.2.jar:/home/hadoop/storm-0.9.2/lib/guava-13.0.jar:/home/hadoop/storm-0.9.2/lib/jetty-util-6.1.26.jar:/home/hadoop/storm-0.9.2/lib/commons-fileupload-1.2.1.jar:/home/hadoop/storm-0.9.2/lib/curator-client-2.4.0.jar:/home/hadoop/storm-0.9.2/lib/curator-framework-2.4.0.jar:/home/hadoop/storm-0.9.2/lib/ring-servlet-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/joda-time-2.0.jar:/home/hadoop/storm-0.9.2/lib/clout-1.0.1.jar:/home/hadoop/storm-0.9.2/lib/slf4j-api-1.6.5.jar:/home/hadoop/storm-0.9.2/lib/commons-codec-1.6.jar:/home/hadoop/storm-0.9.2/lib/servlet-api-2.5.jar:/home/hadoop/storm-0.9.2/lib/logback-classic-1.0.6.jar:/home/hadoop/storm-0.9.2/lib/core.incubator-0.1.0.jar:/home/hadoop/storm-0.9.2/lib/tools.cli-0.2.4.jar:/home/hadoop/storm-0.9.2/lib/ring-core-1.1.5.jar:/home/hadoop/storm-0.9.2/lib/ring-jetty-adapter-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/jgrapht-core-0.9.0.jar:/home/hadoop/storm-0.9.2/lib/clj-time-0.4.1.jar:/home/hadoop/storm-0.9.2/lib/snakeyaml-1.11.jar:/home/hadoop/storm-0.9.2/lib/storm-core-0.9.2-incubating.jar:/home/hadoop/storm-0.9.2/lib/log4j-over-slf4j-1.6.6.jar:/home/hadoop/storm-0.9.2/lib/objenesis-1.2.jar:/home/hadoop/storm-0.9.2/lib/asm-4.0.jar:/home/hadoop/storm-0.9.2:/home/hadoop/storm-0.9.2/conf
-Xmx768m -Dlogfile.name=ui.log -Dlogback.configurationFile=/home/hadoop/storm-0.9.2/logback/cluster.xml backtype.storm.ui.core

[email protected]:~$ storm logviewer &
[3] 8451

[email protected]:~$ Running: /usr/programs/jdk1.7.0_65/bin/java -server -Dstorm.options= -Dstorm.home=/home/hadoop/storm-0.9.2 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/hadoop/storm-0.9.2/lib/commons-io-2.4.jar:/home/hadoop/storm-0.9.2/lib/chill-java-0.3.5.jar:/home/hadoop/storm-0.9.2/lib/commons-lang-2.5.jar:/home/hadoop/storm-0.9.2/lib/kryo-2.21.jar:/home/hadoop/storm-0.9.2/lib/hiccup-0.3.6.jar:/home/hadoop/storm-0.9.2/lib/reflectasm-1.07-shaded.jar:/home/hadoop/storm-0.9.2/lib/zookeeper-3.4.5.jar:/home/hadoop/storm-0.9.2/lib/clojure-1.5.1.jar:/home/hadoop/storm-0.9.2/lib/json-simple-1.1.jar:/home/hadoop/storm-0.9.2/lib/ring-devel-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/math.numeric-tower-0.0.1.jar:/home/hadoop/storm-0.9.2/lib/disruptor-2.10.1.jar:/home/hadoop/storm-0.9.2/lib/commons-logging-1.1.3.jar:/home/hadoop/storm-0.9.2/lib/carbonite-1.4.0.jar:/home/hadoop/storm-0.9.2/lib/netty-3.6.3.Final.jar:/home/hadoop/storm-0.9.2/lib/minlog-1.2.jar:/home/hadoop/storm-0.9.2/lib/jline-2.11.jar:/home/hadoop/storm-0.9.2/lib/clj-stacktrace-0.2.4.jar:/home/hadoop/storm-0.9.2/lib/httpclient-4.3.3.jar:/home/hadoop/storm-0.9.2/lib/netty-3.2.2.Final.jar:/home/hadoop/storm-0.9.2/lib/logback-core-1.0.6.jar:/home/hadoop/storm-0.9.2/lib/jetty-6.1.26.jar:/home/hadoop/storm-0.9.2/lib/commons-exec-1.1.jar:/home/hadoop/storm-0.9.2/lib/compojure-1.1.3.jar:/home/hadoop/storm-0.9.2/lib/tools.macro-0.1.0.jar:/home/hadoop/storm-0.9.2/lib/servlet-api-2.5-20081211.jar:/home/hadoop/storm-0.9.2/lib/tools.logging-0.2.3.jar:/home/hadoop/storm-0.9.2/lib/httpcore-4.3.2.jar:/home/hadoop/storm-0.9.2/lib/guava-13.0.jar:/home/hadoop/storm-0.9.2/lib/jetty-util-6.1.26.jar:/home/hadoop/storm-0.9.2/lib/commons-fileupload-1.2.1.jar:/home/hadoop/storm-0.9.2/lib/curator-client-2.4.0.jar:/home/hadoop/storm-0.9.2/lib/curator-framework-2.4.0.jar:/home/hadoop/storm-0.9.2/lib/ring-servlet-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/joda-time-2.0.jar:/home/hadoop/storm-0.9.2/lib/clout-1.0.1.jar:/home/hadoop/storm-0.9.2/lib/slf4j-api-1.6.5.jar:/home/hadoop/storm-0.9.2/lib/commons-codec-1.6.jar:/home/hadoop/storm-0.9.2/lib/servlet-api-2.5.jar:/home/hadoop/storm-0.9.2/lib/logback-classic-1.0.6.jar:/home/hadoop/storm-0.9.2/lib/core.incubator-0.1.0.jar:/home/hadoop/storm-0.9.2/lib/tools.cli-0.2.4.jar:/home/hadoop/storm-0.9.2/lib/ring-core-1.1.5.jar:/home/hadoop/storm-0.9.2/lib/ring-jetty-adapter-0.3.11.jar:/home/hadoop/storm-0.9.2/lib/jgrapht-core-0.9.0.jar:/home/hadoop/storm-0.9.2/lib/clj-time-0.4.1.jar:/home/hadoop/storm-0.9.2/lib/snakeyaml-1.11.jar:/home/hadoop/storm-0.9.2/lib/storm-core-0.9.2-incubating.jar:/home/hadoop/storm-0.9.2/lib/log4j-over-slf4j-1.6.6.jar:/home/hadoop/storm-0.9.2/lib/objenesis-1.2.jar:/home/hadoop/storm-0.9.2/lib/asm-4.0.jar:/home/hadoop/storm-0.9.2:/home/hadoop/storm-0.9.2/conf
-Xmx128m -Dlogfile.name=logviewer.log -Dlogback.configurationFile=/home/hadoop/storm-0.9.2/logback/cluster.xml backtype.storm.daemon.logviewer

分别去各个节点开启supervisor

[email protected]:~$ ~/storm-0.9.2/bin/storm supervisor &

[email protected]:~$ ~/storm-0.9.2/bin/storm supervisor &

7.访问web界面

输入你的nimbus节点的ip:8080

比如我的是http://namenode:8080

时间： 2024-09-27 22:38:45

storm部署实践的相关文章

Hadoop部署实践:   离线安装 CDH5.1   (待完成)

配置主机映射关系 [[email protected] ~]$ cat /etc/hosts 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 10.15.5.200 master.hadoop 10.15.5.201 slave01.hadoop 10.15.5.202 slave02.hadoop 10.15.5.203 slave03.hadoop 如上有4台主机,每台hosts都已

Storm【实践系列-如何写一个爬虫- 对于Protocol进行的封装】

本章描述:对于Protocol的封装 package com.digitalpebble.storm.crawler.fetcher; import com.digitalpebble.storm.crawler.util.Configuration; public interface Protocol {     public ProtocolResponse getProtocolOutput(String url) throws Exception;          public voi

Docker在Ubuntu的部署实践

Docker在Ubuntu的部署实践 作者:chszs,版权所有,未经同意,不得转载.博主主页:http://blog.csdn.net/chszs 本文讲述Docker在Ubuntu系统上的部署过程.其中,Ubuntu为12.04.5 LTS, Precise Pangolin版. 1.安装Docker # apt-get update # apt-get install docker.io Reading package lists... Done Building dependency t

Storm【实践系列-如何写一个爬虫- Metric 系列】1

package com.digitalpebble.storm.crawler; import backtype.storm.Config; import backtype.storm.metric.MetricsConsumerBolt; import backtype.storm.metric.api.IMetricsConsumer; import backtype.storm.task.IErrorReporter; import backtype.storm.task.OutputCo

Storm【实践系列-如何写一个爬虫】 - ParserBolt

阅读背景: 如果您对爬虫,或则web前端不够了解,请自行google. 代码前提:您需要参阅本ID 所写的前面两篇博文:  Storm[实践系列-如何写一个爬虫] - Fetcher 本章主题: ParserBolt 如何完成的解析,并且如何从前面的组件得到数据,并emit出去. 博文流程:  博文将整个 爬虫系列公开,其过程为: 1 : 代码实现. 2 : 对代码的细节进行解析. 3 : 对真个设计进行回顾,并作总结. 如果您在参看本ID的博文的过程之中,只存在流程 1.那么请继续等待.一旦公

https部署实践 (Let's Encrypt)

1 .获取 Let's Encrypt git clone https://github.com/letsencrypt/letsencrypt cd letsencrypt chmod +x letsencrypt-auto 2 .执行安装证书 ./letsencrypt-auto certonly -a webroot --webroot-path=/home/www/demo.com --email [email protected] -d demo.com -d www.demo.com

Citrix XenApp&XenDesktop 7.15 部署实践指南——第二节·环境介绍

参考之前的的XenServer或vSphere手册配置Hypervisor,并完成虚拟机操作系统模板的创建,然后按照下表创建虚拟机,修改计算机名,配置IP地址,加域:具体创建过程不再此手册中体现:XenServer部分相关内容参考itdali.cn . 所有Windows.Windows Server都强烈建议更新至最新的补丁,这样可避免在Citrix环境下很多问题的发生:所有Windows.Windows Server在制作模板时也都建议安装.net framework 3.5..net fr

Spring Boot Tomcat 容器化部署实践与总结

在平时的工作和学习中经常会构建简单的web应用程序.如果只是HelloWorld级别的程序,使用传统的Spring+SpringMVC框架搭建得话会将大部分的时间花费在搭建框架本身上面,比如引入SpringMVC,配置DispatcheherServlet等.并且这些配置文件都差不多,重复这些劳动似乎意义不大.所以使用Springboot框架来搭建简单的应用程序显得十分的便捷和高效. 前两天在工作中需要一个用于测试文件下载的简单web程序,条件是使用Tomcat Docker Image作为载体

ASP.NET Core在CentOS上的最小化部署实践

原文:ASP.NET Core在CentOS上的最小化部署实践 引言 本文从Linux小白的视角, 在CentOS 7.x服务器上搭建一个Nginx-Powered AspNet Core Web准生产应用. 在开始之前,我们还是重温一下部署原理,正如你所常见的.Net Core 部署图: 在Linux上部署.Net Core App最好的方式是在Linux机器上使用Kestrel 服务在端口5000上支撑web应用: 然后设置Nginx作为反向代理服务器,将输入请求转发给Kestrel服务器,