apache storm 的安装

原文: http://storm.praveendeshmane.co.in/storm/storm-0-10-0-installation-on-ubuntu-14-04.jsp

------------------------------------------

Apache Storm is a distributed stream processing computation framework written predominantly in the Clojure programming language. Originally created by Nathan Marz and team at BackType, the project was open sourced after being acquired by Twitter. It uses custom created "spouts" and "bolts" to define information sources and manipulations to allow batch, distributed processing of streaming data. The initial release was on 17 September 2011.

A Storm application is designed as a "topology" in the shape of a directed acyclic graph (DAG) with spouts and bolts acting as the graph vertices. Edges on the graph are named streams and direct data from one node to another. Together, the topology acts as a data transformation pipeline. At a superficial level the general topology structure is similar to a MapReduce job, with the main difference being that data is processed in real time as opposed to in individual batches.

Pre Requirements

1) A machine with Ubuntu 14.04 LTS operating system.

2) Apcahe ZooKeeper pre installed (How to install ZooKeeper on Ubuntu 14.04)

3) Apache Storm 0.10.0 software (Download Here)

Storm Installation Steps

Step 1 - Installing java 7. Open a terminal (CTRL + ALT + T) and type the following sudo command.

$ sudo apt-get install openjdk-7-jdk

Check it installed properly or not.

$ java -version

Step 2 - Edit $HOME/.bashrc file by adding the java path.

$ sudo gedit $HOME/.bashrc

$HOME/.bashrc file

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

Step 3 - Reload your changed $HOME/.bashrc settings

$ source $HOME/.bashrc

Step 4 - Creating storm directory. Open a terminal (CTRL + ALT + T) and type the following $ sudo command.

$ sudo mkdir /usr/local/storm

Step 5 - Change the ownership and permissions of the directory /usr/local/storm. Here ‘hduser‘ is an Ubuntu username.

$ sudo chown -R hduser /usr/local/storm
$ sudo chmod -R 755 /usr/local/storm

Step 6 - Change the directory to /home/hduser/Desktop , In my case the downloaded apache-storm-0.10.0.tar.gz file is in /home/hduser/Desktop folder. For you it might be in /downloads folder check it.

$ cd /home/hduser/Desktop/

Step 7 - Untar the apache-storm-0.10.0.tar.gz file.

$ tar xzf apache-storm-0.10.0.tar.gz

Step 8 - Move the contents of apache-storm-0.10.0 folder to /usr/local/storm

$ mv apache-storm-0.10.0/* /usr/local/storm

Step 9 - Change the directory to /usr/local/storm

$ cd /usr/local/storm

Step 10 - Create /data, /data/nimbus and /data/supervisor directories.

$ sudo mkdir /usr/local/storm/data
$ sudo mkdir /usr/local/storm/data/nimbus
$ sudo mkdir /usr/local/storm/data/supervisor

Step 11 - Change the ownership and permissions of the directory /data, /data/nimbus and /data/supervisor directories.. Here ‘hduser‘ is an Ubuntu username.

$ sudo chown -R hduser /usr/local/storm/data
$ sudo chown -R hduser /usr/local/storm/data/nimbus
$ sudo chown -R hduser /usr/local/storm/data/supervisor
$ sudo chmod -R 755 /usr/local/storm/data
$ sudo chmod -R 755 /usr/local/storm/data/nimbus
$ sudo chmod -R 755 /usr/local/storm/data/supervisor

Step 12 - Change the directory to /usr/local/storm/conf

$ cd /usr/local/storm/conf

Step 13 - Edit storm.yaml file.

$ gedit storm.yaml

Step 14 - Add these below lines to storm.yaml file. Save and close.

storm.zookeeper.servers:
- "localhost"
storm.local.dir: "/usr/local/storm/data"
nimbus.host: "localhost"
nimbus.thrift.port: 49627
storm.zookeeper.port: 2181
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703

Step 15 - Start ZooKeeper. Open a new terminal (CTRL + ALT + T) and start zookeeper.

$ /usr/local/zookeeper/bin/zkServer.sh start

Step 16 - Open a new terminal (CTRL + ALT + T). Change the directory to /usr/local/storm

$ cd /usr/local/storm

Step 17 - Start nimbus

$ ./bin/storm nimbus

Step 18 - Open a new terminal (CTRL + ALT + T). Change the directory to /usr/local/storm

$ cd /usr/local/storm

Step 19 - Start supervisor

$ ./bin/storm supervisor

Step 20 - Open a new terminal (CTRL + ALT + T). Change the directory to /usr/local/storm

$ cd /usr/local/storm

Step 21 - Start web UI

$ ./bin/storm ui

Step 22 - Check the web UI here. Open a browser and type the following URL.

http://localhost:8080

Please share this blog post and follow me for latest updates on

时间: 2024-10-20 10:11:12

apache storm 的安装的相关文章

Docker在Ubuntu16.04上安装和部署Apache Storm

Docker在Ubuntu16.04上安装和部署Apache Storm 1.添加Docker源 sudo apt-get update 2.增加CA证书 sudo apt-get install apt-transport-https ca-certificates 3.添加GPG Key(一种加密手段) sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897

Apache Storm 1.1.0 中文文档 | ApacheCN

前言  Apache Storm 是一个免费的,开源的,分布式的实时计算系统. 官方文档: http://storm.apache.org 中文文档: http://storm.apachecn.org ApacheCN 最近组织了翻译 Storm 1.1.0 中文文档 的活动,整体 翻译进度 为 96%. 感谢大家参与到该活动中来 感谢无私奉献的 贡献者,才有了这份 Storm 1.1.0 中文文档 感谢一路有你的陪伴,我们才可以做的更好,走的更快,走的更远,我们一直在努力 ... 网页地址:

课程预告:大数据实时处理系统Apache Storm

Storm为分布式实时计算提供了一组通用原语,可被用于"流处理"之中,实时处理消息并更新数据库.这是管理队列及工作者集群的另一种方式. Storm也可被用于"连续计算"(continuous computation),对数据流做连续查询,在计算时就将结果以流的形式输出给用户.它还可被用于"分布式RPC",以并行的方式运行昂贵的运算. Storm可以方便地在一个计算机集群中编写与扩展复杂的实时计算,Storm用于实时处理,就好比 Hadoop 用于

ubuntu14.04LTS 下storm单机版安装配置

1.下载storm 的安装文件 http://www.apache.org/dyn/closer.cgi/incubator/storm/apache-storm-0.9.2-incubating/apache-storm-0.9.2-incubating.tar.gz 1 tar -zxvf apache-storm-0.9.2-incubating.tar.gz 2.修改Storm的配置文件 conf/storm.yaml 1 storm.zookeeper.servers: 2 - "lo

Apache Storm 集群环境搭建

Apache storm 是一个由twitter开源的大数据处理系统,与其他系统不同的是,storm旨在用于分布式实时处理并且与语言无关.笔者所认知的storm使用场景诸 如 实时日志分析.网站用户行为实时分析.实时计算等,目前很多公司也都把storm作为自己的大数据架构的一部分,来实现一些实时业务的处理. 相信大家都和我有一样的认知,那就是现在的技术都是项目驱动模式,没有最好的技术,只有最适合自己项目的技术.下面先跟大家分享一下我对storm的一些简单了解: storm的优点: 1.简单的编程

Apache - Storm

一.Apache Storm Storm是一个分布式的,可靠的,容错的数据流处理系统.Storm集群的输入流由一个被称作spout的组件管理,spout把数据传递给bolt,bolt要么把数据保存到某种存储器,要么把数据传递给其它的bolt.一个Storm集群就是在一连串的bolt之间转换spout传过来的数据. 1.Storm组件 在Storm集群中.有两类节点:主节点master node和工作节点worker nodes.主节点运行Nimbus守护进程,这个守护进程负责在集群中分发代码,为

apache storm基本原理及使用总结

什么是Apache Storm Apache Storm是一个分布式实时大数据处理系统.Storm设计用于在容错和水平可扩展方法中处理大量数据.它是一个流数据框架,具有最高的摄取率.虽然Storm是无状态的,它通过Apache ZooKeeper管理分布式环境和集群状态.通过Storm可以并行地对实时数据执行各种操作.Storm易于部署和操作,并且它可以保证每个消息将通过拓扑至少处理一次. Apache Storm核心概念 Apache Storm从一端读取??实时数据的原始流,并将其传递通过一

Centos6.5 storm单机安装

Centos6.5 storm单机安装 本篇幅讲述Twitter Storm安装配置,也作为自己的笔记. storm的官方安装说明(e文):https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster storm的安装分为单机版和集群版,只是配置稍微有点区别,大致一样. 要使用storm首先要安装以下工具: python.zookeeper.zeromq.jzmq.storm 第一步,安装Python2.7.2 wget

Windows 下 Apache HTTP Server 安装、配置以及与 Tomcat 的整合(附图)

如果您能点开这篇文章,说明您已对熟悉Apache HTTP Server(下文用Apache简称)配置的重要性已很清楚了,本文不在赘述,直接介入正题,请往下阅读: 为便于阅读,列出文章目录: 一.Apache 的安装说明 二.Apache 的基本配置 三.Apache 整合 Tomcat 的配置 以下为正文内容: 一.Apache 的安装说明 1.首先打开Apache官网,进入Windows版本的下载地址页面:http://archive.apache.org/dist/httpd/binari