1.Storm 安装手册和提交Topology

目录

(一)搭建单机storm集群

一、下载所需要的资源

二、资源解压

三、安装JDK

四、安装依赖的库文件

五、安装ZMQ

六、安装jzmq

七、启动zookeeper

八、启动storm

九、在UI端访问

十、编译storm-start jar包

十一、storm-starter源码导出

十二、提交Topology

(二)搭建多机storm集群

一、基本设置

二、修改配置文件

三、启动集群

四、在UI端口访问

五、提交Topology

附一

Storm 安装手册

(一)搭建单机storm集群

一、下载所需要的资源

Storm:

http://storm-project.net/downloads.html

(0.9.3)apache-storm-0.9.3.tar.gz

Zookeeper:

http://www.apache.org/dyn/closer.cgi/zookeeper/

(3.4.6)zookeeper-3.4.6.tar.gz

ZMQ:

http://download.zeromq.org/

(2.1.7)zeromq-2.1.7.tar.gz

jzmq:

http://github.com/nathanmarz/jzmq/archive/master.zip

jzmq-master.zip

二、资源解压

下载完成后将资源放在自己指定的目录下

解压

tar -xvf zookeeper-3.4.6.tar.gz

tar -xvf apache-storm-0.9.3.tar.gz

tar -xvf zeromq-2.1.7.tar.gz

unzip jzmq-master.zip

三、安装JDK

安装openjdk

安装命令:

sudo apt-get install openjdk-7-jdk

安装完成后修改环境变量,修改文件etc/profile

vim /etc/profile

修改以下内容:

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386

export CLASSPATH=$JAVA_HOME/lib

export PATH=$JAVA_HOME/bin:$PATH

修改完成后,使配置文件生效

source /etc/profile

四、安装依赖的库文件

运行以下命令对所需要的库文件进行安装,也可使用yum进行安装

sudo apt-get install libtool

sudo apt-get install autoconf

sudo apt-get install automake

sudo apt-get install g++

sudo apt-get install uuid-dev

sudo apt-get install uuid

sudo apt-get install e2fsprogs

sudo apt-get install python

五、安装ZMQ

进入zero相应目录下,运行以下命令进行安装

cd zeromq-2.1.7

./configure

若出现如下错误

configure: error: cannot link with -luuid, install uuid-dev.

解决方案如下:

yum install uuid-dev 提示找不到该依赖包  (需从网上下载安装yum)

yum install uuid*     直接用* ok

./configure  再次运行configure还是报错

configure: error: cannot link with -luuid, install uuid-dev.

上网搜索可能是还缺另外一个包

yum install e2fsprogs* ok

make

sudo make install

sudo ldconfig

六、安装jzmq

进入jzmq相应目录下,运行以下命令进行安装

cd jzmq-master

./autogen.sh

./configure

make

sudo make install

若make过程出现如下错误

***No rule to make target ‘classdist_noinst.stamp’,needed by ‘org、zeromq/ZMQ.class’. Stop

解决方法,运行如下命令解决

touch src/classdist_noinst.stamp

cd src/org/zeromq

javac *.java

cd –

make

sudo make install

七、启动zookeeper

进入zookeeper相应目录

cd zookeeper-3.4.6/

重命名./conf/zoo_sample.cfg为./conf/zoo.cfg

mv ./conf/zoo_sample.cfg ./conf/zoo.cfg

运行命令启动zookeeper

bin/zkServer.sh start

检查zookeeper是否启动成功zkServer.sh status

bin/zkCli.sh –server 127.0.0.1:2181

在出现的交互式窗口中运行

ls /

八、启动storm

进入storm相应目录,启动storm相应服务

cd apache-storm-0.9.3/bin

./storm nimbus&

./storm supervisor&

./storm ui&

九、在UI端访问

在桌面打开连接http://localhost:8080,此时能够看见Storm UI界面

十、编译storm-start jar包

在运行起来的集群上提交一个Topology,使用storm-starter做示范

安装git工具

sudo apt-get install git

安装leiningen工具(2.0.0以上版本),按照https://github.com/technomancy/leiningen的步骤安装

确保java版本在6以上

下载lein脚本(我下载的lein脚本附在最后,附一),命名为lein:

cd ~/wfm-storm/apache-storm-0.9.3/bin

vim lein

将附一的脚本粘贴到lein文件中并保存

chmod 755 ~/bin/lein

运行lein

./lein

运行lein报如下错误

“Failed to download

https://github.com/technomancy/leiningen/releases/download/2.5.0/leiningen-2.5.0-standalone.jar  (exit code 7) It‘s possible your HTTP client‘s certificate store does not have the correct certificate authority needed. This is often caused by an out-of-date version of libssl. It‘s also possible that you‘re behind a firewall and haven‘t set HTTP_PROXY and HTTPS_PROXY.“

解决方法:

export HTTP_CLIENT="wget --no-check-certificate -O"

./lein

export PATH=" ~/wfm-storm/apache-storm-0.9.3/bin/:$PATH"(在配置文件中加入这句)

下载storm-start源码保存在~/wfm-storm/storm-starter中

cd ~/wfm-storm/

git clone git://github.com/nathanmarz/storm-starter.git

依次执行以下命令创建项目jar包

cd storm-starter/

lein deps //下载相应的依赖包

lein compile //编译

lein install //安装打包

报backtype.storm.LocalCluster和backtype.storm.LocalDRPC类找不到:

Compiling backtype.storm.testing4j

Exception in thread "main" java.lang.ClassNotFoundException: backtype.storm.LocalCluster, compiling:(testing4j.clj:1)

at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3387)

at clojure.lang.Compiler.compile1(Compiler.java:7035)

at clojure.lang.Compiler.compile1(Compiler.java:7025)

at clojure.lang.Compiler.compile(Compiler.java:7097)

at clojure.lang.RT.compile(RT.java:387)

at clojure.lang.RT.load(RT.java:427)

at clojure.lang.RT.load(RT.java:400)

at clojure.core$load$fn__4890.invoke(core.clj:5415)

at clojure.core$load.doInvoke(core.clj:5414)

解决方法:

修改storm-core/src/clj/backtype/storm/testing4j.clj

和storm-core/src/clj/storm/trident/testing.clj,分别引用这两个类:

testing4j.clj:

1 (ns backtype.storm.testing4j

2   (:import [java.util Map List Collection ArrayList])

3   (:require [backtype.storm.LocalCluster :as LocalCluster])

testing.clj:

1  (ns storm.trident.testing)

2   (:require [backtype.storm.LocalDRPC :as LocalDRPC])

安装时出现jdk版本问题

sudo apt-get install openjdk-7-jdk

执行之后会将JDK安装到/usr/lib/jvm/java-7-openjdk-i386/

通过update-alternatives选择系统中使用的java

sudo update-alternatives --config java

有 3 个候选项可用于替换 java (提供 /usr/bin/java)。

选择  路径                         优先级  状态

------------------------------------------------------------

* 0   /usr/lib/jvm/java-6-openjdk-i386/jre/bin/java   1061      自动模式

1    /usr/lib/jvm/java-6-openjdk-i386/jre/bin/java   1061      手动模式

2    /usr/lib/jvm/java-6-sun/jre/bin/java            63        手动模式

3    /usr/lib/jvm/java-7-openjdk-i386/jre/bin/java   1051      手动模式

要维持当前值[*]请按回车键,或者键入选择的编号:3

update-alternatives: 使用 /usr/lib/jvm/java-7-openjdk-i386/jre/bin/java 来提供 /usr/bin/java (java),于 手动模式 中。

实际是将/usr/bin/java链接到JDK7上,保证执行java命令后,实际运行的是我们的目标版本

java -version

除了java外,还有javac、jar、javah、javap、jps、jconsole等

sudo update-alternatives --config javac

sudo update-alternatives --config jar

sudo update-alternatives --config javah

sudo update-alternatives --config javap

sudo update-alternatives --config jps

sudo update-alternatives --config jconsole

补充:jps -lv  用于查看java程序的启动参数,如java版本

修改各个环境变量

sudo vim /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/

export CLASSPATH=$JAVA_HOME/lib

export PATH=$JAVA_HOME/bin:$PATH

source  /etc/profile

sudo vim /etc/environment

CLASSPATH=.:/usr/lib/jvm/java-7-openjdk-amd64/lib#

JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/source /etc/environment

sudo vim ~/.bashrc

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/

export CLASSPATH=$JAVA_HOME/lib

export PATH=$JAVA_HOME/bin:$PATH

source ~/.bashrc

创建好的jar包storm-starter-0.0.1-SNAPSHOT.jar位于target目录下

十一、storm-starter源码导出

获取storm-starter源码

cd ~/wfm-storm

git clone git://github.com/apache/storm.git

cd storm/examples/storm-starter

storm-starter简介

storm-starter包含很多使用storm的例子。如果是第一次使用storm,建议从以下三个例子(topologies)入手:

- ExclamationTopology: 基础的topology,全部都是使用java写的

- WordCountTopology: 基础的topology ,利用multilang使用Python实现的

- ReachTopology: storm之上的复杂的DRPC例子

熟悉这几个例子之后,可看看其他的一些topologies(src/jvm/storm/starter/),其中有更多advanced的实现例子

在maven上使用storm-starter,安装maven(3.x)

下载相应maven版本(http://maven.apache.org/download.cgi),例如apache-maven-3.2.5-bin.tar.gz

解压下载的包

tar -zxf apache-maven-3.2.5-bin.tar.gz

修改环境变量M2_HOME

sudo vim /etc/profile

添加如下代码:

export M2_HOME=/home/wamdm/wfm-storm/apache-maven-3.2.5

export M2=$M2_HOME/bin

export MAVEN_OPTS="-Xms256m -Xmx512m"

export PATH=$M2:$PATH

source /etc/profile

运行mvn –version查看安装是否成功.

在maven上运行topologies

storm-starter topologies可以使用maven-exec-plugin执行

例如:在本地模式下编译并且运行 WordCountTopology

$ mvn compile exec:java -Dstorm.topology=storm.starter.WordCountTopology

也可以使用Maven运行clojure topologies

$ mvn compile exec:java -Dstorm.topology=storm.starter.clj.word_count

打包strom-starter用于storm集群

mvn package

运行例子:

# Example 1: Run the RollingTopWords in local mode (LocalCluster)

$ storm jar storm-starter-*-jar-with-dependencies.jar storm.starter.RollingTopWords

# Example 2: Run the RollingTopWords in remote/cluster mode,

# under the name "production-topology"

$ storm jar storm-starter-*-jar-with-dependencies.jar storm.starter.RollingTopWords production-topology remote

十二、提交Topology

进入storm文件夹的bin目录中,运行以下命令提交Topology

./storm jar ~/project/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCount Topology wordcount

等待提交结束,刷新ui页面,可以看到提交的“Wordcount”Topology,点击Wordcount可以看到其详细的运行情况

停止storm Topology

storm kill :{toponame}

(二)搭建多机storm集群

一、基本设置

首先在每台机器上都搭建单机集群

wamdm1作为Nimbus(192.168.0.11)

wamdm2作为Zookeeper(192.168.0.12)

wamdm3作为Supervisor(192.168.0.13)

wamdm4作为Supervisor(192.168.0.14)

二、修改配置文件

修改wamdm1、wamdm3、wamdm4上的storm.yaml

(~/wfm-storm/apache-storm-0.9.3/conf)

进入conf目录

cd ~/wfm-storm/apache-storm-0.9.3/conf

修改配置文件

vim storm.yaml

将文件中添加如下代码

storm.zookeeper.servers:

- "192.168.0.12"

# - "192.168.0.13"

nimbus.host: "192.168.0.11"

#ui.port: 83

supervisor.slots.ports:

- 6700

- 6701

- 6702

- 6703

可用参数解释:

java.library.path:该配置项配置启动storm所需lib包的路径(默认可不用配置)

storm.zookeeper.servers:该配置项配置了当前集群中所有ZooKeeper机器的IP地址(这里我们只使用wamdm1作为Zookeeper服务器)

nimbus.host:该配置项指明了Nimbus机器的IP地址

ui.port:该配置项配置了Storm UI使用的端口。(默认使用8080端口)

supervisor.slots.ports:该配置项指明了一台Supervisor机器上所有可以使用的slot信息,也就是端口号。表明该机器上最多可以启动4个worker

Storm还提供了很多其他的配置项,需要时可查相应的document

三、启动集群

在wamdm2上启动zookeeper

bin/zkServer.sh start

检查zookeeper是否启动成功zkServer.sh status

bin/zkCli.sh –server 127.0.0.1:2181

在出现的交互式窗口中运行

ls /

在wamdm1上启动nimbus和ui

cd apache-storm-0.9.3/bin

./storm nimbus

./storm ui

在wamdm3上启动supervisor

cd apache-storm-0.9.3/bin

./storm supervisor

在wamdm4上启动supervisor

cd apache-storm-0.9.3/bin

./storm supervisor

四、在UI端口访问

在桌面打开连接http://192.168.0.11:8080,此时能够看见Storm集群

五、提交Topology

在wamdm1(Nimbus所在机器)上提交WordCountTopology到集群中,刷新ui页面,就能够看到提交的Topology了。

附一

lein

  1 #!/usr/bin/env bash
  2
  3 # Ensure this file is executable via `chmod a+x lein`, then place it
  4
  5 # somewhere on your $PATH, like ~/bin. The rest of Leiningen will be
  6
  7 # installed upon first run into the ~/.lein/self-installs directory.
  8
  9 export LEIN_VERSION="2.5.0"
 10
 11 case $LEIN_VERSION in
 12
 13     *SNAPSHOT) SNAPSHOT="YES" ;;
 14
 15     *) SNAPSHOT="NO" ;;
 16
 17 esac
 18
 19 if [[ "$OSTYPE" == "cygwin" ]] || [[ "$OSTYPE" == "msys" ]]; then
 20
 21     delimiter=";"
 22
 23 else
 24
 25     delimiter=":"
 26
 27 fi
 28
 29 if [[ "$OSTYPE" == "cygwin" ]]; then
 30
 31   cygwin=true
 32
 33 else
 34
 35   cygwin=false
 36
 37 fi
 38
 39 function make_native_path {
 40
 41     # ensure we have native paths
 42
 43     if $cygwin && [[ "$1"  == /* ]]; then
 44
 45     echo -n "$(cygpath -wp "$1")"
 46
 47     elif [[ "$OSTYPE" == "msys" && "$1"  == /?/* ]]; then
 48
 49     echo -n "$(sh -c "(cd $1 2</dev/null && pwd -W) || echo $1 | sed ‘s/^\\/\([a-z]\)/\\1:/g‘")"
 50
 51     else
 52
 53     echo -n "$1"
 54
 55     fi
 56
 57 }
 58
 59 #  usage : add_path PATH_VAR [PATH]...
 60
 61 function add_path {
 62
 63     local path_var="$1"
 64
 65     shift
 66
 67     while [ -n "$1" ];do
 68
 69         # http://bashify.com/?Useful_Techniques:Indirect_Variables:Indirect_Assignment
 70
 71         if [[ -z ${!path_var} ]]; then
 72
 73           export ${path_var}="$(make_native_path "$1")"
 74
 75         else
 76
 77           export ${path_var}="${!path_var}${delimiter}$(make_native_path "$1")"
 78
 79         fi
 80
 81     shift
 82
 83     done
 84
 85 }
 86
 87 function download_failed_message {
 88
 89     echo "Failed to download $1 (exit code $2)"
 90
 91     echo "It‘s possible your HTTP client‘s certificate store does not have the"
 92
 93     echo "correct certificate authority needed. This is often caused by an"
 94
 95     echo "out-of-date version of libssl. It‘s also possible that you‘re behind a"
 96
 97     echo "firewall and haven‘t set HTTP_PROXY and HTTPS_PROXY."
 98
 99 }
100
101 function self_install {
102
103   if [ -r "$LEIN_JAR" ]; then
104
105     echo "The self-install jar already exists at $LEIN_JAR."
106
107     echo "If you wish to re-download, delete it and rerun \"$0 self-install\"."
108
109     exit 1
110
111   fi
112
113   echo "Downloading Leiningen to $LEIN_JAR now..."
114
115   mkdir -p "$(dirname "$LEIN_JAR")"
116
117   LEIN_URL="https://github.com/technomancy/leiningen/releases/download/$LEIN_VERSION/leiningen-$LEIN_VERSION-standalone.jar"
118
119   $HTTP_CLIENT "$LEIN_JAR.pending" "$LEIN_URL"
120
121   local exit_code=$?
122
123   if [ $exit_code == 0 ]; then
124
125       # TODO: checksum
126
127       mv -f "$LEIN_JAR.pending" "$LEIN_JAR"
128
129   else
130
131       rm "$LEIN_JAR.pending" 2> /dev/null
132
133       download_failed_message "$LEIN_URL" "$exit_code"
134
135       exit 1
136
137   fi
138
139 }
140
141 if [ `id -u` -eq 0 ] && [ "$LEIN_ROOT" = "" ]; then
142
143     echo "WARNING: You‘re currently running as root; probably by accident."
144
145     echo "Press control-C to abort or Enter to continue as root."
146
147     echo "Set LEIN_ROOT to disable this warning."
148
149     read _
150
151 fi
152
153 NOT_FOUND=1
154
155 ORIGINAL_PWD="$PWD"
156
157 while [ ! -r "$PWD/project.clj" ] && [ "$PWD" != "/" ] && [ $NOT_FOUND -ne 0 ]
158
159 do
160
161     cd ..
162
163     if [ "$(dirname "$PWD")" = "/" ]; then
164
165         NOT_FOUND=0
166
167         cd "$ORIGINAL_PWD"
168
169     fi
170
171 done
172
173 export LEIN_HOME="${LEIN_HOME:-"$HOME/.lein"}"
174
175 for f in "$LEIN_HOME/leinrc" ".leinrc"; do
176
177   if [ -e "$f" ]; then
178
179     source "$f"
180
181   fi
182
183 done
184
185 if $cygwin; then
186
187     export LEIN_HOME=`cygpath -w "$LEIN_HOME"`
188
189 fi
190
191 LEIN_JAR="$LEIN_HOME/self-installs/leiningen-$LEIN_VERSION-standalone.jar"
192
193 # normalize $0 on certain BSDs
194
195 if [ "$(dirname "$0")" = "." ]; then
196
197     SCRIPT="$(which $(basename "$0"))"
198
199 else
200
201     SCRIPT="$0"
202
203 fi
204
205 # resolve symlinks to the script itself portably
206
207 while [ -h "$SCRIPT" ] ; do
208
209     ls=`ls -ld "$SCRIPT"`
210
211     link=`expr "$ls" : ‘.*-> \(.*\)$‘`
212
213     if expr "$link" : ‘/.*‘ > /dev/null; then
214
215         SCRIPT="$link"
216
217     else
218
219         SCRIPT="$(dirname "$SCRIPT"$)/$link"
220
221     fi
222
223 done
224
225 BIN_DIR="$(dirname "$SCRIPT")"
226
227 export LEIN_JVM_OPTS="${LEIN_JVM_OPTS-"-XX:+TieredCompilation -XX:TieredStopAtLevel=1"}"
228
229 # This needs to be defined before we call HTTP_CLIENT below
230
231 if [ "$HTTP_CLIENT" = "" ]; then
232
233     if type -p curl >/dev/null 2>&1; then
234
235         if [ "$https_proxy" != "" ]; then
236
237             CURL_PROXY="-x $https_proxy"
238
239         fi
240
241         HTTP_CLIENT="curl $CURL_PROXY -f -L -o"
242
243     else
244
245         HTTP_CLIENT="wget -O"
246
247     fi
248
249 fi
250
251 # When :eval-in :classloader we need more memory
252
253 grep -E -q ‘^\s*:eval-in\s+:classloader\s*$‘ project.clj 2> /dev/null && 254
255     export LEIN_JVM_OPTS="$LEIN_JVM_OPTS -Xms64m -Xmx512m"
256
257 if [ -r "$BIN_DIR/../src/leiningen/version.clj" ]; then
258
259     # Running from source checkout
260
261     LEIN_DIR="$(dirname "$BIN_DIR")"
262
263     # Need to use lein release to bootstrap the leiningen-core library (for aether)
264
265     if [ ! -r "$LEIN_DIR/leiningen-core/.lein-bootstrap" ]; then
266
267         echo "Leiningen is missing its dependencies."
268
269         echo "Please run \"lein bootstrap\" in the leiningen-core/ directory"
270
271         echo "with a stable release of Leiningen. See CONTRIBUTING.md for details."
272
273         exit 1
274
275     fi
276
277     # If project.clj for lein or leiningen-core changes, we must recalculate
278
279     LAST_PROJECT_CHECKSUM=$(cat "$LEIN_DIR/.lein-project-checksum" 2> /dev/null)
280
281     PROJECT_CHECKSUM=$(sum "$LEIN_DIR/project.clj" "$LEIN_DIR/leiningen-core/project.clj")
282
283     if [ "$PROJECT_CHECKSUM" != "$LAST_PROJECT_CHECKSUM" ]; then
284
285         if [ -r "$LEIN_DIR/.lein-classpath" ]; then
286
287             rm "$LEIN_DIR/.lein-classpath"
288
289         fi
290
291     fi
292
293     # Use bin/lein to calculate its own classpath.
294
295     if [ ! -r "$LEIN_DIR/.lein-classpath" ] && [ "$1" != "classpath" ]; then
296
297         echo "Recalculating Leiningen‘s classpath."
298
299         ORIG_PWD="$PWD"
300
301         cd "$LEIN_DIR"
302
303         LEIN_NO_USER_PROFILES=1 $0 classpath .lein-classpath
304
305         sum "$LEIN_DIR/project.clj" "$LEIN_DIR/leiningen-core/project.clj" > 306
307             .lein-project-checksum
308
309         cd "$ORIG_PWD"
310
311     fi
312
313     mkdir -p "$LEIN_DIR/target/classes"
314
315     export LEIN_JVM_OPTS="$LEIN_JVM_OPTS -Dclojure.compile.path=$LEIN_DIR/target/classes"
316
317     add_path CLASSPATH "$LEIN_DIR/leiningen-core/src/" "$LEIN_DIR/leiningen-core/resources/" 318
319         "$LEIN_DIR/test:$LEIN_DIR/target/classes" "$LEIN_DIR/src" ":$LEIN_DIR/resources"
320
321     if [ -r "$LEIN_DIR/.lein-classpath" ]; then
322
323         add_path CLASSPATH "$(cat "$LEIN_DIR/.lein-classpath" 2> /dev/null)"
324
325     else
326
327         add_path CLASSPATH "$(cat "$LEIN_DIR/leiningen-core/.lein-bootstrap" 2> /dev/null)"
328
329     fi
330
331 else # Not running from a checkout
332
333     add_path CLASSPATH "$LEIN_JAR"
334
335     BOOTCLASSPATH="-Xbootclasspath/a:$LEIN_JAR"
336
337     if [ ! -r "$LEIN_JAR" -a "$1" != "self-install" ]; then
338
339         self_install
340
341     fi
342
343 fi
344
345 # TODO: explain what to do when Java is missing
346
347 export JAVA_CMD="${JAVA_CMD:-"java"}"
348
349 export LEIN_JAVA_CMD="${LEIN_JAVA_CMD:-$JAVA_CMD}"
350
351 if [[ -z "${DRIP_INIT+x}" && "$(basename "$LEIN_JAVA_CMD")" == *drip* ]]; then
352
353     export DRIP_INIT="$(printf -- ‘-e\n(require (quote leiningen.repl))‘)"
354
355     export DRIP_INIT_CLASS="clojure.main"
356
357 fi
358
359 # Support $JAVA_OPTS for backwards-compatibility.
360
361 export JVM_OPTS="${JVM_OPTS:-"$JAVA_OPTS"}"
362
363 # Handle jline issue with cygwin not propagating OSTYPE through java subprocesses: https://github.com/jline/jline2/issues/62
364
365 cygterm=false
366
367 if $cygwin; then
368
369   case "$TERM" in
370
371     rxvt* | xterm* | vt*) cygterm=true ;;
372
373   esac
374
375 fi
376
377 if $cygterm; then
378
379   LEIN_JVM_OPTS="$LEIN_JVM_OPTS -Djline.terminal=jline.UnixTerminal"
380
381   stty -icanon min 1 -echo > /dev/null 2>&1
382
383 fi
384
385 # TODO: investigate http://skife.org/java/unix/2011/06/20/really_executable_jars.html
386
387 # If you‘re packaging this for a package manager (.deb, homebrew, etc)
388
389 # you need to remove the self-install and upgrade functionality or see lein-pkg.
390
391 if [ "$1" = "self-install" ]; then
392
393     if [ -r "$BIN_DIR/../src/leiningen/version.clj" ]; then
394
395         echo "Running self-install from a checkout is not supported."
396
397         echo "See CONTRIBUTING.md for SNAPSHOT-specific build instructions."
398
399         exit 1
400
401     fi
402
403     echo "Manual self-install is deprecated; it will run automatically when necessary."
404
405     self_install
406
407 elif [ "$1" = "upgrade" ] || [ "$1" = "downgrade" ]; then
408
409     if [ "$LEIN_DIR" != "" ]; then
410
411         echo "The upgrade task is not meant to be run from a checkout."
412
413         exit 1
414
415     fi
416
417     if [ $SNAPSHOT = "YES" ]; then
418
419         echo "The upgrade task is only meant for stable releases."
420
421         echo "See the \"Hacking\" section of the README."
422
423         exit 1
424
425     fi
426
427     if [ ! -w "$SCRIPT" ]; then
428
429         echo "You do not have permission to upgrade the installation in $SCRIPT"
430
431         exit 1
432
433     else
434
435         TARGET_VERSION="${2:-stable}"
436
437         echo "The script at $SCRIPT will be upgraded to the latest $TARGET_VERSION version."
438
439         echo -n "Do you want to continue [Y/n]? "
440
441         read RESP
442
443         case "$RESP" in
444
445             y|Y|"")
446
447                 echo
448
449                 echo "Upgrading..."
450
451                 TARGET="/tmp/lein-$$-upgrade"
452
453                 if $cygwin; then
454
455                     TARGET=`cygpath -w $TARGET`
456
457                 fi
458
459                 LEIN_SCRIPT_URL="https://github.com/technomancy/leiningen/raw/$TARGET_VERSION/bin/lein"
460
461                 $HTTP_CLIENT "$TARGET" "$LEIN_SCRIPT_URL"
462
463                 if [ $? == 0 ]; then
464
465                     cmp -s "$TARGET" "$SCRIPT"
466
467                     if [ $? == 0 ]; then
468
469                         echo "Leiningen is already up-to-date."
470
471                     fi
472
473                     mv "$TARGET" "$SCRIPT" && chmod +x "$SCRIPT"
474
475                     exec "$SCRIPT" version
476
477                 else
478
479                     download_failed_message "$LEIN_SCRIPT_URL"
480
481                 fi;;
482
483             *)
484
485                 echo "Aborted."
486
487                 exit 1;;
488
489         esac
490
491     fi
492
493 else
494
495     if $cygwin; then
496
497         # When running on Cygwin, use Windows-style paths for java
498
499         ORIGINAL_PWD=`cygpath -w "$ORIGINAL_PWD"`
500
501     fi
502
503     # apply context specific CLASSPATH entries
504
505     if [ -f .lein-classpath ]; then
506
507         add_path CLASSPATH "$(cat .lein-classpath)"
508
509     fi
510
511     if [ $DEBUG ]; then
512
513         echo "Leiningen‘s classpath: $CLASSPATH"
514
515     fi
516
517     if [ -r .lein-fast-trampoline ]; then
518
519         export LEIN_FAST_TRAMPOLINE=‘y‘
520
521     fi
522
523     if [ "$LEIN_FAST_TRAMPOLINE" != "" ] && [ -r project.clj ]; then
524
525         INPUTS="[email protected] $(cat project.clj) $LEIN_VERSION $(test -f "$LEIN_HOME/profiles.clj" && cat "$LEIN_HOME/profiles.clj")"
526
527         export INPUT_CHECKSUM=$(echo $INPUTS | shasum - | cut -f 1 -d " ")
528
529         # Just don‘t change :target-path in project.clj, mkay?
530
531         TRAMPOLINE_FILE="target/trampolines/$INPUT_CHECKSUM"
532
533     else
534
535         if hash mktemp 2>/dev/null; then
536
537             # Check if mktemp is available before using it
538
539             TRAMPOLINE_FILE="$(mktemp /tmp/lein-trampoline-XXXXXXXXXXXXX)"
540
541         else
542
543             TRAMPOLINE_FILE="/tmp/lein-trampoline-$$"
544
545         fi
546
547         trap "rm -f $TRAMPOLINE_FILE" EXIT
548
549     fi
550
551     if $cygwin; then
552
553         TRAMPOLINE_FILE=`cygpath -w $TRAMPOLINE_FILE`
554
555     fi
556
557     if [ "$INPUT_CHECKSUM" != "" ] && [ -r "$TRAMPOLINE_FILE" ]; then
558
559         if [ $DEBUG ]; then
560
561             echo "Fast trampoline with $TRAMPOLINE_FILE."
562
563         fi
564
565         exec sh -c "exec $(cat $TRAMPOLINE_FILE)"
566
567     else
568
569         export TRAMPOLINE_FILE
570
571         "$LEIN_JAVA_CMD" 572
573             "${BOOTCLASSPATH[@]}" 574
575             -Dfile.encoding=UTF-8 576
577             -Dmaven.wagon.http.ssl.easy=false 578
579             -Dmaven.wagon.rto=10000 580
581             $LEIN_JVM_OPTS 582
583             -Dleiningen.original.pwd="$ORIGINAL_PWD" 584
585             -Dleiningen.script="$SCRIPT" 586
587             -classpath "$CLASSPATH" 588
589             clojure.main -m leiningen.core.main "[email protected]"
590
591         EXIT_CODE=$?
592
593         if $cygterm ; then
594
595           stty icanon echo > /dev/null 2>&1
596
597         fi
598
599         ## TODO: [ -r "$TRAMPOLINE_FILE" ] may be redundant? A trampoline file
600
601         ## is always generated these days.
602
603         if [ -r "$TRAMPOLINE_FILE" ] && [ "$LEIN_TRAMPOLINE_WARMUP" = "" ]; then
604
605             TRAMPOLINE="$(cat $TRAMPOLINE_FILE)"
606
607             if [ "$INPUT_CHECKSUM" = "" ]; then
608
609                 rm $TRAMPOLINE_FILE
610
611             fi
612
613             if [ "$TRAMPOLINE" = "" ]; then
614
615                 exit $EXIT_CODE
616
617             else
618
619                 exec sh -c "exec $TRAMPOLINE"
620
621             fi
622
623         else
624
625             exit $EXIT_CODE
626
627         fi
628
629     fi
630
631 fi
时间: 2024-10-27 11:36:40

1.Storm 安装手册和提交Topology的相关文章

通过 IDE 向 Storm 集群远程提交 topology

转载: http://weyo.me/pages/techs/storm-topology-remote-submission/ 作为一个懒癌晚期患者,虽然 Storm 只需要一条命令的任务提交方式已经够简单了,但还是一直想要有种更简(tou)单(lan)的方式,比如要是在 Windows 下写完代码之后可以直接提交任务而不需要手动把 jar 包拷到服务器上再提交那定是极好的了.谷歌了一下终于在墙外找到了解决方法: Submitting a topology to Remote Storm Cl

storm源码分析之topology提交过程

storm集群上运行的是一个个topology,一个topology是spouts和bolts组成的图.当我们开发完topology程序后将其打成jar包,然后在shell中执行storm jar xxxxxx.jar xxxxxxxClass就可以将jar包上传到storm集群的nimbus上,并执行topology.本文主要分析下topology的jar包是如何上传到nimbus上的.首先我们从storm的jar命令入手,jar命令的实现位于storm根目录的bin/storm文件里.定义如

Storm 安装速记

大数据数据处理最常用的是两类模式:批处理和流计算.在open source领域,批处理最有名的组件自然是大名鼎鼎的Hadoop MapReduce,而流计算则是Storm. Storm是一个分布式的.容错的实时计算系统,目前是Apache 的一个incubator project (http://storm.incubator.apache.org/).介绍Storm基本概念的文章已经很多了,本文就不再赘述.在此仅仅速记一下笔者实际安装 Storm的步骤. 详细步骤: 1.安装Zookeeper

Storm入门教程 第三章 Storm安装部署步骤[转]

本文以Twitter Storm官方Wiki为基础,详细描述如何快速搭建一个Storm集群,其中,项目实践中遇到的问题及经验总结,在相应章节以“注意事项”的形式给出. 3.1 Storm集群组件 Storm集群中包含两类节点:主控节点(Master Node)和工作节点(Work Node).其分别对应的角色如下: 1. 主控节点(Master Node)上运行一个被称为Nimbus的后台程序,它负责在Storm集群内分发代码,分配任务给工作机器,并且负责监控集群运行状态.Nimbus的作用类似

Storm安装及使用

一:安装storm (一)安装一个zookeeper集群 注意:需要先启动zookeeper集群才可以,不然后面容易出错 (二)上传storm的安装包,解压 (三)修改配置文件storm.yaml #所使用的zookeeper集群主机 storm.zookeeper.servers: - "hadoopH5" - "hadoopH6" - "hadoopH7" #nimbus所在的主机名 nimbus.host: "hadoopH5&q

storm安装

0安装JDK wget --no-check-certificate --no-cookie --header "Cookie: oraclelicense=accept-securebackup-cookie;" http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.rpm 使用rpm -ivh jdk-8u45-linux-x64.rpm进行安装 检查安装Javac 1:centOS安装Zer

2017.4.20 远程提交topology成功

新建了一个ubuntu虚拟机开发环境,在ubuntu机器上实现了远程提交topology,但还是无法解决topology本地模式运行的问题, 在windows上也实现了远程提交topology,也是无法解决topology本地模式运行的问题, 要实现远程提交topology,首先要在本地机器安装storm,讲storm命令添加到path路径,然后修改本地storm的配置文件, 让他知道nimbus服务器的地址,topology被提交给nimbus节点,再由nimbus节点分派任务到supervi

Apache-openmeetings2.x-3.x的安装手册

最近在研究了一下openmeetings  结果 太悲催了 提供的google链接被封掉了,最没办法是用代理找找资料吧.看了一下结果 好家伙3.0版本之前还是有一些资料的 但是3.0版本以后的资料就太少了  官网提供的也不是很详细.下面是翻译的一篇官方的安装手册 希望能够给学习或者开发openmeetings的爱好者给予帮助 由于不是职业翻译 有些地方可能不是翻译的很准确 希望大家多多指出 让我们能共同解读一篇好的文章. Apache服务器上安装Windows OpenMeetings     

流式计算-Jstorm提交Topology过程

Topology是Jstorm对有向无环图的抽象,内部封装了数据来源spout和数据处理单元bolt,以及spout和bolt.bolt和bolt之间的关系,它可以被提交到Jstorm集群. 本文以Jstorm自带的SequenceTopology简单介绍一下Jstorm提交topology的过程,本文主要介绍提交过程,不涉及具体业务, 1. SequenceTopology核心方法com.alipay.dw.jstorm.example.sequence.SequenceTopology.Se