hello flume (Ubuntu 下 flume1.5单机版安装以及简单入门示例)

1,下载最新的flume安装包：

wget http://www.apache.org/dist/flume/stable/apache-flume-1.5.2-bin.tar.gz

2，在安装目录解压：

 tar -zxvf apache-flume-1.5.2-bin.tar.gz

3,设置环境变量

export JAVA_HOME=/usr ;
export FLUME_HOME=/home/joeyon/apache-flume-1.5.2-bin;
export PATH=$PATH:FLUME_HOME/bin

4,配置一个简单单机的测试案例

</pre>4.1在conf目录下新建一个文件，example-conf.properties，内容如下：<pre name="code" class="html"># Describe the source
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
# 将数据输出至日志中
a1.sinks.k1.type = logger

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

4.2在conf目录下新建一个源文件file01并向其中写入数据

# Use a channel which buffers events in memory
   touch file01;

   echo "hello world" > file01

4.3 启动agent代理

flume-ng agent -c . -f /home/joeyon/apache-flume-1.5.2-bin/conf/example-conf.properties  -n a1 -Dflume.root.logger=INFO,console

启动成功日志如下：

+ exec /usr/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /data/projects/apache-flume-1.5.2-bin/conf/example-conf.properties -n a1
2015-01-21 13:44:43,231 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2015-01-21 13:44:43,239 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/data/projects/apache-flume-1.5.2-bin/conf/example-conf.properties
2015-01-21 13:44:43,248 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: a1
2015-01-21 13:44:43,249 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2015-01-21 13:44:43,249 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2015-01-21 13:44:43,261 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2015-01-21 13:44:43,262 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2015-01-21 13:44:43,271 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel c1 type memory
2015-01-21 13:44:43,276 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel c1
2015-01-21 13:44:43,277 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source r1, type avro
2015-01-21 13:44:43,297 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: k1, type: logger
2015-01-21 13:44:43,301 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel c1 connected to [r1, k1]
2015-01-21 13:44:43,309 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: localhost, port: 44444 } }} sinkRunners:{k1=SinkRunner: { policy:[email protected] counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2015-01-21 13:44:43,320 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel c1
2015-01-21 13:44:43,383 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2015-01-21 13:44:43,383 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2015-01-21 13:44:43,387 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink k1
2015-01-21 13:44:43,388 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source r1
2015-01-21 13:44:43,388 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:228)] Starting Avro source r1: { bindAddress: localhost, port: 44444 }...
2015-01-21 13:44:43,771 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2015-01-21 13:44:43,771 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started
2015-01-21 13:44:43,772 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:253)] Avro source r1 started.

4.4.启动avro-client客户端向agent代理发送数据：

flume-ng avro-client -c . -H localhost -p 44444 -F /home/joeyon/apache-flume-1.5.2-bin/conf/file01

日志如下：

+ exec /usr/bin/java -Xmx20m -cp '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/lib/*' -Djava.library.path= org.apache.flume.client.avro.AvroCLIClient -H localhost -p 44444 -F /home/joeyon/apache-flume-1.5.2-bin/conf/file01

4.5查看agen控制台输出代理端输出如下则收集成功：

2015-01-21 13:50:13,276 (New I/O server boss #1 ([id: 0xc0b7fffb, /127.0.0.1:44444])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] OPEN
2015-01-21 13:50:13,277 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] BOUND: /127.0.0.1:44444
2015-01-21 13:50:13,278 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 => /127.0.0.1:44444] CONNECTED: /127.0.0.1:3809
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] DISCONNECTED
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] UNBOUND
2015-01-21 13:50:13,703 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x9d115219, /127.0.0.1:3809 :> /127.0.0.1:44444] CLOSED
2015-01-21 13:50:13,704 (New I/O  worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed(NettyServer.java:209)] Connection to /127.0.0.1:3809 disconnected.
2015-01-21 13:50:17,411 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 3E 3E 3E 3E 3E hello world>>>>> }
2015-01-21 13:50:17,411 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 6D 79 20 6E 61 6D 65 20 69 73 20 6A 6F 65 79 6F my name is joeyo }
2015-01-21 13:50:17,412 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 77 65 6C 63 6F 6D 20 74 6F 20 63 68 69 6E 61 3E welcom to china> }
时间： 2024-10-07 14:12:52

hello flume (Ubuntu 下 flume1.5单机版安装以及简单入门示例)的相关文章

Ubuntu 下ftp服务器的安装配置

Ubuntu 下ftp服务器的安装配置 FTP服务器是平时应用最为广泛的服务之一.VSFTP是Very Secure FTP的缩写,意指非常安全的FTP服务.VSFTP功能强大,通过结合本地系统的用户认证模块及其多功能的配置项目,可以快速有效的搭建强大的多用户FTP服务. 首先我们考虑搭建FTP服务需要关注的都有哪些方面?比如我们最经常关注的有: 1.如何添加FTP用户?  2.如何临时冻结某FTP用户?  3.FTP用户登入后的默认目录?是否可以改变?  4.如何锁定FTP用户可访问的目录范围

Linux/ubuntu下的boost库安装

我一直都没有写博客的习惯,最近正好在研究linux下的开发(目前也只是粗粗的研究),且用到了boost库,就乘此机会写点什么,最起码记录一下我在安装boost的一些步骤,主要给和我一样的linux开发新手们提供点借鉴(当然如果看到这篇文档的话 ),作者(jwybobo2007). 1.下载boost库 这个我就不说啥了,去官网看一下就能下到:www.boost.org ,现在的版本更新到了1.44 2.执行:sudo apt-get install build-essential 不为别的,就是

Ubuntu下deb包的安装方法 (zz)

Ubuntu下deb包的安装方法 分类: Ubuntu10使用技巧 2010-10-11 23:49 42969人阅读 评论(3) 收藏 举报 ubuntudebdebianlinux deb是debian linus的安装格式,跟red hat的rpm非常相似,最基本的安装命令是:dpkg -i file.deb dpkg 是Debian Package的简写,是为Debian 专门开发的套件管理系统,方便软件的安装.更新及移除.所有源自Debian的Linux发行版都使用dpkg,例如Ubu

Ubuntu 下iscsi initiator的安装与使用

Ubuntu下比较方便好用的initiator是open iscsi,这里将要简要介绍它的使用方法: 1.安装: sudo apt-get install open-iscsi 2.chap设置 如果ip-san上设置了chap(没有设置chap的话,可以略去这一步),则要修改/etc/iscsi/iscsid.conf,找到以下内容,去掉前面的#,然后将用户名.密码改为san上设置好的密码 node.session.auth.authmethod = CHAPnode.session.auth

【转】Ubuntu下deb包的安装方法

[转]Ubuntu下deb包的安装方法 deb是debian linus的安装格式,跟red hat的rpm非常相似,最基本的安装命令是:dpkg -i file.deb dpkg 是Debian Package的简写,是为Debian 专门开发的套件管理系统,方便软件的安装.更新及移除.所有源自Debian的Linux发行版都使用dpkg,例如Ubuntu.Knoppix 等.以下是一些 Dpkg 的普通用法: 1.dpkg -i <package.deb>安装一个 Debian 软件包,如

ubuntu下源码方式安装php5.4

一.安装前准备 下载php-5.4.13.tar.gz最新版本放到/user/src目录下 二.安装 因为在安装php过程中,会依赖安装很多库,为了不让你反复安装,建议按步骤操作 安装autoconf工具 sudo apt-get install autoconf 安装libxml2库 sudo apt-get install libxml2-dev 安装libbz2-dev sudo apt-get install libbz2-dev 安装curl sudo apt-get install

转载：Ubuntu下deb包的安装方法

转载:Ubuntu下deb包的安装方法,http://blog.csdn.net/kevinhg/article/details/5934462 deb是debian linus的安装格式,跟red hat的rpm非常相似,最基本的安装命令是:dpkg -i file.deb dpkg 是Debian Package的简写,是为Debian 专门开发的套件管理系统,方便软件的安装.更新及移除.所有源自Debian的Linux发行版都使用dpkg,例如Ubuntu.Knoppix 等.以下是一些

04.ubuntu下kvm 命令行安装64位ubuntu报&quot;Couldn&#39;t find hvm kernel for Ubuntu tree.&quot;的问题

1.安装ubuntu时使用的virt-install的配置: virt-install \ --name test4 \ --ram 1024 \ --disk path=/data/01_ubuntu/ubuntu4.img,size=6 \ --vcpus 1 \ --hvm \ --os-type linux \ --network network=default \ --os-variant ubuntuquantal \ --graphics none \ --console pty,

ubuntu下erlang man的安装

下载 http://www.erlang.org/download/otp_doc_man_17.1.tar.gz 找到erlang 安装目录 解压 otp_doc_man_17.1.tar.gz sudo cp -r man /usr/local/lib/erlang erl -man lists 查看结果 ubuntu下erlang man的安装