1、Flume Agent安装（采用spoolDir模式以便获取系统、应用等日志信息）
注：使用jyapp用户安装
一台虚拟机部署了多个JAVA应用同时需要部署多个flume-agent进行监控时，
需要调整如下配置文件：
a flume-agent/conf/app.conf中的spool_Dir参数
b flume-agent/conf/flume-env.sh中的jmxremote.port参数
c logback.xml中fileNamePattern参数
d flume_spooldir.sh中的dir参数
1.1 安装环境
系统:
CentOS release 6.6
软件:
flume-agent.tar.gz
flume-agent.tar.gz和flume_spooldir.sh文件下载路径：ftp://172.21.100.61/pro/flume/flume-agent
1.2 安装步骤
1.2.1 部署Flume Agent
具体脚本（jyapp用户）：
|
cd /home/jyapp mkdir flume_logs tar -zxvf flume-agent.tar.gz cd flume-agent #修改flume环境变量 vi conf/flume-env.sh 具体配置解析：（注意：红色需修改） ------------------------------------------------------ export JAVA_OPTS="-Xms100m -Xmx100m -Dflume.root.logger=INFO,LOGFILE -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=本机IP -Dcom.sun.management.jmxremote.port=9999 ------------------------------------------------------------------------------------- #修改agent脚本 vi conf/app.conf 具体配置解析：（注意：红色需修改） ------------------------------------------------------ app.sources = r1 app.sinks = k1 k2 app.channels = c1 app.sources.r1.type = spooldir app.sources.r1.spoolDir = /home/jyapp/flume_logs #定义监控的日志目录 app.sources.r1.deletePolicy = immediate app.sources.r1.ignorePattern = ^(.)*\\.tmp$ #忽略.tmp文件 app.sources.r1.deserializer = MULTILINE #使用多行反序列化器 app.sources.r1.deserializer.maxLineLength = 60000 app.sources.r1.interceptors = i1 i2 i3 app.sources.r1.interceptors.i1.type = timestamp #设置时间戳 app.sources.r1.interceptors.i2.type = static app.sources.r1.interceptors.i2.key = flume.client.ip app.sources.r1.interceptors.i2.value = 客户机IP地址 #设置IP地址 app.sources.r1.interceptors.i3.type = static app.sources.r1.interceptors.i3.key = flume.client.system.flag app.sources.r1.interceptors.i3.value = 客户机系统标示 #设置系统标示，如S001 app.channels.c1.type = memory app.channels.c1.capacity = 1000 app.channels.c1.transactionCapacity = 1000 app.channels.c1.keep-alive = 3 app.channels.c1.byteCapacityBufferPercentage = 30 app.channels.c1.byteCapacity = 104857600 app.sinks.k1.type = avro app.sinks.k1.hostname = 172.19.100.92 app.sinks.k1.port = 44466 app.sinks.k2.type = avro app.sinks.k2.hostname = 172.19.100.102 app.sinks.k2.port = 44466 app.sources.r1.channels = c1 app.sinks.k1.channel = c1 app.sinks.k2.channel = c1 app.sinkgroups = g1 app.sinkgroups.g1.sinks = k1 k2 app.sinkgroups.g1.processor.type = load_balance app.sinkgroups.g1.processor.backoff = true app.sinkgroups.g1.processor.selector = round_robin ------------------------------------------------------------------------------------- 批量替换命令参考： #修改flume-env.sh sed -i ‘s/172.18.100.159/172.19.100.73/g;s/S000/S001/g‘ #修改app.conf sed -i ‘s/172.18.100.159/172.19.100.73/g;s/S000/S001/g‘ 后台启动agent cd /home/jyapp/flume-agent bin/flume-ng agent --conf conf --conf-file conf/app.conf --name app & 停止agent，使用kill命令 |