org.apache.flume.conf.ConfigurationException: Channel c1 not in active set.

1 错误详细信息

WARN conf.FlumeConfiguration: Could not configure sink  k1 due to: Channel c1 not in active set.
org.apache.flume.conf.ConfigurationException: Channel c1 not in active set.
	at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:685)
	at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:347)
	at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:213)
	at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:127)
	at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:109)
	at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:189)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:89)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/02/21 03:50:54 WARN conf.FlumeConfiguration: Could not configure sink  k2 due to: Channel k2 not in active set.
org.apache.flume.conf.ConfigurationException: Channel k2 not in active set.
	at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:685)
	at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:347)
	at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:213)
	at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:127)
	at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:109)
	at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:189)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:89)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2 错误原因

  配置文件写错了

# set channel for sinks
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = k2

  

  

3 正确应该是配置c2 也就是channel

# set channel for sinks
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

4 完整的配置文件

# List the sources, sinks and channels for the agent
# <Agent>.sources = <Source1>
# <Agent>.sinks = <Sink1> <Sink2>
# <Agent>.channels = <Channel1> <Channel2>
#
# # set list of channels for source (separated by space)
# <Agent>.sources.<Source1>.channels = <Channel1> <Channel2>
#
# # set channel for sinks
# <Agent>.sinks.<Sink1>.channel = <Channel1>
# <Agent>.sinks.<Sink2>.channel = <Channel2>
#
# <Agent>.sources.<Source1>.selector.type = replicating

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1 c2

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /tmp/logs/cmcc.log

# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = cmcc1
a1.sinks.k1.brokerList = hdp1:9092,hdp2:9092,hdp3:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20

a1.sinks.k2.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k2.topic = cmcc2
a1.sinks.k2.brokerList = hdp1:9092,hdp2:9092,hdp3:9092
a1.sinks.k2.requiredAcks = 1
a1.sinks.k1.batchSize = 20

a1.sinks.k2.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k2.topic = cmcc2
a1.sinks.k2.brokerList = hdp1:9092,hdp2:9092,hdp3:9092
a1.sinks.k2.requiredAcks = 1
a1.sinks.k2.batchSize = 20

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.channels.c2.type = file
a1.channels.c2.checkpointDir = /tmp/flume/checkpoint
a1.channels.c2.dataDirs = /tmp/flume/data

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 c2

# set channel for sinks
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
 # #
a1.sources.r1.selector.type = replicating
 #

  

原文地址:https://www.cnblogs.com/QuestionsZhang/p/10413877.html

时间: 2024-11-08 20:17:23

org.apache.flume.conf.ConfigurationException: Channel c1 not in active set.的相关文章

WARN conf.FlumeConfiguration: Could not configure sink sink1 due to: No channel configured for sink: sink1 org.apache.flume.conf.ConfigurationException: No channel configured for sink: sink1

1.错误如下所示,启动flume采集文件到hdfs案例的时候,出现如下所示的错误: 大概是说No channel configured for sink,所以应该是sink哪里配置出现了错误,百度了一下,然后检查了一下自己的配置: 1 18/04/24 08:31:02 WARN conf.FlumeConfiguration: Could not configure sink sink1 due to: No channel configured for sink: sink1 2 org.a

实时事件统计项目:优化flume:用file channel代替mem channel

背景:利用kafka+flume+morphline+solr做实时统计. solr从12月23号开始一直没有数据.查看日志发现,因为有一个同事加了一条格式错误的埋点数据,导致大量error. 据推断,是因为使用mem channel占满,消息来不及处理,导致新来的数据都丢失了. 修改flume使用file channel: kafka2solr.sources = source_from_kafka kafka2solr.channels = file_channel kafka2solr.s

Apache Flume 安装文档、日志收集

简介: 官网 http://flume.apache.org 文档 https://flume.apache.org/FlumeUserGuide.html hadoop 生态系统中,flume 的职责是收集数据,一般用作收集各种日志数据. Source -> Channel -> Sink 这是一个基本的工作流程. Source 定义了数据从哪里来,Channel 是一个数据暂存的位置 ( disk / mem ),Sink 定义将数据流向哪里! 一.flume 安装 shell >

apache flume agent安装

1.Flume Agent安装(采用spoolDir模式以便获取系统.应用等日志信息) 注:使用jyapp用户安装 一台虚拟机部署了多个JAVA应用同时需要部署多个flume-agent进行监控时, 需要调整如下配置文件: a    flume-agent/conf/app.conf中的spool_Dir参数 b    flume-agent/conf/flume-env.sh中的jmxremote.port参数 c    logback.xml中fileNamePattern参数 d    f

apache flume collector 安装

2.Flume Collector安装(通过extends AbstractSink implements Configurable,直接写入数据库) 2.1 安装环境 系统: CentOS release 6.6 软件: flume-collector.tar.gz 2.2 安装步骤 2.2.1 部署Flume Collector 具体脚本(jyapp用户): cd  /home/jyapp tar  -zxvf  flume-collector.tar.gz cd  flume-collec

【Apache Flume系列】Flume-ng failover 以及Load balance测试及注意事项

好久没写博客了.最近在研究storm.flume和kafka.今天给大伙写下我测试flume failover以及load balance的场景以及一些结论: 测试环境包含5个配置文件,也就是5个agent. 一个主的配置文件,也就是我们配置failover以及load balance关系的配置文件(flume-sink.properties),这个文件在下面的场景 会变动,所以这里就不列举出来了,会在具体的场景中写明: 其他4个配置文件类似: #Name the compents on thi

那些年踏过的Apache Flume之路

Flume作为日志采集系统,有着独特的应用和优势,那么Flume在实际的应用和实践中到底是怎样的呢?让我们一起踏上Flume之路. 1.  什么是Apache Flume (1)Apache Flume简单来讲是高性能.分布式的日志采集系统,和sqoop同属于数据采集系统组件,但是sqoop用来采集关系型数据库数据,而Flume用来采集流动型数据. (2)Flume名字来源于原始的近乎实时的日志数据采集工具,现在被广泛用于任何流事件数据的采集,它支持从很多数据源聚合数据到HDFS. (3)Flu

使用Apache Flume抓取数据(1)

使用Apache Flume抓取数据,怎么来抓取呢?不过,在了解这个问题之前,我们必须明确ApacheFlume是什么? 一.什么是Apache Flume Apache Flume是用于数据采集的高性能系统 ,名字来源于原始的近乎实时的日志数据采集工具,现在广泛用于任何流事件数据的采集,支持从很多数据源聚合数据到HDFS. 最初由Cloudera开发 ,在2011年贡献给了Apache基金会 ,在2012年变成了Apache的顶级项目,Flume OG升级换代成了Flume NG. Flume

Flume内置channel,source,sink汇总

由于经常会使用到Flume的一些channel,source,sink,于是为了方便将这些channel,source,sink汇总出来,也共大家访问. Component Interface Type Alias Implementation Class *.Channel memory *.channel.MemoryChannel *.Channel jdbc *.channel.jdbc.JdbcChannel *.Channel file *.channel.file.FileChan