flume配置:
#DBFile DBFile.sources = sources1 DBFile.sinks = sinks1 DBFile.channels = channels1 # DBFile-DB-Source DBFile.sources.sources1.type = spooldir DBFile.sources.sources1.spoolDir =/var/log/apache/flumeSpool//db DBFile.sources.sources1.inputCharset=utf-8 # DBFile-Sink DBFile.sinks.sinks1.type = org.apache.flume.sink.kafka.KafkaSink DBFile.sinks.sinks1.topic = DBFile DBFile.sinks.sinks1.brokerList = hdp01:6667,hdp02:6667,hdp07:6667 DBFile.sinks.sinks1.requiredAcks = 1 DBFile.sinks.sinks1.batchSize = 2000 # DBFile-Channel DBFile.channels.channels1.type = memory DBFile.channels.channels1.capacity = 10000 DBFile.channels.channels1.transactionCapacity = 1000 # DBFile-Source And Sink to the channel DBFile.sources.sources1.channels = channels1 DBFile.sinks.sinks1.channel = channels1
故障现象:第一次上传文件时,flume能很快处理文件,后面上传还是显示文件未处理。如果重启flume服务,又能立刻处理。
经测试,问题的原因在这个配置上:DBFile.sinks.sinks1.requiredAcks = -1 。
requiredAcks 的官方解释:How many replicas must acknowledge a message before its considered successfully written. Accepted values are 0 (Never wait for acknowledgement),
1 (wait for leader only), -1 (wait for all replicas) Set this to -1 to avoid data loss in some cases of leader failure.
把这个值改为1就好了。
时间: 2024-12-29 23:09:18