1. 错误提示
2016-04-21 05:40:51,393 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
java.lang.IllegalStateException: Channel closed [channel=channel1]. Due to java.io.EOFException: null
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:329)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:376)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:79)
at org.apache.flume.channel.file.Log.replay(Log.java:417)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
2016-04-21 05:40:51,972 (pool-4-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:256)] FATAL: Spool Directory source source1: { spoolDir: /home/hadoop_admin/movielog }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.lang.IllegalStateException: Channel closed [channel=channel1]. Due to java.io.EOFException: null
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:329)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:181)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:79)
at org.apache.flume.channel.file.Log.replay(Log.java:417)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
... 7 more
配置文件：
agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 # Each channel‘s type is defined. agent1.channels.channel1.type = file agent1.channels.channel1.checkpointDir = /home/hadoop_admin/flumeTemp/fchannel/spool/checkpoint agent1.channels.channel1.dataDirs = /home/hadoop_admin/flumeTemp/fchannel/spool/data agent1.channels.channel1.capacity = 200000000 agent1.channels.channel1.keep-alive = 30 agent1.channels.channel1.write-timeout = 30 agent1.channels.channel1.checkpoint-timeout = 600 # For each one of the sources, the type is defined agent1.sources.source1.type = spooldir agent1.sources.source1.inputCharset = GBK agent1.sources.source1.spoolDir =/home/hadoop_admin/movielog agent1.sources.source1.fileHeader = true agent1.sources.source1.deletePolicy = immediate agent1.sources.source1.batchSize = 1000 agent1.sources.source1.channels = channel1 # Each sink‘s type must be defined agent1.sinks.sink1.type = hdfs agent1.sinks.sink1.channel = channel1 agent1.sinks.sink1.hdfs.path = hdfs://master:9000/flumeTest agent1.sinks.sink1.hdfs.filePrefix = master- agent1.sinks.sink1.hdfs.writeFormat = Text agent1.sinks.sink1.hdfs.fileType = DataStream agent1.sinks.sink1.hdfs.rollInterval = 0 agent1.sinks.sink1.hdfs.idleTimeout = 0 agent1.sinks.sink1.hdfs.rollCount = 0 agent1.sinks.sink1.hdfs.rollSize = 102400 agent1.sinks.sink1.hdfs.batchSize = 1000 agent1.sinks.sink1.hdfs.callTimeout = 30000
2. 解决方法
链接： http://serverfault.com/questions/690588/flume-error-log-while-using-filechannel
只要把配置文件中的关于channel配置checkpointDir和dataDir目录清空就可以
时间： 2024-10-17 03:06:37