flume内置的ChannelSelector有两种,分别是Replicating和Multiplexing。
Replicating类型的ChannelSelector会针对每一个Event,拷贝到所有的Channel中,这是默认的ChannelSelector。
replicating类型的ChannelSelector例子如下
1 a1.sources = r1 2 a1.channels = c1 c2 # 如果有100个Event,那么c1和c2中都会有这100个事件 3 4 a1.channels.c1.type = memory 5 a1.channels.c1.capacity = 1000 6 a1.channels.c1.transactionCapacity = 100 7 8 9 a1.channels.c2.type = memory 10 a1.channels.c2.capacity = 1000 11 a1.channels.c2.transactionCapacity = 100
Multiplexing类型的ChannelSelector会根据Event中Header中的某个属性决定分发到哪个Channel。
multiplexing类型的ChannelSelector例子如下:
1 a1.sources = r1 2 3 a1.sources.source1.selector.type = multiplexing 4 a1.sources.source1.selector.header = validation # 以header中的validation对应的值作为条件 5 a1.sources.source1.selector.mapping.SUCCESS = c2 # 如果header中validation的值为SUCCESS,使用c2这个channel 6 a1.sources.source1.selector.mapping.FAIL = c1 # 如果header中validation的值为FAIL,使用c1这个channel 7 a1.sources.source1.selector.default = c1 # 默认使用c1这个channel
a1.sources.source1.selector.header = validation # 以header中的validation对应的值作为条件 同理,如下conf文件:
1 # example.conf: A single-node Flume configuration 2 3 # Name the components on this agent 4 a1.sources = r1 5 a1.sinks = k1 6 a1.channels = c1 7 8 # Describe/configure the source 9 a1.sources.r1.type = netcat 10 a1.sources.r1.bind = localhost 11 a1.sources.r1.port = 44444 12 13 a1.sources.r1.interceptors = i1 14 a1.sources.r1.interceptors.i1.type = regex_extractor 15 a1.sources.r1.interceptors.i1.regex = (\\d):(\\d):(\\d) 16 a1.sources.r1.interceptors.i1.serializers = s1 s2 s3 17 a1.sources.r1.interceptors.i1.serializers.s1.name = ip 18 a1.sources.r1.interceptors.i1.serializers.s2.name = domain 19 a1.sources.r1.interceptors.i1.serializers.s3.name = course 20 21 a1.sources.r1.selector.type = multiplexing 22 a1.sources.r1.selector.header = course 23 a1.sources.r1.selector.default = c1 24 25 # Describe the sink 26 a1.sinks.k1.type = logger 27 28 # Use a channel which buffers events in memory 29 a1.channels.c1.type = memory 30 a1.channels.c1.capacity = 1000 31 a1.channels.c1.transactionCapacity = 100 32 33 # Bind the source and sink to the channel 34 a1.sources.r1.channels = c1 35 a1.sinks.k1.channel = c1
source r1中的头部有IP、Domain和cource三种信息,而r1的selector.header = course,表示selector只对IP,Domain和Cource中的Cource进行判断选择,然后再划分channel。
时间: 2024-10-08 08:55:10