最新消息:请随时分享你的乐趣!

flume生产环境配置

技术博客 磊子 1185浏览 0评论
目录
[隐藏]

逻辑

图中flume4 为两个,这样图需要重画。
flume_UserGuide_image02

collector配置

#1. info
ghw.sources = src
ghw.channels =ch1
ghw.sinks = ss1

#2. source
ghw.sources.src.type = avro
ghw.sources.src.bind = 0.0.0.0
ghw.sources.src.port = 4141

#3. channels
ghw.channels.ch1.type = file
ghw.channels.ch1.checkpointDir = /www/hadoop/flume/channel/checkpoint
ghw.channels.ch1.dataDirs = /www/hadoop/flume/channel/data
ghw.channels.ch1.transactionCapacity = 1000

#4. sinks
ghw.sinks.ss1.type = hdfs
ghw.sinks.ss1.hdfs.fileType = DataStream
ghw.sinks.ss1.hdfs.writeFormat = Text
ghw.sinks.ss1.hdfs.path = hdfs://ghdata-namenode:8020/user/flume/data-events/%Y-%m
ghw.sinks.ss1.hdfs.filePrefix = data-%Y-%m-%d
ghw.sinks.ss1.hdfs.useLocalTimeStamp = true
ghw.sinks.ss1.hdfs.inUseSuffix = .doing
ghw.sinks.ss1.hdfs.rollCount=0
#120M
ghw.sinks.ss1.hdfs.rollSize=125829120  
ghw.sinks.ss1.hdfs.rollInterval = 0
ghw.sinks.ss1.hdfs.batchSize=100
ghw.sinks.ss1.hdfs.minBlockReplicas=1


#5. binding
ghw.sources.src.channels = ch1
ghw.sinks.ss1.channel = ch1

agent配置

agent-ghw.sources = src
agent-ghw.channels =ch1
agent-ghw.sinks = ss1 ss2
agent-ghw.sinkgroups = group1
#source
agent-ghw.sources.src.type = exec
agent-ghw.sources.src.command = tail -F /www/applog/data/ghw.data

#3. channels
agent-ghw.channels.ch1.type = file
agent-ghw.channels.ch1.checkpointDir = /www/hadoop/flume/channel/checkpoint
agent-ghw.channels.ch1.dataDirs = /www/hadoop/flume/channel/data
agent-ghw.channels.ch1.transactionCapacity = 1000

#sink-groups
agent-ghw.sinkgroups.group1.sinks=ss1 ss2
agent-ghw.sinkgroups.group1.processor.type = failover
agent-ghw.sinkgroups.group1.processor.priority.k1 = 5
agent-ghw.sinkgroups.group1.processor.priority.k2 = 10
agent-ghw.sinkgroups.group1.processor.maxpenalty = 10000

#sinks
agent-ghw.sinks.ss1.type = avro
agent-ghw.sinks.ss1.channel = ch1
agent-ghw.sinks.ss1.hostname = ghdata-node1
agent-ghw.sinks.ss1.port = 4141

agent-ghw.sinks.ss2.type = avro
agent-ghw.sinks.ss2.channel = ch1
agent-ghw.sinks.ss2.hostname = ghdata-node2
agent-ghw.sinks.ss2.port = 4141

#channels
agent-ghw.sources.src.channels = ch1
agent-ghw.sinks.ss1.channel = ch1
agent-ghw.sinks.ss2.channel = ch1

转载请注明:印迹. » flume生产环境配置

发表我的评论
取消评论

表情