最新消息:请随时分享你的乐趣!

Flume 配置 kafka2hdfs

技术博客 磊子 783浏览 0评论

kafka to hdfs 的flume配置


kafka2hdfs.sources = src1 kafka2hdfs.channels = ch1 kafka2hdfs.sinks = s1 kafka2hdfs.sources.src1.type = org.apache.flume.source.kafka.KafkaSource kafka2hdfs.sources.src1.channels = ch1 kafka2hdfs.sources.src1.zookeeperConnect = localhost:2181 kafka2hdfs.sources.src1.topic = ghwdata kafka2hdfs.sources.src1.groupId = kafka2hdfs kafka2hdfs.sources.src1.kafka.consumer.timeout.ms = 100 kafka2hdfs.sources.src1.kafka.auto.offset.reset = smallest kafka2hdfs.channels.ch1.type = memory kafka2hdfs.channels.ch1.capacity = 1000000 kafka2hdfs.channels.ch1.transactionCapacity = 10000 #kafka2hdfs.channels.ch1.type = file #kafka2hdfs.channels.ch1.checkpointDir =/www/flume/checkpoint #kafka2hdfs.channels.ch1.dataDirs =/www/flume/data kafka2hdfs.sinks.s1.type = hdfs kafka2hdfs.sinks.s1.channel = ch1 kafka2hdfs.sinks.s1.hdfs.path = /flume/events/%y/%m/%d/ kafka2hdfs.sinks.s1.hdfs.filePrefix = events-source kafka2hdfs.sinks.s1.hdfs.round = true kafka2hdfs.sinks.s1.hdfs.roundValue = 10 kafka2hdfs.sinks.s1.hdfs.roundUnit = minute kafka2hdfs.sinks.s1.hdfs.fileType = CompressedStream kafka2hdfs.sinks.s1.hdfs.codeC = gzip kafka2hdfs.sinks.s1.hdfs.rollSize = 524288000 kafka2hdfs.sinks.s1.hdfs.rollInterval = 0 kafka2hdfs.sinks.s1.hdfs.rollHour= 1 kafka2hdfs.sinks.s1.hdfs.rollCount = 512000

转载请注明:印迹. » Flume 配置 kafka2hdfs

发表我的评论
取消评论

表情