Apache Hadoop生态-目录汇总-持续更新

系统环境：centos7

Java环境：Java8

案例只演示通道流程，其中Source，channel，Sink的种类按需调整

1：案例需求-实现聚合拓扑结构

Flume-1 监控文件/tmp/group3/log1.log

Flume-2 监控文件/tmp/group3/log2.log

Flume-1 与 Flume-2 将数据发送给Flume-3，Flume-3 将最终数据打印到控制台。

3：实现步骤：

2.1：实现flume1.conf - sink端口4141

vim flume1.conf# 1：定义组件
a1.sources = r1
a1.sinks = k1
a1.channels = c1# 2：定义source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /tmp/group3/log1.log
a1.sources.r1.shell = /bin/bash -c# 3：定义channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100# 4：定义sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = worker215
a1.sinks.k1.port = 4111# 5：定义关联关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2.2：实现flume2.conf- sink端口4141

vim flume2.conf# 1：定义组件
a2.sources = r1
a2.sinks = k1
a2.channels = c1# 2：定义source
a2.sources.r1.type = exec
a2.sources.r1.command = tail -F /tmp/group3/log2.log
a2.sources.r1.shell = /bin/bash -c# 3：定义channel
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100# 4：定义sink
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = worker215
a2.sinks.k1.port = 4111# 5：定义关联关系
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

2.3：实现flume3.conf - 监听端口4141

vim flume3.conf# 1：定义组件
a3.sources = r1
a3.sinks = k1
a3.channels = c1# 2：定义source
a3.sources.r1.type = avro
a3.sources.r1.bind = worker215
a3.sources.r1.port = 4111# 3：定义channel
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100# 4：定义sink
a3.sinks.k1.type = logger# 5：定义关联关系
a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1

3：启动传输链路

1）启动flume3，监听端口4141-收集其他服务传输的数据
flume-ng agent --name flume3 --conf-file flume3.conf  -Dflume.root.logger=INFO,console2）启动flume1，sink端口4141
flume-ng agent --name flume1 --conf-file flume1.conf3）启动flume2，sink端口4141
flume-ng agent --name flume2 --conf-file flume2.conf