kafka集群搭建与prometheus监控配置

news/2025/1/16 0:04:07/

文章目录

    • 1、基于zookeeper的集群
    • 2、kafka集群安装
      • 2.1 基于Zookeeper集群的配置
      • 2.2 基于KRaft模式集群的配置
      • 2.3、启动Kafka集群
    • 3、kafka_exporter监控组件安装
      • 3.1、安装
      • 3.2、系统服务
      • 3.3、集成到prometheus
    • 4、与Grafana集成

1、基于zookeeper的集群

下载地址：https://zookeeper.apache.org/releases.html#download

tar -zxvf zookeeper-3.4.11.tar.gz -C /usr/local
cp /usr/local/zookeeper-3.4.11/conf/zoo_sample.cfg /usr/local/zookeeper-3.4.11/conf/zoo.cfg
vi /usr/local/zookeeper-3.4.11/conf/zoo.cfg##zoo.cfg内容###
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper-3.4.11/data/
clientPort=2181
server.0=192.168.28.133:2888:3888
server.1=192.168.28.136:2888:3888
server.2=192.168.28.132:2888:3888
##zoo.cfg内容###

防火墙需要打开相关端口

firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --reload

三个节点分别创建目录和myid文件，三个节点的myid值分别为0,1,2，与上边的server.0，server.1，server.2对应

mkdir -p /usr/local/zookeeper-3.4.11/data/
echo 0 > /usr/local/zookeeper-3.4.11/data/myid

三个节点分别启动

# 启动
/usr/local/zookeeper-3.4.11/bin/zkServer.sh start
# 查看状态
/usr/local/zookeeper-3.4.11/bin/zkServer.sh status
# 停止
/usr/local/zookeeper-3.4.11/bin/zkServer.sh stop
# 重启
/usr/local/zookeeper-3.4.11/bin/zkServer.sh restart

2、kafka集群安装

https://kafka.apache.org/downloads

tar -zxvf  kafka_2.13-3.2.3.tgz -C /usr/local
cd /usr/local/kafka_2.13-3.2.3

kafka集群安装支持Zookeeper和KRaft两种方式，可选择其中一种方式

2.1 基于Zookeeper集群的配置

编译配置server.properties

vi /usr/local/kafka_2.13-3.2.3/config/server.properties######## server.properties配置########
# 三个节点的broker.id分别为0,1,2
broker.id=0
# listeners三台机器修改为对应IP
listeners=PLAINTEXT://192.168.28.133:9092
zookeeper.connect=192.168.28.133:2181,192.168.28.136:2181,192.168.28.132:2181
log.dirs=/usr/local/kafka_2.13-3.2.3/logs/
######## server.properties配置########

2.2 基于KRaft模式集群的配置

防火墙

firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --zone=public --add-port=9093/tcp --permanent
firewall-cmd --reload

编译配置server.properties

vi /usr/local/kafka_2.13-3.2.3/config/kraft/server.properties# server.properties配置##################
process.roles=broker,controller
# ３台机器的node.id分别为1,2,3，不能重复
node.id=1
controller.quorum.voters=1@192.168.28.133:9093,2@192.168.28.136:9093,3@192.168.28.132:9093
# ３台机器的listeners修改为对应的IP
listeners=PLAINTEXT://192.168.28.133:9092,CONTROLLER://192.168.28.133:9093
log.dirs=/usr/local/kafka_2.13-3.2.3/kraft-combined-logs
# server.properties配置##################

生成集群统一UUID号：

./bin/kafka-storage.sh random-uuid
# 生成uuid:rrGzprV0RPyWTVek9gyTcg

用上边生成的UUID格式化kafka存储目录(所有节点都需要执行)

./bin/kafka-storage.sh format -t rrGzprV0RPyWTVek9gyTcg -c ./config/kraft/server.properties

2.3、启动Kafka集群

命令

cd /usr/local/kafka_2.13-3.2.3
# 1、启动
# （1）、zookeeper集群启动
bin/kafka-server-start.sh -daemon config/server.properties 
# （2）、KRaft集群启动
bin/kafka-server-start.sh -daemon ./config/kraft/server.properties # 2、关闭
# （1）、zookeeper集群关闭
bin/kafka-server-stop.sh config/server.properties 
# （2）、KRaft集群关闭
bin/kafka-server-stop.sh ./config/kraft/server.properties # 3、主题
# （1）、创建主题  分区数partitions 分区的副本数replication-factor
$ bin/kafka-topics.sh --create --topic topic-test --zookeeper 192.168.28.133:2181,192.168.28.136:2181,192.168.28.132:2181 --replication-factor 3 --partitions 3
# // 2.2版本建议使用--bootstrap-server代替--zookeeper
$ bin/kafka-topics.sh --create --topic topic-test --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --replication-factor 3 --partitions 3
# （2）列出所有主题
$ bin/kafka-topics.sh --list --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092
# （3）查询主题
$ bin/kafka-topics.sh --describe --topic topic-test --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092
# （4）增加主题的partition数
$ bin/kafka-topics.sh --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --alter --topic topic-test --partitions 5 
# （5）查看主题指定分区 offset 的最大值或最小值，  time 为 -1 时表示最大值，为 -2 时表示最小值：
$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic topic-test --time -1 --broker-list 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --partitions 0 # 4、生产消息
$ bin/kafka-console-producer.sh --topic topic-test --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092
This is my first test
This is my second test# 5、读取消息
# （1）从头开始
$ bin/kafka-console-consumer.sh --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --topic topic-test --from-beginning
# （2）从尾部开始,需要指定分区
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-test --offset latest --partition 0
# （3）取指定个数
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-test --offset latest --partition 0 --max-messages 1 
# （4）指定Group
$ bin/kafka-console-consumer.sh --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --topic topic-test -group group_test --from-beginning# 6、消费者Group#
# （2）消费者Group列表
$ bin/kafka-consumer-groups.sh --list --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092
# （3）查看Group详情[用来判断是否有延迟数据]
$ bin/kafka-consumer-groups.sh --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --group test_group --describe
# （4）删除Group
$ bin/kafka-consumer-groups.sh --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --group test_group --delete# 7、平衡Leader
$ bin/kafka-preferred-replica-election.sh --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092
#或 --partition:指定需要重新分配leader的partition编号
$ bin/kafka-leader-election.sh --bootstrap-server 192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092 --topic test --partition=2 --election-type preferred
# 8、自带压测工具
$ bin/kafka-producer-perf-test.sh --topic test --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=192.168.28.133:9092,192.168.28.136:9092,192.168.28.132:9092

3、kafka_exporter监控组件安装

3.1、安装

下载地址：https://github.com/danielqsj/kafka_exporter

# 解压
tar -zxvf kafka_exporter-1.6.0.linux-amd64.tar.gz -C /usr/local
# 启动
/usr/local/kafka_exporter-1.6.0.linux-amd64/kafka_exporter --kafka.server=192.168.28.133:9092 --web.listen-address=:9308
# 查看监控结果
curl http://127.0.0.1:9308/metrics

可用启动参数

参数值说明
kafka.serverkafka:9092Addresses (host:port) of Kafka server
kafka.version2.0.0Kafka broker version
sasl.enabledfalseConnect using SASL/PLAIN
sasl.handshaketrueOnly set this to false if using a non-Kafka SASL proxy
sasl.usernameSASL user name
sasl.passwordSASL user password
sasl.mechanismSASL mechanism can be plain, scram-sha512, scram-sha256
sasl.service-nameService name when using Kerberos Auth
sasl.kerberos-config-pathKerberos config path
sasl.realmKerberos realm
sasl.keytab-pathKerberos keytab file path
sasl.kerberos-auth-typeKerberos auth type. Either ‘keytabAuth’ or ‘userAuth’
tls.enabledfalseConnect to Kafka using TLS
tls.server-nameUsed to verify the hostname on the returned certificates unless tls.insecure-skip-tls-verify is given. The kafka server’s name should be given
tls.ca-fileThe optional certificate authority file for Kafka TLS client authentication
tls.cert-fileThe optional certificate file for Kafka client authentication
tls.key-fileThe optional key file for Kafka client authentication
tls.insecure-skip-tls-verifyfalseIf true, the server’s certificate will not be checked for validity
server.tls.enabledfalseEnable TLS for web server
server.tls.mutual-auth-enabledfalseEnable TLS client mutual authentication
server.tls.ca-fileThe certificate authority file for the web server
server.tls.cert-fileThe certificate file for the web server
server.tls.key-fileThe key file for the web server
topic.filter.*Regex that determines which topics to collect
group.filter.*Regex that determines which consumer groups to collect
web.listen-address:9308Address to listen on for web interface and telemetry
web.telemetry-path/metricsPath under which to expose metrics
log.enable-saramafalseTurn on Sarama logging
use.consumelag.zookeeperfalseif you need to use a group from zookeeper
zookeeper.serverlocalhost:2181Address (hosts) of zookeeper server
kafka.labelsKafka cluster name
refresh.metadata30sMetadata refresh interval
offset.show-alltrueWhether show the offset/lag for all consumer group, otherwise, only show connected consumer groups
concurrent.enablefalseIf true, all scrapes will trigger kafka operations otherwise, they will share results. WARN: This should be disabled on large clusters
topic.workers100Number of topic workers
verbosity0Verbosity log level

3.2、系统服务

vi /usr/lib/systemd/system/kafka_exporter.service

[Unit]
Description=Prometheus Kafka Exporter
After=network.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/kafka_exporter-1.6.0.linux-amd64/kafka_exporter --kafka.server=192.168.28.133:9092 --kafka.server=192.168.28.136:9092 --kafka.server=192.168.28.132:9092 --web.listen-address=:9308
Restart=on-failure
[Install]
WantedBy=multi-user.target

启动服务

systemctl daemon-reload           # 重新加载配置文件
systemctl enable kafka_exporter   # 设置开机启动
systemctl disable kafka_exporter  # 取消开机启动
systemctl start kafka_exporter    # 启动服务
systemctl stop kafka_exporter     # 关闭服务
systemctl status kafka_exporter   # 查看状态

3.3、集成到prometheus

vi /usr/local/prometheus-2.37.0.linux-amd64/prometheus.yml

  - job_name: 'kafka_export'static_configs:- targets: ['192.168.245.139:9308']labels:app: 'zxt_prod'

4、与Grafana集成

使用监控模板：https://grafana.com/grafana/dashboards/7589
在这里插入图片描述


http://www.ppmy.cn/news/256910.html

相关文章

Prometheus通过Kafka_exporter监控kafka并发送钉钉报警

Prometheus通过Kafka_exporter监控kafka并发送钉钉报警

1.下载kafka_exporter相关包： wget https://github.com/danielqsj/kafka_exporter/releases/download/v1.4.2/kafka_exporter-1.4.2.linux-amd64.tar.gz2.解压 注意：一个kafka集群（CDH kafka集群同样适用），只需要配置…
阅读更多...
rk3288-android9-ov13850

rk3288-android9-ov13850

说明：最近调试了下rk3288-android9下的ov13850,碰到了一些坑，现整理记录下，避免以后继续踩坑。 一. 代码路径 1.camera代码路径 2.hal3_camera层代码路径 3.camera_engine代码路径 二：修改内容 1. hal3用到的camera3_profile.x…
阅读更多...
prometheus 监控kafka（章节六）

prometheus 监控kafka（章节六）

前言 由于业务与ELK都使用了Kafka作为消息队列，因此考虑到业务的稳定性与可用性，使用prometheus监控kafka集群。使用的监控方式为：kafka_exporterprometheus。 提示 如果监控kafka集群的话，kafka_exporter只需在集群的一个节点安…
阅读更多...
kafka_exporter部署

kafka_exporter部署

文章目录 简单介绍环境软件包下载CSDN 部署解压 配置为系统服务kafka_exporter.service启动 Grafana监控配置prometheus.yml修改prometheus监控&插件方案总结 简单介绍 kafka_exporter是prometheus官方提供的用于监控kafka运行状态的exporter 环境 系统版本CentOS7.6.181…
阅读更多...
二、Prometheus常用exporter安装详解

二、Prometheus常用exporter安装详解

目录 一、node_exporter 1.安装配置 2.节点添加 3.状态查询 二、elasticsearch_exporter 1.安装配置 2.节点添加 3.状态查询 三、redis_exporter 1.安装配置 2.节点添加 3.状态查询 四、rabbitmq_exporter 1.安装配置 2.节点添加 3.状态查询 五、kafka_exporte…
阅读更多...
Prometheus+Grafana监控一网打尽

Prometheus+Grafana监控一网打尽

PrometheusGrafana监控 一、Prometheus介绍二、监控组件node-exportermysqld-exportercadvisorprometheus 三、Grafana 展示平台docker启动配置Data sources导入Dashboard模板Linux主机监控Mysql监控Nginx监控Redis监控PostgreSQL监控Kafka监控ElasticSearch监控 一、Prometheu…
阅读更多...
基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus监控

基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus监控

转发于：基于PrometheusGrafana搭建可视化监控服务 (一) Prometheus监控 https://www.cnblogs.com/huligong1234/p/15142981.html目录 基于PrometheusGrafana搭建可视化监控服务 (一) Prometheus及Exporter安装 一、概述 二、安装Prometheus 2.1.安装Prometheus 2.2…
阅读更多...
Prometheus监控平台配置kafka_exporter

Prometheus监控平台配置kafka_exporter

转载：(48条消息) Prometheus监控平台配置kafka_exporter_Word哥的博客-CSDN博客_kafka_exporter Kafka现有开源的集群监控方案：kafka-manager、kafka-monitor、kafka-eagle、KafkaOffsetMonitor，但有所限制监控指标被固化，不易扩…
阅读更多...
最新文章

Copyright @ 2022~2023