监控k8s controller和scheduler,创建serviceMonitor以及Rules

news/2025/2/19 10:08:51/

目录

一、修改kube-controller和kube-schduler的yaml文件

二、创建service、endpoint、serviceMonitor

三、Prometheus验证

四、创建PrometheusRule资源

五、Prometheus验证


直接上干货

一、修改kube-controller和kube-schduler的yaml文件

注意:修改时要一个节点一个节点的修改,等上一个修改的节点服务正常启动后再修改下个节点

kube-controller文件路径:/etc/kubernetes/manifests/kube-controller-manager.yaml
kube-scheduler文件路径:/etc/kubernetes/manifests/kube-scheduler.yamlvim /etc/kubernetes/manifests/kube-controller-manager.yaml
vim /etc/kubernetes/manifests/kube-scheduler.yaml

修改后系统监听的端口从127.0.0.1变成了0.0.0.0

二、创建service、endpoint、serviceMonitor

kube-controller-monitor.yaml

apiVersion: v1
kind: Service
metadata:labels:k8s-app: kube-controller-managername: kube-controller-manage-monitornamespace: kube-system
spec:ports:- name: https-metricsport: 10257protocol: TCPtargetPort: 10257sessionAffinity: Nonetype: ClusterIP
--- 
apiVersion: v1
kind: Endpoints
metadata:labels:k8s-app: kube-controller-managername: kube-controller-manage-monitornamespace: kube-system
subsets:
- addresses:- ip: 10.50.238.191- ip: 10.50.107.48- ip: 10.50.140.151ports:- name: https-metricsport: 10257protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:k8s-app: kube-controller-managername: kube-controller-managernamespace: kube-system
spec:endpoints:- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokeninterval: 30sport: https-metricsscheme: httpstlsConfig:insecureSkipVerify: truejobLabel: k8s-appnamespaceSelector:matchNames:- kube-systemselector:matchLabels:k8s-app: kube-controller-manager

kube-scheduler-monitor.yaml

apiVersion: v1
kind: Service
metadata:labels:k8s-app: kube-schedulername: kube-scheduler-monitornamespace: kube-system
spec:ports:- name: https-metricsport: 10259protocol: TCPtargetPort: 10259sessionAffinity: Nonetype: ClusterIP
--- 
apiVersion: v1
kind: Endpoints
metadata:labels:k8s-app: kube-schedulername: kube-scheduler-monitornamespace: kube-system
subsets:
- addresses:- ip: 10.50.238.191- ip: 10.50.107.48- ip: 10.50.140.151ports:- name: https-metricsport: 10259protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:k8s-app: kube-schedulername: kube-schedulernamespace: kube-system
spec:endpoints:- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokeninterval: 30sport: https-metricsscheme: httpstlsConfig:insecureSkipVerify: truejobLabel: k8s-appnamespaceSelector:matchNames:- kube-systemselector:matchLabels:k8s-app: kube-scheduler

root@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/serviceMonitor/kubernetes-cluster# kubectl  apply -f ./
service/kube-controller-manage-monitor created
endpoints/kube-controller-manage-monitor created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
service/kube-scheduler-monitor created
endpoints/kube-scheduler-monitor created
servicemonitor.monitoring.coreos.com/kube-scheduler created

三、Prometheus验证

四、创建PrometheusRule资源

kube-controller-rules.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:annotations:meta.helm.sh/release-namespace: cattle-monitoring-systemprometheus-operator-validated: "true"generation: 3labels:app: rancher-monitoringapp.kubernetes.io/instance: rancher-monitoringapp.kubernetes.io/part-of: rancher-monitoringname: kube-controller-managernamespace: cattle-monitoring-system
spec:groups:- name: kube-controller-manager.rulerules:- alert: K8SControllerManagerDownexpr: absent(up{job="kube-controller-manager"} == 1)for: 1mlabels:severity: criticalcluster: manage-prodannotations:description: There is no running K8S controller manager. Deployments and replication controllers are not making progress.summary: No kubernetes controller manager are reachable- alert: K8SControllerManagerDownexpr: up{job="kube-controller-manager"} == 0for: 1mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is down. {{ $labels.instance }} isn't reachablesummary: kubernetes controller manager is down- alert: K8SControllerManagerUserCPUexpr: sum(rate(container_cpu_user_seconds_total{pod=~"kube-controller-manager.*",container_name!="POD"}[5m]))by(pod) > 5for: 5mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is user cpu time > 5s. {{ $labels.instance }} isn't reachablesummary: kubernetes controller 负载较高超过5s- alert: K8SControllerManagerUseMemoryexpr: sum(rate(container_memory_usage_bytes{pod=~"kube-controller-manager.*",container_name!="POD"}[5m])/1024/1024)by(pod) > 20for: 5mlabels:severity: infocluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is use memory More than 20MBsummary: kubernetes controller 使用内存超过20MB- alert: K8SControllerManagerQueueTimedelayexpr: histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job="kubernetes-controller-manager"}[5m])) by(le)) > 10for: 5mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is QueueTimedelay More than 10ssummary: kubernetes controller 队列停留时间超过10秒,请检查ControllerManager

kube-scheduler-rules.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:annotations:meta.helm.sh/release-namespace: cattle-monitoring-systemprometheus-operator-validated: "true"generation: 3labels:app: rancher-monitoringapp.kubernetes.io/instance: rancher-monitoringapp.kubernetes.io/part-of: rancher-monitoringname: kube-schedulernamespace: cattle-monitoring-system
spec:groups:- name: kube-scheduler.rulerules:- alert: K8SSchedulerDownexpr: absent(up{job="kube-scheduler"} == 1)for: 1mlabels:severity: criticalcluster: manage-prodannotations:description: "There is no running K8S scheduler. New pods are not being assigned to nodes."summary: "all k8s scheduler is down"- alert: K8SSchedulerDownexpr: up{job="kube-scheduler"} == 0for: 1mlabels:severity: warningcluster: manage-prodannotations:description: "K8S scheduler {{ $labels.instance }} is no running. New pods are not being assigned to nodes."summary: "k8s scheduler {{ $labels.instance }} is down"- alert: K8SSchedulerUserCPUexpr: sum(rate(container_cpu_user_seconds_total{pod=~"kube-scheduler.*",container_name!="POD"}[5m]))by(pod) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: "kubernetes scheduler {{ $labels.instance }} is user cpu time > 1s. {{ $labels.instance }} isn't reachable"summary: "kubernetes scheduler 负载较高超过1s,当前值为{{$value}}"- alert: K8SSchedulerUseMemoryexpr: sum(rate(container_memory_usage_bytes{pod=~"kube-scheduler.*",container_name!="POD"}[5m])/1024/1024)by(pod) > 20for: 5mlabels:severity: infocluster: manage-prodannotations:current_value: '{{$value}}'description: "kubernetess scheduler {{ $labels.instance }} is use memory More than 20MB"summary: "kubernetes scheduler 使用内存超过20MB,当前值为{{$value}}MB"- alert: K8SSchedulerPodPendingexpr: sum(scheduler_pending_pods{job="kubernetes-scheduler"})by(queue) > 5for: 5mlabels:severity: infocluster: manage-prodannotations:current_value: '{{$value}}'description: "kubernetess scheduler {{ $labels.instance }} is Pending pod More than 5"summary: "kubernetes scheduler pod无法调度 > 5,当前值为{{$value}}"- alert: K8SSchedulerPodPendingexpr: sum(scheduler_pending_pods{job="kubernetes-scheduler"})by(queue) > 10for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }} is Pending pod More than 10summary: "kubernetes scheduler pod无法调度 > 10,当前值为{{$value}}"- alert: K8SSchedulerPodPendingexpr: sum(rate(scheduler_binding_duration_seconds_count{job="kubernetes-scheduler"}[5m])) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }}summary: "kubernetes scheduler pod 无法绑定调度有问题,当前值为{{$value}}"- alert: K8SSchedulerVolumeSpeedexpr: sum(rate(scheduler_volume_scheduling_duration_seconds_count{job="kubernetes-scheduler"}[5m])) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }}summary: "kubernetes scheduler pod Volume 速度延迟,当前值为{{$value}}"- alert: K8SSchedulerClientRequestSlowexpr: histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{job="kubernetes-scheduler"}[5m])) by (verb, url, le)) > 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: '{{$value}}'description: kubernetess scheduler {{ $labels.instance }}summary: "kubernetes scheduler 客户端请求速度延迟,当前值为{{$value}}"
root@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# kubectl  apply -f kube-controller-rules.yaml 
prometheusrule.monitoring.coreos.com/kube-apiserver-rules configured
root@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# kubectl  apply -f kube-scheduler-rules.yaml 
prometheusrule.monitoring.coreos.com/kube-apiserver-rules configured
root@10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# 

五、Prometheus验证


http://www.ppmy.cn/news/1276669.html

相关文章

<二>Python的进制转换和类型转换

1.进制 Python里的数据类型: 整型(int) 浮点型(fLoat) 复数(complex) 字符串(str) 布尔(bool) 列表(List) 元组(tuple) 字典(dict) 集合(set) 整型就是整数。计算机其实只能保存二进制0和1,一个二进制表示一个比特(Byte)。为了方便数据的表示,同时计算机…

玩转大数据19:数据治理与元数据管理策略

随着大数据时代的到来,数据已经成为企业的重要资产。然而,如何有效地管理和利用这些数据,成为了一个亟待解决的问题。数据治理和元数据管理是解决这个问题的关键。 1.数据治理的概念和重要性 数据治理是指对数据进行全面、系统、规范的管理…

java中各种类型用Stream流求最大值最小值

java中各种类型用Stream流求最大值最小值 一、BigDecimal 求最大值和最小值 1. stream().reduce()实现 List<BigDecimal> list new ArrayList<>(Arrays.asList(new BigDecimal("1"), new BigDecimal("2"))); BigDecimal max list.stream()…

前后端分离下的鸿鹄电子招投标系统:使用Spring Boot、Mybatis、Redis和Layui实现源码与立项流程

在数字化时代&#xff0c;采购管理也正经历着前所未有的变革。全过程数字化采购管理成为了企业追求高效、透明和规范的关键。该系统通过Spring Cloud、Spring Boot2、Mybatis等先进技术&#xff0c;打造了从供应商管理到采购招投标、采购合同、采购执行的全过程数字化管理。通过…

Linux基础第一章(第二部分)

目录 一、通配符 1、详细解释ls -l即ll 2、通配符 3、alias&#xff1a;临时修改别名 4、du&#xff08;disk usage&#xff09;&#xff1a;查看当前所在文件夹的总磁盘占用量 5、mkdir&#xff1a;创建目录 6、touch&#xff1a;①创建文件。②刷新文件时间 7、硬链接…

MATLAB版本、labview版本、UHD版本 互相对应

LabVIEWMATLABUHD2019R2021bUHD_3.15.0.0-vendor2020R2022bUHD_4.1.0.4-vendorR2023bUHD_4.2.0.0-vendor 更换固件 MATLAB 更换固件指令 status sdruload(Device "X310", IPAddress 192.168.10.2)

Python自动化神器入门

在日常生活和工作中&#xff0c;我们经常需要重复执行一些鼠标和键盘操作。今天&#xff0c;我们就来介绍一款非常实用的Python库——PyAutoGUI&#xff0c;它可以帮助我们实现鼠标和键盘的自动化操作。 一、PyAutoGUI简介 PyAutoGUI是一个用于模拟鼠标和键盘操作的Python库。…

PSP - 结构生物学中的机器学习 (NIPS MLSB Workshop 2023.12)

欢迎关注我的CSDN&#xff1a;https://spike.blog.csdn.net/ 本文地址&#xff1a;https://spike.blog.csdn.net/article/details/135120094 Machine Learning in Structural Biology (机器学习在结构生物学中) 网址&#xff1a;https://www.mlsb.io/ Workshop at the 37th Co…