Kubernetes实战——集群监控和可视化管理

一、Kube-Prometheus

1、版本兼容性介绍

prometheus-toc" style="margin-left:40px;">2、安装 kube-prometheus

3、安装Ingress，实现访问

二、K8s安装ELK日志收集

1、安装Elasticsearch

2、安装Logstash

3、安装Filebeat

4、安装Kibina

三、Dashboard安装与使用

1、安装

2、创建token

3、使用

四、kubesphere安装与使用

1、官方文档

2、准备工作

2.1、Default StorageClass安装

3、安装

4、卸载

5、使用

5.1、主界面

5.2、可插拔组件

5.3、平台管理

5.4、访问控制

5.5、通知设置

五、附录

prometheus%E6%BA%90%E7%A0%81-toc" style="margin-left:40px;">1、kube-prometheus源码

prometheus%E6%89%80%E9%9C%80%E9%95%9C%E5%83%8F-toc" style="margin-left:40px;">2、kube-prometheus所需镜像

3、Dashboard配置文件

4、openebs-operator.yaml

5、default-storage-class.yaml

6、修改镜像名称

六、参考

一、Kube-Prometheus

1、版本兼容性介绍

kube-prometheus 堆栈	Kubernetes 1.23	Kubernetes 1.24	Kubernetes 1.25	Kubernetes 1.26	Kubernetes 1.27	Kubernetes 1.28	Kubernetes 1.29	Kubernetes 1.30	Kubernetes 1.31
release-0.11	✔	✔	✗	十	十	十	十	十	十
release-0.12	✗	✔	✔	十	十	十	十	十	十
release-0.13	✗	✗	十	✔	✔	✔	十	十	十
release-0.14	✗	✗	十	✔	✔	✔	✔	✔	✔
main	✗	✗	十	十	✔	✔	✔

prometheus">2、安装 kube-prometheus

#下载资源 下载不下来的见附录
wget https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.11.0.tar.gz#解压文件
tar -xf v0.11.0.tar.gz #重命名
mv kube-prometheus-0.11.0 kube-prometheus#进入目录
cd kube-prometheus/manifests#创建setup
kubectl create -f setup/#创建kube-prometheus资源
kubectl apply -f ../manifests/#查看资源
kubectl get all -n monitoring

注意：

有些镜像可能在国内拉取不下来，我会把镜像打包放到附录，直接导入镜像即可。
安装需要足够的空间，请清理掉没用的的Pod，或者增大虚拟机的内存。

#导入镜像 
docker load -i <压缩包名称>

3、安装Ingress，实现访问

apiVersion: networking.k8s.io/v1
kind: Ingress  #类型为ingress
metadata:name: prometheus-ingress #ingress名称namespace: monitoring  #命名空间annotations:nginx.ingress.kubernetes.io/rewrite-target: /
spec:ingressClassName: nginxrules:- host: grafana.wssnail.com  #grafana访问的域名http:paths:- pathType: Prefix #类型path: /backend:service:name: grafana  #需要与grafana-service.yml中配置的服务名保持一致port:number: 3000 #需要与grafana-service.yml中配置的端口持一致- host: prometheus.wssnail.com  #prometheus访问的域名http:paths:- pathType: Prefixpath: /backend:service:name: prometheus-k8s #需要与prometheus-service.yaml中配置的服务名称保持一致port:number: 9090  #需要与prometheus-service.yaml中配置的端口称保持一致- host: alertmanager.wssnail.com  #alertmanager访问的域名http:paths:- pathType: Prefixpath: /backend:service:name: alertmanager-main  #需要与alertmanager-service.yaml配置的服务名保持一致port:number: 9093  #需要与alertmanager-service.yaml配置的端口保持一致

#创建资源，配置hosts
kubectl create -f prometheus-ingress.yaml #配置host,其中ip为ingress-nginx的Pod所在的节点ip
192.168.139.207  grafana.wssnail.com
192.168.139.207  prometheus.wssnail.com
192.168.139.207  alertmanager.wssnail.com

二、K8s安装ELK日志收集

1、安装Elasticsearch

apiVersion: v1
kind: Namespace
metadata:name: kube-logging
--- 
apiVersion: v1 
kind: Service 
metadata: name: elasticsearch-logging namespace: kube-logging labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "Elasticsearch" 
spec: ports: - port: 9200 protocol: TCP targetPort: db selector: k8s-app: elasticsearch-logging 
--- 
# RBAC authn and authz 
apiVersion: v1 
kind: ServiceAccount 
metadata: name: elasticsearch-logging namespace: kube-logging labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile 
--- 
kind: ClusterRole 
apiVersion: rbac.authorization.k8s.io/v1 
metadata: name: elasticsearch-logging labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile 
rules: 
- apiGroups: - "" resources: - "services" - "namespaces" - "endpoints" verbs: - "get" 
--- 
kind: ClusterRoleBinding 
apiVersion: rbac.authorization.k8s.io/v1 
metadata: namespace: kube-logging name: elasticsearch-logging labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile 
subjects: 
- kind: ServiceAccount name: elasticsearch-logging namespace: kube-logging apiGroup: "" 
roleRef: kind: ClusterRole name: elasticsearch-logging apiGroup: "" 
--- 
# Elasticsearch deployment itself 
apiVersion: apps/v1 
kind: StatefulSet #使用statefulset创建Pod 
metadata: name: elasticsearch-logging #pod名称,使用statefulSet创建的Pod是有序号有顺序的 namespace: kube-logging  #命名空间 labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile srv: srv-elasticsearch 
spec: serviceName: elasticsearch-logging #与svc相关联，这可以确保使用以下DNS地址访问Statefulset中的每个pod (es-cluster-[0,1,2].elasticsearch.elk.svc.cluster.local) replicas: 1 #副本数量,单节点 selector: matchLabels: k8s-app: elasticsearch-logging #和pod template配置的labels相匹配 template: metadata: labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" spec: serviceAccountName: elasticsearch-logging containers: - image: docker.io/library/elasticsearch:7.9.3 name: elasticsearch-logging resources: # need more cpu upon initialization, therefore burstable class limits: cpu: 1000m memory: 2Gi requests: cpu: 100m memory: 500Mi ports: - containerPort: 9200 name: db protocol: TCP - containerPort: 9300 name: transport protocol: TCP volumeMounts: - name: elasticsearch-logging mountPath: /usr/share/elasticsearch/data/   #挂载点 env: - name: "NAMESPACE" valueFrom: fieldRef: fieldPath: metadata.namespace - name: "discovery.type"  #定义单节点类型 value: "single-node" - name: ES_JAVA_OPTS #设置Java的内存参数，可以适当进行加大调整 value: "-Xms512m -Xmx2g"  volumes: - name: elasticsearch-logging hostPath: path: /data/es/nodeSelector: #如果需要匹配落盘节点可以添加 nodeSelect es: data tolerations: - effect: NoSchedule operator: Exists # Elasticsearch requires vm.max_map_count to be at least 262144. # If your OS already sets up this number to a higher value, feel free # to remove this init container. initContainers: #容器初始化前的操作 - name: elasticsearch-logging-init image: alpine:3.6 command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"] #添加mmap计数限制，太低可能造成内存不足的错误 securityContext:  #仅应用到指定的容器上，并且不会影响Volume privileged: true #运行特权容器 - name: increase-fd-ulimit image: busybox imagePullPolicy: IfNotPresent command: ["sh", "-c", "ulimit -n 65536"] #修改文件描述符最大数量 securityContext: privileged: true - name: elasticsearch-volume-init #es数据落盘初始化，加上777权限 image: alpine:3.6 command: - chmod - -R - "777" - /usr/share/elasticsearch/data/ volumeMounts: - name: elasticsearch-logging mountPath: /usr/share/elasticsearch/data/

#给Node1节点打标签
kubectl label no node1 es=data#创建资源
kubectl create -f es.yaml #查看资源
kubectl get po -n kube-logging

2、安装Logstash

--- 
apiVersion: v1 
kind: Service 
metadata: name: logstash namespace: kube-logging 
spec: ports: - port: 5044 targetPort: beats selector: type: logstash clusterIP: None 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: name: logstash namespace: kube-logging 
spec: selector: matchLabels: type: logstash template: metadata: labels: type: logstash srv: srv-logstash spec: containers: - image: docker.io/kubeimages/logstash:7.9.3 #该镜像支持arm64和amd64两种架构 name: logstash ports: - containerPort: 5044 name: beats command: - logstash - '-f' - '/etc/logstash_c/logstash.conf' env: - name: "XPACK_MONITORING_ELASTICSEARCH_HOSTS" value: "http://elasticsearch-logging:9200"  #这里配置的是es的服务名称，如果是夸命名空间需要加上命名空间volumeMounts: - name: config-volume mountPath: /etc/logstash_c/ - name: config-yml-volume mountPath: /usr/share/logstash/config/ - name: timezone mountPath: /etc/localtime resources: #logstash一定要加上资源限制，避免对其他业务造成资源抢占影响 limits: cpu: 1000m memory: 2048Mi requests: cpu: 512m memory: 512Mi volumes: - name: config-volume configMap: name: logstash-conf items: - key: logstash.conf path: logstash.conf - name: timezone hostPath: path: /etc/localtime - name: config-yml-volume configMap: name: logstash-yml items: - key: logstash.yml path: logstash.yml --- 
apiVersion: v1 
kind: ConfigMap 
metadata: name: logstash-conf namespace: kube-logging labels: type: logstash 
data: logstash.conf: |- input {beats { port => 5044 } } filter {# 处理 ingress 日志 if [kubernetes][container][name] == "nginx-ingress-controller" {json {source => "message" target => "ingress_log" }if [ingress_log][requesttime] { mutate { convert => ["[ingress_log][requesttime]", "float"] }}if [ingress_log][upstremtime] { mutate { convert => ["[ingress_log][upstremtime]", "float"] }} if [ingress_log][status] { mutate { convert => ["[ingress_log][status]", "float"] }}if  [ingress_log][httphost] and [ingress_log][uri] {mutate { add_field => {"[ingress_log][entry]" => "%{[ingress_log][httphost]}%{[ingress_log][uri]}"} } mutate { split => ["[ingress_log][entry]","/"] } if [ingress_log][entry][1] { mutate { add_field => {"[ingress_log][entrypoint]" => "%{[ingress_log][entry][0]}/%{[ingress_log][entry][1]}"} remove_field => "[ingress_log][entry]" }} else { mutate { add_field => {"[ingress_log][entrypoint]" => "%{[ingress_log][entry][0]}/"} remove_field => "[ingress_log][entry]" }}}}# 处理以srv进行开头的业务服务日志 if [kubernetes][container][name] =~ /^srv*/ { json { source => "message" target => "tmp" } if [kubernetes][namespace] == "kube-logging" { drop{} } if [tmp][level] { mutate{ add_field => {"[applog][level]" => "%{[tmp][level]}"} } if [applog][level] == "debug"{ drop{} } } if [tmp][msg] { mutate { add_field => {"[applog][msg]" => "%{[tmp][msg]}"} } } if [tmp][func] { mutate { add_field => {"[applog][func]" => "%{[tmp][func]}"} } } if [tmp][cost]{ if "ms" in [tmp][cost] { mutate { split => ["[tmp][cost]","m"] add_field => {"[applog][cost]" => "%{[tmp][cost][0]}"} convert => ["[applog][cost]", "float"] } } else { mutate { add_field => {"[applog][cost]" => "%{[tmp][cost]}"} }}}if [tmp][method] { mutate { add_field => {"[applog][method]" => "%{[tmp][method]}"} }}if [tmp][request_url] { mutate { add_field => {"[applog][request_url]" => "%{[tmp][request_url]}"} } }if [tmp][meta._id] { mutate { add_field => {"[applog][traceId]" => "%{[tmp][meta._id]}"} } } if [tmp][project] { mutate { add_field => {"[applog][project]" => "%{[tmp][project]}"} }}if [tmp][time] { mutate { add_field => {"[applog][time]" => "%{[tmp][time]}"} }}if [tmp][status] { mutate { add_field => {"[applog][status]" => "%{[tmp][status]}"} convert => ["[applog][status]", "float"] }}}mutate { rename => ["kubernetes", "k8s"] remove_field => "beat" remove_field => "tmp" remove_field => "[k8s][labels][app]" }}output { elasticsearch { hosts => ["http://elasticsearch-logging:9200"] codec => json index => "logstash-%{+YYYY.MM.dd}" #索引名称以logstash+日志进行每日新建 }} 
---apiVersion: v1 
kind: ConfigMap 
metadata: name: logstash-yml namespace: kube-logging labels: type: logstash 
data: logstash.yml: |- http.host: "0.0.0.0" xpack.monitoring.elasticsearch.hosts: http://elasticsearch-logging:9200

#创建资源
kubectl create -f logstash.yaml#查看资源
kubectl get po -n kube-logging

3、安装Filebeat

--- 
apiVersion: v1 
kind: ConfigMap 
metadata: name: filebeat-config namespace: kube-logging labels: k8s-app: filebeat 
data: filebeat.yml: |- filebeat.inputs: - type: container enable: truepaths: - /var/log/containers/*.log #这里是filebeat采集挂载到pod中的日志目录 processors: - add_kubernetes_metadata: #添加k8s的字段用于后续的数据清洗 host: ${NODE_NAME}matchers: - logs_path: logs_path: "/var/log/containers/" #output.kafka:  #如果日志量较大，es中的日志有延迟，可以选择在filebeat和logstash中间加入kafka #  hosts: ["kafka-log-01:9092", "kafka-log-02:9092", "kafka-log-03:9092"] # topic: 'topic-test-log' #  version: 2.0.0 output.logstash: #因为还需要部署logstash进行数据的清洗，因此filebeat是把数据推到logstash中 hosts: ["logstash:5044"] enabled: true 
--- 
apiVersion: v1 
kind: ServiceAccount 
metadata: name: filebeat namespace: kube-logging labels: k8s-app: filebeat
--- 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole 
metadata: name: filebeat labels: k8s-app: filebeat 
rules: 
- apiGroups: [""] # "" indicates the core API group resources: - namespaces - pods verbs: ["get", "watch", "list"] 
--- 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding 
metadata: name: filebeat 
subjects: 
- kind: ServiceAccount name: filebeat namespace: kube-logging 
roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io 
--- 
apiVersion: apps/v1 
kind: DaemonSet 
metadata: name: filebeat namespace: kube-logging labels: k8s-app: filebeat 
spec: selector: matchLabels: k8s-app: filebeat template: metadata: labels: k8s-app: filebeat spec: serviceAccountName: filebeat terminationGracePeriodSeconds: 30 containers: - name: filebeat image: docker.io/kubeimages/filebeat:7.9.3 #该镜像支持arm64和amd64两种架构 args: [ "-c", "/etc/filebeat.yml", "-e","-httpprof","0.0.0.0:6060" ] #ports: #  - containerPort: 6060 #    hostPort: 6068 env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: ELASTICSEARCH_HOST value: elasticsearch-logging - name: ELASTICSEARCH_PORT value: "9200" securityContext: runAsUser: 0 # If using Red Hat OpenShift uncomment this: #privileged: true resources: limits: memory: 1000Mi cpu: 1000m requests: memory: 100Mi cpu: 100m volumeMounts: - name: config #挂载的是filebeat的配置文件 mountPath: /etc/filebeat.yml readOnly: true subPath: filebeat.yml - name: data #持久化filebeat数据到宿主机上 mountPath: /usr/share/filebeat/data - name: varlibdockercontainers #这里主要是把宿主机上的源日志目录挂载到filebeat容器中,如果没有修改docker或者containerd的runtime进行了标准的日志落盘路径，可以把mountPath改为/var/lib mountPath: /var/libreadOnly: true - name: varlog #这里主要是把宿主机上/var/log/pods和/var/log/containers的软链接挂载到filebeat容器中 mountPath: /var/log/ readOnly: true - name: timezone mountPath: /etc/localtime volumes: - name: config configMap: defaultMode: 0600 name: filebeat-config - name: varlibdockercontainers hostPath: #如果没有修改docker或者containerd的runtime进行了标准的日志落盘路径，可以把path改为/var/lib path: /var/lib- name: varlog hostPath: path: /var/log/ # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart - name: inputs configMap: defaultMode: 0600 name: filebeat-inputs - name: data hostPath: path: /data/filebeat-data type: DirectoryOrCreate - name: timezone hostPath: path: /etc/localtime tolerations: #加入容忍能够调度到每一个节点 - effect: NoExecute key: dedicated operator: Equal value: gpu - effect: NoSchedule operator: Exists

#创建资源
kubectl create -f filebeat.yaml#查看资源
kubectl get po -n kube-logging

4、安装Kibina

---
apiVersion: v1
kind: ConfigMap
metadata:namespace: kube-loggingname: kibana-configlabels:k8s-app: kibana
data:kibana.yml: |-server.name: kibanaserver.host: "0"i18n.locale: zh-CN                      #设置默认语言为中文elasticsearch:hosts: ${ELASTICSEARCH_HOSTS}         #es集群连接地址，由于我这都都是k8s部署且在一个ns下，可以直接使用service name连接
--- 
apiVersion: v1 
kind: Service 
metadata: name: kibana namespace: kube-logging labels: k8s-app: kibana kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "Kibana" srv: srv-kibana 
spec: type: NodePortports: - port: 5601 protocol: TCP targetPort: ui selector: k8s-app: kibana 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: name: kibana namespace: kube-logging labels: k8s-app: kibana kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile srv: srv-kibana 
spec: replicas: 1 selector: matchLabels: k8s-app: kibana template: metadata: labels: k8s-app: kibana spec: containers: - name: kibana image: docker.io/kubeimages/kibana:7.9.3 #该镜像支持arm64和amd64两种架构 resources: # need more cpu upon initialization, therefore burstable class limits: cpu: 1000m requests: cpu: 100m env: - name: ELASTICSEARCH_HOSTS value: http://elasticsearch-logging:9200 ports: - containerPort: 5601 name: ui protocol: TCP volumeMounts:- name: configmountPath: /usr/share/kibana/config/kibana.ymlreadOnly: truesubPath: kibana.ymlvolumes:- name: configconfigMap:name: kibana-config#此处配置个ingress 避免通过外网访问不到的情况下处理
--- 
apiVersion: networking.k8s.io/v1
kind: Ingress 
metadata: name: kibana namespace: kube-logging 
spec: ingressClassName: nginxrules: - host: kibana.wssnail.comhttp: paths: - path: / pathType: Prefixbackend: service:name: kibana port:number: 5601

#创建资源kubectl create -f kibana.yaml#查看资源kubectl get all -n kube-logging

三、Dashboard安装与使用

1、安装

#下载资源文件
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
#修改service类型为type: NodePort 完整文件见附录#查看资源
kubectl get all -n kubernetes-dashboard

2、创建token

apiVersion: v1 
kind: ServiceAccount 
metadata: labels: k8s-app: kubernetes-dashboard name: dashboard-admin namespace: kubernetes-dashboard 
--- 
apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRoleBinding 
metadata: name: dashboard-admin-cluster-role 
roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin 
subjects: - kind: ServiceAccountname: dashboard-adminnamespace: kubernetes-dashboard

#创建资源
kubectl create -f dashboard-admin.yaml#查看账户信息
kubectl describe serviceaccount dashboard-admin -n kubernetes-dashboard#查看token
kubectl describe secrets dashboard-admin-token-krndx -n kubernetes-dashboard

注意：这里访问需要使用https

3、使用

四、kubesphere安装与使用

1、官方文档

https://kubesphere.io/zh/docs/v3.4

2、准备工作

您的 Kubernetes 版本必须为：v1.20.x、v1.21.x、v1.22.x、v1.23.x、* v1.24.x、* v1.25.x 和 * v1.26.x。带星号的版本可能出现边缘节点部分功能不可用的情况。因此，如需使用边缘节点，推荐安装 v1.23.x。
确保您的机器满足最低硬件要求：CPU > 1 核，内存 > 2 GB。
在安装之前，需要配置 Kubernetes 集群中的默认存储类型。

2.1、Default StorageClass安装

# 安装 iSCSI 协议客户端（OpenEBS 需要该协议提供存储支持） 所有节点都执行
yum install iscsi-initiator-utils -y# 设置开机启动
systemctl enable --now iscsid# 启动服务
systemctl start iscsid# 查看服务状态
systemctl status iscsid# 安装 OpenEBS 
kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml# 查看状态（下载镜像可能需要一些时间）
kubectl get all -n openebs# 在主节点创建本地 storage class
kubectl apply -f default-storage-class.yaml

附录 openebs-operator.yaml

附录 default-storage-class.yaml

3、安装

kubesphere-installer.yaml

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:name: clusterconfigurations.installer.kubesphere.io
spec:group: installer.kubesphere.ioversions:- name: v1alpha1served: truestorage: trueschema:openAPIV3Schema:type: objectproperties:spec:type: objectx-kubernetes-preserve-unknown-fields: truestatus:type: objectx-kubernetes-preserve-unknown-fields: truescope: Namespacednames:plural: clusterconfigurationssingular: clusterconfigurationkind: ClusterConfigurationshortNames:- cc---
apiVersion: v1
kind: Namespace
metadata:name: kubesphere-system---
apiVersion: v1
kind: ServiceAccount
metadata:name: ks-installernamespace: kubesphere-system---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: ks-installer
rules:
- apiGroups:- ""resources:- '*'verbs:- '*'
- apiGroups:- appsresources:- '*'verbs:- '*'
- apiGroups:- extensionsresources:- '*'verbs:- '*'
- apiGroups:- batchresources:- '*'verbs:- '*'
- apiGroups:- rbac.authorization.k8s.ioresources:- '*'verbs:- '*'
- apiGroups:- apiregistration.k8s.ioresources:- '*'verbs:- '*'
- apiGroups:- apiextensions.k8s.ioresources:- '*'verbs:- '*'
- apiGroups:- tenant.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- certificates.k8s.ioresources:- '*'verbs:- '*'
- apiGroups:- devops.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- monitoring.coreos.comresources:- '*'verbs:- '*'
- apiGroups:- logging.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- jaegertracing.ioresources:- '*'verbs:- '*'
- apiGroups:- storage.k8s.ioresources:- '*'verbs:- '*'
- apiGroups:- admissionregistration.k8s.ioresources:- '*'verbs:- '*'
- apiGroups:- policyresources:- '*'verbs:- '*'
- apiGroups:- autoscalingresources:- '*'verbs:- '*'
- apiGroups:- networking.istio.ioresources:- '*'verbs:- '*'
- apiGroups:- config.istio.ioresources:- '*'verbs:- '*'
- apiGroups:- iam.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- notification.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- auditing.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- events.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- core.kubefed.ioresources:- '*'verbs:- '*'
- apiGroups:- installer.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- storage.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- security.istio.ioresources:- '*'verbs:- '*'
- apiGroups:- monitoring.kiali.ioresources:- '*'verbs:- '*'
- apiGroups:- kiali.ioresources:- '*'verbs:- '*'
- apiGroups:- networking.k8s.ioresources:- '*'verbs:- '*'
- apiGroups:- edgeruntime.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- types.kubefed.ioresources:- '*'verbs:- '*'
- apiGroups:- monitoring.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- application.kubesphere.ioresources:- '*'verbs:- '*'
- apiGroups:- alerting.kubesphere.ioresources:- '*'verbs:- '*'---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: ks-installer
subjects:
- kind: ServiceAccountname: ks-installernamespace: kubesphere-system
roleRef:kind: ClusterRolename: ks-installerapiGroup: rbac.authorization.k8s.io---
apiVersion: apps/v1
kind: Deployment
metadata:name: ks-installernamespace: kubesphere-systemlabels:app: ks-installer
spec:replicas: 1selector:matchLabels:app: ks-installertemplate:metadata:labels:app: ks-installerspec:serviceAccountName: ks-installercontainers:- name: installerimage: kubesphere/ks-installer:v3.4.1imagePullPolicy: "IfNotPresent"resources:limits:cpu: "1"memory: 1Girequests:cpu: 20mmemory: 100MivolumeMounts:- mountPath: /etc/localtimename: host-timereadOnly: truevolumes:- hostPath:path: /etc/localtimetype: ""name: host-time

cluster-configuration.yaml

---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:name: ks-installernamespace: kubesphere-systemlabels:version: v3.4.1
spec:persistence:storageClass: ""        # If there is no default StorageClass in your cluster, you need to specify an existing StorageClass here.authentication:# adminPassword: ""     # Custom password of the admin user. If the parameter exists but the value is empty, a random password is generated. If the parameter does not exist, P@88w0rd is used.jwtSecret: ""           # Keep the jwtSecret consistent with the Host Cluster. Retrieve the jwtSecret by executing "kubectl -n kubesphere-system get cm kubesphere-config -o yaml | grep -v "apiVersion" | grep jwtSecret" on the Host Cluster.local_registry: ""        # Add your private registry address if it is needed.
#  dev_tag: ""               # Add your kubesphere image tag you want to install, by default it's same as ks-installer release version.etcd:monitoring: false       # Enable or disable etcd monitoring dashboard installation. You have to create a Secret for etcd before you enable it.endpointIps: localhost  # etcd cluster EndpointIps. It can be a bunch of IPs here.port: 2379              # etcd port.tlsEnable: truecommon:core:console:enableMultiLogin: true  # Enable or disable simultaneous logins. It allows different users to log in with the same account at the same time.port: 30880type: NodePort# apiserver:            # Enlarge the apiserver and controller manager's resource requests and limits for the large cluster#  resources: {}# controllerManager:#  resources: {}redis:enabled: falseenableHA: falsevolumeSize: 2Gi # Redis PVC size.openldap:enabled: falsevolumeSize: 2Gi   # openldap PVC size.minio:volumeSize: 20Gi # Minio PVC size.monitoring:# type: external   # Whether to specify the external prometheus stack, and need to modify the endpoint at the next line.endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090 # Prometheus endpoint to get metrics data.GPUMonitoring:     # Enable or disable the GPU-related metrics. If you enable this switch but have no GPU resources, Kubesphere will set it to zero.enabled: falsegpu:                 # Install GPUKinds. The default GPU kind is nvidia.com/gpu. Other GPU kinds can be added here according to your needs.kinds:- resourceName: "nvidia.com/gpu"resourceType: "GPU"default: truees:   # Storage backend for logging, events and auditing.# master:#   volumeSize: 4Gi  # The volume size of Elasticsearch master nodes.#   replicas: 1      # The total number of master nodes. Even numbers are not allowed.#   resources: {}# data:#   volumeSize: 20Gi  # The volume size of Elasticsearch data nodes.#   replicas: 1       # The total number of data nodes.#   resources: {}enabled: falselogMaxAge: 7             # Log retention time in built-in Elasticsearch. It is 7 days by default.elkPrefix: logstash      # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log.basicAuth:enabled: falseusername: ""password: ""externalElasticsearchHost: ""externalElasticsearchPort: ""opensearch:   # Storage backend for logging, events and auditing.# master:#   volumeSize: 4Gi  # The volume size of Opensearch master nodes.#   replicas: 1      # The total number of master nodes. Even numbers are not allowed.#   resources: {}# data:#   volumeSize: 20Gi  # The volume size of Opensearch data nodes.#   replicas: 1       # The total number of data nodes.#   resources: {}enabled: truelogMaxAge: 7             # Log retention time in built-in Opensearch. It is 7 days by default.opensearchPrefix: whizard      # The string making up index names. The index name will be formatted as ks-<opensearchPrefix>-logging.basicAuth:enabled: trueusername: "admin"password: "admin"externalOpensearchHost: ""externalOpensearchPort: ""dashboard:enabled: falsealerting:                # (CPU: 0.1 Core, Memory: 100 MiB) It enables users to customize alerting policies to send messages to receivers in time with different time intervals and alerting levels to choose from.enabled: false         # Enable or disable the KubeSphere Alerting System.# thanosruler:#   replicas: 1#   resources: {}auditing:                # Provide a security-relevant chronological set of records，recording the sequence of activities happening on the platform, initiated by different tenants.enabled: false         # Enable or disable the KubeSphere Auditing Log System.# operator:#   resources: {}# webhook:#   resources: {}devops:                  # (CPU: 0.47 Core, Memory: 8.6 G) Provide an out-of-the-box CI/CD system based on Jenkins, and automated workflow tools including Source-to-Image & Binary-to-Image.enabled: false         # Enable or disable the KubeSphere DevOps System.jenkinsCpuReq: 0.5jenkinsCpuLim: 1jenkinsMemoryReq: 4GijenkinsMemoryLim: 4Gi  # Recommend keep same as requests.memory.jenkinsVolumeSize: 16Gievents:                  # Provide a graphical web console for Kubernetes Events exporting, filtering and alerting in multi-tenant Kubernetes clusters.enabled: false         # Enable or disable the KubeSphere Events System.# operator:#   resources: {}# exporter:#   resources: {}ruler:enabled: truereplicas: 2#   resources: {}logging:                 # (CPU: 57 m, Memory: 2.76 G) Flexible logging functions are provided for log query, collection and management in a unified console. Additional log collectors can be added, such as Elasticsearch, Kafka and Fluentd.enabled: false         # Enable or disable the KubeSphere Logging System.logsidecar:enabled: truereplicas: 2# resources: {}metrics_server:                    # (CPU: 56 m, Memory: 44.35 MiB) It enables HPA (Horizontal Pod Autoscaler).enabled: false                   # Enable or disable metrics-server.monitoring:storageClass: ""                 # If there is an independent StorageClass you need for Prometheus, you can specify it here. The default StorageClass is used by default.node_exporter:port: 9100# resources: {}# kube_rbac_proxy:#   resources: {}# kube_state_metrics:#   resources: {}# prometheus:#   replicas: 1  # Prometheus replicas are responsible for monitoring different segments of data source and providing high availability.#   volumeSize: 20Gi  # Prometheus PVC size.#   resources: {}#   operator:#     resources: {}# alertmanager:#   replicas: 1          # AlertManager Replicas.#   resources: {}# notification_manager:#   resources: {}#   operator:#     resources: {}#   proxy:#     resources: {}gpu:                           # GPU monitoring-related plug-in installation.nvidia_dcgm_exporter:        # Ensure that gpu resources on your hosts can be used normally, otherwise this plug-in will not work properly.enabled: false             # Check whether the labels on the GPU hosts contain "nvidia.com/gpu.present=true" to ensure that the DCGM pod is scheduled to these nodes.# resources: {}multicluster:clusterRole: none  # host | member | none  # You can install a solo cluster, or specify it as the Host or Member Cluster.network:networkpolicy: # Network policies allow network isolation within the same cluster, which means firewalls can be set up between certain instances (Pods).# Make sure that the CNI network plugin used by the cluster supports NetworkPolicy. There are a number of CNI network plugins that support NetworkPolicy, including Calico, Cilium, Kube-router, Romana and Weave Net.enabled: false # Enable or disable network policies.ippool: # Use Pod IP Pools to manage the Pod network address space. Pods to be created can be assigned IP addresses from a Pod IP Pool.type: none # Specify "calico" for this field if Calico is used as your CNI plugin. "none" means that Pod IP Pools are disabled.topology: # Use Service Topology to view Service-to-Service communication based on Weave Scope.type: none # Specify "weave-scope" for this field to enable Service Topology. "none" means that Service Topology is disabled.openpitrix: # An App Store that is accessible to all platform tenants. You can use it to manage apps across their entire lifecycle.store:enabled: false # Enable or disable the KubeSphere App Store.servicemesh:         # (0.3 Core, 300 MiB) Provide fine-grained traffic management, observability and tracing, and visualized traffic topology.enabled: false     # Base component (pilot). Enable or disable KubeSphere Service Mesh (Istio-based).istio:  # Customizing the istio installation configuration, refer to https://istio.io/latest/docs/setup/additional-setup/customize-installation/components:ingressGateways:- name: istio-ingressgatewayenabled: falsecni:enabled: falseedgeruntime:          # Add edge nodes to your cluster and deploy workloads on edge nodes.enabled: falsekubeedge:        # kubeedge configurationsenabled: falsecloudCore:cloudHub:advertiseAddress: # At least a public IP address or an IP address which can be accessed by edge nodes must be provided.- ""            # Note that once KubeEdge is enabled, CloudCore will malfunction if the address is not provided.service:cloudhubNodePort: "30000"cloudhubQuicNodePort: "30001"cloudhubHttpsNodePort: "30002"cloudstreamNodePort: "30003"tunnelNodePort: "30004"# resources: {}# hostNetWork: falseiptables-manager:enabled: true mode: "external"# resources: {}# edgeService:#   resources: {}gatekeeper:        # Provide admission policy and rule management, A validating (mutating TBA) webhook that enforces CRD-based policies executed by Open Policy Agent.enabled: false   # Enable or disable Gatekeeper.# controller_manager:#   resources: {}# audit:#   resources: {}terminal:# image: 'alpine:3.15' # There must be an nsenter program in the imagetimeout: 600         # Container timeout, if set to 0, no timeout will be used. The unit is seconds

#下载资源
wget https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yamlwget  https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml#创建资源kubectl apply -f kubesphere-installer.yamlkubectl apply -f cluster-configuration.yaml

查看日志，出现以下内容证明安装完成。

#查看日志
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f#查看资源
kubectl get all -n kubesphere-system

注意：安装时间有点长，耐心等待