概述

在Kubernetes集群中部署MySQL和Mongodb的StatefulSet服务。

MySQL有官方文档的指引 + 其他网站博客的指引实现

Mongodb修改operator的Deployment进行简单的实现

MySQL-StatefulSet

参考官方文档：运行一个有状态的应用程序 | Kubernetes

深入剖析Kubernetes学习笔记：StatefulSet-MySQL集群（20） - 活的潇洒80 - 博客园 (cnblogs.com)

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:labels:kubernetes.io/metadata.name: mysqlname: mysql
spec:finalizers:- kubernetes
status:phase: Active

mysql-class.yaml

创建动态存储卷：storeageclass。

存在部署依赖，需要先安装服务：nfs-subdir-external-provisioner

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: nfs-mysqlnamespace: mysqlannotations:storageclass.kubernetes.io/is-default-class: "false"  ## 是否设置为默认的storageclass
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
#provisioner: nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:archiveOnDelete: "true"pathPattern: "${.PVC.namespace}/${.PVC.name}/${.PVC.annotations.nfs.io/storage-path}"# pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"# pathPattern: "${.PVC.namespace}-${.PVC.name}"

mysql-pvc.yaml

参考官方文档：存储类 | Kubernetes

存在部署依赖，需要先安装服务：nfs-subdir-external-provisioner

自动创建的，下面data-mysql-0为例子：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:annotations:pv.kubernetes.io/bind-completed: "yes"pv.kubernetes.io/bound-by-controller: "yes"volume.beta.kubernetes.io/storage-class: nfs-mysqlvolume.beta.kubernetes.io/storage-provisioner: k8s-sigs.io/nfs-subdir-external-provisionerfinalizers:- kubernetes.io/pvc-protectionname: data-mysql-0namespace: mysql
spec:accessModes:- ReadWriteManyresources:requests:storage: 10MistorageClassName: nfs-mysqlvolumeMode: Filesystem

configmap.yaml

参考文档：运行一个有状态的应用程序 | Kubernetes

参考文档：使用StatefulSet搭建MySQL8集群 - 简书 (jianshu.com)

目前使用的是5.7.43的mysql

apiVersion: v1
kind: ConfigMap
metadata:name: mysqlnamespace: mysqllabels:app: mysql
data:master.cnf: |# Apply this config only on the master.[client]default-character-set=utf8mb4[mysql]default-character-set=utf8mb4[mysqld]# 打开binlog日志log-bin# binlog_expire_logs_seconds=2592000max_connections=10000# 在容器里面需要设置下时区default-time-zone='+8:00'character-set-client-handshake=FALSEcharacter-set-server=utf8mb4collation-server=utf8mb4_unicode_ciinit_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'slave.cnf: |# Apply this config only on slaves.[client]default-character-set=utf8mb4[mysql]default-character-set=utf8mb4[mysqld]# 机器设置只读super-read-onlymax_connections=10000default-time-zone='+8:00'character-set-client-handshake=FALSEcharacter-set-server=utf8mb4collation-server=utf8mb4_unicode_ciinit_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'

service.yaml

根据官方文档部署svc，无头服务 + ClusterIP

# 为 StatefulSet 成员提供稳定的 DNS 表项的无头服务（Headless Service）
apiVersion: v1
kind: Service
metadata:name: mysqlnamespace: mysqllabels:app: mysqlapp.kubernetes.io/name: mysql
spec:ports:- name: mysqlport: 3306clusterIP: Noneselector:app: mysql
---
# 用于连接到任一 MySQL 实例执行读操作的客户端服务
# 对于写操作，你必须连接到主服务器：mysql-0.mysql
apiVersion: v1
kind: Service
metadata:name: mysql-readnamespace: mysqllabels:app: mysqlapp.kubernetes.io/name: mysqlreadonly: "true"
spec:ports:- name: mysqlport: 3306selector:app: mysql

secret.yaml

官方文档的 MYSQL_ALLOW_EMPTY_PASSWORD 是通过明文的方式传递，需要修复为secret的模式登录，参考文档：Secret | Kubernetes

创建命令：

kubectl create secret generic mysql-secret -n mysql  --dry-run=client --from-literal=MYSQL_ALLOW_EMPTY_PASSWORD=1 -o yaml

 结果输出为：

apiVersion: v1
data:MYSQL_ALLOW_EMPTY_PASSWORD: MQ==
kind: Secret
metadata:creationTimestamp: nullname: mysql-secretnamespace: mysql

使用 Secret 安全地分发凭据 | Kubernetes 配置statefulset

...env:- name: MYSQL_ALLOW_EMPTY_PASSWORDvalueFrom:secretKeyRef:name: mysql-secretkey: MYSQL_ALLOW_EMPTY_PASSWORD
...

详细参考下面的statefulset.yaml

StatefulSet.yaml

修改后的配置文件

与官方的文档主要差异如下：

• 配置了镜像拉取策略：imagePullPolicy: Always
• 密码文件使用secret的环境变量方式传递而非明文：secretKeyRef
• 使用的是image为MySQL:5.7.43
• 缩小了副本数：replicas: 2
• 添加了节点反亲和的策略：nodeAffinity + operator: NotIn
• 更新了镜像地址： image
• 使用了storeageclass的方式记录data：storageClassName + 更新了访问模式ReadWriteMany

apiVersion: apps/v1
kind: StatefulSet
metadata:name: mysqlnamespace: mysql
spec:selector:matchLabels:app: mysqlapp.kubernetes.io/name: mysqlserviceName: mysqlreplicas: 2template:metadata:labels:app: mysqlapp.kubernetes.io/name: mysqlspec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: NotInvalues:- master01initContainers:- name: init-mysqlimage: mysql:5.7.43command:- bash- "-c"- |set -ex# 基于 Pod 序号生成 MySQL 服务器的 ID。[[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1ordinal=${BASH_REMATCH[1]}echo [mysqld] > /mnt/conf.d/server-id.cnf# 添加偏移量以避免使用 server-id=0 这一保留值。echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf# 将合适的 conf.d 文件从 config-map 复制到 emptyDir。if [[ $ordinal -eq 0 ]]; thencp /mnt/config-map/master.cnf /mnt/conf.d/elsecp /mnt/config-map/slave.cnf /mnt/conf.d/fivolumeMounts:- name: confmountPath: /mnt/conf.d- name: config-mapmountPath: /mnt/config-map- name: clone-mysqlimage: reporsitory:5000/xtrabackup:1.0imagePullPolicy: Alwayscommand:- bash- "-c"- |set -ex# 如果已有数据，则跳过克隆。[[ -d /var/lib/mysql/mysql ]] && exit 0# 跳过主实例（序号索引 0）的克隆。[[ `hostname` =~ -([0-9]+)$ ]] || exit 1ordinal=${BASH_REMATCH[1]}[[ $ordinal -eq 0 ]] && exit 0# 从原来的对等节点克隆数据。ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql# 准备备份。xtrabackup --prepare --target-dir=/var/lib/mysqlvolumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dcontainers:- name: mysqlimage: mysql:5.7.43env:- name: MYSQL_ALLOW_EMPTY_PASSWORDvalueFrom:secretKeyRef:name: mysql-secretkey: MYSQL_ALLOW_EMPTY_PASSWORDports:- name: mysqlcontainerPort: 3306volumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dresources:requests:cpu: 500mmemory: 1GilivenessProbe:exec:command: ["mysqladmin", "ping"]initialDelaySeconds: 30periodSeconds: 10timeoutSeconds: 5readinessProbe:exec:# 检查我们是否可以通过 TCP 执行查询（skip-networking 是关闭的）。command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]initialDelaySeconds: 5periodSeconds: 2timeoutSeconds: 1- name: xtrabackupimage: reporsitory:5000/xtrabackup:1.0imagePullPolicy: Alwaysports:- name: xtrabackupcontainerPort: 3307command:- bash- "-c"- |set -excd /var/lib/mysql# 确定克隆数据的 binlog 位置（如果有的话）。if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then# XtraBackup 已经生成了部分的 “CHANGE MASTER TO” 查询# 因为我们从一个现有副本进行克隆。(需要删除末尾的分号!)cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in# 在这里要忽略 xtrabackup_binlog_info （它是没用的）。rm -f xtrabackup_slave_info xtrabackup_binlog_infoelif [[ -f xtrabackup_binlog_info ]]; then# 我们直接从主实例进行克隆。解析 binlog 位置。[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1rm -f xtrabackup_binlog_info xtrabackup_slave_infoecho "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.infi# 检查我们是否需要通过启动复制来完成克隆。if [[ -f change_master_to.sql.in ]]; thenecho "Waiting for mysqld to be ready (accepting connections)"until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; doneecho "Initializing replication from clone position"mysql -h 127.0.0.1 \-e "$(<change_master_to.sql.in), \MASTER_HOST='mysql-0.mysql', \MASTER_USER='root', \MASTER_PASSWORD='', \MASTER_CONNECT_RETRY=10; \START SLAVE;" || exit 1# 如果容器重新启动，最多尝试一次。mv change_master_to.sql.in change_master_to.sql.origfi# 当对等点请求时，启动服务器发送备份。exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"volumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dresources:requests:cpu: 100mmemory: 100Mivolumes:- name: confemptyDir: {}- name: config-mapconfigMap:name: mysqlvolumeClaimTemplates:- metadata:name: dataspec:storageClassName: nfs-mysqlaccessModes: ["ReadWriteMany"]resources:requests:storage: 10Mi

Error: pvc的绑定错误

使用默认的官方的yaml文件会发生报错，需要配置PV-Claim:

一直报错：

修改配置PV-Claim。yaml文件：

这里应该不用修改PVC，因为pvc是自动创建，应该是修改statefulset中的storageClassName: nfs-mysql 实现修复。

Error: pause:3.6 镜像无法下载

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox i                                     mage "registry.aliyuncs.com/google_containers/pause:3.6": failed to pull image "registry.aliyuncs.com/google_containers/pause:3.6": failed to pull and unpack imag                                     e "registry.aliyuncs.com/google_containers/pause:3.6": failed to resolve reference "registry.aliyuncs.com/google_containers/pause:3.6": failed to do request: Head                                      "https://registry.aliyuncs.com/v2/google_containers/pause/manifests/3.6": dial tcp: lookup registry.aliyuncs.com on [fe80::10%ens34]:53: read udp [fe80::d21d:3ae                                     :ea99:27c6%ens34]:51322->[fe80::10%ens34]:53: i/o timeout

因为被调度到master01节点，该节点没有pause:3.6镜像，所以启动失败，添加反亲和策略：

将 Pod 指派给节点 | Kubernetes

解决办法如下：

Error: xtrabackup:1.0镜像下载失败

更换以下镜像源：

使用StatefulSet搭建MySQL8集群 - 简书 (jianshu.com)

mzmuer/xtrabackup:1.0

又或者docker search:

但仍然是失败的，核心应该是修改： /etc/containerd/config.toml 。

因为目前集群使用的是containerd，镜像仓库是docker的registry，需要修改对应的配置：

1. 在仓库主机中将镜像ist0ne/xtrabackup下载下来
2. 使用docker tag 更新标签
3. 从新推送到自己的镜像仓库中

Error: init-container 启动失败

官方文档使用的是：primary.cnf + replica.cnf

其他参考文档使用的是：master.cnf + slave.cnf

我参考的是国内其他参考文档的内容，所以要进行修改

Error:The node was low on resource: ephemeral-storage

临时存储空间不足，下载不了镜像

当时master02 --> master03 --> node04 --> node05 顺序集群自动逐步尝试部署。

Error: ErrImagePull

添加镜像拉取策略：imagePullPolicy: Always

修改containerd的配置文件，一般为：/etc/containerd/config.toml

随后重启服务： systemctl restart containerd

因为部署的是docker的镜像仓库：相关的daemon.json如图：

修复后：

Error: unknown variable 'binlog_expire_logs_seconds=2592000'

非官方的参考文档用的是mysql:8的镜像，其中配置文件写binlog_expire_logs_seconds=2592000

目前，我自己使用的镜像是MySQL:5.7.43的镜像，这里可能存在配置参数不支持的问题

修改配置文件：kubectl edit configmap mysql -n mysql

Error: pvc 一直处于pending状态

相关nfs服务已经处于资源吃紧的边缘：

缩减资源的使用，statefulset的副本需要缩小，由原来的3个缩小为2个。

成功结果：

验证：

参考文档

深入剖析Kubernetes学习笔记：StatefulSet-MySQL集群（20） - 活的潇洒80 - 博客园 (cnblogs.com)

kubernetes【statefulset搭建MySQL集群】 - 掘金 (juejin.cn)

使用pv + nfs服务器的案例：k8s-使用statefulset部署mysql主从集群5.7 - gg888666 - 博客园 (cnblogs.com)

使用storageclass + PV的案例：K8S 部署 Statefulset mysql-腾讯云开发者社区-腾讯云 (tencent.com)

有状态的节点控制器 -- StatefulSet 及其网络状态 (qq.com)

有状态的节点控制器 StatefulSet 的存储状态 (qq.com)

实战 Kubernetes StatefulSet -- MySQL 主从集群搭建-腾讯云开发者社区-腾讯云 (tencent.com)

Kubernetes：使用StatefulSet搭建MySQL集群（一主多从） - Cooper_73 - 博客园 (cnblogs.com)

Kubernetes：使用StatefulSet搭建MySQL集群（一主多从） - Cooper_73 - 博客园 (cnblogs.com)

Mongodb-StatefulSet

文件来源

Install and Configure the Kubernetes Operator — MongoDB Kubernetes Operator upcoming

Install the MongoDB Enterprise Kubernetes Operator — MongoDB Kubernetes Operator upcoming

mongodb-enterprise-kubernetes/mongodb-enterprise.yaml at master · mongodb/mongodb-enterprise-kubernetes (github.com)

crd.yaml

mongodb-enterprise-kubernetes/crds.yaml at master · mongodb/mongodb-enterprise-kubernetes (github.com)

来源于官网文件，使用kubectl apply -f 直接部署 

mongodb-enterprise.yaml

参考的视频：statefulset 学亮-哔哩哔哩_Bilibili

将官方网站的deployment文件进行了修改，主要是：

• 增加了创建namespace的yaml部分
• 修改了Deployment为StatefulSet
• 增加了无头服务Service

---
apiVersion: v1
kind: Namespace
metadata:labels:kubernetes.io/metadata.name: mongodbname: mongodb
spec:finalizers:- kubernetes
status:phase: Active
---
# Source: enterprise-operator/templates/operator-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: mongodb-enterprise-operatornamespace: mongodb
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: mongodb-enterprise-operator-mongodb-webhook
rules:- apiGroups:- "admissionregistration.k8s.io"resources:- validatingwebhookconfigurationsverbs:- get- create- update- delete- apiGroups:- ""resources:- servicesverbs:- get- list- watch- create- update- delete
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: mongodb-enterprise-operator-mongodb-webhook-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: mongodb-enterprise-operator-mongodb-webhook
subjects:- kind: ServiceAccountname: mongodb-enterprise-operatornamespace: mongodb
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: mongodb-enterprise-operatornamespace: mongodb
rules:- apiGroups:- ""resources:- servicesverbs:- get- list- watch- create- update- delete- apiGroups:- ""resources:- secrets- configmapsverbs:- get- list- create- update- delete- watch- apiGroups:- appsresources:- statefulsetsverbs:- create- get- list- watch- delete- update- apiGroups:- ""resources:- podsverbs:- get- list- watch- delete- deletecollection- apiGroups:- mongodb.comverbs:- "*"resources:- mongodb- mongodb/finalizers- mongodbusers- opsmanagers- opsmanagers/finalizers- mongodbmulticluster- mongodbmulticluster/finalizers- mongodb/status- mongodbusers/status- opsmanagers/status- mongodbmulticluster/status
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: mongodb-enterprise-operatornamespace: mongodb
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: mongodb-enterprise-operator
subjects:- kind: ServiceAccountname: mongodb-enterprise-operatornamespace: mongodb# This ClusterRoleBinding is necessary in order to use validating
# webhooks—these will prevent you from applying a variety of invalid resource
# definitions. The validating webhooks are optional so this can be removed if
# necessary.
---
# Source: enterprise-operator/templates/database-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: mongodb-enterprise-appdbnamespace: mongodb
---
# Source: enterprise-operator/templates/database-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: mongodb-enterprise-database-podsnamespace: mongodb
---
# Source: enterprise-operator/templates/database-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: mongodb-enterprise-ops-managernamespace: mongodb
---
# Source: enterprise-operator/templates/database-roles.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: mongodb-enterprise-appdbnamespace: mongodb
rules:- apiGroups:- ""resources:- secretsverbs:- get- apiGroups:- ""resources:- podsverbs:- patch- delete- get
---
# Source: enterprise-operator/templates/database-roles.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: mongodb-enterprise-appdbnamespace: mongodb
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: mongodb-enterprise-appdb
subjects:- kind: ServiceAccountname: mongodb-enterprise-appdbnamespace: mongodb
---
# Source: enterprise-operator/templates/operator.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:name: mongodb-enterprise-operatornamespace: mongodb
spec:serviceName: mongodbreplicas: 3selector:matchLabels:app.kubernetes.io/component: controllerapp.kubernetes.io/name: mongodb-enterprise-operatorapp.kubernetes.io/instance: mongodb-enterprise-operatortemplate:metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/name: mongodb-enterprise-operatorapp.kubernetes.io/instance: mongodb-enterprise-operatorspec:serviceAccountName: mongodb-enterprise-operatorsecurityContext:runAsNonRoot: truerunAsUser: 2000containers:- name: mongodb-enterprise-operatorimage: "quay.io/mongodb/mongodb-enterprise-operator-ubi:1.22.0"imagePullPolicy: Alwaysargs:- -watch-resource=mongodb- -watch-resource=opsmanagers- -watch-resource=mongodbuserscommand:- /usr/local/bin/mongodb-enterprise-operatorresources:limits:cpu: 1100mmemory: 1Girequests:cpu: 500mmemory: 200Mienv:- name: OPERATOR_ENVvalue: prod- name: WATCH_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: CLUSTER_CLIENT_TIMEOUTvalue: "10"- name: IMAGE_PULL_POLICYvalue: Always# Database- name: MONGODB_ENTERPRISE_DATABASE_IMAGEvalue: quay.io/mongodb/mongodb-enterprise-database-ubi- name: INIT_DATABASE_IMAGE_REPOSITORYvalue: quay.io/mongodb/mongodb-enterprise-init-database-ubi- name: INIT_DATABASE_VERSIONvalue: 1.0.19- name: DATABASE_VERSIONvalue: 2.0.2# Ops Manager- name: OPS_MANAGER_IMAGE_REPOSITORYvalue: quay.io/mongodb/mongodb-enterprise-ops-manager-ubi- name: INIT_OPS_MANAGER_IMAGE_REPOSITORYvalue: quay.io/mongodb/mongodb-enterprise-init-ops-manager-ubi- name: INIT_OPS_MANAGER_VERSIONvalue: 1.0.12# AppDB- name: INIT_APPDB_IMAGE_REPOSITORYvalue: quay.io/mongodb/mongodb-enterprise-init-appdb-ubi- name: INIT_APPDB_VERSIONvalue: 1.0.18- name: OPS_MANAGER_IMAGE_PULL_POLICYvalue: Always- name: AGENT_IMAGEvalue: "quay.io/mongodb/mongodb-agent-ubi:12.0.25.7724-1"- name: MONGODB_IMAGEvalue: mongodb-enterprise-server- name: MONGODB_REPO_URLvalue: quay.io/mongodb- name: MDB_IMAGE_TYPEvalue: ubi8- name: PERFORM_FAILOVERvalue: "true"
---
apiVersion: v1
kind: Service
metadata:name: mongodbnamespace: mongodb
spec:selector:app.kubernetes.io/name: mongodb-enterprise-operatortype: ClusterIPclusterIP: Noneports:- port: 27017targetPort: 27017

成功部署的结果：