k8s中修复mongodb启动失败

背景

同事反馈 dev环境的yapi不能登录，看了一下是同事两年前用helm搭建的。单副本使用。
排查发现是后端数据库mongodb数据库挂掉。

root@dev-k8s-master03:~# kubectl get svc 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mongodb       ClusterIP   10.101.87.7     <none>        27017/TCP        2y39d
yapi-svc      NodePort    10.108.180.80   <none>        3000:31781/TCP   2y39d

故障现象

 2024-04-12T09:52:26.257+0000 I CONTROL  [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory2024-04-12T09:52:26.257+0000 I CONTROL  [initandlisten] options: { config: "/opt/bitnami/mongodb/conf/mongodb.conf", net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: "/opt/bitnami/mongodb/tmp" } }, processManagement: { fork: false, pidFilePath: "/opt/bitnami/mongodb/tmp/mongodb.pid" }, security: { authorization: "enabled" }, setParameter: { enableLocalhostAuthBypass: "false" }, storage: { dbPath: "/bitnami/mongodb/data/db", directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/opt/bitnami/mongodb/logs/mongodb.log", quiet: false, verbosity: 0 } }2024-04-12T09:52:26.258+0000 W STORAGE  [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.2024-04-12T09:52:26.259+0000 I STORAGE  [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.2024-04-12T09:52:26.259+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.2024-04-12T09:52:26.259+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options2024-04-12T09:52:26.390+0000 W STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.2024-04-12T09:52:26.390+0000 W STORAGE  [initandlisten] WiredTiger metadata corruption detected2024-04-12T09:52:26.390+0000 F STORAGE  [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair2024-04-12T09:52:26.390+0000 F -        [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 7012024-04-12T09:52:26.390+0000 F -        [initandlisten]

2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory

2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] options: { config: “/opt/bitnami/mongodb/conf/mongodb.conf”, net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: “/opt/bitnami/mongodb/tmp” } }, processManagement: { fork: false, pidFilePath: “/opt/bitnami/mongodb/tmp/mongodb.pid” }, security: { authorization: “enabled” }, setParameter: { enableLocalhostAuthBypass: “false” }, storage: { dbPath: “/bitnami/mongodb/data/db”, directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, logRotate: “reopen”, path: “/opt/bitnami/mongodb/logs/mongodb.log”, quiet: false, verbosity: 0 } }

2024-04-12T09:52:26.258+0000 W STORAGE [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.

2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.

2024-04-12T09:52:26.259+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.

2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.

2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] WiredTiger metadata corruption detected

2024-04-12T09:52:26.390+0000 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair

2024-04-12T09:52:26.390+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 701

2024-04-12T09:52:26.390+0000 F - [initandlisten]

分析&思路

看前面的E级别的log信息，存储引擎读取root page失败。unable to read root page from file:WiredTiger.wt

找到这个mongo挂载的nfs存储，先将数据库文件做一个备份。

单副本的mongo并且一直不断重启，网上搜到的使用mogod --repair执行修复根本不能进入到容器中执行的。

在同一个pod中使用跟mongo同样的镜像做一个二次封装，命名为mongodebug 把这个镜像放在mongo的pod中，使得这个pod存在两个容器，然后挂载同一个共享存储，进入到mongodebug执行mongod命令进行修复

实施

必要条件

挂载持久卷存储的容器路径/bitnami/mongodb/data/db

制作mongodebug容器镜像

目的：需要一个可执行的mongd命令

拉镜像，运行，找到容器启动脚本，覆盖掉。
防止读取同一份配置文件不能启动

root@master01:~# docker run -it bitnami/mongodb:4.0.14-debian-9-r24 /bin/bash 10:11:48.80 10:11:48.80 Welcome to the Bitnami mongodb container10:11:48.81 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb10:11:48.81 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues10:11:48.81 Send us your feedback at containers@bitnami.com10:11:48.81 I have no name!@7b9781de0eed:/$ 
I have no name!@7b9781de0eed:/$ 
I have no name!@7b9781de0eed:/$ ls
bin  bitnami  boot  dev  entrypoint.sh	etc  home  lib	lib64  libbitnami.sh  libcomponent.sh  libfile.sh  libfs.sh  liblog.sh	libmongodb.sh  libnet.sh  libos.sh  libservice.sh  libvalidations.sh  libversion.sh  licenses  media  mnt  opt	postunpack.sh  proc  root  run	run.sh	sbin  setup.sh	srv  sys  tmp  usr  varI have no name!@7b9781de0eed:/$ cat entrypoint.sh 
#!/bin/bash
# shellcheck disable=SC1091set -o errexit
set -o nounset
set -o pipefail# Load libraries
. /libbitnami.sh
. /libmongodb.sh# Load MongoDB env. variables
eval "$(mongodb_env)"print_welcome_pageif [[ "$*" = "/run.sh" ]]; theninfo "** Starting MongoDB setup **"/setup.shinfo "** MongoDB setup finished! **"
fiecho ""
exec "$@"I have no name!@7b9781de0eed:/$ cat run.sh 
#!/bin/bash
# shellcheck disable=SC1091set -o errexit
set -o nounset
set -o pipefail. /libmongodb.sh
. /libos.sh# Load MongoDB env. variables
eval "$(mongodb_env)"cmd=$(command -v mongod)flags=("--config=$MONGODB_CONF_FILE")if [[ -n "${MONGODB_EXTRA_FLAGS:-}" ]]; thenread -r -a extra_flags <<< "$MONGODB_EXTRA_FLAGS"flags+=("${extra_flags[@]}")
fiinfo "** Starting MongoDB **"
if am_i_root; thenexec gosu "$MONGODB_DAEMON_USER" "${cmd}" "${flags[@]}"
elseexec "${cmd}" "${flags[@]}"
fi
I have no name!@7b9781de0eed:/$ cmd=$(command -v mongod)
I have no name!@7b9781de0eed:/$ echo $cmd
/opt/bitnami/mongodb/bin/mongod
I have no name!@7b9781de0eed:/$ echo $MONGODB_CONF_FILE
/opt/bitnami/mongodb/conf/mongodb.conf
I have no name!@7b9781de0eed:/$ ls
bin  bitnami  boot  dev  entrypoint.sh	etc  home  lib	lib64  libbitnami.sh  libcomponent.sh  libfile.sh  libfs.sh  liblog.sh	libmongodb.sh  libnet.sh  libos.sh  libservice.sh  libvalidations.sh  libversion.sh  licenses  media  mnt  opt	postunpack.sh  proc  root  run	run.sh	sbin  setup.sh	srv  sys  tmp  usr  var
I have no name!@7b9781de0eed:/$ ps -ef 
UID          PID    PPID  C STIME TTY          TIME CMD
1001           1       0  0 10:11 pts/0    00:00:00 /bin/bash
1001          25       1  0 10:25 pts/0    00:00:00 ps -ef
I have no name!@7b9781de0eed:/$ exit
exit

dockerfile文件

root@master01:/data/mongodb# cat Dockerfile 
FROM bitnami/mongodb:4.0.14-debian-9-r24
USER root
RUN echo "tail -f /etc/hosts" > /setup.sh
RUN echo "tail -f /etc/hosts" > /run.sh

构建镜像&推送到一个具有公共读的镜像仓库中

root@master01:/data/mongodb# docker build -t xxx/xxx/mongodb-debug:v1 .

测试

root@master01:/data/mongodb# docker run -it xxx/xxx/mongodb-debug:v1 /bin/bash 11:15:17.58 11:15:17.58 Welcome to the Bitnami mongodb container11:15:17.58 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb11:15:17.58 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues11:15:17.58 Send us your feedback at containers@bitnami.com11:15:17.58 root@8af93d798c63:/# ls /bitnami/
mongodb
root@8af93d798c63:/# cd bitnami/
root@8af93d798c63:/bitnami# ls
mongodb

编辑部署文件

编辑mongodb的deployment文件，将自定义的mongodb镜像做为第二个容器镜像，并配置pvc

kubectl edit deployment mongodbcontainers:- name: mongodbdebugimage: xxx/istio_test/mongodb-debug:v1volumeMounts:- name: datamountPath: /bitnami/mongodb- name: mongodbimage: xxx/xxx/mongodb:4.0.14-debian-9-r24

进入自定义的容器内部执行修复

root@dev-k8s-master03:/data# kubectl exec -it mongodb-7f97f799d9-qz82v -c mongodbdebug -- bash 
root@mongodb-7f97f799d9-qz82v:/# ls bitnami/mongodb/data/db/
WiredTiger			      collection-11--5099670552419067160.wt  collection-4-5162049135609455324.wt    index-1--8902867768542019650.wt   index-2--5099670552419067160.wt	index-30--5099670552419067160.wt  index-44--5099670552419067160.wt  index-56--5099670552419067160.wt  index-7-5162049135609455324.w
.........root@mongodb-7f97f799d9-qz82v:/bitnami/mongodb/data/db# mongod --dbpath /bitnami/mongodb/data/db --repair2024-04-12T11:23:29.167+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] MongoDB starting : pid=40 port=27017 dbpath=/bitnami/mongodb/data/db 64-bit host=mongodb-7f97f799d9-qz82v
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] db version v4.0.14
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] git version: 1622021384533dade8b3c89ed3ecd80e1142c132
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] modules: none
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] build environment:
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     distmod: debian92
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     distarch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/bitnami/mongodb/data/db" } }
2024-04-12T11:23:29.175+0000 I STORAGE  [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2024-04-12T11:23:29.175+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31648M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-04-12T11:23:29.284+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
省略日志.......
2024-04-12T11:23:34.236+0000 W STORAGE  [initandlisten] Modifications made by repair:
2024-04-12T11:23:34.236+0000 W STORAGE  [initandlisten]   WiredTiger metadata salvaged
2024-04-12T11:23:34.236+0000 I STORAGE  [initandlisten] finished checking dbs
2024-04-12T11:23:34.236+0000 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2024-04-12T11:23:34.238+0000 I STORAGE  [initandlisten] Shutting down session sweeper thread
2024-04-12T11:23:34.238+0000 I STORAGE  [initandlisten] Finished shutting down session sweeper thread
2024-04-12T11:23:34.378+0000 I STORAGE  [initandlisten] shutdown: removing fs lock...
2024-04-12T11:23:34.380+0000 I CONTROL  [initandlisten] now exiting
2024-04-12T11:23:34.380+0000 I CONTROL  [initandlisten] shutting down with code:0

新的报错

mongodb提示读取数据没权限，看了一下是由于构建镜像时候用了root用户执行命令，导致修改后的文件的所有者是root，chown xxx mongodb数据库文件即可

验证

root@dev-k8s-master03:/data# kubectl get pod 
NAME                           READY   STATUS             RESTARTS   AGE
mongodb-68d99d8cf9-wlcn2       1/1     Running            0          2d16h
yapi-pod-6f889c4b68-6xgqc      1/1     Running            0          2d18h

yapi也能正常登录了。

refer

mongodb社区
https://www.mongodb.com/community/forums/t/my-mongodb-not-working-properly-wt-error-non-specific-wiredtiger-error/145434/4

官网修复意外断电导致的mongo报错
https://www.mongodb.com/zh-cn/docs/manual/tutorial/recover-data-following-unexpected-shutdown/

mongodb镜像层
https://hub.docker.com/layers/bitnami/mongodb/4.0.14-debian-9-r24/images/sha256-fd839bcba360b2f470d003706f21ad4b8b1306be0b74e3ab936ff69b72f2e5d3?context=explore