问题

出现下面告警信息

 ceph -scluster:id:     xxxxxxxxxxxxxxxxxxxxxxxhealth: HEALTH_WARN7 large omap objects

获取问题 pool

# ceph health detail
HEALTH_WARN 7 large omap objects; 4 clients failing to respond to cache pressure
[WRN] LARGE_OMAP_OBJECTS: 7 large omap objects7 large objects found in pool 'k8s-metadata'

获取有问题 pg 信息

结果返回时间跟你当前集群拥有 PG 数量成正比
PG 越少，返回速度越快

for i in `ceph pg ls-by-pool k8s-metadata | tail -n +2 | head -n -2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i query | grep num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ": 1"
11.18a: 1,
11.1fc: 1,
11.215: 1,
11.44f: 1,
11.5d7: 1,
11.808: 1,
11.c23: 1,

获取每个 PG 信息

通过下面命令得知对应 OSD 信息

# for id in 11.18a 11.1fc 11.215 11.44f 11.5d7 11.808 11.c23doceph pg map $iddoneosdmap e64761 pg 11.18a (11.18a) -> up [44,215,107] acting [44,215,107]
osdmap e64761 pg 11.1fc (11.1fc) -> up [240,50,196] acting [240,50,196]
osdmap e64761 pg 11.215 (11.215) -> up [20,56,198] acting [20,56,198]
osdmap e64761 pg 11.44f (11.44f) -> up [64,102,89] acting [64,102,89]
osdmap e64761 pg 11.5d7 (11.5d7) -> up [71,28,228] acting [71,28,228]
osdmap e64761 pg 11.808 (11.808) -> up [11,8,219] acting [11,8,219]
osdmap e64761 pg 11.c23 (11.c23) -> up [28,43,108] acting [28,43,108]

针对 osd 执行 deep-scrub 操作

# for id in 44 215 107 240 50 196 20 56 198 64 102 89 71 28 228 11 8 219 28 43 108
doceph osd deep-scrub osd.$id
doneinstructed osd(s) 44 to deep-scrub
instructed osd(s) 215 to deep-scrub
instructed osd(s) 107 to deep-scrub
instructed osd(s) 240 to deep-scrub
instructed osd(s) 50 to deep-scrub
instructed osd(s) 196 to deep-scrub
instructed osd(s) 20 to deep-scrub
instructed osd(s) 56 to deep-scrub
instructed osd(s) 198 to deep-scrub
instructed osd(s) 64 to deep-scrub
instructed osd(s) 102 to deep-scrub
instructed osd(s) 89 to deep-scrub
instructed osd(s) 71 to deep-scrub
instructed osd(s) 28 to deep-scrub
instructed osd(s) 228 to deep-scrub
instructed osd(s) 11 to deep-scrub
instructed osd(s) 8 to deep-scrub
instructed osd(s) 219 to deep-scrub
instructed osd(s) 28 to deep-scrub
instructed osd(s) 43 to deep-scrub
instructed osd(s) 108 to deep-scrub