ceph集群报 Monitor clock skew detected 错误问题排查,解决
告警信息如下:
[[email protected] ceph]# ceph -w
cluster ddc1b10b-6d1a-4ef9-8a01-d561512f3c1d
health HEALTH_WARN
clock skew detected on mon.ceph-100-81, mon.ceph-100-82
Monitor clock skew detected
monmap e1: 3 mons at {ceph-100-80=172.16.100.80:6789/0,ceph-100-81=172.16.100.81:6789/0,ceph-100-82=172.16.100.82:6789/0}
election epoch 22, quorum 0,1,2 ceph-100-80,ceph-100-81,ceph-100-82
mdsmap e21: 1/1/1 up {0=ceph-100-80=up:active}, 2 up:standby
osdmap e116: 6 osds: 6 up, 6 in
pgmap v205: 576 pgs, 3 pools, 1962 bytes data, 20 objects
234 MB used, 269 GB / 269 GB avail
576 active+clean
1:添加配置参数:
vim /etc/ceph/ceph.conf
[mon.ceph-100-80]
host = ceph-100-80
mon_data = /var/lib/ceph/mon/ceph-ceph-100-80/
mon_addr = 172.16.100.80:6789
# 添加内容如下:
mon clock drift allowed = 2
mon clock drift warn backoff = 30
2:同步配置文件
ceph-deploy --overwrite-conf admin ceph-100-{80..82}
3:重启mon 服务
/etc/init.d/ceph restart mon
4:验证:
[[email protected] ceph]# ceph -w
cluster ddc1b10b-6d1a-4ef9-8a01-d561512f3c1d
health HEALTH_OK
monmap e1: 3 mons at {ceph-100-80=172.16.100.80:6789/0,ceph-100-81=172.16.100.81:6789/0,ceph-100-82=172.16.100.82:6789/0}
election epoch 24, quorum 0,1,2 ceph-100-80,ceph-100-81,ceph-100-82
mdsmap e21: 1/1/1 up {0=ceph-100-80=up:active}, 2 up:standby
osdmap e116: 6 osds: 6 up, 6 in
pgmap v205: 576 pgs, 3 pools, 1962 bytes data, 20 objects
234 MB used, 269 GB / 269 GB avail
576 active+clean
再次查看,告警内容消失。
问题总结:
本问题主要是mon节点服务器,时间偏差比较大导致,本次遇到问题为测试环境,通过修改ceph对时间偏差阀值,规避的告警信息,线上业务环境,注意排查服务器时间同步问题。