ceph osd盘挂掉，无法修复

由于网络更换ip段，导致ceph启动时monitor进程无法启动，解决了这个问题后，ceph能够启动，但一块osd盘坏掉了，错误日志如下：

2014-12-24 10:53:30.353262 7f3fbbd78800  0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 9794
2014-12-24 10:53:30.359829 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) backend xfs (magic 0x58465342)
2014-12-24 10:53:30.536660 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is supported and appears to work
2014-12-24 10:53:30.536683 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is disabled via ‘filestore fiemap‘ config option
2014-12-24 10:53:30.537600 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: syscall(SYS_syncfs, fd) fully supported
2014-12-24 10:53:30.537684 7f3fbbd78800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_feature: extsize is disabled by conf
2014-12-24 10:53:30.537684 7f3fbbd78800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_feature: extsize is disabled by conf
2014-12-24 10:53:30.592079 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-12-24 10:53:40.394856 7f3fbbd78800 -1 os/FileStore.cc: In function ‘int FileStore::_check_replay_guard(int, const SequencerPosition&)‘ thread 7f3fbbd78800 time 2014-12-24 10:53:40.391498
os/FileStore.cc: 2181: FAILED assert(!m_filestore_fail_eio || r != -5)

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (FileStore::_check_replay_guard(int, SequencerPosition const&)+0x2d3) [0x730633]
 2: (FileStore::_check_replay_guard(coll_t, ghobject_t, SequencerPosition const&)+0x148) [0x742918]
 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x58a4) [0x75e7a4]
 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x763894]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x47c) [0x77578c]
 6: (FileStore::mount()+0x2baa) [0x751d4a]
 7: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x64e75a]
 8: (main()+0x33e4) [0x613044]
 9: (__libc_start_main()+0xfd) [0x31e6a1ed1d]
 10: /usr/bin/ceph-osd() [0x60ec19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -21> 2014-12-24 10:53:30.348526 7f3fbbd78800  5 asok(0x5278230) register_command perfcounters_dump hook 0x5200010
   -20> 2014-12-24 10:53:30.348569 7f3fbbd78800  5 asok(0x5278230) register_command 1 hook 0x5200010
   -19> 2014-12-24 10:53:30.348579 7f3fbbd78800  5 asok(0x5278230) register_command perf dump hook 0x5200010
   -18> 2014-12-24 10:53:30.348590 7f3fbbd78800  5 asok(0x5278230) register_command perfcounters_schema hook 0x5200010
   -17> 2014-12-24 10:53:30.348600 7f3fbbd78800  5 asok(0x5278230) register_command 2 hook 0x5200010
   -19> 2014-12-24 10:53:30.348579 7f3fbbd78800  5 asok(0x5278230) register_command perf dump hook 0x5200010
   -18> 2014-12-24 10:53:30.348590 7f3fbbd78800  5 asok(0x5278230) register_command perfcounters_schema hook 0x5200010
   -17> 2014-12-24 10:53:30.348600 7f3fbbd78800  5 asok(0x5278230) register_command 2 hook 0x5200010
   -16> 2014-12-24 10:53:30.348609 7f3fbbd78800  5 asok(0x5278230) register_command perf schema hook 0x5200010

   -15> 2014-12-24 10:53:30.348618 7f3fbbd78800  5 asok(0x5278230) register_command config show hook 0x5200010
asok(0x5278230) register_command config show hook 0x5200010
   -14> 2014-12-24 10:53:30.348629 7f3fbbd78800  5    -14> 2014-12-24 10:53:30.348629 7f3fbbd78800  5 asok(0x5278230) register_command config set hook 0x5200010
   -13> 2014-12-24 10:53:30.348638 7f3fbbd78800  5 asok(0x5278230) register_command config get hook 0x5200010asok(0x5278230) register_command config get hook 0x5200010
   -12> 2014-12-24 10:53:30.348647 7f3fbbd78800  5 asok(0x5278230) register_command config diff hook 0x5200010asok(0x5278230) register_command config diff hook 0x5200010
   -11> 2014-12-24 10:53:30.348657 7f3fbbd78800  5 asok(0x5278230) register_command log flush hook 0x5200010asok(0x5278230) register_command log flush hook 0x5200010
   -10> 2014-12-24 10:53:30.348666 7f3fbbd78800  5 asok(0x5278230) register_command log dump hook 0x5200010asok(0x5278230) register_command log dump hook 0x5200010
    -9> 2014-12-24 10:53:30.348675 7f3fbbd78800  5 asok(0x5278230) register_command log reopen hook 0x5200010
    -8> 2014-12-24 10:53:30.353262 7f3fbbd78800  0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 9794
    -7> 2014-12-24 10:53:30.355214 7f3fbbd78800  1 finished global_init_daemonize
    -6> 2014-12-24 10:53:30.359829 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) backend xfs (magic 0x58465342)
    -5> 2014-12-24 10:53:30.536660 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is supported and appears to work
    -4> 2014-12-24 10:53:30.536683 7f3fbbd78800  0     -5> 2014-12-24 10:53:30.536660 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is supported and appears to work
    -4> 2014-12-24 10:53:30.536683 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: FIEMAP ioctl is disabled via ‘filestore fiemap‘ config option
    -3> 2014-12-24 10:53:30.537600 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/o    -3> 2014-12-24 10:53:30.537600 7f3fbbd78800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_features: syscall(SYS_syncfs, fd) fully supported
    -2> 2014-12-24 10:53:30.537684 7f3fbbd78800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-4) detect_feature: extsize is disabled by conf

    -1> 2014-12-24 10:53:30.592079 7f3fbbd78800  0 filestore(/var/lib/ceph/osd/ceph-4) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled

     0> 2014-12-24 10:53:40.394856 7f3fbbd78800 -1 os/FileStore.cc: In function ‘int FileStore::_check_replay_guard(int, const SequencerPosition&)‘ thread 7f3fbbd78800 time 2014-12-24 10:53:40.391498
os/FileStore.cc: 2181: FAILED assert(!m_filestore_fail_eio || r != -5)

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (FileStore::_check_replay_guard(int, SequencerPosition const&)+0x2d3) [0x730633]
 2: (FileStore::_check_replay_guard(coll_t, ghobject_t, SequencerPosition const&)+0x148) [0x742918]
 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x58a4) [0x75e7a4]
 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x763894]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x47c) [0x77578c]
 6: (FileStore::mount()+0x2baa) [0x751d4a]
 7: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x64e75a]
 8: (main()+0x33e4) [0x613044]
 9: (__libc_start_main()+0xfd) [0x31e6a1ed1d]
 10: /usr/bin/ceph-osd() [0x60ec19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 noneos/FileStore.cc: In function ‘int FileStore::_check_replay_guard(int, const SequencerPosition&)‘ thread 7f3fbbd78800 time 2014-12-24 10:53:40.391498
os/FileStore.cc: 2181: FAILED assert(!m_filestore_fail_eio || r != -5)

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (FileStore::_check_replay_guard(int, SequencerPosition const&)+0x2d3) [0x730633]
 2: (FileStore::_check_replay_guard(coll_t, ghobject_t, SequencerPosition const&)+0x148) [0x742918]
 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x58a4) [0x75e7a4]
 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x763894]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x47c) [0x77578c]
 6: (FileStore::mount()+0x2baa) [0x751d4a]
 7: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x64e75a]
 8: (main()+0x33e4) [0x613044]
 9: (__libc_start_main()+0xfd) [0x31e6a1ed1d]
 10: /usr/bin/ceph-osd() [0x60ec19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   1/ 5 mds_balancer
   1/ 5 mds_balancer
   1/ 5 mds_locker   1/ 5 mds_locker
   1/ 5 mds_log   1/ 5 mds_log
   1/ 5 mds_log_expire

   1/ 5 mds_migrator   1/ 5 mds_migrator
   0/ 0 buffer

   0/ 0 timer

   0/ 1 filer

   0/ 1 striper

   0/ 1 objecter

   0/ 5 rados

   0/ 0 rbd

   0/ 5 rbd_replay

   0/ 0 journaler

   0/ 5 objectcacher
   0/ 5 client   0/ 5 client
   0/ 0 osd   0/ 0 osd
   0/ 0 optracker   0/ 0 optracker
   0/ 0 objclass   0/ 0 objclass
   0/ 0 filestore   0/ 0 filestore
   1/ 3 keyvaluestore
   1/ 3 keyvaluestore
   0/ 0 journal

   0/ 0 ms

   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp

   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   1/ 5 crypto
   0/ 0 finisher

   0/ 0 heartbeatmap

   0/ 0 perfcounter

   1/ 5 rgw

   1/10 civetweb

   1/ 5 javaclient

   0/ 0 asok

   0/ 0 throttle

   0/ 0 refs

  -2/-2 (syslog threshold)

  -1/-1 (stderr threshold)

  max_recent     10000

  max_new         1000

  log_file /var/log/ceph/ceph-osd.4.log

--- end dump of recent events ---
2014-12-24 10:53:40.400275 7f3fbbd78800 -1 *** Caught signal (Aborted) **
 in thread 7f3fbbd78800

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: /usr/bin/ceph-osd() [0xa02425]
 2: /lib64/libpthread.so.0() [0x31e6e0f710]
 3: (gsignal()+0x35) [0x31e6a32925]
 4: (abort()+0x175) [0x31e6a34105]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x31f2abea7d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2014-12-24 10:53:40.400275 7f3fbbd78800 -1 *** Caught signal (Aborted) **
 in thread 7f3fbbd78800

 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: /usr/bin/ceph-osd() [0xa02425]
 2: /lib64/libpthread.so.0() [0x31e6e0f710]
 3: (gsignal()+0x35) [0x31e6a32925]
 4: (abort()+0x175) [0x31e6a34105]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x31f2abea7d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   1/ 5 mds_balancer

   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   1/ 5 mds_migrator
   0/ 0 buffer

   0/ 0 timer
   0/ 1 filer

   0/ 1 striper

   0/ 1 objecter

   0/ 5 rados

   0/ 0 rbd

   0/ 5 rbd_replay

   0/ 0 journaler

   0/ 5 objectcacher

   0/ 5 client

   0/ 0 osd

   0/ 0 optracker

   0/ 0 objclass

   0/ 0 filestore

   1/ 3 keyvaluestore

   0/ 0 journal

   0/ 0 ms

   0/ 0 mon

   0/ 0 monc

   0/ 0 paxos
   0/ 0 tp   0/ 0 tp
   0/ 0 auth   0/ 0 auth
   1/ 5 crypto   1/ 5 crypto
   0/ 0 finisher   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter   0/ 0 heartbeatmap
   0/ 0 perfcounter
   1/ 5 rgw

   1/10 civetweb

   1/ 5 javaclient

   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.4.log
--- end dump of recent events ---

暂时无法解决，有遇到过类似问题的朋友吗？

时间： 2024-08-25 16:16:10

ceph osd盘挂掉，无法修复的相关文章

ip改变引起的ceph monitor异常及osd盘崩溃的总结

公司搬家,所有服务器的ip改变.对ceph服务器配置好ip后启动,发现monitor进程启动失败,monitor进程总是试图绑定到以前的ip地址,那当然不可能成功了.开始以为服务器的ip设置有问题,在改变hostname.ceph.conf等方法无果后,逐步分析发现,是monmap中的ip地址还是以前的ip,ceph通过读取monmap来启动monitor进程,所以需要修改monmap.方法如下: #Add the new monitor locations # monmaptool --cre

ISCSI网盘RAW格式修复

ISCSI网盘RAW格式修复: cmd--chkdsk G:/F  (G为ISCSI网盘)

Ceph osd异常退出故障处理

如果一个硬盘故障导致osd节点出现如下的down状态,且一直无法恢复( reweight列等于0,表示osd已经out此集群) [[email protected] ~]# ceph osd tree # id    weight  type name       up/down reweight -1      4       root default -2      1               host os-node5 0       1                       o

更换硬盘跳线，ceph osd 不能正常启动

1.环境说明 使用kolla 部署了ceph,由于osd 0占用于 sata 0通道,需要将系统盘与osd 0的跳线交换,在跳线交换后,osd 0不能正常启动. 2.原因分析: 交换跳线前,osd 0的设备文件为/dev/sda2 ,交换跳线后变为/dev/sdc2,osd在启动时,--osd-journal /dev/sda2,指定日志设备,由于跳线变更后,日志分区设备名称变为/dev/sdc2,而osd 启动依然使用/dev/sda2作日志设备,故启动失败. 3.解决办法: 删除原有dock

Win8.1 + CentOS7 双系统 U盘安装--引导修复

安装完重启后,竟然没有引导!!!(难道是由于Windows 8在BIOS中使用了统一的 UEFI 接口??但是,Linux基金会早就宣布将会提供一种全新的 方式使用 UEFI Secure Boot选项安装Linux系统.她两可以和平共处啊) 本人对 UEFI 引导不太了解(UEFI boot 比 Legacy boot 要强大 ),只能另想办法了!!最终,决定采用 easyBCD 来引导 CentOS 7. 1.失败的尝试(收获巨大,得到了启动的方式) 第一次尝试中,以为只要简单的添加 gru

ceph osd 删除

1.简介     一般 情况下,osd 是不需要 进行 删除 操作的,但是了,凡事 无绝对:很多时候 还是需要 进行删除 操作(例如,对已经 部署好的 存储 集群 重新 进行 方案的 调整,就需要 对 osd 进行 删除 操作,重新 进行 调整)     2.out osd ## 将osd out ## reweight 会被置为 0,集群会进行数据迁移 ## 相当于进行 "ceph osd reweight <osdname (id|osd.id)> <float[0.0-1

ceph osd weight、reweight 数值调整

1.简介 ????ceph 集群搭建成功后,很多时候 pg 分布是 不均匀的,此时 就需要 我们 通过 手动调整 weight值,从而 达到 pg相对均衡(本文 针对 有ceph 基础的人,所以 命令 不会 进行解释)???? # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.20181 root default -7 0.05499 host mon0 0 hdd 0.01799 osd.0 up 1.

U盘 出现RD盘符 容量修复

前几天,用PowerISO制作了一个U盘启动,制作完之后,一看U盘,怎么只剩下几M了,原本我的U盘是8G的,怎么会用了那么多的容量-_-!!!!,后来我就把U盘给格式化,结果盘符都出不来了,插到别的电脑U盘也显示不出来了. 后来自己鼓捣以下,安装个硬盘分区工具DiskGenius,在DiskGenius中看到U盘的盘符是RD1,我也不知道是神马东东来的,格式化还会出错,然后就乱点那些神马磁盘修复之类,但太久了,没耐心等(俺是最没耐心的^-^),后来就点击HDD-ZIP 制作U盘启动,就可以把原来

U盘目录损坏修复方法

一大早打开U盘发现无法保存文档,继而弹出目录损坏的窗口,要求格式化. 现提供解决方法: 1.窗口左下方windows键,输入框输入chkdsk F: /f,这里F代表U盘名为F,根据具体情况写. 2.回车,等待20秒.成功修复.