Proper Way to Remove an OSD
Run ‘ceph osd tree‘ to list OSD devices and look for the number of the failed OSD or the OSD to be removed.
# ceph osd tree
We are going to remove OSD.8 for example. Login to the OSD node first and start the following steps.
# ceph osd crush reweight osd.8 0
Once the command is executed, use ‘ceph -w‘ or ‘ceph -s‘ to monitor status. When it backed to healthy state (all the PG must be in active+clean state), we can start to remove the OSD.
# ceph osd out 8
# systemctl stop [email protected]
# ceph osd crush remove osd.8
# ceph auth del osd.8
# ceph osd rm 8
# umount /var/lib/ceph/osd/ceph-8
You may wipe out the disk by running
# sgdisk --zap-all -- /dev/vdb
# sgdisk --clear --mbrtogpt /dev/vdb
or just by going back to ceph admin node to run
# ceph-deploy disk zap Ceph01:vdb
If this is a failed hard drive, you may need to turn off the node for replacing it with a good one.
If the ceph.conf contains the OSD we have just removed, modify the ceph.conf appropriately and then inject the updated ceph.conf to all the Ceph nodes.
Add OSD
The simplest way to add OSD is using ‘ceph-deploy‘ command as below,
# ceph-deploy disk zap Ceph01:/dev/vdb
# ceph-deploy --overwrite-conf osd create Ceph01:vdb:[journal]
You may also add OSD from OSD node.
# ceph-disk prepare /dev/vdb [journal]
# ceph-disk activate /dev/vdb1
Note: Avoid to adopt the procedure of ‘Adding OSD Manually‘ on official Ceph website, there will be ownership issue on block device (e.q., /dev/sdb is not owned by ceph:ceph).