Glusterfs 分布式存储部署 是存储当中可以选择的一种 现在很多虚拟化 云计算都在用软件存储 例如 ceph
Glusterfs 等等 今天我们部署一下Glusterfs环境
GlusterFs安装
环境信息：
| Glusterfs-A | 10.20.0.200 |
| Glusterfs-B | 10.20.0.201 |
| Glusterfs-C | 10.20.0.202 |
服务器版本信息
# cat /etc/redhat-release CentOS release 6.6 (Final)
# uname -r
2.6.32-504.el6.x86_64
软件下载地址
http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.6/CentOS/epel-6.6/x86_64
依赖包安装
直接用Yum安装 每台都需要安装
# yum -y install libibverbs librdmacm xfsprogs nfs-utils rpcbind libaio liblvm2app lvm2-devel
软件安装
安装服务器端软件包
glusterfs-3.4.6-1.el6.x86_64.rpm glusterfs-api-3.4.6-1.el6.x86_64.rpm glusterfs-cli-3.4.6-1.el6.x86_64.rpm glusterfs-fuse-3.4.6-1.el6.x86_64.rpm glusterfs-libs-3.4.6-1.el6.x86_64.rpm glusterfs-server-3.4.6-1.el6.x86_64.rpm
安装客户端软件包
glusterfs-3.4.6-1.el6.x86_64.rpm glusterfs-fuse-3.4.6-1.el6.x86_64.rpm glusterfs-libs-3.4.6-1.el6.x86_64.rpm
磁盘配置
磁盘分区：
EXT4格式化工具目前最大支持16TB
使用parted进行磁盘分区
分区格式化
每个服务器上面都需要至少添加一块硬盘 我们这边是用的1块8G硬盘 现在每台都格式化
# mkfs.ext4 -L /brick1 /dev/sdb
mke2fs 1.41.12 (17-May-2010)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=/brick1
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
524288 inodes, 2097152 blocks
104857 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2147483648
64 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
自动挂载分区
# vim /etc/fstab LABEL=/brick1 /brick1 ext4 defaults 0 0
mount文件系统
# mkdir /brick1
# mount -L /brick1 /brick1/
# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
47G 759M 44G 2% /
tmpfs 491M 0 491M 0% /dev/shm
/dev/sda1 477M 28M 425M 6% /boot
/dev/sdb 7.8G 18M 7.4G 1% /brick1
基础测试包安装 性能测试包
工具软件
Atop,iperf,sysstat
dd,lozone,fio,postmark
工具安装
#rpm -ivh http://mirrors.ustc.edu.cn/fedora/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum install sysstat
Gcc -o postmark postmark-1.52.c
有些可以用yum安装
安装GlusterFS软件包
每台创建存放软件的目录 我们目前安装的是3.4.6的版本
#mdkir /home/tools/3.4.6 -p#cd /home/tools/3.4.6 # ll total 2164 -rw-r--r-- 1 root root 1027320 Oct 4 11:13 glusterfs-3.4.6-1.el6.x86_64.rpm -rw-r--r-- 1 root root 66680 Oct 4 11:13 glusterfs-api-3.4.6-1.el6.x86_64.rpm -rw-r--r-- 1 root root 108872 Oct 4 11:14 glusterfs-cli-3.4.6-1.el6.x86_64.rpm -rw-r--r-- 1 root root 88452 Oct 4 11:14 glusterfs-fuse-3.4.6-1.el6.x86_64.rpm -rw-r--r-- 1 root root 124288 Oct 4 11:14 glusterfs-geo-replication-3.4.6-1.el6.x86_64.rpm -rw-r--r-- 1 root root 227900 Oct 4 11:14 glusterfs-libs-3.4.6-1.el6.x86_64.rpm -rw-r--r-- 1 root root 50128 Oct 4 11:14 glusterfs-rdma-3.4.6-1.el6.x86_64.rpm -rw-r--r-- 1 root root 504744 Oct 4 11:14 glusterfs-server-3.4.6-1.el6.x86_64.rpm
直接rpm安装
# rpm -ivh *.rpm warning: glusterfs-3.4.6-1.el6.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 4ab22bb3: NOKEY Preparing... ########################################### [100%] 1:glusterfs-libs ########################################### [ 13%] 2:glusterfs ########################################### [ 25%] 3:glusterfs-fuse ########################################### [ 38%] 4:glusterfs-cli ########################################### [ 50%] 5:glusterfs-server ########################################### [ 63%] 6:glusterfs-geo-replicati########################################### [ 75%] 7:glusterfs-api ########################################### [ 88%] 8:glusterfs-rdma ########################################### [100%]
安装成功启动GlusterFS
# service glusterd start Starting glusterd: [ OK ] # chkconfig glusterd on
组建集群扩展
在GlusterFS-01服务器上面操作 选择10.20.0.200 作为当前集群节点
添加另外两台服务器
# gluster peer probe 10.20.0.201 peer probe: success # gluster peer probe 10.20.0.202 peer probe: success
GlusterFS-01服务器查看添加状态
# gluster peer status Number of Peers: 2 Hostname: 10.20.0.201 Port: 24007 Uuid: c4d9edfd-d6ae-4628-8ab3-5a9fbb5f18fc State: Peer in Cluster (Connected) Hostname: 10.20.0.202 Port: 24007 Uuid: c0b425bb-a7b4-4dae-bc19-ea6031bf02b9 State: Peer in Cluster (Connected)
GlusterFS-02服务器查看添加状态
# gluster peer status Number of Peers: 2 Hostname: 10.20.0.200 Port: 24007 Uuid: 7ad4db05-ee75-44e6-8213-e58072b62d33 State: Peer in Cluster (Connected) Hostname: 10.20.0.202 Uuid: c0b425bb-a7b4-4dae-bc19-ea6031bf02b9 State: Peer in Cluster (Connected)
GlusterFS-03服务器查看添加状态
# gluster peer status Number of Peers: 2 Hostname: 10.20.0.200 Port: 24007 Uuid: 7ad4db05-ee75-44e6-8213-e58072b62d33 State: Peer in Cluster (Connected) Hostname: 10.20.0.201 Uuid: c4d9edfd-d6ae-4628-8ab3-5a9fbb5f18fc State: Peer in Cluster (Connected)
表示成功
如果添加不成功 排除错误：
1，每台服务器是否能否Ping通
2，防火墙是否关闭iptables selinux
3，glusterd服务是否启动成功
创建卷
- 确定创建卷的类型
- 确定创建卷的brick列表
- 确实创建卷的网络类型（TCP/RDMA）
Gluster volume create创建卷
在GlusterFS-01服务器上面操作 可以任选一台
# gluster volume create dht-vol1 10.20.0.200:/brick1/b1 10.20.0.201:/brick1/b2 10.20.0.202:/brick1/b 3 volume create: dht-vol1: success: please start the volume to access data
查看卷状态信息
# gluster volume info Volume Name: dht-vol1 Type: Distribute Volume ID: 7d947cf1-c743-477e-a28d-7d030572b655 Status: Created Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 10.20.0.200:/brick1/b1 Brick2: 10.20.0.201:/brick1/b2 Brick3: 10.20.0.202:/brick1/b3
启动卷
# gluster volume start dht-vol1 volume start: dht-vol1: success
时间： 2024-10-10 14:03:03