六、安装存储管理管理软件,并导出磁盘
[[email protected] ~]# fdisk /dev/xvdc \\存储节点挂有磁盘,空间1T
[[email protected] ~]# yum installscsi-target-utils -y \\安装target管理端
[[email protected] ~]# vim /etc/tgt/targets.conf \\配置导出磁盘的信息
其中backing-store指定共享的磁盘,incominguser指定用于验证的用户和密码,initiator-address允许连接的地址
[[email protected] ~]# /etc/rc.d/init.d/tgtd start &&chkconfig tgtd on
[[email protected] ~]# tgtadm --lld iscsi --mode target --op show \\查看是否导出成功
[[email protected] ~]# for i in {1..3}; do ssh openstack.node$i "yum -yinstall iscsi-initiator-utils"; done \\节点安装iscsi客户端软件
[[email protected] ~]# vim /etc/iscsi/iscsid.conf \\所有节点配置文件加上以下3行,设置账户密码
node.session.auth.authmethod = CHAP
node.session.auth.username = openstack
node.session.auth.password =oepnstack
[[email protected] ~] iscsiadm -mdiscovery -t st -p 192.168.249.64 //发现共享设备
192.168.249.64:3260,1 iqn.2015-05.com.si-tech:target1
[[email protected] ~]iscsiadm -m node –l iqn.2015-05.com.si-tech:target1 \\注册iscsi共享设备
[[email protected] ~]tgtadm --lld iscsi --op show --mode conn --tid 1 \\iscsi服务器端查看共享情况
[[email protected] ~]# netstat -nlatp | grep 3260 连接情况
[[email protected] ~]# fdisk -l \\ 在各个节点上面都会多出个iscsi设备
Disk /dev/sda: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
七、在节点上创建clvm逻辑卷并创建gfs2集群文件系统
[[email protected] ~]# pvcreate /dev/sda \\创建pv
Physical volume"/dev/sda" successfully created
[[email protected] ~]# vgcreate openstackvg0 /dev/sda \\创建vg
Clustered volume group "openstackvg0"successfully created
[[email protected] ~]# lvcreate -L +1000Gopenstackvg0 -n openstacklv0 \\创建大小为1T的lv
[[email protected] iscsi]# lvmconf--enable-cluster 每个节点开启集群LVM功能,才能mklv
[[email protected] iscsi]# lvcreate-L +100G openstackvg0 -n openstacklv0
Logical volume "openstacklv0" created
登陆其他节点验证,可以看到已经都有了
[[email protected] ~]# mkfs.gfs2 -j 3 -p lock_dlm -t openstack:gfs2/dev/openstackvg0/openstacklv0
-p:用来指定gfs的锁机制
-j:指定journal个数(可加入节点数),一般情况下应留有冗余,否则后期还得再调整
查看journals:# gfs2_tool journals /openstack
增加journals:# gfs2_jadd –j 1 /openstack ##增加一个journals
-t:格式为ClusterName:FS_Path_Name
ClusterName:应与前面cluster.conf中指定的集群名称相同;
FS_Path_Name:这个块设备mount的路径;
最后一个参数是指定逻辑卷的详细路径
\\创建gfs2集群文件系统,并设置节点为3个,锁协议为lock_dlm,openstack为我们的集群名,gfs2为一个标志也可以任意
[[email protected] usr]# mount/dev/openstackvg0/openstacklv0 /openstack/
fs is for a different cluster
error mounting lockproto lock_dlm
这是因为我们使用的集群时gfs所以改为gfs
[[email protected] usr]# mount /dev/openstackvg0/openstacklv0 /openstack/
[[email protected] usr]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_openstack-lv_root 35G 4.5G 29G 14% /
tmpfs 7.6G 32M 7.6G 1% /dev/shm
/dev/xvda1 485M 35M 426M 8% /boot
/dev/xvdd1 4.2G 4.2G 0 100% /mnt
/dev/mapper/openstackvg0-openstacklv0 110G 388M 110G 1% /openstack
echo "/dev/openstackvg0/openstacklv0 /openstack gfs2 defaults 00" >> /etc/fstab
八、集群管理
1)、当基于clvm的gfs2文件系统不够用时,如何增加
[[email protected]~]# lvextend -L +10G /dev/openstackvg0/openstacklv0
Extending logical volume openstacklv0 to 110.00 GiB
Logical volume openstacklv0successfully resized
[[email protected] ~]# gfs2_grow /openstack \\同步文件系统
FS: Mount Point: /openstack
FS: Device: /dev/dm-2
FS: Size: 28835838(0x1b7fffe)
FS: RG size: 65533 (0xfffd)
DEV: Size: 31457280(0x1e00000)
The file system grew by 10240MB.
gfs2_grow complete. gfs2_grow complete.
[[email protected] ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_openstack-lv_root 35G 4.5G 29G 14% /
tmpfs 7.6G 32M 7.6G 1% /dev/shm
/dev/xvda1 485M 35M 426M 8% /boot
/dev/xvdd1 4.2G 4.2G 0 100% /mnt
/dev/mapper/openstackvg0-openstacklv0 110G 388M 110G 1% /openstack
2)、当节点不够用时,如果添加一个新的节点加入集群
步骤如下:
1)、安装ricci
[[email protected] ~]# yum install ricci -y
2)、登录luci web,添加ricci
3)、导入共享存储设备
[[email protected] ~]# iscsiadm -m discovery -t st -p 192.168.249.64
[[email protected]~]# scp/etc/iscsi/iscsid.conf node4:/etc/iscsi/
[[email protected] ~]# iscsiadm -m node –l iqn.2015-05.com.si-tech:target1
4)、在已成功挂载的节点上增加节点数,并实现挂载使用
(注意:如果系统看不到/dev/openstackvg0/openstacklv0,则重新启动节点系统即可生效)
[[email protected] ~]# mount /dev/cvg0/clv0 /openstack/ \ \节点数不够
Too many nodes mounting filesystem, no free journals
[[email protected] ~]# gfs2_jadd -j 1 /optenstack \\增加一个节点数
Filesystem: /opt
Old Journals 3
NewJournals 4
[[email protected] ~]# mount /dev/cvg0/clv0 /opt/
[[email protected] ~]# ll /opt/
total 4
-rw-r--r-- 1 root root 210 Jun 8 00:42 test.txt
[[email protected] ~]#
3、)、测试基于gfs2文件系统的集群节点是否支持同时读写操作
[[email protected] ~]# echo node1 >>/openstack/test.txt\\节点1模拟向test.txt文件写入node1
[[email protected] ~]#echo node2 >>/openstack/test.txt \\节点2模拟向test.txt文件写入node2
[[email protected] ~]# tail -f /openstack/test.txt \\节点3模拟读出节点1和节点2同时写入的数据
node1
node2
4)、如果数据很重要并且磁盘空间也充足可以考虑使用cmirror实现存储高可用这里简述一下步骤
存储节点添加备份磁盘,修改targets.conf文件
数据节点使用iscsiadm命令发现并使用备份磁盘
数据节点需要按cmirror软件,并备份:yum install cmirror –y
lvconvert -m 1 /dev/openstackvg0/openstacklv0 /dev/sda /dev/sdb \\创建先有lv的mirror
[[email protected] ~]# dmsetup ls –tree \\查看现有iscsi导出设备的状态为mirror型
cvg0-clv0 (253:1)
├─cvg0-clv0_mimage_1 (253:5)
│└─iscsi2 (253:2)
│├─ (8:48)
│└─ (8:64)
├─cvg0-clv0_mimage_0 (253:4)
│└─iscsi1 (253:0)
│├─ (8:32)
│└─ (8:16)
└─cvg0-clv0_mlog (253:3)
└─iscsi2 (253:2)
├─ (8:48)
└─ (8:64)