在上一篇中我们介绍了Glusterfs在CentOS7上的安装，并且提到Glusterfs client端有三种：Native client,NFS,Samba， 今天我们就来一起学习下这三种方式。
在开始之前，我已经做好了一个Volume:
[[email protected] ~]# gluster volume create repvol replica 2 > node01.lab.example.com:/bricks/thinvol1/brick > node02.lab.example.com:/bricks/thinvol1/brick volume create: repvol: success: please start the volume to access data [[email protected] ~]# gluster volume start repvol volume start: repvol: success [[email protected] ~]# gluster volume info repvol Volume Name: repvol Type: Replicate Volume ID: 7015b7fe-8039-4d5d-b698-e8321df5289c Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: node01.lab.example.com:/bricks/thinvol1/brick Brick2: node02.lab.example.com:/bricks/thinvol1/brick Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on
1 Native mount client
1). server 端无需做额外设置，只需正常安装，配置glusterfs服务即可
2). Client 端需要安装glusterfs client,这里我们还是把node02看做client
[[email protected] ~]# yum install -y glusterfs glusterfs-fuse [[email protected] ~]# mount -t glusterfs node01.lab.example.com:repvol /mnt [[email protected] ~]# df -h | grep mnt node01.lab.example.com:repvol 2.0G 33M 2.0G 2% /mnt
Native mount的一大优势是支持“高可用”，虽然我们挂载的时使用的命令是指定的node01.lab.example.com, 但是当node01不可用时，glusterfs volume 还是可以work的。我们可以stop node01上的glusterd service模拟node01出现故障，看是否影响volume repvol的使用。
[[email protected] ~]# systemctl stop glusterd
[[email protected] /]# df -h | grep mnt node01.lab.example.com:repvol 2.0G 33M 2.0G 2% /mnt [[email protected] /]# cd /mnt [[email protected] mnt]# touch test.file [[email protected] mnt]#
从上面可以看出volume repvol在node01 不可用时，还依然可用。
2 NFS
在查看volume详细信息时，我们会发现有一个选项“nfs.disable: on”,从字面上就可以理解默认是禁用了nfs的
[[email protected] /]# gluster volume info repvol Volume Name: repvol Type: Replicate Volume ID: 7015b7fe-8039-4d5d-b698-e8321df5289c Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: node01.lab.example.com:/bricks/thinvol1/brick Brick2: node02.lab.example.com:/bricks/thinvol1/brick Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on
手动放开nfs,并重启volume
[[email protected] ~]# gluster volume set repvol nfs.disable off volume set: success [[email protected] ~]# gluster volume stop repvol Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: repvol: success [[email protected] ~]# gluster volume start repvol volume start: repvol: success [[email protected] ~]# gluster volume info repvol | grep nfs.disable nfs.disable: off [[email protected] ~]#
同时server上会启动这样一个进程做一个nfs共享以便提供nfs挂载
[[email protected] ~]# ps -aux | grep nfs | grep -v grep root 20064 0.1 6.5 581640 65028 ? Ssl 04:23 0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/d8b498939d669fdc978b5b3d96d5aa9d.socket
放开防火墙限制
[[email protected] ~]# firewall-cmd --add-service=nfs --add-service=rpc-bind success
在node02上使用nfs方式挂载volume repvol
[[email protected] /]# mount -t nfs node01.lab.example.com:repvol /mnt [[email protected] /]# df -h | grep mnt node01.lab.example.com:repvol 2.0G 33M 2.0G 2% /mnt
这里可以看出nfs挂载似乎也不复杂，而且client上不需要安装额外的glusterfs组件。
3 Samba
将option "nfs.disable"恢复默认设置，并重启volume
[[email protected] glusterfs]# gluster volume reset repvol nfs.disable volume reset: success: reset volume successful
[[email protected] ~]# gluster volume stop repvol Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: repvol: success
[[email protected] ~]# gluster volume start repvol volume start: repvol: success
[[email protected] ~]# gluster volume info repvol | grep nfs.disable nfs.disable: on
3.1在node01上安装samba,启动samba service并放开防火墙限制
[[email protected] ~]# yum install -y samba [[email protected] ~]# systemctl start smb [[email protected] ~]# firewall-cmd --add-service=samba
3.2修改volume option并重启volume, glusterd service
编辑/etc/glusterfs/glusterd.vol 文件加入下面一行
option rpc-auth-allow-insecure on
[[email protected] ~]# gluster volume set repvol server.allow-insecure on [[email protected] ~]# gluster volume set repvol storage.batch-fsync-delay-usec 0
[[email protected] ~]# gluster volume set repvol stat-prefetch off
[[email protected] ~]# systemctl restart glusterd [[email protected] ~]# gluster volume stop repvol [[email protected] ~]# gluster volume start repvol
---查看一下samba配置文件，可以看到最下面多出一个关于repvol的共享：
[[email protected] ~]# cat /etc/samba/smb.conf
......
[gluster-repvol]
comment = For samba share of volume repvol
vfs objects = glusterfs
glusterfs:volume = repvol
glusterfs:logfile = /var/log/samba/glusterfs-repvol.%M.log
glusterfs:loglevel = 7
path = /
read only = no
guest ok = yes
......
3.3创建一个samba user
[[email protected] ~]# adduser -s /sbin/nologin smbuser
[[email protected] ~]# smbpasswd -a smbuser New SMB password: Retype new SMB password: Added user smbuser.
在node01 server 上挂载volume repvol到/mnt目录下，并设修改目录权限如下
[[email protected] ~]# mount -t glusterfs node01.lab.example.com:repvol /mnt [[email protected] ~]# chown smbuser: /mnt [[email protected] ~]# umount /mnt
3.4 在node02上安装samba-client,cifs-utils
[[email protected] ~]# yum install -y cifs-utils [[email protected] ~]# yum install -y samba-client*
验证samba共享目录
[[email protected] ~]# smbclient -L node01.lab.example.com -U%
Domain=[SAMBA] OS=[Windows 6.1] Server=[Samba 4.4.4]
Sharename Type Comment
--------- ---- -------
print$ Disk Printer Drivers
IPC$ IPC IPC Service (Samba 4.4.4)
gluster-repvol Disk For samba share of volume repvol
Domain=[SAMBA] OS=[Windows 6.1] Server=[Samba 4.4.4]
Server Comment
--------- -------
Workgroup Master
--------- -------
使用smbuser进行挂载
[[email protected] ~]# mount -t cifs -o user=smbuser,pass=redhat //node01.lab.example.com/gluster-repvol /mnt Retrying with upper case share name mount error(6): No such device or address Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
查看samba日志：/var/log/samba/log.smbd发现了如下一些messages:
[2017/07/25 08:39:56.358336, 0] ../lib/util/modules.c:48(load_module)
Error loading module ‘/usr/lib64/samba/vfs/glusterfs.so‘: /usr/lib64/samba/vfs/glusterfs.so: cannot open shared object file: No such file or directory
google了很久，得到提示需要安装samba-vfs-glusterfs.x86_64，然后yum进行安装
再次尝试挂载
[[email protected] ~]# mount -t cifs //node01.lab.example.com/gluster-repvol /mnt -o username=smbuser,password=redhat mount error(5): Input/output error Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
发现错误已经不同了，再次找到node01下面的samba日志，发现
[[email protected] samba]# cd /var/log/samba/ [[email protected] samba]# ls cores glusterfs-repvol.192.168.133.135.log log.smbd old
192.168.133.135即为node02的ip，打开这个glusterfs-repvol.192.168.133.135.log后发现如下错误：
[2017-07-25 13:09:46.473070] E [socket.c:3097:socket_connect] 0-gfapi: connection attempt on failed, (Permission denied)
[2017-07-25 13:09:46.473307] I [MSGID: 104025] [glfs-mgmt.c:752:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all volfile servers [Transport endpoint is not connected]
[2017-07-25 13:09:46.474733] I [MSGID: 101191] [event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2017-07-25 13:09:46.473303] E [MSGID: 104024] [glfs-mgmt.c:735:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with remote-host: localhost (Transport endpoint is not connected) [Transport endpoint is not connected]
权限有问题！！！而权限无外乎就是目录本身权限和selinux权限，于是关闭了node01上的selinux，然后尝试挂载，这次挂载成功。哈哈。。。不过selinux这里的设置还没有研究出该怎么配置，先记在这里。
[[email protected] samba]# setenforce 0
[[email protected] ~]# mount -t cifs //node01.lab.example.com/gluster-repvol /mnt -o username=smbuser,password=redhat [[email protected] ~]# df -h | grep mnt //node01.lab.example.com/gluster-repvol 2.0G 33M 2.0G 2% /mnt [[email protected] ~]#