1、初识分布式文件系统
在生产环境中，后端存储服务器服务器是一个很重要的环节。如果一旦前段服务器需要大量进行IO操作，会对后端存储的文件系统造成很大的压力。一旦后端的存储文件系统down机，那会导致整个生产环境奔溃。因为在生产环境中，后端的存储文件系统会成为一个单点故障。而为了避免因为单点故障导致整个生产环境无法正常使用，所有大家都会采用分布式文件系统，以避免因为单点故障导致生产环境奔溃。
2、mogilefs组成部分
MogileFS由3个部分组成：
(1) server：主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它通过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它其实是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将所有的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。
(2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。
(3) 客户端API：MogileFS的客户端API很多，例如Perl、PHP、Java、Python等，用这个模块可以编写客户端程序，实现文件的备份管理功能等。
存储主机(节点)
这个是 MogileFS 存储文件存放在这些机器上,也是 mogstored 节点,也叫 Storage Server,一台存储主要都要启动一个 mogstored 服务.扩容就是增加这些机器.
设备(device)
一个存储节点,以就是上面的主机,可以有多个 device, 就是用来存放文件的目录(例如挂载的目录),每个设备都有一个设备id,需要在 mogstored 的配置文件中的 docroot 配置的项目 指定的目录下面创建相应的设备的目录,目录名为 $docroot/dev$id,设备是不能删除的.只能将其设备的状态的值置为dead,当一个设备 dead 之后,就真的 dead了,里面的数据也无法恢复了,且这个dead了的设备的 id 也不能再用.
3、分布式文件系统需要的程序包
MogileFS-Server-2.46-2.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm
MogileFS-Utils-2.19-1.el6.noarch.rpm
Perlbal-1.78-1.el6.noarch.rpm
Perlbal-doc-1.78-1.el6.noarch.rpm
perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm
perl-IO-AIO-3.71-2.el6.x86_64
perl-MogileFS-Client-1.14-1.el6.noarch.rpm
perl-Net-Netmask-1.9015-8.el6.noarch.rpm
perl-Perlbal-1.78-1.el6.noarch.rpm
4、接着去安装所提供的所有的软件包
[[email protected] mogilefs]# yum install perl-IO-AIO perl-Net-Netmask perl-Sys-Syslog
[[email protected] mogilefs]# yum install MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm
perl-MogileFS-Client-1.14-1.el6.noarch.rpm MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm
或者在安装包的目录下把所有rpm一次性安装
[[email protected] mogilefs]# yum install *.rpm
5、接着去mysql数据库上进行授权
[[email protected] ~]# mysql
mysql> grant all on *.* to ‘root‘@‘172.16.17.%‘ identified by ‘xiaopeng‘;
Query OK, 0 rows affected (0.00 sec)
mysql> create database mogilefs;
Query OK, 1 row affected (0.03 sec)
mysql> grant all on mogilefs.* to ‘moguser‘@‘172.16.17.%‘ identified by ‘mogpass‘;
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
[[email protected] ~]#
6、数据库授权完成之后，接着去mogilefs文件系统的主控制台上，对数据库进行初始化。（建议此命令慎用，如果mogilefs文件系统在运行的时候，小心使用。）
[[email protected] mogilefs]# mogdbsetup --dbhost=172.16.17.144 --dbrootpass=‘xiaopeng‘ --dbuser=‘moguser‘ --dbpass=‘mogpass‘
This will attempt to setup or upgrade your MogileFS database.
It won‘t destroy existing data.
Run with --help for more information. Run with --yes to shut up these prompts.
Continue? [N/y]: y
完成之后，可以去数据库确认初始化情况
[[email protected] mogilefs]#
mysql> use mogilefs
Database changed
mysql> show tables;
+----------------------+
| Tables_in_mogilefs |
+----------------------+
7、数据库准备完成之后，接着去修改控制台文件
[[email protected] ~]# vim /etc/mogilefs/mogilefsd.conf
daemonize = 1
pidfile = /var/run/mogilefsd/mogilefsd.pid
db_dsn = DBI:mysql:mogilefs:host=172.16.100.68
db_user = moguser
db_pass = mogpass
listen = 0.0.0.0:7001
conf_port = 7001
query_jobs = 5
delete_jobs = 1
replicate_jobs = 2
reaper_jobs = 1
8、主配置文件修改完成之后，接着所有的存储节点都需要进行一下的配置
[[email protected] ~]# mkdir -pv /data/mogilefs/dev1
[[email protected] ~]# chown -R mogilefs.mogilefs /data/mogilefs/
[[email protected] ~]# vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data/mogilefs/
9、对于mogstord的启动文件（所有的节点都的mogstored的启动脚本文件需要加以下的几项）
为脚本添加：
pidfile=/var/run/mogilefsd/mogstored.pid
echo $(pidof mogstored) > ${pidfile}
rm -f ${pidfile}
10、所有的配置修改完成之后，接着去启动服务器，进行配置资源
[[email protected] ~]# service mogstored start
Starting mogstored [ OK ]
[[email protected] ~]# ss -tanlp
[[email protected] init.d]# service mogilefsd start
Starting mogilefsd [ OK ]
[[email protected] ~]# mogadm check
Checking trackers...
127.0.0.1:7001 ... OK
Checking hosts...
No devices found on tracker(s).
[[email protected] ~]# clear
追踪节点所能适应的管理命令
[[email protected] ~]# mogadm --tracker=172.16.17.141:7001
mogadm check Check the state of the MogileFS world.
mogadm stats Show MogileFS system statistics. mogadm host ...
host add ... Add a host to MogileFS.
host delete ... Delete a host.
host list List all hosts.
host mark ... Change the status of a host. (equivalent to ‘modify --status‘)
host modify ... Modify a host‘s properties.
mogadm device ...
device add ... Add a device to a host.
device list ... List all devices, for each host.
device mark ... Mark a device as {alive,dead,down,drain,readonly}
device modify ... Modify a device‘s properties.
device summary ... List the summary of devices, for each host.
mogadm domain ...
domain add ... Add a domain (namespace)
domain delete ... Delete a domain.
domain list List all hosts.
mogadm class ...
class add ... Add a file class to a domain.
class delete ... Delete a file class from a domain.
class list List all classes, for each domain.
class modify ... Modify properties of a file class.
mogadm slave ...
slave add ... Add a slave node for store usage
slave delete ... Delete a slave node for store usage
slave list List current store slave nodes.
slave modify ... Modify a slave node for store usage
mogadm fsck ...
fsck clearlog Clear the fsck log
fsck printlog Display the fsck log
fsck reset ... Reset fsck position back to the beginning
fsck start Start (or resume) background fsck
fsck status Show fsck status
fsck stop Stop (pause) background fsck
fsck taillog Tail the fsck log
mogadm rebalance ...
rebalance policy ... Add or adjust the current policy
rebalance reset Reset an existing policy
rebalance settings Display rebalance settings
rebalance start Start a rebalance job
rebalance status Show status of current rebalance job
rebalance stop Stop a rebalance job
rebalance test Show what devices the current policy would match
mogadm settings ...
settings list List all server settings
settings set ... Set server setting ‘key‘ to ‘value‘.
11、首先需要把节点设备添加至管理设备中。使其能够正常管理
[[email protected] ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.140 --ip=172.16.17.140 --status=alive
[[email protected] ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.141 --ip=172.16.17.141 --status=alive
[[email protected] ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.144 --ip=172.16.17.144 --status=alive
[[email protected] ~]#
添加完成之后，去确认设备添加情况
[[email protected] ~]# mogadm host list
172.16.17.140 [1]: alive
IP: 172.16.17.140:7500
172.16.17.141 [2]: alive
IP: 172.16.17.141:7500
172.16.17.144 [3]: alive
IP: 172.16.17.144:7500
12、后端存储的节点添加完成之后，接着去添加设备文件。
[[email protected] ~]# mogadm device add 172.16.17.140 1
[[email protected] ~]# mogadm device add 172.16.17.141 2
[[email protected] ~]# mogadm device add 172.16.17.144 3
[[email protected] ~]# mogadm device list
172.16.17.140 [1]: alive
used(G) free(G) total(G) weight(%)
dev1: alive 0.280 18.280 18.561 100
172.16.17.141 [2]: alive
used(G) free(G) total(G) weight(%)
dev2: alive 0.280 18.280 18.561 100
172.16.17.144 [3]: alive
used(G) free(G) total(G) weight(%)
dev3: alive 1.671 16.890 18.561 100
再次检查mogadm的状态信息
[[email protected] ~]# mogadm check
Checking trackers...
127.0.0.1:7001 ... OK
Checking hosts...
[ 1] 172.16.17.140 ... OK
[ 2] 172.16.17.141 ... OK
[ 3] 172.16.17.144 ... OK
Checking devices...
host device size(G) used(G) free(G) use% ob state I/O%
---- ------------ ---------- ---------- ---------- ------ ---------- -----
[ 1] dev1 18.561 0.281 18.280 1.51% writeable 0.0
[ 2] dev2 18.561 0.281 18.280 1.51% writeable 0.0
[ 3] dev3 18.561 1.671 16.890 9.00% writeable 1.7
---- ------------ ---------- ---------- ---------- ------
total: 55.683 2.233 53.451 4.01%
[[email protected] ~]#
13、添加domain域空间存储目录
[[email protected] ~]# mogadm domain add files
[[email protected] ~]# mogadm domain add images
[[email protected] ~]# mogadm domain list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
files default 2 MultipleHosts() NONE
images default 2 MultipleHosts() NONE
在创建domain的时候，根绝自己的需要进行创建。
14、创建好域名空间之后，如果需要有创建class类别。也能创建，这样能够分类别存储文件
基本使用方法
[[email protected] ~]# mogadm class help
mogadm class add <domain> <class> [opts] Add a file class to a domain.
<class> Name of class to add.
<domain> Domain to add class to.
--hashtype=s Hash algorithm string (‘MD5‘, ‘NONE‘).
--mindevcount=i Minimum number of replicas.
--replpolicy=s Replication policy string.
添加几个基本的类试试
[[email protected] ~]# mogadm class add images jpg
[[email protected] ~]# mogadm class add images png
[[email protected] ~]# mogadm class add images gif
[[email protected] ~]# mogadm class add files text --mindevcount=1
[[email protected] ~]# mogadm class add files html --replpolicy="MultipleHosts(3)"
[[email protected] ~]# mogadm class list
domain class mindevcount replpolicy hashtype
files default 2 MultipleHosts() NONE
files html 2 MultipleHosts(3) NONE
files plaintext 1 MultipleHosts() NONE
images default 2 MultipleHosts() NONE
images gif 2 MultipleHosts() NONE
images jpeg 2 MultipleHosts() NONE
images png 2 MultipleHosts() NONE
15、如果基本的属性定义完成之后，接着可以尝试去上传资源试试
[[email protected] ~]# mogupload
Usage: /usr/bin/mogupload --trackers=host --domain=foo --key=‘/hello.jpg‘ --file=‘./hello.jpg‘
[[email protected] ~]# mogupload --trackers=172.16.17.140 --domain=files --key=‘/fstab.txt‘ --file=‘/etc/fstab‘ --class=plaintext
上传完成之后，确认上传情况。能够，会生成http连接。但是连接会比较特殊。
[[email protected] ~]# mogfileinfo --trackers=172.16.17.140 --domain=files --key=‘/fstab.txt‘
- file: /fstab.txt
class: plaintext
devcount: 1
domain: files
fid: 2
key: /fstab.txt
length: 921
- http://172.16.17.141:7500/dev2/0/000/000/0000000002.fid
[[email protected] ~]#
[[email protected] mogilefs]# mogupload --trackers=172.16.17.140 --domain=images --key=‘/1.jpg‘ --file=‘1.jpg‘ --class=jpeg
[[email protected] mogilefs]# mogfileinfo --trackers=172.16.17.140 --domain=images --key=‘/1.jpg‘
- file: /1.jpg
class: jpeg
devcount: 2
domain: images
fid: 3
key: /1.jpg
length: 170625
- http://172.16.17.140:7500/dev1/0/000/000/0000000003.fid
- http://172.16.17.141:7500/dev2/0/000/000/0000000003.fid
[[email protected] mogilefs]# mogstats --db_dsn="DBI:mysql:mogilefs:host=172.16.17.144" --db_user="moguser" --db_pass="mogpass" --verbose --stats="all"
Statistics for devices...
device host files status
---------- ---------------- ------------ ----------
dev1 172.16.17.140 1 alive
dev2 172.16.17.141 2 alive
---------- ---------------- ------------ ----------
Statistics for file ids...
Max file id: 3
Statistics for files...
domain class files size (m) fullsize (m)
-------------------- ----------- ---------- ----------- -------------
files plaintext 1 0 0
images jpeg 1 0 0
-------------------- ----------- ---------- ----------- -------------
Statistics for replication...
domain class devcount files
-------------------- ----------- ---------- ----------
files plaintext 1 1
images jpeg 2 1
-------------------- ----------- ---------- ----------
如果想要通过更加便捷的方式进行上传数据需要，程序员自己写API接口
前段基于nginx进行反代mogilefs文件系统
1、需要的安装包组
[[email protected] ~]# ll
-rw-r--r-- 1 root root 832104 Dec 4 15:32 nginx-1.8.0.tar.gz
-rw-r--r-- 1 root root 11208 Dec 4 18:54 nginx_mogilefs_module-1.0.4.tar.gz
需要注意的一点是，对于nginx而言，如果想实现对后端的mogilefs服务器的代理需要nginx_mogilefs_module模块。否则不能实现此功能；同时，还需要注意的一点是模板的版本需要严格要求，模块的版本信息要和nginx版本信息匹配否则会报内核溢出之类的错误
2015/12/04 18:36:42 [alert] 1880#0: worker process 1890 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1892 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1894 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1896 exited on signal 11 (core dumped)
2、在安装之前需要解决需要的依赖环境
[[email protected] ~]# yum groupinstall "Development Tools" "Server Platform Deveopment"
[[email protected] ~]# yum install openssl-devel pcre-devel
还需要创建一个nginx用户
[[email protected] ~]# groupadd -r nginx
[[email protected] ~]# useradd -r -g nginx nginx
3、接着解压安装包进行安装
[[email protected] ~]# ./configure \
--prefix=/usr \
--sbin-path=/usr/sbin/nginx \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/tmp/nginx/client/ \
--http-proxy-temp-path=/var/tmp/nginx/proxy/ \
--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
--http-scgi-temp-path=/var/tmp/nginx/scgi \
--with-pcre \
--with-debug \
--add-module=
这里需要特别注意的一项是，在编译安装nginx的时候，需要为期指定mogilefs的模块相对于的文件--add-module=../nginx_mogilefs_module-1.0.4
编译完成之后进行
Make && make install
4、为nginx提供一个启动脚本如下
#!/bin/sh
# nginx - this script starts and stops the nginx daemon
# chkconfig: - 85 15
# description: Nginx is an HTTP(S) server, HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /etc/nginx/nginx.conf
# config: /etc/sysconfig/nginx
# pidfile: /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
nginx="/usr/sbin/nginx"
prog=$(basename $nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() {
# make required directories
user=`nginx -V 2>&1 | grep "configure arguments:" | sed ‘s/[^*]*--user=\([^ ]*\).*/\1/g‘ -`
options=`$nginx -V 2>&1 | grep ‘configure arguments:‘`
for opt in $options; do
if [ `echo $opt | grep ‘.*-temp-path‘` ]; then
value=`echo $opt | cut -d "=" -f 2`
if [ ! -d "$value" ]; then
# echo "creating" $value
mkdir -p $value && chown -R $user $value
fi
fi
done
}
start() {
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
make_dirs
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
configtest || return $?
stop
sleep 1
start
}
reload() {
configtest || return $?
echo -n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac
而后为此脚本赋予执行权限：
[[email protected] ~]# chmod +x /etc/rc.d/init.d/nginx
添加至服务管理列表，并让其开机自动启动：
[[email protected] ~]# chkconfig --add nginx
[[email protected] ~]# chkconfig nginx on
而后就可以启动服务并测试了：
[[email protected] ~]# service nginx start
5、接着去修改nginx的主配置文件
upstream trackers {
server 172.16.17.140:7001 weight=1;
server 172.16.17.141:7001 weight=1;
server 172.16.17.144:7001 weight=1;
}
Server {
location /images {
mogilefs_tracker trackers;
mogilefs_domain images;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
location /files {
mogilefs_tracker trackers;
mogilefs_domain files;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
}
6、配置结束之后，重启服务器可以去测试反代效果如何？
查看反代日志
[[email protected] ~]# tail -2 /var/log/nginx/access.log
172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /images/love.jpg HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"
172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /images/love.jpg HTTP/1.1" 200 190166 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"
[[email protected] ~]#
7、接着尝试down机一台存储服务器测试效果如何
[[email protected] ~]# service mogstored stop
Stop mogstored [ OK ]
对于前段访问后端存储服务器不会出现任何问题
注意点：对于这个版本的mogilefs分布式文件系统，在centos7上，只能存储一份，不能有其他多余的副本