FastDFs 开源的轻量级分布式文件系统部署

1.技术背景介绍;

FastDFS是一个开源的轻量级分布式文件系统,它对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务,如相册网站、视频网站等等。
FastDFS为互联网量身定制,充分考虑了冗余备份、负载均衡、线性扩容等机制,并注重高可用、高性能等指标,使用FastDFS很容易搭建一套高性能的文件服务器集群提供文件上传、下载等服务。

2.基础环境介绍;

主机名 IP地址规划 相关软件源码包汇总 服务器角色

Nginx_fastdfs1
192.168.1.190 Nginx.1.8.0.tar.gz
fastdfs .v5.05.tar.gz
libfastcommon-master.zip
fastdfs-nginx-module_v1.16.tar.gz Storage/tracker

Nginx_fastdfs2
192.168.1.200 Storage/tracker

3.软件功能介绍;
(1) libfastcommon 源码包介绍;

libfastcommon是从开源软件FastDFS和FastDHT中提取出来的公共C函数库。
它提供了字符串处理、链表、哈希表、日志记录、配置文件读取、socket操作、base64编码、url编码、md5等常用函数库。还包括http的一个简单的client,可以获取url内容(目前仅支持GET方式)。
这些代码已经在FastDFS和FastDHT中使用,比较简洁、稳定。FastDFS已经在Linux、FreeBSD、AIX下进行了编译并通过了测试.

(2)fastdfs-nginx-module_v1 介绍;
我们在使用FastDFS部署一个分布式文件系统的时候,通过FastDFS的客户端API来进行文件的上传、下载、删除等操作。同时通过FastDFS 的HTTP服务器来提供HTTP服务。但是FastDFS的HTTP服务较为简单,无法提供负载均衡等高性能的服务,所以FastDFS的开发者——淘宝 的架构师余庆同学,为我们提供了Nginx上使用的FastDFS模块。其使用非常简单。

(3) fastdfs 介绍;
FastDFS是一个开源的轻量级分布式文件系统,它对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务,如相册网站、视频网站等等。
stDFS服务端有两个角色:跟踪器(tracker)和存储节点(storage)。跟踪器主要做调度工作,在访问上起负载均衡的作用。
存储节点存储文件,完成文件管理的所有功能:存储、同步和提供存取接 口,FastDFS同时对文件的metadata进行管理。所谓文件的meta data就是文件的相关属性,以键值对(key valuepair)方式表示,如:width=1024,其中的key为width,value为1024。文件metadata是文件属性列表,可以包含多个键值对。
跟踪器和存储节点都可以由一台或多台服务器构成。跟踪器和存储节点中的服务器均可以随时增加或下线而不会影响线上服务。其中跟踪器中的所有服务器都是对等的,可以根据服务器的压力情况随时增加或减少。
为了支持大容量,存储节点(服务器)采用了分卷(或分组)的组织方式。存储系统由一个或多个卷组成,卷与卷之间的文件是相互独立的,所有卷的文件容量累加就是整个存储系统中的文件容量。一个卷可以由一台或多台存储服务器组成,一个卷下的存储服务器中的文件都是相同的,卷中的多台存储服务器起到了冗余备份和负载均衡的作用。
在卷中增加服务器时,同步已有的文件由系统自动完成,同步完成后,系统自动将新增服务器切换到线上提供服务。
当存储空间不足或即将耗尽时,可以动态添加卷。只需要增加一台或多台服务器,并将它们配置为一个新的卷,这样就扩大了存储系统的容量。
FastDFS中的文件标识分为两个部分:卷名和文件名,二者缺一不可。

(4).上传交互过程;

  1. client询问tracker上传到的storage,不需要附加参数;
  2. tracker返回一台可用的storage;
  3. client直接和storage通讯完成文件上传。
    FastDFS file download

(5).下载交互过程

  1. client询问tracker下载文件的storage,参数为文件标识(卷名和文件名);
  2. tracker返回一台可用的storage;
  3. client直接和storage通讯完成文件下载。
    需要说明的是,client为使用FastDFS服务的调用方,client也应该是一台服务器,它对tracker和storage的调用均为服务器间的调用

3.部署环境准备;

注意:(两台机器同样执行以下操作)

(1).安装环境依赖包

yum install -y zlib zlib-devel pcre pcre-devel gcc gcc-c++ openssl openssl-devel libevent libevent-devel perl unzip

(2).创建fastdfs 运行用户;

useradd -s /sbin/nologin fastdfs

(3).创建数据存储目录;

mkdir -p /export/fastdfs/{storage,tracker}
[[email protected]_Fastdfs1 ~]# ll /export/fastdfs/
total 8
drwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 22:09 storage
drwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 21:12 tracker

[[email protected]_Fastdfs2 ~]# ll /export/fastdfs/
total 8
drwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 21:59 storage
drwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 21:03 tracker

4.Fastdfs部署详情;

注意:(2台服务器需要同样执行如下操作)

(1).部署libfastcommon;
mkdir /usr/local/fastdfs_packages
cd /usr/local/fastdfs_packages
wget https://github.com/happyfish100/libfastcommon/archive/master.zip
unzip master.zip
cd libfastcommon-master/
./make.sh
./make.sh install

(2)部署fastdfs
cd /usr/local/fastdfs_packages
wget http://sourceforge.net/projects/fastdfs/files/FastDFS%20Server%20Source
%20Code/FastDFS%20Server%20with%20PHP%20Extension%20Source%20Code%20V5.05/
FastDFS_v5.05.tar.gz
tar zxf FastDFS_v5.05.tar.gz && cd FastDFS
./make.sh && ./make.sh install
\cp pa conf/.conf /etc/fdfs/
cd /etc/fdfs/
mv
.sample /opt/ #备份模板文件;
chown -R fastdfs: /export/fastdfs

注意:配置tracker 和 storage
Nginx_fastdfs1 和Nginx_fastdfs2分别配置;

(3) Nginx_fastdfs1机器(配置tracker 和 storage 配置文件;)

[[email protected]_Fastdfs1 fdfs]# tree /etc/fdfs/
/etc/fdfs/
├── client.conf
├── mod_fastdfs.conf
├── storage.conf
└── tracker.conf
0 directories, 4 files

storage 配置

[[email protected]_Fastdfs1 /]# cat /etc/fdfs/storage.conf
group_name=group1
base_path=/export/fastdfs/storage
store_path0=/export/fastdfs/storage
tracker_server=192.168.1.190:22122
tracker_server=192.168.1.200:22122
http.server_port=80
disabled=false
bind_addr=192.168.1.190
client_bind=true
port=23000
connect_timeout=30
network_timeout=60
heart_beat_interval=30
stat_report_interval=60
max_connections=256
buff_size = 256KB
work_threads=2
disk_rw_separated = true
disk_rw_direct = false
disk_reader_threads = 1
disk_writer_threads = 1
sync_wait_msec=50
sync_interval=0
sync_start_time=00:00
sync_end_time=23:59
write_mark_file_freq=500
store_path_count=1
subdir_count_per_path=256
log_level=info
run_by_group=
run_by_user=
allow_hosts=*
file_distribute_path_mode=0
file_distribute_rotate_count=100
fsync_after_written_bytes=0
sync_log_buff_interval=10
sync_binlog_buff_interval=10
sync_stat_file_interval=300
thread_stack_size=512KB
upload_priority=10
if_alias_prefix=
check_file_duplicate=0
key_namespace=FastDFS
keep_alive=0
http.disabled=false
httphttp.domain_name=
http.trunk_size=256KB
http.need_find_content_type=true

tracker.conf 配置;

[[email protected]_Fastdfs1 fdfs]# cat /etc/fdfs/tracker.conf
base_path=/export/fastdfs/tracker
disabled=false
bind_addr=192.168.1.190
port=22122
connect_timeout=30
network_timeout=60
max_connections=256
work_threads=2
store_lookup=2
store_group=group2
store_server=0
store_path=0
download_server=0
reserved_storage_space = 4GB
log_level=info
run_by_group=
run_by_user=
allow_hosts=*
sync_log_buff_interval = 10
check_active_interval = 120
thread_stack_size = 64KB
storage_ip_changed_auto_adjust = true
storage_sync_file_max_delay = 86400
storage_sync_file_max_time = 300
use_trunk_file = false
slot_min_size = 256
slot_max_size = 16MB
trunk_file_size = 64MB
http.disabled=false
http.server_port=8080
http.check_alive_interval=30
http.check_alive_type=tcp
http.check_alive_uri=/status.html
http.need_find_content_type=true

mod_fastdfs.conf 配置

[[email protected]_Fastdfs1 /]# cat /etc/fdfs/mod_fastdfs.conf
group_name=group1
base_path=/export/fastdfs/storage
store_path0=/export/fastdfs/storage
tracker_server=192.168.1.190:22122
tracker_server=192.168.1.200:22122
url_have_group_name = true

client.conf 配置

[[email protected]_Fastdfs1 fdfs]# cat client.conf
tracker_server=192.168.1.190:22122
tracker_server=192.168.1.200:22122

(3) Nginx_fastdfs2机器(配置tracker 和 storage 配置文件;)

[[email protected]_Fastdfs2 ~]# tree /etc/fdfs/
/etc/fdfs/
├── client.conf
├── mod_fastdfs.conf
├── storage.conf
└── tracker.conf
0 directories, 4 files

storage.conf 配置;

[[email protected]_Fastdfs2 ~]# cat /etc/fdfs/storage.conf
base_path=/export/fastdfs/storage
store_path0=/export/fastdfs/storage
tracker_server=192.168.1.200:22122
tracker_server=192.168.1.190:22122
http.server_port=80
disabled=false
group_name=group1
bind_addr=192.168.1.200
client_bind=true
port=23000
connect_timeout=30
network_timeout=60
heart_beat_interval=30
stat_report_interval=60
max_connections=256
buff_size = 256KB
work_threads=2
disk_rw_separated = true
disk_rw_direct = false
disk_reader_threads = 1
disk_writer_threads = 1
sync_wait_msec=50
sync_interval=0
sync_start_time=00:00
sync_end_time=23:59
write_mark_file_freq=500
store_path_count=1
subdir_count_per_path=256
log_level=info
run_by_group=
run_by_user=
allow_hosts=*
file_distribute_path_mode=0
file_distribute_rotate_count=100
fsync_after_written_bytes=0
sync_log_buff_interval=10
sync_binlog_buff_interval=10
sync_stat_file_interval=300
thread_stack_size=512KB
upload_priority=10
if_alias_prefix=
check_file_duplicate=0
key_namespace=FastDFS
keep_alive=0
http.disabled=false
httphttp.domain_name=
http.trunk_size=256KB
http.need_find_content_type=true

tracker.conf 配置

[[email protected]_Fastdfs2 ~]# cat /etc/fdfs/tracker.conf
base_path=/export/fastdfs/tracker
disabled=false
bind_addr=192.168.1.200
port=22122
connect_timeout=30
network_timeout=60
max_connections=256
work_threads=2
store_lookup=2
store_group=group2
store_server=0
store_path=0
download_server=0
reserved_storage_space = 4GB
log_level=info
run_by_group=
run_by_user=
allow_hosts=*
sync_log_buff_interval = 10
check_active_interval = 120
thread_stack_size = 64KB
storage_ip_changed_auto_adjust = true
storage_sync_file_max_delay = 86400
storage_sync_file_max_time = 300
use_trunk_file = false
slot_min_size = 256
slot_max_size = 16MB
trunk_file_size = 64MB
http.disabled=false
http.server_port=8080
http.check_alive_interval=30
http.check_alive_type=tcp
http.check_alive_uri=/status.html
http.need_find_content_type=true

mod_fastdfs.conf 配置

[[email protected]_Fastdfs2 ~]# cat /etc/fdfs/mod_fastdfs.conf
group_name=group1
base_path=/export/fastdfs/storage
store_path0=/export/fastdfs/storage
tracker_server=192.168.1.200:22122
tracker_server=192.168.1.190:22122
url_have_group_name = true
client.conf 配置

[[email protected]_Fastdfs2 ~]# cat /etc/fdfs/client.conf
tracker_server=192.168.1.200:22122
tracker_server=192.168.1.190:22122

5.Fastdfs结合nginx;

(1).软件部署以及配置调整;
cd /usr/local/fastdfs_packages
wget http://nginx.org/download/nginx-1.8.0.tar.gz
http://sourceforge.net/projects/fastdfs/files/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz/download
tar zxf fastdfs-nginx-module_v1.16.tar.gz

注意:修改模块中对应的路径,否则模块不能正确安装加载;
cd fastdfs-nginx-module/src
vi conf #更改如下,去掉local,并指定lib64(64系统)
CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/
CORE_LIBS="$CORE_LIBS -L/usr/lib64 -lfastcommon -lfdfsclient"

(2).nginx 添加fastdfs-nginx-module模块;
ulimit -SHn 102400
useradd -s /sbin/nologin nginx
tar zxf nginx-1.8.0.tar.gz
cd nginx-1.8.0
./configure --user=nginx --group=nginx --add-module=../fastdfs-nginx-module/src/ --prefix=/usr/local/nginx --with-http_stub_status_module--with-http_ssl_module --with-http_realip_module
make && make install

(3).nginx 配置fastdfs;

nginx_Fastdfs1 配置;

[[email protected]_Fastdfs1 nginx-1.8.0]# cat /usr/local/nginx/conf/nginx.conf
user nginx;
worker_processes 1;
events {
worker_connections 1024;
}
http{
server {
listen 80;
server_name 192.168.1.190;
root /webdata/fastdfs.com;
index index.html index.htm;
location / {
if (!-e $request_filename) {
rewrite ^(.)$ /index.html last;
}
location /group1/M00 {
root /export/fastdfs/storage/data/;
ngx_fastdfs_module;
}
location ~ .
.(gif|jpg|jpeg|png|bmp|swf|js)$ {

fastcgi_cache cache_one; #nginx.conf 开启cache才行,否则然启动报错

   fastcgi_cache_valid 200 10m;
   fastcgi_cache_valid 304 3m;
   fastcgi_cache_valid 301 302 1h;
   fastcgi_cache_valid any 1m;
   fastcgi_cache_min_uses 1;
   fastcgi_cache_use_stale error timeout invalid_header http_500;
   fastcgi_cache_key $host$request_uri;
   access_log off;
   }
   }
access_log off;

}
}

nginx_Fastdfs2 配置;

[[email protected]_Fastdfs2]# cat /usr/local/nginx/conf/nginx.conf
events {
worker_connections 1024;
}
http{
server {
listen 80;
server_name 192.168.1.200;
root /webdata/fastdfs.com;
index index.html index.htm;
location / {
if (!-e $request_filename) {
rewrite ^(.)$ /index.html last;
}
location /group1/M00 {
root /export/fastdfs/storage/data/;
ngx_fastdfs_module;
}
location ~ .
.(gif|jpg|jpeg|png|bmp|swf|js)$ {
#fastcgi_cache cache_one; #nginx.conf 开启cache,否则启动报错
fastcgi_cache_valid 200 10m;
fastcgi_cache_valid 304 3m;
fastcgi_cache_valid 301 302 1h;
fastcgi_cache_valid any 1m;
fastcgi_cache_min_uses 1;
fastcgi_cache_use_stale error timeout invalid_header http_500;
fastcgi_cache_key $host$request_uri;
access_log off;
}
}
access_log off;
}
}

6.fastdfs服务管理;
(1)编写启动脚本;
[[email protected]_Fastdfs1]# cat /etc/init.d/fdfs_storaged
#!/bin/bash

fdfs_storaged Starts fdfs_storaged

chkconfig: 2345 99 01

description: FastDFS storage server

BEGIN INIT INFO

Provides: $fdfs_storaged

END INIT INFO

Source function library.

. /etc/init.d/functions
PRG=/usr/local/fastdfs_packages/FastDFS/storage/fdfs_storaged
CONF=/etc/fdfs/storage.conf
if [ ! -f $PRG ]; then
echo "file $PRG does not exist!"
exit 2
fi
if [ ! -f /usr/local/fastdfs_packages/FastDFS/stop.sh ]; then
echo "file /usr/local/fastdfs_packages/FastDFS/stop.sh does not exist!"
exit 2
fi
if [ ! -f /usr/local/fastdfs_packages/FastDFS/restart.sh ]; then
echo "file /usr/local/fastdfs_packages/FastDFS/restart.sh does not exist!"
exit 2
fi
if [ ! -f $CONF ]; then
echo "file $CONF does not exist!"
exit 2
fi

CMD="$PRG $CONF"
RETVAL=0

start() {
echo -n $"Starting FastDFS storage server: "
$CMD &
RETVAL=$?
echo
return $RETVAL
}
stop() {
/usr/local/fastdfs_packages/FastDFS/stop.sh $CMD
RETVAL=$?
return $RETVAL
}
rhstatus() {
status fdfs_storaged
}
restart() {
/usr/local/fastdfs_packages/FastDFS/restart.sh $CMD &
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart|reload)
restart
;;
condrestart)
restart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit 1
esac
exit $?
[[email protected]_Fastdfs1]# cat /etc/init.d/fdfs_trackerd
#!/bin/bash

fdfs_trackerd Starts fdfs_trackerd

chkconfig: 2345 99 01

description: FastDFS tracker server

BEGIN INIT INFO

Provides: $fdfs_trackerd

END INIT INFO

Source function library.

. /etc/init.d/functions
PRG=/usr/local/fastdfs_packages/FastDFS/tracker/fdfs_trackerd
CONF=/etc/fdfs/tracker.conf

if [ ! -f $PRG ]; then
echo "file $PRG does not exist!"
exit 2
fi

if [ ! -f /usr/local/fastdfs_packages/FastDFS/stop.sh ]; then
echo "file /usr/local/bin/stop.sh does not exist!"
exit 2
fi

if [ ! -f /usr/local/fastdfs_packages/FastDFS/restart.sh ]; then
echo "file /usr/local/bin/restart.sh does not exist!"
exit 2
fi

if [ ! -f $CONF ]; then
echo "file $CONF does not exist!"
exit 2
fi

CMD="$PRG $CONF"
RETVAL=0

start() {
echo -n $"Starting FastDFS tracker server: "
$CMD &
RETVAL=$?
echo
return $RETVAL
}
stop() {
/usr/local/fastdfs_packages/FastDFS/stop.sh $CMD
RETVAL=$?
return $RETVAL
}
rhstatus() {
status fdfs_trackerd
}
restart() {
/usr/local/fastdfs_packages/FastDFS/restart.sh $CMD &
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart|reload)
restart
;;
condrestart)
restart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit 1
esac
exit $?

(2).fastdfs相关服务启动;
/etc/init.d/fdfs_trackerd start
/etc/init.d/fdfs_storaged start
/usr/local/nginx/sbin/nginx

(3)查看进程运行情况;
[[email protected]_Fastdfs1 /]# netstat -anplt |grep -E "nginx|fdfs"
tcp 0 0 192.168.1.200:23000 0.0.0.0: LISTEN 1945/fdfs_storaged
tcp 0 0 192.168.1.200:22122 0.0.0.0:
LISTEN 1937/fdfs_trackerd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1798/nginx
tcp 0 0 192.168.1.200:35036 192.168.1.200:22122 ESTABLISHED 1945/fdfs_storaged
tcp 0 0 192.168.1.200:22122 192.168.1.200:35036 ESTABLISHED 1937/fdfs_trackerd
tcp 0 0 192.168.1.200:43282 192.168.1.190:22122 ESTABLISHED 1945/fdfs_storaged
tcp 0 0 192.168.1.200:51407 192.168.1.190:23000 ESTABLISHED 1945/fdfs_storaged
tcp 0 0 192.168.1.200:22122 192.168.1.190:44927 ESTABLISHED 1937/fdfs_trackerd
tcp 0 0 192.168.1.200:22122 192.168.1.190:40724 ESTABLISHED 1937/fdfs_trackerd
tcp 0 0 192.168.1.200:23000 192.168.1.190:54332 ESTABLISHED 1945/fdfs_storaged

[[email protected]_Fastdfs2 ~]# netstat -anplt |grep -E "nginx|fdfs"
tcp 0 0 192.168.1.190:22122 0.0.0.0: LISTEN 1603/fdfs_trackerd
tcp 0 0 0.0.0.0:80 0.0.0.0:
LISTEN 1573/nginx
tcp 0 0 192.168.1.190:23000 0.0.0.0:* LISTEN 1612/fdfs_storaged
tcp 0 0 192.168.1.190:22122 192.168.1.190:54669 ESTABLISHED 1603/fdfs_trackerd
tcp 0 0 192.168.1.190:23000 192.168.1.200:51407 ESTABLISHED 1612/fdfs_storaged
tcp 0 0 192.168.1.190:22122 192.168.1.200:43282 ESTABLISHED 1603/fdfs_trackerd
tcp 0 0 192.168.1.190:40724 192.168.1.200:22122 ESTABLISHED 1603/fdfs_trackerd
tcp 0 0 192.168.1.190:44927 192.168.1.200:22122 ESTABLISHED 1612/fdfs_storaged
tcp 0 0 192.168.1.190:54332 192.168.1.200:23000 ESTABLISHED 1612/fdfs_storaged
tcp 0 0 192.168.1.190:54669 192.168.1.190:22122 ESTABLISHED 1612/fdfs_storaged

7.fastdfs 功能测试;
nginx_Fastdfs1 机器测试;
[[email protected]_Fastdfs1]# fdfs_test /etc/fdfs/client.conf upload test.jsp
This is FastDFS client test program v5.05
Copyright (C) 2008, Happy Fish / YuQing
FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.csource.org/
for more detail.
[2016-06-07 03:04:04] DEBUG - base_path=/tmp, connect_timeout=30, network_timeout=30, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
tracker_query_storage_store_list_without_group:
server 1. group_name=, ip_addr=192.168.1.190, port=23000
server 2. group_name=, ip_addr=192.168.1.200, port=23000
group_name=group1, ip_addr=192.168.1.190, port=23000
storage_upload_by_filename
group_name=group1, remote_filename=M00/00/00/wKgBvldVySWAObaSAAAAAAAAAAA284.jsp
source ip address: 192.168.1.190
file timestamp=2016-06-07 03:04:05
file size=0
file crc32=0
example file url: http://192.168.1.190/group1/M00/00/00/wKgBvldVySWAObaSAAAAAAAAAAA284.jsp
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/wKgBvldVySWAObaSAAAAAAAAAAA284_big.jsp
source ip address: 192.168.1.190
file timestamp=2016-06-07 03:04:05
file size=0
file crc32=0
example file url: http://192.168.1.190/group1/M00/00/00/wKgBvldVySWAObaSAAAAAAAAAAA284_big.jsp

nginx_Fastdfs2 机器测试;
[[email protected]_Fastdfs2 ~]# fdfs_test /etc/fdfs/client.conf upload test.jsp
This is FastDFS client test program v5.05
Copyright (C) 2008, Happy Fish / YuQing
FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.csource.org/
for more detail.
[2016-06-07 03:06:33] DEBUG - base_path=/tmp, connect_timeout=30, network_timeout=30, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
tracker_query_storage_store_list_without_group:
server 1. group_name=, ip_addr=192.168.1.190, port=23000
server 2. group_name=, ip_addr=192.168.1.200, port=23000
group_name=group1, ip_addr=192.168.1.200, port=23000
storage_upload_by_filename
group_name=group1, remote_filename=M00/00/00/wKgByFdVybiAcFDfAAAAAAAAAAA902.jsp
source ip address: 192.168.1.200
file timestamp=2016-06-07 03:06:32
file size=0
file crc32=0
example file url: http://192.168.1.200/group1/M00/00/00/wKgByFdVybiAcFDfAAAAAAAAAAA902.jsp
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/wKgByFdVybiAcFDfAAAAAAAAAAA902_big.jsp
source ip address: 192.168.1.200
file timestamp=2016-06-07 03:06:32
file size=0
file crc32=0
example file url: http://192.168.1.200/group1/M00/00/00/wKgByFdVybiAcFDfAAAAAAAAAAA902_big.jsp

原文地址:http://blog.51cto.com/breaklinux/2150199

时间: 2024-08-08 12:25:50

FastDFs 开源的轻量级分布式文件系统部署的相关文章

FastDFS轻量级分布式文件系统部署

FastDFS介绍 FastDFS 是一个由 C 语言实现的开源轻量级分布式文件系统,作者余庆,支持 Linux.FreeBSD.AID 等 Unix 系统,解决了大数据存储和读写负载均衡等问题,适合存储 4KB~500MB 之间的小文件,如图片网站.短视频网站.文档.app 下载站等,UC.京东.支付宝.迅雷.酷狗等都有使用. FastDFS上传下载的流程 4.4 利用Java客户端调用FastDFS 服务器安装完毕后,咱们通过Java调用fastdfs 加载Maven依赖 fastdfs 没

轻量级分布式文件系统FastDFS使用安装说明手册(新手入门级)

轻量级分布式文件系统FastDFS使用安装说明手册(新手入门级) 实验室所在的课题组以研究云计算为主,但所有的研究都是在基于理论的凭空想像,缺少分布式环境的平台的实践,云计算神马的都是浮云了.因此,我想借助Hadoop把实验室的服务器搭建出一个基础的分布式实验环境,方便于以后进行实验测试神马的.无意中,看到了一款开源的轻量级分布式文件系统FastDFS,它是用纯C语言实现,详细信息可参见博客:分布式文件系统FastDFS架构剖析.通过收集资料,自己动手做了一些安装和测试使用,现在对前人的一些经验

DFS分布式文件系统--部署篇

DFS分布式文件系统--部署篇 续DFS分布式文件系统--基础篇 三.DFS部署实例 在VMwareWorkstation 12.0虚拟环境下,建立五台Windows Server 2008 R2虚拟机. NameSrv01:192.168.0.180/24,DC,DNS: NameSrv02:192.168.0.181/24,辅助DC,DNS:域名FromHeart.Com. 这两台DC为命名空间服务器. ShareSrv01:192.168.0.182/24:ShareSrv02:192.1

GlusterFS分布式文件系统部署

理论部分 一:GlusterFS概述1.1:Glusterfs简介?? GlusterFS是Scale-Out存储解决方案Gluster的核心,它是一个开源的分布式文件系统,具有强大的横向扩展能力,通过扩展能够支持数PB存储容量和处理数千客户端.GlusterFS借助TCP/IP或InfiniBandRDMA网络将物理分布的存储资源聚集在一起,使用单一全局命名空间来管理数据.1.2:Glusterfs特点1:扩展性和高性能?? ??GlusterFS利用双重特性来提供几TB至数PB的高扩展存储解

Nginx+MogileFS分布式文件系统部署实验

实验要求 1.  部署mogilefs分布式文件系统 2.  nginx代理 实验环境 1.拓扑 2.网络地址规划 node1:172.16.76.10 nginx (centos6.8) node2:172.16.76.20 Trackers .storage Node   (centos7.2) node3:172.16.76.30 storage Node                   (centos7.2) node4:172.16.76.40 storage Node  (Mar

分布式文件系统fastDFS部署

一.部署环境:(CentOS 6.9) tracker server:10.10.1.105 storage server:10.10.1.106  10.10.1.107 client:10.10.1.105 二.FastDFS: 开源的轻量级分布式文件系统: 三个角色: tracker server :跟踪服务器,调度,在内存中记录所有存储组和存储服务器的状态信息: tracker.conf storage server :存储服务器,文件(data)和文件属性(metadata),一组服务

【架构设计】分布式文件系统 FastDFS的原理和安装使用

本文地址 分享提纲: 1.概述 2. 原理 3. 安装 4. 使用 5. 参考文档 1. 概述 1.1)[常见文件系统] Google了一下,流行的开源分布式文件系统有很多,介绍如下: -- mogileFS:Key-Value型元文件系统,不支持FUSE,应用程序访问它时需要API,主要用在web领域处理海量小图片,效率相比mooseFS高很多. -- fastDFS:国人 余庆老师(GitHub)在mogileFS的基础上进行改进的key-value型文件系统,同样不支持FUSE,提供比mo

一些流行的分布式文件系统(Hadoop、Lustre、MogileFS、FreeNAS、FastDFS、GoogleFS)

1.故事的起源 时间过的很快,距离上一次项目的大规模升级和调整虽然已经过去了几年,但是总感觉就发生在昨天,但是系统已经再次需要进行扩展.数据规模的扩大化,运行条件的复杂化,运维保障体系的升级化,原来有不少内容都需要调整,使用一种合适的分布式文件系统已经进入我们的视野. 在网上找了一些资料,也请了一些业内的专家座谈和调研,我们最终选择的是分布式文件系统是Hadoop,顺便也把一些其他的调研的系统也记录下,将来要使用也可以作为一个参考. 当前比较流行的分布式文件系统包括:Hadoop.Lustre.

记录:CentOS 7 安装配置分布式文件系统 FastDFS 5.1.1

CentOS 7 安装配置分布式文件系统 FastDFS 5.1.1 软件下载:http://download.csdn.net/download/qingchunwuxian1993/9897458 yum-y install net-tools.x86_64 前言 项目中用到文件服务器,有朋友推荐用FastDFS,所以就了解学习了一番,感觉确实颇为强大,在此再次感谢淘宝资深架构师余庆大神开源了如此优秀的轻量级分布式文件系统,本篇文章就记录一下FastDFS的最新版本5.1.1在CentOS7