FastDFS功能简介:
FastDFS是一个开源的轻量级分布式文件系统,它对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务,如相册网站、视频网站等等。
FastDFS特点:
FastDFS主要解决了大容量的文件(主要是图片、视频、音频等小文件)存储和高并发访问的问题,并在文件存取时实现了负载均衡。与其它类GFS系统相比,FastDFS最大的特点在于它是一个轻量级的系统,体现在以下几个方面。
首先,FastDFS的结构比较简单,主要由Client、Tracker server和Storage server三部分组成。Client通过Tracker server得到Storage server的信息,然后直接与Storage server通信访问文件,避免了Tracker server成为瓶颈。
第二,FastDFS不对文件进行分块存储,与支持文件分块存储的DFS相比,更加简洁高效。
第三,FastDFS中的文件ID是由Storage server生成后返回给客户端的。文件ID中包含了组号、文件相对路径和文件名等(文件ID中还包含文件大小、时间戳、源Storage server IP地址、文件内容校验码、随机数等),client可以根据文件ID直接定位到文件所在的组(但具体通过哪个Storage server下载需要询问Tracker server根据负载均衡原则指定)。因此FastDFS不需要存储文件索引信息。而其他文件系统则通常需要由NameServer存储文件索引信息,如mogileFS采用MySQL数据库来存储文件索引以及系统相关的信息,而MySQL很容易成为系统瓶颈。
FastDFS特性:
文件不分块存储,上传的文件和OS文件系统中的文件一一对应
支持相同内容的文件只保存一份,节约磁盘空间
下载文件支持HTTP协议,可以使用内置Web Server,也可以和其他Web Server配合使用
支持在线扩容
支持主从文件
存储服务器上可以保存文件属性(meta-data)V2.0网络通信采用libevent,支持大并发访问,整体性能更好
二、安装部署
1.libfastcommon支持
wget https://github.com/happyfish100/libfastcommon/archive/master.zip unzip master.zip cd libfastcommon-master/ ./make.sh ./make.sh install
2、FastDFS安装
wget http://osdn.jp/projects/sfnet_fastdfs/downloads/FastDFS%20Server%20Source%20Code/FastDFS%20Server%20with%20PHP%20Extension%20Source%20Code%20V5.05/FastDFS_v5.05.tar.gz tar -zxvf FastDFS_v5.05.tar.gz
cd FastDFS/ vim make.sh ./make.sh ./make.sh instal
创建数据文件和日志文件目录
mkdir -pv /data/fastdfs
3、启动FastDFS
启动tracker
cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart
成功提示
[2015-05-12 09:42:28] INFO - FastDFS v5.05, base_path=/home/yuqing/fastdfs....
[2015-05-12 09:42:28] INFO - local_host_ip_count: 3, 127.0.0.1 10.170.219.123 123.56.123.123
启动storage
cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf vim /etc/fdfs/storage.conf /usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
4、上传并查看
cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf vim /etc/fdfs/client.conf tracker_server=192.168.133.171:22122 => 192.168.100.151:22122 // 自己的服务器IP地址+端口 /usr/bin/fdfs_test /etc/fdfs/client.conf upload /etc/passwd
//成功提示
file size=34254
file crc32=2553486104
example file url: http://192.168.100.53:8080/group1/M00/00/00/CqrbklVRaTiAVsRkAACFzpgzGxg03754_big.h
也可以以下命令来监控服务器的状态
[[email protected] ~]# fdfs_monitor /etc/fdfs/client.conf [2016-04-18 16:06:11] DEBUG - base_path=/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=1, server_index=0 tracker server is 192.168.100.151:22122 group count: 1 Group 1: group name = group1 disk total space = 18949 MB disk free space = 16307 MB trunk free space = 0 MB storage server count = 1 active server count = 1 storage server port = 23000 storage HTTP port = 80 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 192.168.100.153 ip_addr = 192.168.100.153 (localhost) ACTIVE http domain = version = 5.08 join time = 2016-04-18 14:20:26 up time = 2016-04-18 15:39:05 total storage = 18949 MB free storage = 16307 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 80 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 0 connection.max_count = 2 total_upload_count = 8 success_upload_count = 8 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 8 success_set_meta_count = 8 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 3115422 success_upload_bytes = 3115422 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 8 success_file_open_count = 8 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 16 success_file_write_count = 16 last_heart_beat_time = 2016-04-18 16:06:31 last_source_update = 2016-04-18 15:49:48 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00
二、在storage server上安装nginx
特别说明:其实,storage中安装nginx,主要是为了为提供http的访问服务,同时解决group中storage服务器的同步延迟问题。
而tracker中安装nginx,主要是为了提供http访问的反向代理、负载均衡以及缓存服务
1、安装nginx关于fastdfs集合的扩展模块,注意这个模块nginx默认的编译模块中没有,需手动下载安装
(1)安装Nginx前期环境及扩展模块
yum -y install pcre-devel openssl openssl-devel gcc unzip fastdfs-nginx-module-master.zip tar -zxvf nginx-1.8.1.tar.gz cd nginx-1.8.1 ./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/fastdfs-nginx-module-master/src/ make && make install
(2)复制fastdfs集合的扩展模块到fastDFS的配置文件中,并修改如下:
cp /usr/local/src/fastdfs-nginx-module-master/src/mod_fastdfs.conf /etc/fdfs/ vi /etc/fdfs/mod_fastdfs.conf base_path=/tmp 改为 base_path=/data/fastdfs tracker_server=tracker:22122 改为 tracker_server=192.168.100.151:22122 store_path0=/home/yuqing/fastdfs 改为 store_path0=/data/fastdfs
(3)设置软连接
ln -s /data/fastdfs/data/ /data/fastdfs/data/M00
(4)复制fastdfs扩展模块中的http.conf mime.type两文件到fastdfs配置文件中
cp /usr/local/src/FastDFS/conf/http.conf mime.types /etc/fdfs/
(5)配置Nginx 在Server中加入
vi /usr/local/nginx/conf/nginx.conf location /group1/M00{ alias /data/fastdfs/data; #ngx_fastdfs_module
;
}
(6)启动Nginx访问
******************************************************************
注意:如果不能访问改fastdfs的模块配置文件
[[email protected] ~]# vim /etc/fdfs/mod_fastdfs.conf
url_have_group_name = false 改为 true #关于啥意思,配置文件中有解释
参考文档:http://www.linux178.com/storage/fastdfs-nginx-cache.html
***************************************************************