1、机器
192.168.1.211 Proxy Node
192.168.1.212 Storage Node
192.168.1.213 Storage Node
192.168.1.214 Storage Node
系统为SLES11sp1
2、配置软件源
因为公司服务器无法连外网，所以配置局域网源和本地源来搭建环境
上传ISO镜像文件到各台机器
SLES-11-SP4-DVD-x86_64-GM-DVD1.iso
每台机器挂载镜像，配置本地源
# mkdir /mnt/cdrom
# mount -o loop SLES-11-SP4-DVD-x86_64-GM-DVD1.iso /mnt/cdrom
# mount
/home/SLES-11-SP4-DVD-x86_64-GM-DVD1.iso on /mnt/cdrom type iso9660 (ro)
# vi /etc/zypp/repos.d/Local-iso.repo
[local-iso]
name=local iso repo
enabled=1
autorefresh=1
baseurl=file:/mnt/cdrom/
type=yast2
gpgcheck=0
http局域网源
/etc/zypp/repos.d # cat Icehouse.repo
[Icehouse]
name=Icehouse
enabled=1
autorefresh=1
baseurl=http://192.168.1.206:8080/download.opensuse.org/repositories/Cloud/OpenStack/Icehouse/SLE_11_SP3/
type=rpm-md
3、创建swift用户并配置权限
创建组
# groupadd swift
创建用户
# useradd -g swift -d /home/swift -s /bin/bash -m swift
为swift赋予sudo
# vi /etc/sudoers
在root ALL=(ALL) ALL一行下添加
swift ALL=(ALL) NOPASSWD:ALL
# passwd swift
5、基础配置
注意关闭SElinux和防火墙
分别在四台机器上安装依赖
# su - swift
sudo zypper install curlgcc memcached rsync sqlite3 xfsprogs Git-core libffi-dev python-setuptools
sudo zypper install python-coverage python-dev python-nose python-simplejson python-xattr python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip python-dnspython python-mock python-swiftclient openstack-swift
注：
python-swiftclient是客户端工具，可以在服务器上安装使服务器只充当客户端。
每个节点上配置
~> sudo chown -R swift:swift /etc/swift
~> /etc/swift> cat swift.conf
[swift-hash]
#random unique strings that can neverchange(DO NOT LOSE)
swift_hash_path_prefix = ‘od -t x8 -N 8 -An < /dev/random‘
swift_hash_path_suffix = ‘od -t x8 -N 8 -An < /dev/random‘
6、安装配置proxy
sudo zypper install openstack-swift-proxy memcached
启动memcached
/usr/sbin # # ./memcached -d -m 10 -u swift -l 192.168.1.211 11211 -c 256 -P /tmp/memcached/pid
注：
memcached为什么这样启动，而不是在配置文件里写入是因为我发现写入配置文件后启动服务并没有生效。
/usr/sbin # service memcached status
Checking for service memcached
running
/usr/sbin # netstat -an | grep 11211
tcp 0 0 192.168.1.211:11211 0.0.0.0:* LISTEN
udp 0 0 192.168.1.211:11211 0.0.0.0:*
查看配置后的文件
/etc/swift> grep -v "^#" /etc/swift/proxy-server.conf | grep -v "^$"
[DEFAULT]
user = swift
bind_port = 8090
workers = 8
[pipeline:main]
pipeline= healthcheck proxy-logging cache tempauth proxy-logging proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = ture
account_autocreate = true
[filter:tempauth]
use = egg:swift#tempauth
user_system_root= testpass .admin http://192.168.1.211:8090/v1/AUTH_system
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:cache]
use = egg:swift#memcache
memcache_servers = 192.168.1.211:11211
[filter:ratelimit]
use = egg:swift#ratelimit
[filter:domain_remap]
use = egg:swift#domain_remap
[filter:catch_errors]
use = egg:swift#catch_errors
[filter:cname_lookup]
use = egg:swift#cname_lookup
[filter:staticweb]
use = egg:swift#staticweb
[filter:tempurl]
use = egg:swift#tempurl
[filter:formpost]
use = egg:swift#formpost
[filter:name_check]
use = egg:swift#name_check
[filter:list-endpoints]
use = egg:swift#list_endpoints
[filter:proxy-logging]
use = egg:swift#proxy_logging
[filter:bulk]
use = egg:swift#bulk
[filter:container-quotas]
use = egg:swift#container_quotas
[filter:slo]
use = egg:swift#slo
[filter:dlo]
use = egg:swift#dlo
[filter:account-quotas]
use = egg:swift#account_quotas
[filter:gatekeeper]
use = egg:swift#gatekeeper
[filter:container_sync]
use = egg:swift#container_sync
创建account、container以及object rings
> cd /etc/swift
/etc/swift> sudo swift-ring-builder account.builder create 18 3 1
/etc/swift> sudo swift-ring-builder container.builder create 18 3 1
/etc/swift> sudo swift-ring-builder object.builder create 18 3 1
注：
18代表2的18次幂，这个数字取决与你希望一个ring中会有多少个partition，3代表object的副本数，1代表至少一个小时后才能被移动。
让ring记录每个storage存储设备
/etc/swift> export ZONE=1
/etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.212
/etc/swift> export WEIGHT=100
/etc/swift> export DEVICE=sdb1
/etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
WARNING: No region specified for z1-192.168.1.212:6002/sdb1. Defaulting to region 1.
Device d4r1z1-192.168.1.212:6002R192.168.1.212:6002/sdb1_"" with 100.0 weight got id 4
/etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
WARNING: No region specified for z1-192.168.1.212:6001/sdb1. Defaulting to region 1.
Device d0r1z1-192.168.1.212:6001R192.168.1.212:6001/sdb1_"" with 100.0 weight got id 0
/etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
WARNING: No region specified for z1-192.168.1.212:6000/sdb1. Defaulting to region 1.
Device d0r1z1-192.168.1.212:6000R192.168.1.212:6000/sdb1_"" with 100.0 weight got id 0
/etc/swift> export ZONE=2
/etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.213
/etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
WARNING: No region specified for z2-192.168.1.213:6002/sdb1. Defaulting to region 1.
Device d1r1z2-192.168.1.213:6002R192.168.1.213:6002/sdb1_"" with 100.0 weight got id 1
/etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
WARNING: No region specified for z2-192.168.1.213:6001/sdb1. Defaulting to region 1.
Device d1r1z2-192.168.1.213:6001R192.168.1.213:6001/sdb1_"" with 100.0 weight got id 1
/etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
WARNING: No region specified for z2-192.168.1.213:6000/sdb1. Defaulting to region 1.
Device d1r1z2-192.168.1.213:6000R192.168.1.213:6000/sdb1_"" with 100.0 weight got id 1
/etc/swift> export ZONE=3
/etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.214
/etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
WARNING: No region specified for z3-192.168.1.214:6002/sdb1. Defaulting to region 1.
Device d12r1z3-192.168.1.214:6002R192.168.1.214:6002/sdb1_"" with 100.0 weight got id 12
/etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
WARNING: No region specified for z3-192.168.1.214:6001/sdb1. Defaulting to region 1.
Device d2r1z3-192.168.1.214:6001R192.168.1.214:6001/sdb1_"" with 100.0 weight got id 2
/etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
WARNING: No region specified for z3-192.168.1.214:6000/sdb1. Defaulting to region 1.
Device d2r1z3-192.168.1.214:6000R192.168.1.214:6000/sdb1_"" with 100.0 weight got id 2
平衡ring
/etc/swift> sudo swift-ring-builder account.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.
/etc/swift> sudo swift-ring-builder container.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.
/etc/swift> sudo swift-ring-builder object.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.
如果操作失误也可以移除
/etc/swift> sudo swift-ring-builder account.builder remove z1-192.168.1.212:6020
d0r1z1-192.168.1.212:6020R192.168.1.212:6020/sdb1_"" marked for removal and will be removed next rebalance.
不过我发现最后一个无法删除，只能在新建另外一个，然后删除想要删除的
删除后需要rebalance
/etc/swift> sudo swift-ring-builder account.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.
当然也可以加入设备名
/etc/swift> sudo swift-ring-builder account.builder remove z1-192.168.1.212:6020/sdb1
d0r1z1-192.168.1.212:6020R192.168.1.212:6020/sdb1_"" marked for removal and will be removed next rebalance.
拷贝ring到其他所有节点
所有节点
sudo chown -R swift:swift /etc/swift
scp swift/*.ring.gz [email protected]:/etc/swift
scp swift/*.ring.gz [email protected]:/etc/swift
scp swift/*.ring.gz [email protected]:/etc/swift
开启proxy
/etc/swift> sudo swift-init proxy start
需要注意的是对于account.builder 、container.builder以及object.builder的命名一定要严格按照小写，并且名字不能修改，不然会出现类似下面的错误
sudo swift-init proxy start
Starting proxy-server...(/etc/swift/proxy-server.conf)
WARNING: SSL should only be enabled for testing purposes. Use external SSL termination for a production deployment.
Traceback (most recent call last):
File "/usr/bin/swift-proxy-server", line 23, in <module>
sys.exit(run_wsgi(conf_file, ‘proxy-server‘, default_port=8080, **options))
......
IOError: [Errno 2] No such file or directory: ‘/etc/swift/container.ring.gz‘
因为/etc/swift/目录下是Container.ring.gz
7、安装配置storage节点（三个节点安装配置）
sudo zypper install openstack-swift-account openstack-swift-container openstack-swift-object python-xml
sudo mkdir -p /srv/node/sdb1
sudo chown swift:swift /srv/node/sdb1
设备配置xfs卷
机器为底层lvm逻辑卷格式
# lvcreate -n swiftlv -L 10G vg0
# mkfs.xfs /dev/mapper/vg0-swiftlv
把挂载信息写入配置文件
# vi /etc/fstab
/dev/mapper/vg0-swiftlv /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0
重新加载配置文件
# mount -a
查看
# mount
/dev/mapper/vg0-swiftlv on /srv/node/sdb1 type xfs (rw,noatime,nodiratime,nobarrier,logbufs=8)
按实际情况修改rsync配置文件
/etc> sudo vi rsyncd.conf
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.1.212
[account]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/object.lock
其他两台注意：
address = 192.168.1.213
address = 192.168.1.214
/etc> sudo vi default/rsync
RSYNC_ENABLE=true
# service rsyncd start
/etc> rsync rsync://[email protected]
account
container
object
/etc> rsync rsync://[email protected]
account
container
object
/etc> rsync rsync://[email protected]
account
container
object
修改account、container以及object配置文件
~> grep -v "^#" /etc/swift/account-server.conf |grep -v "^$"
[DEFAULT]
user = swift
bind_ip = 192.168.1.212
bind_port = 6002
swift_dir = /etc/swift
devices = /srv/node
workers = 2
[pipeline:main]
pipeline = account-server
[app:account-server]
use = egg:swift#account
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:recon]
use = egg:swift#recon
[account-replicator]
[account-auditor]
[account-reaper]
~> grep -v "^#" /etc/swift/container-server.conf |grep -v "^$"
[DEFAULT]
user = swift
bind_ip = 192.168.1.212
bind_port = 6001
swift_dir = /etc/swift
devices = /srv/node
workers = 2
[pipeline:main]
pipeline = container-server
[app:container-server]
use = egg:swift#container
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:recon]
use = egg:swift#recon
[container-replicator]
[container-updater]
[container-auditor]
[container-sync]
~> grep -v "^#" /etc/swift/object-server.conf |grep -v "^$"
[DEFAULT]
user = swift
bind_ip = 192.168.1.212
bind_port = 6000
swift_dir = /etc/swift
devices = /srv/node
workers = 2
[pipeline:main]
pipeline = object-server
[app:object-server]
use = egg:swift#object
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:recon]
use = egg:swift#recon
[object-replicator]
[object-updater]
[object-auditor]
拷贝配置文件到其他两台机器，记得其他两台IP做修改
启动存储服务
etc/swift> sudo swift-init all start
Starting container-updater...(/etc/swift/container-server.conf)
Starting account-auditor...(/etc/swift/account-server.conf)
Starting object-replicator...(/etc/swift/object-server.conf)
Unable to locate config for proxy-server
Starting container-replicator...(/etc/swift/container-server.conf)
Starting object-auditor...(/etc/swift/object-server.conf)
Starting object-expirer...(/etc/swift/object-expirer.conf)
Starting container-auditor...(/etc/swift/container-server.conf)
Starting container-server...(/etc/swift/container-server.conf)
Starting account-server...(/etc/swift/account-server.conf)
Starting account-reaper...(/etc/swift/account-server.conf)
Starting container-sync...(/etc/swift/container-server.conf)
Starting account-replicator...(/etc/swift/account-server.conf)
Starting object-updater...(/etc/swift/object-server.conf)
Starting object-server...(/etc/swift/object-server.conf)
可以使用curl来和存储进行交互
curl -k -v -H‘X-Storage-User: system:root‘ -H ‘X-Storage-Pass: testpass‘ https://192.168.1.211:8080/auth/v1.0
curl -k -v -H‘X-Storage-User: system:root‘ -H ‘X-Storage-Pass: testpass‘
http://192.168.1.211:8080/auth/v1.0
如果curl出错：
[email protected]:~> curl -k -v -H‘X-Storage-User: system:root‘ -H ‘X-Storage-Pass: testpass‘ https://192.168.1.211:8080/auth/v1.0
* Protocol "https" not supported or disabled in libcurl
* Closing connection -1
curl: (1) Protocol "https" not supported or disabled in libcurl
则需要编译curl
# tar jxf curl-7.19.0.tar.bz2
# cd curl-7.19.0/
# ./configure --prefix=/usr/local/curl --with-ssl=/usr/local/openssl
curl version: 7.19.0
Host setup: x86_64-unknown-linux-gnu
Install prefix: /usr/local
Compiler: gcc
SSL support: enabled (OpenSSL)
SSH support: no (--with-libssh2)
zlib support: enabled
krb4 support: no (--with-krb4*)
GSSAPI support: no (--with-gssapi)
SPNEGO support: no (--with-spnego)
c-ares support: no (--enable-ares)
ipv6 support: enabled
IDN support: no (--with-libidn)
Build libcurl: Shared=yes, Static=yes
Built-in manual: no (--enable-manual)
Verbose errors: enabled (--disable-verbose)
SSPI support: no (--enable-sspi)
ca cert bundle: no
ca cert path: /etc/ssl/certs/
LDAP support: no (--enable-ldap / --with-ldap-lib / --with-lber-lib)
LDAPS support: no (--enable-ldaps)
# make && make install
/usr/local/bin # mv curl curl.bak
/usr/bin # mv curl curl.bak
# ln -s /usr/local/curl/bin/curl /usr/bin/curl
检测swift是否正常工作，出现像下面的输出就说明正常
/etc/swift> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass stat
Account: AUTH_system
Containers: 0
Objects: 0
Bytes: 0
Content-Type: text/plain; charset=utf-8
X-Timestamp: 1490690568.46981
X-Trans-Id: tx59301de1e9244b70a2065-0058da2208
X-Put-Timestamp: 1490690568.46981
新建一级子目录
swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass post container1
忘存储目录上传文件
swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container1 /etc/swift/*.ring.gz
查看存储里的目录和文件
swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list
下载存储目录里的文件到本地
swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass download container1