# Redis Clutser #
## 一、Redis Cluster集群 ##
参考资料：
http://www.cnblogs.com/lykxqhh/p/5690923.html
Redis集群搭建的方式有多种，例如使用zookper等，但从redis3.0之后版本支持redis cluster集群，Redis Cluster采用无中心结构，每个节点保存数据和整个集群状态，每个节点都和其他所有节点连接。其redis cluster架构图如下：
其结构特点：
1、所有的redis节点彼此互联(PING-PONG机制)，内部使用二进制协议优化传输速度和带宽。
2、节点的fail机制是通过集群中超过半数的节点检测失效时才生效。
3、客户端与redis节点直连，不需要中间件proxy层。客户端不需要连接集群所有节点，连接集群中其中一和可用节点即可。
4、redis cluster把所有的物理节点映射到[0-16384]slot上(不一定平均分配)，cluster负责维护node<->slot<->value。
5、Redis集群预分配好16384个桶，当需要在Redis集群中放置一个key-value时，根据CRC16(key)mod 16384的值，决定一个key放到哪个桶中。
1、redis cluster节点分配
----------
现在我们是三个主节点分别是A、B、C三个节点，他们可以是一台机器上的三个端口，也可以是三台不同的服务器。那么，采用哈希槽(hash slot)的方式来分配16384个slot的话，它们三个节点分别承担的slot区间是：
节点A覆盖0-5460；
节点B覆盖5461-10922；
节点C覆盖10923-16384.
获取数据：
如果存入一个值，按照redis clsuter哈希槽的算法：CRC16(‘key‘)%16384=6782。那么就会把这个key分配存储在B上了。同样，当我连接(A、B、C)任何一个节点想获取‘key‘这个key时，也会这样的算法，然后内部跳转到B节点上获取数据。
新增一个主节点：
新增一个主节点D，redis cluster的这种做法是从各个节点的前面各拿取一部分slot到D上，我会在接下来的实践中实验。大致就会变成这样：
A节点覆盖1365-5460；
B节点覆盖6827-10922；
C节点覆盖12888-16384；
D节点覆盖0-1364，5461-6826，10923-12287
同样删除一个节点也是类似，移动完成后就可以删除这个节点了。
2、Redis Cluster主从模式
----------
redis cluster为了保证数据的高可用性，加入了主从模式，一个主节点对应一个或多个从节点，主节点提取数据存储，从节点则是从主节点拉去数据备份，当这个主节点挂掉后，就会有这个从节点选取一个来充当主节点，从而保证集群不会挂掉。
上面那个例子里，集群有ABC三个主节点，如果这三个节点都没有加入从节点，如果B挂掉了，我们就无法访问整个集群了。A和B的slot也无法访问了。
所以我们在集群建立的时候，一定要为每个主节点都添加了从节点，比如像这样，集群包含主节点A、B、C以及从节点A1、B1、C1，那么即使B挂掉系统也可以继续正确工作。
B1节点替代了B节点，所以Redis集群将会选择B1节点作为新的主节点，集群将会继续正确地提供服务。当B重新开启后，它就会变成B1的从节点。
不过需要注意，如果节点B和B1同时挂掉了，Redis集群就无法正确地提供服务了。
## 二、redis集群的搭建 ##
集群中至少应该有奇数个节点，所以至少有三个节点，每个节点至少有一个备份节点，所以下面使用6节点(主节点、备份节点由redis-cluster集群确定)。
下面使用redis-3.2.5.tar.gz
1、安装redis节点指定端口
----------
实验环境：
10.69.213.127 bj05-ops-mongodb04.test.gomeplus.com
cd /opt
wget http://download.redis.io/releases/redis-3.2.5.tar.gz
tar -zxf redis-3.2.5.tar.gz
cd redis-3.2.5
make PREFIX=/usr/local/redis install
mkdir -p /gomeo2o/data/redis/9000/db
mkdir -p /gomeo2o/logs/redis/9000
mkdir -p /usr/local/redis/etc
rm -rf /usr/sbin/redis-*
cat > /usr/local/redis/etc/redis_9000.conf <<EOF
daemonize yes
pidfile "/gomeo2o/logs/redis/9000/redis_9000.pid"
tcp-backlog 511
timeout 0
tcp-keepalive 0
loglevel debug
logfile "/gomeo2o/logs/redis/9000/redis_9000.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /gomeo2o/data/redis/9000/db
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
maxclients 10000
maxmemory 1073741824
# 密码认证
#requirepass "KaIJgcSVDv67GePw"
#masterauth "KaIJgcSVDv67GePw"
#loglevel debug
# 最简集群配置
bind 0.0.0.0
port 9000
cluster-enabled yes
# 这个文件会自动生成在db目录下
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-require-full-coverage no
EOF
#复制六份，并修改端口和对应的路径
cd /usr/local/redis/etc/
cp redis_9000.conf redis_9001.conf
cp redis_9000.conf redis_9002.conf
cp redis_9000.conf redis_9003.conf
cp redis_9000.conf redis_9004.conf
cp redis_9000.conf redis_9005.conf
cp redis_9000.conf redis_9006.conf
cd /usr/local/redis/etc/
cp redis_9000.conf redis_9001.conf
cp redis_9000.conf redis_9002.conf
cp redis_9000.conf redis_9003.conf
cp redis_9000.conf redis_9004.conf
cp redis_9000.conf redis_9005.conf
cp redis_9000.conf redis_9006.conf
sed -i ‘s/9000/9001/g‘ redis_9001.conf
sed -i ‘s/9000/9002/g‘ redis_9002.conf
sed -i ‘s/9000/9003/g‘ redis_9003.conf
sed -i ‘s/9000/9004/g‘ redis_9004.conf
sed -i ‘s/9000/9005/g‘ redis_9005.conf
sed -i ‘s/9000/9006/g‘ redis_9006.conf
#创建所需要的目录：
mkdir -p /gomeo2o/data/redis/9000/db
mkdir -p /gomeo2o/logs/redis/9000
mkdir -p /gomeo2o/data/redis/9001/db
mkdir -p /gomeo2o/logs/redis/9001
mkdir -p /gomeo2o/data/redis/9002/db
mkdir -p /gomeo2o/logs/redis/9002
mkdir -p /gomeo2o/data/redis/9003/db
mkdir -p /gomeo2o/logs/redis/9003
mkdir -p /gomeo2o/data/redis/9004/db
mkdir -p /gomeo2o/logs/redis/9004
mkdir -p /gomeo2o/data/redis/9005/db
mkdir -p /gomeo2o/logs/redis/9005
mkdir -p /gomeo2o/data/redis/9006/db
mkdir -p /gomeo2o/logs/redis/9006
# 检查配置
cd /usr/local/redis/etc/
grep ‘/900‘ *
[[email protected] redis-3.2.5]# cd /usr/local/redis/etc/
[[email protected] etc]# ll
total 4
-rw-r--r-- 1 root root 1449 Dec 1 18:47 redis_9000.conf
[[email protected] etc]# cp redis_9000.conf redis_9001.conf
[[email protected] etc]# cp redis_9000.conf redis_9002.conf
[[email protected] etc]# cp redis_9000.conf redis_9003.conf
[[email protected] etc]# cp redis_9000.conf redis_9004.conf
[[email protected] etc]# cp redis_9000.conf redis_9005.conf
[[email protected] etc]# cp redis_9000.conf redis_9006.conf
[[email protected] etc]# sed -i ‘s/9000/9001/g‘ redis_9001.conf
[[email protected] etc]# sed -i ‘s/9000/9001/g‘ redis_9001.conf
[[email protected] etc]# sed -i ‘s/9000/9002/g‘ redis_9002.conf
[[email protected] etc]# sed -i ‘s/9000/9003/g‘ redis_9003.conf
[[email protected] etc]# sed -i ‘s/9000/9004/g‘ redis_9004.conf
[[email protected] etc]# sed -i ‘s/9000/9005/g‘ redis_9005.conf
[[email protected] etc]# sed -i ‘s/9000/9006/g‘ redis_9006.conf
[[email protected] etc]# cd /usr/local/redis/etc/
[[email protected] etc]# grep ‘/900‘ *
redis_9000.conf:pidfile "/gomeo2o/logs/redis/9000/redis_9000.pid"
redis_9000.conf:logfile "/gomeo2o/logs/redis/9000/redis_9000.log"
redis_9000.conf:dir /gomeo2o/data/redis/9000/db
redis_9001.conf:pidfile "/gomeo2o/logs/redis/9001/redis_9001.pid"
redis_9001.conf:logfile "/gomeo2o/logs/redis/9001/redis_9001.log"
redis_9001.conf:dir /gomeo2o/data/redis/9001/db
redis_9002.conf:pidfile "/gomeo2o/logs/redis/9002/redis_9002.pid"
redis_9002.conf:logfile "/gomeo2o/logs/redis/9002/redis_9002.log"
redis_9002.conf:dir /gomeo2o/data/redis/9002/db
redis_9003.conf:pidfile "/gomeo2o/logs/redis/9003/redis_9003.pid"
redis_9003.conf:logfile "/gomeo2o/logs/redis/9003/redis_9003.log"
redis_9003.conf:dir /gomeo2o/data/redis/9003/db
redis_9004.conf:pidfile "/gomeo2o/logs/redis/9004/redis_9004.pid"
redis_9004.conf:logfile "/gomeo2o/logs/redis/9004/redis_9004.log"
redis_9004.conf:dir /gomeo2o/data/redis/9004/db
redis_9005.conf:pidfile "/gomeo2o/logs/redis/9005/redis_9005.pid"
redis_9005.conf:logfile "/gomeo2o/logs/redis/9005/redis_9005.log"
redis_9005.conf:dir /gomeo2o/data/redis/9005/db
redis_9006.conf:pidfile "/gomeo2o/logs/redis/9006/redis_9006.pid"
redis_9006.conf:logfile "/gomeo2o/logs/redis/9006/redis_9006.log"
redis_9006.conf:dir /gomeo2o/data/redis/9006/db
[[email protected] etc]# mkdir -p /gomeo2o/data/redis/9001/db
[[email protected] etc]# mkdir -p /gomeo2o/logs/redis/9001
[[email protected] etc]#
[[email protected] etc]# mkdir -p /gomeo2o/data/redis/9002/db
[[email protected] etc]# mkdir -p /gomeo2o/logs/redis/9002
[[email protected] etc]#
[[email protected] etc]# mkdir -p /gomeo2o/data/redis/9003/db
[[email protected] etc]# mkdir -p /gomeo2o/logs/redis/9003
[[email protected] etc]#
[[email protected] etc]# mkdir -p /gomeo2o/data/redis/9004/db
[[email protected] etc]# mkdir -p /gomeo2o/logs/redis/9004
[[email protected] etc]#
[[email protected] etc]# mkdir -p /gomeo2o/data/redis/9005/db
[[email protected] etc]# mkdir -p /gomeo2o/logs/redis/9005
[[email protected] etc]#
[[email protected] etc]# mkdir -p /gomeo2o/data/redis/9006/db
[[email protected] etc]# mkdir -p /gomeo2o/logs/redis/9006
[[email protected] etc]#
# 系统内核配置
echo "vm.overcommit_memory=1" >> /etc/sysctl.conf
echo never > /sys/kernel/mm/transparent_hugepage/enabled
sysctl -p
cat >>/etc/rc.local <<EOF
echo never > /sys/kernel/mm/transparent_hugepage/enabled
EOF
#启动实例
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9000.conf
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9001.conf
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9002.conf
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9003.conf
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9004.conf
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9005.conf
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9006.conf
ps -ef|grep -v ‘grep‘ |grep ‘redis‘
[[email protected] etc]# ps -ef|grep -v ‘grep‘ |grep ‘redis‘
root 31100 1 0 19:12 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9000 [cluster]
root 31104 1 0 19:12 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9001 [cluster]
root 31108 1 0 19:12 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9002 [cluster]
root 31112 1 0 19:12 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9003 [cluster]
root 31116 1 0 19:12 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9004 [cluster]
root 31120 1 0 19:12 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9005 [cluster]
root 31124 1 0 19:12 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9006 [cluster]
#查看日志
cat /gomeo2o/logs/redis/9000/redis_9000.log
cat /gomeo2o/logs/redis/9001/redis_9001.log
cat /gomeo2o/logs/redis/9002/redis_9002.log
cat /gomeo2o/logs/redis/9003/redis_9003.log
cat /gomeo2o/logs/redis/9004/redis_9004.log
cat /gomeo2o/logs/redis/9005/redis_9005.log
cat /gomeo2o/logs/redis/9006/redis_9006.log
#以9005端口为例
[[email protected] etc]# cat /gomeo2o/logs/redis/9006/redis_9006.log
31124:M 01 Dec 19:12:04.920 * No cluster configuration found, I‘m d4395d6492bbd56956a4e3a8cf073107afe4f38a
_._
_.-``__ ‘‘-._
_.-`` `. `_. ‘‘-._ Redis 3.2.5 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ‘‘-._
( ‘ , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|‘` _.-‘| Port: 9006
| `-._ `._ / _.-‘ | PID: 31124
`-._ `-._ `-./ _.-‘ _.-‘
|`-._`-._ `-.__.-‘ _.-‘_.-‘|
| `-._`-._ _.-‘_.-‘ | http://redis.io
`-._ `-._`-.__.-‘_.-‘ _.-‘
|`-._`-._ `-.__.-‘ _.-‘_.-‘|
| `-._`-._ _.-‘_.-‘ |
`-._ `-._`-.__.-‘_.-‘ _.-‘
`-._ `-.__.-‘ _.-‘
`-._ _.-‘
`-.__.-‘
31124:M 01 Dec 19:12:04.922 # Server started, Redis version 3.2.5
31124:M 01 Dec 19:12:04.922 * The server is now ready to accept connections on port 9006
31124:M 01 Dec 19:12:04.922 - 0 clients connected (0 slaves), 1221904 bytes in use
31124:M 01 Dec 19:12:09.936 - 0 clients connected (0 slaves), 1221904 bytes in use
31124:M 01 Dec 19:12:14.949 - 0 clients connected (0 slaves), 1221904 bytes in use
......
......
......
[[email protected] etc]#
# 关闭实例
/usr/local/redis/bin/redis-cli -c -p 9000 shutdown
/usr/local/redis/bin/redis-cli -c -p 9001 shutdown
/usr/local/redis/bin/redis-cli -c -p 9002 shutdown
/usr/local/redis/bin/redis-cli -c -p 9003 shutdown
/usr/local/redis/bin/redis-cli -c -p 9004 shutdown
/usr/local/redis/bin/redis-cli -c -p 9005 shutdown
/usr/local/redis/bin/redis-cli -c -p 9006 shutdown
ps -ef|grep redis |grep -v ‘grep‘
# 需要联网，安装支持语言和库
yum -y install ruby ruby-rdoc
yum -y install rubygems
gem sources --add https://ruby.taobao.org/ --remove http://rubygems.org/
gem sources -l
gem install redis
# 创建进群成员
cd /opt/redis-3.2.5/src/
./redis-trib.rb create --replicas 1 10.125.141.137:9000 10.125.141.137:9001 10.125.141.137:9002 10.125.141.137:9003 10.125.141.137:9004 10.125.141.137:9005
#使用create命令 --replicas 1 参数表示为每个主节点创建一个从节点，其他参数是实例的地址集合。
[[email protected] redis-3.2.5]# cd /opt/redis-3.2.5/src/
[[email protected] src]# ./redis-trib.rb create --replicas 1 10.125.141.137:9000 10.125.141.137:9001 10.125.141.137:9002 10.125.141.137:9003 10.125.141.137:9004 10.125.141.137:9005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.125.141.137:9000
10.125.141.137:9001
10.125.141.137:9002
Adding replica 10.125.141.137:9003 to 10.125.141.137:9000
Adding replica 10.125.141.137:9004 to 10.125.141.137:9001
Adding replica 10.125.141.137:9005 to 10.125.141.137:9002
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-5460 (5461 slots) master
M: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots:5461-10922 (5462 slots) master
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
S: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
replicates 09c4cec212b76ff325b92ea6c62526c4104ae8cd
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
Can I set the above configuration? (type ‘yes‘ to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
S: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots: (0 slots) slave
replicates 09c4cec212b76ff325b92ea6c62526c4104ae8cd
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] src]#
上面显示创建成功，有3个主节点，3个从节点，每个节点都是成功连接状态。
3个主节点[M]以及分配的哈希卡槽如下：
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
3个从节点[S]以及附属的主节点如下：
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
S: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots: (0 slots) slave
replicates 09c4cec212b76ff325b92ea6c62526c4104ae8cd
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
## 三、redis集群的测试 ##
1、测试存取值
客户端连接集群redis-cli需要带上 -c ，redis-cli -c -p 端口号
[[email protected] local]# /usr/local/redis/bin/redis-cli -c -p 9000
127.0.0.1:9000> set names andy
-> Redirected to slot [6659] located at 10.125.141.137:9001
OK
10.125.141.137:9001> get names
"andy"
10.125.141.137:9001>
根据redis-cluster的key值分配，name应该分配到节点10.125.141.137:9001[5461-10922]上，上面显示redis cluster自动从9000跳转到了9001节点。
我们可以测试一下9005从节点获取name值
[[email protected] local]# /usr/local/redis/bin/redis-cli -c -p 9005
127.0.0.1:9005> get names
-> Redirected to slot [6659] located at 10.125.141.137:9001
"andy"
10.125.141.137:9001>
9005是9002的从节点，从上面看也是自动跳转至9001获取值，这也是redis cluster的特点，它是去中心化，每个节点都是对等的，连接哪个节点都可以获取和设置数据。
## 四、集群节点选举 ##
现在模拟将9001节点挂掉，按照redis-cluster原理会选举会将9001的从节点9004选举为主节点。
[[email protected] local]# ps -ef | grep redis |grep -v ‘grep‘
root 31100 1 0 19:12 ? 00:00:03 /usr/local/redis/bin/redis-server 0.0.0.0:9000 [cluster]
root 31104 1 0 19:12 ? 00:00:03 /usr/local/redis/bin/redis-server 0.0.0.0:9001 [cluster]
root 31108 1 0 19:12 ? 00:00:03 /usr/local/redis/bin/redis-server 0.0.0.0:9002 [cluster]
root 31112 1 0 19:12 ? 00:00:03 /usr/local/redis/bin/redis-server 0.0.0.0:9003 [cluster]
root 31116 1 0 19:12 ? 00:00:03 /usr/local/redis/bin/redis-server 0.0.0.0:9004 [cluster]
root 31120 1 0 19:12 ? 00:00:03 /usr/local/redis/bin/redis-server 0.0.0.0:9005 [cluster]
root 31124 1 0 19:12 ? 00:00:01 /usr/local/redis/bin/redis-server 0.0.0.0:9006 [cluster]
[[email protected] local]#
[[email protected] local]# kill 31104
再查看集群中的9001节点
[[email protected]bj02-ops-mgotest01 local]# /usr/local/redis/bin/redis-cli -c -p 9001
Could not connect to Redis at 127.0.0.1:9001: Connection refused
Could not connect to Redis at 127.0.0.1:9001: Connection refused
not connected> exit
[[email protected] local]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:5461-10922 (5462 slots) master
0 additional replica(s)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] local]#
可以看到集群连接不了9001节点，而9004由原来的S转换为M节点，代替了原来的9001节点。我们可以获取name值：
[[email protected] local]# /usr/local/redis/bin/redis-cli -c -p 9005
127.0.0.1:9005> get names
-> Redirected to slot [6659] located at 10.125.141.137:9004
"andy"
10.125.141.137:9004> exit
[[email protected] local]#
从9005节点连入，自动跳转到9004节点，并且获取name值。
现在我们将7002节点恢复，看是否会自动加入集群中以及充当的M还是S节点。
[[email protected] local]# /usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9001.conf
[[email protected] local]# ps -ef|grep redis|grep -v ‘grep‘
root 31100 1 0 19:12 ? 00:00:06 /usr/local/redis/bin/redis-server 0.0.0.0:9000 [cluster]
root 31108 1 0 19:12 ? 00:00:06 /usr/local/redis/bin/redis-server 0.0.0.0:9002 [cluster]
root 31112 1 0 19:12 ? 00:00:06 /usr/local/redis/bin/redis-server 0.0.0.0:9003 [cluster]
root 31116 1 0 19:12 ? 00:00:06 /usr/local/redis/bin/redis-server 0.0.0.0:9004 [cluster]
root 31120 1 0 19:12 ? 00:00:06 /usr/local/redis/bin/redis-server 0.0.0.0:9005 [cluster]
root 31124 1 0 19:12 ? 00:00:02 /usr/local/redis/bin/redis-server 0.0.0.0:9006 [cluster]
root 31252 1 0 19:50 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9001 [cluster]
[[email protected] local]#
查看集群状态
/usr/local/redis/bin/redis-cli -c -p 9000 cluster nodes
[[email protected] local]# /usr/local/redis/bin/redis-cli -c -p 9000 cluster nodes
09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001 slave 67f093514bf6499f701f11f40b18937098b983c0 0 1480593304173 7 connected
c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005 slave c311d5160a2f4f1df8fea93e49c97fdbad5da920 0 1480593305175 6 connected
adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000 myself,master - 0 0 1 connected 0-5460
67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004 master - 0 1480593303672 7 connected 5461-10922
bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003 slave adb0a771e41b8284adbed417f89eb8c8ce943835 0 1480593305175 4 connected
c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002 master - 0 1480593304673 3 connected 10923-16383
[[email protected] local]#
测试健康状况
./redis-trib.rb check 127.0.0.1:9000
[[email protected] local]# /opt/redis-3.2.5/src/redis-trib.rb check 127.0.0.1:9000
>>> Performing Cluster Check (using node 127.0.0.1:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 127.0.0.1:9000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] local]#
可以看到9001节点变成了67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004的从节点。
## 五、集群节点添加 ##
节点新增包括新增主节点、从节点两种情况。以下分别做一下测试：
1、新增主节点
----------
新增一个节点9006作为主节点修改配置文件(之前已经准备好了)
[[email protected] local]# ps -ef|grep 9006 |grep -v ‘grep‘
root 31124 1 0 19:12 ? 00:00:02 /usr/local/redis/bin/redis-server 0.0.0.0:9006 [cluster]
[[email protected] local]#
这里的10.125.141.137:9006为新增节点，10.125.141.137:9003这个是指集群中的其中一个节点，用来表示是哪个集群。理论上那个集群的任何一个节点都可以。
[[email protected] local]# /opt/redis-3.2.5/src/redis-trib.rb add-node 10.125.141.137:9006 10.125.141.137:9003
>>> Adding node 10.125.141.137:9006 to cluster 10.125.141.137:9003
>>> Performing Cluster Check (using node 10.125.141.137:9003)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.125.141.137:9006 to make it join the cluster.
[OK] New node added correctly.
[[email protected] local]#
可以看到9006加入这个Cluster，并成为一个新的节点。可以check以下9006节点状态
[[email protected] local]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9006
>>> Performing Cluster Check (using node 10.125.141.137:9006)
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots: (0 slots) master
0 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] local]#
上面信息可以看到有4个M节点，3个S节点，9006成为了M主节点，它没有附属的从节点，而且Cluster并未给7007分配哈希卡槽（0 slots）。
可以从客户端连接集群查看一下，集群节点的连接情况
[[email protected] local]# /usr/local/redis/bin/redis-cli -c -p 9006 cluster nodes
09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001 slave 67f093514bf6499f701f11f40b18937098b983c0 0 1480594174584 7 connected
bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003 slave adb0a771e41b8284adbed417f89eb8c8ce943835 0 1480594173582 1 connected
c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002 master - 0 1480594173080 3 connected 10923-16383
adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000 master - 0 1480594172578 1 connected 0-5460
d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006 myself,master - 0 0 0 connected
c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005 slave c311d5160a2f4f1df8fea93e49c97fdbad5da920 0 1480594174082 3 connected
67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004 master - 0 1480594174083 7 connected 5461-10922
[[email protected] local]#
redis-cluster在新增节点时并未分配卡槽，需要我们手动对集群进行重新分片迁移数据，需要重新分片命令reshard
/opt/redis-3.2.5/src/redis-trib.rb reshard 10.125.141.137:9000
这个命令是用来迁移slot节点的，后面的10.125.141.137:9000是表示是哪个集群，端口填［9000-9006］都可以，执行结果如下：
[[email protected] local]# /opt/redis-3.2.5/src/redis-trib.rb reshard 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots: (0 slots) master
0 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
它提示我们需要迁移多少slot到9006上，我们平分16384个哈希槽给4个节点：16384/4 = 4096，我们需要移动4096个槽点到9006上。
接着会问，要把这些哈希槽迁移到哪里去，需要填写目的机器的id号。这里填写9006对应的机器id号码。
What is the receiving node ID? d4395d6492bbd56956a4e3a8cf073107afe4f38a
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? d4395d6492bbd56956a4e3a8cf073107afe4f38a
Please enter all the source node IDs.
Type ‘all‘ to use all the nodes as source nodes for the hash slots.
Type ‘done‘ once you entered all the source nodes IDs.
Source node #1:all
redis-trib 会向你询问重新分片的源节点（source node），即要从集群的哪个节点中取出 4096 个哈希槽，还是从全部节点提取4096个哈希槽， 并将这些槽移动到9006节点上面。
如果我们不打算从特定的节点上取出指定数量的哈希槽，那么可以向redis-trib输入 all，这样的话， 集群中的所有主节点都会成为源节点，redis-trib从各个源节点中各取出一部分哈希槽，凑够4096个，然后移动到9006节点上：
Source node #1:all
然后开始从别的主节点迁移哈希槽，并且确认。
......
......
Moving slot 12277 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12278 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12279 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12280 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12281 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12282 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12283 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12284 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12285 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12286 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Moving slot 12287 from c311d5160a2f4f1df8fea93e49c97fdbad5da920
Do you want to proceed with the proposed reshard plan (yes/no)? yes
确认之后，redis-trib就开始执行分片操作，将哈希槽一个一个从源主节点移动到目标主节点9006。
重新分片结束后我们可以check以下节点的分配情况。
[[email protected] local]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] local]#
可以看到
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
可以看到9006节点分片的哈希槽片不是连续的，间隔的移动。
[[email protected] local]# /usr/local/redis/bin/redis-cli -c -p 9006
127.0.0.1:9006> keys *
1) "names"
127.0.0.1:9006>
可以看到将9004的names[6659]移动到9006节点上，主节点7007添加成功。
2、新增从节点
----------
新增一个节点9007节点，使用add-node --slave命令。
cp /usr/local/redis/etc/redis_9000.conf /usr/local/redis/etc/redis_9007.conf
sed -i ‘s/9000/9007/g‘ /usr/local/redis/etc/redis_9007.conf
grep 9007 /usr/local/redis/etc/redis_9007.conf
mkdir -p /gomeo2o/logs/redis/9007/
mkdir -p /gomeo2o/logs/redis/9007/
/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9007.conf
[[email protected] local]# cp /usr/local/redis/etc/redis_9000.conf /usr/local/redis/etc/redis_9007.conf
[[email protected] local]# sed -i ‘s/9000/9007/g‘ /usr/local/redis/etc/redis_9007.conf
[[email protected] local]# grep 9007 /usr/local/redis/etc/redis_9007.conf
pidfile "/gomeo2o/logs/redis/9007/redis_9007.pid"
logfile "/gomeo2o/logs/redis/9007/redis_9007.log"
dir /gomeo2o/data/redis/9007/db
port 9007
[[email protected] ~]# mkdir -p /gomeo2o/logs/redis/9007/
[[email protected] ~]# mkdir -p /gomeo2o/data/redis/9007/db
[[email protected] ~]# /usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9007.conf
[[email protected] ~]# ps -ef|grep 9007|grep -v ‘grep‘
root 1006 1 0 17:32 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9007 [cluster]
[[email protected] ~]#
redis-trib增加从节点的命令为：
/opt/redis-3.2.5/src/redis-trib.rb add-node --slave --master-id $[nodeid] 10.125.141.137:9007 10.125.141.137:9000
nodeid为要加到master主节点的node id，10.125.141.137:9007为新增的从节点，10.125.141.137:9000为集群的一个节点（集群的任意节点都行），用来辨识是哪个集群；如果没有给定那个主节点--master-id的话，redis-trib将会将新增的从节点随机到从节点较少的主节点上。
现在我们添加一下9007，看是否会自动加到没有从节点的9006主节点上。
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] ~]#
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb add-node --slave 10.125.141.137:9007 10.125.141.137:9000
>>> Adding node 10.125.141.137:9007 to cluster 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 10.125.141.137:9006
>>> Send CLUSTER MEET to node 10.125.141.137:9007 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 10.125.141.137:9006.
[OK] New node added correctly.
[[email protected] ~]#
可以看到9006已经有了一个从节点，从节点是9007
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S: 22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007
slots: (0 slots) slave
replicates d4395d6492bbd56956a4e3a8cf073107afe4f38a
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
1 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] ~]#
再测试一下指定主节点添加从节点，给9006增加9008从节点。
cp /usr/local/redis/etc/redis_9000.conf /usr/local/redis/etc/redis_9008.conf
sed -i ‘s/9000/9008/g‘ /usr/local/redis/etc/redis_9008.conf
grep 900* /usr/local/redis/etc/redis_9008.conf
mkdir -p /gomeo2o/logs/redis/9008/
mkdir -p /gomeo2o/data/redis/9008/db
ps -ef|grep 9008 |grep -v ‘grep‘
[[email protected] ~]# cp /usr/local/redis/etc/redis_9000.conf /usr/local/redis/etc/redis_9008.conf
[[email protected] ~]# sed -i ‘s/9000/9008/g‘ /usr/local/redis/etc/redis_9008.conf
[[email protected] ~]# grep 900* /usr/local/redis/etc/redis_9008.conf
pidfile "/gomeo2o/logs/redis/9008/redis_9008.pid"
logfile "/gomeo2o/logs/redis/9008/redis_9008.log"
save 900 1
dir /gomeo2o/data/redis/9008/db
port 9008
[[email protected] ~]#
[[email protected] ~]# mkdir -p /gomeo2o/logs/redis/9008/
[[email protected] ~]# mkdir -p /gomeo2o/data/redis/9008/db
[[email protected] ~]# ps -ef|grep 9008 |grep -v ‘grep‘
root 1088 1 0 18:22 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:9008 [cluster]
[[email protected] ~]#
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb add-node --slave --master-id d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9008 10.125.141.137:9006
>>> Adding node 10.125.141.137:9008 to cluster 10.125.141.137:9006
>>> Performing Cluster Check (using node 10.125.141.137:9006)
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S: 22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007
slots: (0 slots) slave
replicates d4395d6492bbd56956a4e3a8cf073107afe4f38a
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.125.141.137:9008 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 10.125.141.137:9006.
[OK] New node added correctly.
[[email protected] ~]#
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 0800e962aaeb41ede634f267acf7d1280ab0e3c3 10.125.141.137:9008
slots: (0 slots) slave
replicates d4395d6492bbd56956a4e3a8cf073107afe4f38a
S: 22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007
slots: (0 slots) slave
replicates d4395d6492bbd56956a4e3a8cf073107afe4f38a
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] ~]#
显示从节点9008节点添加到9006主节点下面，可以看一下9006的从节点，如下：可以看出来，9006已经有两个从节点，分别是10.125.141.137:9008和10.125.141.137:9007。
[[email protected] ~]# /usr/local/redis/bin/redis-cli -c -p 9000 cluster nodes
adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000 myself,master - 0 0 1 connected 1365-5460
67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004 master - 0 1480674797185 7 connected 6827-10922
c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002 master - 0 1480674798689 3 connected 12288-16383
0800e962aaeb41ede634f267acf7d1280ab0e3c3 10.125.141.137:9008 slave d4395d6492bbd56956a4e3a8cf073107afe4f38a 0 1480674799191 8 connected
22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007 slave d4395d6492bbd56956a4e3a8cf073107afe4f38a 0 1480674798689 8 connected
09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001 slave 67f093514bf6499f701f11f40b18937098b983c0 0 1480674799190 7 connected
c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005 slave c311d5160a2f4f1df8fea93e49c97fdbad5da920 0 1480674798188 6 connected
bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003 slave adb0a771e41b8284adbed417f89eb8c8ce943835 0 1480674799191 4 connected
d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006 master - 0 1480674798187 8 connected 0-1364 5461-6826 10923-12287
[[email protected] ~]#
我们测试一下9006节点挂掉，看和7009那个成为主节点。
[[email protected] ~]# ps -ef | grep redis |grep -v ‘grep‘
root 1006 1 0 17:32 ? 00:00:16 /usr/local/redis/bin/redis-server 0.0.0.0:9007 [cluster]
root 1088 1 0 18:22 ? 00:00:05 /usr/local/redis/bin/redis-server 0.0.0.0:9008 [cluster]
root 31100 1 0 Dec01 ? 00:04:28 /usr/local/redis/bin/redis-server 0.0.0.0:9000 [cluster]
root 31108 1 0 Dec01 ? 00:04:26 /usr/local/redis/bin/redis-server 0.0.0.0:9002 [cluster]
root 31112 1 0 Dec01 ? 00:04:18 /usr/local/redis/bin/redis-server 0.0.0.0:9003 [cluster]
root 31116 1 0 Dec01 ? 00:04:25 /usr/local/redis/bin/redis-server 0.0.0.0:9004 [cluster]
root 31120 1 0 Dec01 ? 00:04:21 /usr/local/redis/bin/redis-server 0.0.0.0:9005 [cluster]
root 31124 1 0 Dec01 ? 00:04:13 /usr/local/redis/bin/redis-server 0.0.0.0:9006 [cluster]
root 31252 1 0 Dec01 ? 00:04:11 /usr/local/redis/bin/redis-server 0.0.0.0:9001 [cluster]
[[email protected] ~]# kill -9 31124
[[email protected] ~]#
[[email protected] ~]# /usr/local/redis/bin/redis-cli -c -p 9000 cluster nodes
adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000 myself,master - 0 0 1 connected 1365-5460
67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004 master - 0 1480676007001 7 connected 6827-10922
c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002 master - 0 1480676006497 3 connected 12288-16383
0800e962aaeb41ede634f267acf7d1280ab0e3c3 10.125.141.137:9008 master - 0 1480676007502 9 connected 0-1364 5461-6826 10923-12287
22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007 slave 0800e962aaeb41ede634f267acf7d1280ab0e3c3 0 1480676007001 9 connected
09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001 slave 67f093514bf6499f701f11f40b18937098b983c0 0 1480676007502 7 connected
c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005 slave c311d5160a2f4f1df8fea93e49c97fdbad5da920 0 1480676008005 6 connected
bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003 slave adb0a771e41b8284adbed417f89eb8c8ce943835 0 1480676008508 4 connected
d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006 master,fail - 1480675926833 1480675926634 8 disconnected
[[email protected] ~]#
可以看到9008代替9006成了主节点。重启9006之后，会自动变成9008的从节点。
[[email protected] ~]# /usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_9006.conf
[[email protected] ~]# /usr/local/redis/bin/redis-cli -c -p 9000 cluster nodes
adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000 myself,master - 0 0 1 connected 1365-5460
67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004 master - 0 1480676134530 7 connected 6827-10922
c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002 master - 0 1480676135032 3 connected 12288-16383
0800e962aaeb41ede634f267acf7d1280ab0e3c3 10.125.141.137:9008 master - 0 1480676134028 9 connected 0-1364 5461-6826 10923-12287
22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007 slave 0800e962aaeb41ede634f267acf7d1280ab0e3c3 0 1480676133025 9 connected
09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001 slave 67f093514bf6499f701f11f40b18937098b983c0 0 1480676133026 7 connected
c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005 slave c311d5160a2f4f1df8fea93e49c97fdbad5da920 0 1480676134029 6 connected
bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003 slave adb0a771e41b8284adbed417f89eb8c8ce943835 0 1480676134028 4 connected
d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006 slave 0800e962aaeb41ede634f267acf7d1280ab0e3c3 0 1480676134932 9 connected
[[email protected] ~]#
验证了之前的测试。
## 六、节点的移除 ##
和节点添加一样，移除节点也有移除主节点，从节点。
1、移除主节点
----------
/opt/redis-3.2.5/src/redis-trib.rb del-node host:port node_id
例如:
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb del-node 10.125.141.137:9000 0800e962aaeb41ede634f267acf7d1280ab0e3c3
10.125.141.137:9000是集群节点，node-id为要删除的主节点。 和添加节点不同，移除节点node-id是必需的，测试删除9008主节点：
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb del-node 10.125.141.137:9000 0800e962aaeb41ede634f267acf7d1280ab0e3c3
>>> Removing node 0800e962aaeb41ede634f267acf7d1280ab0e3c3 from cluster 10.125.141.137:9000
[ERR] Node 10.125.141.137:9008 is not empty! Reshard data away and try again.
[[email protected] ~]#
redis cluster提示9008已经有数据了，不能够被删除，需要将他的数据转移出去，也就是和新增主节点一样需重新分片。
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb reshard 10.125.141.137:9008
>>> Performing Cluster Check (using node 10.125.141.137:9008)
M: 0800e962aaeb41ede634f267acf7d1280ab0e3c3 10.125.141.137:9008
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
2 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007
slots: (0 slots) slave
replicates 0800e962aaeb41ede634f267acf7d1280ab0e3c3
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots: (0 slots) slave
replicates 0800e962aaeb41ede634f267acf7d1280ab0e3c3
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID?
执行以后会提示我们移除的大小，因为9008占用了4096个槽点。输入4096，提示移动的node id，填写9007的node id。
What is the receiving node ID? 22c190eefeba4987df1ce9762a07da90b9afbe08
The specified node is not known or not a master, please retry.
What is the receiving node ID?
这里需要填写一个主节点的id号。这里我们填写9000的id。
What is the receiving node ID? adb0a771e41b8284adbed417f89eb8c8ce943835
Please enter all the source node IDs.
Type ‘all‘ to use all the nodes as source nodes for the hash slots.
Type ‘done‘ once you entered all the source nodes IDs.
Source node #1:0800e962aaeb41ede634f267acf7d1280ab0e3c3
Source node #2:done
......
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes
......
......
这里需要填写被删除的节点的id号。确认之后会一个一个将9008的卡槽移到到9000上。此时9008的数据已经全部迁移到了9000上，查看验证，此时9008已经没有槽位了。而9008上有8192个槽位。
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9001
>>> Performing Cluster Check (using node 10.125.141.137:9001)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: 0800e962aaeb41ede634f267acf7d1280ab0e3c3 10.125.141.137:9008
slots: (0 slots) master
0 additional replica(s)
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-6826,10923-12287 (8192 slots) master
3 additional replica(s)
S: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
S: 22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] ~]#
在执行删除操作，此时成功删除节点9008.
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb del-node 10.125.141.137:9008 0800e962aaeb41ede634f267acf7d1280ab0e3c3
>>> Removing node 0800e962aaeb41ede634f267acf7d1280ab0e3c3 from cluster 10.125.141.137:9008
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[[email protected] ~]#
已经删除了9008节点。
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9001 >>> Performing Cluster Check (using node 10.125.141.137:9001)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-6826,10923-12287 (8192 slots) master
3 additional replica(s)
S: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
S: 22c190eefeba4987df1ce9762a07da90b9afbe08 10.125.141.137:9007
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] ~]#
可以看到9008已经连接不了；而9008的从节点9007自动分配到了9000主节点中，9000现在有3个从节点。
2、移除从节点
----------
比如移除9000的从节点9007节点。
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb del-node 10.125.141.137:9007 22c190eefeba4987df1ce9762a07da90b9afbe08
>>> Removing node 22c190eefeba4987df1ce9762a07da90b9afbe08 from cluster 10.125.141.137:9007
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[[email protected] ~]#
删除从节点比较方便，现在redis-cluster中有3个主节点，4个从节点，如下：
[[email protected] ~]# /opt/redis-3.2.5/src/redis-trib.rb check 10.125.141.137:9000
>>> Performing Cluster Check (using node 10.125.141.137:9000)
M: adb0a771e41b8284adbed417f89eb8c8ce943835 10.125.141.137:9000
slots:0-6826,10923-12287 (8192 slots) master
2 additional replica(s)
M: 67f093514bf6499f701f11f40b18937098b983c0 10.125.141.137:9004
slots:6827-10922 (4096 slots) master
1 additional replica(s)
M: c311d5160a2f4f1df8fea93e49c97fdbad5da920 10.125.141.137:9002
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 09c4cec212b76ff325b92ea6c62526c4104ae8cd 10.125.141.137:9001
slots: (0 slots) slave
replicates 67f093514bf6499f701f11f40b18937098b983c0
S: c03a00b628b4c65b68f24f249e4694db937b0c5c 10.125.141.137:9005
slots: (0 slots) slave
replicates c311d5160a2f4f1df8fea93e49c97fdbad5da920
S: bec729e8b454b190db07997661f003b869391b9b 10.125.141.137:9003
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
S: d4395d6492bbd56956a4e3a8cf073107afe4f38a 10.125.141.137:9006
slots: (0 slots) slave
replicates adb0a771e41b8284adbed417f89eb8c8ce943835
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[[email protected] ~]#
ok，测试到这儿吧。