Redis的集群(伸缩)

Redis集群提供了节点的扩容和收缩方案,在不影响集群对外服务的情况下,可以为集群添加节点进行扩容,也可以下线节点进行缩容。其中的原理可理解为槽和对应的数据在不同节点间移动。

扩容集群

Redis的集群(搭建)中搭建了6个节点,其中3个主节点分别维护自己负责的槽和数据,为了后续测试,填充若干测试数据。

$ for i in $(seq 1 70000); do redis-cli -p 6879 -c set key:migrate:test:${i} ${i}; done

$ redis-trib.rb info 127.0.0.1:6879

127.0.0.1:6879 (90cb860b...) -> 20304 keys | 5461 slots | 1 slaves.

127.0.0.1:6881 (fa2acce2...) -> 20201 keys | 5461 slots | 1 slaves.

127.0.0.1:6880 (3f121a67...) -> 20174 keys | 5462 slots | 1 slaves.

[OK] 60679 keys in 3 masters.

3.70 keys per slot on average.

若加入1个节点实现集群扩容时,要通过相关命令把一部分槽和数据迁移给新节点,按照3个过程进行。

1. 准备新节点

准备6885和6886两个新端口(节点),运行在集群模式下。

2. 加入集群

127.0.0.1:6879> cluster meet 127.0.0.1 6885

127.0.0.1:6879> cluster meet 127.0.0.1 6886

127.0.0.1:6879> cluster nodes

e090ec47b8e66e415d69e9452c9c8e7deccd3624 127.0.0.1:6886 master - 0 1532846452277 0 connected

79c8cf3c11ede4962a2d690c2a2545b86c2f56ed 127.0.0.1:6885 master - 0 1532846453289 0 connected

...

90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879 myself,master - 0 0 1 connected 0-5460

新节点刚开始都是主节点状态,由于没有负责的槽,不能接受读写操作。对于新节点的后续操作一般有两种选择,一个是为它迁移槽和数据实现扩容,一个是作为其它主节点的从节点负责故障转移。

在正式环境建议使用redis-trib.rb add-node命令加入新节点,该命令会对新节点是否包含数据或已经加入其它集群进行检查。

$ redis-trib.rb add-node 127.0.0.1:6885 127.0.0.1:6879

>>> Adding node 127.0.0.1:6885 to cluster 127.0.0.1:6879

...

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

>>> Send CLUSTER MEET to node 127.0.0.1:6885 to make it join the cluster.

[OK] New node added correctly.

$ redis-trib.rb add-node 127.0.0.1:6886 127.0.0.1:6879

>>> Adding node 127.0.0.1:6886 to cluster 127.0.0.1:6879

...

M: 99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885

slots: (0 slots) master

0 additional replica(s)

...

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

>>> Send CLUSTER MEET to node 127.0.0.1:6886 to make it join the cluster.

[OK] New node added correctly.

3. 迁移槽和数据

新节点加入集群后,需为其迁移槽和相关数据,迁移过程是集群扩容最核心的环节,按照3个步骤进行。

(1)槽迁移计划

加入6885节点后,原有节点负责的槽数量从6380变为4096个。

(2)迁移数据

数据迁移过程是逐个槽进行的,每个槽数据迁移的流程如下:

1)对目标节点发送cluster setslot {slot} importing {sourceNodeId}命令,让目标节点准备导入槽的数据。

2)对源节点发送cluster setslot {slot} migrating {targetNodeId}命令,让源节点准备迁出槽的数据。

3)源节点循环执行cluster getkeysinslot {slot} {count}命令,获取count个属于槽{slot}的键。

4)在源节点上执行migrate {targetIp} {targetPort} "" 0 {timeout} keys {keys...}命令,把获取的键通过Pipeline机制批量迁移到目标节点。

5)重复执行步骤3)和4),直到槽下所有的键值数据迁移到目标节点。

6)向集群内所有主节点发送cluster setslot {slot} node {targetNodeId}命令,通知槽分配给目标节点。

根据上面流程,手动使用命令把源节点6879负责的槽4096迁移到目标节点6885中。

1)目标节点准备导入槽4096数据。

127.0.0.1:6885> cluster setslot 4096 importing 90cb860b7f4ff516304c577bc1e514dc95ecd09b

确认槽4096导入状态开启。

127.0.0.1:6885> cluster nodes

99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885 myself,master - 0 0 0 connected [4096-<-90cb860b7f4ff516304c577bc1e514dc95ecd09b]

...

90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879 master - 0 1532849864072 1 connected 0-5460

2)源节点准备导出槽4096数据。

127.0.0.1:6879> cluster setslot 4096 migrating 99ea0df1d9683affb1271a5092fc8b15b378adba

确认槽4096导出状态开启。

127.0.0.1:6879> cluster nodes

99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885 master - 0 1532850180009 0 connected

...

90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879 myself,master - 0 0 1 connected 0-5460 [4096->-99ea0df1d9683affb1271a5092fc8b15b378adba]

127.0.0.1:6879>

3)批量获取槽4096对应的键,这里获取4个处于该槽的键。

127.0.0.1:6879> cluster getkeysinslot 4096 4

1) "key:migrate:test:13752"

2) "key:migrate:test:16020"

3) "key:migrate:test:20791"

4) "key:migrate:test:5512"

确认这4个键存在于源节点,不在目标节点上。

127.0.0.1:6879> mget key:migrate:test:13752 key:migrate:test:16020 key:migrate:test:20791 key:migrate:test:5512

1) "13752"

2) "16020"

3) "20791"

4) "5512"

$ redis-cli -p 6885 -c

127.0.0.1:6885> mget key:migrate:test:13752 key:migrate:test:16020 key:migrate:test:20791 key:migrate:test:5512

-> Redirected to slot [4096] located at 127.0.0.1:6879

1) "13752"

2) "16020"

3) "20791"

4) "5512"

批量迁移这4个键。

127.0.0.1:6879> migrate 127.0.0.1 6885 "" 0 5000 keys key:migrate:test:13752 key:migrate:test:16020 key:migrate:test:20791 key:migrate:test:5512

再次查看这4个键,已不再源节点。

127.0.0.1:6879> mget key:migrate:test:13752 key:migrate:test:16020 key:migrate:test:20791 key:migrate:test:5512

(error) ASK 4096 127.0.0.1:6885

通知所有主节点槽4096指派给目标节点6885。

127.0.0.1:6879> cluster setslot 4096 node 99ea0df1d9683affb1271a5092fc8b15b378adba

127.0.0.1:6880> cluster setslot 4096 node 99ea0df1d9683affb1271a5092fc8b15b378adba

127.0.0.1:6881> cluster setslot 4096 node 99ea0df1d9683affb1271a5092fc8b15b378adba

127.0.0.1:6885> cluster setslot 4096 node 99ea0df1d9683affb1271a5092fc8b15b378adba

确认源节点6879不再负责槽4096,改为目标节点6885负责。

127.0.0.1:6879> cluster nodes

99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885 master - 0 1532851434584 9 connected 4096

...

90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879 myself,master - 0 0 1 connected 0-4095 4097-5460

实际迁移过程中会涉及大量槽,每个槽会有非常多的键,因此redis-trib.rb reshard提供了槽重分片功能,reshard命令简化了槽迁移的过程,剩下槽迁移使用redis-trib.rb完成。

$ redis-trib.rb reshard 127.0.0.1:6879

...

M: 99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885

slots:4096 (1 slots) master

0 additional replica(s)

M: 558b0fb8d44933e694b46c15d05e595ce5ae4fab 127.0.0.1:6886

slots: (0 slots) master

0 additional replica(s)

...

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

How many slots do you want to move (from 1 to 16384)? 4096

What is the receiving node ID? 99ea0df1d9683affb1271a5092fc8b15b378adba

Please enter all the source node IDs.

Type 'all' to use all the nodes as source nodes for the hash slots.

Type 'done' once you entered all the source nodes IDs.

Source node #1:90cb860b7f4ff516304c577bc1e514dc95ecd09b

Source node #2:3f121a67fab0d74f0d31b69326259e687902e1b3

Source node #3:fa2acce219d088e2b33756dac2e85ca92936a8dd

Source node #4:done

Ready to move 4096 slots.

Source nodes:

M: 90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879

slots:0-4095,4097-5460 (5460 slots) master

1 additional replica(s)

M: 3f121a67fab0d74f0d31b69326259e687902e1b3 127.0.0.1:6880

slots:5461-10922 (5462 slots) master

1 additional replica(s)

M: fa2acce219d088e2b33756dac2e85ca92936a8dd 127.0.0.1:6881

slots:10923-16383 (5461 slots) master

1 additional replica(s)

Destination node:

M: 99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885

slots:4096 (1 slots) master

0 additional replica(s)

Resharding plan:

Moving slot 5461 from 3f121a67fab0d74f0d31b69326259e687902e1b3

...

Moving slot 11090 from fa2acce219d088e2b33756dac2e85ca92936a8dd

...

Moving slot 1364 from 90cb860b7f4ff516304c577bc1e514dc95ecd09b

Do you want to proceed with the proposed reshard plan (yes/no)? yes

Moving slot 5461 from 127.0.0.1:6880 to 127.0.0.1:6885: ..

...

Moving slot 12177 from 127.0.0.1:6881 to 127.0.0.1:6885: ....

...

Moving slot 1364 from 127.0.0.1:6879 to 127.0.0.1:6885: .....

查看节点和槽新的映射关系。

127.0.0.1:6879> cluster nodes

99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885 master - 0 1532852546583 9 connected 0-1364 4096 5461-6826 10923-12287

558b0fb8d44933e694b46c15d05e595ce5ae4fab 127.0.0.1:6886 master - 0 1532852550630 8 connected

fa2acce219d088e2b33756dac2e85ca92936a8dd 127.0.0.1:6881 master - 0 1532852548097 3 connected 12288-16383

3f121a67fab0d74f0d31b69326259e687902e1b3 127.0.0.1:6880 master - 0 1532852549619 2 connected 6827-10922

...

90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879 myself,master - 0 0 1 connected 1365-4095 4097-5460

迁移后使用redis-trib.rb rebalance命令检查节点间槽的均衡性。

$ redis-trib.rb rebalance 127.0.0.1:6879

...

[OK] All 16384 slots covered.

*** No rebalancing needed! All nodes are within the 2.0% threshold.

(3)添加从节点

把节点6886作为6885的从节点,保证整个集群的高可用。

127.0.0.1:6886> cluster replicate 99ea0df1d9683affb1271a5092fc8b15b378adba

查看节点6886状态已成为6885的从节点,至此扩容完成。

127.0.0.1:6886> cluster nodes

99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885 master - 0 1532852938340 9 connected 0-1364 4096 5461-6826 10923-12287

558b0fb8d44933e694b46c15d05e595ce5ae4fab 127.0.0.1:6886 myself,slave 99ea0df1d9683affb1271a5092fc8b15b378adba 0 0 8 connected

...

收缩集群

收缩集群意味着从现有集群中安全下线部分节点。首先要确定下线节点是否有负责的槽,若有,需把槽迁移到其它节点,保证节点下线后整个集群槽和节点映射的完整性。当下线节点不再负责槽或本身是从节点时,就可以通知集群内其它节点忘记下线节点,当所有节点忘记该节点后可以正常关闭。下面按照2个过程进行。

(1)下线迁移槽

下线节点要把自己负责的槽迁移到其它节点,原理和节点扩容槽迁移过程一致。如把6881和6884节点下线,6881是主节点,负责槽(12288-16383),6884是它的从节点。下线6881节点之前,要把它负责的槽迁移到6879,6880和6885这3个节点。由于每次执行reshard命令只能有一个目标节点,因此要执行3次reshard命令,分别迁移1365,1365和1366个槽。

$ redis-trib.rb reshard 127.0.0.1:6879

...

How many slots do you want to move (from 1 to 16384)? 1365

What is the receiving node ID? 90cb860b7f4ff516304c577bc1e514dc95ecd09b

Please enter all the source node IDs.

Type 'all' to use all the nodes as source nodes for the hash slots.

Type 'done' once you entered all the source nodes IDs.

Source node #1:fa2acce219d088e2b33756dac2e85ca92936a8dd

Source node #2:done

Ready to move 1365 slots.

Source nodes:

M: fa2acce219d088e2b33756dac2e85ca92936a8dd 127.0.0.1:6881

slots:12288-16383 (4096 slots) master

1 additional replica(s)

Destination node:

M: 90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879

slots:1365-4095,4097-5460 (4095 slots) master

1 additional replica(s)

Resharding plan:

Moving slot 12288 from fa2acce219d088e2b33756dac2e85ca92936a8dd

Moving slot 12289 from fa2acce219d088e2b33756dac2e85ca92936a8dd

Do you want to proceed with the proposed reshard plan (yes/no)? yes

...

Moving slot 13651 from 127.0.0.1:6881 to 127.0.0.1:6879: ..

Moving slot 13652 from 127.0.0.1:6881 to 127.0.0.1:6879: ...

槽迁移完成后,6879节点接管了6881节点的1365个槽12288-13652。

继续把1365个,和1366个槽迁移到6880节点,和6885节点。

$ redis-trib.rb reshard 127.0.0.1:6879

...

How many slots do you want to move (from 1 to 16384)? 1365

What is the receiving node ID? 3f121a67fab0d74f0d31b69326259e687902e1b3

Please enter all the source node IDs.

Type 'all' to use all the nodes as source nodes for the hash slots.

Type 'done' once you entered all the source nodes IDs.

Source node #1:fa2acce219d088e2b33756dac2e85ca92936a8dd

Source node #2:done

...

Do you want to proceed with the proposed reshard plan (yes/no)? yes

...

$ redis-trib.rb reshard 127.0.0.1:6879

...

How many slots do you want to move (from 1 to 16384)? 1366

What is the receiving node ID? 99ea0df1d9683affb1271a5092fc8b15b378adba

Please enter all the source node IDs.

Type 'all' to use all the nodes as source nodes for the hash slots.

Type 'done' once you entered all the source nodes IDs.

Source node #1:fa2acce219d088e2b33756dac2e85ca92936a8dd

Source node #2:done

...

Do you want to proceed with the proposed reshard plan (yes/no)? yes

...

到此为止,6881节点所有的槽全部迁出完成,集群状态如下:

127.0.0.1:6885> cluster nodes

99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885 myself,master - 0 0 12 connected 0-1364 4096 5461-6826 10923-12287 15018-16383

...

fa2acce219d088e2b33756dac2e85ca92936a8dd 127.0.0.1:6881 master - 0 1532862908561 3 connected

904db05d81825413702f7eac960cd2f656b217f7 127.0.0.1:6884 slave 99ea0df1d9683affb1271a5092fc8b15b378adba 0 1532862911596 12 connected

90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879 master - 0 1532862913617 10 connected 1365-4095 4097-5460 12288-13652

3f121a67fab0d74f0d31b69326259e687902e1b3 127.0.0.1:6880 master - 0 1532862907549 11 connected 6827-10922 13653-15017

(2)忘记节点

当下线主节点具有从节点时,需要把该从节点指向到其它主节点。对于主从节点都下线的情况,要先下线从节点再下线主节点,防止不必要的切换。对于6881和6884节点下线操作,命令如下:

$ redis-trib.rb del-node 127.0.0.1:6879 904db05d81825413702f7eac960cd2f656b217f7

>>> Removing node 904db05d81825413702f7eac960cd2f656b217f7 from cluster 127.0.0.1:6879

>>> Sending CLUSTER FORGET messages to the cluster...

>>> SHUTDOWN the node.

$ redis-trib.rb del-node 127.0.0.1:6879 fa2acce219d088e2b33756dac2e85ca92936a8dd

...

节点下线后,集群最终的状态。

127.0.0.1:6885> cluster nodes

99ea0df1d9683affb1271a5092fc8b15b378adba 127.0.0.1:6885 myself,master - 0 0 12 connected 0-1364 4096 5461-6826 10923-12287 15018-16383

558b0fb8d44933e694b46c15d05e595ce5ae4fab 127.0.0.1:6886 slave 99ea0df1d9683affb1271a5092fc8b15b378adba 0 1532863551984 12 connected

90cb860b7f4ff516304c577bc1e514dc95ecd09b 127.0.0.1:6879 master - 0 1532863552997 10 connected 1365-4095 4097-5460 12288-13652

c88a8bbe719e337e9015aa84aab40db06878b728 127.0.0.1:6882 slave 90cb860b7f4ff516304c577bc1e514dc95ecd09b 0 1532863555528 10 connected

3f121a67fab0d74f0d31b69326259e687902e1b3 127.0.0.1:6880 master - 0 1532863556034 11 connected 6827-10922 13653-15017

200da7d61d40c384a3e55b74434bf229333a5fe8 127.0.0.1:6883 slave 3f121a67fab0d74f0d31b69326259e687902e1b3 0 1532863555023 11 connected

若感兴趣可关注订阅号”数据库最佳实践”(DBBestPractice).

原文地址:http://blog.51cto.com/coveringindex/2151871

时间: 2024-10-08 03:18:02

Redis的集群(伸缩)的相关文章

Redis的集群(故障转移)

Redis集群自身实现了高可用,当集群内少量节点出现故障时通过自动故障转移保证集群可以正常对外提供服务. 故障发现 1. 主观下线 当cluster-node-timeout时间内某节点无法与另一个节点顺利完成ping消息通信时,则将该节点标记为主观下线状态. 2. 客观下线 当某个节点判断另一个节点主观下线后,该节点的下线报告会通过Gossip消息传播.当接收节点发现消息体中含有主观下线的节点,其会尝试对该节点进行客观下线,依据下线报告是否在有效期内(如果在cluster-node-timeo

redis搭建集群并用TreeSoft管理

前言:redis作为一款高效的NOSQL数据库已经深入贯彻和落实到我们的日常开发代码中,作为缓存.时间控制.数据仓库.队列等使用方法层出不穷,简直是开写代码.居家旅行之必备良药.曾经,我们的项目都是单体的,直到后来逐渐演变为微服务-- 一个将我们的工程解耦成多个工程的体系.然后随着我们项目的访问量越来越高,后台的吞吐量也越来越大.如果我们还采用的单体redis,性能很容易形成瓶颈.如何突破单体redis带来的功能受限?如何突破性能带来的问题?这时我们就可以考虑横向扩展,搭建redis集群.正所谓

10.Redis分布式集群

10.Redis分布式集群10.1 数据分布10.1.1 数据分布理论10.1.2 Redis数据分区10.1.3 集群功能限制10.2 搭建集群10.2.1 准备节点10.2.2 节点握手10.2.3 分配槽10.2.4 用redis-trib.rb搭建集群10.3 节点通信10.3.1 通信流程10.3.2 Gossip消息10.3.3 节点选择10.4 集群伸缩10.4.1 伸缩原理10.4.2 扩容集群10.4.3 收缩集群10.5 请求路由10.5.1 请求重定向10.5.2 Smar

(五)Redis Cluster 集群

Redis Cluster 需求:1.请求量过大 100w/s 2.数据量大 1.数据分布 哈希分布特点: 数据分散度高 键值分布业务无关 无法顺序访问 支持批量操作 顺序分布特点: 数据分散度易倾斜 键值业务相关 可顺序访问 支持批量操作 2.数据分区 节点取余分区特点: 客户端分片:哈希 + 取余 节点伸缩数据迁移率过大 节点伸缩采用翻倍扩展 一致性哈希特点: 使用token环,顺时针定位节点 节点伸缩时影响临近节点,存在数据迁移 翻倍伸缩,保证最小迁移数据和负债均衡 虚拟哈希分区特点: 预

Redis Cluster 集群数据分片机制

复制粘贴自: https://www.e-learn.cn/content/redis/2344485, 点击链接访问原文 仅供个人学习参考之用, 高级开发不得不懂的Redis Cluster数据分片机制 Redis 集群简介 Redis Cluster 是 Redis 的分布式解决方案,在 3.0 版本正式推出,有效地解决了 Redis 分布式方面的需求. Redis Cluster 一般由多个节点组成,节点数量至少为 6 个才能保证组成完整高可用的集群,其中三个为主节点,三个为从节点.三个主

Redis Cluster集群部署搭建

在Oracle的路上走了许多年,换换感觉,尝试一下新的知识,也是一个不错的感觉.Redis,一个超轻量化的内存数据库,只做一小块数据库功能实现,却非常优秀的一个产品.今天,就分享一下安装Redis集群的过程. 搭建redis集群,建议至少需要准备3台服务器,共搭建6个节点,3个master,3个slave,并且要求3个master节点不能全部跑到同一台服务器上,保证节点安全,3台服务器的配置相同,使用redistest账号搭建,对应的端口是7000/7001/7002端口 我的集群分配如下,每个

java:redis(redis的集群配置)

服务器集群作用: 服务器集群就是指将很多服务器集中起来一起进行同一种服务,在客户端看来就象是只有一个服务器 集群可以利用多个计算机进行并行计算从而获得很高的计算速度,也可以用多个计算机做备份,从而使得任何一个机器坏了整个系统还是能正常运行.一旦在服务器上安装并运行了群集服务,该服务器即可加入群集.群集化操作可以减少单点故障数量,并且实现了群集化资源的高可用性. redis的集群配置: (.编辑network文件 HOSTNAME=redis(自己定义的hostname) vi /etc/sysc

redis cluster 集群重启关闭

找遍了redis cluster官方文档,没发现有关集群重启和关闭的方法.为啥会没有呢,猜测redis cluster至少要三个节点才能运行,三台同时挂掉的可能性比较小,只要不同时挂掉,挂掉的机器修复后在加入集群,集群都能良好的运作,万一同时挂掉,数据又没有备份的话,就有大麻烦了. redis cluster集群中的节点基本上都对等的,没有管理节点.如果要让所有节点都关闭,只能关闭进程了# pkill -9 redis 把所有集群都关闭,然后在重新启动,会报以下错误 # redis-trib.r

JFinal redis cluster集群插件

JFinal 框架到了2.1版本号,可是依旧仅仅支持redis的主从集群,没有看到Cluster集群的插件.笔者照着主从的插件方式,改了改,实现了个简单的插件,先使用起来,兴许会更新完好版本号. 插件地址:点击打开链接 附上源代码: package com.sxt.jfinal.rediscluster; import java.util.Set; import org.apache.commons.pool2.impl.GenericObjectPoolConfig; import redis