Redis cluster Specification 笔记

　　ref： http://redis.io/topics/cluster-spec

1. 设计目标： 高性能；线性扩展；不支持合并操作；写操作安全：小概率丢弃；（对于每个key）只要有一个slave工作，就可用；

Redis Cluster is a distributed implementation of Redis with the following goals, in order of importance in the design:

• High performance and linear scalability up to 1000 nodes.
• No merge operations in order to play well with values size and semantics typical of the Redis data model.
• Write safety: the system tries to retain all the writes originating from clients connected with the majority of the nodes. However there are small windows where acknowledged writes can be lost.
• Availability: Redis Cluster is able to survive to partitions where the majority of the master nodes are reachable and there is at least a reachable slave for every master node that is no longer reachable.

What is described in this document is implemented in the unstable branch of the Github Redis repository. Redis Cluster has now entered the beta stage, so new betas are released every month and can be found in the download page of the Redis web site.

2. 使用hash tag实现将一个key总是路由到固定的节点；

3. 通信协议：

• 通过一个“cluster bus”的二进制协议通信；
• 每个几点都和其他所有节点建立tcp链接（在这点就不是线性扩展的）；
• client发起链接给任何请求都是允许的，但是节点不做proxy功能，而是像http那样，返回一个重定向的错误信息；

4. 安全写： 存在两个丢数据的可能：

Redis Cluster tries hard to retain all the writes that are performed by clients connected to the majority of masters, with two exceptions:

1) A write may reach a master, but while the master may be able to reply to the client, the write may not be propagated to slaves via the asynchronous replication used between master and slave nodes. If the master dies without the write reaching the slaves, the write is lost forever in case the master is unreachable for a long enough period that one of its slaves is promoted.

2) Another theoretically possible failure mode where writes are lost is the following:

• A master is unreachable because of a partition.
• It gets failed over by one of its slaves.
• After some time it may be reachable again.
• A client with a not updated routing table may write to it before the master is converted to a slave (of the new master) by the cluster.

5 可用性： 当网络分裂时，包含多数server的一侧可以正常使用，另一侧不可以使用； 不适用于大规模网络故障的场景； 对任何一个key，只要有一个master或slave存在，就能正常访问；

6. 性能： （每个节点）与单个redis基本相同（这就是所谓的性能线性增长）；

7. 为什么不支持merge操作： 性能考虑；

8. key的分布：先CRC然后取模分配从16k的分片（slot），然后在分配到各个node上；

HASH_SLOT = CRC16(key) mod 16384

The CRC16 is specified as follows:

• Name: XMODEM (also known as ZMODEM or CRC-16/ACORN)
• Width: 16 bit
• Poly: 1021 (That is actually x¹⁶ + x¹² + x⁵ + 1)
• Initialization: 0000
• Reflect Input byte: False
• Reflect Output CRC: False
• Xor constant to output CRC: 0000
• Output for "123456789": 31C3

9 。 keys hash tag: 在key “{tag}otherString”中，tag就是hash tags，用于计算这个key的slot位置，为了实现先同tag的key映射到相同的slot中。10。 node属性：node的标识是一个随机数，第一次运行时写入到配置文件，并保持不变；

Every node has other associated information that all the other nodes know:

• The IP address and TCP port where the node is located.
• A set of flags.
• A set of hash slots served by the node.
• Last time we sent a ping packet using the cluster bus.
• Last time we received a pong packet in reply.
• The time at which we flagged the node as failing.
• The number of slaves of this node.
• The master node ID, if this node is a slave (or 0000000... if it is a master).

11. Cluster topology 拓扑： 全连接且长tcp链接。

Redis cluster is a full mesh where every node is connected with every other node using a TCP connection.

In a cluster of N nodes, every node has N-1 outgoing TCP connections, and N-1 incoming connections.

These TCP connections are kept alive all the time and are not created on demand.

12. 节点间通信： 新节点加入时，只有管理员才能发起MEET消息；MEET消息会在cluster中传播。

13. 重定向策略

A Redis client is free to send queries to every node in the cluster, including slave nodes. The node will analyze the query, and if it is acceptable (that is, only a single key is mentioned in the query) it will see what node is responsible for the hash slot where the key belongs.

If the hash slot is served by the node, the query is simply processed, otherwise the node will check its internal hash slot -> node ID map and will reply to the client with a MOVED error.

A MOVED error is like the following:

GET x
-MOVED 3999 127.0.0.1:6381

14. key迁移：

The following subcommands are available:

• CLUSTER ADDSLOTS slot1 [slot2] ... [slotN]
• CLUSTER DELSLOTS slot1 [slot2] ... [slotN]
• CLUSTER SETSLOT slot NODE node
• CLUSTER SETSLOT slot MIGRATING node
• CLUSTER SETSLOT slot IMPORTING node

• CLUSTER GETKEYSINSLOT slot count

• MIGRATE target_host target_port key target_database id timeout

15. Ask redirection: 查询一个key的位置

16. Client处理重定向： client应该适当记录key与slot的关系（减少redirect的次数），并且处理redirect的错误信息;　　

17 . 多key。Multiple keys operations

Using hash tags clients are free to use multiple-keys operations. For example the following operation is valid:

MSET {user:1000}.name Angela {user:1000}.surname White

18 容错：

1. 节点间心跳检测：随机发给随机数量的节点，使得整个cluster的总心跳数在N的规模；
2. 心跳报文内容:

The common header has the following information:

• Node ID, that is a 160 bit pseudorandom string that is assigned the first time a node is created and remains the same for all the life of a Redis Cluster node.
• The currentEpoch and configEpoch field, that are used in order to mount the distributed algorithms used by Redis Cluster (this is explained in details in the next sections). If the node is a slave the configEpoch is the last known configEpoch of the master.
• The node flags, indicating if the node is a slave, a master, and other single-bit node information.
• A bitmap of the hash slots served by a given node, or if the node is a slave, a bitmap of the slots served by its master.
• Port: the sender TCP base port (that is, the port used by Redis to accept client commands, add 10000 to this to obtain the cluster port).
• State: the state of the cluster from the point of view of the sender (down or ok).
• The master node ID, if this is a slave.

19 失效节点检测： PFAIL/FAIL标志。当A心跳检测B失败，A标志为B为PFAIL；然后A询问其他的node，如果多数节点返回B为PFAIL，则A标志B为FAIL，并通知其他所有的node B为FAIL。

This mechanism is used in order to escalate a PFAIL condition to a FAIL condition, when the following set of conditions are met:

• Some node, that we‘ll call A, has another node B flagged as PFAIL.
• Node A collected, via gossip sections, information about the state of B from the point of view of the majority of masters in the cluster.
• The majority of masters signaled the PFAIL or PFAIL condition within NODE_TIMEOUT * FAIL_REPORT_VALIDITY_MULT time.

If all the above conditions are true, Node A will:

• Mark the node as FAIL.
• Send a FAIL message to all the reachable nodes.

The FAIL message will force every receiving node to mark the node in FAIL state.

20. 逻辑时钟：Cluster epoch

21. Slave提升为Master： 检测到master失效-》slave发起选举-》赢得选举的slave将自己变为master. 选举过程：

• slave A发送FAILOVER_AUTH_REQUEST给其他所有的master，并等待回应（至少NODE_TIMEOUT*2时长）；
• 其他master收到FAILOVER_AUTH_REQUEST请求后，决定如果同意就回应FAILOVER_AUTH_ACK消息，并且在2*NODE_TIMEOUT时间内不在同意其他请求（和zk的类似）
• slave A 如果收到大多少（超过半数）master的ack回应，则赢得选举，广播自己赢得选举的消息。 然后就可以升为master了。

22. key slot分配与信息传播。

Rule 1: If an hash slot is unassigned, and a known node claims it, I‘ll modify my hash slot table to associate the hash slot to this node.

Rule 2: If an hash slot is already assigned, and a known node is advertising it using a configEpoch that is greater than theconfigEpoch advertised by the current owner of the slot, I‘ll rebind the hash slot to the new node.

22. publish and subscribe: 可以向任何一个节点publish或subscribe，cluter内部会通知到正确的节点；

地方

Redis cluster Specification 笔记