zookeeper学习系列：四、Paxos算法和zookeeper的关系

一、问题起源

淘宝搜索的博客 http://www.searchtb.com/2011/01/zookeeper-research.html  提到Paxos是zookeeper的灵魂

有一篇文章标题更是以“Zookeeper全解析——Paxos作为灵魂” 作为标题，认为是zookeeper的基础：

“

Google的Chubby，Apache的Zookeeper都是基于它的理论来实现的，Paxos还被认为是到目前为止唯一的分布式一致性算法，其它的算法都是Paxos的改进或简化。有个问题要提一下，Paxos有一个前提：没有拜占庭将军问题。就是说Paxos只有在一个可信的计算环境中才能成立，这个环境是不会被入侵所破坏的。

”

Paxos 这个算法是Leslie Lamport在1990年提出的一种基于消息传递的一致性算法.Paxos 算法解决的问题是一个分布式系统如何就某个值（决议）达成一致。

part-time parliament

Paxos Made Simple里这样描述Paxos算法执行过程：

1. prepare 阶段：
   1. proposer 选择一个提案编号 n 并将 prepare 请求发送给 acceptors 中的一个多数派；
   2. acceptor 收到 prepare 消息后，如果提案的编号大于它已经回复的所有 prepare 消息，则 acceptor 将自己上次接受的提案回复给 proposer，并承诺不再回复小于 n 的提案；
2. 批准阶段：
   1. 当一个 proposer 收到了多数 acceptors 对 prepare 的回复后，就进入批准阶段。它要向回复 prepare 请求的 acceptors 发送 accept 请求，包括编号 n 和根据 P2c 决定的 value（如果根据 P2c 没有已经接受的 value，那么它可以自由决定 value）。
   2. 在不违背自己向其他 proposer 的承诺的前提下，acceptor 收到 accept 请求后即接受这个请求。

wiki上是两个阶段，论文里却是说三阶段，而且默认就有了个proposer相当于leader。查资料有大侠列出了第三个阶段(http://www.wuzesheng.com/?p=2724)：

3. Learn阶段:

• 当各个Acceptor达到一致之后，需要将达到一致的结果通知给所有的Learner.

zookeeper采用org.apache.zookeeper.server.quorum.FastLeaderElection作为其缺省选举算法

这篇文章http://csrd.aliapp.com/?p=162&replytocom=782 直接用paxos实现作为标题，提到  zookeeper在选举leader的时候采用了paxos算法(主要是fast paxos)

偶然看到下边有人反驳：

魏讲文：“

FastLeaderElection根本不是Paxos，也不是Fast Paxos的实现。
FastLeaderElection源码与Paxos的论文相去甚远。

Paxos与 FastPaxos算法中也有一个leader选举的问题。

FastLeaderElection对于zookeeper来讲，只是相当于Paxos中的leader选举。

”

二、资料证实

好的，查查资料，分析源码开始调研

首先是魏讲文的反驳：

There is a very common misunderstanding that the leader election algorithm in zookeeper is paxos or fast paxos. The leader election algorithm is not paxos or fast paxos, please consider the following facts:

1. There is no the concept of proposal number in the leader election in zookeeper. the proposal number is a key concept to paxos. Some one think the epoch is the proposal number, but different followers may produce proposal with the same epoch which is not allowed in paxos.
2. Fast paxos requires at least 3t + 1 acceptors, where t is the number of servers which are allowed to fail [3]. This is conflict with the fact that a zookeeper cluster with 3 servers works well even if one server failed.
3. The leader election algorithm must make sure P1. However paxos does provide such guarantee.
4. In paxos, a leader is also required to achieve progress. There are some similarities between the leader in paxos and the leader in zookeeper. Even if more than one servers believe they are the leader, the consistency is preserved both in zookeeper and in paxos. this is very clearly discussed in [1] and [2].

然后是作者三次对比

1）邮件列表

Our protocol instead, has only two phases, just like a two-phase
commit protocol. Of course, for Paxos, we can ignore the first phase in runs in
which we have a single proposer as we can run phase 1 for multiple instances at
a time, which is what Ben called previously Multi-Paxos, I believe. The trick
with skipping phase 1 is to deal with leader switching. 

2）出书的访谈

We made a few interesting observations about Paxos when contrasting it to Zab, like problems you could run into if you just implemented Paxos alone. Not that Paxos is broken or anything, just that in our setting, there were some properties it was not giving us. Some people still like to map Zab to Paxos, and they are not completely off, but the way we see it, Zab matches a service like ZooKeeper well.

zk的分布式一致性算法有了个名称叫Zab

3）论文

We use an algorithm that shares some of the character- istics of Paxos, but that combines transaction logging needed for consensus with write-ahead logging needed for data tree recovery to enable an efficient implementa- tion.

三、分析

在我理解首先在选举时，并不能用到paxos算法，paxos里选总统也好，zk选leader也好，跟搞个提案让大部分人同意是有区别的。选主才能保证不会出现多proposer的并发提案冲突

谁去作为proposer发提案？是paxos算法进行下去的前提。而提出提案让大部分follower同意则可用到类似paxos的算法实现一致性。zookeeper使用的是Zab算法实现一致性。

zk的选主策略：

there can be at most one leader (proposer) at any time, and we guarantee this by making sure
that a quorum of replicas recognize the leader as a leader by committing to an
epoch change. This change in epoch also allows us to get unique zxids since the
epoch forms part of the zxid. 

每个server有一个id，收到提交的事务时则有一个zxid，随更新数据的变动，事务编号递增，server id各不同。首先选zxid最大的作为leader，如果zxid比较不出来，则选server id最大的为leader

再看代码：

四、zookeeper数据一致性原理分析

了解完选主的做法后，来了解下同步数据的做法，同步数据则采用Zab协议：Zookeeper Atomic broadcast protocol，是个简单易懂的两阶段提交：

1. The leader sends a PROPOSAL message, p, to all followers.
2. Upon receiving p, a follower responds to the leader with an ACK, informing the

   leader that it has accepted the proposal.

3. Uponreceivingacknowledgmentsfromaquorum(thequorumincludestheleader

   itself), the leader sends a message informing the followers to COMMIT it.

跟paxos的区别是leaer发送给所有follower，而不是大多数，所有follower都要确认并通知leader，而不是大多数。

保证机制：按顺序广播的两个事务， T 和 T? ，T在前则T? 生效前必须提交T。如果有一个server 提交了T 和 T? ，则所有其他server必须也在T?前提交T。

[1] Reed, B., & Junqueira, F. P. (2008). A simple totally ordered broadcast protocol. Second
Workshop on Large-Scale Distributed Systems and Middleware (LADIS 2008). Yorktown
Heights, NY: ACM. ISBN: 978-1-60558-296-2.
[2] Lamport, L. Paxos made simple. ACM SIGACT News 32, 4 (Dec. 2001), 1825.
[3] F. Junqueira, Y. Mao, and K. Marzullo. Classic paxos vs. fast paxos: caveat emptor. In
Proceedings of the 3rd USENIX/IEEE/IFIP Workshop on Hot Topics in System Dependability
(HotDep.07). Citeseer, 2007.