ZooKeeper集群高可用性测试与实践：从规划到故障模拟

#作者：任少近

ZooKeeper集群高可用性测试的主要目的是确保在分布式环境中，ZooKeeper服务能够持续提供一致性和高可用性的协调服务。

ZooKeeper集群环境规划

节点ip	ZooKeeper版本	java版本	对外端口	集群通信端口	集群选举端口
192.168.x.x	ZooKeeper-3.6.1	1.8.0_332	2181	2888	3888
192.168.x.y	ZooKeeper-3.6.1	1.8.0_332	2181	2888	3888
192.168.x.z	ZooKeeper-3.6.1	1.8.0_332	2181	2888	3888

注：这里以裸金属部署方式进行测试验证，ZooKeeper集群部署参考3.3.2集群模式章节

1.集群数据一致性测试

ZooKeeper集群数据一致性测试的主要目的是确保在分布式环境中，ZooKeeper能够提供强一致性的数据存储和协调服务。测试确保 ZooKeeper 在不同节点上的数据副本之间保持一致，无论客户端连接到哪个节点，都能够读取到相同的数据。

192.168.x.y节点连接并创建数据

# ./zkCli.sh -server 192.168.x.y:2181
/usr/bin/java
Connecting to 192.168.x.y:2181
[zk: 192.168.x.y:2181(CONNECTED) 0]
[zk: 192.168.x.y:2181(CONNECTED) 0] create /zktest 'test'
Created /zktest
[zk: 192.168.x.y:2181(CONNECTED) 1] ls /zktest
[]
[zk: 192.168.x.y:2181(CONNECTED) 2] ls /
[zktest, ZooKeeper]
[zk: 192.168.x.y:2181(CONNECTED) 3] get /zktest
test
[zk: 192.168.x.y:2181(CONNECTED) 4]

192.168.x.x节点查看数据，然后再进行修改数据

# ./zkCli.sh -server 192.168.x.x:2181
/usr/bin/java
Connecting to 192.168.x.x:2181
[zk: 192.168.x.x:2181(CONNECTED) 0] ls /
[zktest, ZooKeeper]
[zk: 192.168.x.x:2181(CONNECTED) 1] get /zktest
test
[zk: 192.168.x.x:2181(CONNECTED) 2] set /zktest 'test2'
[zk: 192.168.x.x:2181(CONNECTED) 3] get /zktest
test2
[zk: 192.168.x.x:2181(CONNECTED) 4]

192.168.x.z节点查看数据

# ./zkCli.sh -server 192.168.x.z:2181
/usr/bin/java
Connecting to 192.168.x.z:2181
[zk: 192.168.x.z:2181(CONNECTED) 0] ls /
[zktest, ZooKeeper]
[zk: 192.168.x.z:2181(CONNECTED) 1] get /zktest
test2
[zk: 192.168.x.z:2181(CONNECTED) 2]

总结：ZooKeeper 集群中的每个节点都维护着相同的数据副本，称为 ZNode。无论在哪个节点上执行数据操作（添加、删除、修改），这些更改都会自动传播到其他节点。这种自动同步确保了数据在整个集群中的一致性。客户端无需担心选择特定节点来进行写操作，因为数据会自动同步到其他节点，确保所有节点都具有相同的数据副本。

2.集群节点故障测试

验证高可用性：测试节点故障时，ZooKeeper 集群是否能够继续提供协调和一致性服务，以确保系统的高可用性。节点故障测试有助于评估集群在节点损坏时的表现和恢复能力。
测试自动故障检测和恢复：验证集群是否能够自动检测故障节点，并将工作负载迁移到健康节点，以维持服务的连续性。这有助于确保集群对节点故障具有适当的故障恢复机制。

192.168.x.x上查看ZooKeeper状态

# /usr/local/ZooKeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

192.168.x.y上查看ZooKeeper状态

# /usr/local/ZooKeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

192.168.x.y上查看ZooKeeper状态

# /usr/local/ZooKeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

注：通过查看ZooKeeper集群状态得知，ZooKeeper集群leader角色在192.168.x.y节点上。

在具有leader角色服务节点192.168.x.y上，停止ZooKeeper服务

# /usr/local/ZooKeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

# /usr/local/ZooKeeper/bin/zkServer.sh stop
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Stopping ZooKeeper ... STOPPED

查看ZooKeeper集群状态得知，leader角色已转移到192.168.x.z

# /usr/local/ZooKeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

192.168.x.z节点修改数据

# ./zkCli.sh -server 192.168.x.z:2181
/usr/bin/java
Connecting to 192.168.x.z:2181
[zk: 192.168.x.z:2181(CONNECTED) 0] ls /
[zktest, ZooKeeper]
[zk: 192.168.x.z:2181(CONNECTED) 1] get /zktest
test2
[zk: 192.168.x.z:2181(CONNECTED) 2] set /zktest 'test3'
[zk: 192.168.x.z:2181(CONNECTED) 3] get /zktest
test3
[zk: 192.168.x.z:2181(CONNECTED) 4] create /test 'hello'
Created /test
[zk: 192.168.x.z:2181(CONNECTED) 5] get /test
hello
[zk: 192.168.x.z:2181(CONNECTED) 6] delete /zktest
[zk: 192.168.x.z:2181(CONNECTED) 7] ls /
[test, ZooKeeper]
[zk: 192.168.x.z:2181(CONNECTED) 8]

192.168.x.y节点服务启动ZooKeeper服务及查看服务状态

# /usr/local/ZooKeeper/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Starting ZooKeeper ... STARTED

# /usr/local/ZooKeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/ZooKeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

192.168.x.y节点上查看数据

# ./zkCli.sh -server 192.168.x.y:2181
/usr/bin/java
Connecting to 192.168.x.y:2181
[zk: 192.168.x.y:2181(CONNECTED) 0] ls /
[test, ZooKeeper]
[zk: 192.168.x.y:2181(CONNECTED) 1] get /test
hello
[zk: 192.168.x.y:2181(CONNECTED) 2]

结论：ZooKeeper 集群的设计旨在确保高可用性和数据的自动同步。即使发生一台节点的故障，整个集群仍然能够提供协调和一致性服务，不会对用户的使用产生重大影响。这种容错性和自动恢复性使 ZooKeeper 成为构建分布式系统中的可靠基础设施之一。