Zookeeper Quick Start

Posted on | Edited on | In

The Vagrant demo file please see my git repo Infratree

Introduction

Zookeeper quorum architecture, at least 3 nodes cluster setup via Vagrant for Kafka use. Hght level information about zk:

  • distributed key value store
  • has voting mechanism
  • used by many big data tools

Zookeeper role in Kafka:

  • broker registration, heart-beating check
  • maintaining a list of topics alongside
  • leader election
  • store kafka cluster id
  • store ACLs if security is enabled
  • quotas config if enabled

使用kafka自带的zookeeper 还是 独立的zookeeper呢？见这个回答参考. 工作项目中还是用的独立的zk. Zookeeper is going to be removed from kafka, see this article.

More references please see Zookeeper main page. System Requirements for version 3.6.2: ZooKeeper (also Kafka) runs in Java, release 1.8 or greater (JDK 8 LTS, JDK 11 LTS, JDK 12 - Java 9 and 10 are not supported).

Archive download link. For example, here uses Zookeeper version 3.6.2.

Performance Factors

Latency is key for zookeeper:

  • fast disk
  • no RAM swap
  • separate disk for snapshots and logs
  • high performance network
  • resonable number of zk servers
  • isolation zk process from others

Cluster Setup

Architecture

Clustered (Multi-Server) Setup. 这里面说了很多注意事项, 比如Java heap size to avoid swapping or disable swapping.

The architecture diagram of this experiment, co-locate zk and kafka in the same node, this is not recommended on production.

1
2
3
4
5
6
7
8
9
10
11
12
// zk leader can be anyone of them

  192.168.20.20        192.168.20.21         192.168.20.22
|--------------|    |-----------------|    |----------------|
| zk server1<--|----|--> zk server2<--|----|-->zk server3   | 
|  follower    |    |     leader      |    |    follower    |
|              |    |     /  |  \     |    |                |
|              |    |    /   |   \    |    |                |
|           /--|----|---/    |    \---|----|--\             |
|  kafka   /   |    |    kafka        |    |   \kafka       |
|  broker1     |    |    broker2      |    |    broker3     |
|--------------|    |-----------------|    |----------------|

Config

Generate zoo.cfg file in each node:

1
2
3
4
5
6
7
# download release binary version 3.6.2
cd /root
wget https://archive.apache.org/dist/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz
tar -zxf apache-zookeeper-3.6.2-bin.tar.gz

cd /root/apache-zookeeper-3.6.2-bin/conf
cp zoo_sample.cfg zoo.cfg

Create zk data, log and conf directories in each zookeeper node:

1
2
3
mkdir -p /root/zk/data
mkdir -p /root/zk/log
mkdir -p /root/zk/conf

Edit zoo.cfg file for each zookeeper instance, use the same configuration for them:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# the basic time unit in milliseonds used by zk
tickTime=2000
#Leader-Follower初始通信时限 tickTime * 10 = 20 seconds
initLimit=10
#Leader-Follower同步通信时限 tickTime * 5 = 10 seconds
syncLimit=5

# data dir
dataDir=/root/zk/data
# log dir
dataLogDir=/root/zk/log

# client port, 3 nodes use the same port number
clientPort=2181
#maxClientCnxns=60

# broker id and IP address, or using hostname from /etc/hosts
# for cluster, borker id must start from 1, not 0

# 2888: connect the individual follower nodes to the leader node
# 3888: used for leader election in the ensemble
# can by any port number
server.1=192.168.20.20:2888:3888
server.2=192.168.20.21:2888:3888
server.3=192.168.20.22:2888:3888

In zk data directory, create myid for each zk instance, much be unqiue:

1
2
3
4
5
6
# current in 192.168.20.20
echo 1 > /root/zk/data/myid
# run on 192.168.20.21
echo 2 > /root/zk/data/myid
# run on 192.168.20.22
echo 3 > /root/zk/data/myid

Commands

Run zk service commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
cd /root/apache-zookeeper-3.6.2-bin/bin
# start
# default uses zoo.cfg config file
# run in background by default, has flag to run in foreground
./zkServer.sh start [path/to/zoo.cfg]
# stop
./zkServer.sh stop [path/to/zoo.cfg]

# status check
# Mode: standalone, follower, leader
./zkServer.sh status [path/to/zoo.cfg]

# test zk is good
# ipv6 + port
./zkCli.sh -server :2181
# then run
# you will see some output like [zookeeper], etc
ls /

[x] why port 2181 is bound on ipv6? 其他环境上也是如此。

After start zookeeper, check ./zkServer.sh status to see if current node is leader or follower. Run ./zkCli.sh, you can execute zk CLI, see this reference.

./zkCli.sh 也可用于从外部连接一个zk server, 只要指定accessable IP 和 port即可，demo中由于就在本机，所以其实是localhost.

If start failed, see <zk package>/logs/zookeeper_audit.log, If you don’t start another 2 zk instance, you will see periodically exception errors in <zk package>/logs/zookeeper-root-server-kafka1.out, it will be resolved after you start all of them. The log setting is by log4j configuration from <zk package>/conf/log4j.properties.

After the cluster is up and running, you can check the ports:

1
2
3
4
5
6
7
8
9
10
# see 2181, 2888, 3888
# -i: network socket
# -P: show port number
# -n: shpw ip address
lsof -i -P -n

# scan 2181
# -z: scan only
# -v: verbose
nc -zv <zk instance ip> 2181

在旧版本中，可以使用four letter words去检查一些状态，比如:

1
2
# are you ok
echo ruok | nc locahost 2181

新版本已经切换到AdminServer. The AdminServer is enabled by default, access by http 8080 port:

1
2
3
4
5
6
7
8
# list all commands
http://192.168.20.20:8080/commands

# for example:
# state
http://192.168.20.20:8080/commands/stat
# config
http://192.168.20.20:8080/commands/configuration

ZooNavigator is a web based ZooKeeper UI and editor/browser with many features. You can launched it by docker container.

Run as Daemon

Set zookeeper as system daemon so that it will be launched every time on system boots.

systemd zookeeper cluster setup on ubuntu, see here.

Basically speaking, first generate a zookeeper service file, for example zookeeper.service, the prefix zookeeper is the service name used in systemctl command. Place this file in /etc/systemd/system folder and owned by root.

Double curly brackets is placeholder in jinja2 template.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
[Unit]
Description=Zookeeper Daemon
Documentation=http://zookeeper.apache.org
# boot after these services
# for example, here we rely consul and dnsmasq services
After=network-online.target consul.service dnsmasq.service
StartLimitInterval=200
StartLimitBurst=5

[Service]
# in zkServer.sh, it uses `&` to fork and exec
Type=forking
# systemd can identify the main process of the daemon
# in zkServer.sh it will echo pid to this file
PIDFile={{ zookeeper_data_dir }}/zookeeper-server.pid

User={{ zookeeper_user }}
Group={{ zookeeper_group }}
WorkingDirectory=/opt/zookeeper

# environment variables may be needed
# or configure in zoo.cfg file
Environment=ZOOPIDFILE={{ zookeeper_data_dir }}/zookeeper-server.pid
Environment=ZOOKEEPER_HOME=/opt/zookeeper
Environment=ZOOKEEPER_CONF={{ zookeeper_conf_dir }}
Environment=ZOOCFGDIR={{ zookeeper_conf_dir }}
Environment=CLASSPATH=$CLASSPATH:$ZOOKEEPER_CONF:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
Environment=ZOO_LOG_DIR={{ zookeeper_log_dir }}
Environment=ZOO_LOG4J_PROP=INFO,ROLLINGFILE
Environment=JVMFLAGS=-Dzookeeper.log.threshold=INFO
Environment=ZOO_DATADIR_AUTOCREATE_DISABLE=true

# start, stop and reload commands
ExecStart=/opt/zookeeper/bin/zkServer.sh start {{ zookeeper_conf_dir }}/zoo.cfg
ExecStop=/opt/zookeeper/bin/zkServer.sh stop {{ zookeeper_conf_dir }}/zoo.cfg
ExecReload=/opt/zookeeper/bin/zkServer.sh restart {{ zookeeper_conf_dir }}/zoo.cfg

# OOM killer: -1000 disable ~ 1000 very likely
OOMScoreAdjust=-500
Restart=on-failure
RestartSec=30

[Install]
# usually this is your system default target
WantedBy=multi-user.target

More detail about systemd please search and see my systemd blog.

Then you must enable zookeeper starts on boot:

1
2
3
systemctl daemon-reload
# enable start on boot
systemctl enable zookeeper

Other systemd commands:

1
2
3
4
5
6
7
8
systemctl start zookeeper
systemctl stop zookeeper
systemctl restart zookeeper

# reload config without restart
systemctl reload zookeeper
# first try relaod, if not supports then restart
systemctl reload-or-restart zookeeper
0%