ES8集群部署与使用-zookeeper集群部署与使用
🌟ES8单点部署实战
参考链接:https://www.elastic.co/guide/en/elasticsearch/reference/8.18/deb.html
准备机器
ip | 主机名 | 配置 |
---|---|---|
10.0.0.94 | elk94 | 2C4G |
10.0.0.95 | elk95 | 2C4G |
10.0.0.96 | elk96 | 2C4G |
下载软件包
官网下载:https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.18.6-amd64.deb
[root@elk94 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.18.6-amd64.deb
安装ES8
[root@elk94 ~]# dpkg -i elasticsearch-8.18.6-amd64.deb
生成的密码,我们保留好:CkugKRWSr6sn01KfDr1o
启动ES服务
[root@elk94 ~]# systemctl daemon-reload
[root@elk94 ~]# systemctl enable --now elasticsearch.service
访问测试
[root@elk94 ~]# curl -k https://10.0.0.94:9200 -u "elastic:iVzoK-8a1eOF+O87lHcw"
🌟ES8集群部署实战
停止ES单点服务
1.停止ES服务
[root@elk94 ~]# systemctl stop elasticsearch.service2.查看监听端口
[root@elk94 ~]# ss -ntl | egrep "9200|9300"3.删除数据目录
[root@elk94 ~]# rm -rf /var/lib/elasticsearch/*
拷贝软件包到其他节点
[root@elk94 ~]# scp elasticsearch-8.18.6-amd64.deb 10.0.0.95:~
[root@elk94 ~]# scp elasticsearch-8.18.6-amd64.deb 10.0.0.96:~
其他节点安装ES8软件包
[root@elk95 ~]# dpkg -i elasticsearch-8.18.6-amd64.deb
[root@elk96 ~]# dpkg -i elasticsearch-8.18.6-amd64.deb
修改ES集群的配置文件
vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es8-cluster
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.0.94"]
cluster.initial_master_nodes: ["10.0.0.94"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:enabled: truekeystore.path: certs/http.p12
xpack.security.transport.ssl:enabled: trueverification_mode: certificatekeystore.path: certs/transport.p12truststore.path: certs/transport.p12
检查配置文件
egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
启动ES服务
[root@elk94 ~]# systemctl enable --now elasticsearch.service
[root@elk94 ~]# ss -ntl | egrep "9200|9300"
生成token
[root@elk94 ~]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTAuMC4wLjk0OjkyMDAiXSwiZmdyIjoiYTRiZWUxODkxMzk3ZjgyNzBiYzMzYmY0MjUzY2JiMjE3OTY3ZjBmYWIxYjgxMzRkN2QzMzEwMWJmY2Y3ZTg4NiIsImtleSI6InprQmlEWmtCMV9CaXpmSVpXaUhlOkdpSFY4YjlUQUZpSU5Kc19qOGk5RWcifQ==
新加入节点使用token注册
被加入节点使用令牌加入,目的是为了拷贝配置文件
[root@elk95 ~]# /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTAuMC4wLjk0OjkyMDAiXSwiZmdyIjoiYTRiZWUxODkxMzk3ZjgyNzBiYzMzYmY0MjUzY2JiMjE3OTY3ZjBmYWIxYjgxMzRkN2QzMzEwMWJmY2Y3ZTg4NiIsImtleSI6InprQmlEWmtCMV9CaXpmSVpXaUhlOkdpSFY4YjlUQUZpSU5Kc19qOGk5RWcifQ==[root@elk96 ~]# /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTAuMC4wLjk0OjkyMDAiXSwiZmdyIjoiYTRiZWUxODkxMzk3ZjgyNzBiYzMzYmY0MjUzY2JiMjE3OTY3ZjBmYWIxYjgxMzRkN2QzMzEwMWJmY2Y3ZTg4NiIsImtleSI6InprQmlEWmtCMV9CaXpmSVpXaUhlOkdpSFY4YjlUQUZpSU5Kc19qOGk5RWcifQ==
经过测试发现集群并没有加入成功
[root@elk96 ~]# curl -k https://10.0.0.94:9200/_cat/nodes -u "elastic:CkugKRWSr6sn01KfDr1o"
手动修改ES集群配置文件
[root@elk94 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: oldboyedu-linux99-es8-cluster
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.0.94","10.0.0.95","10.0.0.96"]
cluster.initial_master_nodes: ["10.0.0.94"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:enabled: truekeystore.path: certs/http.p12
xpack.security.transport.ssl:enabled: trueverification_mode: certificatekeystore.path: certs/transport.p12truststore.path: certs/transport.p12
同步配置文件到所有节点
[root@elk94 ~]# scp /etc/elasticsearch/elasticsearch.yml 10.0.0.95:/etc/elasticsearch/
[root@elk94 ~]# scp /etc/elasticsearch/elasticsearch.yml 10.0.0.96:/etc/elasticsearch/
重启ES集群
[root@elk94 ~]# systemctl restart elasticsearch.service
[root@elk95 ~]# systemctl restart elasticsearch.service
[root@elk96 ~]# systemctl restart elasticsearch.service
再次测试验证
curl -k https://10.0.0.94:9200/_cat/nodes -u "elastic:CkugKRWSr6sn01KfDr1o"
🌟ES8重置管理员elastic密码
重置密码
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
This tool will reset the password of the [elastic] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y # 手动输入字母'y'Password for the [elastic] user successfully reset.
New value: AQmNtJ2v=grGHGB89e2c # 这是新密码
用旧密码将无法访问
curl -k https://10.0.0.94:9200/_cat/nodes?v -u "elastic:iVzoK-8a1eOF+O87lHcw"
使用新密码是可以正常访问的
curl -k https://10.0.0.94:9200/_cat/nodes?v -u "elastic:AQmNtJ2v=grGHGB89e2c"
🌟部署kibana对接ES8集群
下载kibana
[root@elk94 ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-8.18.6-amd64.deb
安装kibana
[root@elk94 ~]# dpkg -i kibana-8.18.6-amd64.deb
修改kibana的配置文件
[root@elk94 ~]# vim /etc/kibana/kibana.yml
server.host: "0.0.0.0"
i18n.locale: "zh-CN"
启动kibana
[root@elk94 ~]# systemctl enable kibana.service --now
生成kiban专用的token
[root@elk94 ~]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTAuMC4wLjk0OjkyMDAiXSwiZmdyIjoiYTRiZWUxODkxMzk3ZjgyNzBiYzMzYmY0MjUzY2JiMjE3OTY3ZjBmYWIxYjgxMzRkN2QzMzEwMWJmY2Y3ZTg4NiIsImtleSI6IjNBdHZEWmtCZzhPdVlYWUM0WktoOlVBbjlnNlk0SzFyVE9Qek9VWUVsZ2cifQ==
访问kibana的webUI
http://10.0.0.94:5601/
kiban服务器获取校验码
[root@elk94 ~]# /usr/share/kibana/bin/kibana-verification-code
🌟filebeat对接ES8实战
下载filebeat
[root@elk95 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.18.6-amd64.deb
安装filebeat
[root@elk95 ~]# dpkg -i filebeat-8.18.6-amd64.deb
创建api-key
AwuFDZkBg8OuYXYCzZPM:O1P8XATTDJzZro6Fj0gORA
编写filebeat的配置文件
[root@elk95 ~]# vim /etc/filebeat/01-tcp-to-es.yaml
filebeat.inputs:
- type: tcphost: "0.0.0.0:9000"output.elasticsearch:hosts:- https://10.0.0.94:9200- https://10.0.0.95:9200- https://10.0.0.96:9200api_key: "AwuFDZkBg8OuYXYCzZPM:O1P8XATTDJzZro6Fj0gORA"index: es8-apikey-001# 跳过客户端证书校验。ssl.verification_mode: nonesetup.ilm.enabled: false
setup.template.name: "es8-apikey"
setup.template.pattern: "es8-apikey-*"
setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 0
启动filebeat实例
[root@elk95 ~]# filebeat -e -c /etc/filebeat/01-tcp-to-es.yaml
发送测试数据
[root@elk94 ~]# echo xixi | nc 10.0.0.95 9000
kibana查看数据
🌟logstash对接ES8集群
下载logstash
官网下载:https://artifacts.elastic.co/downloads/logstash/logstash-8.18.6-amd64.deb
[root@elk95 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-8.18.6-amd64.deb
安装logstash
[root@elk95 ~]# dpkg -i logstash-8.18.6-amd64.deb
创建符号链接
[root@elk95 ~]# ln -svf /usr/share/logstash/bin/logstash /usr/local/bin/
创建api-key
建议基于图形化创建,后期可以考虑使用API创建
AwuFDZkBg8OuYXYCzZPM:O1P8XATTDJzZro6Fj0gORA
编写Logstash配置文件
cat > /etc/logstash/conf.d/01-tcp-to-es.yaml <<EOF
input { tcp {port => 8888}
} output { if [type] == "dba" {elasticsearch {hosts => ["https://10.0.0.94:9200","https://10.0.0.95:9200","https://10.0.0.96:9200"]index => "zhu-es8-apikey-002"api_key => "AwuFDZkBg8OuYXYCzZPM:O1P8XATTDJzZro6Fj0gORA"ssl => truessl_certificate_verification => false}
}
EOF
启动Logstash
[root@elk96 ~]# logstash -f /etc/logstash/conf.d/01-tcp-to-es.yaml
发送测试数据
[root@elk94 ~]# echo haha | nc 10.0.0.96 8888
🌟ES8和ES7对比
1️⃣ES8默认启用了https,支持认证等功能;
2️⃣ES8新增’elasticsearch-reset-password’脚本,对于elastic用户重置密码更加简单;
3️⃣ES8新增’elasticsearch-create-enrollment-token’脚本,可以为组件创建token信息,比如kibana,node组件;
4️⃣ES8新增kibana新增’kibana-verification-code’用于生成校验码。
5️⃣kibana支持更多的语言:English (default) “en”, Chinese “zh-CN”, Japanese “ja-JP”, French “fr-FR”
6️⃣kibana的webUI更加丰富,支持AI助手,手动创建索引等功能;
7️⃣kibana不用创建索引模式就可以直接查看数据,系统会生成一个临时的视图以供查看;
8️⃣ES8集群部署时,需要借助’elasticsearch-reconfigure-node’脚本来加入已存在的集群,默认就是单master节点的配置;
🌟ELFK架构采集系统日志
创建api-key
创建zhu-es8-apikey-syslog索引的apikey
sys: 2AsqDpkBg8OuYXYCH6ER:90EhDgoopSqrU_effytCxQ
创建zhu-es8-apikey-auth索引的apikey
auth: 2gssDpkBg8OuYXYCeaGs:GMdZw1qEINar0NUnuU6BpQ
编写Logstash配置文件
cat > /etc/logstash/conf.d/02-beats-to-es.yaml <<'EOF'
input { beats {port => 6666type => auth}beats {port => 7777type => syslog}
} output { if [type] == "auth" {elasticsearch {hosts => ["https://10.0.0.94:9200","https://10.0.0.95:9200","https://10.0.0.96:9200"]index => "zhu-es8-apikey-auth"api_key => "2gssDpkBg8OuYXYCeaGs:GMdZw1qEINar0NUnuU6BpQ"ssl => truessl_certificate_verification => false}} else {elasticsearch {hosts => ["https://10.0.0.94:9200","https://10.0.0.95:9200","https://10.0.0.96:9200"]index => "zhu-es8-apikey-syslog"api_key => "2AsqDpkBg8OuYXYCH6ER:90EhDgoopSqrU_effytCxQ"ssl => truessl_certificate_verification => false}}
}
EOF
启动logstash
logstash -r -f /etc/logstash/conf.d/02-beats-to-es.yaml
编写filebeat配置文件采集auth日志
cat > /etc/filebeat/02-auth-to-logstash.yaml <<'EOF'
filebeat.inputs:
- type: filestreampaths:- /var/log/auth.log# 将数据发送给Logstash
output.logstash:hosts: ["10.0.0.96:6666"]
EOF
编写filebeat配置文件采集syslog日志
cat > /etc/filebeat/02-sys-to-logstash.yaml <<'EOF'
filebeat.inputs:
- type: filestreampaths:- /var/log/syslog# 将数据发送给Logstash
output.logstash:hosts: ["10.0.0.96:7777"]
EOF
运行多实例
filebeat -e -c /etc/filebeat/02-sys-to-logstash.yaml
filebeat -e -c /etc/filebeat/02-auth-to-logstash.yaml
创建角色
创建用户关联角色
登录用户
使用xixi用户登录
使用haha用户登录
🌟zookeeper单点部署
什么是zookeeper
ZooKeeper是一个集中式服务,用于维护配置信息、命名、提供分布式同步和提供组服务
所有这些类型的服务都以某种形式被分布式应用程序使用
主要的应用场景: Kafka,HDFS HA,YARN HA,HBase,Solr,…
官网地址: https://zookeeper.apache.org/
下载zookeeper
官方网站下载:https://dlcdn.apache.org/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz
[root@elk91 ~]# wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz
解压软件包
[root@elk91 ~]# tar xf apache-zookeeper-3.8.4-bin.tar.gz -C /usr/local/
配置环境变量
1.修改环境变量子配置文件
vim /etc/profile.d/zk.sh
#!/bin/bash
export ZK_HOME=/usr/local/apache-zookeeper-3.8.4-bin
export JAVA_HOME=/usr/share/elasticsearch/jdk
export PATH=$PATH:${ZK_HOME}/bin:${JAVA_HOME}/bin2.加载环境变量
source /etc/profile.d/zk.sh3.检查
java --version
准备配置文件
[root@elk91 ~]# cp /usr/local/apache-zookeeper-3.8.4-bin/conf/zoo{_sample,}.cfg
启动zookeeper服务
zkServer.sh start
查看zookeeper状态
zkServer.sh status
登录测试
zkCli.sh -server 10.0.0.91:2181
🌟zookeeper的基本使用
创建zookeeper node
[zk: 10.0.0.91:2181(CONNECTED) 1] create /haha ha
Created /haha
查看zookeeper node列表
[zk: 10.0.0.91:2181(CONNECTED) 9] ls /
[haha, zookeeper]
查看zookeeper node数据
[zk: 10.0.0.91:2181(CONNECTED) 4] get /haha
haha
修改zookeeper node数据
[zk: 10.0.0.91:2181(CONNECTED) 3] set /haha haha
[zk: 10.0.0.91:2181(CONNECTED) 4] get /haha
haha
删除zookeeper node
[zk: 10.0.0.91:2181(CONNECTED) 5] delete
delete deleteall
[zk: 10.0.0.91:2181(CONNECTED) 5] delete /haha
[zk: 10.0.0.91:2181(CONNECTED) 6] ls /
[zookeeper]
delete只能删除空目录,deleteall可以递归删除目录
创建层级的zookeeper node
[zk: 10.0.0.91:2181(CONNECTED) 7] create /haha
Created /haha
[zk: 10.0.0.91:2181(CONNECTED) 8] set /haha haha
[zk: 10.0.0.91:2181(CONNECTED) 9] ls /
[haha, zookeeper]
[zk: 10.0.0.91:2181(CONNECTED) 10] create /haha/xixi xixi
Created /haha/xixi
🌟zookeeper的集群部署
集群模式节点数量选择
参考链接: https://zookeeper.apache.org/doc/current/zookeeperOver.html
停止单点服务
zkServer.sh stop
修改zookeeper的配置文件
vim /usr/local/apache-zookeeper-3.8.4-bin/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
server.11=10.0.0.91:2888:3888
server.22=10.0.0.92:2888:3888
server.33=10.0.0.93:2888:3888
将程序和环境变量文件同步到集群的其他节点
scp -r /usr/local/apache-zookeeper-3.8.4-bin/ 10.0.0.92:/usr/local
scp -r /usr/local/apache-zookeeper-3.8.4-bin/ 10.0.0.93:/usr/local
scp /etc/profile.d/zk.sh 10.0.0.92:/etc/profile.d/
scp /etc/profile.d/zk.sh 10.0.0.93:/etc/profile.d/
所有节点准备数据目录
[root@elk91 ~]# mkdir /var/lib/zookeeper && echo 11 > /var/lib/zookeeper/myid
[root@elk92 ~]# mkdir /var/lib/zookeeper && echo 22 > /var/lib/zookeeper/myid
[root@elk93 ~]# mkdir /var/lib/zookeeper && echo 33 > /var/lib/zookeeper/myid
检查集群的状态
zkServer.sh status
客户端链接测试
zkCli.sh -server 10.0.0.91:2181,10.0.0.92:2181,10.0.0.93:2181
验证zookeeper集群高可用
开启两个终端指向不同的节点分别写入数据测试
[root@elk92 ~]# zkCli.sh -server 10.0.0.92:2181
[root@elk93 ~]# zkCli.sh -server 10.0.0.93:2181
停止leader节点观察是否可用
[root@elk91 ~]# zkCli.sh -server 10.0.0.91:2181,10.0.0.92:2181,10.0.0.93:2181
停止服务观察终端是否可用
[root@elk93 ~]# zkServer.sh stop
测试验证故障容忍节点数量
[root@elk92 ~]# zkServer.sh stop
[root@elk93 ~]# zkServer.sh stop
zookeeper集群要保证半数以上存活机制
一个集群如果想要容忍N台故障,则要求有2N+1台服务器
举例子:
如果集群容忍挂掉1个节点,则至少要准备2*1 + 1 = 3台服务器
如果集群容忍挂掉2个节点,则至少要准备2*2 + 1 = 5台服务器
🌟zookeeper的leader选举流程
1️⃣启动集群是默认都认为自己是leader,进行leader选举
2️⃣选举时各节点要暴露出自己的: myid,zxid
3️⃣先比较zxid事务ID,越大就优先被选举为leader
4️⃣如果zxid比不出来,则比较myid,myid越大则优先成为leader
5️⃣当zookeeper集群半数以上节点参与选举完成,则leader就确认下来了
🌟zookeeper的watch机制及node类型
watch监听子节点是否变化,节点数据是否变化
node类型:临时和永久
终端1监听事件
[zk: 10.0.0.93:2181(CONNECTED) 1] ls -w /haha
终端2触发事件
[zk: 10.0.0.92:2181(CONNECTED) 0] create /haha/haha
Created /haha/haha
[zk: 10.0.0.92:2181(CONNECTED) 1]
观察终端1出现结果
[zk: 10.0.0.93:2181(CONNECTED) 0] ls /haha
[heihei, xixi]
[zk: 10.0.0.93:2181(CONNECTED) 1] ls -w /haha
[heihei, xixi]
[zk: 10.0.0.93:2181(CONNECTED) 2]
WATCHER::WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/haha
类似watch还支持监控数据变化
[zk: 10.0.0.93:2181(CONNECTED) 2] get -w /haha/xixi
null
[zk: 10.0.0.93:2181(CONNECTED) 3]
WATCHER::WatchedEvent state:SyncConnected type:NodeDataChanged path:/haha/xixi
🌟zookeeper的类型
临时的zookeeper node
当链接断开,zookeeper node在一定时间范围(30s)内自动删除
永久的zookeeper node
当链接断开后,zookeeper node数据并不丢失
默认情况下,我们创建的 都是永久的zookeeper node
[zk: 10.0.0.93:2181(CONNECTED) 5] stat /linshi
cZxid = 0x400000005
ctime = Wed Sep 03 08:57:38 UTC 2025
mZxid = 0x400000005
mtime = Wed Sep 03 08:57:38 UTC 2025
pZxid = 0x400000005
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x21000060bad10000
dataLength = 0
numChildren = 0
[zk: 10.0.0.93:2181(CONNECTED) 6] create /yongjiu
Created /yongjiu
[zk: 10.0.0.93:2181(CONNECTED) 7] stat /yongjiu
cZxid = 0x400000006
ctime = Wed Sep 03 09:02:52 UTC 2025
mZxid = 0x400000006
mtime = Wed Sep 03 09:02:52 UTC 2025
pZxid = 0x400000006
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: 10.0.0.93:2181(CONNECTED) 8]
当前连接的session 会话ID。当当前会话断开30s(negotiated timeout)后会自动删除
当前会话ID为:0x21000060bad10000
断开会话重新连接,临时的node自动删除