ELK 日志管理系统相关内容总结

ELK 日志管理系统相关内容总结

一、日志管理背景与需求

1. 传统日志管理痛点：日志分散在不同服务器，传统登录单台服务器查看的方式繁琐低效；rsyslog 虽能实现日志集中管理，但集中后的日志统计与检索（如按变动时间周期统计网站 PV、UV 数据，合并多 realserver 日志后统计）存在困难，wc、grep、awk 等命令在高要求场景下力不从心
2. 核心需求：需要专业的集中式日志收集、分析、展示系统，满足日志集中管理、灵活统计（如多服务器日志合并统计）、图形化展示数据的需求

二、ELK 系统基础认知

1. ELK 组成：由 Elasticsearch（ES）、Logstash、Kibana 三款开源软件组成，常搭配轻量级日志收集工具 Filebeat，官网下载地址为https://www.elastic.co/cn/downloads

2. 各组件核心功能

   组件	核心功能
   Elasticsearch（ES）	分布式搜索引擎与文档数据库，提供数据存储、快速搜索与分析功能
   Logstash	日志采集与处理工具，通过 Input（导入日志源）、Filter（过滤，非必需）、Output（导出，必需）插件，采集多种格式数据（结构化、半结构化、非结构化）并输出到指定目的地（如 ES）
   Kibana	日志图形化展示工具，支持通过浏览器访问，可视化 Elasticsearch 中的数据
   Filebeat	轻量级日志收集工具，适用于资源有限的服务器，辅助 Logstash 采集日志

3. 工作流程：Logstash/Filebeat 从应用服务器收集日志，经过滤后输出给 Elasticsearch 集群存储，用户通过 Kibana 访问 Elasticsearch 获取并查看图形化日志数据

4. 应用场景：解决分散日志的集中管理、多维度统计分析（如网站 PV/UV 统计）、日志数据可视化展示等问题，适用于运维人员监控系统与业务状态

具体示例：

四台机器(内存建议大于1G，比如1.5G; filebeat服务器可为1G) :

静态IP(要求能上公网,最好用虚拟机的NAT网络类型上网)
主机名及主机名绑定

IP 地址	主机名	部署组件
192.168.100.10	vm1.cluster.com	Kibana
192.168.100.20	vm2.cluster.com	Elasticsearch
192.168.100.30	vm3.cluster.com	Logstash

已关闭防火墙与selinux

设置时间同步

elasticsearch部署
第1步: 在elasticsearch服务器上(我这里为vm2),确认jdk(使用系统自带的openjdk就OK)

[root@vm2 ~]# rpm -qa | grep openjdk
java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64
[root@vm2 ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

第2步: es的安装,配置：（注：这里是上传）

[root@vm2 ~]# rz -E
rz waiting to receive.
[root@vm2 ~]# rpm -ivh elasticsearch-6.5.2.rpm 
warning: elasticsearch-6.5.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...1:elasticsearch-0:6.5.2-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemdsudo systemctl daemon-reloadsudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executingsudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch

第3步: 单机es的配置与服务启动：

[root@vm2 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-cluster       //可以自定义一个集群名称,不配置的话默认会取名为elasticsearchpath.data: /var/lib/elasticsearchpath.logs: /var/log/elasticsearchnetwork.host: 0.0.0.0           //打开注释，并修改为监听所有http.port: 9200                 //打开注释，监听端口9200
[root@vm2 ~]# systemctl start elasticsearch
[root@vm2 ~]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

启动有点慢和卡，稍等1分钟左右，查看到以下端口则表示启动OK

[root@vm2 ~]# netstat -ntlup | grep java
tcp6       0      0 :::9200                 :::*                    LISTEN      57963/java          
tcp6       0      0 :::9300                 :::*                    LISTEN      57963/java          

9200则是数据传输端口

9300端口是集群通信端口（我们暂时还没有配置集群，现在是单点elasticsearch）

第4步: 查看状态
使用curl命令或浏览器访问http://192.168.100.20:9200/_cluster/health?pretty地址(IP为ES服务器IP)

[root@vm2 ~]# curl http://192.168.100.20:9200/_cluster/health?pretty
{"cluster_name" : "elk-cluster","status" : "green","timed_out" : false,"number_of_nodes" : 1,"number_of_data_nodes" : 1,"active_primary_shards" : 0,"active_shards" : 0,"relocating_shards" : 0,"initializing_shards" : 0,"unassigned_shards" : 0,"delayed_unassigned_shards" : 0,"number_of_pending_tasks" : 0,"number_of_in_flight_fetch" : 0,"task_max_waiting_in_queue_millis" : 0,"active_shards_percent_as_number" : 100.0
}

elasticsearch集群部署

可以使用两台或两台以上ES做集群, 以下就是两台ES做集群的配置

首先在ES集群所有节点都安装ES

[root@vm1 ~]# rz -E
rz waiting to receive.
[root@vm1 ~]# rpm -ivh elasticsearch-6.5.2.rpm 
warning: elasticsearch-6.5.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...1:elasticsearch-0:6.5.2-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemdsudo systemctl daemon-reloadsudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executingsudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch[root@vm1 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-clusternode.name: 192.168.100.10       //本机IP或主机名
node.master: false                //指定不为master节点path.data: /var/lib/elasticsearchpath.logs: /var/log/elasticsearchnetwork.host: 0.0.0.0http.port: 9200discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]   //集群所有节点IP

[root@vm2 ~]# vim /etc/elasticsearch/elasticsearch.yml 
......
cluster.name: elk-clusternode.name: 192.168.100.20       //本机IP或主机名
node.master: true                //指定为master节点path.data: /var/lib/elasticsearchpath.logs: /var/log/elasticsearchnetwork.host: 0.0.0.0http.port: 9200discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]   //集群所有节点IP

启动或重启服务：

[root@vm1 ~]# systemctl restart elasticsearch
[root@vm1 ~]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

[root@vm2 ~]# systemctl restart elasticsearch

查看状态：

查看节点信息：

通过curl或浏览器访问http://192.168.100.20:9200/_cat/nodes?v(ip为ES节点IP,如果有ES集群,则为ES任意节
点IP)

[root@vm2 ~]# curl http://192.168.100.20:9200/_cat/nodes?v
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.100.20           26          69   0    0.06    0.06     0.05 mdi       *      192.168.100.20
192.168.100.10           25          68   0    0.00    0.03     0.05 di        -      192.168.100.10

查看索引信息：
通过curl或浏览器访问http://10.1.1.12:9200/_cat/indices?v：

[root@vm2 ~]# curl http://192.168.100.20:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

默认现在没有任何索引

新增索引：

[root@vm2 ~]# curl -X PUT http://192.168.100.20:9200/nginx_access_log
{"acknowledged":true,"shards_acknowledged":true,"index":"nginx_access_log"}[root@vm2 ~]# curl http://192.168.100.20:9200/_cat/indices?v
health status index            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   nginx_access_log DywHuNoaRB6fEU4zmi0oIQ   5   1          0            0      2.2kb          1.1kb

green：所有的主分片和副本分片都已分配。你的集群是100%可用的
yellow：所有的主分片已经分片了，但至少还有一个副本是缺失的。不会有数据丢失，所以搜索结果依
然是完整的。不过，你的高可用性在某种程度上被弱化。如果 更多的 分片消失，你就会丢数据了。把
yellow 想象成一个需要及时调查的警告
red：至少一个主分片（以及它的全部副本）都在缺失中。这意味着你在缺少数据：搜索只能返回部分数
据，而分配到这个分片上的写入请求会返回一个异常

删除索引：

[root@vm2 ~]# curl -X DELETE http://192.168.100.20:9200/nginx_access_log
{"acknowledged":true}

ES查询语句（拓展了解）
ES提供一种可用于执行查询JSON式的语言，被称为Query DSL
针对elasticsearch的操作，可以分为增、删、改、查四个动作

查询匹配条件：
match_all
from,size
match
bool
range

查询应用案例：
导入数据源
使用官方提供的示例数据：
下载并导入进elasticsearch：（注：这里是直接上传）

[root@vm2 ~]# rz -E
rz waiting to receive.导入进elasticsearch
[root@vm2 ~]# curl -H "Content-Type: application/json" -XPOST "192.168.100.20:9200/bank/_doc/_bulk?pretty&refresh" --data-binary "@accounts.json"查询确认
[root@vm2 ~]# curl "192.168.100.20:9200/_cat/indices?v"
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   bank  dC8RjIl1QWeSMmpIpV-vsw   5   1       1000            0    949.4kb        474.7kb

查询bank索引的数据(使用查询字符串进行查询)

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?q=*&sort=account_number:asc&pretty"
{"took" : 98,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1000,"max_score" : null,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "0","_score" : null,"_source" : {"account_number" : 0,"balance" : 16623,"firstname" : "Bradshaw","lastname" : "Mckenzie","age" : 29,"gender" : "F","address" : "244 Columbus Place","employer" : "Euron","email" : "[e-mail]","city" : "Hobucken","state" : "CO"},"sort" : [0]},{"_index" : "bank","_type" : "_doc","_id" : "1","_score" : null,"_source" : {"account_number" : 1,"balance" : 39225,"firstname" : "Amber","lastname" : "Duke","age" : 32,"gender" : "M","address" : "880 Holmes Lane","employer" : "Pyrami","email" : "[e-mail]","city" : "Brogan","state" : "IL"},"sort" : [1]},{"_index" : "bank","_type" : "_doc","_id" : "2","_score" : null,"_source" : {"account_number" : 2,"balance" : 28838,"firstname" : "Roberta","lastname" : "Bender","age" : 22,"gender" : "F","address" : "560 Kingsway Place","employer" : "Chillium","email" : "[e-mail]","city" : "Bennett","state" : "LA"},"sort" : [2]},{"_index" : "bank","_type" : "_doc","_id" : "3","_score" : null,"_source" : {"account_number" : 3,"balance" : 44947,"firstname" : "Levine","lastname" : "Burks","age" : 26,"gender" : "F","address" : "328 Wilson Avenue","employer" : "Amtap","email" : "[e-mail]","city" : "Cochranville","state" : "HI"},"sort" : [3]},{"_index" : "bank","_type" : "_doc","_id" : "4","_score" : null,"_source" : {"account_number" : 4,"balance" : 27658,"firstname" : "Rodriquez","lastname" : "Flores","age" : 31,"gender" : "F","address" : "986 Wyckoff Avenue","employer" : "Tourmania","email" : "[e-mail]","city" : "Eastvale","state" : "HI"},"sort" : [4]},{"_index" : "bank","_type" : "_doc","_id" : "5","_score" : null,"_source" : {"account_number" : 5,"balance" : 29342,"firstname" : "Leola","lastname" : "Stewart","age" : 30,"gender" : "F","address" : "311 Elm Place","employer" : "Diginetic","email" : "[e-mail]","city" : "Fairview","state" : "NJ"},"sort" : [5]},{"_index" : "bank","_type" : "_doc","_id" : "6","_score" : null,"_source" : {"account_number" : 6,"balance" : 5686,"firstname" : "Hattie","lastname" : "Bond","age" : 36,"gender" : "M","address" : "671 Bristol Street","employer" : "Netagy","email" : "[e-mail]","city" : "Dante","state" : "TN"},"sort" : [6]},{"_index" : "bank","_type" : "_doc","_id" : "7","_score" : null,"_source" : {"account_number" : 7,"balance" : 39121,"firstname" : "Levy","lastname" : "Richard","age" : 22,"gender" : "M","address" : "820 Logan Street","employer" : "Teraprene","email" : "[e-mail]","city" : "Shrewsbury","state" : "MO"},"sort" : [7]},{"_index" : "bank","_type" : "_doc","_id" : "8","_score" : null,"_source" : {"account_number" : 8,"balance" : 48868,"firstname" : "Jan","lastname" : "Burns","age" : 35,"gender" : "M","address" : "699 Visitation Place","employer" : "Glasstep","email" : "[e-mail]","city" : "Wakulla","state" : "AZ"},"sort" : [8]},{"_index" : "bank","_type" : "_doc","_id" : "9","_score" : null,"_source" : {"account_number" : 9,"balance" : 24776,"firstname" : "Opal","lastname" : "Meadows","age" : 39,"gender" : "M","address" : "963 Neptune Avenue","employer" : "Cedward","email" : "[e-mail]","city" : "Olney","state" : "OH"},"sort" : [9]}]}
}说明：
默认结果为10条
_search 属于一类API，用于执行查询操作
q=* ES批量索引中的所有文档
sort=account_number:asc 表示根据account_number按升序对结果排序
pretty调整显示格式

查询bank索引的数据 (使用json格式进行查询)：

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search" -H 'Content-Type: application/json' -d'
> {
> "query": { "match_all": {} },
> "sort": [
> { "account_number": "asc" }
> ]
> }
> '      注意：最后为单引号
{"took":10,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1000,"max_score":null,"hits":[{"_index":"bank","_type":"_doc","_id":"0","_score":null,"_source":{"account_number":0,"balance":16623,"firstname":"Bradshaw","lastname":"Mckenzie","age":29,"gender":"F","address":"244 Columbus Place","employer":"Euron","email":"[e-mail]","city":"Hobucken","state":"CO"},"sort":[0]},{"_index":"bank","_type":"_doc","_id":"1","_score":null,"_source":{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"[e-mail]","city":"Brogan","state":"IL"},"sort":[1]},{"_index":"bank","_type":"_doc","_id":"2","_score":null,"_source":{"account_number":2,"balance":28838,"firstname":"Roberta","lastname":"Bender","age":22,"gender":"F","address":"560 Kingsway Place","employer":"Chillium","email":"[e-mail]","city":"Bennett","state":"LA"},"sort":[2]},{"_index":"bank","_type":"_doc","_id":"3","_score":null,"_source":{"account_number":3,"balance":44947,"firstname":"Levine","lastname":"Burks","age":26,"gender":"F","address":"328 Wilson Avenue","employer":"Amtap","email":"[e-mail]","city":"Cochranville","state":"HI"},"sort":[3]},{"_index":"bank","_type":"_doc","_id":"4","_score":null,"_source":{"account_number":4,"balance":27658,"firstname":"Rodriquez","lastname":"Flores","age":31,"gender":"F","address":"986 Wyckoff Avenue","employer":"Tourmania","email":"[e-mail]","city":"Eastvale","state":"HI"},"sort":[4]},{"_index":"bank","_type":"_doc","_id":"5","_score":null,"_source":{"account_number":5,"balance":29342,"firstname":"Leola","lastname":"Stewart","age":30,"gender":"F","address":"311 Elm Place","employer":"Diginetic","email":"[e-mail]","city":"Fairview","state":"NJ"},"sort":[5]},{"_index":"bank","_type":"_doc","_id":"6","_score":null,"_source":{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"[e-mail]","city":"Dante","state":"TN"},"sort":[6]},{"_index":"bank","_type":"_doc","_id":"7","_score":null,"_source":{"account_number":7,"balance":39121,"firstname":"Levy","lastname":"Richard","age":22,"gender":"M","address":"820 Logan Street","employer":"Teraprene","email":"[e-mail]","city":"Shrewsbury","state":"MO"},"sort":[7]},{"_index":"bank","_type":"_doc","_id":"8","_score":null,"_source":{"account_number":8,"balance":48868,"firstname":"Jan","lastname":"Burns","age":35,"gender":"M","address":"699 Visitation Place","employer":"Glasstep","email":"[e-mail]","city":"Wakulla","state":"AZ"},"sort":[8]},{"_index":"bank","_type":"_doc","_id":"9","_score":null,"_source":{"account_number":9,"balance":24776,"firstname":"Opal","lastname":"Meadows","age":39,"gender":"M","address":"963 Neptune Avenue","employer":"Cedward","email":"[e-mail]","city":"Olney","state":"OH"},"sort":[9]}]}}

问题: 怎么将上面json格式进行pretty查询?

查询匹配动作及案例：
match_all
匹配所有文档。默认查询
示例：查询所有，默认只返回10个文档

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
> {
> "query": { "match_all": {} }
> }
> '
{"took" : 6,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1000,"max_score" : 1.0,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "25","_score" : 1.0,"_source" : {"account_number" : 25,"balance" : 40540,"firstname" : "Virginia","lastname" : "Ayala","age" : 39,"gender" : "F","address" : "171 Putnam Avenue","employer" : "Filodyne","email" : "[e-mail]","city" : "Nicholson","state" : "PA"}},{"_index" : "bank","_type" : "_doc","_id" : "44","_score" : 1.0,"_source" : {"account_number" : 44,"balance" : 34487,"firstname" : "Aurelia","lastname" : "Harding","age" : 37,"gender" : "M","address" : "502 Baycliff Terrace","employer" : "Orbalix","email" : "[e-mail]","city" : "Yardville","state" : "DE"}},{"_index" : "bank","_type" : "_doc","_id" : "99","_score" : 1.0,"_source" : {"account_number" : 99,"balance" : 47159,"firstname" : "Ratliff","lastname" : "Heath","age" : 39,"gender" : "F","address" : "806 Rockwell Place","employer" : "Zappix","email" : "[e-mail]","city" : "Shaft","state" : "ND"}},{"_index" : "bank","_type" : "_doc","_id" : "119","_score" : 1.0,"_source" : {"account_number" : 119,"balance" : 49222,"firstname" : "Laverne","lastname" : "Johnson","age" : 28,"gender" : "F","address" : "302 Howard Place","employer" : "Senmei","email" : "[e-mail]","city" : "Herlong","state" : "DC"}},{"_index" : "bank","_type" : "_doc","_id" : "126","_score" : 1.0,"_source" : {"account_number" : 126,"balance" : 3607,"firstname" : "Effie","lastname" : "Gates","age" : 39,"gender" : "F","address" : "620 National Drive","employer" : "Digitalus","email" : "[e-mail]","city" : "Blodgett","state" : "MD"}},{"_index" : "bank","_type" : "_doc","_id" : "145","_score" : 1.0,"_source" : {"account_number" : 145,"balance" : 47406,"firstname" : "Rowena","lastname" : "Wilkinson","age" : 32,"gender" : "M","address" : "891 Elton Street","employer" : "Asimiline","email" : "[e-mail]","city" : "Ripley","state" : "NH"}},{"_index" : "bank","_type" : "_doc","_id" : "183","_score" : 1.0,"_source" : {"account_number" : 183,"balance" : 14223,"firstname" : "Hudson","lastname" : "English","age" : 26,"gender" : "F","address" : "823 Herkimer Place","employer" : "Xinware","email" : "[e-mail]","city" : "Robbins","state" : "ND"}},{"_index" : "bank","_type" : "_doc","_id" : "190","_score" : 1.0,"_source" : {"account_number" : 190,"balance" : 3150,"firstname" : "Blake","lastname" : "Davidson","age" : 30,"gender" : "F","address" : "636 Diamond Street","employer" : "Quantasis","email" : "[e-mail]","city" : "Crumpler","state" : "KY"}},{"_index" : "bank","_type" : "_doc","_id" : "208","_score" : 1.0,"_source" : {"account_number" : 208,"balance" : 40760,"firstname" : "Garcia","lastname" : "Hess","age" : 26,"gender" : "F","address" : "810 Nostrand Avenue","employer" : "Quiltigen","email" : "[e-mail]","city" : "Brooktrails","state" : "GA"}},{"_index" : "bank","_type" : "_doc","_id" : "222","_score" : 1.0,"_source" : {"account_number" : 222,"balance" : 14764,"firstname" : "Rachelle","lastname" : "Rice","age" : 36,"gender" : "M","address" : "333 Narrows Avenue","employer" : "Enaut","email" : "[e-mail]","city" : "Wright","state" : "AZ"}}]}
}

query告诉我们查询什么

match_all是我们查询的类型

match_all查询仅仅在指定的索引的所有文件进行搜索

from，size
除了query参数外，还可以传递其他参数影响查询结果，比如前面提到的sort，接下来使用的size：

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match_all": {} },
> "size": 1
> }
> '
{"took" : 6,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1000,"max_score" : 1.0,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "25","_score" : 1.0,"_source" : {"account_number" : 25,"balance" : 40540,"firstname" : "Virginia","lastname" : "Ayala","age" : 39,"gender" : "F","address" : "171 Putnam Avenue","employer" : "Filodyne","email" : "[e-mail]","city" : "Nicholson","state" : "PA"}}]}
}

查询1条数据

指定位置与查询条数

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match_all": {} },
> "from": 0, 
> "size": 2
> }
> '
{"took" : 6,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1000,"max_score" : 1.0,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "25","_score" : 1.0,"_source" : {"account_number" : 25,"balance" : 40540,"firstname" : "Virginia","lastname" : "Ayala","age" : 39,"gender" : "F","address" : "171 Putnam Avenue","employer" : "Filodyne","email" : "[e-mail]","city" : "Nicholson","state" : "PA"}},{"_index" : "bank","_type" : "_doc","_id" : "44","_score" : 1.0,"_source" : {"account_number" : 44,"balance" : 34487,"firstname" : "Aurelia","lastname" : "Harding","age" : 37,"gender" : "M","address" : "502 Baycliff Terrace","employer" : "Orbalix","email" : "[e-mail]","city" : "Yardville","state" : "DE"}}]}
}

from 0表示从第1个开始

size 指定查询的个数

示例: 查询account_number从第501条到510条的数据：

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query": { "match_all": {} },
"from": 500,
"size": 10,
"sort": [
{ "account_number": "asc" }
]
}
' 2>/dev/null | grep account_number"account_number" : 500,"account_number" : 501,"account_number" : 502,"account_number" : 503,"account_number" : 504,"account_number" : 505,"account_number" : 506,"account_number" : 507,"account_number" : 508,"account_number" : 509,

匹配查询字段
返回_source字段中的片段字段

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match_all": {} },
> "_source": ["account_number", "balance"]
> }
> '
{"took" : 5,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1000,"max_score" : 1.0,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "25","_score" : 1.0,"_source" : {"account_number" : 25,"balance" : 40540}},{"_index" : "bank","_type" : "_doc","_id" : "44","_score" : 1.0,"_source" : {"account_number" : 44,"balance" : 34487}},{"_index" : "bank","_type" : "_doc","_id" : "99","_score" : 1.0,"_source" : {"account_number" : 99,"balance" : 47159}},{"_index" : "bank","_type" : "_doc","_id" : "119","_score" : 1.0,"_source" : {"account_number" : 119,"balance" : 49222}},{"_index" : "bank","_type" : "_doc","_id" : "126","_score" : 1.0,"_source" : {"account_number" : 126,"balance" : 3607}},{"_index" : "bank","_type" : "_doc","_id" : "145","_score" : 1.0,"_source" : {"account_number" : 145,"balance" : 47406}},{"_index" : "bank","_type" : "_doc","_id" : "183","_score" : 1.0,"_source" : {"account_number" : 183,"balance" : 14223}},{"_index" : "bank","_type" : "_doc","_id" : "190","_score" : 1.0,"_source" : {"account_number" : 190,"balance" : 3150}},{"_index" : "bank","_type" : "_doc","_id" : "208","_score" : 1.0,"_source" : {"account_number" : 208,"balance" : 40760}},{"_index" : "bank","_type" : "_doc","_id" : "222","_score" : 1.0,"_source" : {"account_number" : 222,"balance" : 14764}}]}
}

match
基本搜索查询，针对特定字段或字段集合进行搜索
查询编号为20的账户

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match": { "account_number": 20 } }
> }
> '
{"took" : 11,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1,"max_score" : 1.0,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "20","_score" : 1.0,"_source" : {"account_number" : 20,"balance" : 16418,"firstname" : "Elinor","lastname" : "Ratliff","age" : 36,"gender" : "M","address" : "282 Kings Place","employer" : "Scentric","email" : "[e-mail]","city" : "Ribera","state" : "WA"}}]}
}

返回地址中包含mill的账户

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query": { "match": { "address": "mill" } }   
}
'
{"took" : 11,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 4,"max_score" : 4.89784,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "472","_score" : 4.89784,"_source" : {"account_number" : 472,"balance" : 25571,"firstname" : "Lee","lastname" : "Long","age" : 32,"gender" : "F","address" : "288 Mill Street","employer" : "Comverges","email" : "[e-mail]","city" : "Movico","state" : "MT"}},{"_index" : "bank","_type" : "_doc","_id" : "136","_score" : 4.8485627,"_source" : {"account_number" : 136,"balance" : 45801,"firstname" : "Winnie","lastname" : "Holland","age" : 38,"gender" : "M","address" : "198 Mill Lane","employer" : "Neteria","email" : "[e-mail]","city" : "Urie","state" : "IL"}},{"_index" : "bank","_type" : "_doc","_id" : "970","_score" : 4.388994,"_source" : {"account_number" : 970,"balance" : 19648,"firstname" : "Forbes","lastname" : "Wallace","age" : 28,"gender" : "M","address" : "990 Mill Road","employer" : "Pheast","email" : "[e-mail]","city" : "Lopezo","state" : "AK"}},{"_index" : "bank","_type" : "_doc","_id" : "345","_score" : 4.388994,"_source" : {"account_number" : 345,"balance" : 9812,"firstname" : "Parker","lastname" : "Hines","age" : 38,"gender" : "M","address" : "715 Mill Avenue","employer" : "Baluba","email" : "[e-mail]","city" : "Blackgum","state" : "KY"}}]}
}

返回地址有包含mill或lane的所有账户

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query": { "match": { "address": "mill lane" } }      //空格就是或的关系
}
'
{"took" : 12,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 19,"max_score" : 8.398771,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "136","_score" : 8.398771,"_source" : {"account_number" : 136,"balance" : 45801,"firstname" : "Winnie","lastname" : "Holland","age" : 38,"gender" : "M","address" : "198 Mill Lane","employer" : "Neteria","email" : "[e-mail]","city" : "Urie","state" : "IL"}},{"_index" : "bank","_type" : "_doc","_id" : "472","_score" : 4.89784,"_source" : {"account_number" : 472,"balance" : 25571,"firstname" : "Lee","lastname" : "Long","age" : 32,"gender" : "F","address" : "288 Mill Street","employer" : "Comverges","email" : "[e-mail]","city" : "Movico","state" : "MT"}},{"_index" : "bank","_type" : "_doc","_id" : "556","_score" : 4.4402957,"_source" : {"account_number" : 556,"balance" : 36420,"firstname" : "Collier","lastname" : "Odonnell","age" : 35,"gender" : "M","address" : "591 Nolans Lane","employer" : "Sultraxin","email" : "[e-mail]","city" : "Fulford","state" : "MD"}},{"_index" : "bank","_type" : "_doc","_id" : "934","_score" : 4.4402957,"_source" : {"account_number" : 934,"balance" : 43987,"firstname" : "Freida","lastname" : "Daniels","age" : 34,"gender" : "M","address" : "448 Cove Lane","employer" : "Vurbo","email" : "[e-mail]","city" : "Snelling","state" : "NJ"}},{"_index" : "bank","_type" : "_doc","_id" : "970","_score" : 4.388994,"_source" : {"account_number" : 970,"balance" : 19648,"firstname" : "Forbes","lastname" : "Wallace","age" : 28,"gender" : "M","address" : "990 Mill Road","employer" : "Pheast","email" : "[e-mail]","city" : "Lopezo","state" : "AK"}},{"_index" : "bank","_type" : "_doc","_id" : "345","_score" : 4.388994,"_source" : {"account_number" : 345,"balance" : 9812,"firstname" : "Parker","lastname" : "Hines","age" : 38,"gender" : "M","address" : "715 Mill Avenue","employer" : "Baluba","email" : "[e-mail]","city" : "Blackgum","state" : "KY"}},{"_index" : "bank","_type" : "_doc","_id" : "908","_score" : 4.388994,"_source" : {"account_number" : 908,"balance" : 45975,"firstname" : "Mosley","lastname" : "Holloway","age" : 31,"gender" : "M","address" : "929 Eldert Lane","employer" : "Anivet","email" : "[e-mail]","city" : "Biehle","state" : "MS"}},{"_index" : "bank","_type" : "_doc","_id" : "637","_score" : 4.388994,"_source" : {"account_number" : 637,"balance" : 3169,"firstname" : "Kathy","lastname" : "Carter","age" : 27,"gender" : "F","address" : "410 Jamison Lane","employer" : "Limage","email" : "[e-mail]","city" : "Ernstville","state" : "WA"}},{"_index" : "bank","_type" : "_doc","_id" : "449","_score" : 4.3719764,"_source" : {"account_number" : 449,"balance" : 41950,"firstname" : "Barnett","lastname" : "Cantrell","age" : 39,"gender" : "F","address" : "945 Bedell Lane","employer" : "Zentility","email" : "[e-mail]","city" : "Swartzville","state" : "ND"}},{"_index" : "bank","_type" : "_doc","_id" : "742","_score" : 4.3719764,"_source" : {"account_number" : 742,"balance" : 24765,"firstname" : "Merle","lastname" : "Wooten","age" : 26,"gender" : "M","address" : "317 Pooles Lane","employer" : "Tropolis","email" : "[e-mail]","city" : "Bentley","state" : "ND"}}]}
}

bool

bool must 查询的字段必须同时存在
查询包含mill和lane的所有账户

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": {
> "bool": {
> "must": [
> { "match": { "address": "mill" } },
> { "match": { "address": "lane" } }
> ]
> }
> }
> }
> '
{"took" : 14,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1,"max_score" : 8.398771,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "136","_score" : 8.398771,"_source" : {"account_number" : 136,"balance" : 45801,"firstname" : "Winnie","lastname" : "Holland","age" : 38,"gender" : "M","address" : "198 Mill Lane","employer" : "Neteria","email" : "[e-mail]","city" : "Urie","state" : "IL"}}]}
}

bool should 查询的字段仅存在一即可
查询包含mill或lane的所有账户

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": {
> "bool": {
> "should": [
> { "match": { "address": "mill" } },
> { "match": { "address": "lane" } }
> ]
> }
> }
> }
> '
{"took" : 4,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 19,"max_score" : 8.398771,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "136","_score" : 8.398771,"_source" : {"account_number" : 136,"balance" : 45801,"firstname" : "Winnie","lastname" : "Holland","age" : 38,"gender" : "M","address" : "198 Mill Lane","employer" : "Neteria","email" : "[e-mail]","city" : "Urie","state" : "IL"}},{"_index" : "bank","_type" : "_doc","_id" : "472","_score" : 4.89784,"_source" : {"account_number" : 472,"balance" : 25571,"firstname" : "Lee","lastname" : "Long","age" : 32,"gender" : "F","address" : "288 Mill Street","employer" : "Comverges","email" : "[e-mail]","city" : "Movico","state" : "MT"}},{"_index" : "bank","_type" : "_doc","_id" : "556","_score" : 4.4402957,"_source" : {"account_number" : 556,"balance" : 36420,"firstname" : "Collier","lastname" : "Odonnell","age" : 35,"gender" : "M","address" : "591 Nolans Lane","employer" : "Sultraxin","email" : "[e-mail]","city" : "Fulford","state" : "MD"}},{"_index" : "bank","_type" : "_doc","_id" : "934","_score" : 4.4402957,"_source" : {"account_number" : 934,"balance" : 43987,"firstname" : "Freida","lastname" : "Daniels","age" : 34,"gender" : "M","address" : "448 Cove Lane","employer" : "Vurbo","email" : "[e-mail]","city" : "Snelling","state" : "NJ"}},{"_index" : "bank","_type" : "_doc","_id" : "970","_score" : 4.388994,"_source" : {"account_number" : 970,"balance" : 19648,"firstname" : "Forbes","lastname" : "Wallace","age" : 28,"gender" : "M","address" : "990 Mill Road","employer" : "Pheast","email" : "[e-mail]","city" : "Lopezo","state" : "AK"}},{"_index" : "bank","_type" : "_doc","_id" : "345","_score" : 4.388994,"_source" : {"account_number" : 345,"balance" : 9812,"firstname" : "Parker","lastname" : "Hines","age" : 38,"gender" : "M","address" : "715 Mill Avenue","employer" : "Baluba","email" : "[e-mail]","city" : "Blackgum","state" : "KY"}},{"_index" : "bank","_type" : "_doc","_id" : "908","_score" : 4.388994,"_source" : {"account_number" : 908,"balance" : 45975,"firstname" : "Mosley","lastname" : "Holloway","age" : 31,"gender" : "M","address" : "929 Eldert Lane","employer" : "Anivet","email" : "[e-mail]","city" : "Biehle","state" : "MS"}},{"_index" : "bank","_type" : "_doc","_id" : "637","_score" : 4.388994,"_source" : {"account_number" : 637,"balance" : 3169,"firstname" : "Kathy","lastname" : "Carter","age" : 27,"gender" : "F","address" : "410 Jamison Lane","employer" : "Limage","email" : "[e-mail]","city" : "Ernstville","state" : "WA"}},{"_index" : "bank","_type" : "_doc","_id" : "449","_score" : 4.3719764,"_source" : {"account_number" : 449,"balance" : 41950,"firstname" : "Barnett","lastname" : "Cantrell","age" : 39,"gender" : "F","address" : "945 Bedell Lane","employer" : "Zentility","email" : "[e-mail]","city" : "Swartzville","state" : "ND"}},{"_index" : "bank","_type" : "_doc","_id" : "742","_score" : 4.3719764,"_source" : {"account_number" : 742,"balance" : 24765,"firstname" : "Merle","lastname" : "Wooten","age" : 26,"gender" : "M","address" : "317 Pooles Lane","employer" : "Tropolis","email" : "[e-mail]","city" : "Bentley","state" : "ND"}}]}
}

range
指定区间内的数字或者时间
操作符：gt大于，gte大于等于，lt小于，lte小于等于

查询余额大于或等于20000且小于等于30000的账户

[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": {
> "bool": {
> "must": { "match_all": {} },
> "filter": {
> "range": {
> "balance": {
> "gte": 20000,
> "lte": 30000
> }
> }
> }
> }
> }
> }
> '
{"took" : 9,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 217,"max_score" : 1.0,"hits" : [{"_index" : "bank","_type" : "_doc","_id" : "253","_score" : 1.0,"_source" : {"account_number" : 253,"balance" : 20240,"firstname" : "Melissa","lastname" : "Gould","age" : 31,"gender" : "M","address" : "440 Fuller Place","employer" : "Buzzopia","email" : "[e-mail]","city" : "Lumberton","state" : "MD"}},{"_index" : "bank","_type" : "_doc","_id" : "400","_score" : 1.0,"_source" : {"account_number" : 400,"balance" : 20685,"firstname" : "Kane","lastname" : "King","age" : 21,"gender" : "F","address" : "405 Cornelia Street","employer" : "Tri@Tribalog","email" : "kaneking@[e-mail]","city" : "Gulf","state" : "VT"}},{"_index" : "bank","_type" : "_doc","_id" : "520","_score" : 1.0,"_source" : {"account_number" : 520,"balance" : 27987,"firstname" : "Brandy","lastname" : "Calhoun","age" : 32,"gender" : "M","address" : "818 Harden Street","employer" : "Maxemia","email" : "[e-mail]","city" : "Sidman","state" : "OR"}},{"_index" : "bank","_type" : "_doc","_id" : "645","_score" : 1.0,"_source" : {"account_number" : 645,"balance" : 29362,"firstname" : "Edwina","lastname" : "Hutchinson","age" : 26,"gender" : "F","address" : "892 Pacific Street","employer" : "Essensia","email" : "[e-mail]","city" : "Dowling","state" : "NE"}},{"_index" : "bank","_type" : "_doc","_id" : "734","_score" : 1.0,"_source" : {"account_number" : 734,"balance" : 20325,"firstname" : "Keri","lastname" : "Kinney","age" : 23,"gender" : "M","address" : "490 Balfour Place","employer" : "Retrotex","email" : "[e-mail]","city" : "Salunga","state" : "PA"}},{"_index" : "bank","_type" : "_doc","_id" : "784","_score" : 1.0,"_source" : {"account_number" : 784,"balance" : 25291,"firstname" : "Mabel","lastname" : "Thornton","age" : 21,"gender" : "M","address" : "124 Louisiana Avenue","employer" : "Zolavo","email" : "[e-mail]","city" : "Lynn","state" : "AL"}},{"_index" : "bank","_type" : "_doc","_id" : "880","_score" : 1.0,"_source" : {"account_number" : 880,"balance" : 22575,"firstname" : "Christian","lastname" : "Myers","age" : 35,"gender" : "M","address" : "737 Crown Street","employer" : "Combogen","email" : "[e-mail]","city" : "Abrams","state" : "OK"}},{"_index" : "bank","_type" : "_doc","_id" : "14","_score" : 1.0,"_source" : {"account_number" : 14,"balance" : 20480,"firstname" : "Erma","lastname" : "Kane","age" : 39,"gender" : "F","address" : "661 Vista Place","employer" : "Stockpost","email" : "[e-mail]","city" : "Chamizal","state" : "NY"}},{"_index" : "bank","_type" : "_doc","_id" : "19","_score" : 1.0,"_source" : {"account_number" : 19,"balance" : 27894,"firstname" : "Schwartz","lastname" : "Buchanan","age" : 28,"gender" : "F","address" : "449 Mersereau Court","employer" : "Sybixtex","email" : "[e-mail]","city" : "Greenwich","state" : "KS"}},{"_index" : "bank","_type" : "_doc","_id" : "204","_score" : 1.0,"_source" : {"account_number" : 204,"balance" : 27714,"firstname" : "Mavis","lastname" : "Deleon","age" : 39,"gender" : "F","address" : "400 Waldane Court","employer" : "Lotron","email" : "[e-mail]","city" : "Stollings","state" : "LA"}}]}
}

elasticsearch-head
elasticsearch-head是集群管理、数据可视化、增删改查、查询语句可视化工具。从ES5版本后安装方式
和ES2以上的版本有很大的不同,在ES2中可以直接在bin目录下执行plugin install xxxx 来进行安装,但是
在ES5中这种安装方式变了,要想在ES5中安装Elasticsearch Head必须要安装NodeJs,然后通过NodeJS来
启动Head。
官网地址:https://github.com/mobz/elasticsearch-head

elasticsearch-head安装
下载相关软件并上传到服务器
官网有安装说明,可以通过git安装,也可以下载zip包解压安装
下载相应的软件包,并拷贝到ES集群的一个节点上(我这里拷贝到192.168.100.20这台,也就是vm2上)

安装nodejs（注：这里是上传）

[root@vm2 ~]# ls
accounts.json    Documents                initial-setup-ks.cfg            Pictures   Videos
anaconda-ks.cfg  Downloads                Music                           Public
Desktop          elasticsearch-6.5.2.rpm  node-v10.24.1-linux-x64.tar.xz  Templates
[root@vm2 ~]# tar xf node-v10.24.1-linux-x64.tar.xz -C /usr/local/
[root@vm2 ~]# mv /usr/local/node-v10.24.1-linux-x64/ /usr/local/nodejs/
[root@vm2 ~]# ls /usr/local/nodejs/bin/npm
/usr/local/nodejs/bin/npm         //确认有此命令
[root@vm2 ~]# ln -s /usr/local/nodejs/bin/npm /bin/npm
[root@vm2 ~]# ln -s /usr/local/nodejs/bin/node /bin/node

安装es-head
安装方法1(需要网速好):

[root@vm2 ~]# cd /etc/yum.repos.d/
[root@vm2 yum.repos.d]# ls
CentOS-Base.repo  CentOS-Debuginfo.repo  CentOS-Media.repo    CentOS-Vault.repo
CentOS-CR.repo    CentOS-fasttrack.repo  CentOS-Sources.repo
[root@vm2 yum.repos.d]# rm -rf *
[root@vm2 yum.repos.d]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed
100  2523  100  2523    0     0  10638      0 --:--:-- --:--:-- --:--:-- 10600
[root@vm2 yum.repos.d]# cd
[root@vm2 ~]# yum -y install git
.....

[root@vm2 ~]# git clone https://github.com/mobz/elasticsearch-head.git
Cloning into 'elasticsearch-head'...
remote: Enumerating objects: 4377, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 4377 (delta 12), reused 34 (delta 12), pack-reused 4337 (from 1)
Receiving objects: 100% (4377/4377), 2.54 MiB | 696.00 KiB/s, done.
Resolving deltas: 100% (2429/2429), done.
[root@vm2 ~]# cd elasticsearch-head/先使用npm安装grunt
npm(node package manager):node包管理工具,类似yum
Grunt是基于Node.js的项目构建工具
[root@vm2 elasticsearch-head]# npm install -g grunt-cli
/usr/local/nodejs/bin/grunt -> /usr/local/nodejs/lib/node_modules/grunt-cli/bin/grunt
+ grunt-cli@1.5.0
added 56 packages from 68 contributors in 19.056s安装时间较久,还会在网上下载phantomjs包
[root@vm2 elasticsearch-head]# npm install

安装可能有很多错误,我这里出现了下面的错误(重点是注意红色的ERR!,黄色的WARN不用管)

解决方法：

[root@vm2 elasticsearch-head]# npm install phantomjs-prebuilt@2.1.16 --ignorescript
此命令执行后不用再返回去执行npm install了，直接开始启动[root@vm2 elasticsearch-head]# nohup npm run start &
[5] 63180

安装方法二 : (网速特别慢导致安装时间过长的话可以尝试以下方法)

git clone慢的话就使用下载好的zip压缩包解压安装
[root@vm2 ~]# unzip elasticsearch-head-master.zip -d /usr/local/
[root@vm2 ~]# mv /usr/local/elasticsearch-head-master/ /usr/local/es-head/
[root@vm2 ~]# cd /usr/local/es-head/
[root@vm2 es-head]# npm install -g grunt-cli --
registry=http://registry.npm.taobao.org[root@vm2 es-head]# npm install --registry=http://registry.npm.taobao.org
当安装出现下载phantomjs软件包特别慢的时候，可以ctrl+c取消，拷贝下载好的phantomjs包到特定位置
再重新安装
[root@vm2 es-head]# cp phantomjs-2.1.1-linux-x86_64.tar.bz2 /tmp/phantomjs/
注意：phantomjs请改成自己的绝对路径
[root@vm2 es-head]# npm install --registry=http://registry.npm.taobao.org[root@vm2 es-head]# nohup npm run start &

注意: 运行 nohup npm run start & 必须要先cd到es-head的目录

第4步:浏览器访问
浏览器访问 http://es-head节点IP:9100 ,并在下面的地址里把localhost改为es-head节点IP(浏览器与
es-head不是同一节点就要做)

第5步: 修改ES集群配置文件,并重启服务：

[root@vm1 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-clusternode.name: 192.168.100.10
node.master: falsepath.data: /var/lib/elasticsearchpath.logs: /var/log/elasticsearchnetwork.host: 0.0.0.0http.port: 9200discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]http.cors.enabled: true
http.cors.allow-origin: "*"      //加上最后这两句
~  

[root@vm2 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-clusternode.name: 192.168.100.20
node.master: truepath.data: /var/lib/elasticsearchpath.logs: /var/log/elasticsearchnetwork.host: 0.0.0.0http.port: 9200discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]http.cors.enabled: true
http.cors.allow-origin: "*"     //加上最后这两句
~  

[root@vm1 ~]# systemctl restart elasticsearch

[root@vm2 ~]# systemctl restart elasticsearch

第6步: 再次连接就可以看到信息了

新建个索引试试

删除此索引

es-head查询验证：

logstash简介
logstash是一个开源的数据采集工具,通过数据源采集数据.然后进行过滤,并自定义格式输出到目的地。
数据分为:

1. 结构化数据 如:mysql数据库里的表等
2. 半结构化数据 如: xml,yaml,json等
3. 非结构化数据 如:文档,图片,音频,视频等
   logstash可以采集任何格式的数据,当然我们这里主要是讨论采集系统日志,服务日志等日志类型数据

官方产品介绍:https://www.elastic.co/cn/products/logstash

input插件: 用于导入日志源 (配置必须)
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
filter插件: 用于过滤(不是配置必须的)https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
output插件: 用于导出(配置必须)
https://www.elastic.co/guide/en/logstash/current/output-plugins.html

logstash部署

在logstash服务器上确认openjdk安装：

[root@vm3 ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

在logstash服务器上安装logstash：（注：这里是上传）：

[root@vm3 ~]# rz -E
rz waiting to receive.[root@vm3 ~]# rpm -ivh logstash-6.5.2.rpm 
warning: logstash-6.5.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...1:logstash-1:6.5.2-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

配置logstash主配置文件：

[root@vm3 ~]# vim /etc/logstash/logstash.yml 
......
path.data: /var/lib/logstashpath.config: /etc/logstash/conf.d     //打开注释,并加上配置目录路径http.host: "192.168.100.30"   //打开注释,并改为本机IP(这是用于xpack监控用,但要收费,所以在这里不配
置也可以)path.logs: /var/log/logstash

启动测试：

[root@vm3 ~]# cd /usr/share/logstash/bin
使用下面的空输入和空输出启动测试一下
[root@vm3 bin]# ./logstash -e 'input {stdin {}} output {stdout {}}'
运行后，输入字符将被stdout做为标准输出内容输出
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2025-10-23 19:52:30.811 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2025-10-23 19:52:30.820 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.5.2"}
[INFO ] 2025-10-23 19:52:32.668 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2025-10-23 19:52:32.795 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x59e19f73 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2025-10-23 19:52:32.835 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2025-10-23 19:52:32.982 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
abc
{"@version" => "1","message" => "abc","@timestamp" => 2025-10-23T11:52:41.921Z,"host" => "vm3.example.com"
}
123
{"@version" => "1","message" => "123","@timestamp" => 2025-10-23T11:52:46.314Z,"host" => "vm3.example.com"
}

关闭启动

测试能启动成功后,ctrl+c取消,则关闭了

另一种验证方法：

[root@vm3 ~]# vim /etc/logstash/conf.d/test.conf
input {stdin {}
}
filter {
}
output {stdout {codec => rubydebug}
}
~ 
[root@vm3 ~]# cd /usr/share/logstash/bin
[root@vm3 bin]# pwd
/usr/share/logstash/bin
[root@vm3 bin]# ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/test.conf -t
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2025-10-23T20:15:12,379][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2025-10-23T20:15:13,314][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash--path.settings 指定logstash主配置文件目录
-f 指定片段配置文件
-t 测试配置文件是否正确
codec => rubydebug这句可写可不定,默认就是这种输出方式

[root@vm3 bin]# ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2025-10-23T20:17:04,902][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2025-10-23T20:17:04,912][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.2"}
[2025-10-23T20:17:04,926][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"236820fc-5c98-483f-8551-d4fd0e76598b", :path=>"/var/lib/logstash/uuid"}
[2025-10-23T20:17:06,416][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2025-10-23T20:17:06,478][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x16d5eb4d sleep>"}
The stdin plugin is now waiting for input:
[2025-10-23T20:17:06,503][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2025-10-23T20:17:06,659][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
haha
{"@timestamp" => 2025-10-23T12:18:11.474Z,"@version" => "1","message" => "haha","host" => "vm3.example.com"
}
hehe
{"@timestamp" => 2025-10-23T12:18:18.993Z,"@version" => "1","message" => "hehe","host" => "vm3.example.com"
}-r参数很强大,会动态装载配置文件,也就是说启动后,可以不用重启修改配置文件

日志采集
采集messages日志
这里以/var/log/messages为例,只定义input输入和output输出,不考虑过滤

[root@vm3 bin]# vim /etc/logstash/conf.d/test.conf 
input {file {path => "/var/log/messages"start_position => "beginning"}
}       
output {elasticsearch{hosts => ["192.168.100.20:9200"]index => "test-%{+YYYY.MM.dd}"}           
}               
~  
[root@vm3 bin]# ./logstash --path.settings /etc/logstash/ -r -f /etc/logstash/conf.d/test.conf &
[4] 11335后台运行如果要杀掉,请使用pkill java或ps查看PID再kill -9清除

通过浏览器访问es-head验证 ：

请自行练习验证:
1, 在logstash那台服务器上做一些操作(比如,重启下sshd服务), 让/var/log/message有新的日志信息，
然后验证es-head里的数据
结果: 会自动更新, 浏览器刷新就能在es-head上看到更新的数据
2, kill掉logstash进程(相当于关闭), 也做一些操作让/var/log/message日志有更新，然后再次启动
logstash
结果: 会自动连上es集群, es-head里也能查看到数据的更新

采集多日志源

[root@vm3 bin]# vim /etc/logstash/conf.d/test.conf 
input {file {path => "/var/log/messages"start_position => "beginning"type => "messages"}file {path => "/var/log/yum.log"start_position => "beginning"type => "yum"}
}
filter {
}
output {if [type] == "messages" {elasticsearch {hosts => ["192.168.100.20:9200","192.168.100.10:9200"]index => "messages-%{+YYYY-MM-dd}"}}if [type] == "yum" {elasticsearch {hosts => ["192.168.100.20:9200","192.168.100.10:9200"]index => "yum-%{+YYYY-MM-dd}"}}
}
~