当前位置： 首页 > news >正文

ELK运维之路(Filebeat第一章-7.17.24)

news 2025/10/22 9:23:40

1. 部署Filebeat

1.1 下载解压Filebeat

# 下载filebeat并解压到自己习惯的目录
root@ubuntu2204test99:~# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.24-linux-x86_64.tar.gz
root@ubuntu2204test99:~# mv filebeat-7.17.24-linux-x86_64 filebeat-7.17.24
root@ubuntu2204test99:~# mv filebeat-7.17.24 /usr/local/
root@ubuntu2204test99:~# mkdir /usr/local/filebeat-7.17.24/{logs,data}

1.2 使用Systemctl接管Filebeat服务

# 配置systemd启动filebeat
vi /lib/systemd/system/filebeat.service
[Unit]
Description=filebeat is a lightweight shipper for metrics.
Documentation=https://www.elastic.co/products/beats/filebeat
Wants=network-online.target
After=network-online.target
[Service]
Environment="BEAT_LOG_OPTS=-e"
Environment="BEAT_CONFIG_OPTS=-c /usr/local/filebeat-7.17.24/filebeat.yml"
Environment="BEAT_PATH_OPTS=-path.home /usr/local/filebeat-7.17.24 -path.config /usr/local/filebeat-7.17.24 -path.data /usr/local/filebeat-7.17.24/data -path.logs /usr/local/filebeat-7.17.24/logs"
ExecStart=/usr/local/filebeat-7.17.24/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
Restart=always
[Install]
WantedBy=multi-user.target# 测试启动
root@ubuntu2204k8s220:~# systemctl daemon-reload
root@ubuntu2204k8s220:~# systemctl start filebeat.service
root@ubuntu2204k8s220:~# systemctl enable filebeat.service# 查看启动状态
root@ubuntu2204test99:~# systemctl status filebeat.service
● filebeat.service - filebeat is a lightweight shipper for metrics.Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)Active: active (running) since Thu 2025-10-09 09:55:56 UTC; 14s agoDocs: https://www.elastic.co/products/beats/filebeatMain PID: 542553 (filebeat)Tasks: 9 (limit: 9388)Memory: 32.4MCPU: 258msCGroup: /system.slice/filebeat.service└─542553 /usr/local/filebeat-7.17.24/filebeat -e -c /usr/local/filebeat-7.17.24/filebeat.yml -path.home /usr/local/filebeat-7.17.24 -path.config /usr/local>Oct 09 09:55:59 ubuntu2204test99 filebeat[542553]: 2025-10-09T09:55:59.685Z        INFO        memlog/store.go:119        Loading data file of '/usr/local/filebeat-7.17>
Oct 09 09:55:59 ubuntu2204test99 filebeat[542553]: 2025-10-09T09:55:59.685Z        INFO        memlog/store.go:124        Finished .....
lines 1-21/21 (END)

2. Filebeat默认配置

root@ubuntu2204test99:~# egrep -v "^*#|^$" /usr/local/filebeat-7.17.24/filebeat.yml
filebeat.inputs:
- type: filestreamid: my-filestream-idenabled: falsepaths:- /var/log/*.log
filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: false
setup.template.settings:index.number_of_shards: 1
setup.kibana:
output.elasticsearch:hosts: ["localhost:9200"]
processors:- add_host_metadata:when.not.contains.tags: forwarded- add_cloud_metadata: ~- add_docker_metadata: ~- add_kubernetes_metadata: ~

3.采集日志示例

将内容直接输出到ES当中

3.1 采集常规格式的Nginx日志

3.1.1 常规格式日志

root@ubuntu2204test99:~# cat nginx_log/nginx01.log
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/select2/select2.full.min.js HTTP/1.1" 200 78556 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/markdown-it.min.js HTTP/1.1" 200 47613 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/select2/select2.full.min.js HTTP/1.1" 200 78556 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/markdown-it.min.js HTTP/1.1" 200 47613 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"

3.1.2 Filebeat配置

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-nginxlog-base.yml
filebeat.inputs:
- type: logenable: truetags: ["nginx"]paths:- /root/nginx_log/nginx01.logfields:python: truejava: falsefields_under_root: false
output.elasticsearch:hosts: ["http://192.168.1.100:9200","http://192.168.1.101:9200","http://192.168.1.102:9200"]username: "" # 账号password: "" # 密码index: "filebeat-nginxlog-base-%{+yyyy.MM.dd}"setup.ilm.enabled: falsesetup.template.name: "filebeat-nginxlog-base"
setup.template.pattern: "filebeat-nginxlog-base*"setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 1index.number_of_replicas: 1# 临时启动filebeat
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# pwd
/usr/local/filebeat-7.17.24
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# ./filebeat -e -c filebeat-nginxlog-base.yml --path.data /tmp/filebeat01/

image-20251016155656555

image-20251016155656555

image-20251016145413084

image-20251016145413084

image-20251016155802221

image-20251016155802221

image-20251016155828824

image-20251016155828824

image-20251016155942995

image-20251016155942995

3.2 采集Json格式的Nginx日志

这里有一个前提，需要提前将Nginx的日志格式修改为Json，如果不知道如何修改可以参考网上

3.2.1 Nginx Json日志格式

# Nginx Json格式日志示例
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat /root/nginx_log/nginxjson.log
{"@timestamp":"2024-11-26T15:47:34+08:00","@source":"192.168.1.39","@nginx_fields":{"http_x_forwarded_for":"","request":"POST /api/v4/jobs/request HTTP/1.1","status":"204","body_bytes_sent":"0","http_referer":"","client":"192.168.1.47","request_time":"0.001","upstream_response_time":"0.002","upstream_addr":"192.168.1.40:10080","request_method":"POST","domain":"gitlab.muscledog.top","url":"/api/v4/jobs/request","args":"","request_body":"{\"info\":{\"name\":\"gitlab-runner\",\"version\":\"17.3.1\",\"revision\":\"66269445\",\"platform\":\"linux\",\"architecture\":\"amd64\",\"executor\":\"shell\",\"shell\":\"bash\",\"features\":{\"variables\":true,\"image\":false,\"services\":false,\"artifacts\":true,\"cache\":true,\"fallback_cache_keys\":true,\"shared\":true,\"upload_multiple_artifacts\":true,\"upload_raw_artifacts\":true,\"session\":true,\"terminal\":true,\"refspecs\":true,\"masking\":true,\"proxy\":false,\"raw_variables\":true,\"artifacts_exclude\":true,\"multi_build_steps\":true,\"trace_reset\":true,\"trace_checksum\":true,\"trace_size\":true,\"vault_secrets\":true,\"cancelable\":true,\"return_exit_code\":true,\"service_variables\":false,\"service_multiple_aliases\":false,\"image_executor_opts\":false,\"service_executor_opts\":false,\"cancel_gracefully\":true},\"config\":{\"gpus\":\"\"}},\"token\":\"glrt-BetamQqihBBxfUghD5p6\",\"system_id\":\"s_9a48451a140f\",\"last_update\":\"ac9b5d563d539d7a3a08029a60d796ff\"}","http_user_agent":"gitlab-runner 17.3.1 (17-3-stable; go1.22.5; linux/amd64)","remote_addr":"192.168.1.47","proxy_add_x_forwarded_for":"192.168.1.47"}}
{"@timestamp":"2024-11-26T15:47:37+08:00","@source":"192.168.1.39","@nginx_fields":{"http_x_forwarded_for":"","request":"POST /api/v4/jobs/request HTTP/1.1","status":"204","body_bytes_sent":"0","http_referer":"","client":"192.168.1.47","request_time":"0.001","upstream_response_time":"0.002","upstream_addr":"192.168.1.40:10080","request_method":"POST","domain":"gitlab.muscledog.top","url":"/api/v4/jobs/request","args":"","request_body":"{\"info\":{\"name\":\"gitlab-runner\",\"version\":\"17.3.1\",\"revision\":\"66269445\",\"platform\":\"linux\",\"architecture\":\"amd64\",\"executor\":\"shell\",\"shell\":\"bash\",\"features\":{\"variables\":true,\"image\":false,\"services\":false,\"artifacts\":true,\"cache\":true,\"fallback_cache_keys\":true,\"shared\":true,\"upload_multiple_artifacts\":true,\"upload_raw_artifacts\":true,\"session\":true,\"terminal\":true,\"refspecs\":true,\"masking\":true,\"proxy\":false,\"raw_variables\":true,\"artifacts_exclude\":true,\"multi_build_steps\":true,\"trace_reset\":true,\"trace_checksum\":true,\"trace_size\":true,\"vault_secrets\":true,\"cancelable\":true,\"return_exit_code\":true,\"service_variables\":false,\"service_multiple_aliases\":false,\"image_executor_opts\":false,\"service_executor_opts\":false,\"cancel_gracefully\":true},\"config\":{\"gpus\":\"\"}},\"token\":\"glrt-BetamQqihBBxfUghD5p6\",\"system_id\":\"s_9a48451a140f\",\"last_update\":\"ac9b5d563d539d7a3a08029a60d796ff\"}","http_user_agent":"gitlab-runner 17.3.1 (17-3-stable; go1.22.5; linux/amd64)","remote_addr":"192.168.1.47","proxy_add_x_forwarded_for":"192.168.1.47"}}

3.2.2 Filebeat配置

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-nginxlog-json.yml
filebeat.inputs:
- type: logenable: truetags: ["nginxjson"]paths:- /root/nginx_log/nginxjson.logfields:python: truejava: falsefields_under_root: falsejson.keys_under_root: true #对Json格式的日志进行解析并放在顶级字段,如果不是json格式会有大量报错setup.ilm.enabled: false
output.elasticsearch:hosts: ["http://192.168.1.99:9201","http://192.168.1.99:9202","http://192.168.1.99:9203"]username: "elastic" # 账号password: "123456" # 密码index: "filebeat-nginxlog-json-%{+yyyy.MM.dd}"setup.template.name: "filebeat-nginxlog-json"
setup.template.pattern: "filebeat-nginxlog-json*"setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 1index.number_of_replicas: 1
# 临时启动filebeat
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# pwd
/usr/local/filebeat-7.17.24
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# ./filebeat -e -c filebeat-nginxlog-json.yml --path.data /tmp/filebeat01/

可以看到json格式的日志，被解析后会非常清晰的进行展示，而不是像普通问个格式那样都堆在一块。目前我们公司在nginx日志方面是使用的json格式的。

3.3 使用Filebeat模块采集Nginx日志

Filebeat的模块会对Nginx日志进行序列化解析处理，然后再录入到ES，一般模块这种不太建议采用,仅做了解

3.3.1 修改模块的内容(需要先开启Filebeat对应模块)

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# pwd
/usr/local/filebeat-7.17.24
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# vi modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-nginx.html- module: nginx# Access logsaccess:enabled: truevar.paths: ["/root/nginx_log/nginx01.log"]# Set custom paths for the log files. If left empty,# Filebeat will choose the paths depending on your OS.#var.paths:# Error logserror:enabled: false# Set custom paths for the log files. If left empty,# Filebeat will choose the paths depending on your OS.#var.paths:# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logsingress_controller:enabled: false# Set custom paths for the log files. If left empty,# Filebeat will choose the paths depending on your OS.#var.paths:

3.3.2 Filebeat模板

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-nginxlog-modules.yml
# 加载模块配置
filebeat.config.modules:path: /usr/local/filebeat-7.17.24/modules.d/*.ymlreload.enabled: false#reload.period: 10s
output.elasticsearch:hosts: ["http://192.168.1.99:9201","http://192.168.1.99:9202","http://192.168.1.99:9203"]username: "elastic"password: "123456"index: "filebeat-nginxlog-modules-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "filebeat-nginxlog-modules"
setup.template.pattern: "filebeat-nginxlog-modules*"
setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 2

http://www.dtcms.com/a/490205.html

相关文章：

  • 开源接口管理工具深度横评，swagger vs PostIn哪个更适合你？
  • list的使用和模拟实现
  • 群辉怎么做网站服务器专做投放广告网站
  • 【34】MFC入门到精通——MFC 控件 ComboBox 运行点击控件下拉框 “终止“、“重试“、“忽略“、“引发异常”
  • 论文见解：REACT：在语言模型中协同推理和行动
  • Megatron-DeepSpeed 方案
  • 停止检索！刚刚这本期刊被数据库剔除！
  • Flink-Kafka 数据倾斜问题解决方案
  • html 实现鼠标滑动点亮横轴
  • 连接 TDengine 遇到报错 “failed to connect to server, reason: Connection refused” 怎么办？
  • Web自动化（三方库：Selenium）
  • 网站单页在线制作软件长尾关键词是什么意思
  • 生成静态页面网站源码信息门户网站是什么
  • 在阿里云CentOS服务器上使用Certbot为Nginx配置SSL证书
  • 如何优化网站打开速度网站运营和管理
  • 华为数通认证学习难吗？需掌握哪些核心知识点？
  • 【vue】导出excel
  • WiFi模块远程连接APP：wifi模块的应用
  • 关闭电脑的“快速启动”功能
  • 【MongoDB 基本语法】数据库和集合的基本操作--探秘 MongoDB
  • 提示词管理器设计：从需求到用户体验的高效落地逻辑
  • idea 网站开发无远低代码开发平台
  • FFmpeg 基本API avio_open函数内部调用流程分析
  • 27.Redisson基本使用和可重入性
  • 凡客做网站阎良网站建设公司
  • Uniapp微信小程序开发：全局变量的使用
  • 【优先队列介绍】
  • 初识HTNL
  • 笔试强训(四)
  • 【深入浅出PyTorch】--7.2.PyTorch可视化2