ELK运维之路(Filebeat第二章-7.17.24)
1.采集Java日志
1.1 Java日志
2025-10-09T06:05:38.705828345Z 2025-10-09 06:05:38.705+0000 [id=68919] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.plugins.nodejs.tools.NodeJSInstaller
2025-10-09T06:05:38.706397912Z 2025-10-09 06:05:38.706+0000 [id=68919] INFO hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
2025-10-10T06:05:17.126071592Z 2025-10-10 06:05:17.117+0000 [id=75798] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
2025-10-10T06:05:19.284996467Z 2025-10-10 06:05:19.284+0000 [id=75798] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
2025-10-10T06:05:21.690331304Z 2025-10-10 06:05:21.689+0000 [id=75798] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.plugins.gradle.GradleInstaller
2025-10-10T06:05:23.790708459Z 2025-10-10 06:05:23.790+0000 [id=75798] WARNING h.m.DownloadService$Downloadable#updateNow: No tool installer metadata found for jenkins.plugins.nodejs.tools.MirrorNodeJSInstaller
2025-10-10T06:05:26.010118395Z 2025-10-10 06:05:26.008+0000 [id=75798] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.plugins.nodejs.tools.NodeJSInstaller
2025-10-11T06:05:14.258254636Z 2025-10-11 06:05:14.256+0000 [id=82672] INFO hudson.util.Retrier#start: Attempt #1 to do the action check updates server
2025-10-11T06:05:18.285376626Z 2025-10-11 06:05:18.278+0000 [id=82672] INFO hudson.util.Retrier#start: The attempt #1 to do the action check updates server failed with an allowed exception:
2025-10-11T06:05:18.285425973Z java.net.SocketException: Network is unreachableat java.base/sun.nio.ch.Net.connect0(Native Method)at java.base/sun.nio.ch.Net.connect(Unknown Source)at java.base/sun.nio.ch.Net.connect(Unknown Source)at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)at java.base/java.net.SocksSocketImpl.connect(Unknown Source)at java.base/java.net.Socket.connect(Unknown Source)at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)at java.base/sun.net.NetworkClient.doConnect(Unknown Source)at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)at java.base/sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)at java.base/sun.net.www.protocol.https.HttpsClient.New(Unknown Source)at java.base/tractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown at hudson.model.DownloadService.loadJSON(DownloadService.java:122)at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:240)at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:235)at hudson.PluginManager.checkUpdatesServer(PluginManager.java:2179)at hudson.util.Retrier.start(Retrier.java:62)at hudson.PluginManager.doCheckUpdatesServer(PluginManager.java:2150)at jenkins.DailyCheck.execute(DailyCheck.java:93)at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:110)at java.base/java.lang.Thread.run(Unknown Source)
2025-10-11T06:05:18.285644291Z 2025-10-11 06:05:18.280+0000 [id=82672] INFO hudson.util.Retrier#start: Calling the listener of the allowed exception 'Network is unreachable' at the attempt #1 to do the action check updates server
2025-10-11T06:05:18.285654331Z 2025-10-11 06:05:18.280+0000 [id=82672] INFO hudson.util.Retrier#start: Attempted the action check updates server for 1 time(s) with no success
2025-10-11T06:05:18.285660731Z 2025-10-11 06:05:18.281+0000 [id=82672] SEVERE hudson.PluginManager#doCheckUpdatesServer: Error checking update sites for 1 attempt(s). Last exception was: SocketException: Network is unreachable
2025-10-12T06:05:14.255756740Z 2025-10-12 06:05:14.254+0000 [id=89527] INFO hudson.util.Retrier#start: Attempt #1 to do the action check updates server
2025-10-12T06:05:28.705898887Z 2025-10-12 06:05:28.704+0000 [id=89527] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
2025-10-12T06:05:31.712896068Z 2025-10-12 06:05:31.712+0000 [id=89527] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Ant.AntInstaller1.1.1 采集常规Java日志
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-javalog-base.yml
filebeat.inputs:
- type: logenable: truetags: ["javalogbase"]paths:- /root/java_log/java-log.logfields:project: javalog: javabaselogfields_under_root: falseoutput.elasticsearch:hosts: ["http://192.168.1.99:9201","http://192.168.1.99:9202","http://192.168.1.99:9203"]username: "elastic"password: "123456"index: "filebeat-javalog-base-%{+yyyy.MM.dd}"setup.ilm.enabled: falsesetup.template.name: "filebeat-javalog-base"
setup.template.pattern: "filebeat-javalog-base*"setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 2
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# ./filebeat -e -c filebeat-javalog-base.yml --path.data /tmp/filebeat-javabase
image-20251016181018529
1.1.2 采集收集多行日志(一般为Java错误日志)
虽然正常抓取到了java日志，但是会发现一个问题当java报错的时候是多行报错，那你再抓取日志的时候就把本该在一起的报错分成n行抓取，造成日志无法阅读，为了解决这个问题那么就要对日志开启多行匹配，然后通过正则来匹配错误日志格式
image-20251016181238333
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-javalog-mutli.yml
filebeat.inputs:
- type: logenable: truetags: ["javalogmutil"]paths:- /root/java_log/java-log.logmultiline.type: pattern # 多行匹配方式 还有一个count模式multiline.pattern: '^\d{4}-\d{2}-\d{2}' # 匹配正则，日志开头2025-10-11#multiline.pattern: '^\[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2},\d{3}\]' # 日志开头[2022-05-11T16:11:33,652]multiline.negate: true # 贪婪匹配multiline.match: after # 匹配方式 还有个before#日志的过滤#include_lines: ['^ERR', '^WARN', 'info'] # 日志中包含指定内容才会进行采集，这里是必须以ERR和WARN开头(匹配会区分大小写)#include_lines: ['info'] # 日志中只要包含info就会采集#exclude_lines: ['^DBG'] # 当有指定字段，会排除不采集
output.elasticsearch:hosts: ["http://192.168.1.99:9201","http://192.168.1.99:9202","http://192.168.1.99:9203"]username: "elastic"password: "123456"index: "filebeat-javalog-mutli-%{+yyyy.MM.dd}"setup.ilm.enabled: falsesetup.template.name: "filebeat-javalog-mutli"
setup.template.pattern: "filebeat-javalog-mutli*"setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 2
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# ./filebeat -e -c filebeat-javalog-mutli.yml --path.data /tmp/filebeat-javamutil
image-20251016182332536
image-20251016183457901
2.Nginx错误日志收集(通过过滤)
在收集Nginx的错误日志的时候有很多无用的信息，是我们不需要的，我们分析下Nginx的错误日志，下面的日志我们可以看到除了error信息，还有有warn info 等其他状态的信息出现。那么我们可以通过error关键字来只抓取有error状态的日志信息，这样可以减少无效日志采集
2.1 Nginx错误日志
2025/06/18 12:54:14 [warn] 25921#25921: *3098732 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000029, client: 192.168.1.188, server: nextcloudv.wolf.eg, request: "PUT /remote.php/dav/files/admin/%E7%AC%94%E8%AE%B0%E6%96%87%E4%BB%B6/Gitlab/%E5%9B%BE%E7%89%87%E6%96%87%E4%BB%B6/01%E8%8A%82%E7%82%B901.png HTTP/1.1", host: "nextcloudv.wolf.eg"
2025/06/18 12:54:17 [warn] 25921#25921: *3098732 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000030, client: 192.168.1.188, server: nextcloudv.wolf.eg, request: "PUT /remote.php/dav/files/admin/%E7%AC%94%E8%AE%B0%E6%96%87%E4%BB%B6/Gitlab/%E5%9B%BE%E7%89%87%E6%96%87%E4%BB%B6/01%E8%8A%82%E7%82%B903.png HTTP/1.1", host: "nextcloudv.wolf.eg"
2025/06/18 12:54:28 [warn] 25921#25921: *3098732 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000031, client: 192.168.1.188, server: nextcloudv.wolf.eg, request: "PUT /remote.php/dav/files/admin/%E7%AC%94%E8%AE%B0%E6%96%87%E4%BB%B6/Gitlab/Jenkins%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90%E4%BB%8E%E5%85%A5%E9%97%A8%E5%88%B0%E7%B2%BE%E9%80%9A.pdf HTTP/1.1", host: "nextcloudv.wolf.eg"
2025/06/18 12:54:37 [warn] 25920#25920: *3098724 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000032, client: 192.168.1.188, server: nextcloudv.wolf.eg, request: "PUT /remote.php/dav/files/admin/%E7%AC%94%E8%AE%B0%E6%96%87%E4%BB%B6/%E8%BF%90%E7%BB%B4%E5%AF%BC%E5%9B%BE/nginx.km HTTP/1.1", host: "nextcloudv.wolf.eg"
2025/06/18 18:36:11 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/difyv.wolf.eg.com.conf:2
2025/06/18 18:36:11 [emerg] 15411#15411: invalid port in upstream "http://192.168.1.18:80" in /usr/local/nginx/conf/conf.d/difyv.wolf.eg.com.conf:24
2025/06/18 18:39:05 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/difyv.wolf.eg.com.conf:2
2025/06/18 18:39:05 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/drawio.muscledog.ink.conf:2
2025/06/18 18:39:05 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/fastgpt.wolf.eg.conf:2
2025/06/18 18:39:05 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/gitlab.wolf.eg.conf:2
2025/07/02 17:26:59 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/difyv.wolf.eg.com.conf:2
2025/07/02 17:26:59 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/drawio.muscledog.ink.conf:2
2025/07/02 17:26:59 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/fastgpt.wolf.eg.conf:2
2025/07/02 17:26:59 [warn] 15411#15411: protocol options redefined for 0.0.0.0:443 in /usr/local/nginx/conf/conf.d/gitlab.wolf.eg.conf:2
2025/07/07 18:43:38 [warn] 14392#14392: *5392785 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000033, client: 192.168.1.188, server: nextcloudv.wolf.eg, request: "PUT /remote.php/dav/files/admin/%E6%88%91%E7%9A%84%E9%85%8D%E7%BD%AE/PicGo-2.4.0-beta.10-x64.dmg HTTP/1.1", host: "nextcloudv.wolf.eg"
2025/07/09 20:26:45 [error] 14391#14391: *5654551 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.13, server: chatnext.wolf.eg, request: "GET / HTTP/2.0", upstream: "http://192.168.1.18:8090/", host: "chatnext.wolf.eg"
2025/07/10 10:04:00 [error] 14391#14391: *5721682 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.13, server: chatnext.wolf.eg, request: "GET / HTTP/2.0", upstream: "http://192.168.1.18:8090/", host: "chatnext.wolf.eg"2.2 Filebeat配置
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-01.yml
filebeat.inputs:
- type: logenable: truetags: ["nginx-access"]paths:- /root/nginx_log/nginx01.logfields:log: nginx-accessfields_under_root: falsejson.keys_under_root: true- type: logtags: ["nginx-error"]paths:- /root/nginx_log/error.loginclude_lines: ['\[error\]']fields: log: nginx-errorfields_under_root: truejson.keys_under_root: trueoutput.elasticsearch:hosts: ["http://192.168.1.99:9201","http://192.168.1.99:9202","http://192.168.1.99:9203"]username: "elastic"password: "123456"indices:- index: "filebeat-nginx-access-%{+yyyy.MM.dd}"when.contains:tags: "nginx-access"- index: "filebeat-nginx-error-%{+yyyy.MM.dd}"when.contains:tags: "nginx-error"setup.ilm.enabled: falsesetup.template.name: "filebeat-nginxlog"
setup.template.pattern: "filebeat-nginxlog*"setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 2在抓取Nginx错误入职的时候会根据 include_lines: 的条件只抓取包含[error]关键字的错误日志
3.FileBeat 输出至 Redis
filebeat.inputs:
- type: filestreampaths:- /root/nginx_log/nginxaccess.logtags: ['access']parsers: - ndjson:keys_under_root: trueoutput.redis:hosts: ["192.168.1.47:6379"]password: "my_password"key: "filebeat-log"db: 3 # 数据库编号timeout: 54.FileBeat 输出至 Logsatsh
output.logstash:hosts: ["127.0.0.1:5044"]