当前位置：首页 > news >正文

logstash采集springboot微服务日志

news 2025/7/28 7:07:05

前面一篇文章介绍了logstash采集nginx日志，这次介绍一下通过logstash采集springboot微服务中指定格式的日志。

需求

现有日志示例格式如下：

2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}
2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}
2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}

需要采集“logstashVO:”之后的JSON格式字符串，该功能主要通过网关服务收集用户访问行为，当然根据用户需求可以做的事情还可以很多，这里就不一一列举。

logstash配置

input {file {path => ["/data/logs/spring.log"]type => "test"start_position => "beginning"}
}
filter {if "c.test.authgateway.filter.GateWayFilter  : logstashVO:" not in [message] { drop {}}ruby {code => "event.set('@timestamp', LogStash::Timestamp.at(event.get('@timestamp').time.localtime + 8*60*60))"}grok {match => { "message" => "c\.test\.authgateway\.filter\.GateWayFilter  \: logstashVO\:%{GREEDYDATA:logstashVO}"}}json {source => "logstashVO"}json {source => "body"target => "body"}mutate {remove_field => ["@version","event","host","logstashVO"]}   
}output {stdout {codec => rubydebug}kafka {bootstrap_servers => "10.16.39.121:9092"topic_id => "test2-logs"codec => jsonacks => "all"retries => 5batch_size => 16384}
}

注意这里的输入input.file.path对应的路径一定要是镜像中的路径/data/logs/，而不能为宿主机里的路径/data/public/logstash/logs

启动镜像

docker run --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
--privileged=true --network=host --restart=always --name logstash \
-v /data/public/logstash/config/:/usr/share/logstash/config/ \
-v /data/public/logstash/logs/:/data/logs/ \
-d logstash:8.4.0

检查日志采集

启动后，可以通过Offset Explorer连接Kafka检查是否正确采集到了相关的日志。
在这里插入图片描述

测试

echo '2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}' >> /data/public/logstash/logs/spring.log

执行一下，可以看到新日志进入到Kafka中
在这里插入图片描述
写入Kafka中日志格式如下：

{"@timestamp": "2025-07-26T17:48:58.874280406Z","body": {},"url": "/test/findUserInfo.do","fullName": "TEST","log": {"file": {"path": "/data/logs/spring.log"}},"type": "test","time": 1753067797629,"userId": 1088479580471164928,"message": "2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{\"body\":\"{}\",\"fullName\":\"TEST\",\"time\":1753067797629,\"url\":\"/test/findUserInfo.do\",\"userId\":1088479580471164928}"
}