当前位置： 首页 > news >正文

fluentd采集K8S日志

news 来源：原创 2025/5/23 13:49:31

1 前言

发现fluentd是一个不错的日志采集转换工具，也非常的轻量。

在这里插入图片描述

支持输入（Input）、过滤（Filter）、解析（Parser）、输出（Output）的灵活扩展。

说白话就是功能比filebeat丰富，比logstash轻量，从内存使用情况就能明显发现

那么就开始使用fluentd采集K8S的日志吧，原理docker的日志默认输出到了

[root@k8s-node04 ~]# head  -100 /var/lib/docker/containers/feb5be2528d96c21d6e77508d0e5e25d9fc5aada95a8cea8191fcc77f3154dc1/feb5be2528d96c21d6e77508d0e5e25d9fc5aada95a8cea8191fcc77f3154dc1-json.log
{"log":"2025-05-17 09:47:20.649  INFO 1 --- [nio-8081-exec-3] com.willlink.web.core.mvc.WebLogAspect   : 参数 : [1, 10, null, null, null, null]\n","stream":"stdout","time":"2025-05-17T01:47:20.65052207Z"}
{"log":"  Creating a new SqlSession\n","stream":"stdout","time":"2025-05-17T01:47:20.650531313Z"}
{"log":"  SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@47925525] was not registered for synchronization because synchronization is not active\n","stream":"stdout","time":"2025-05-17T01:47:20.650532873Z"}
{"log":"  JDBC Connection [HikariProxyConnection@2075305405 wrapping com.mysql.jdbc.JDBC4Connection@2ddf3e2e] will not be managed by Spring\n","stream":"stdout","time":"2025-05-17T01:47:20.650534233Z"}
{"log":"  ==\u003e  Preparing: SELECT COUNT(*) AS total FROM t_pile_type WHERE flag = 1\n","stream":"stdout","time":"2025-05-17T01:47:20.650535501Z"}
{"log":"  ==\u003e Parameters:\n","stream":"stdout","time":"2025-05-17T01:47:20.650537251Z"}

2 日志整理需求

日志整理需求

  • 日志是json格式，分为3个字段，log是程序的原始日志，最后的\n是docker自己加上去的，第二个字段stream是程序的标准输出，time是UTC时间，注意这个时间是要处理的，不然会比北京时间慢8个小时。这个日期与日期、宿主机的时间无关，哪怕你的时间都设置对了，最后日志的时间都是不正确的。因此后期使用fluentd加上8个小时

  • 还有这里需要多行合并，因为这里的日志首行是已日期开头的，其他行都是信息打印，例如java的错误日志，数据库查询结果都要往上合并，所以合并规则就是log是否包含日期(或已日期开头)

  • 增加log_level字段，用于标识这条日志的日志级别，例如INFO ERROR WARN …这些在log字段内有

  • 增加K8S元数据信息，例如这个容器的docker ID，镜像地址，pod ID，pod IP，名称空间，容器IP…，默认会把所有的K8S元信息添加进去，因此需要进一步的去除无关信息

  • fluentd写入的日志统一命名为k8s-pod-%Y-%m-%d，索引按照日期滚动

3 创建pod

3.1 创建elasticsearch

1.准备elastice。前置条件需要配置动态存储类SC。如果你不想吧ES部署到K8S，也可以使用传统部署es的方式

apiVersion: apps/v1
kind: StatefulSet
metadata:name: es-clusternamespace: kube-logging
spec:serviceName: elasticsearchreplicas: 3selector:matchLabels:app: elasticsearchtemplate:metadata:labels:app: elasticsearchspec:containers:- name: elasticsearchimage: elasticsearch:7.17.1imagePullPolicy: IfNotPresentresources:limits:cpu: 2000mrequests:cpu: 500mports:- containerPort: 9200name: restprotocol: TCP- containerPort: 9300name: inter-nodeprotocol: TCPvolumeMounts:- name: datamountPath: /usr/share/elasticsearch/data- name: localtimereadOnly: truemountPath: /etc/localtimeenv:- name: cluster.namevalue: k8s-logs- name: node.namevalueFrom:fieldRef:fieldPath: metadata.name- name: discovery.seed_hostsvalue: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"- name: cluster.initial_master_nodesvalue: "es-cluster-0,es-cluster-1,es-cluster-2"- name: ES_JAVA_OPTSvalue: "-Xms512m -Xmx512m"initContainers:- name: fix-permissionsimage: busyboximagePullPolicy: IfNotPresentcommand: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]securityContext:privileged: truevolumeMounts:- name: datamountPath: /usr/share/elasticsearch/data- name: increase-vm-max-mapimage: busyboximagePullPolicy: IfNotPresentcommand: ["sysctl", "-w", "vm.max_map_count=262144"]securityContext:privileged: true- name: increase-fd-ulimitimage: busyboximagePullPolicy: IfNotPresentcommand: ["sh", "-c", "ulimit -n 65536"]securityContext:privileged: truevolumes:- name: localtimehostPath:path: /etc/localtime          volumeClaimTemplates:- metadata:name: datalabels:app: elasticsearchspec:accessModes: [ "ReadWriteOnce" ]storageClassName: nfs-storageresources:requests:storage: 100Gi

3.2 创建fluentd pod

准备configmap

1.创建configmap也就是fluetnd的配置文件

kind: ConfigMap
apiVersion: v1
metadata:name: fluentd-confignamespace: kube-logging
data:containers.input.conf: |-<source>@id fluentd-containers.log@type tail# 使用了hostpath存储，所以可以访问到pod的日志path "/var/log/containers/*.log"pos_file /var/log/es-containers.log.postag raw.kubernetes.*# 从日志文件的第一行开始读取，false表示只读取增量日志，不会从头读read_from_head false<parse>@type multi_format<pattern># 以json格式匹配日志format jsontime_key timetime_format %Y-%m-%dT%H:%M:%S.%N%zkeep_time_key false</pattern># 如果不是json格式日志的兜底匹配方案<pattern>format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/time_format %Y-%m-%dT%H:%M:%S.%N%z</pattern></parse></source><match raw.kubernetes.**>@id raw.kubernetes@type detect_exceptionsremove_tag_prefix rawmessage logstream streammultiline_flush_interval 5max_bytes   500000max_lines   1000</match><filter **>@id filter_concat@type concatkey log# 多行合并规则，看是否有日志出现multiline_start_regexp /(\d{4}-\d{2}-\d{2}|==>|<==)/separator ""flush_interval 5s</filter><filter kubernetes.**>@id filter_kubernetes_metadata@type kubernetes_metadataskip_container_metadata trueskip_labels trueskip_master_url true</filter><filter kubernetes.**>@id filter_k8s_cleanup@type record_transformer# 移除不需要的K8S信息(可选)remove_keys $.kubernetes.pod_name,$.kubernetes.namespace_id,$.docker,$.stream</filter><filter kubernetes.**># CHANGED: 新增 - 将kubernetes字段提升一级@id filter_promote_k8s_fields@type record_transformerenable_ruby true<record>container_name  ${record['kubernetes']['container_name']}namespace_name  ${record['kubernetes']['namespace_name']}pod_id          ${record['kubernetes']['pod_id']}pod_ip          ${record['kubernetes']['pod_ip']}host            ${record['kubernetes']['host']}</record>renew_record false  # CHANGED: 保留原始字段，非覆盖remove_keys $.kubernetes</filter><filter kubernetes.**>  # CHANGED: 新增 - 提取log等级@id filter_log_level@type record_transformerenable_ruby true<record>log_level ${record['log'] =~ /(INFO|DEBUG|ERROR|WARN|error|debug|info|warning)/ ? $1 : 'UNKNOWN'}</record></filter>output.conf: |-<match **>@id elasticsearch@type elasticsearch@log_level errortype_name _doc# 如果你的elasticsearch在k8s内host elasticsearchport 9200# 如果es设置了访问密码#user elastic#password ****# 不是勇logstash默认的索引名称logstash_format falseindex_name k8s-pod-%Y.%m.%dtime_key timetime_key_format %Y-%m-%dT%H:%M:%S%zinclude_timestamp true<buffer time>@type filepath /var/log/fluentd-buffers/kubernetes.system.bufferflush_mode intervalretry_type exponential_backoffflush_thread_count 2flush_interval 5sretry_foreverretry_max_interval 30chunk_limit_size 2Mtotal_limit_size 500Moverflow_action blocktimekey 1dtimekey_use_utc false</buffer></match>fluent.conf: |-<system>log_level error</system><label @FLUENT_LOG><match fluent.*>@type stdout</match></label>

设置RBAC

因为获取日志的元信息是从api-server组件获取的，需要配置授权service account

apiVersion: v1
kind: ServiceAccount
metadata:name: fluentd-esnamespace: kube-logginglabels:k8s-app: fluentd-esaddonmanager.kubernetes.io/mode: Reconcile
---kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: fluentd-eslabels:k8s-app: fluentd-esaddonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:- ""resources:- "namespaces"- "pods"verbs:- "get"- "watch"- "list"
---kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: fluentd-eslabels:k8s-app: fluentd-esaddonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccountname: fluentd-esnamespace: kube-loggingapiGroup: ""
roleRef:kind: ClusterRolename: fluentd-esapiGroup: ""

创建daemonset

apiVersion: apps/v1
kind: DaemonSet
metadata:name: fluentd-es-v3.1.0namespace: kube-logginglabels:k8s-app: fluentd-esversion: v3.1.0addonmanager.kubernetes.io/mode: Reconcile
spec:selector:matchLabels:k8s-app: fluentd-esversion: v3.1.0template:metadata:labels:k8s-app: fluentd-esversion: v3.1.0spec:securityContext:seccompProfile:type: RuntimeDefaultpriorityClassName: system-node-criticalserviceAccountName: fluentd-escontainers:- name: fluentd-es#image: quay.io/fluentd_elasticsearch/fluentd:v3.1.0image: quay.io/fluentd_elasticsearch/fluentd:v4.7.2imagePullPolicy: IfNotPresentenv:- name: FLUENTD_ARGSvalue: --no-supervisor -qresources:limits:memory: 1024Micpu: 800mrequests:cpu: 100mmemory: 200MivolumeMounts:- name: varlogmountPath: /var/log- name: varlibdockercontainersmountPath: /var/lib/docker/containersreadOnly: true- name: config-volumemountPath: /etc/fluent/fluent.confsubPath: fluent.conf- name: config-volumemountPath: /etc/fluent/config.d/containers.input.confsubPath: containers.input.conf- name: config-volumemountPath: /etc/fluent/config.d/output.confsubPath: output.conf- name: localtimemountPath: /etc/localtimeports:- containerPort: 24231name: prometheusprotocol: TCPlivenessProbe:tcpSocket:port: prometheusinitialDelaySeconds: 5timeoutSeconds: 10readinessProbe:tcpSocket:port: prometheusinitialDelaySeconds: 5timeoutSeconds: 10terminationGracePeriodSeconds: 30volumes:- name: varloghostPath:path: /var/log- name: varlibdockercontainershostPath:path: /var/lib/docker/containers- name: config-volumeconfigMap:name: fluentd-configmap- name: localtimehostPath:path: /etc/localtime

4 查看日志

这是运行了一段时间的采集结果，分片和副本可以自己在kibana设置索引模板

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

查看json字段，只保留了必要的K8S信息

{"_index": "k8s-pod-2025-05-19","_type": "_doc","_id": "Eh__55YB_RI1PaU21I5o","_version": 1,"_score": 1,"_source": {"log": "2025-05-19 18:03:40.819 [nioEventLoopGroup-6-31] INFO  c.c.n.k.CustomDelimiterDecoder - 解析报文完成，channelId = 6adbc0e1, oldReadableBytes = 1115，nowReadableBytes = 0\n","datetime": "2025-05-19T18:03:40.820231799+0800","container_name": "automotive-netty","namespace_name": "automotive-dev","pod_id": "dca185db-e1a4-4392-9d84-53b4971c2b93","pod_ip": "172.17.217.71","host": "k8s-node04","log_level": "INFO","@timestamp": "2025-05-19T18:03:40.820233305+08:00"},"fields": {"log": ["2025-05-19 18:03:40.819 [nioEventLoopGroup-6-31] INFO  c.c.n.k.CustomDelimiterDecoder - 解析报文完成，channelId = 6adbc0e1, oldReadableBytes = 1115，nowReadableBytes = 0\n"],"pod_ip": ["172.17.217.71"],"log_level": ["INFO"],"pod_ip.keyword": ["172.17.217.71"],"log_level.keyword": ["INFO"],"container_name.keyword": ["automotive-netty"],"namespace_name": ["automotive-dev"],"pod_id.keyword": ["dca185db-e1a4-4392-9d84-53b4971c2b93"],"datetime": ["2025-05-19T10:03:40.820Z"],"@timestamp": ["2025-05-19T10:03:40.820Z"],"container_name": ["automotive-netty"],"host": ["k8s-node04"],"log.keyword": ["2025-05-19 18:03:40.819 [nioEventLoopGroup-6-31] INFO  c.c.n.k.CustomDelimiterDecoder - 解析报文完成，channelId = 6adbc0e1, oldReadableBytes = 1115，nowReadableBytes = 0\n"],"namespace_name.keyword": ["automotive-dev"],"host.keyword": ["k8s-node04"],"pod_id": ["dca185db-e1a4-4392-9d84-53b4971c2b93"]}
}

相关文章：

  • 软考 系统架构设计师系列知识点之杂项集萃（67）
  • 如何在PyCharm2025中设置conda的多个Python版本
  • net Core》》包与库 LibMan、NPM
  • 机器学习-KNN算法
  • 智能指针RAII
  • RISC-V IDE MRS2 开发笔记一：volatile关键字的使用
  • elementUI 中el-date-picker和el-select的样式调整
  • 自由开发者计划 001：创建一个用于查看 Jupyter Notebook 的谷歌浏览器插件
  • 从零搭建SpringBoot Web 单体项目【基础篇】2、SpringBoot 整合数据库
  • transformer归一化层优化：深度解读 RMSNorm （Root Mean Square Layer Normalization，均方根层归一化）
  • R语言学习--Day05--绘图技巧
  • Vue.js教学第八章：深入掌握Vue组件生命周期
  • 【沉浸式求职学习day46】【华为5.7暑期机试题目讲解】
  • 机器学习第二十一讲：正则化 → 给模型带定位手环防走极端路线
  • 【ffmpeg】SPS与PPS的概念
  • java中定时任务的实现及使用场景
  • 现网割接步骤，慢慢总结版
  • Spring Boot接口通用返回值设计与实现最佳实践
  • spring中的Interceptor使用说明
  • svn 提交后报错 ： is scheduled for addtion,but is missing
  • 自适应网站一般做多大尺寸/国内重大新闻十条
  • 网站备案哪个部门/如何自己开发一个网站
  • 江苏专业做网站的公司/百度网盘怎么找片
  • 网站备案把二级域名放在国外/seo站长工具综合查询
  • 微信公众号的微网站开发/优化法治化营商环境
  • wordpress更新配置文件/搜索优化seo