Kubernetes V1.24+ Docker运行时 grafana容器指标显示异常

参考:
https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/

在 Kubernetes v1.24 版本中，内建组件 dockershim 被移除。 默认的容器运行时从 Docker 切换到了 containerd。如果您的集群使用 Docker 作为容器运行时，并且您的应用程序依赖于 dockershim 提供的指标,
那么部署kube-prometheus-stack 监控栈时，会出现容器指标显示异常的问题。

故障复现

根据kubernetes版本和kube-prometheus的兼容矩阵安装合适版本的kube-prometheus-stack, 容器指标却采集不到。

Compatibility Matrix

The following Kubernetes versions are supported and work as we test against these versions in their respective branches. But note that other versions might work!

kube-prometheus stack	Kubernetes 1.22	Kubernetes 1.23	Kubernetes 1.24	Kubernetes 1.25	Kubernetes 1.26	Kubernetes 1.27	Kubernetes 1.28
release-0.10	✔	✔	✗	✗	x	x	x
release-0.11	✗	✔	✔	✗	x	x	x
release-0.12	✗	✗	✔	✔	x	x	x
release-0.13	✗	✗	✗	x	✔	✔	✔
main	✗	✗	✗	x	x	✔	✔

比如我的版本是kubernetes 1.26, 那么我就需要安装kube-prometheus-stack release-0.13 版本。

如果我的容器运行时是containerd, 那么正常情况，容器指标显示是没有问题的。因为kube-prometheus-stack就是根据标准kubernetes版本去测试的。

但是如果你的容器运行时用了docker, 那么就会出现容器指标显示异常的问题。

根本原因

容器的指标是通过kubelet的metrics-endpoint暴露的，路径为/metrics/cadvisor,由内置cadvisor提供。

本来默认是containerd运行时，如果你的运行时换成了docker，就会采集容器指标变成：

容器的PromSQL查询语句：

sum(container_memory_working_set_bytes{job="kubelet", metrics_path="/metrics/cadvisor", cluster="$cluster", namespace="$namespace", pod="$pod", container!="", image!=""}) by (container)

以docker运行时查询出来的容器指标，会有多个标签是空值,如下所示：

container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20aada51_3ac1_4ab7_8ec6_87b090585825.slice",image="",name="",namespace="monitoring",pod="node-exporter-v25n7"} 327.638786615 176252734624

其中的container, image标签都是空值，因此面板监控显示的容器指标就是异常的。

获取kubelet cadvisor 指标

apiVersion: v1
kind: Secret
metadata:name: cadvisor-reader-tokennamespace: defaultannotations:kubernetes.io/service-account.name: cadvisor-reader  # 关联到目标 ServiceAccount
type: kubernetes.io/service-account-token

创建一个ServiceAccount,用于kubelet认证。

# 保存为 custom-kubelet-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: custom-kubelet-reader
rules:
- apiGroups: [""]resources:- "nodes/proxy"           # 允许访问节点代理（包含 cAdvisor 接口）- "nodes/metrics"         # 节点 metrics 接口- "nodes/stats"           # 节点统计信息（包含容器详细数据）- "nodes/log"             # 节点日志（可选）- "nodes/spec"            # 节点规格（可选）verbs: ["get", "list", "watch"]---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: default-cadvisor-reader-binding  # 与之前同名，会覆盖旧绑定
subjects:
- kind: ServiceAccountname: cadvisor-readernamespace: default
roleRef:kind: ClusterRolename: custom-kubelet-reader  # 引用自定义角色apiGroup: rbac.authorization.k8s.io

• get-metrics.sh

# 定义 Secret 名称（已确认存在）
SECRET_NAME=cadvisor-reader-token# 提取 Token 并解码（base64 解码）
TOKEN=$(kubectl get secret $SECRET_NAME -n default -o jsonpath='{.data.token}' | base64 -d)curl -k -H "Authorization: Bearer $TOKEN" https://192.168.0.4:10250/metrics/cadvisor

就可以获取到cadvisor指标了。

如何解决？

需要安装cadvisor, 来替代kubelet默认提供的cadvisor。

参考：https://github.com/google/cadvisor/tree/master/deploy/kubernetes

创建cadvisor ServiceAccount 和 Namespace

apiVersion: v1
kind: ServiceAccount
metadata:name: cadvisornamespace: monitoring
---
apiVersion: v1
kind: Namespace
metadata:name: cadvisor

创建cadvisor ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app: cadvisorname: cadvisor
rules:
- apiGroups:- policyresourceNames:- cadvisorresources:- podsecuritypoliciesverbs:- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app: cadvisorname: cadvisor
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cadvisor
subjects:
- kind: ServiceAccountname: cadvisornamespace: monitoring

创建cadvisor DaemonSet

apiVersion: apps/v1 # for Kubernetes versions before 1.9.0 use apps/v1beta2
kind: DaemonSet
metadata:name: cadvisornamespace: monitoring
spec:selector:matchLabels:name: cadvisortemplate:metadata:labels:name: cadvisorspec:serviceAccountName: cadvisorpriorityClassName: system-node-criticaltolerations:- key: "CriticalAddonsOnly"operator: "Exists"containers:- name: cadvisorimage: dockerhub.kubekey.local/cadvisor/cadvisor:v0.45.0args:- --housekeeping_interval=10s                           # kubernetes default args- --max_housekeeping_interval=15s- --event_storage_event_limit=default=0- --event_storage_age_limit=default=0- --enable_metrics=app,cpu,disk,diskIO,memory,network,process- --docker_only                                         # only show stats for docker containers- --store_container_labels=false- --whitelisted_container_labels=io.kubernetes.container.name, io.kubernetes.pod.name,io.kubernetes.pod.namespaceresources:requests:memory: 400Micpu: 400mlimits:memory: 2000Micpu: 800mvolumeMounts:- name: rootfsmountPath: /rootfsreadOnly: true- name: var-runmountPath: /var/runreadOnly: true- name: sysmountPath: /sysreadOnly: true- name: dockermountPath: /var/lib/dockerreadOnly: true- name: diskmountPath: /dev/diskreadOnly: trueports:- name: httpcontainerPort: 8080protocol: TCPautomountServiceAccountToken: falseterminationGracePeriodSeconds: 30volumes:- name: rootfshostPath:path: /- name: var-runhostPath:path: /var/run- name: syshostPath:path: /sys- name: dockerhostPath:path: /var/lib/docker- name: diskhostPath:path: /dev/disk

创建cadvisor Service 和 ServiceMonitor

apiVersion: v1
kind: Service
metadata:name: cadvisorlabels:app: cadvisornamespace: monitoring
spec:selector:name: cadvisortype: NodePortports:- name: cadvisorport: 8080protocol: TCPtargetPort: 8080
---
apiVersion: v1
kind: Service
metadata:name: cadvisorlabels:app: cadvisornamespace: monitoring
spec:selector:name: cadvisortype: NodePortports:- name: cadvisorport: 8080protocol: TCPtargetPort: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:app: cadvisorname: cadvisornamespace: cadvisor
spec:endpoints:- metricRelabelings:- action: replacesourceLabels:- container_label_io_kubernetes_pod_nametargetLabel: pod- action: replacesourceLabels:- container_label_io_kubernetes_container_nametargetLabel: container- action: replacesourceLabels:- container_label_io_kubernetes_pod_namespacetargetLabel: namespace- action: labeldropregex: container_label_io_kubernetes_pod_name- action: labeldropregex: container_label_io_kubernetes_container_name- action: labeldropregex: container_label_io_kubernetes_pod_namespace- action: replaceregex: 'k8s_[^_]+_([^_]+)_.*'replacement: $1sourceLabels:- nametargetLabel: podport: cadvisorrelabelings:- action: replacesourceLabels:- __meta_kubernetes_pod_node_nametargetLabel: node- action: replacereplacement: /metrics/cadvisorsourceLabels:- __metrics_path__targetLabel: metrics_path- action: replacereplacement: kubeletsourceLabels:- jobtargetLabel: jobnamespaceSelector:matchNames:- monitoringselector:matchLabels:app: cadvisor

给promethues用户足够的权限获取cadvisor命名空间下的所有资源：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: prometheus-all-resources-readonly
rules:
- apiGroups: ["*"]  # 匹配所有 API 组（核心组、apps、networking.k8s.io 等）resources: ["*"]  # 匹配所有资源类型（pods、services、endpoints、deployments 等）verbs: ["get", "list", "watch"]  # 仅授予只读操作权限
---
# 绑定到 prometheus-k8s ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: prometheus-k8s-all-readonly
subjects:
- kind: ServiceAccountname: prometheus-k8snamespace: monitoring
roleRef:kind: ClusterRolename: prometheus-all-resources-readonlyapiGroup: rbac.authorization.k8s.io

为什么要创建一个cadvisor命名空间？

这是因为monitoring下的ServiceMonitor对象，Prometheus-Operator监控不到。

查看cadvisor收集到的指标

通过cadvisor的NodePort端口，访问cadvisor的指标。

http://<cadvisor-node-ip>:<node-port>/metrics

我们会看到如下指标：

container_network_receive_bytes_total{container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_pod_namespace="kube-system",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60538598_0b4c_49c0_9577_c6b600e287d6.slice/docker-5e54348343024541ec3f9dbe65649f77a701f7a8808c84f8fbdfeaa20f625b20.scope",image="gcr.io/pause:3.9",interface="califffbc9ee416",name="k8s_POD_nodelocaldns-8trx7_kube-system_60538598-0b4c-49c0-9577-c6b600e287d6_1"} 5.168659e+07 1762570436207

可以看到container、image、namespace字段都有了正确的默认值。

但是pod字段为空，不过我们可以通过name字段来推测它的值，它的组成模式k8s_POD_nodelocaldns-8trx7_kube-system_60538598-0b4c-49c0-9577-c6b600e287d6_1为k8s_$deployment_$pod_name_$namespace_$pod_uid。

那么此时，我们可以通过给添加metricRelabelings来提取pod字段(yaml已包含)：

    - metricRelabelings:- action: replaceregex: 'k8s_[^_]+_([^_]+)_.*'replacement: $1sourceLabels:- nametargetLabel: pod

观察prometheus是否采集到了cadvisor的指标：

如果已经采集到了，那么此时我们还有一件事要做，就是找到kubelet的ServiceMonitor对象，删除默认的采集规则

使用kubectl get servicemonitor -n monitoring kubelet -o yaml来获取kubelet的ServiceMonitor对象。

删除路径为/metrics/cadvisor的采集规则,也就是如下这部分。

- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokenhonorLabels: trueinterval: 1mmetricRelabelings:- action: keepregex: >-container_cpu_usage_seconds_total|container_memory_usage_bytes|container_memory_cache|container_network_.+_bytes_total|container_memory_working_set_bytes|container_cpu_cfs_.*periods_total|container_processes.*|container_threads.*sourceLabels:- __name__path: /metrics/cadvisorport: https-metricsrelabelings:- sourceLabels:- __metrics_path__targetLabel: metrics_path- action: labeldropregex: (service|endpoint)scheme: httpstlsConfig:insecureSkipVerify: true

校验指标

Okay，所有准备工作已经完成，现在我们可以进入prometheus的web界面，查看是否采集到了cadvisor的指标。

container_memory_working_set_bytes{container="nfs-client-provisioner", endpoint="cadvisor", id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3db91060_4f1e_41b0_becc_6f4a69e86319.slice/docker-f1bc1a1d88993cc65eb09ae5b2a7f47cf596c21c7f9990d6523ba1fa997c28ab.scope", image="sha256:932b0bface75b80e713245d7c2ce8c44b7e127c075bd2d27281a16677c8efef3", instance="10.233.82.36:8080", job="kubelet", metrics_path="/metrics/cadvisor", name="k8s_nfs-client-provisioner_nfs-provisioner-nfs-client-provisioner-97d76bf9-jx7h8_default_3db91060-4f1e-41b0-becc-6f4a69e86319_1", namespace="default", node="node01", pod="nfs-provisioner-nfs-client-provisioner-97d76bf9-jx7h8", service="cadvisor"}

如图所示：我们发现prometheus之前采集的指标中为空的label，比如image、container、pod、namespace等都有了正确的值。

查看面板：

CPU & 内存指标显示正常：

网络流量指标显示正常：