Prometheus + Grafana 监控常用服务
一、引言
Prometheus监控常见服务的原理主要包括服务暴露指标和Prometheus抓取指标。一方面，被监控服务通过自身提供的监控接口或借助Exporter将服务的性能指标等数据以HTTP协议的方式暴露出来；另一方面，Prometheus根据配置好的采集任务，定期去拉取这些服务暴露的指标数据，将其存储在本地的时间序列数据库中，之后用户就可以通过PromQL查询来分析这些数据，实现对服务的监控。
二、Prometheus监控MySQL服务
1. 安装数据库
这里就不展示怎么安装啦~
2. 启动数据库
[root@elk93 ~]# /etc/init.d/mysql.server start
Starting mysql.server (via systemctl): mysql.server.service.
[root@elk93 ~]#
[root@elk93 ~]# ss -ntl | grep 3306
LISTEN 0 151 *:3306 *:*
LISTEN 0 70 *:33060 *:*
[root@elk93 ~]# # 创建用户并且授权，这里我就用已经存在的用户。
mysql> SELECT user,host,plugin FROM mysql.user;
+------------------+-----------+-----------------------+
| user | host | plugin |
+------------------+-----------+-----------------------+
| linux96 | % | mysql_native_password |
| nacos | % | mysql_native_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session | localhost | caching_sha2_password |
| mysql.sys | localhost | caching_sha2_password |
| root | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
6 rows in set (0.00 sec)mysql>
mysql> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO linux96;
Query OK, 0 rows affected (0.00 sec)# 查看权限
mysql> SHOW GRANTS FOR linux96;
+-------------------------------------------------------------------+
| Grants for linux96@% |
+-------------------------------------------------------------------+
| GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO `linux96`@`%` |
| GRANT ALL PRIVILEGES ON `nova_kafka`.* TO `linux96`@`%` |
+-------------------------------------------------------------------+3. 下载mysql_exporter
[root@elk93 ~]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.17.2/mysqld_exporter-0.17.2.linux-amd64.tar.gz3.解压程序
[root@elk91 ~]# tar xf mysqld_exporter-0.17.2.linux-amd64.tar.gz -C /usr/local/bin/ mysqld_exporter-0.17.2.linux-amd64/mysqld_exporter --strip-components=1mysqld_exporter-0.17.2.linux-amd64/mysqld_exporter显式指定要解压的归档文件内的具体路径。即从压缩包中仅提取 mysqld_exporter-0.17.2.linux-amd64/ 目录下的 mysqld_exporter 文件。
--strip-components=1移除解压路径中的前 1层目录。[root@elk91 ~]#
[root@elk91 ~]# ll /usr/local/bin/
total 17936
drwxr-xr-x 2 root root 4096 Mar 27 10:56 ./
drwxr-xr-x 18 root root 4096 Mar 24 18:03 ../
-rwxr-xr-x 1 1001 1002 18356306 Feb 26 15:16 mysqld_exporter*
[root@elk91 ~]# 4. 启动MySQL exporter
# MySQL-exporter可以和要暴露http(s)接口服务在同一个节点，这里分开是为了证明，不在同一个节点也可以。
[root@elk91 ~]# cat .my.cnf
[client]
user=linux96
password=novacao666
host=10.0.0.93
[root@elk91 ~]# [root@elk91 ~]# mysqld_exporter --config.my-cnf="/root/.my.cnf"
time=2025-03-27T03:09:09.709Z level=INFO source=mysqld_exporter.go:239 msg="Starting mysqld_exporter" version="(version=0.17.2, branch=HEAD, revision=e84f4f22f8a11089d5f04ff9bfdc5fc042605773)"
time=2025-03-27T03:09:09.709Z level=INFO source=mysqld_exporter.go:240 msg="Build context" build_context="(go=go1.23.6, platform=linux/amd64, user=root@18b69b4b0fea, date=20250226-07:16:19, tags=unknown)"
time=2025-03-27T03:09:09.709Z level=INFO source=mysqld_exporter.go:252 msg="Scraper enabled" scraper=global_status
......
time=2025-03-27T03:09:09.710Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:91045. 测试访问
# 一般数据量大的都是正确的，如果是几十几百就说明我们暴露的数据不太对。
[root@elk93 ~]# curl -s http://10.0.0.91:9104/metrics| wc -l
2573
[root@elk93 ~]# 6. 配置Prometheus监控MySQL项
[root@prometheus-server31 ~]# tail -5 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-mysql-exporterstatic_configs:- targets: - 10.0.0.91:9104# 热加载
[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload7. 验证监控结果并导入grafana模板
# 访问http://10.0.0.31:9090/targets?search=grafana导入模板ID 17320
18949
14057
7362
三、Prometheus监控zookeeper集群
zookeeper没有Prometheus提供的exporter组件，但是他自带metrics暴露方式；打开以下注释即可。
1. 修改zookeeper集群的配置信息
[root@elk91 ~]# vim /usr/local/zookeeper/conf/zoo.cfg
[root@elk91 ~]#
[root@elk91 ~]# tail -5 /usr/local/zookeeper/conf/zoo.cfg
# zookeeper没有Prometheus提供的exporter组件，但是他自带metrics暴露方式；打开以下注释即可。
# https://prometheus.io Metrics Exporter
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpHost=0.0.0.0
metricsProvider.httpPort=7000
metricsProvider.exportJvmInfo=true
[root@elk91 ~]# # 拷贝配置文件到其他集群的节点
[root@elk91 ~]# scp /usr/local/zookeeper/conf/zoo.cfg 10.0.0.92:/usr/local/zookeeper/conf/
[root@elk91 ~]# scp /usr/local/zookeeper/conf/zoo.cfg 10.0.0.93:/usr/local/zookeeper/conf/2. 重启zookeeper服务
[root@elk91 ~]# zkServer.sh restart
[root@elk92 ~]# zkServer.sh restart
[root@elk93 ~]# zkServer.sh restart 3. 访问测试
http://10.0.0.91:7000/metrics
http://10.0.0.92:7000/metrics
http://10.0.0.93:7000/metrics4. 配置prometheus监控zookeeper
[root@prometheus-server31 ~]# tail -7 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-zk-exporterstatic_configs:- targets: - 10.0.0.91:7000- 10.0.0.92:7000- 10.0.0.93:7000
.....
[root@prometheus-server31 ~]# # 热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 5. 验证监控结果并导入grafana模板
5.验证配置是否生效
http://10.0.0.31:9090/targets?search=
# 看到Prometheus的targets出现zookeeper集群就行6.grafana导入模板ID
10465
四、Prometheus监控kafka集群
1. 启动kafka集群
[root@elk91 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.13-3.9.0/config/server.properties
[root@elk91 ~]# ss -ntl | grep 9092
LISTEN 0 50 [::ffff:10.0.0.91]:9092 *:*
[root@elk91 ~]# [root@elk92 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.13-3.9.0/config/server.properties
[root@elk92 ~]# ss -ntl | grep 9092
LISTEN 0 50 [::ffff:10.0.0.92]:9092 *:*
[root@elk92 ~]# [root@elk93 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.13-3.9.0/config/server.properties
[root@elk93 ~]# ss -ntl | grep 9092
LISTEN 0 50 [::ffff:10.0.0.93]:9092 *:*
[root@elk93 ~]# 2. 下载kafka exporter
wget https://github.com/danielqsj/kafka_exporter/releases/download/v1.9.0/kafka_exporter-1.9.0.linux-amd64.tar.gz3.解压软件包
[root@elk92 ~]# tar xf kafka_exporter-1.9.0.linux-amd64.tar.gz -C /usr/local/bin/ kafka_exporter-1.9.0.linux-amd64/kafka_exporter --strip-components=1
[root@elk92 ~]#
[root@elk92 ~]# ll /usr/local/bin/
total 24520
drwxr-xr-x 2 root root 4096 Mar 27 11:44 ./
drwxr-xr-x 14 root root 4096 Mar 18 14:38 ../
-rwxr-xr-x 1 1001 fwupd-refresh 25099148 Feb 17 11:04 kafka_exporter*
[root@elk92 ~]# 3. 启动kafka exporter
# 这里我们指定了集群信息，和kafka的版本信息
[root@elk92 ~]# kafka_exporter --kafka.server=10.0.0.91:9092 --kafka.server=10.0.0.92:9092 --kafka.server=10.0.0.93:9092 --kafka.version="3.9.0"4. 访问kafka exporter的WebUI
# 出现大量信息即可
http://10.0.0.92:9308/metrics5. 配置Prometheus监控kafka集群
[root@prometheus-server31 ~]# tail -5 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-kafka-exporterstatic_configs:- targets: - 10.0.0.92:9308
...
# 热加载
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 6. 验证监控结果并导入grafana模板
# 出现kafka集群的targets即可
http://10.0.0.31:9090/targets?search=8.grafana导入模板ID
12460
10122
五、prometheus监控nacos服务
nacos服务是自己有提供metrics接口，在配置文件中打开注释即可
参考链接: 监控手册 | Nacos 官网
1. 启动nacos集群
[root@elk91 ~]# /usr/local/nacos/bin/startup.sh
[root@elk92 ~]# /usr/local/nacos/bin/startup.sh
[root@elk93 ~]# /usr/local/nacos/bin/startup.sh
访问测试 ：http://10.0.0.93:8848/2. 修改nacos服务启用Metrics接口
# nacos和zk服务都是自己有提供metrics接口，在配置文件中打开注释即可[root@elk91 ~]# vim /usr/local/nacos/conf/application.properties
...
management.endpoints.web.exposure.include=prometheus,health
...
# 将配置文件拷贝给集群的其他节点
[root@elk91 ~]# scp /usr/local/nacos/conf/application.properties 10.0.0.92:/usr/local/nacos/conf/
[root@elk91 ~]# scp /usr/local/nacos/conf/application.properties 10.0.0.92:/usr/local/nacos/conf/3. 重启服务
[root@elk91 ~]# /usr/local/nacos/bin/shutdown.sh
[root@elk92 ~]# /usr/local/nacos/bin/shutdown.sh
[root@elk93 ~]# /usr/local/nacos/bin/shutdown.sh [root@elk91 ~]# /usr/local/nacos/bin/startup.sh
[root@elk92 ~]# /usr/local/nacos/bin/startup.sh
[root@elk93 ~]# /usr/local/nacos/bin/startup.sh 4. 访问nacos的WebUI
http://10.0.0.91:8848/actuator/prometheus
http://10.0.0.92:8848/actuator/prometheus
http://10.0.0.93:8848/actuator/prometheus5. 配置Prometheus监控nacos集群
# 在Prometheus配置文件中，他默认只会发现到是以/metrics结尾的URI，所以这里我们要指引一下Prometheus。加一行配置文件指定。
[root@prometheus-server31 ~]# tail -8 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-nacos-exportermetrics_path: "/actuator/prometheus"static_configs:- targets: - 10.0.0.91:8848- 10.0.0.92:8848- 10.0.0.93:8848......# 热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 6. 验证监控结果并导入grafana模板
http://10.0.0.31:9090/targets?search=7.grafana导入模板ID
13221
六、 Prometheus监控docker原生Metrics
参考链接:
https://docs.docker.com/reference/cli/dockerd/#daemon-metrics
1. 停止docker
[root@elk93 ~]# systemctl stop docker2. 修改docker启动脚本启用Metrics监控指标
[root@elk93 ~]# cat /lib/systemd/system/docker.service
[Unit]
Description=novacao linux Docke Engine
Documentation=https://docs.docker.com
Wants=network-online.target[Service]
Type=notify
ExecStart=/usr/bin/dockerd --metrics-addr 0.0.0.0:5200
[Install]
WantedBy=multi-user.target
[root@elk93 ~]#
[root@elk93 ~]# systemctl daemon-reload
[root@elk93 ~]#
[root@elk93 ~]# systemctl restart docker.service
[root@elk93 ~]#
[root@elk93 ~]# ss -ntl | grep 5200
LISTEN 0 4096 *:5200 *:*
[root@elk93 ~]# 3. 访问docker的Metrics接口
http://10.0.0.93:5200/metrics[root@node-exporter41 ~]# curl -s http://10.0.0.93:5200/metrics| wc -l
589
[root@node-exporter41 ~]# 4. prometheus监控docker服务
[root@prometheus-server31 ~]# tail -5 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-docker-exporterstatic_configs:- targets: - 10.0.0.93:5200[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 5. 验证监控结果并导入grafana模板
http://10.0.0.31:9090/targets?search=8.grafana导入模板ID
9621# 这里Prometheus提供的原生Metrics接口数据显示不全，所以我们有第二种解决方案。
Prometheus基于cAdvisor监控Docker容器
1. cAdvisor是什么？
cAdvisor（Container Advisor）是由Google开源的一款轻量级容器监控工具，主要用于实时收集、分析和展示容器运行时资源使用情况与性能数据，帮助用户了解容器化应用的行为和资源消耗。
2. 在docker节点拉取镜像
[root@elk92 ~]# docker pull gcr.io/cadvisor/cadvisor-amd64:v0.52.1
v0.52.1: Pulling from cadvisor/cadvisor-amd64
44cf07d57ee4: Already exists
.....
Status: Downloaded newer image for gcr.io/cadvisor/cadvisor-amd64:v0.52.1
gcr.io/cadvisor/cadvisor-amd64:v0.52.1
[root@elk92 ~]# 3. 运行cAdvisor
[root@elk92 ~]# docker run \--volume=/:/rootfs:ro \--volume=/var/run:/var/run:ro \--volume=/sys:/sys:ro \--volume=/var/lib/docker/:/var/lib/docker:ro \--volume=/dev/disk/:/dev/disk:ro \--publish=18080:8080 \--detach=true \--name=cadvisor \--privileged \--device=/dev/kmsg \gcr.io/cadvisor/cadvisor-amd64:v0.52.1[root@elk92 ~]# docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3857cc9ff34e gcr.io/cadvisor/cadvisor-amd64:v0.52.1 "/usr/bin/cadvisor -…" 2 minutes ago Up About a minute (healthy) 0.0.0.0:18080->8080/tcp, :::18080->8080/tcp cadvisor
[root@elk92 ~]#
4. 访问测试
[root@elk92 ~]# curl -s http://10.0.0.92:18080/metrics | wc -l
4885
[root@elk92 ~]# 5. 访问cAdVisor的webUI测试
http://10.0.0.92:18080/containers/6. 配置Prometheus监控docker容器
[root@prometheus-server31 ~]# tail -6 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-cadVisor-exporterstatic_configs:- targets: - 10.0.0.92:18080- 10.0.0.93:18080# 热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 7. 验证监控结果并导入grafana模板
http://10.0.0.31:9090/targets?search=6.grafana导入模板ID
11600 
七、prometheus监控nginx
方案1>nginx-module-vts
1. 编译安装NGINX
1.1 安装编译工具
[root@elk91 ~]# apt -y install git wget gcc make zlib1g-dev build-essential libtool openssl libssl-dev1.2 克隆nginx-module-vts模块# 配置虚拟机VPN
export https_proxy=http://10.0.0.1:7890
export http_proxy=http://10.0.0.1:7890[root@elk91 ~]# env| grep http
https_proxy=http://10.0.0.1:7890
http_proxy=http://10.0.0.1:7890
[root@elk91 ~]# [root@elk91 ~]# git clone https://github.com/vozlt/nginx-module-vts.git
Cloning into 'nginx-module-vts'...
remote: Enumerating objects: 1134, done.
remote: Counting objects: 100% (262/262), done.
remote: Compressing objects: 100% (87/87), done.
remote: Total 1134 (delta 206), reused 175 (delta 175), pack-reused 872 (from 2)
Receiving objects: 100% (1134/1134), 1.22 MiB | 935.00 KiB/s, done.
Resolving deltas: 100% (746/746), done.
[root@elk91 ~]# 1.3 下载nginx软件包
[root@elk91 ~]# wget http://nginx.org/download/nginx-1.27.3.tar.gz1.4 解压nginx
[root@elk91 ~]# tar xf nginx-1.27.3.tar.gz 1.5 配置nginx
[root@elk91 ~]# cd nginx-1.27.3/
[root@elk91 nginx-1.27.3]#
[root@elk91 nginx-1.27.3]# ./configure --prefix=/softwares/nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --without-http_rewrite_module --with-http_stub_status_module --without-http_gzip_module --with-file-aio --with-stream --with-stream_ssl_module --with-stream_realip_module --add-module=/root/nginx-module-vts1.6 编译并安装nginx
make -j 2 && make install2. 修改nginx的配置文件
[root@elk91 nginx-1.27.3]# vim /softwares/nginx/conf/nginx.conf
...
http {vhost_traffic_status_zone;upstream novacao-promethues {server 10.0.0.31:9090;}...server {...location / {root html;# index index.html index.htm;proxy_pass http://novacao-promethues;}location /nginx_status {vhost_traffic_status_display;vhost_traffic_status_display_format html;}}
}3. 启动NGINX
[root@elk91 nginx-1.27.3]# /softwares/nginx/sbin/nginx -t
nginx: the configuration file /softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /softwares/nginx/conf/nginx.conf test is successful
[root@elk91 nginx-1.27.3]# 1.9 启动nginx
[root@elk91 nginx-1.27.3]# systemctl disable --now nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable nginx
Removed /etc/systemd/system/multi-user.target.wants/nginx.service.
[root@elk91 nginx-1.27.3]#
[root@elk91 nginx-1.27.3]# /softwares/nginx/sbin/nginx
[root@elk91 nginx-1.27.3]#
4. 访问nginx的状态页面
http://10.0.0.91/nginx_status/
http://10.0.0.91/nginx_status/format/prometheus5. 修改Prometheus的配置文件
[root@prometheus-server31 ~]# tail -6 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-nginx-vts-modulesmetrics_path: "/nginx_status/format/prometheus"static_configs:- targets: - 10.0.0.91:80# 热加载
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 6. 验证监控结果并导入grafana模板
# 查看Prometheus
http://10.0.0.31:9090/targets?search=1.12 导入模板ID
9785
方案2>安装nginx-vtx-exporter
# 前提也是需要编译好NGINX
1. 下载nginx-vtx-exporter
[root@elk92 ~]# wget https://github.com/sysulq/nginx-vts-exporter/releases/download/v0.10.8/nginx-vtx-exporter_0.10.8_linux_amd64.tar.gz[root@elk92 ~]# tar xf nginx-vtx-exporter_0.10.8_linux_amd64.tar.gz -C /usr/local/bin/ nginx-vtx-exporter
[root@elk92 ~]#
[root@elk92 ~]# ll /usr/local/bin/nginx-vtx-exporter
-rwxr-xr-x 1 1001 avahi 7950336 Jul 11 2023 /usr/local/bin/nginx-vtx-exporter*
[root@elk92 ~]# 2. 运行nginx-vtx-exporter
[root@elk92 ~]# nginx-vtx-exporter -nginx.scrape_uri=http://10.0.0.91/nginx_status/format/json
2025/03/27 16:25:19 Starting nginx_vts_exporter (version=, branch=, revision=)
2025/03/27 16:25:19 Build context (go=go1.20.5, user=, date=)
2025/03/27 16:25:19 Starting Server at : :9913
2025/03/27 16:25:19 Metrics endpoint: /metrics
2025/03/27 16:25:19 Metrics namespace: nginx
2025/03/27 16:25:19 Scraping information from : http://10.0.0.91/nginx_status/format/json3. 访问nginx exporter的WebUI
http://10.0.0.93:9913/metrics
4. 配置prometheus采集nginx数据
[root@prometheus-server31 ~]# tail -12 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-nginx-vts-modulesmetrics_path: "/nginx_status/format/prometheus"static_configs:- targets: - 10.0.0.91:80- job_name: "linux96-nginx-vtx-exporter"static_configs:- targets: - 10.0.0.92:9913[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload5. 验证监控结果并导入grafana模板
导入grafana模板
2949八、Prometheus监控redis
1. 部署Redis服务
1.拉取redis镜像
[root@elk92 ~]# docker pull redis:7.4.2-alpine
7.4.2-alpine: Pulling from library/redis
...........
Digest: sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952
Status: Downloaded newer image for redis:7.4.2-alpine
docker.io/library/redis:7.4.2-alpine2.部署redis服务
[root@elk92 ~]# docker run -d --network host --name redis-server redis:7.4.2-alpine
14f6cd2c24f0735bb5bfd189730898b15380101dd11516ad2af3b7776a33a75a
[root@elk92 ~]#
[root@elk92 ~]# ss -ntl | grep 6379
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:*
LISTEN 0 511 [::]:6379 [::]:*
[root@elk92 ~]# 2. 部署redis-exporter
wget https://github.com/oliver006/redis_exporter/releases/download/v1.69.0/redis_exporter-v1.69.0.linux-amd64.tar.gz4.解压软件包
[root@elk93 ~]# tar xf redis_exporter-v1.69.0.linux-amd64.tar.gz -C /usr/local/bin/ redis_exporter-v1.69.0.linux-amd64/redis_exporter --strip-components=1
[root@elk93 ~]#
[root@elk93 ~]# ll /usr/local/bin/redis_exporter
-rwxr-xr-x 1 mysql fwupd-refresh 9638072 Mar 15 12:41 /usr/local/bin/redis_exporter*
[root@elk93 ~]# 3. 运行redis exporter
[root@elk93 ~]# redis_exporter -redis.addr redis://10.0.0.92:6379
INFO[0000] Redis Metrics Exporter v1.69.0 build date: 2025-03-15-04:40:26 sha1: 71dbe37fb14a4ae2537c1790a239dc1e568ffba5 Go: go1.24.1 GOOS: linux GOARCH: amd64
INFO[0000] Providing metrics at :9121/metrics 4. 访问redis exporter的WebUI
http://10.0.0.93:9121/metrics
5. 配置Prometheus监控redis容器指标
[root@prometheus-server31 ~]# tail -7 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-redis-exporterstatic_configs:- targets: - 10.0.0.93:9121 [root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 6. 验证监控结果并导入grafana模板
验证Prometheus配置是否生效
http://10.0.0.31:9090/targets?search=9.grafana导入模板ID
763
14091
14615
彩蛋、 grafana安装插件
1. 图表报错提示
Panel plugin not found: natel-discrete-panel说明需要安装对应的插件，使用grafana客户端工具安装即可。2. 安装指定插件
[root@prometheus-server31 ~]# ll /var/lib/grafana/
total 5236
drwxr-xr-x 5 grafana grafana 4096 Mar 27 16:53 ./
drwxr-xr-x 61 root root 4096 Mar 26 11:59 ../
drwxr-x--- 3 grafana grafana 4096 Mar 26 12:00 alerting/
drwx------ 2 grafana grafana 4096 Mar 26 12:00 csv/
-rw-r----- 1 grafana grafana 5337088 Mar 27 16:53 grafana.db
drwx------ 2 grafana grafana 4096 Mar 26 12:00 png/
[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# grafana-cli plugins install natel-discrete-panel
✔ Downloaded and extracted natel-discrete-panel v0.1.1 zip successfully to /var/lib/grafana/plugins/natel-discrete-panelPlease restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# ll /var/lib/grafana/
total 5240
drwxr-xr-x 6 grafana grafana 4096 Mar 27 16:56 ./
drwxr-xr-x 61 root root 4096 Mar 26 11:59 ../
drwxr-x--- 3 grafana grafana 4096 Mar 26 12:00 alerting/
drwx------ 2 grafana grafana 4096 Mar 26 12:00 csv/
-rw-r----- 1 grafana grafana 5337088 Mar 27 16:53 grafana.db
drwxr-xr-x 3 root root 4096 Mar 27 16:56 plugins/ # 安装后会将插件存储在本地。
drwx------ 2 grafana grafana 4096 Mar 26 12:00 png/
[root@prometheus-server31 ~]# 3. 重启grafana使得插件生效
[root@prometheus-server31 ~]# systemctl restart grafana-server.service
[root@prometheus-server31 ~]# 九、Prometheus监控MongoDB
1. 部署mongoDB服务
1.下载mongoDB镜像
[root@elk92 ~]# docker pull mongo:8.0.6-noble
8.0.6-noble: Pulling from library/mongo
.................
Status: Downloaded newer image for mongo:8.0.6-noble
docker.io/library/mongo:8.0.6-noble
[root@elk92 ~]# 2.部署mongoDB服务
[root@elk92 ~]# docker run -d --name mongodb-server --network host mongo:8.0.6-noble
4b0f00dea78bb571c216c344984ced026c1210c94db147fdc9e32f549e3135de
[root@elk92 ~]#
[root@elk92 ~]# docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4b0f00dea78b mongo:8.0.6-noble "docker-entrypoint.s…" 2 seconds ago Up 1 second mongodb-server
[root@elk92 ~]#
[root@elk92 ~]# ss -ntl | grep 27017
LISTEN 0 4096 0.0.0.0:27017 0.0.0.0:*
[root@elk92 ~]#
2. 部署MongoDB的exporter
https://github.com/percona/mongodb_exporter/releases/download/v0.43.1/mongodb_exporter-0.43.1.linux-amd64.tar.gz解压软件包
[root@elk91 ~]# tar xf mongodb_exporter-0.43.1.linux-amd64.tar.gz -C /usr/local/bin/ mongodb_exporter-0.43.1.linux-amd64/mongodb_exporter --strip-components=1
[root@elk91 ~]#
[root@elk91 ~]# ll /usr/local/bin/mongodb_exporter
-rwxr-xr-x 1 1001 geoclue 20467864 Dec 13 20:10 /usr/local/bin/mongodb_exporter*3. 准备MongoDB测试数据
[root@elk91 ~]# vim insert_data.js
// 创建数据库和集合
db = db.getSiblingDB('testdb');// 插入用户数据（10条）
for (let i = 1; i <= 10; i++) {db.users.insertOne({name: `user${i}`,email: `user${i}@example.com`,age: Math.floor(Math.random() * 50 + 18),created_at: new Date()});
}// 插入订单数据（10条）
for (let i = 1; i <= 10; i++) {db.orders.insertOne({user_id: i,product: `product${Math.floor(Math.random() * 5)}`,amount: Math.random() * 100,status: ["pending", "completed", "cancelled"][Math.floor(Math.random() * 3)],created_at: new Date()});
}// 插入产品数据（5条）
const products = ["product0", "product1", "product2", "product3", "product4"];
products.forEach((product, index) => {db.products.insertOne({name: product,price: Math.random() * 50 + 20,stock: Math.floor(Math.random() * 100)});
});// 创建索引（可选）
db.users.createIndex({ email: 1 });
db.orders.createIndex({ user_id: 1 });################################################################## 将脚本复制到容器内
docker cp insert_data.js mongodb-server:/tmp/insert_data.js# 执行脚本（使用 mongosh）
docker exec -it mongodb-server mongosh --quiet /tmp/insert_data.js
4. 运行mongodb-exporter
mongodb_exporter --mongodb.uri=mongodb://10.0.0.92:27017 --log.level=info --collector.dbstats --collector.diagnosticdata --collector.replicasetstatus --collector.topmetrics --collector.currentopmetrics --collector.exporter-metrics --collector.collstats --collector.indexstats --mongodb.collstats-colls=testdb.users,testdb.orders,testdb.products --mongodb.indexstats-colls=testdb.users,testdb.orders#####################################3验证mongoDB-exporter的WebUI
http://10.0.0.91:9216/metrics5. 配置Prometheus监控mongoDB容器
[root@prometheus-server31 ~]# tail -7 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-mongodb-exporterstatic_configs:- targets: - 10.0.0.91:9216[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload6. 验证监控结果并导入grafana模板
grafana导入模板ID
16504当然如果实在找不到，就只能自己定义Dashboard。比如: "mongodb_up"十、Prometheus监控ElasticSearch集群
保障ES集群状态正常
[root@elk91 ~]# curl https://10.0.0.91:9200/_cat/nodes -u elastic:123456 -k
10.0.0.93 59 33 45 7.11 2.18 0.75 cdfhilmrstw - elk93
10.0.0.91 78 44 4 0.73 0.51 0.29 cdfhilmrstw * elk91
10.0.0.92 54 31 34 1.49 0.51 0.18 cdfhilmrstw - elk921. 下载elasticsearch_exporter服务插件
[root@node-exporter41 ~]# wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.9.0/elasticsearch_exporter-1.9.0.linux-amd64.tar.gz
2.解压软件包
[root@node-exporter41 ~]# tar xf elasticsearch_exporter-1.9.0.linux-amd64.tar.gz -C /usr/local/bin/ elasticsearch_exporter-1.9.0.linux-amd64/elasticsearch_exporter --strip-components=1
[root@node-exporter41 ~]# ll /usr/local/bin/
total 14728
drwxr-xr-x 2 root root 4096 Mar 27 02:42 ./
drwxr-xr-x 10 root root 4096 Feb 16 2024 ../
-rwxr-xr-x 1 1001 fwupd-refresh 15069336 Mar 3 10:01 elasticsearch_exporter*2. 启动elasticsearch-exporter
[root@node-exporter41 ~]# elasticsearch_exporter --es.uri="https://elastic:123456@10.0.0.91:9200" --es.ssl-skip-verify
time=2025-03-27T02:42:46.739Z level=INFO source=clusterinfo.go:211 msg="triggering initial cluster info call"
time=2025-03-27T02:42:46.739Z level=INFO source=clusterinfo.go:182 msg="providing consumers with updated cluster info label"
time=2025-03-27T02:42:46.750Z level=INFO source=main.go:244 msg="started cluster info retriever" interval=5m0s
.....
time=2025-03-27T02:14:19.093Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:91143. 配置Prometheus的job服务发现
[root@prometheus-server31 ~]# tail -5 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-es-exporterstatic_configs:- targets: - 10.0.0.41:91145.热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 4. 验证监控结果并导入grafana模板
验证Prometheus配置是否生效
http://10.0.0.31:9090/targets?search=7.grafana导入模板ID
14191
9746
十一、小结
Prometheus监控服务分为三种情况。
1. 第一种情况
在Prometheus官网里面提供了exporter的组件，我们只需要部署好组件然后把要监控的服务使用http(s)的方式暴露出去即可，然后Prometheus修改配置文件，配置好服务发现，最后Grafana套用官方的模版ID即可。（当然也可以自己去制作dashboard展示）。
# Prometheus的exporter官网
https://prometheus.io/download/
https://prometheus.io/docs/instrumenting/exporters/
# Grafana的模版官网
https://grafana.com/dashboards2. 第二种情况
有些服务自己本身就支持云原生的Prometheus监控，在自己的配置文件中就支持metrics的接口暴露。只需要在配置文件中修改或者取消注释好配置即可，然后Prometheus修改配置文件，配置好服务发现，最后Grafana套用官方的模版ID即可。（当然也可以自己去制作dashboard展示）。
例如：nacos，zookeeper3. 第三种情况
第一二种情况都不能达到预期的标准，那么他就会有第三种服务为其metrics提供接口。
例如：docker的cAdvisor
希望这篇文章可以帮助您在生产环境中高效率的解决问题。
最后有关K8S的如何监控可以看我的基于 Operator 部署 Prometheus 实现 K8S 监控-CSDN博客
这篇文章！