OpenTelemetry日志采集和链路跟踪部署与问题解决文档
OpenTelemetry日志采集和链路跟踪部署与问题解决文档
目录
- 系统架构概述
- 环境准备
- 部署步骤
- Jaeger + Elasticsearch 链路追踪集成
- Elasticsearch索引架构深度解析
- Prometheus Push 模式 Metrics 收集
- 问题排查与解决
- 验证与测试
- 使用指南
- 运维工具集
- 常见问题FAQ
- 维护建议
系统架构概述
技术栈
- 应用程序: Spring Boot 3.1.5 + Java 17
- 链路跟踪: OpenTelemetry Java Agent 1.13.1
- 数据收集: OpenTelemetry Collector 0.135.0
- 数据存储:
- Elasticsearch 8.14.3 (Logs + Traces存储)
- Prometheus v3.5.0 (Metrics)
- Jaeger All-in-One (链路追踪UI + ES存储后端)
- 可视化:
- Jaeger UI (链路追踪可视化)
- Prometheus UI (指标监控)
- 运行环境: Windows 11 + Docker Desktop
数据流架构
┌─────────────────┐ OTLP/gRPC ┌─────────────────┐
│ Spring Boot │─────────────────→│ OpenTelemetry │
│ Application │ localhost:4317 │ Collector │
│ (Port: 9013) │ │ (Docker) │
└─────────────────┘ └─────────────────┘│┌──────────────────┼──────────────────┐│ │ │▼ ▼ ▼┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│ Jaeger UI │ │ Elasticsearch │ │ Prometheus ││ (localhost:16686│ │ (192.168.3.4:9200)│ │ (localhost:9090││ Traces可视化 │ │ Logs存储 │ │ Metrics存储 │└─────────────────┘ └─────────────────┘ └─────────────────┘│ ▲ ▲│ │ │OTLP/gRPC (4319) 存储Traces Remote Write API│ │▼ │┌─────────────────┐ ││ Jaeger │────────────────┘│ All-in-One │ ES Client API│ OTLP接收+ES存储 │└─────────────────┘
Elasticsearch索引架构图
Elasticsearch 8.14.3
├── 日志数据流 (自动创建)
│ ├── logs-generic.otel-default
│ │ ├── 索引模板: "logs" (内置，优先级100)
│ │ ├── 组件模板: logs@mappings + logs@settings + ecs@mappings
│ │ └── ILM策略: "logs" (自动关联)
│ └── 索引: .ds-logs-generic.otel-default-2025.09.18-000001
│
├── 链路数据流 (手动创建)
│ ├── traces-generic.otel-default
│ │ ├── 索引模板: "traces-generic.otel" (手动，优先级100)
│ │ ├── 组件模板: otel-traces-mappings (手动)
│ │ └── ILM策略: "logs" (手动配置)
│ └── 索引: .ds-traces-generic.otel-default-2025.09.19-000001
│
└── 监控数据流 (专用模板)├── metrics-generic.otel-default│ ├── 索引模板: "metrics-generic.otel" (专用，优先级200) ✅│ ├── 组件模板: otel-metrics-mappings (专用嵌套结构) ✅│ └── ILM策略: "logs" (手动配置)└── 索引: .ds-metrics-generic.otel-default-2025.09.19-000001
环境准备
1. 软件依赖
- Docker Desktop (已安装)
- Java 17 JDK
- OpenTelemetry Java Agent JAR包
2. 目录结构
D:/soft/
├── opentelemetry-javaagent.jar # OpenTelemetry Java Agent
├── springboot3.jar # Spring Boot应用JAR包
└── otelcol-otlp_0.132.2_windows_amd64/└── otelcol-config.yml # Collector配置文件
部署步骤
1. Elasticsearch部署
# 拉取并启动Elasticsearch容器
docker run -d \--name elasticsearch \-p 9200:9200 \-p 9300:9300 \-e "discovery.type=single-node" \-e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \docker.elastic.co/elasticsearch/elasticsearch:8.14.3
2. OpenTelemetry Collector配置
配置文件 (otelcol-config.yml)
receivers:otlp:protocols:grpc:endpoint: 0.0.0.0:4317http:endpoint: 0.0.0.0:4318cors:allowed_origins:- "*"exporters:elasticsearch/logs:endpoints: ["https://192.168.3.4:9200"]tls:insecure: falseinsecure_skip_verify: truemapping:mode: otelauth:authenticator: basicauthnum_workers: 4retry:enabled: trueinitial_interval: 1smax_interval: 30selasticsearch/traces:endpoints: ["https://192.168.3.4:9200"]tls:insecure: falseinsecure_skip_verify: truemapping:mode: otelauth:authenticator: basicauthnum_workers: 4retry:enabled: trueinitial_interval: 1smax_interval: 30selasticsearch/metrics:endpoints: ["https://192.168.3.4:9200"]tls:insecure: falseinsecure_skip_verify: truemapping:mode: otelauth:authenticator: basicauthnum_workers: 4retry:enabled: trueinitial_interval: 1smax_interval: 30sdebug:verbosity: detailedextensions:basicauth:client_auth:username: elasticpassword: ********processors:batch:timeout: 5ssend_batch_size: 1000send_batch_max_size: 5000memory_limiter:check_interval: 5slimit_percentage: 80spike_limit_percentage: 25resourcedetection:detectors: [ env, system ]timeout: 2stransform:error_mode: ignorelog_statements:- context: logstatements:- set(attributes["@timestamp"], FormatTime(time, "%Y-%m-%dT%H:%M:%S.%LZ"))connectors:spanmetrics:service:extensions: [basicauth]pipelines:traces:receivers: [ otlp ]processors: [ memory_limiter, resourcedetection, transform, batch ]exporters: [ debug, elasticsearch/traces ]metrics:receivers: [ otlp ]processors: [ memory_limiter, resourcedetection, batch ]exporters: [ elasticsearch/metrics ]logs:receivers: [ otlp ]processors: [ memory_limiter, resourcedetection, transform, batch ]exporters: [ elasticsearch/logs ]telemetry:logs:level: infometrics:level: basic
配置说明
Receivers 接收器配置:
otlp.protocols.grpc: OTLP gRPC协议，监听4317端口otlp.protocols.http: OTLP HTTP协议，监听4318端口，启用CORS支持
Exporters 导出器配置:
- Elasticsearch导出器: 配置了logs、traces、metrics三个导出器
endpoints: ES集群地址 (HTTP协议)mapping.mode: otel: 使用OpenTelemetry映射模式auth.authenticator: 使用basicauth认证num_workers: 4: 并发工作线程数retry: 重试机制配置
- Debug导出器: 详细日志输出，用于调试
Extensions 扩展配置:
basicauth: ES认证配置，包含用户名和密码
Processors 处理器配置:
batch: 批处理配置 (超时5s，批大小1000-5000)memory_limiter: 内存限制 (80%使用率，25%峰值)resourcedetection: 资源检测 (环境和系统信息)transform: 数据转换 (添加@timestamp字段)
Service 服务配置:
- 三条数据管道: traces、metrics、logs各自独立处理
- Telemetry: 收集器自身的遥测配置
启动Collector容器
docker run -d \--name collector \-p 4317:4317 \-p 4318:4318 \-p 13133:13133 \-p 55679:55679 \-v "D:/soft/otelcol-otlp_0.132.2_windows_amd64/otelcol-config.yml:/etc/otelcol-contrib/config.yaml" \otel/opentelemetry-collector-contrib:0.135.0
3. Spring Boot应用启动
启动命令
java -Dspring.profiles.active=local \-javaagent:D:/soft/opentelemetry-javaagent.jar \-Dotel.service.name=springboot3 \-Dotel.traces.exporter=otlp \-Dotel.metrics.exporter=otlp \-Dotel.logs.exporter=otlp \-Dotel.exporter.otlp.compression=gzip \-Dotel.exporter.otlp.protocol=grpc \-Dotel.exporter.otlp.endpoint=http://127.0.0.1:4317 \-Dotel.javaagent.logging=application \-jar D:/soft/springboot3.jar
Jaeger + Elasticsearch 链路追踪集成
概述
Jaeger + Elasticsearch 集成方案提供了最佳的链路追踪体验，结合了Jaeger强大的链路分析UI和Elasticsearch可靠的数据存储能力。
架构优势
Java应用 → OpenTelemetry Collector → Jaeger → Elasticsearch↓Jaeger UI (查询展示)
与直接ES存储方案对比
| 特性 | 直接ES存储 | Jaeger+ES方案 |
|---|---|---|
| UI体验 | 需要Kibana查询 | 🟢 专业链路追踪UI |
| 数据分析 | 🟡 通用JSON查询 | 🟢 链路依赖分析 |
| 服务拓扑 | ❌ 需要手动构建 | 🟢 自动生成服务依赖图 |
| 性能分析 | 🟡 需要复杂聚合 | 🟢 内置性能指标 |
| 配置复杂度 | 🔴 需要复杂索引模板 | 🟢 开箱即用 |
| 数据持久化 | 🟢 ES原生存储 | 🟢 ES后端存储 |
完整部署流程
1. 部署Jaeger All-in-One容器
完整启动命令: