当前位置： 首页 > news >正文

第二章 ELK安装部署与环境配置

news 2025/9/12 13:08:02

目录

  • 环境准备
  • Elasticsearch安装
  • Kibana安装
  • Logstash安装
  • Beats安装
  • 集群配置
  • 安全配置
  • 性能调优
  • 监控配置
  • 故障排除

环境准备

系统要求

硬件要求：

最小配置:CPU: 2核心内存: 4GB磁盘: 20GB网络: 1Gbps推荐配置:CPU: 8核心内存: 32GB磁盘: 500GB SSD网络: 10Gbps生产环境:CPU: 16核心+内存: 64GB+磁盘: 1TB+ NVMe SSD网络: 10Gbps+

操作系统支持：

  • Linux (推荐): CentOS 7+, Ubuntu 18.04+, RHEL 7+
  • Windows: Windows Server 2016+
  • macOS: 10.14+

Java环境配置

安装OpenJDK 11/17：

# CentOS/RHEL
sudo yum install java-11-openjdk java-11-openjdk-devel# Ubuntu/Debian
sudo apt update
sudo apt install openjdk-11-jdk# 验证安装
java -version
javac -version

环境变量配置：

# 编辑环境变量
sudo vim /etc/environment# 添加以下内容
JAVA_HOME=/usr/lib/jvm/java-11-openjdk
PATH=$PATH:$JAVA_HOME/bin# 重新加载环境变量
source /etc/environment# 验证配置
echo $JAVA_HOME

系统优化

内核参数调优：

# 编辑系统限制
sudo vim /etc/security/limits.conf# 添加以下内容
elastic soft nofile 65536
elastic hard nofile 65536
elastic soft nproc 4096
elastic hard nproc 4096
elastic soft memlock unlimited
elastic hard memlock unlimited# 编辑系统参数
sudo vim /etc/sysctl.conf# 添加以下内容
vm.max_map_count=262144
vm.swappiness=1
net.core.somaxconn=65535
net.ipv4.tcp_max_syn_backlog=65535# 应用配置
sudo sysctl -p

创建用户和目录：

# 创建elastic用户
sudo useradd -m -s /bin/bash elastic# 创建安装目录
sudo mkdir -p /opt/elastic
sudo chown -R elastic:elastic /opt/elastic# 创建数据目录
sudo mkdir -p /var/lib/elasticsearch
sudo mkdir -p /var/log/elasticsearch
sudo chown -R elastic:elastic /var/lib/elasticsearch
sudo chown -R elastic:elastic /var/log/elasticsearch

Elasticsearch安装

1. 下载和安装

使用包管理器安装：

# 添加Elastic仓库
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list# 更新包列表
sudo apt update# 安装Elasticsearch
sudo apt install elasticsearch

手动下载安装：

# 下载Elasticsearch
cd /opt/elastic
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.11.0-linux-x86_64.tar.gz# 解压
tar -xzf elasticsearch-8.11.0-linux-x86_64.tar.gz
mv elasticsearch-8.11.0 elasticsearch# 设置权限
chown -R elastic:elastic elasticsearch

2. 配置文件

主配置文件 (elasticsearch.yml)：

# ======================== Elasticsearch Configuration =========================# ---------------------------------- Cluster -----------------------------------
cluster.name: elk-cluster# ------------------------------------ Node ------------------------------------
node.name: node-1
node.roles: [ master, data, ingest ]# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch# ---------------------------------- Memory ------------------------------------
bootstrap.memory_lock: true# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300# --------------------------------- Discovery ----------------------------------
discovery.type: single-node
# discovery.seed_hosts: ["host1", "host2"]
# cluster.initial_master_nodes: ["node-1", "node-2"]# ---------------------------------- Security ----------------------------------
xpack.security.enabled: true
xpack.security.enrollment.enabled: truexpack.security.http.ssl:enabled: truekeystore.path: certs/http.p12xpack.security.transport.ssl:enabled: trueverification_mode: certificatekeystore.path: certs/transport.p12truststore.path: certs/transport.p12# ---------------------------------- Various -----------------------------------
action.destructive_requires_name: true
indices.query.bool.max_clause_count: 10000

JVM配置 (jvm.options)：

# JVM heap size
-Xms4g
-Xmx4g# GC configuration
-XX:+UseG1GC
-XX:G1HeapRegionSize=16m
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:+UseZGC# Memory settings
-XX:+AlwaysPreTouch
-Xss1m# GC logging
-Xlog:gc*,gc+age=trace,safepoint:gc.log:utctime,pid,tid,level
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=32
-XX:GCLogFileSize=64m# Heap dumps
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch# Security
-Djava.security.policy=all.policy

3. 启动和验证

启动服务：

# 使用systemd启动
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
sudo systemctl status elasticsearch# 手动启动
su - elastic
cd /opt/elastic/elasticsearch
./bin/elasticsearch -d

验证安装：

# 检查集群状态
curl -X GET "localhost:9200/_cluster/health?pretty"# 检查节点信息
curl -X GET "localhost:9200/_nodes?pretty"# 检查索引
curl -X GET "localhost:9200/_cat/indices?v"

Kibana安装

1. 下载和安装

# 使用包管理器
sudo apt install kibana# 手动安装
cd /opt/elastic
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.11.0-linux-x86_64.tar.gz
tar -xzf kibana-8.11.0-linux-x86_64.tar.gz
mv kibana-8.11.0 kibana
chown -R elastic:elastic kibana

2. 配置文件

主配置文件 (kibana.yml)：

# =================== System: Kibana Server ===================
server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana-server"# =================== System: Elasticsearch ===================
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password"# =================== System: SSL ===================
server.ssl.enabled: true
server.ssl.certificate: "/path/to/kibana-server.crt"
server.ssl.key: "/path/to/kibana-server.key"elasticsearch.ssl.certificateAuthorities: ["/path/to/ca.crt"]
elasticsearch.ssl.verificationMode: certificate# =================== System: Logging ===================
logging.appenders:file:type: filefileName: /var/log/kibana/kibana.loglayout:type: jsonlogging.root:appenders:- default- filelevel: info# =================== System: Other ===================
pid.file: /var/run/kibana/kibana.pid
path.data: /var/lib/kibana# =================== System: Security ===================
xpack.security.encryptionKey: "something_at_least_32_characters"
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"
xpack.reporting.encryptionKey: "something_at_least_32_characters"# =================== System: Monitoring ===================
monitoring.ui.container.elasticsearch.enabled: true
monitoring.ui.container.logstash.enabled: true

3. 启动和验证

# 启动Kibana
sudo systemctl enable kibana
sudo systemctl start kibana
sudo systemctl status kibana# 检查日志
sudo tail -f /var/log/kibana/kibana.log# 访问Web界面
# http://localhost:5601

Logstash安装

1. 下载和安装

# 使用包管理器
sudo apt install logstash# 手动安装
cd /opt/elastic
wget https://artifacts.elastic.co/downloads/logstash/logstash-8.11.0-linux-x86_64.tar.gz
tar -xzf logstash-8.11.0-linux-x86_64.tar.gz
mv logstash-8.11.0 logstash
chown -R elastic:elastic logstash

2. 配置文件

主配置文件 (logstash.yml)：

# =================== Node identity ===================
node.name: logstash-1# =================== Data path ===================
path.data: /var/lib/logstash
path.logs: /var/log/logstash
path.settings: /etc/logstash# =================== Pipeline Settings ===================
pipeline.workers: 4
pipeline.batch.size: 1000
pipeline.batch.delay: 50# =================== Pipeline Configuration ===================
path.config: /etc/logstash/conf.d/*.conf
config.reload.automatic: true
config.reload.interval: 3s# =================== Logging ===================
log.level: info
path.logs: /var/log/logstash# =================== HTTP API ===================
http.host: "0.0.0.0"
http.port: 9600# =================== Monitoring ===================
monitoring.enabled: true
monitoring.elasticsearch.hosts: ["https://localhost:9200"]
monitoring.elasticsearch.username: logstash_system
monitoring.elasticsearch.password: your_password

管道配置示例 (/etc/logstash/conf.d/apache.conf)：

input {beats {port => 5044}file {path => "/var/log/apache2/access.log"start_position => "beginning"sincedb_path => "/dev/null"}
}filter {if [fields][log_type] == "apache" {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }}date {match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]}geoip {source => "clientip"target => "geoip"}useragent {source => "agent"target => "useragent"}mutate {convert => { "response" => "integer" }convert => { "bytes" => "integer" }}}
}output {elasticsearch {hosts => ["https://localhost:9200"]index => "apache-logs-%{+YYYY.MM.dd}"user => "logstash_writer"password => "your_password"ssl => truecacert => "/path/to/ca.crt"}stdout {codec => rubydebug}
}

3. JVM配置

JVM设置 (jvm.options)：

# Heap size
-Xms2g
-Xmx2g# GC settings
-XX:+UseG1GC
-XX:+UseStringDeduplication# Memory settings
-XX:+AlwaysPreTouch# GC logging
-Xlog:gc*,gc+age=trace,safepoint:gc.log:utctime,pid,tid,level# Heap dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/logstash

Beats安装

1. Filebeat安装

# 下载和安装
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-linux-x86_64.tar.gz
tar -xzf filebeat-8.11.0-linux-x86_64.tar.gz
mv filebeat-8.11.0-linux-x86_64 /opt/elastic/filebeat

Filebeat配置 (filebeat.yml)：

# =================== Filebeat inputs ===================
filebeat.inputs:
- type: logenabled: truepaths:- /var/log/apache2/*.log- /var/log/nginx/*.logfields:log_type: webfields_under_root: truemultiline.pattern: '^\d{4}-\d{2}-\d{2}'multiline.negate: truemultiline.match: after- type: logenabled: truepaths:- /var/log/app/*.logfields:log_type: applicationfields_under_root: true# =================== Filebeat modules ===================
filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: truereload.period: 10s# =================== Processors ===================
processors:- add_host_metadata:when.not.contains.tags: forwarded- add_docker_metadata: ~- add_kubernetes_metadata: ~# =================== Outputs ===================
output.logstash:hosts: ["localhost:5044"]# output.elasticsearch:
#   hosts: ["https://localhost:9200"]
#   username: "filebeat_writer"
#   password: "your_password"
#   ssl.certificate_authorities: ["/path/to/ca.crt"]# =================== Logging ===================
logging.level: info
logging.to_files: true
logging.files:path: /var/log/filebeatname: filebeatkeepfiles: 7permissions: 0644

2. Metricbeat安装

# 下载和安装
wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.11.0-linux-x86_64.tar.gz
tar -xzf metricbeat-8.11.0-linux-x86_64.tar.gz
mv metricbeat-8.11.0-linux-x86_64 /opt/elastic/metricbeat

Metricbeat配置 (metricbeat.yml)：

# =================== Metricbeat modules ===================
metricbeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: truereload.period: 10s# =================== System module ===================
metricbeat.modules:
- module: systemmetricsets:- cpu- load- memory- network- process- process_summary- socket_summary- filesystem- fsstatenabled: trueperiod: 10sprocesses: ['.*']- module: dockermetricsets:- container- cpu- diskio- healthcheck- info- memory- networkhosts: ["unix:///var/run/docker.sock"]period: 10senabled: true# =================== Outputs ===================
output.elasticsearch:hosts: ["https://localhost:9200"]username: "metricbeat_writer"password: "your_password"ssl.certificate_authorities: ["/path/to/ca.crt"]# =================== Processors ===================
processors:- add_host_metadata: ~- add_docker_metadata: ~

集群配置

1. Elasticsearch集群

Master节点配置：

# elasticsearch.yml for master node
cluster.name: elk-production
node.name: master-1
node.roles: [ master ]network.host: 192.168.1.10
http.port: 9200
transport.port: 9300discovery.seed_hosts: ["192.168.1.10", "192.168.1.11", "192.168.1.12"]
cluster.initial_master_nodes: ["master-1", "master-2", "master-3"]gateway.expected_master_nodes: 3
gateway.expected_data_nodes: 6
gateway.recover_after_master_nodes: 2
gateway.recover_after_data_nodes: 4

Data节点配置：

# elasticsearch.yml for data node
cluster.name: elk-production
node.name: data-1
node.roles: [ data, ingest ]network.host: 192.168.1.20
http.port: 9200
transport.port: 9300discovery.seed_hosts: ["192.168.1.10", "192.168.1.11", "192.168.1.12"]# 数据节点特定配置
indices.memory.index_buffer_size: 30%
indices.memory.min_index_buffer_size: 96mb
indices.fielddata.cache.size: 40%

2. 负载均衡配置

Nginx配置：

upstream elasticsearch {server 192.168.1.20:9200 max_fails=3 fail_timeout=30s;server 192.168.1.21:9200 max_fails=3 fail_timeout=30s;server 192.168.1.22:9200 max_fails=3 fail_timeout=30s;
}upstream kibana {server 192.168.1.30:5601 max_fails=3 fail_timeout=30s;server 192.168.1.31:5601 max_fails=3 fail_timeout=30s;
}server {listen 80;server_name elasticsearch.example.com;location / {proxy_pass http://elasticsearch;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_connect_timeout 30s;proxy_send_timeout 30s;proxy_read_timeout 30s;}
}server {listen 80;server_name kibana.example.com;location / {proxy_pass http://kibana;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_connect_timeout 30s;proxy_send_timeout 30s;proxy_read_timeout 30s;}
}

安全配置

1. 启用安全功能

# 生成证书
cd /opt/elastic/elasticsearch
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12# 设置密码
./bin/elasticsearch-setup-passwords interactive

2. 用户和角色管理

# 创建自定义角色
curl -X POST "localhost:9200/_security/role/logstash_writer" -H 'Content-Type: application/json' -d'
{"cluster": ["manage_index_templates", "monitor", "manage_ilm"],"indices": [{"names": [ "logstash-*" ],"privileges": ["write","create","create_index","manage","manage_ilm"]}]
}'# 创建用户
curl -X POST "localhost:9200/_security/user/logstash_internal" -H 'Content-Type: application/json' -d'
{"password" : "your_password","roles" : [ "logstash_writer" ],"full_name" : "Internal Logstash User"
}'

性能调优

1. 系统级优化

# 禁用swap
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab# 文件描述符限制
echo "elastic soft nofile 65536" >> /etc/security/limits.conf
echo "elastic hard nofile 65536" >> /etc/security/limits.conf# 虚拟内存设置
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p

2. Elasticsearch优化

# elasticsearch.yml优化配置
bootstrap.memory_lock: true
indices.memory.index_buffer_size: 30%
indices.memory.min_index_buffer_size: 96mb
indices.fielddata.cache.size: 40%
indices.queries.cache.size: 10%
indices.requests.cache.size: 2%# 线程池配置
thread_pool:write:size: 8queue_size: 1000search:size: 13queue_size: 1000

监控配置

1. 集群监控

# 启用监控
echo "xpack.monitoring.collection.enabled: true" >> /etc/elasticsearch/elasticsearch.yml
echo "xpack.monitoring.collection.interval: 10s" >> /etc/elasticsearch/elasticsearch.yml# 重启服务
sudo systemctl restart elasticsearch

2. 监控脚本

#!/bin/bash
# elk-monitor.shELASTICSEARCH_URL="http://localhost:9200"
KIBANA_URL="http://localhost:5601"
LOGSTASH_URL="http://localhost:9600"# 检查Elasticsearch健康状态
check_elasticsearch() {echo "Checking Elasticsearch..."health=$(curl -s "$ELASTICSEARCH_URL/_cluster/health" | jq -r '.status')if [ "$health" = "green" ] || [ "$health" = "yellow" ]; thenecho "✓ Elasticsearch is healthy: $health"elseecho "✗ Elasticsearch is unhealthy: $health"return 1fi
}# 检查Kibana状态
check_kibana() {echo "Checking Kibana..."status=$(curl -s "$KIBANA_URL/api/status" | jq -r '.status.overall.state')if [ "$status" = "green" ]; thenecho "✓ Kibana is healthy"elseecho "✗ Kibana is unhealthy: $status"return 1fi
}# 检查Logstash状态
check_logstash() {echo "Checking Logstash..."status=$(curl -s "$LOGSTASH_URL" | jq -r '.status')if [ "$status" = "green" ]; thenecho "✓ Logstash is healthy"elseecho "✗ Logstash is unhealthy: $status"return 1fi
}# 主函数
main() {echo "ELK Stack Health Check - $(date)"echo "================================"check_elasticsearchcheck_kibanacheck_logstashecho "================================"echo "Health check completed"
}main

故障排除

1. 常见问题

Elasticsearch启动失败：

# 检查日志
sudo tail -f /var/log/elasticsearch/elk-cluster.log# 检查JVM内存
jps -v | grep Elasticsearch# 检查端口占用
netstat -tlnp | grep 9200# 检查磁盘空间
df -h# 检查文件描述符
ulimit -n

内存不足问题：

# 调整JVM堆大小
sudo vim /etc/elasticsearch/jvm.options
# 修改 -Xms 和 -Xmx 参数# 检查系统内存
free -h# 检查swap使用
swapon --show

2. 诊断工具

# Elasticsearch诊断
curl -X GET "localhost:9200/_cluster/health?pretty"
curl -X GET "localhost:9200/_nodes/stats?pretty"
curl -X GET "localhost:9200/_cat/indices?v"
curl -X GET "localhost:9200/_cat/shards?v"# 性能分析
curl -X GET "localhost:9200/_nodes/hot_threads"
curl -X GET "localhost:9200/_cluster/pending_tasks"

3. 日志分析

# 实时监控日志
tail -f /var/log/elasticsearch/*.log
tail -f /var/log/kibana/kibana.log
tail -f /var/log/logstash/logstash-plain.log# 错误日志过滤
grep -i error /var/log/elasticsearch/*.log
grep -i exception /var/log/elasticsearch/*.log

总结

本章详细介绍了ELK Stack的完整安装部署过程，包括：

关键要点

  1. 环境准备: 系统要求、Java环境、系统优化
  2. 组件安装: Elasticsearch、Kibana、Logstash、Beats
  3. 配置管理: 主配置文件、JVM参数、管道配置
  4. 集群部署: 多节点配置、负载均衡、高可用
  5. 安全配置: 认证授权、SSL/TLS、用户管理
  6. 性能调优: 系统优化、内存配置、线程池
  7. 监控运维: 健康检查、性能监控、故障排除

最佳实践

  • 合理规划硬件资源和网络架构
  • 严格按照官方文档进行配置
  • 定期备份配置文件和数据
  • 建立完善的监控和告警机制
  • 制定详细的运维操作手册

下一章我们将学习数据收集与处理，包括Beats和Logstash的详细使用方法。


文章转载自：

http://70RIJchx.rgpsq.cn
http://kCP89Tlc.rgpsq.cn
http://qWZ7FeHs.rgpsq.cn
http://DBW9iGhn.rgpsq.cn
http://zoI9RsDG.rgpsq.cn
http://ngFlywMj.rgpsq.cn
http://Ly2R7UoV.rgpsq.cn
http://TnicRiBC.rgpsq.cn
http://ycb1BOPS.rgpsq.cn
http://yY7AWPRn.rgpsq.cn
http://xDMmcGTF.rgpsq.cn
http://DPqlzcnj.rgpsq.cn
http://Zcvo92ds.rgpsq.cn
http://fjU4dtWU.rgpsq.cn
http://H2HbrqQL.rgpsq.cn
http://EtqygOOp.rgpsq.cn
http://LF8vmRmy.rgpsq.cn
http://rBjTb2A3.rgpsq.cn
http://SivuiJyx.rgpsq.cn
http://6UpYPWgS.rgpsq.cn
http://XLMontVh.rgpsq.cn
http://AoUINXq6.rgpsq.cn
http://fFA4gxDN.rgpsq.cn
http://RNJRAOM9.rgpsq.cn
http://4ZDSl3XM.rgpsq.cn
http://hMLxfkwR.rgpsq.cn
http://lmjv3m0Y.rgpsq.cn
http://fgm1qsDT.rgpsq.cn
http://8BeinAL4.rgpsq.cn
http://MHuoPRa2.rgpsq.cn
http://www.dtcms.com/a/379423.html

相关文章：

  • I2C 总线
  • 设计模式——七大常见设计原则
  • 请创建一个视觉精美、交互流畅的进阶版贪吃蛇游戏
  • 利用美团龙猫添加xlsx的sheet.xml读取sharedStrings.xml中共享字符串输出到csv功能
  • 时序数据库：定义与基本特点
  • 【WorkManager】Android 后台任务调度的核心组件指南
  • python项目批量安装包和生成requirements.txt文件
  • 零部件力学测试系统参数
  • 3D Web轻量引擎HOOPS赋能BIM/工程施工：实现超大模型的轻量化加载与高效浏览！
  • Java Web应用的安全性与防护措施！
  • 填写简历信息
  • 优先算法——专题十一：字符串
  • [Spring Cloud][3]从零开始简单工程搭建实践详解，远程调用
  • 为什么要显示调用析构函数
  • MySQL 数据完整性与约束：从基础到实战，守护数据准确性
  • Python中的“占位符”艺术：深入理解pass关键字的妙用
  • 构建企业级Python离线包仓库：从下载到服务部署全流程指南
  • C++面向对象之多态
  • 个人自留笔记——git操作
  • 命令模式，餐厅订单管理系统C++
  • Android EDLA测试命令总结
  • opencv基础实践；银行卡号识别
  • 【录屏软件】 实用工具推荐——电脑录屏软件班迪（Bandicam）录屏图文安装指南
  • 微服务事务管理实践与 Seata 框架解析
  • 今日行情明日机会——20250911
  • P4105 [HEOI2014] 南园满地堆轻絮
  • Docker 命令核心语法、常用命令
  • Windows安装Chroma DB
  • 60_基于深度学习的羊群计数统计系统（yolo11、yolov8、yolov5+UI界面+Python项目源码+模型+标注好的数据集）
  • Linux 命令 top、vmstat、iostat、free、iftop 正常用法和退出.