服务器硬盘爆满100%问题解决
问题
在工作中遇到一个服务器，服务器硬盘100%，查找哪个目录文件中占用大量空间。发现加起来才150G，硬盘空间大概有500G。
处理问题，排查是否有某个进程正在删除文件，进程卡住了，所以过滤一下有哪些进程，进行删除。
排查
目前硬盘占用100%
/]# df -lh
文件系统 容量 已用 可用 已用% 挂载点
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 72M 16G 1% /dev/shm
tmpfs 16G 1.7G 15G 11% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-root 495G 494G 1.1G 100% /
/dev/sda1 1014M 216M 799M 22% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/1002查看占用的应用程序
/]# lsof -n / |grep deleted
java 1519 elasticsearch 198r FIFO 253,0 0t0 370874022 /tmp/elasticsearch-15125760050432573356/controller_log_1519 (deleted)
java 1519 elasticsearch 199w FIFO 253,0 0t0 370874023 /tmp/elasticsearch-15125760050432573356/controller_command_1519 (deleted)
controlle 2161 elasticsearch 2w FIFO 253,0 0t0 370874022 /tmp/elasticsearch-15125760050432573356/controller_log_1519 (deleted)
controlle 2161 elasticsearch 3w FIFO 253,0 0t0 370874022 /tmp/elasticsearch-15125760050432573356/controller_log_1519 (deleted)
controlle 2161 elasticsearch 5r FIFO 253,0 0t0 370874023 /tmp/elasticsearch-15125760050432573356/controller_command_1519 (deleted)
clckhouse 7807 clickhouse 3w REG 253,0 1048576066 805499428 /var/log/clickhouse-server/clickhouse-server.log.0 (deleted)
java 13742 root 40w REG 253,0 5412 805533253 /root/logs/rocketmqlogs/namesrv_default.log (deleted)
mysqld 14236 mysql 5u REG 253,0 0 270724638 /tmp/ibu23Tlr (deleted)
mysqld 14236 mysql 6u REG 253,0 0 270724639 /tmp/ibgoyhTx (deleted)
mysqld 14236 mysql 7u REG 253,0 0 270726018 /tmp/ibIlnFqE (deleted)
mysqld 14236 mysql 8u REG 253,0 0 270726023 /tmp/ib8YqztS (deleted)
mysqld 14236 mysql 12u REG 253,0 0 270726024 /tmp/ibYVQrc0 (deleted)
java 14599 root 1w REG 253,0 327890149848 1105447 /root/kafka/kafka/efak/kms/logs/catalina.out (deleted)
java 14599 root 2w REG 253,0 327890149848 1105447 /root/kafka/kafka/efak/kms/logs/catalina.out (deleted)
java 14599 root 8w REG 253,0 2555 1105542 /root/kafka/kafka/efak/kms/logs/localhost.2024-09-11.log (deleted)
java 14599 root 9w REG 253,0 0 1105450 /root/kafka/kafka/efak/kms/logs/manager.2023-11-07.log (deleted)
java 14599 root 10w REG 253,0 0 1105451 /root/kafka/kafka/efak/kms/logs/host-manager.2023-11-07.log (deleted)
java 14599 root 62w REG 253,0 92888 1105261 /root/kafka/kafka/efak/kms/logs/localhost_access_log.2024-09-11.txt (deleted)
java 14599 root 78w REG 253,0 10647384 271536236 /root/kafka/kafka/efak/logs/error.log.2024-09-11 (deleted)
java 14599 root 180w REG 253,0 477577 271999126 /root/kafka/kafka/efak/logs/error.log.2023-11-07 (deleted)
java 14599 root 328w REG 253,0 10647384 271536236 /root/kafka/kafka/efak/logs/error.log.2024-09-11 (deleted)杀死卡主的进程号
/]# kill -9 14599 14236查看杀死进程号后的硬盘空间
/]# df -lh
文件系统 容量 已用 可用 已用% 挂载点
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 72M 16G 1% /dev/shm
tmpfs 16G 1.7G 15G 11% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-root 495G 189G 307G 39% /
/dev/sda1 1014M 216M 799M 22% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/1002至此硬盘100%已解决。