day031-Shell自动化编程-数组与案例

1. 数组

• 数组是shell编程中一种特殊的变量，可以存储多个多种类型的数据
• 数组的元素与下标位置、和元素名字（在awk中使用）有关
• 应用场景
  • 用于存放相关的数据
  • 获取用户连续的输入

1.1 基础使用

# 创建数组变量
[root@aliyun-ubuntu ~]# arr=(10 2.4 -2 0.33 skx "abc")
# 根据下标获取数组中的元素
[root@aliyun-ubuntu ~]# echo ${arr[0]}
10
[root@aliyun-ubuntu ~]# echo ${arr[-1]}
abc
[root@aliyun-ubuntu ~]# echo ${arr[-2]}
skx
# 输出所有数组元素
[root@aliyun-ubuntu ~]# echo ${arr[*]}
10 2.4 -2 0.33 skx abc
[root@aliyun-ubuntu ~]# echo ${arr[@]}
10 2.4 -2 0.33 skx abc
# 输出数组长度
[root@aliyun-ubuntu ~]# echo ${#arr[*]}
6

1.2 案例-批量检查ip/域名，ip/域名存放在数组中

• read -a arr：按空格分割，读取多个数据，组成数组变量，变量名为arr

[root@aliyun-ubuntu /server/scripts/array_shell]# cat check_ip_array.sh 
#!/bin/bash
##############################################################
# File Name: check_ip_array.sh
# Version: V1.0
# Author: SunKexu
# Organization: www.oldboyedu.com
# Description:array
##############################################################
export LANG=en_US.UTF-8
# vars
# get ips from user
read -p "input ips/domains:" -a ips# check vars
function check_vars(){[ -z "${ips}" ] && {echo "input ips"exit 1}return $?
}
# check ip
function check_ips(){for ip in ${ips[*]}doif ping -c 1 -W 1 $ip >/dev/null 2>&1 ;thenecho "$ip ping ok"elseecho "$ip ping failed"fidonereturn $?
}
# main function
function main(){check_varscheck_ipsreturn $?
}
main
[root@aliyun-ubuntu /server/scripts/array_shell]# bash check_ip_array.sh 
input ips/domains:
input ips
[root@aliyun-ubuntu /server/scripts/array_shell]# bash check_ip_array.sh 
input ips/domains:baidu.com jd.com 520skx.com abc12345.com
baidu.com ping ok
jd.com ping ok
520skx.com ping ok
abc12345.com ping failed

1.3 技巧-curl命令的选项有多个,选项中还有引号"",使用数组可以避免引号导致的问题

[root@aliyun-ubuntu /server/scripts/array_shell]#  curl -X POST \-H "Content-Type:application/x-www-form-urlencoded" \-d "key=045a8a29190663ed52c2ee82f22d739d" \"https://apis.tianapi.com/pyqwenan/index"
{"code":200,"msg":"success","result":{"content":"人一生只有唯一的一个结果，不同的是过程。","source":"佚名"
[root@aliyun-ubuntu /server/scripts/array_shell]# curl_arr=(-X POST -H "Content-Type:application/x-www-form-urlencoded" -d "key=045a8a29190663ed52c2ee82f22d739d" "https://apis.tianapi.com/pyqwenan/index")
[root@aliyun-ubuntu /server/scripts/array_shell]# curl ${curl_arr[*]}
{"code":200,"msg":"success","result":{"content":"世间最美好的事莫过于，在最好的年华遇见了那个对的人。","source":"原创"}}

2. Debug

• bash -x：显示脚本中命令执行过程
• set -x/+x：精准查看某个命令的执行过程

# 脚本命令：
……
set -x
命令……
set +x
……

3. 案例-书写检查僵尸进程与处理僵尸进程的脚本

[root@aliyun-ubuntu /server/scripts/array_shell]# cat check_zombie2.sh
#!/bin/bash
##############################################################
# File Name: check_zombie2.sh
# Version: V1.0
# Author: SunKexu
# Organization: www.oldboyedu.com
# Description:check zombie
##############################################################
export LANG=en_US.UTF-8# vars
# temp file
tmp_file=`mktemp`
# check zombie num
zombie_count=`top -bn1 |awk 'NR==2{print $(NF-1)}'`
if [ $zombie_count -eq 0 ];thenecho "no zombie process"exit
fi
# deal with zombie
zombie_pids=`ps aux |awk '$8~/Z/{print $2}'`
reboot=0
for pid in ${zombie_pids}
doppid=`ps -ef |awk -v p=$pid '$2==p {print $3}'`name=`ps -ef |awk -v p=$pid '$2==p {print $8}'`if [ $ppid -eq 1 ];thenecho "zombie:$pid,$name,need reboot"let reboot++elseecho "zombie:$pid,$name,kill parent process $ppid could deal with"echo "$ppid" >> $tmp_filefi
done#
if [ $reboot -ge 1 ];thenecho "There are zombie processes in the system that need to be restarted to resolve"
fiecho "zombie ppid:"
cat "$tmp_file"
read -p "Should we clean up now?(y/n)" clean
[ "$clean" = 'y' ] && {cat "$tmp_file" |xargs kill[ $? -eq 0 ] && echo "clean success" || echo "clean failed"
}
[root@aliyun-ubuntu /server/scripts/array_shell]# bash check_zombie2.sh 
zombie:35471,[zombie],kill parent process 35470 could deal with
zombie:35473,[zombie],kill parent process 35472 could deal with
zombie:35475,[zombie],kill parent process 35474 could deal with
zombie:35477,[zombie],kill parent process 35476 could deal with
zombie:35479,[zombie],kill parent process 35478 could deal with
zombie:35481,[zombie],kill parent process 35480 could deal with
zombie:35483,[zombie],kill parent process 35482 could deal with
zombie:35485,[zombie],kill parent process 35484 could deal with
zombie:35487,[zombie],kill parent process 35486 could deal with
zombie ppid:
35470
35472
35474
35476
35478
35480
35482
35484
35486
Should we clean up now?(y/n)y
clean success

4. 案例-磁盘空间不足

检查:
1.是否有block导致的磁盘空间不足
df -h |awk -F '[ %]+' '$5>=90 && NR>1'
2.是否有inode导致的磁盘空间不足
df -i |awk -F '[ %]+' '$5>=90 && NR>1'
处理:
情况1:df -h + find / -type f -size +1g
处理
情况2:lsof |grep delete 判断大小大于1g
情况3:df -i +find 大于10MB目录

---

[root@aliyun-ubuntu /server/scripts/array_shell]# cat check_disk_space.sh
#!/bin/bash
##############################################################
# File Name: check_disk_space.sh
# Version: V1.0
# Author: SunKexu
# Organization: www.oldboyedu.com
# Description:check disk
##############################################################
export LANG=en_US.UTF-8# block
block_used=`df -h / |awk -F '[ %]' 'NR==2{print $(NF-2)}'`
if [ $block_used -gt 60 ];thenecho "Insufficient disk space:${block_used}"big_files=`find / -type f -size +1G 2>/dev/null |grep -Ev '/proc|/sys' |xargs ls -l`size_total=`find / -type f -size +1G 2>/dev/null |grep -Ev '/proc|/sys' |xargs ls -l |awk '{sum=sum+$5}END{print sum/1024/1024/1024"GB"}'`echo "big file:${big_files}"echo "file size:${size_total}"
elseecho "ok"
fi
# inode
inode_used=`df -i / |awk -F '[ %]' 'NR==2{print $(NF-2)}'`
if [ "$inode_used" -ge 60 ];thenecho "Insufficient disk inode:${inode_used}"big_dirs=`find / -type d -size +1M`files_total=`find / -type d -size +1M |xargs ls |wc -l`echo "A directory with a large number of small files:${big_dirs}"echo "Number of files in the directory:${files_total}"
elseecho "ok"
fi
# Files that have not been completely deleted
# find big file
delete=`lsof |grep delete |awk '$(NF-3)>=1*1024^3' |wc -l`
if [ "$delete" -ge 1 ];thenlsof |grep delete |awk '$(NF-3)>=1*1024^3{print "command:"$1,"pid:"$2,"file name:"$(NF-1)}'echo "restart service"
elseecho "ok"
fi[root@aliyun-ubuntu /server/scripts/array_shell]# bash check_disk_space.sh
ok
ok
ok

5. sed补充

• sed的代码块使用单引号，但是这样不能解析外部变量；可使用双引号

5.1 案例-修改配置文件的参数

• grep -w：word，按照单词精确查找
  • -c：显示行数，相当于wc -l
• \b：表示边界

[root@aliyun-ubuntu /server/scripts/array_shell]# cat chcfg.sh     
#!/bin/bash
##############################################################
# File Name: chcfg.sh
# Version: V1.0
# Author: SunKexu
# Organization: www.oldboyedu.com
# Description:change config 
##############################################################
export LANG=en_US.UTF-8# vars
src=$1
dest=$2
file=$3# command
# check args
check_args(){if [ $# -ne 3 ];then echo "Usage:$0 src dest file"exit 1fi
}# check file and content
check_file_content(){if [ ! -f $file ];thenecho "$file file does not exist"exit 2ficount=`grep -Ev '^$|#' $file |grep -wc "$src"`if [ $count -eq 0 ];thenecho "${file} no ${src}"exit 3fi
}# replace
change_config(){old=`grep -Ev '^$|#' $file |grep -w "$src" |awk -F '[ =]' '{print $2}'`sed -i.bak "/\b${src}\b/s#${old}#${dest}#g" $file
}# main function
main(){check_args $*check_file_contentset -xchange_configset +x
}main $*

6. awk补充：awk数组

• awk数组的元素名称和元素值要加上双引号
• 遍历数组时，n表示数组下标名称，不是元素值
• 每个代码用分号分割

[root@aliyun-ubuntu ~]# awk 'BEGIN{arr["name"]="tom";arr["age"]="25";arr["addr"]="BJ";for(n in arr) print n,arr[n]}'
age 25
addr BJ
name tom

6.1 案例-统计每个域名出现的次数

• awk：先计算，再输出

[root@aliyun-ubuntu /server/scripts/log]# cat url.txt 
http://www.etiantian.org/index.html
http://www.etiantian.org/1.html
http://post.etiantian.org/index.html
http://mp3.etiantian.org/index.html
http://www.etiantian.org/3.html
http://post.etiantian.org/2.html

---

• awk是按行读取的，每读取一行，就取出第二列，作为url数组的元素名称，且该元素名称对应的元素值加1

[root@aliyun-ubuntu /server/scripts/log]# awk -F '/+' '{url[$2]++}END{for(u in url) print u,url[u]}' url.txt 
mp3.etiantian.org 1
www.etiantian.org 3
post.etiantian.org 2

6.2 案例-对access.log文件中统计ip流量

• 第10列是ip对应的流量数据

[root@aliyun-ubuntu /server/scripts/log]# awk '{ips[$1]+=$10}END{for(ip in ips) print ip,ips[ip]}' 1000w-oldboy-lidao996-access.log |sort -k2rn |head |column -t
220.181.51.41  11965949
220.181.51.57  11950521
220.181.51.58  6220735
220.181.51.42  6152526
61.135.165.36  4055321
61.135.165.5   4032632
61.135.169.5   4029569
61.135.165.42  4026574
61.135.165.37  4022820
61.135.165.39  4017123

7. 思维导图

【金山文档】 思维导图 https://www.kdocs.cn/l/co3I7PtpTYQX