抓虫:unshared后执行命令dump
文章目录
- 已知
- 简单看看
- 再看
已知
# unshare -m bash -c "mount --bind /opt/cmt/.venv/lib64 /lib64 && ls"
段错误 (核心已转储)
bash
挂掉了
# coredumpctl
TIME PID UID GID SIG COREFILE EXE
Wed 2025-09-10 10:32:31 CST 66363 0 0 11 none /usr/bin/bash
-m, --mount[=file]Unshare the mount namespace. If file is specified, then a persistent namespace is created by a bind mount. Note that file hasto be located on a filesystem with the propagation flag set to private. Use the command findmnt -o+PROPAGATION when not sureabout the current setting. See also the examples below.
unshare -m
会创建一个新的mount ns
,后面会执行的mount
会只在新的ns有效
开发人员疑问,创建新的ns后,bash
进程先正常执行,猜测bash
执行其他进程时候会fork
再execve
,为什么bash
会dump???即使是dump,可能mount
后环境问题不应该是ls
进程出问题吗???
简单看看
分步执行没有问题
# unshare -m bash
# mount --bind /opt/cmt/.venv/lib64 /lib64 && ls
anaconda-ks.cfg install_panel.sh rear-localhost.log test userdata.txt
先strace
看一下,执行到mount
没有问题,正常退出,看起来是要执行ls
时候出现了问题
# strace -f -e fork -e clone -e execve unshare -m bash -c "mount --bind /opt/cmt/.venv/lib64 /lib64 && ls"
execve("/usr/bin/unshare", ["unshare", "-m", "bash", "-c", "mount --bind /opt/cmt/.venv/lib6"...], 0x7ffeb12f22f8 /* 23 vars */) = 0
execve("/usr/local/sbin/bash", ["bash", "-c", "mount --bind /opt/cmt/.venv/lib6"...], 0x7fff39c963e8 /* 23 vars */) = -1 ENOENT (没有那个文件或目录)
execve("/usr/local/bin/bash", ["bash", "-c", "mount --bind /opt/cmt/.venv/lib6"...], 0x7fff39c963e8 /* 23 vars */) = -1 ENOENT (没有那个文件或目录)
execve("/usr/sbin/bash", ["bash", "-c", "mount --bind /opt/cmt/.venv/lib6"...], 0x7fff39c963e8 /* 23 vars */) = -1 ENOENT (没有那个文件或目录)
execve("/usr/bin/bash", ["bash", "-c", "mount --bind /opt/cmt/.venv/lib6"...], 0x7fff39c963e8 /* 23 vars */) = 0
strace: Process 66503 attached
[pid 66503] execve("/usr/bin/mount", ["mount", "--bind", "/opt/cmt/.venv/lib64", "/lib64"], 0x559b627273a0 /* 23 vars */) = 0
[pid 66503] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=66503, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV (core dumped) +++
段错误 (核心已转储)
去掉strace
过滤看以下,报错是SEGV_MAPERR
,刚好之前好几次munmap
,取消映射
strace unshare -m bash -c "mount --bind /opt/cmt/.venv/lib64 /lib64 && ls"
openat(AT_FDCWD, "/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 5
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260y\0\0\0\0\0\0"..., 832) = 832
lseek(5, 156256, SEEK_SET) = 156256
read(5, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32) = 32
fstat(5, {st_mode=S_IFREG|0755, st_size=193024, ...}) = 0
lseek(5, 156256, SEEK_SET) = 156256
read(5, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 2266608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7f62f7195000
mprotect(0x7f62f71bc000, 2093056, PROT_NONE) = 0
mmap(0x7f62f73bb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x26000) = 0x7f62f73bb000
mmap(0x7f62f73bd000, 5616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f62f73bd000
close(5) = 0
openat(AT_FDCWD, "/lib64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 5
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P#\0\0\0\0\0\0"..., 832) = 832
lseek(5, 532816, SEEK_SET) = 532816
read(5, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32) = 32
fstat(5, {st_mode=S_IFREG|0755, st_size=553480, ...}) = 0
lseek(5, 532816, SEEK_SET) = 532816
read(5, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 2634280, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7f62f6f11000
mprotect(0x7f62f6f94000, 2093056, PROT_NONE) = 0
mmap(0x7f62f7193000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x82000) = 0x7f62f7193000
close(5) = 0
mprotect(0x7f62f7193000, 4096, PROT_READ) = 0
mprotect(0x7f62f73bb000, 4096, PROT_READ) = 0
mprotect(0x7f62f75c5000, 4096, PROT_READ) = 0
mprotect(0x7f62f7812000, 20480, PROT_READ) = 0
mprotect(0x7f62f7a2f000, 4096, PROT_READ) = 0
munmap(0x7f62f9d8b000, 25771) = 0
munmap(0x7f62f8098000, 2445528) = 0
munmap(0x7f62f7c8a000, 2113848) = 0
munmap(0x7f62f7a31000, 2458880) = 0
munmap(0x7f62f7e8f000, 2132928) = 0
munmap(0x7f62f7819000, 2192080) = 0
munmap(0x7f62f75c7000, 2430720) = 0
munmap(0x7f62f73bf000, 2125840) = 0
munmap(0x7f62f7195000, 2266608) = 0
munmap(0x7f62f6f11000, 2634280) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV (core dumped) +++
段错误 (核心已转储)
进程找了一些/lib64
下的动态库,映射到内存,然后取消映射
openat(AT_FDCWD, "/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 5
mmap(NULL, 2266608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7f62f7195000
openat(AT_FDCWD, "/lib64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 5
mmap(NULL, 2634280, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7f62f6f11000
......
munmap(0x7f62f7195000, 2266608) = 0
munmap(0x7f62f6f11000, 2634280) = 0
问题是:
strace
显示munmap
好像返回了0,好像成功了,返回成功了为什么还会sigsegv
呢???
难道是还有一个munmap
没成功,strace
没显示出来???
看一下bash
进程的munmap
,看起来是没有多余的munmap
系统调用了,排除munmap
系统调用入口时就抛出异常没触发ptrace
跟踪导致strace
没显示。
bpftrace -e 'tracepoint:syscalls:sys_enter_munmap {printf("%d %s %d %d\n", pid, comm, args->addr, args->len)}'
Attaching 1 probe...
......
67402 bash 1380843520 2266608
67402 bash 1378205696 2634280 // 和上面对应,都是最后一条 munmap
还是看一下内核何时触发的sigsegv
吧。。。
再看
刚才bpftrace
已经看到只有这个出错的bash
会发起munmap
系统调用,所以,当bash
进程munmap
系统调用时,暂停bash
# bpftrace -e 'tracepoint:syscalls:sys_enter_munmap / comm == "bash" / {signal(20); printf("%d %s\n", pid, comm); exit();}' --unsafe
Attaching 1 probe...
67481 bash
另一个终端执行一下出错的命令,就会暂停
# unshare -m bash -c "mount --bind /opt/cmt/.venv/lib64 /lib64 && ls "[1]+ 已停止 unshare -m bash -c "mount --bind /opt/cmt/.venv/lib64 /lib64 && ls "
暂停后附加:
# gdb -p 68924
(gdb) c
Program received signal SIGSEGV, Segmentation fault.
0x00007efdffacd15b in usranalyse_log_syscall_exec (syscallName=syscallName@entry=0x7efdffad210a "execve",filename=filename@entry=0x560fe3df3270 "/usr/bin/ls", argv=argv@entry=0x560fe3df0910, envp=envp@entry=0x560fe3df03a0,result=result@entry=0x7ffc9d1567dc) at log.c:111
111 log.c: 没有那个文件或目录.
(gdb) bt
#0 0x00007efdffacd15b in usranalyse_log_syscall_exec (syscallName=syscallName@entry=0x7efdffad210a "execve",filename=filename@entry=0x560fe3df3270 "/usr/bin/ls", argv=argv@entry=0x560fe3df0910, envp=envp@entry=0x560fe3df03a0,result=result@entry=0x7ffc9d1567dc) at log.c:111
#1 0x00007efdffacd5f6 in usranalyse_log_syscall_execve (filename=filename@entry=0x560fe3df3270 "/usr/bin/ls", argv=argv@entry=0x560fe3df0910,envp=envp@entry=0x560fe3df03a0) at log.c:101
#2 0x00007efdffad1d95 in execve (filename=0x560fe3df3270 "/usr/bin/ls", argv=0x560fe3df0910, envp=0x560fe3df03a0) at usr_protect.c:115
#3 0x0000560fe1cfe286 in shell_execve (command=command@entry=0x560fe3df3270 "/usr/bin/ls", args=0x560fe3df0910, env=0x560fe3df03a0)at execute_cmd.c:5485
#4 0x0000560fe1d00382 in execute_disk_command (cmdflags=<optimized out>, fds_to_close=0x560fe3df2930, async=0, pipe_out=-1, pipe_in=-1,command_line=0x560fe3df0580 "ls", redirects=0x0, words=0x560fe3df2cf0) at execute_cmd.c:5279
#5 execute_simple_command (simple_command=<optimized out>, pipe_in=pipe_in@entry=-1, pipe_out=pipe_out@entry=-1, async=async@entry=0,fds_to_close=fds_to_close@entry=0x560fe3df2930) at execute_cmd.c:4407
#6 0x0000560fe1d011a6 in execute_command_internal (command=0x560fe3df2780, asynchronous=0, pipe_in=-1, pipe_out=-1,fds_to_close=0x560fe3df2930) at execute_cmd.c:819
#7 0x0000560fe1d02a56 in execute_command (command=0x560fe3df2780) at execute_cmd.c:409
#8 0x0000560fe1d02727 in execute_connection (fds_to_close=0x560fe3df2850, pipe_out=-1, pipe_in=-1, asynchronous=0, command=0x560fe3df2820)at execute_cmd.c:2687
#9 execute_command_internal (command=0x560fe3df2820, asynchronous=asynchronous@entry=0, pipe_in=pipe_in@entry=-1, pipe_out=pipe_out@entry=-1,fds_to_close=fds_to_close@entry=0x560fe3df2850) at execute_cmd.c:988
#10 0x0000560fe1d4ae34 in parse_and_execute (string=<optimized out>, from_file=0x560fe1d8d86d "-c", flags=4) at evalstring.c:421
#11 0x0000560fe1ce9802 in run_one_command (command=<optimized out>) at /usr/include/bits/string_fortified.h:90
#12 0x0000560fe1ce86a4 in main (argc=3, argv=0x7ffc9d156ee8, env=0x7ffc9d156f08) at shell.c:734
log.c: 没有那个文件或目录.
,最近的一个栈是0x00007efdffacd15b
,查找这个地址属于哪个库,安装这个库的debug包
(gdb) info proc
process 68924# cat /proc/68924/maps
7efdffac7000-7efdffad5000 r-xp 00000000 fd:00 35932266 /usr/local/usranalyse/lib/libusranalyse.so
很奇怪,不知道这个文件哪来的,readelf -d/ldd
也没看到谁依赖这个动态库
# rpm -qf /usr/local/usranalyse/lib/libusranalyse.so
文件 /usr/local/usranalyse/lib/libusranalyse.so 不属于任何软件包
发现是读取/etc/ld.so.preload
这里后才映射这个文件的
# strace -f file bash 2>&1 |grep usran -C 5
access("/etc/ld.so.preload", R_OK) = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=44, ...}) = 0
mmap(NULL, 44, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x7fd42d8b2000
close(3) = 0
openat(AT_FDCWD, "/usr/local/usranalyse/lib/libusranalyse.so", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0PY\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=299016, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd42d8b0000
mmap(NULL, 2157240, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd42d478000
mprotect(0x7fd42d486000, 2093056, PROT_NONE) = 0
原来如此,开发者可能自己添加的,这个动态库没有对应的rpm包,没有对应源码,这个动态库劫持了libc
里的execve
,可能做了一些什么导致了挂掉
# cat /etc/ld.so.preload/usr/local/usranalyse/lib/libusranalyse.so
google
搜这个动态库,三条结果都指向宝塔面板。。。
libusranalyse.so cannot be preloaded
WP Cron Job - Help setting up
【已完成】升级后出现ERROR: ld.so: object
/etc/ld.so.preload
里的内容移除下,再试一次:
# unshare -m bash -c "mount --bind /opt/cmt/.venv/lib64 /lib64 && ls"
anaconda-ks.cfg bash bash-4.4.20-2.el8.src.rpm install_panel.sh rear-localhost.log rpmbuild test texinfo-6.5-6.el8.src.rpm userdata.txt
ok,完结
问题出在宝塔面报有个动态库会劫持libc
里的一些系统调用封装,可能宝塔这个功能写的不完善,某些情况下会dump。。。