kdump+coredunp+bpftrace

kdump

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/kernel_administration_guide/kernel_crash_dump_guide#sect-kdump-config-cli-default-action

https://blog.csdn.net/wtl666_6/article/details/129737794

https://docs.kernel.org/dev-tools/gdb-kernel-debugging.html

https://www.kernel.org/pub/linux/kernel/people/jwessel/kdb/

https://accelazh.github.io/kernel/Build-Linux-Kernel-and-Live-Debugging

https://github.com/Rhydon1337/linux-kernel-debugging

https://nxdong.com/linux-coredump/

https://zhuanlan.zhihu.com/p/354552321

https://forums.raspberrypi.com/viewtopic.php?t=375048

crash

https://blog.csdn.net/weixin_42915431/article/details/105666507

https://blog.csdn.net/vic_qxz/article/details/112771650

@@@@@@@@@@@@@@@@@

https://docs.redhat.com/zh-cn/documentation/red_hat_enterprise_linux/7/html/kernel_administration_guide/chap-installing-configuring-kdump#sect-kdump-install

https://buildlogs.centos.org/c7-kernels.x86_64/kernel/20200916135108/5.4.65-200.el7.x86_64/

http://debuginfo.centos.org/

https://www.alibabacloud.com/help/en/alinux/getting-started/modify-and-compile-the-rpm-package-of-the-alibaba-cloud-linux-kernel

https://zhuanlan.zhihu.com/p/469860384

https://winddoing.github.io/post/cb2d9d77.html

https://www.zhangfangzhou.cn/how-to-compile-linux-kernel-make-rpm.html

https://linux.cn/article-16252-1.html

https://blog.csdn.net/vic_qxz/article/details/112771650

https://blog.csdn.net/weixin_42915431/article/details/105666507

https://zhuanlan.zhihu.com/p/354552321

https://stackoverflow.com/questions/76500644/how-to-build-debug-symbols-from-source

https://serverfault.com/questions/1041080/how-to-build-kernel-debuginfo-from-upstream-kernel

https://nxdong.com/linux-coredump/

https://github.com/Rhydon1337/linux-kernel-debugging

https://accelazh.github.io/kernel/Build-Linux-Kernel-and-Live-Debugging

https://serverfault.com/questions/251134/how-to-compile-the-kernel-with-debug-symbols

https://blog.csdn.net/wtl666_6/article/details/129737794

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/kernel_administration_guide/kernel_crash_dump_guide#sect-kdump-config-cli-memory

https://zhuanlan.zhihu.com/p/555089668

https://docs.amd.com/r/en-US/57368-uProf-user-guide/Overview?tocId=uJrUpSwNOSdOfcxoqidI1Q

https://www.kernel.org/pub/linux/kernel/people/jwessel/kdb/

https://community.nxp.com/t5/i-MX-Processors/Build-Linux-Kernel-for-Debugging-in-Yocto/m-p/615085

https://docs.kernel.org/dev-tools/gdb-kernel-debugging.html

https://forums.raspberrypi.com/viewtopic.php?t=375048

https://superuser.com/questions/1780125/linux-kernel-compilation-debug-info-generated-albeit-disabling-debug-info

make localmodconfig
scripts/config --disable DEBUG_INFO
make bindeb-pkg

# readelf -S vmlinux-5.4.177-1.el7.elrepo.x86_64 | grep debug
[64] .debug_frame PROGBITS 0000000000000000 02a00030
[65] .rela.debug_frame RELA 0000000000000000 042014b0\

为什么rsyslog服务停了，因为lib lib64还原了，已经排查过了也找了椒图的 rsyslog的人都说没问题这个也认了，有没有努力过呢看dmesg.log messages.log没有发现开kdump也没有记录。下面是kdump sysctl配置基本都涵盖了。个人怀疑可能是哪里load 内核模块一直过不去kenel死了所以才没kdump。
kernel.panic=1
kernel.unknown_nmi_panic=1
kernel.panic_on_unrecovered_nmi=1
kernel.panic_on_io_nmi=1
kernel.softlockup_panic = 1
kernel.hung_task_panic=1
kernel.sysrq = 1
kernel.nmi_watchdog = 1
kernel.watchdog_thresh = 1

业务要的是稳定这是背景。问题的根本原因大概率还是安全的内核模块原因。你们要说装发出来配合你们，但是麻烦也查查行不不能只讲收集日志不管业务OK？？？？
1. kernel.panic=0:
解释：设置内核在发生 panic（内核崩溃）后不自动重启。0 表示不重启。
用途：用于调试目的，允许在崩溃后检查系统状态。
2. kernel.unknown_nmi_panic=1:
解释：设置内核在接收到未知 NMI（非屏蔽中断）时触发 panic。1 表示触发 panic。
用途：用于处理未知 NMI 事件，通常用于硬件故障检测。
kernel.panic_on_unrecovered_nmi=1:
解释：设置内核在无法恢复的 NMI 事件时触发 panic。1 表示触发 panic。
用途：确保在严重硬件故障时系统停止运行，以便进行故障排查。
kernel.panic_on_io_nmi=1:
解释：设置内核在 I/O NMI 事件时触发 panic。1 表示触发 panic。
用途：用于处理 I/O 相关的 NMI 事件，通常与硬件问题相关。
5. kernel.softlockup_panic=1:
解释：设置内核在检测到软锁死（soft lockup）时触发 panic。1 表示触发 panic。
用途：在系统检测到软锁死时重启，以避免系统长时间无响应。
kernel.hung_task_panic=1:
解释：设置内核在检测到挂起任务时触发 panic。1 表示触发 panic。
用途：用于检测和处理长时间挂起的任务，防止系统卡死。
kernel.sysrq=1:
解释：启用 SysRq（系统请求）键的功能。1 表示启用。
用途：允许使用 SysRq 键执行特定的内核命令，通常用于调试和紧急恢复。
kernel.nmi_watchdog=1:
解释：启用 NMI 看门狗。1 表示启用。
用途：用于检测和报告系统锁死情况。
9. kernel.watchdog_thresh=1:
解释：设置看门狗阈值（以秒为单位）。1 表示 1 秒。
用途：用于调整看门狗的灵敏度，较低的值可以更快地检测到系统问题。

grubby --info=ALL|grep ^kernel
grubby --set-default /boot/vmlinuz-5.10.134-16.1.an8.x86_64
grubby --update-kernel=ALL --args="crashkernel=1G"
grubby --default-kernel
grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

手动触发重启
echo c | sudo tee /proc/sysrq-trigger

是否支持nmi
cat /proc/sys/kernel/nmi_watchdog

确保 kdump 服务已启用并正在运行。
sudo systemctl status kdump
确保 crashkernel 参数在内核启动参数中正确设置，并且为 kdump 保留了足够的内存。
cat /proc/cmdline
确保 /etc/kdump.conf 中的配置正确，特别是转储文件的保存路径。
检查 initramfs：
确保 initramfs 包含 kdump 所需的模块和工具。使用 dracut 重新生成 initramfs：
sudo dracut -f

]# grep KEXEC /boot/config-$(uname -r)
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_ARCH_HAS_KEXEC_PURGATORY=y
# CONFIG_KEXEC_SIG is not set
CONFIG_KEXEC_JUMP=y
CONFIG_KEXEC_CORE=y

使用 kexec 工具来加载和启动新的内核。以下是一个基本的使用示例：

sudo kexec -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname -r).img --append="$(cat /proc/cmdline)"

　sudo kexec -e

# 进入/boot目录
cd /boot

# 解压bz2文件
bunzip2 -k vmlinux-5.4.177-1.el7.elrepo.x86_64.bz2

# 或者保留原文件并解压到新文件
bzcat vmlinux-5.4.177-1.el7.elrepo.x86_64.bz2 > vmlinux-5.4.177-1.el7.elrepo.x86_64

# 检查debug段
readelf -S vmlinux-5.4.177-1.el7.elrepo.x86_64 | grep debug

# 检查文件类型
file vmlinux-5.4.177-1.el7.elrepo.x86_64
!!!https://www.cnblogs.com/the-capricornus/p/17130474.html

@@@@@@@@@@@@@@@@@@@@@@

https://www.cnblogs.com/tiantao36/p/18426875

@@@@@@@@@@@@@@@@@@@@@@@@@

debuginfo

https://lishiwen4.github.io/linux-kernel/kernel-debug

https://forums.raspberrypi.com/viewtopic.php?t=375048

https://www.kernel.org/pub/linux/kernel/people/jwessel/kdb/CompilingAKernel.html

https://accelazh.github.io/linux/Build-Linux-Kernel-and-Live-Debugging

https://www.kernel.org/doc/html/v4.17/admin-guide/bug-hunting.html

https://docs.kernel.org/admin-guide/bug-hunting.html

https://stackoverflow.com/questions/12589605/debug-info-for-loadable-kernel-modules

# make sure at least below options are enabled
vim .config
CONFIG_DEBUG_KERNEL=y
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_DEBUG_INFO=y

内核编译

https://kernelnewbies.org/KernelBuild

https://www.kernel.org/doc/Documentation/kbuild/modules.txt

https://docs.redhat.com/zh-cn/documentation/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/proc_compiling-custom-kernel-modules_managing-kernel-modules

https://zhuanlan.zhihu.com/p/536391938

https://zhuanlan.zhihu.com/p/636318310

https://zhuanlan.zhihu.com/p/637842323

https://winddoing.github.io/post/cb2d9d77.html

https://github.com/armbian/build/issues/5218

https://lore.kernel.org/lkml/20220128214131.580131-1-keescook@chromium.org/t/

https://zhuanlan.zhihu.com/p/637842323

https://blog.csdn.net/2401_84003809/article/details/137398907

bpftrace

https://developer.aliyun.com/article/1592368

https://github.com/bpftrace/bpftrace/blob/master/INSTALL.md

mongosniff

https://www.mongodb.org.cn/manual/201.html

tcprecv

https://people.computing.clemson.edu/~westall/853/notes/tcprecv.pdf

bcc

https://pwl999.github.io/2018/10/16/bpf_bcc/

kernel-debug

https://lishiwen4.github.io/linux-kernel/kernel-debug

go-sniffer

https://github.com/40t/go-sniffer.git

mytrace

https://github.com/mYu4N/mytracer

系统卡顿

https://www.cnblogs.com/tiantao36/p/18426875

20250218

https://blog.csdn.net/2401_84003809/article/details/137398907

https://docs.redhat.com/zh-cn/documentation/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/proc_compiling-custom-kernel-modules_managing-kernel-modules#proc_compiling-custom-kernel-modules_managing-kernel-modules

https://lishiwen4.github.io/linux-kernel/kernel-debug

https://zhuanlan.zhihu.com/p/536391938

https://winddoing.github.io/post/cb2d9d77.html

https://zhuanlan.zhihu.com/p/637842323

https://blog.csdn.net/2401_84003809/article/details/137398907

网络分析

https://github.com/mYu4N/mytracer

https://github.com/bpftrace/bpftrace/blob/master/INSTALL.md

https://github.com/iovisor/bcc/blob/master/INSTALL.md

https://mp.weixin.qq.com/s/tVy0oTxhgrd5cjZsGPCyUA

进程状态

https://github.com/0voice/linux_kernel_wiki/blob/main/文章/进程管理/Linux进程状态总结.md

rpmbuild

https://blog.51cto.com/u_13999641/6173299

https://www.cnblogs.com/ray-mmss/p/10422969.html
https://github.com/iovisor/bpftrace/blob/master/INSTALL.md

https://zhuanlan.zhihu.com/p/469860384
dmesg -T|grep 10.252.33.214
/usr/local/src/linux-5.4.177
cp /boot/config-3.10.0-862.el7.x86_64 ./.config
make clean
make mrproper
make menuconfig | make oldconfig
make -j 32
make modules_install
make install
make rpm-pkg -j 32

https://mp.weixin.qq.com/s/r_WtwE6XISjEg-g8yCoPyg

./scripts/config -d COMPILE_TEST
./scripts/config -e DEBUG_KERNEL
./scripts/config -e FRAME_POINTER
./scripts/config -e KGDB
./scripts/config -e KGDB_SERIAL_CONSOLE
./scripts/config -e DEBUG_INFO
./scripts/config -e BPF
./scripts/config -e BPF_SYSCALL
./scripts/config -e BPF_JIT
./scripts/config -e HAVE_EBPF_JIT
./scripts/config -e BPF_EVENTS
./scripts/config -e FTRACE_SYSCALLS
./scripts/config -e FUNCTION_TRACER
./scripts/config -e HAVE_DYNAMIC_FTRACE
./scripts/config -e DYNAMIC_FTRACE
./scripts/config -e HAVE_KPROBES
./scripts/config -e KPROBES
./scripts/config -e KPROBE_EVENTS
./scripts/config -e ARCH_SUPPORTS_UPROBES
./scripts/config -e UPROBES
./scripts/config -e UPROBE_EVENTS
./scripts/config -e DEBUG_FS
./scripts/config -m NET_CLS_BPF
./scripts/config -m NET_ACT_BPF
./scripts/config -e HAVE_BPF_JIT
./scripts/config -e IKHEADERS
./scripts/config -e DEBUG_KMEMLEAK
./scripts/config -d DEBUG_KMEMLEAK_DEFAULT_OFF
./scripts/config -e KEXEC
./scripts/config -e CRASH_DUMP
./scripts/config -e BPF_LSM
./scripts/config -e BPF_PRELOAD

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
rpmbuild
https://blog.51cto.com/u_13999641/6173299

@@@@@@@@@@@@@@@@@@@@@@@@@@@@
./include/config/kernel.release

https://mirror.hostart.az/AltArch/7.8.2003/kernel/x86_64/Packages/
https://buildlogs.centos.org/c7-kernels.x86_64/kernel/20200916135108/5.4.65-200.el7.x86_64/

yum clean all
yum install make
yum install gcc
yum install flex
yum install bison
yum install lrzsz
yum install elfutils-libelf-devel
yum install openssl-devel
yum install bc

[root@k8s-node1 ~]# awk -F \' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
0 : CentOS Linux (4.18.12-1.el7.elrepo.x86_64) 7 (Core)
1 : CentOS Linux (3.10.0-957.el7.x86_64) 7 (Core)
2 : CentOS Linux (0-rescue-48003444b3ab47bfae9148b4d94764a4) 7 (Core)
You have new mail in /var/spool/mail/root
[root@k8s-node1 ~]#

查看当前default的entry
[root@k8s-node1 ~]# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-957.el7.x86_64) 7 (Core)
[root@k8s-node1 ~]#

修改为指定的entry
[root@k8s-node1 ~]# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-957.el7.x86_64) 7 (Core)
[root@k8s-node1 ~]# grub2-set-default 0
[root@k8s-node1 ~]# grub2-editenv list
saved_entry=0
[root@k8s-node1 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

5.4.65-200.el7.elrepo.x86_64

ansible -i inventory all -m shell -a "sudo /usr/bin/awk -F\"'\" '/^menuentry / {print \$2}' /etc/grub2.cfg" -b

awk -F\' '/menuentry / {print $2}' /etc/grub2.cfg

CONFIG_BPF_SYSCALL

yum install gcc gcc-c++ gcc-gfortran glibc-devel glibc-headers libquadmath-devel libtool systemtap systemtap-devel

find . | cpio -c -o | xz -9 --format=lzma > ../initrd.img