ubuntu 20.04 搭建crash dump问题分析环境

news/2025/2/28 6:15:47/

ubuntu 20.04 搭建crash dump问题分析环境

  • 1 安装依赖软件
    • 1.1 linux-dump
    • 1.2 kexec-tools
    • 1.3 安装crash工具
    • 1.4 安装gdb调试工具
    • 1.5 安装ubuntu内核调试符号
      • 1.5.1 GPG 秘钥导入
      • 1.5.2 添加仓库配置
      • 1.5.3 更新软件包
      • 1.5.4 下载和安装内核调试符号
      • 1.5.5 验证内核调试符号已经被安装
    • 1.6 配置转储内存大小
  • 2 触发系统异常验证
    • 2.1 开启kdump服务
    • 2.2 查看kdump服务当前的状态
    • 2.3 手动触发crash dump
      • 2.3.1 切换为 root 用户
      • 2.3.2 触发crash dump
  • 3 kernel dump文件分析
    • 3.1 crash dump文件的位置
    • 3.2 crash工具分析crash dump文件
  • 4 遇到的问题以及解决办法
    • 4.1 'makeinfo' is missing on your system
      • 4.1.1 问题现象
      • 4.1.2 解决方法
    • 4.2 crash工具和主机安装的gdb版本不一致导致的异常
      • 4.2.1 问题的现象
      • 4.2.2 解决办法
    • 4.3 /dev/mem: Operation not permitted
      • 4.3.1 问题的现象
      • 4.3.2 解决方法

主机环境和内核版本信息：

test@test:~/software/crash$ uname -r
5.15.0-74-generic
test@test:~/software/crash$test@test:~/software/crash$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
test@test:~/software/crash$

1 安装依赖软件

参考了Ubuntu Kernel crash dump这篇文章

1.1 linux-dump

sudo apt install linux-crashdump

1.2 kexec-tools

sudo apt-get install kexec-tools

1.3 安装crash工具

sudo apt install crash

1.4 安装gdb调试工具

sudo apt-get install gdb

1.5 安装ubuntu内核调试符号

sudo apt-get install linux-image-$(uname -r)-dbgsym

如果通过上面的命令安装ubuntu内核调试符号，则需要通过下面的方法去安装，参考文章：安装ubuntu内核调试符号

1.5.1 GPG 秘钥导入

确保您拥有系统的 GPG 密钥。适用于16.04 及更高版本的 Ubuntu ：

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622

对于旧的发布版本:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C0

1.5.2 添加仓库配置

codename=$(lsb_release -c | awk  '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

1.5.3 更新软件包

sudo apt-get update

1.5.4 下载和安装内核调试符号

sudo apt-get install linux-image-$(uname -r)-dbgsym

1.5.5 验证内核调试符号已经被安装

包含调试信息的文件名为 vmlinux-XXX-debug，其中 XXX 是内核版本。该文件存储在 /usr/lib/debug/boot 目录下。

1.6 配置转储内存大小

修改配置/etc/default/grub.d/kdump-tools.cfg，以支持kernel crash dump现场保存。

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=384M-:512M"

2 触发系统异常验证

2.1 开启kdump服务

systemctl start kdump-tools-dump.servicesystemctl enable kdump-tools-dump.service

2.2 查看kdump服务当前的状态

test@test:~/software/crash$ service kdump-tools-dump status
● kdump-tools-dump.service - Kernel crash dump capture serviceLoaded: loaded (/lib/systemd/system/kdump-tools-dump.service; static; vendor preset: enabled)Active: active (exited) since Tue 2023-11-14 10:19:58 CST; 56s agoMain PID: 126662 (code=exited, status=0/SUCCESS)Tasks: 0 (limit: 37610)Memory: 0BCGroup: /system.slice/kdump-tools-dump.service11月 14 10:19:58 test systemd[1]: Starting Kernel crash dump capture service...
11月 14 10:19:58 test kdump-tools[126662]: Starting kdump-tools:
11月 14 10:19:58 test kdump-tools[126669]:  * Cannot change symbolic links when kdump is loaded
11月 14 10:19:58 test systemd[1]: Finished Kernel crash dump capture service.
test@test:~/software/crash$

2.3 手动触发crash dump

2.3.1 切换为 root 用户

sudo -s

2.3.2 触发crash dump

echo 1 > /proc/sys/kernel/sysrqecho c > /proc/sysrq-trigger

3 kernel dump文件分析

3.1 crash dump文件的位置

test@test:~/software/crash$ ls /var/crash/
202311132045  kexec_cmd                                         linux-image-5.15.0-74-generic-202311132357.crash          _opt_kingsoft_wps-office_office6_wpscloudsvr.1000.uploaded  _usr_bin_crash.0.uploaded
202311132231  linux-image-5.15.0-74-generic-202311132045.crash  _opt_kingsoft_wps-office_office6_wpscloudsvr.1000.crash   _usr_bin_crash.0.crash
202311132357  linux-image-5.15.0-74-generic-202311132231.crash  _opt_kingsoft_wps-office_office6_wpscloudsvr.1000.upload  _usr_bin_crash.0.upload
test@test:~/software/crash$

3.2 crash工具分析crash dump文件

test@test:~/software/crash$ sudo crash -d /usr/bin/gdb /usr/lib/debug/boot/vmlinux-5.15.0-74-generic /var/crash/202311132357/dump.202311132357 crash 8.0.3++
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:<http://www.gnu.org/software/gdb/documentation/>.For help, type "help".
Type "apropos word" to search for commands related to "word"...KERNEL: /usr/lib/debug/boot/vmlinux-5.15.0-74-genericDUMPFILE: /var/crash/202311132357/dump.202311132357  [PARTIAL DUMP]CPUS: 16DATE: Mon Nov 13 23:56:58 CST 2023UPTIME: 00:11:00
LOAD AVERAGE: 0.71, 0.84, 0.58TASKS: 1636NODENAME: testRELEASE: 5.15.0-74-genericVERSION: #81~20.04.2-Ubuntu SMP Fri May 26 19:56:20 UTC 2023MACHINE: x86_64  (2900 Mhz)MEMORY: 31.8 GBPANIC: "Kernel panic - not syncing: sysrq triggered crash"PID: 10269COMMAND: "bash"TASK: ffff89a4d1334d40  [THREAD_INFO: ffff89a4d1334d40]CPU: 0STATE: TASK_RUNNING (PANIC)crash> bt
PID: 10269    TASK: ffff89a4d1334d40  CPU: 0    COMMAND: "bash"#0 [ffff94d441397c48] machine_kexec at ffffffff9688afe0#1 [ffff94d441397ca8] __crash_kexec at ffffffff96998c02#2 [ffff94d441397d78] panic at ffffffff974cf4b6#3 [ffff94d441397df8] sysrq_handle_crash at ffffffff96feea5a#4 [ffff94d441397e08] __handle_sysrq.cold at ffffffff97524bf2#5 [ffff94d441397e50] write_sysrq_trigger at ffffffff96fef548#6 [ffff94d441397e68] proc_reg_write at ffffffff96c23b57#7 [ffff94d441397e88] vfs_write at ffffffff96b83a76#8 [ffff94d441397ec0] ksys_write at ffffffff96b85de7#9 [ffff94d441397f00] __x64_sys_write at ffffffff96b85e8a
#10 [ffff94d441397f10] do_syscall_64 at ffffffff97571d39
#11 [ffff94d441397f28] do_syscall_64 at ffffffff97571d49
#12 [ffff94d441397f50] entry_SYSCALL_64_after_hwframe at ffffffff97600099RIP: 00007fc202171077  RSP: 00007ffe9fcff018  RFLAGS: 00000246RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007fc202171077RDX: 0000000000000002  RSI: 000055c60be1a9e0  RDI: 0000000000000001RBP: 000055c60be1a9e0   R8: 000000000000000a   R9: 0000000000000001R10: 000055c60b144017  R11: 0000000000000246  R12: 0000000000000002R13: 00007fc2022506a0  R14: 00007fc20224c4a0  R15: 00007fc20224b8a0ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
crash>

4 遇到的问题以及解决办法

4.1 ‘makeinfo’ is missing on your system

4.1.1 问题现象

/home/test/software/crash/gdb-10.2/missing: line 81: makeinfo: command not found
WARNING: 'makeinfo' is missing on your system.You should only need it if you modified a '.texi' file, orany other file indirectly affecting the aspect of the manual.You might want to install the Texinfo package:<http://www.gnu.org/software/texinfo/>The spurious makeinfo call might also be the consequence ofusing a buggy 'make' (AIX, DU, IRIX), in which case you mightwant to install GNU make:<http://www.gnu.org/software/make/>
make[5]: *** [Makefile:542: bfd.info] Error 127
make[4]: *** [Makefile:1643: info-recursive] Error 1
make[3]: *** [Makefile:2771: all-bfd] Error 2
make[3]: *** Waiting for unfinished jobs....
config.status: creating Makefile
config.status: creating import/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing default commands
make[2]: *** [Makefile:860: all] Error 2
crash build failed
make[1]: *** [Makefile:263: gdb_merge] Error 1
make: *** [Makefile:254: all] Error 2

4.1.2 解决方法

sudo apt-get update
sudo apt-get install texinfo

4.2 crash工具和主机安装的gdb版本不一致导致的异常

4.2.1 问题的现象

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...WARNING: kernel relocated [344MB]: patching 145829 gdb minimal_symbol valuesplease wait... (patching 145829 gdb minimal_symbol values) Segmentation faulttest@test:~/software/crash$ gdb --version
GNU gdb (Ubuntu 10.2-0ubuntu1~20.04~1) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
test@test:~/software/crash$

4.2.2 解决办法

sudo apt purge crashgit clone https://github.com/crash-utility/crash.gitcd crashmake -j8
sudo make install
crash --version

4.3 /dev/mem: Operation not permitted

4.3.1 问题的现象

sudo crash /usr/lib/debug/boot/vmlinux-5.15.0-74-genericcrash 7.2.8
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.crash: /dev/mem: Operation not permitted

4.3.2 解决方法

参考1.6节，修改/etc/default/grub.d/kdump-tools.cfg文件，添加下面的配置以支持

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT` crashkernel=384M-:512M"

然后修改/dev/mem的权限

sudo chmod 777 /dev/mem

http://www.ppmy.cn/news/1234726.html

相关文章

Java --- JVM之垃圾回收相关知识概念

Java --- JVM之垃圾回收相关知识概念

目录 一、System.gc() 二、内存溢出与内存泄漏 2.1、内存溢出 2.2、内存泄漏 三、Stop the world 四、垃圾回收的并行与并发 4.1、并发 4.2、并行 4.3、并行 vs 并发 4.4、垃圾回收的并发与并行 五、安全点与安全区域 5.1、安全点 5.2、安全区域 六、引用 6.1…
阅读更多...
数据结构重点知识点复习——第二章 线性表（顺序表）

数据结构重点知识点复习——第二章 线性表（顺序表）

目录 一、线性表 二、基本概念 2.1顺序表的插入 思路 代码 复杂度 2.2顺序表的删除 思路 代码 复杂度 查找 一、线性表 二、基本概念 线性表的顺序存储又称为顺序表。 它是用一组地址连续的存储单元&#xff08;比如C语言里面的数组&#xff09;&#xff0c;依次存储线性表…
阅读更多...
【GUI】-- 13 贪吃蛇小游戏之食物及成绩判断

【GUI】-- 13 贪吃蛇小游戏之食物及成绩判断

GUI编程 04 贪吃蛇小游戏 4.4 第四步&#xff1a;食物及成绩判断 首先&#xff0c;添加食物与分数的数据定义&#xff1a; //食物的坐标int foodX;int foodY;Random random new Random();//积分面板数据结构int score;在初始化方法中&#xff0c;添加(画出)食物与分数&…
阅读更多...
【NLP】GPT 模型如何工作

【NLP】GPT 模型如何工作

介绍 2021 年&#xff0c;我使用 GPT 模型编写了最初的几行代码&#xff0c;那时我意识到文本生成已经达到了拐点。我要求 GPT-3 总结一份很长的文档&#xff0c;并尝试了几次提示。我可以看到结果比以前的模型先进得多&#xff0c;这让我对这项技术感到兴奋&#xff0c;并渴望…
阅读更多...
用于 syslog 收集的协议：TCP、UDP、RELP

用于 syslog 收集的协议：TCP、UDP、RELP

系统日志是从 Linux/Unix 设备和其他网络设备&#xff08;如交换机、路由器和防火墙&#xff09;生成的日志 可以通过将 syslog 聚合到称为 syslog 服务器、syslog 守护程序或 syslogd 的服务器来集中 syslog。在TCP、UDP和RELP协议的帮助下&#xff0c;系统日志从设备传输到系…
阅读更多...
MacOS 成为恶意软件活动的目标

MacOS 成为恶意软件活动的目标

Malwarebytes 警告称&#xff0c;一个针对 Mac 操作系统 (OS) 的数据窃取程序正在通过虚假的网络浏览器更新分发给毫无戒心的目标。 Atomic Stealer&#xff0c;也称为 AMOS&#xff0c;是 Mac OS 上流行的窃取程序。 Atomic Stealer (AMOS) 恶意软件最近被发现使用“ClearFa…
阅读更多...
力扣第462题 最小操作次数使数组元素相等 II C++ 排序基础 附Java代码

力扣第462题 最小操作次数使数组元素相等 II C++ 排序基础 附Java代码

题目 462. 最小操作次数使数组元素相等 II 中等 相关标签 数组 数学 排序 给你一个长度为 n 的整数数组 nums &#xff0c;返回使所有数组元素相等需要的最小操作数。 在一次操作中&#xff0c;你可以使数组中的一个元素加 1 或者减 1 。 示例 1&#xff1a; 输入&a…
阅读更多...
浅谈WPF之各种Template

浅谈WPF之各种Template

前几天写了一篇文章【浅谈WPF之控件模板和数据模板】&#xff0c;有粉丝反馈说这两种模板容易弄混&#xff0c;不知道什么时候该用控件模块&#xff0c;什么时候该用数据模板&#xff0c;以及template和itemtemplate之间的关系等&#xff0c;今天专门写一篇文章&#xff0c;简述…
阅读更多...
最新文章

Copyright @ 2022~2023