前言
最近在适配 ubuntu系统，记录一下其crash的安装。
linux_9">一、apt 安装dbgsym vnlinux
# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiversedeb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiversedeb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
$ sudo apt install ubuntu-dbgsym-keyring
$ sudo apt-get update
# apt -y install linux-image-$(uname -r)-dbgsym
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:linux-image-unsigned-5.15.0-122-generic-dbgsym
The following NEW packages will be installed:linux-image-5.15.0-122-generic-dbgsym linux-image-unsigned-5.15.0-122-generic-dbgsym
0 upgraded, 2 newly installed, 0 to remove and 42 not upgraded.
Need to get 1,061 MB of archives.
After this operation, 6,917 MB of additional disk space will be used.
Get:1 http://ddebs.ubuntu.com jammy-updates/main amd64 linux-image-unsigned-5.15.0-122-generic-dbgsym amd64 5.15.0-122.132 [1,061 MB]
linux_37">二、使用.ddeb包安装dbgsym vnlinux
也可以直接在 http://ddebs.ubuntu.com/pool/main/l/linux/下载对应的.ddeb包：
dpkg -i linux-image-unsigned-5.15.0-122-generic-dbgsym_5.15.0-122.132_amd64.ddeb
三、dbgsym发行版
在http://ddebs.ubuntu.com/pool/main/l/linux/查看支持的dbgsym vmlinux：
| 发行版 | 内核版本 |
|---|---|
| 14.04 | 3.13.0 |
| 16.04 | 4.4.0 |
| 18.04 | 4.15.0 |
| 20.04 | 5.4.0 |
| 22.04 | 5.15.0 |
| 24.04 | 6.8.0 |
其他内核版本，请参考：http://ddebs.ubuntu.com/pool/main/l/
四、crash调试
# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy# uname -r
6.5.0-18-generic
我这里用ubuntu22.04 的6.5.0-18-generic内核调试为例：
/** 64-bit SYSCALL instruction entry. Up to 6 arguments in registers.** This is the only entry point used for 64-bit system calls. The* hardware interface is reasonably well designed and the register to* argument mapping Linux uses fits well with the registers that are* available when SYSCALL is used.** SYSCALL instructions can be found inlined in libc implementations as* well as some other programs and libraries. There are also a handful* of SYSCALL instructions in the vDSO used, for example, as a* clock_gettimeofday fallback.** 64-bit SYSCALL saves rip to rcx, clears rflags.RF, then saves rflags to r11,* then loads new ss, cs, and rip from previously programmed MSRs.* rflags gets masked by a value from another MSR (so CLD and CLAC* are not needed). SYSCALL does not save anything on the stack* and does not change rsp.** Registers on entry:* rax system call number* rcx return address* r11 saved rflags (note: r11 is callee-clobbered register in C ABI)* rdi arg0* rsi arg1* rdx arg2* r10 arg3 (needs to be moved to rcx to conform to C ABI)* r8 arg4* r9 arg5* (note: r12-r15, rbp, rbx are callee-preserved in C ABI)** Only called from user space.** When user can change pt_regs->foo always force IRET. That is because* it deals with uncanonical addresses better. SYSRET has trouble* with them due to bugs in both AMD and Intel CPUs.*/SYM_CODE_START(entry_SYSCALL_64)UNWIND_HINT_ENTRYENDBRswapgs/* tss.sp2 is scratch space. */movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)SWITCH_TO_KERNEL_CR3 scratch_reg=%rspmovq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rspSYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL)ANNOTATE_NOENDBR/* Construct struct pt_regs on stack */pushq $__USER_DS /* pt_regs->ss */pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* pt_regs->sp */pushq %r11 /* pt_regs->flags */pushq $__USER_CS /* pt_regs->cs */pushq %rcx /* pt_regs->ip */
SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)......
SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)pushq %rax /* pt_regs->orig_ax */PUSH_AND_CLEAR_REGS rax=$-ENOSYS/* IRQs are off. */movq %rsp, %rdi/* Sign extend the lower 32bit as syscall numbers are treated as int */movslq %eax, %rsi/* clobbers %rax, make sure it is after saving the syscall nr */IBRS_ENTERUNTRAIN_RETcall do_syscall_64 /* returns with IRQs disabled */......
看其vmlinux：
cat /boot/System.map-6.5.0-18-generic | grep entry_SYSCALL_64_after_hwframe
ffffffff82200078 T entry_SYSCALL_64_after_hwframe
ffffffff82200078: 50 push %rax
ffffffff82200079: 57 push %rdi
ffffffff8220007a: 56 push %rsi
ffffffff8220007b: 52 push %rdx
ffffffff8220007c: 51 push %rcx
ffffffff8220007d: 6a da push $0xffffffffffffffda
ffffffff8220007f: 41 50 push %r8
ffffffff82200081: 41 51 push %r9
ffffffff82200083: 41 52 push %r10
ffffffff82200085: 41 53 push %r11
ffffffff82200087: 53 push %rbx
ffffffff82200088: 55 push %rbp
ffffffff82200089: 41 54 push %r12
ffffffff8220008b: 41 55 push %r13
ffffffff8220008d: 41 56 push %r14
ffffffff8220008f: 41 57 push %r15
ffffffff82200091: 31 f6 xor %esi,%esi
ffffffff82200093: 31 d2 xor %edx,%edx
......
ffffffff822000d1: 90 nop
ffffffff822000d2: 90 nop
ffffffff822000d3: 90 nop
ffffffff822000d4: 90 nop
ffffffff822000d5: 90 nop
ffffffff822000d6: 90 nop
ffffffff822000d7: 90 nop
ffffffff822000d8: 90 nop
ffffffff822000d9: 90 nop
ffffffff822000da: 90 nop
ffffffff822000db: 90 nop
ffffffff822000dc: 90 nop
ffffffff822000dd: 90 nop
ffffffff822000de: 90 nop
ffffffff822000df: 90 nop
ffffffff822000e0: 90 nop
ffffffff822000e1: e8 9a 4a f1 ff call 0xffffffff82114b80
ffffffff822000e6: 90 nop
ffffffff822000e7: 90 nop
ffffffff822000e8: 90 nop
ffffffff822000e9: 90 nop
ffffffff822000ea: 90 nop
到了ffffffff822000d1处，全是 nop指令：
# cat /boot/System.map-6.5.0-18-generic | grep '\<do_syscall_64\>'
ffffffff82114b80 T do_syscall_64
而我看crash：
crash> dis entry_SYSCALL_64_after_hwframe
0xffffffff97400078 <entry_SYSCALL_64_after_hwframe>: push %rax
0xffffffff97400079 <entry_SYSCALL_64_after_hwframe+1>: push %rdi
0xffffffff9740007a <entry_SYSCALL_64_after_hwframe+2>: push %rsi
0xffffffff9740007b <entry_SYSCALL_64_after_hwframe+3>: push %rdx
0xffffffff9740007c <entry_SYSCALL_64_after_hwframe+4>: push %rcx
0xffffffff9740007d <entry_SYSCALL_64_after_hwframe+5>: push $0xffffffffffffffda
0xffffffff9740007f <entry_SYSCALL_64_after_hwframe+7>: push %r8
0xffffffff97400081 <entry_SYSCALL_64_after_hwframe+9>: push %r9
0xffffffff97400083 <entry_SYSCALL_64_after_hwframe+11>: push %r10
0xffffffff97400085 <entry_SYSCALL_64_after_hwframe+13>: push %r11
0xffffffff97400087 <entry_SYSCALL_64_after_hwframe+15>: push %rbx
0xffffffff97400088 <entry_SYSCALL_64_after_hwframe+16>: push %rbp
0xffffffff97400089 <entry_SYSCALL_64_after_hwframe+17>: push %r12
0xffffffff9740008b <entry_SYSCALL_64_after_hwframe+19>: push %r13
0xffffffff9740008d <entry_SYSCALL_64_after_hwframe+21>: push %r14
0xffffffff9740008f <entry_SYSCALL_64_after_hwframe+23>: push %r15
0xffffffff97400091 <entry_SYSCALL_64_after_hwframe+25>: xor %esi,%esi
0xffffffff97400093 <entry_SYSCALL_64_after_hwframe+27>: xor %edx,%edx
0xffffffff97400095 <entry_SYSCALL_64_after_hwframe+29>: xor %ecx,%ecx
0xffffffff97400097 <entry_SYSCALL_64_after_hwframe+31>: xor %r8d,%r8d
0xffffffff9740009a <entry_SYSCALL_64_after_hwframe+34>: xor %r9d,%r9d
0xffffffff9740009d <entry_SYSCALL_64_after_hwframe+37>: xor %r10d,%r10d
0xffffffff974000a0 <entry_SYSCALL_64_after_hwframe+40>: xor %r11d,%r11d
0xffffffff974000a3 <entry_SYSCALL_64_after_hwframe+43>: xor %ebx,%ebx
0xffffffff974000a5 <entry_SYSCALL_64_after_hwframe+45>: xor %ebp,%ebp
0xffffffff974000a7 <entry_SYSCALL_64_after_hwframe+47>: xor %r12d,%r12d
0xffffffff974000aa <entry_SYSCALL_64_after_hwframe+50>: xor %r13d,%r13d
0xffffffff974000ad <entry_SYSCALL_64_after_hwframe+53>: xor %r14d,%r14d
0xffffffff974000b0 <entry_SYSCALL_64_after_hwframe+56>: xor %r15d,%r15d
0xffffffff974000b3 <entry_SYSCALL_64_after_hwframe+59>: mov %rsp,%rdi
0xffffffff974000b6 <entry_SYSCALL_64_after_hwframe+62>: movslq %eax,%rsi
0xffffffff974000b9 <entry_SYSCALL_64_after_hwframe+65>: xchg %ax,%ax
0xffffffff974000bb <entry_SYSCALL_64_after_hwframe+67>: mov $0x48,%ecx
0xffffffff974000c0 <entry_SYSCALL_64_after_hwframe+72>: mov %gs:0x1fbd0,%rdx
0xffffffff974000c9 <entry_SYSCALL_64_after_hwframe+81>: mov %edx,%eax
0xffffffff974000cb <entry_SYSCALL_64_after_hwframe+83>: shr $0x20,%rdx
0xffffffff974000cf <entry_SYSCALL_64_after_hwframe+87>: wrmsr
0xffffffff974000d1 <entry_SYSCALL_64_after_hwframe+89>: jmp 0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>
0xffffffff974000d3 <entry_SYSCALL_64_after_hwframe+91>: int3
0xffffffff974000d4 <entry_SYSCALL_64_after_hwframe+92>: int3
0xffffffff974000d5 <entry_SYSCALL_64_after_hwframe+93>: int3
0xffffffff974000d6 <entry_SYSCALL_64_after_hwframe+94>: int3
0xffffffff974000d7 <entry_SYSCALL_64_after_hwframe+95>: int3
0xffffffff974000d8 <entry_SYSCALL_64_after_hwframe+96>: int3
0xffffffff974000d9 <entry_SYSCALL_64_after_hwframe+97>: int3
0xffffffff974000da <entry_SYSCALL_64_after_hwframe+98>: int3
0xffffffff974000db <entry_SYSCALL_64_after_hwframe+99>: int3
0xffffffff974000dc <entry_SYSCALL_64_after_hwframe+100>: int3
0xffffffff974000dd <entry_SYSCALL_64_after_hwframe+101>: int3
0xffffffff974000de <entry_SYSCALL_64_after_hwframe+102>: int3
0xffffffff974000df <entry_SYSCALL_64_after_hwframe+103>: int3
0xffffffff974000e0 <entry_SYSCALL_64_after_hwframe+104>: int3
0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>: call 0xffffffff97314b80 <do_syscall_64>
在0xffffffff974000d1处nop指令变成了jmp：
0xffffffff974000d1 <entry_SYSCALL_64_after_hwframe+89>: jmp 0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>
......
0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>: call 0xffffffff97314b80 <do_syscall_64>
与UNTRAIN_RET指令有关。
（2）
/** Save all registers in pt_regs. Return GSBASE related information* in EBX depending on the availability of the FSGSBASE instructions:** FSGSBASE R/EBX* N 0 -> SWAPGS on exit* 1 -> no SWAPGS on exit** Y GSBASE value at entry, must be restored in paranoid_exit** R14 - old CR3* R15 - old SPEC_CTRL*/
SYM_CODE_START(paranoid_entry)ANNOTATE_NOENDBRUNWIND_HINT_FUNCPUSH_AND_CLEAR_REGS save_ret=1ENCODE_FRAME_POINTER 8/** Always stash CR3 in %r14. This value will be restored,* verbatim, at exit. Needed if paranoid_entry interrupted* another entry that already switched to the user CR3 value* but has not yet returned to userspace.** This is also why CS (stashed in the "iret frame" by the* hardware at entry) can not be used: this may be a return* to kernel code, but with a user CR3 value.** Switching CR3 does not depend on kernel GSBASE so it can* be done before switching to the kernel GSBASE. This is* required for FSGSBASE because the kernel GSBASE has to* be retrieved from a kernel internal table.*/SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14/** Handling GSBASE depends on the availability of FSGSBASE.** Without FSGSBASE the kernel enforces that negative GSBASE* values indicate kernel GSBASE. With FSGSBASE no assumptions* can be made about the GSBASE value when entering from user* space.*/ALTERNATIVE "jmp .Lparanoid_entry_checkgs", "", X86_FEATURE_FSGSBASE/** Read the current GSBASE and store it in %rbx unconditionally,* retrieve and set the current CPUs kernel GSBASE. The stored value* has to be restored in paranoid_exit unconditionally.** The unconditional write to GS base below ensures that no subsequent* loads based on a mispredicted GS base can happen, therefore no LFENCE* is needed here.*/SAVE_AND_SET_GSBASE scratch_reg=%rax save_reg=%rbxjmp .Lparanoid_gsbase_done.Lparanoid_entry_checkgs:/* EBX = 1 -> kernel GSBASE active, no restore required */movl $1, %ebx/** The kernel-enforced convention is a negative GSBASE indicates* a kernel value. No SWAPGS needed on entry and exit.*/movl $MSR_GS_BASE, %ecxrdmsrtestl %edx, %edxjs .Lparanoid_kernel_gsbase/* EBX = 0 -> SWAPGS required on exit */xorl %ebx, %ebxswapgs
.Lparanoid_kernel_gsbase:FENCE_SWAPGS_KERNEL_ENTRY
.Lparanoid_gsbase_done:/** Once we have CR3 and %GS setup save and set SPEC_CTRL. Just like* CR3 above, keep the old value in a callee saved register.*/IBRS_ENTER save_reg=%r15UNTRAIN_RET_FROM_CALLRET
SYM_CODE_END(paranoid_entry)
查看vmlinux：
# cat /boot/System.map-6.5.0-18-generic | grep '\<paranoid_entry\>'
ffffffff822013b0 T paranoid_entry
ffffffff822013b0: 56 push %rsi
ffffffff822013b1: 48 8b 74 24 08 mov 0x8(%rsp),%rsi
ffffffff822013b6: 48 89 7c 24 08 mov %rdi,0x8(%rsp)
ffffffff822013bb: 52 push %rdx
ffffffff822013bc: 51 push %rcx
ffffffff822013bd: 50 push %rax
ffffffff822013be: 41 50 push %r8
ffffffff822013c0: 41 51 push %r9
ffffffff822013c2: 41 52 push %r10
ffffffff822013c4: 41 53 push %r11
ffffffff822013c6: 53 push %rbx
ffffffff822013c7: 55 push %rbp
ffffffff822013c8: 41 54 push %r12
ffffffff822013ca: 41 55 push %r13
ffffffff822013cc: 41 56 push %r14
ffffffff822013ce: 41 57 push %r15
ffffffff822013d0: 56 push %rsi
ffffffff822013d1: 31 f6 xor %esi,%esi
ffffffff822013d3: 31 d2 xor %edx,%edx
......
ffffffff82201484: 90 nop
ffffffff82201485: 90 nop
ffffffff82201486: 90 nop
ffffffff82201487: 90 nop
ffffffff82201488: 90 nop
ffffffff82201489: 90 nop
ffffffff8220148a: 90 nop
ffffffff8220148b: 90 nop
ffffffff8220148c: 90 nop
ffffffff8220148d: 90 nop
ffffffff8220148e: 90 nop
ffffffff8220148f: 90 nop
ffffffff82201490: 90 nop
ffffffff82201491: 90 nop
ffffffff82201492: 90 nop
到了ffffffff82201484处都是nop指令。
crash查看：
crash> dis paranoid_entry
0xffffffff974013b0 <paranoid_entry>: push %rsi
0xffffffff974013b1 <paranoid_entry+1>: mov 0x8(%rsp),%rsi
0xffffffff974013b6 <paranoid_entry+6>: mov %rdi,0x8(%rsp)
0xffffffff974013bb <paranoid_entry+11>: push %rdx
0xffffffff974013bc <paranoid_entry+12>: push %rcx
0xffffffff974013bd <paranoid_entry+13>: push %rax
0xffffffff974013be <paranoid_entry+14>: push %r8
0xffffffff974013c0 <paranoid_entry+16>: push %r9
0xffffffff974013c2 <paranoid_entry+18>: push %r10
0xffffffff974013c4 <paranoid_entry+20>: push %r11
0xffffffff974013c6 <paranoid_entry+22>: push %rbx
0xffffffff974013c7 <paranoid_entry+23>: push %rbp
0xffffffff974013c8 <paranoid_entry+24>: push %r12
0xffffffff974013ca <paranoid_entry+26>: push %r13
0xffffffff974013cc <paranoid_entry+28>: push %r14
0xffffffff974013ce <paranoid_entry+30>: push %r15
0xffffffff974013d0 <paranoid_entry+32>: push %rsi
0xffffffff974013d1 <paranoid_entry+33>: xor %esi,%esi
0xffffffff974013d3 <paranoid_entry+35>: xor %edx,%edx
......
0xffffffff97401484 <paranoid_entry+212>: jmp 0xffffffff97401493 <paranoid_entry+227>
0xffffffff97401486 <paranoid_entry+214>: int3
0xffffffff97401487 <paranoid_entry+215>: int3
0xffffffff97401488 <paranoid_entry+216>: int3
0xffffffff97401489 <paranoid_entry+217>: int3
0xffffffff9740148a <paranoid_entry+218>: int3
0xffffffff9740148b <paranoid_entry+219>: int3
0xffffffff9740148c <paranoid_entry+220>: int3
0xffffffff9740148d <paranoid_entry+221>: int3
0xffffffff9740148e <paranoid_entry+222>: int3
0xffffffff9740148f <paranoid_entry+223>: int3
0xffffffff97401490 <paranoid_entry+224>: int3
0xffffffff97401491 <paranoid_entry+225>: int3
0xffffffff97401492 <paranoid_entry+226>: int3
0xffffffff97401493 <paranoid_entry+227>: ret
0xffffffff97401494 <paranoid_entry+228>: int3
0xffffffff97401495 <paranoid_entry+229>: int3
0xffffffff97401496 <paranoid_entry+230>: int3
0xffffffff97401497 <paranoid_entry+231>: int3
到了0xffffffff97401484：
0xffffffff97401484 <paranoid_entry+212>: jmp 0xffffffff97401493 <paranoid_entry+227>
0xffffffff97401493 <paranoid_entry+227>: ret
也是nop指令变成了jmp指令。
这与UNTRAIN_RET_FROM_CALL指令有关。
参考资料
https://ebpf.top/post/ubuntu_kdump_crash/
