VirtualBox + kgdb analysis of Linux kernel (v3.4.0-rc3)

https://tthtlc.wordpress.com/2012/06/16/virtualbox-kgdb-analysis-of-linux-kernel-v3-4-0-rc3/

Posted June 16, 2012 by Peter Teoh in kernel_general.

After compiling the kernel inside VirtualBox, copy out the vmlinux to the host, where “gdb” command is to be issued.

Add “kgdboc=ttyS0,115200 kgdbwait” at the end of the kernel line inside the /boot/grub/grub.cfg. Eg:

linux /boot/xxxxx kgdboc=ttyS0,115200 kgdbwait

Then issue at host level:

socat -d -d /tmp/serial1204 pty

after that “/dev/pts/6” is printed:

2012/06/17 08:04:20 socat[6051.18446744073196852992] N opening connection to AF=1 "/tmp/serial1204"
2012/06/17 08:04:20 socat[6051.18446744073196852992] N successfully connected from local address AF=1 "\x02\xE0"
2012/06/17 08:04:20 socat[6051.18446744073196852992] N successfully connected via s\xD1]\x7E
2012/06/17 08:04:20 socat[6051.18446744073196852992] N PTY is /dev/pts/6
2012/06/17 08:04:20 socat[6051.18446744073196852992] N starting data transfer loop with FDs [3,3] and [4,4]


Use that information below (at the host level):

gdb ./vmlinux (where vmlinux is the compiled vmlinux image from inside the VirtualBox).

(gdb) set remotebaud 115200
(gdb) target remote /dev/pts/6
Remote debugging using /dev/pts/6
kgdb_breakpoint () at kernel/debug/debug_core.c:987
987 kernel/debug/debug_core.c: No such file or directory.
in kernel/debug/debug_core.c


From above, “No such file” found is because we are not located at the root of the kernel source code. Restarting “gdb ./vmlinux” at the root of the kernel source code, and using the “list” command in gdb, we can list the source code.

(gdb) list
982 void kgdb_breakpoint(void)
983 {
984 atomic_inc(&kgdb_setting_breakpoint);
985 wmb(); /* Sync point before breakpoint */
986 arch_kgdb_breakpoint();
987 wmb(); /* Sync point after breakpoint */
988 atomic_dec(&kgdb_setting_breakpoint);
989 }
990 EXPORT_SYMBOL_GPL(kgdb_breakpoint);
991


From above we can see that the very first breakpoint when kgdb stopped is at 986. So all codes that execute before this point cannot be traced by kgdb. As for how this kgdb_breakpoint() is reached, can be seen from the stacktrace:

(gdb) bt
#0 kgdb_breakpoint () at kernel/debug/debug_core.c:987
#1 0xffffffff810cf486 in kgdb_initial_breakpoint (
new_dbg_io_ops=0xffffffff81c7f9e0) at kernel/debug/debug_core.c:885
#2 kgdb_register_io_module (new_dbg_io_ops=0xffffffff81c7f9e0)
at kernel/debug/debug_core.c:927
#3 0xffffffff813e4d94 in configure_kgdboc ()
at drivers/tty/serial/kgdboc.c:197
#4 0xffffffff81d2709e in init_kgdboc () at drivers/tty/serial/kgdboc.c:219
#5 0xffffffff8100203f in do_one_initcall (fn=0xffffffff81d2708a )
at init/main.c:678
#6 0xffffffff81cf2d53 in do_initcall_level (unused=)
at init/main.c:753
#7 do_initcalls (unused=) at init/main.c:761
#8 do_basic_setup (unused=) at init/main.c:780
#9 kernel_init (unused=) at init/main.c:863
#10 0xffffffff816608e4 in ?? () at arch/x86/kernel/entry_64.S:1204
#11 0x0000000000000000 in ?? ()


Essentially the caller is kernel_init()—> do_basic_setup()—> do_initcalls()—>do_initcall_level()—>do_one_initcall()—>init_kgdboc()—>configure_kgdboc()—>kgdb_register_io_module()—>kgdb_initial_breakpoint()—>kgdb_breakpoint().

By setting breakpoints at “printk”, we can see all the stacktrace when the kernel is printing its output to the dmesg:

(gdb) break printk
Breakpoint 1 at 0xffffffff8164c7c6: file kernel/printk.c, line 753.
(gdb) cont
Continuing.
[New Thread 19]
[Switching to Thread 19]

From above, we can see there is a switching of execution from one thread to another.


Breakpoint 1, printk (
fmt=0xffffffff819ed780 "<6>Refined TSC clocksource calibration: %lu.%03lu MHz.\n") at kernel/printk.c:753
753 }
(gdb) bt
#0 printk (
fmt=0xffffffff819ed780 "<6>Refined TSC clocksource calibration: %lu.%03lu MHz.\n") at kernel/printk.c:753
#1 0xffffffff8101a4a1 in tsc_refine_calibration_work (
work=<value optimized out>) at arch/x86/kernel/tsc.c:915
#2 0xffffffff8106d26a in process_one_work (worker=<value optimized out>,
work=0xffffffff81c18e60) at kernel/workqueue.c:1866
#3 0xffffffff8106e2d5 in worker_thread (__worker=0xffff88003dbb4b80)
at kernel/workqueue.c:1977
#4 0xffffffff81072d63 in kthread (_create=0xffff88003da57d28)
at kernel/kthread.c:121
#5 0xffffffff816608e4 in ?? () at arch/x86/kernel/entry_64.S:1204
#6 0x0000000000000000 in ?? ()
(gdb) cont
Continuing.
[Switching to Thread 1]

Breakpoint 1, printk (fmt=0xffffffff81a37d88 "<6>brd: module loaded\n")
at kernel/printk.c:753
753 }
(gdb) bt
#0 printk (fmt=0xffffffff81a37d88 "<6>brd: module loaded\n")
at kernel/printk.c:753
#1 0xffffffff81d2818b in brd_init () at drivers/block/brd.c:624
#2 0xffffffff8100203f in do_one_initcall (fn=0xffffffff81d2803c <brd_init>)
at init/main.c:678
#3 0xffffffff81cf2d53 in do_initcall_level (unused=<value optimized out>)
at init/main.c:753
#4 do_initcalls (unused=<value optimized out>) at init/main.c:761
#5 do_basic_setup (unused=<value optimized out>) at init/main.c:780
#6 kernel_init (unused=<value optimized out>) at init/main.c:863
#7 0xffffffff816608e4 in ?? () at arch/x86/kernel/entry_64.S:1204
#8 0x0000000000000000 in ?? ()
(gdb) cont
Continuing.
[Switching to Thread 19]

Now boot up all the way by deleting all the breakpoints:

(gdb) delete
Delete all breakpoints? (y or n) y
(gdb) cont
Continuing.