1 .\" Copyright (c) 2003 Greg Lehey
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
32 .Nd external kernel debugger
34 .Cd "makeoptions DEBUG=-g"
39 kernel debugger is a variation of
40 .Xr gdb 1 Pq Pa ports/devel/gdb
41 which understands some aspects of the
44 It can be used in a number of ways:
47 It can be used to examine the memory of the processor on which it runs.
49 It can be used to analyse a processor dump after a panic.
51 It can be used to debug another system interactively via a serial or firewire
53 In this mode, the processor can be stopped and single stepped.
55 With a firewire link, it can be used to examine the memory of a remote system
56 without the participation of that system.
57 In this mode, the processor cannot be stopped and single stepped, but it can be
58 of use when the remote system has crashed and is no longer responding.
61 When used for remote debugging,
63 requires the presence of the
66 Commands exist to switch between
70 .Sh PREPARING FOR DEBUGGING
71 When debugging kernels, it is practically essential to have built a kernel with
73 .Pq Cd "makeoptions DEBUG=-g" .
74 It is easiest to perform operations from the kernel build directory, by default
75 .Pa /usr/obj/usr/src/sys/GENERIC .
77 First, ensure you have a copy of the debug macros in the directory:
81 This command performs some transformations on the macros installed in
82 .Pa /usr/src/tools/debugscripts
83 to adapt them to the local environment.
84 .Ss "Inspecting the environment of the local machine"
85 To look at and change the contents of the memory of the system you are running
88 .Dl "gdb -k -wcore kernel.debug /dev/mem"
90 In this mode, you need the
93 .Xr gdb 1 Pq Pa ports/devel/gdb
97 is a kernel data file.
98 You can look at live data, and if you include the
100 option, you can change it at your peril.
101 The system does not stop (obviously), so a number of things will not work.
102 You can set breakpoints, but you cannot
104 execution, so they will not work.
105 .Ss "Debugging a crash dump"
106 By default, crash dumps are stored in the directory
108 Investigate them from the kernel build directory with:
110 .Dl "gdb -k kernel.debug /var/crash/vmcore.29"
112 In this mode, the system is obviously stopped, so you can only look at it.
113 .Ss "Debugging a live system with a remote link"
114 In the following discussion, the term
116 refers to the system running the debugger, and
118 refers to the live system being debugged.
120 To debug a live system with a remote link, the kernel must be compiled with the
124 .Cd "options BREAK_TO_DEBUGGER"
125 enables the debugging machine stop the debugged machine once a connection has
126 been established by pressing
128 .Ss "Debugging a live system with a remote serial link"
129 When using a serial port for the remote link on the i386 platform, the serial
130 port must be identified by setting the flag bit
132 for the specified interface.
133 Generally, this port will also be used as a serial console (flag bit
136 .Pa /boot/device.hints
139 .Dl hint.sio.0.flags="0x90"
140 .Ss "Debugging a live system with a remote firewire link"
141 As with serial debugging, to debug a live system with a firewire link, the
142 kernel must be compiled with the option
145 A number of steps must be performed to set up a firewire link:
148 Ensure that both systems have
150 support, and that the kernel of the remote system includes the
155 If they are not compiled into the kernel, load the KLDs:
157 .Dl "kldload firewire"
159 On the remote system only:
160 .Bd -literal -offset indent
165 You should see something like this in the
167 output of the remote system:
168 .Bd -literal -offset indent
170 fwohci0: node_id=0x8800ffc0, gen=2, non CYCLEMASTER mode
171 firewire0: 2 nodes, maxhop <= 1, cable IRM = 1
172 firewire0: bus manager 1
173 firewire0: New S400 device ID:00c04f3226e88061
174 dcons_crom0: <dcons configuration ROM> on firewire0
175 dcons_crom0: bus_addr 0x22a000
178 It is a good idea to load these modules at boot time with the following entry in
179 .Pa /boot/loader.conf :
181 .Dl dcons_crom_enable="YES"
183 This ensures that all three modules are loaded.
184 There is no harm in loading
188 on the local system, but if you only want to load the
190 module, include the following in
191 .Pa /boot/loader.conf :
193 .Dl firewire_enable="YES"
197 to find the firewire node corresponding to the remote machine.
198 On the local machine you might see:
199 .Bd -literal -offset indent
201 2 devices (info_len=2)
203 1 0x00c04f3226e88061 0
204 0 0x000199000003622b 1
207 The first node is always the local system, so in this case, node 0 is the remote
209 If there are more than two systems, check from the other end to find which node
210 corresponds to the remote system.
211 On the remote machine, it looks like this:
212 .Bd -literal -offset indent
214 2 devices (info_len=2)
216 0 0x000199000003622b 0
217 1 0x00c04f3226e88061 1
220 Next, establish a firewire connection with
223 .Dl "dconschat -br -G 5556 -t 0x000199000003622b"
225 .Li 0x000199000003622b
226 is the EUI64 address of the remote node, as determined from the output of
229 When started in this manner,
231 establishes a local tunnel connection from port
233 to the remote debugger.
234 You can also establish a console port connection with the
236 option to the same invocation
240 manpage for further details.
245 does not return control to the user.
246 It displays error messages and console output for the remote system, so it is a
247 good idea to start it in its own window.
249 Finally, establish connection:
250 .Bd -literal -offset indent
252 GNU gdb 5.2.1 (FreeBSD)
253 .Em "(political statements omitted)"
254 Ready to go. Enter 'tr' to connect to the remote target
255 with /dev/cuau0, 'tr /dev/cuau1' to connect to a different port
256 or 'trf portno' to connect to the remote target with the firewire
257 interface. portno defaults to 5556.
259 Type 'getsyms' after connection to load kld symbols.
261 If you are debugging a local system, you can use 'kldsyms' instead
262 to load the kld symbols. That is a less obnoxious interface.
269 macro assumes a connection on port 5556.
270 If you want to use a different port (by changing the invocation of
275 For example, if you want to use port 4711, run
279 .Dl "dconschat -br -G 4711 -t 0x000199000003622b"
281 Then establish connection with:
282 .Bd -literal -offset indent
283 (gdb) tr localhost:4711
287 .Ss "Non-cooperative debugging a live system with a remote firewire link"
288 In addition to the conventional debugging via firewire described in the previous
289 section, it is possible to debug a remote system without its cooperation, once
290 an initial connection has been established.
291 This corresponds to debugging a local machine using
293 It can be very useful if a system crashes and the debugger no longer responds.
294 To use this method, set the
297 .Va hw.firewire.fwmem.eui64_hi
299 .Va hw.firewire.fwmem.eui64_lo
300 to the upper and lower halves of the EUI64 ID of the remote system,
302 From the previous example, the remote machine shows:
303 .Bd -literal -offset indent
305 2 devices (info_len=2)
307 0 0x000199000003622b 0
308 1 0x00c04f3226e88061 1
312 .Bd -literal -offset indent
313 # sysctl -w hw.firewire.fwmem.eui64_hi=0x00019900
314 hw.firewire.fwmem.eui64_hi: 0 -> 104704
315 # sysctl -w hw.firewire.fwmem.eui64_lo=0x0003622b
316 hw.firewire.fwmem.eui64_lo: 0 -> 221739
319 Note that the variables must be explicitly stated in hexadecimal.
320 After this, you can examine the remote machine's state with the following input:
321 .Bd -literal -offset indent
322 # gdb -k kernel.debug /dev/fwmem0.0
323 GNU gdb 5.2.1 (FreeBSD)
324 .Em "(messages omitted)"
325 Reading symbols from /boot/kernel/dcons.ko...done.
326 Loaded symbols for /boot/kernel/dcons.ko
327 Reading symbols from /boot/kernel/dcons_crom.ko...done.
328 Loaded symbols for /boot/kernel/dcons_crom.ko
329 #0 sched_switch (td=0xc0922fe0) at /usr/src/sys/kern/sched_4bsd.c:621
333 In this case, it is not necessary to load the symbols explicitly.
334 The remote system continues to run.
336 The user interface to
339 .Xr gdb 1 Pq Pa ports/devel/gdb ,
341 .Xr gdb 1 Pq Pa ports/devel/gdb
343 This section discusses only the extensions for kernel debugging that get
344 installed in the kernel build directory.
345 .Ss "Debugging environment"
346 The following macros manipulate the debugging environment:
347 .Bl -tag -width indent
351 This command is only meaningful when performing remote debugging.
355 information for the target machine and invite user to paste it back in.
356 This is required because
358 does not allow data to be passed to shell scripts.
359 It is necessary for remote debugging and crash dumps; for local memory debugging
364 Read in the symbol tables for the debugging machine.
365 This does not work for
366 remote debugging and crash dumps; use
369 .It Ic tr Ar interface
370 Debug a remote system via the specified serial or firewire interface.
372 Debug a remote system via serial interface
375 Debug a remote system via serial interface
378 Debug a remote system via firewire interface at default port 5556.
385 are convenience commands which invoke
387 .Ss "The current process environment"
388 The following macros are convenience functions intended to make things easier
390 .Xr gdb 1 Pq Pa ports/devel/gdb
392 .Bl -tag -width indent
394 Select stack frame 0 and show assembler-level details.
396 Select stack frame 1 and show assembler-level details.
398 Select stack frame 2 and show assembler-level details.
400 Select stack frame 3 and show assembler-level details.
402 Select stack frame 4 and show assembler-level details.
404 Select stack frame 5 and show assembler-level details.
406 Show 12 words in hex, starting at current
410 List the next 10 instructions from the current
414 Show the register contents and the first four parameters of the current stack
417 Show the first parameter of current stack frame in various formats.
419 Show the second parameter of current stack frame in various formats.
421 Show the third parameter of current stack frame in various formats.
423 Show the fourth parameter of current stack frame in various formats.
425 Show the fifth parameter of current stack frame in various formats.
427 Show the last 12 words on stack in hexadecimal.
429 Show the register contents and the first ten parameters.
431 Single step 1 instruction (over calls) and show next instruction.
433 Single step 1 instruction (through calls) and show next instruction.
435 .Ss "Examining other processes"
436 The following macros access other processes.
440 does not understand the concept of multiple processes, so they effectively
444 .Bl -tag -width indent
446 Show a backtrace for the process
449 Show backtraces for all processes in the system.
451 Show a backtrace for the process previously selected with
454 Show a backtrace from the
457 .It Ic defproc Ar pid
458 Specify the PID of the process for some other commands in this section.
462 of the stack of the process previously selected with
465 Show some PCB contents of the process
468 .Ss "Examining data structures"
470 .Xr gdb 1 Pq Pa ports/devel/gdb
471 commands to look at most data structures.
472 The macros in this section are
473 convenience functions which typically display the data in a more readable
474 format, or which omit less interesting parts of the structure.
475 .Bl -tag -width indent
477 Show information about the buffer header pointed to by the variable
479 in the current frame.
485 in the current frame.
487 Show detailed information about the buffer header
489 pointed at by the local variable
492 Show summary information about the buffer header
494 pointed at by the parameter
497 Print a number of fields from the buffer header pointed at in by the pointer
499 in the current environment.
501 Show some information of the
503 pointed to by the local variable
506 .Ss "Miscellaneous macros"
507 .Bl -tag -width indent
509 Check unallocated memory for modifications.
510 This assumes that the kernel has been compiled with
511 .Cd "options DIAGNOSTIC" .
512 This causes the contents of free memory to be set to
515 Print the system message buffer.
516 This corresponds to the
519 This macro used to be called
521 It can take a very long time over a serial line,
522 and it is even slower via firewire
523 or local memory due to inefficiencies in
525 When debugging a crash dump or over firewire, it is not necessary to start
527 to access the message buffer: instead, use an appropriate variation of
528 .Bd -literal -offset indent
529 dmesg -M /var/crash/vmcore.0 -N kernel.debug
530 dmesg -M /dev/fwmem0.0 -N kernel.debug
535 utility without options.
537 Print the command name of the current process.
540 This corresponds in concept, but not in appearance, to the
543 When debugging a crash dump or over firewire, it is not necessary to start
547 output: instead, use an appropriate variation of
548 .Bd -literal -offset indent
549 ps -M /var/crash/vmcore.0 -N kernel.debug
550 ps -M /dev/fwmem0.0 -N kernel.debug
553 Kludge for writing macros.
554 When writing macros, it is convenient to paste them
558 Unfortunately, if the macro is already defined,
564 It will not give up until you answer
566 This command is that answer.
567 It does nothing else except to print a warning
568 message to remind you to remove it again.
571 .Xr gdb 1 Pq Pa ports/devel/gdb ,
580 This man page was written by
581 .An Greg Lehey Aq Mt grog@FreeBSD.org .
584 .Xr gdb 1 Pq Pa ports/devel/gdb
586 was never designed to debug kernels, and it is not a very good match.
591 implementation is very inefficient, and many operations are slow.
593 Serial debugging is even slower, and race conditions can make it difficult to
594 run the link at more than 9600 bps.
595 Firewire connections do not have this problem.
599 In general, the person who wrote them did so while looking for a specific
600 problem, so they may not be general enough, and they may behave badly when used
601 in ways for which they were not intended, even if those ways make sense.
603 Many of these commands only work on the ia32 architecture.