[C++][linux debugging] GDB wrap up - 2018

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? -- Brian Kernighan

GDB

manual: https://sourceware.org/gdb/current/onlinedocs/gdb/
Build atop ptrace.
When program being traced receives a signal, it is suspended and the tracer gets notified(through waitpid).

When the inferior receives a signal, it stops and gdb gets control.

Usually gdb returns to the prompt, but what it will do depends on the signal and how it is configured.

Two not so special signals:

1. SIGINT is generated when hitting Ctrl-C
2. 2. SIGTRAP is generated when the inferior hits a breakpoint or is single stepped.

https://sourceware.org/gdb/current/onlinedocs/gdb/TUI-Keys.html

-g and -O are orthogonal. 

-Og is optimised but doesn't mess up debug.
-ggdb3 is better than -g
$ clang++ -ggdb3 test.cpp
$ gdb a.out

TUI command:

$ list // list source code.
$ Ctrl-x-a // TUI Interface.
$ Ctrl-x-o // Move to source view.
$ Ctrl-x-s
$ Ctrl-x-2 // cycle through different UI layout.
$ Ctrl-x-1 // cycle back to single window layout.
$ Ctrl-l // refresh screen.
$ Ctrl-p/n //previous/next command.
$ shell ps

traverse:

$ next
$ tbreak // tmp breakpoint.
$ rbreak // regex breakpoint.
$ command // list commands when breakpoint hits.
$ silent // suppress output.
$ save breakpoints // save bp to script.
$ save history // as it is.
$ info line foo.c:42 // show PC for line.
$ info line * $pc // show line begin/end for current Program Counter.
$ frame // Select and print a stack frame.

signals:

$ info signals
set signal disposition:
$ handle SIGINT stop print pass

Breakpoints and watch points:

$ watch foo // stop when foo is modified
$ watch -l foo // watch location
$ rwatch foo // stop when foo is read
$ watch foo thread 42 // stop when thread 42 modifies foo
$ watch foo if foo > 42 // stop when foo is > 42

Thread apply:

$ thread apply 1-4 print $sp // $sp as stack pointer.
$ thread apply all backtrace
$ thread apply all backtrace full
$ info threads // show the current running program threads info

Dynamic Printf:

Using dprintf to put printf in the code without recompiling,
e.g
$ dprintf mutex_lock, "%p, %u\n", m, m-val
Control how the printf happen:
$ set dprintf-style gdb|call|agent
$ set dprintf-function fprintf
$ set dprintf-channel mylog

Calling inferior function:

$ call foo()
$ print foo()
$ print bar + foo // C++ operator overload detected by gdb.
$ print errno

Below calls malloc() as well.
$ call strcpy(buffer, "Hi!\n")

Catchpoints:

Like breakpoints but catch certain events, e.g C++ exceptions
$ catch catch // stop when C++ exceptions caught.
$ catch syscall nanosleep // stop at nanosleep system call.
$ catch syscall 100 // stop at system call number 100.

Remote debugging:

$ gdbserver localhost:2000 ./a.out // server
$ target remote localhost:2000 // client

Multiprocess Debugging:

$ set follow-fork-mode child|parent
$ set detach-on-fork off
$ info inferiors
$ inferior N
$ set follow-exec-mode new|same
$ add-inferior <cnt> <name>
$ remove-inferior N
$ clone-inferior
$ print $_inferior

GDB with embedded python interface:

https://stackoverflow.com/questions/12574253/c-gdb-python-pretty-printing-tutorial
$ python gdb.execute()
$ python gdb.parse_and_eval()
$ python help('gdb')

Valgrind:

$ valgrind ./a.out
With build-in GDB:
$ valgrind --vgdb=full --vgdb-error=0 ./a.out

Tools:
Cachegrind: cache profiler, simulates L1, D1, L2 caches
Callgrind: like Cachegrind but with call-graphs
Massif: heap profiler
Helgrind: Spot race conditions in multi-thread program.
DRD: Data Race Detector. Like Helgrind, use less memory.
Lackey : Simple Valgrind tool that does various kinds of basic program measurement.

Sanitizers:

Build into the compiler.
AddressSanitizer, MemorySanitizer, ThreadSanitizer, LeakSanitizer
$ clang++ -g -fsanitize=address test.cpp

rr: 

has limitation, only on baremetal Intel x86 CPU.

ftrace:

Function Tracer, trace various kernel functions.
Lots of pre-defined events
Controlled through /sys/kernel/debug/tracing

strace:

based on ptrace
$ strace -k cmd   // show backtrace for each syscall.

ltrace:

trace dynamic library calls of a process.

perf trace:

build on perf. faster than strace.

fortify:

memcpy, strcpy, etc. do bounds checking where it can.
Slow.
$ clang++ -D_FORTIFY_SOURCE=1