A script snippet for aggregating GDB backtraces
Note: the bt-aggregate tool has been deprecated and replaced by the pmp tool, which can do all that and more.
A short time ago in a galaxy nearby, Domas Mituzas wrote about contention profiling with GDB stack traces. Mark Callaghan found the technique useful, and contributed an awk script (in the comments) to aggregate stack traces and identify which things are blocking most threads. I’ve used it myself a time or five. But I’ve found myself wanting it to be fancier, for various reasons. So I wrote a little utility that can aggregate and pretty-print backtraces. It can handle unresolved symbols, and aggregate by only the first N lines of the stack trace. Here’s an example of a mysqld instance that’s really, really frozen up:
bt-aggregate -4 samples/backtrace.txt | head -n12
2396 threads with the following stack trace:
#0 0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005f2bd8 in open_table ()
#2 0x00000000005f3fb4 in open_tables ()
#3 0x00000000005f4247 in open_and_lock_tables_derived ()
4 threads with the following stack trace:
#0 0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000780099 in os_event_wait_low ()
#2 0x000000000077de42 in os_aio_simulated_handle ()
#3 0x000000000074a261 in fil_aio_wait ()
Further Reading:
Tnx, it’s useful, but on a production server i see only
“1 threads with the following stack trace” records.
like this:
ssh db2 gdb -batch -ex “‘thr app all bt full’” \
-p ‘`pgrep mysqld|tail -n 1`’|bt-aggregate
1 threads with the following stack trace:
…..
array = (os_aio_array_t *) 0x4769a30
…..
1 threads with the following stack trace:
…..
array = (os_aio_array_t *) 0x4770d80
as you can see if the symbol table exists the addresses of a local variables may be different in two equal threads.
PS: i use mysql binary builds from the dev.mysqld.com site.
Nickolay Ihalainen
30 Aug 09 at 5:42 pm
I haven’t seen a stack with array() in it, can you contribute a sample? You can either email it to me or post it to an issue report on http://code.google.com/p/maatkit/
Xaprb
30 Aug 09 at 7:15 pm
Is this bt-aggregate or mk-aggregate. For pileups, using a prefix of the trace as you do is required to make sense of the output. I had to do that when there were 2000 threads while debugging http://bugs.mysql.com/bug.php?id=46459
Mark Callaghan
30 Aug 09 at 10:47 pm
This is bt-aggregate (bt = backtrace). Naming suggestions are welcome.
These little snippets of code that aren’t really mysql-specific have been finding their way into a util/ directory in the Maatkit SVN repository, and not treated as first-class citizens. Someday I’ll figure out what to do with them, and probably make easy wget-able URLs like maatkit.org/util/foo.
Xaprb
30 Aug 09 at 11:09 pm
I have one of these I wrote for Spinn3r back in the day…. it aggregates and reports stack traces from Java stacks.
I honestly can’t live without it……
Kevin Burton
30 Aug 09 at 11:52 pm
Yeah, Java stack traces are another whole ball of wax. Java, the land of 1000-deep stack traces ;-)
Xaprb
31 Aug 09 at 7:33 am
Hi Baron,
I am lazy, but I know you are not!
It would be great if you could include the sample taking in your program, so that I just have to specify the proc ID… :)
Paul McCullagh
1 Sep 09 at 3:27 am
Paul,
gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $(pidof mysqld)Xaprb
1 Sep 09 at 9:43 am
Hey your back trace script is not on the URL listed above. Could you reupload it?
Dathan Pattishall
7 May 10 at 7:07 pm
It’s at Aspersa now. I fixed the link.
Xaprb
8 May 10 at 6:56 am
I’ve actually deprecated the bt-aggregate tool and replaced it with one called pmp, which is better. I re-fixed the link above.
Xaprb
22 Sep 10 at 10:16 pm