Xaprb » Daniel Nichter http://www.xaprb.com/blog Stay curious! Thu, 09 Feb 2012 10:55:47 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 mk-query-digest uses less memory http://www.xaprb.com/blog/2010/05/19/mk-query-digest-uses-less-memory/ http://www.xaprb.com/blog/2010/05/19/mk-query-digest-uses-less-memory/#comments Thu, 20 May 2010 02:29:56 +0000 Xaprb http://www.xaprb.com/blog/?p=1860 Further Reading:]]> Daniel changed mk-query-digest to use much less memory. It parsed and aggregated a 2GB MySQL slow query log file in a few dozen megabytes of memory for me yesterday. Thanks to Facebook for sponsoring this work.

Further Reading:

]]> http://www.xaprb.com/blog/2010/05/19/mk-query-digest-uses-less-memory/feed/ 3 A script snippet to relative-ize numbers embedded in text http://www.xaprb.com/blog/2009/09/01/a-script-snippet-to-relative-ize-numbers-embedded-in-text/ http://www.xaprb.com/blog/2009/09/01/a-script-snippet-to-relative-ize-numbers-embedded-in-text/#comments Wed, 02 Sep 2009 00:29:46 +0000 Xaprb http://www.xaprb.com/blog/?p=1269 Further Reading:]]> A lot of times I’m looking at several time-series samples of numbers embedded in free-form text, and I want to know how the numbers change over time. For example, two samples of SHOW INNODB STATUS piped through grep wait might contain the following:

Mutex spin waits 0, rounds 143359179688, OS waits 634106844
RW-shared spins 1224152309, OS waits 38278807; RW-excl spins 2432166425, OS waits 35264871
Mutex spin waits 0, rounds 143386303439, OS waits 634292093
RW-shared spins 1224197048, OS waits 38281423; RW-excl spins 2432347936, OS waits 35271423

How much have the numbers changed in the second sample? My head is too lazy to do that math. So Daniel Nichter and I whipped up Yet Another Snippet to self-discover patterns of text and numbers, and compare each line against the previous line that matches the same pattern. Let’s fetch it:

wget http://maatkit.googlecode.com/svn/trunk/util/rel

Now give it the above input, and it’ll print out something useful (emphasis mine):

Mutex spin waits 0, rounds 143359179688, OS waits 634106844
RW-shared spins 1224152309, OS waits 38278807; RW-excl spins 2432166425, OS waits 35264871
Mutex spin waits 0, rounds 27123751, OS waits 185249
RW-shared spins 44739, OS waits 2616; RW-excl spins 181511, OS waits 6552

My lazy brain likes that much better.

Further Reading:

]]> http://www.xaprb.com/blog/2009/09/01/a-script-snippet-to-relative-ize-numbers-embedded-in-text/feed/ 7 Maatkit version 2325 released http://www.xaprb.com/blog/2008/09/19/maatkit-version-2325-released/ http://www.xaprb.com/blog/2008/09/19/maatkit-version-2325-released/#comments Fri, 19 Sep 2008 15:37:43 +0000 Xaprb http://www.xaprb.com/blog/2008/09/19/maatkit-version-2325-released/ Further Reading:]]> Download Maatkit

There’s a new release with a lot of goodies — speed, efficiency, user-friendliness, and new features. In particular some of Percona’s clients have sponsored features for things they need such as the ability to more frequently verify that slaves are in sync with their masters. If you need features, please ask Percona about it.

Daniel Nichter has written most of the code for this release. Having a dedicated developer is really moving the project forward. Thanks Daniel!

If you check the issues list on the project’s website you’ll see we have some ambitious plans for the project, mostly driven by real-world needs from Percona’s clients, many of whom run very large data centers and such. It makes me really happy that Maatkit is solving serious problems for which no other solution exists.

Here’s the changelog:

Changelog for mk-archiver:

2008-09-19: version 1.0.11

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-audit:

2008-09-19: version 0.9.2

   * Checks /etc/sysctl.conf for default ipv4 system variables (issue 56).
   * No longer reports false-positive duplicate system variables (issue 58).
   * Added --socket.
   * The mysqld binary was not detected correctly from 'ps' output.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-deadlock-logger:

2008-09-19: version 1.0.12

   * Added --pid option.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-duplicate-key-checker:

2008-09-19: version 1.1.8

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-find:

2008-09-19: version 0.9.13

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-heartbeat:

2008-09-19: version 1.0.11

   * Dies if heartbeat table does not have at least one row (issue 45).
   * Added --pid option.
   * Using debug (MKDEBUG=1) and --daemonize now causes script to die.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-parallel-dump:

2008-09-19: version 1.0.10

   * Added option --biggestfirst which is enabled by default (issue 31).
   * --noflushlock and --nolocktables caused a global flush and lock (issue 12).
   * Made debugging code more efficient in non-debug mode.
   * Fixed various bugs in shared code.
   * Moved all command-line documentation to POD.

Changelog for mk-parallel-restore:

2008-09-17: version 1.0.9

   * Made restores resume by default if possible (issue 30).
   * Added --noresume option (issue 30).
   * Added --[no]atomicresume option (issue 30).
   * --progress is reported by bytes instead of chunk count (issue 32).
   * --progress counting and prediction was improved.
   * LOAD DATA queries show bytes done and db.tbl name in SQL comment.
   * CHARACTER SET was declared in the wrong place for csv LOAD DATA query.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-query-profiler:

2008-09-19: version 1.1.12

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-show-grants:

2008-09-19: version 1.0.12

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-slave-delay:

2008-09-19: version 1.0.9

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-slave-find:

2008-09-19: version 1.0.3

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-slave-move:

2008-09-19: version 0.9.3

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-slave-prefetch:

2008-09-19: version 1.0.4

   * Added --pid option.
   * Using debug (MKDEBUG=1) and --daemonize now causes script to die.
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-slave-restart:

2008-09-19: version 1.0.9

   * Added --pid option (issue 18).
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-table-checksum:

2008-09-19: version 1.1.29

   * Added --argtable, --offset and --modulo for issue 53.
   * Added --nouseindex option (issue 8).
   * Fixed processing of column options (issue 4, patch by Travis Whitton)
   * Changed --emptyrepltbl to always completely empty the table (issue 21).
   * Added --schema option for issue 5.
   * Added --since and --sincecolumn options for issue 53.
   * Added --probability option for issue 53.
   * A missing table caused a crash (issue 35).
   * Functions specified with --function were not optimized (issue 43).
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-table-sync:

2008-09-19: version 1.0.9

   * Added --nouseindex option (issue 8).
   * Dies if any dest table has triggers unless --ignore-triggers (issue 37).
   * Added better support for CRC32 (issue 43).
   * Made debugging code more efficient in non-debug mode.

Changelog for mk-visual-explain:

2008-09-19: version 1.0.10

   * Updates to shared code.
   * Made debugging code more efficient in non-debug mode.

Further Reading:

]]> http://www.xaprb.com/blog/2008/09/19/maatkit-version-2325-released/feed/ 3 How Maatkit benefits from test-driven development http://www.xaprb.com/blog/2008/08/18/how-maatkit-benefits-from-test-driven-development/ http://www.xaprb.com/blog/2008/08/18/how-maatkit-benefits-from-test-driven-development/#comments Mon, 18 Aug 2008 13:54:24 +0000 Xaprb http://www.xaprb.com/blog/2008/08/18/how-maatkit-benefits-from-test-driven-development/ Further Reading:]]> Over in Maatkit-land, Daniel Nichter and I practice test-first programming, AKA test-driven development. That is, we write tests for each new feature or to catch regressions on each bug we fix. And — this is crucial — we write the tests before we write the code.* The tests should initially fail, which is a validation that the new code actually works and the tests actually verify this. If we don’t first write a failing testcase, then our code lacks a very important guarantee: “if you break this code, then the test case will tell you so.” (A test that doesn’t fail when the code fails isn’t worth writing.)

Most of the time when I do this, I write a test, it fails because I haven’t written any code yet, and I then go do some kind of clean-room coding. Then I run the test and it’s busted, and I have to go back to the code and figure out why, and after a few more tries I get it working. And then it feels great. (That’s the other thing about test-first coding. It’s really satisfying, like cooking the perfect dinner, arranging the plates beautifully and then eating.)

This time I wanted to write a pure-Perl implementation of CRC32, and embed it in mk-table-checksum. We try really hard never to rely on external modules, even modules that ought to be distributed with Perl itself. That keeps Maatkit as portable as possible and makes sure there is no installation hell. You can generally just get and run the Maatkit tools with no installation. So I referred to an existing CRC32 implementation, in Digest::Crc32. I wrote a test by referring to the value I got from MySQL’s built-in CRC32:

mysql> select crc32('hello world');
+----------------------+
| crc32('hello world') |
+----------------------+
|            222957957 | 
+----------------------+
1 row in set (0.00 sec)

Here’s the test:

is($c->crc32('hello world'), 222957957, 'CRC32 of hello world');

CRC32 is CRC32, so my code better agree with a working implementation. And then I wrote the code, which is a refactoring of the math in the module I linked to above. And then I ran the test, and it Just Passed with no further ado. w00t! This is pretty much a historic first for me! I thought at first that I’d screwed something up with the test, but I checked again. This is like getting a hole-in-one for me :-) So I just thought I’d share it with you. It feels awesome.

If you’re not doing test-first coding, you ought to give it a try. If you are conscientious about writing tests first, your code will always be easy to test. If you don’t, you write untestable code. Then it’s tough or impossible to ever get tests on it, and you spend the rest of your life wasting time on stupid bugs and slow, fearful development, never knowing what else you are breaking with your “fixes.”

Test-driven development is one reason The Rimm-Kaufman Group’s in-house bidding system blows away their competition. (RKG is my previous employer.) The comprehensive unit-test suite lets you know right away if you’ve broken something. That keeps the code clean and makes it possible to be extremely productive. I remember once when one of my co-workers there implemented a major feature in a very short time. It was also incredibly helpful when sharding the databases (anyone ever done this without a test suite? Would you like to share about how much of your systems broke during sharding? It was almost a non-event at RKG). The people I worked with before I joined RKG looked at me like an alien when I tried to explain that this was possible.

If you’re thinking that your code is not “that kind of code,” that “only certain kinds of code lend themselves to unit tests,” then stop. I’ve heard this before, and you’re wrong. It’s only “untestable” because you didn’t write tests first. Write tests first, and your code — all of it! — will be “that kind of code” that is testable. It’s hard. No one says it’s not; good programming is much harder than sloppy programming. But it’s well worth it.

Converting untested, untestable code into tested code is not so much fun, though. And in my experience you’ll rarely be rewarded for it, and your coworkers will not appreciate you raising the bar for them. Maybe you need a new job. I hear RKG is hiring. Did I mention that their codebase is built from the ground up on unit tests?

* OK, we’re not perfectly disciplined about this, but we’re pretty good about it.

Further Reading:

]]> http://www.xaprb.com/blog/2008/08/18/how-maatkit-benefits-from-test-driven-development/feed/ 3 Maatkit version 2152 released http://www.xaprb.com/blog/2008/08/12/maatkit-version-2152-released/ http://www.xaprb.com/blog/2008/08/12/maatkit-version-2152-released/#comments Tue, 12 Aug 2008 04:56:59 +0000 Xaprb http://www.xaprb.com/blog/2008/08/12/maatkit-version-2152-released/ Further Reading:]]> Download Maatkit

Maatkit version 2152 is ready for download. This release is also known as the “is this project really alive?” release. We thought we should delay until MySQL released a new Community Server version. Just kidding — it has nothing to do with that.

This release is also very significant in that it’s the first one that has large code contributions by someone other than myself. As you may know, Percona (my employer) has hired the very talented Daniel Nichter, author of mysqlreport and other goodies, to help with Maatkit. So far it is a match made in heaven, and Daniel did most of the coding for this release.

This is also our first release since Ask helped me move the project (thank you Ask!) to Google Code. That means you finally get a decent interface for entering issues, etc, etc. The only thing remaining on Sourceforge at this point is the online documentation, which I will probably move to maatkit.org soon. But more importantly, it means the developers have a decent interface for issues, etc etc. Sourceforge is just a bloody nightmare — their site keeps getting harder and harder to use, both as a developer and as a user. It had gotten to the point where simply adding the files to the site for download would take me hours. I tried to automate it, in true Perl fashion, but their make-a-release forms resisted my every effort. I cannot say what a relief it is to have usable project hosting that gets out of my way and lets me work. A double thanks to Ask for pushing me over the edge on this — it had been on my mind a long time. And thanks to Google, too, for a great project management interface.

Also note that the Sourceforge forums and mailing lists are dead. Google Groups is the preferred replacement.

Keep reporting those bugs and feature requests!

As you might expect, the changelog for such a long release cycle is, er, large. There’s a lot of new stuff here. I’d like to highlight the new features in mk-parallel-dump and mk-parallel-restore — which I just used to reduce a job that would have taken weeks down to mere days — and a lot of new code in mk-table-sync, as well as the up-and-coming mk-audit, which is in release-early/often mode.

Changelog for mk-archiver:

2008-08-11: version 1.0.10

   * Files downloaded directly from SVN crashed due to version information.
   * Added more information to --statistics and changed --whyquit slightly.

Changelog for mk-audit:

2008-08-11: version 0.9.1

   * Files downloaded directly from SVN crashed due to version information.
   * Added useful functionality.

Changelog for mk-deadlock-logger:

2008-08-11: version 1.0.11

   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-duplicate-key-checker:

2008-08-11: version 1.1.7

   * Files downloaded directly from SVN crashed due to version information.
   * Full-text indexes were not treated specially (issue 10).

Changelog for mk-fifo-split:

2008-08-11: version 1.0.1

   * Files downloaded directly from SVN crashed due to version information.
   * Added --offset option.
   * --statistics didn't calculate lines/sec properly.
   * Removed --sleep; EOF doesn't mean anything to a non-terminal.

Changelog for mk-find:

2008-08-11: version 0.9.12

   * Files downloaded directly from SVN crashed due to version information.
   * Added --exec_dsn so you can execute SQL on a different server.

Changelog for mk-heartbeat:

2008-08-11: version 1.0.10

   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-parallel-dump:

2008-08-11: version 1.0.9

   * Files downloaded directly from SVN crashed due to version information.
   * Added --progress option.
   * CHANGE MASTER TO in 00_master_data.sql used the I/O thread position.
   * Added features to permit resuming of dumps.
   * --age without --sets did the opposite of what it should (isssue 7)
   * --stopslave died after complaining the slave was not running.

Changelog for mk-parallel-restore:

2008-08-11: version 1.0.8

   * Files downloaded directly from SVN crashed due to version information.
   * Added --progress option.

Changelog for mk-query-profiler:

2008-08-11: version 1.1.11

   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-show-grants:

2008-08-11: version 1.0.11

   * Files downloaded directly from SVN crashed due to version information.
   * Anonymous users were not permitted (issue 28).

Changelog for mk-slave-delay:

2008-08-11: version 1.0.8

   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-slave-find:

2008-08-11: version 1.0.2

   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-slave-move:

2008-08-11: version 0.9.2

   * The -m option was not recognized as an alias for --timeout.
   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-slave-prefetch:

2008-08-11: version 1.0.3

   * Files downloaded directly from SVN crashed due to version information.
   * Added the --numprefix option for use in sharded data stores.
   * The Rotate binary log event type was not handled.

Changelog for mk-slave-restart:

2008-08-11: version 1.0.8

   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-table-checksum:

2008-08-11: version 1.1.28

   * Files downloaded directly from SVN crashed due to version information.

Changelog for mk-table-sync:

2008-08-11: version 1.0.8

   * Files downloaded directly from SVN crashed due to version information.
   * --synctomaster did not abort when unable to discover the master.
   * An error waiting for the master to catch up caused other tables to fail.
   * Added --bufferinmysql to help make GroupBy algorithm more efficient.
   * Added safety checks to prevent changing data on a slave server.
   * Added --skipslavecheck to prevent safety checks on destination server.
   * Made the GroupBy algorithm the default replacement for Stream.
   * Added the GroupBy algorithm, which can sync tables without unique keys.
   * Syncing could stop and leave a row to delete in the destination.
   * Generate command-line help from the POD.

Changelog for mk-visual-explain:

2008-08-11: version 1.0.9

   * Files downloaded directly from SVN crashed due to version information.

Further Reading:

]]> http://www.xaprb.com/blog/2008/08/12/maatkit-version-2152-released/feed/ 2