$ while read line; do clear; perl MyTestScript.t; done

This works with any language, not just perl — just replace the test command with the right one. ALT-TAB, press Enter, ALT-TAB back to your editor.

Part 2 of my tip is to make it really easy to drop into the debugger if you want. Notice the small change here:

$ while read line; do clear; perl $line MyTestScript.t; done

Now instead of pressing Enter, you can type “-d” and press Enter. Presto, you’re in the debugger. This also works for any language that has a built-in debugger. Of course, you can also pass any other arguments you want, such as enabling profiling.

Further Reading:

• How Maatkit benefits from test-driven development
• How to unit-test code that interacts with a database
• Google Test Automation Conference, Day 1
• Planned change in Maatkit & Aspersa development
• Four companies to sponsor Maatkit development

Note: As commenters have pointed out, I’m not necessarily using “unit” in the agreed-upon way here. Everything I say can be applied to ultra-pure unit testing too, but I go beyond that. I will hold fast to my assertions about mocking though *grin*

Is it really impossible or even hard?

I disagree. In one of my previous articles I said The Rimm-Kaufman Group, my previous employer, has a comprehensive unit-test suite. When I say comprehensive I mean it: database interaction is fully tested, too. I know because I was heavily involved in building it. Even extremely complex things like big reports that are generated from lots of data are tested. And believe me, sharding the databases would have been much harder without complete code coverage. It’s really not that complicated to unit-test against a database, and it’s so worth it. Here are some hints about how you can do this.

There are many ways to do it, but I’ll just describe the basics of the system I helped build. There are several moving parts to the test suite (“smoke“), but one of them sets a magical environment variable. And then, all code that connects to a database server magically gets back a different database connection from the create_me_a_connection() function. This is because there is a database connection abstraction library that respects the environment variable. It’s really pretty simple for the most part; instead of doing DBI->connect(…) you just call this function, which is a thin wrapper that hands back a connection object.

This wrapper is itself unit-tested thoroughly, too. This ensures that when some code is being run from a test, it cannot (I mean cannot!) connect to a production database, and vice versa. There are some conventions about production and test servers that make sure the abstraction library can tell for sure. If there’s any confusion, of course, it will die in a non-recoverable way. Safety first.

Building a good development environment

Just as each developer has their own copy of the code from version control, each developer has their own private database server running on the dev machine. There are some simple conventions that make this possible: Unix user ID plus a constant for the port number, etc. It’s really quite easy. The private database server is a slightly modified version of Giuseppe Maxia’s MySQL Sandbox tool. It can be torn down and set up afresh as desired. It is wiped clean and re-filled at the start of every test, with a small, tightly focused dataset carefully chosen to represent the conditions the code is supposed to work with. (Each test has its own dataset).

If this sounds like a system that can’t work on a large scale, well, it does. That’s the secret sauce that I won’t reveal in this post. (It’s my past employer after all, and I can’t go revealing everything about them can I?) You just have to be smart about it. When a database is central to your business, you either figure out how to get this right, or you pay the consequences in lost time and poor code quality.

I and the other developers there (another secret: it’s a small team; small teams build great things) built several quick utilities to help develop unit tests against a database. There are utilities to get a minimal necessary dataset for testing and dump it into a file that can be loaded by the test. There are utilities that can migrate schemas and update the tests to match the schema changes. And so on, and so on. This is possible because of careful planning for testability, and really smart things like super-consistent and sensible naming conventions for database objects. (Ruby On Rails owes a lot of its success to simple things like this, too. Conventions are really powerful.) Maybe I’ll write about the database naming conventions some other time — I have to credit Alan Rimm-Kaufman a lot for designing those conventions. It was a stroke of genius.

Things to avoid

There are several things I do not recommend doing when you unit-test code that talks to a database. I’ll just mention a couple:

Don’t mock anything! In general I think mocking is the devil. Most of the mock objects I’ve ever seen reflected a propensity to test an implementation instead of a behavior, which is also the devil. Write all your code to test a test instance of something real, and do not mock up a database to test against. It is a rabbit-hole that you will not emerge from easily.
• Never let a test connect to a production database. Never, ever. Worlds of hurt will follow. Not only are you risking your production data, but what about the risk to your code? You’re testing against things that will almost certainly change and break your tests; and you’re possibly polluting your live data with testing data and/or changing live data from the tests.
• I also recommend developing unit tests for your current database functionality if you’re thinking about changing it much. Don’t like MySQL’s lax error handling? Plan to set the SQL_MODE to something stricter? Dive into that database abstraction library and make your tests run in strict mode first by setting SQL_MODE on every new connection that’s created when running inside a test; fix all the breakage in the test suite; feel sure that your code isn’t going to break in production. That was easy!

Summary

Once your creative juices get flowing, you’ll see tons of places your unit test suite can help you out.

If you’re in the Oracle or SQL Server world, or any other world where you can’t just set up and discard database instances at will due to licensing problems, you’re going to have to be a little more inventive. But you can still do it. (Don’t you wish you’d chosen Freedom?) And unit tests are just as beneficial for apps based on Oracle as they are for MySQL.

Have fun! Go forth and test some more!

Further Reading:

• How to write unit tests for ease of refactoring
• How Maatkit benefits from test-driven development
• A productivity tip for test-driven development
• When documentation is code
• Google Test Automation Conference, Day 1

Most of the time when I do this, I write a test, it fails because I haven’t written any code yet, and I then go do some kind of clean-room coding. Then I run the test and it’s busted, and I have to go back to the code and figure out why, and after a few more tries I get it working. And then it feels great. (That’s the other thing about test-first coding. It’s really satisfying, like cooking the perfect dinner, arranging the plates beautifully and then eating.)

This time I wanted to write a pure-Perl implementation of CRC32, and embed it in mk-table-checksum. We try really hard never to rely on external modules, even modules that ought to be distributed with Perl itself. That keeps Maatkit as portable as possible and makes sure there is no installation hell. You can generally just get and run the Maatkit tools with no installation. So I referred to an existing CRC32 implementation, in Digest::Crc32. I wrote a test by referring to the value I got from MySQL’s built-in CRC32:

mysql> select crc32('hello world');
+----------------------+
| crc32('hello world') |
+----------------------+
|            222957957 | 
+----------------------+
1 row in set (0.00 sec)

Here’s the test:

is($c->crc32('hello world'), 222957957, 'CRC32 of hello world');

CRC32 is CRC32, so my code better agree with a working implementation. And then I wrote the code, which is a refactoring of the math in the module I linked to above. And then I ran the test, and it Just Passed with no further ado. w00t! This is pretty much a historic first for me! I thought at first that I’d screwed something up with the test, but I checked again. This is like getting a hole-in-one for me :-) So I just thought I’d share it with you. It feels awesome.

If you’re not doing test-first coding, you ought to give it a try. If you are conscientious about writing tests first, your code will always be easy to test. If you don’t, you write untestable code. Then it’s tough or impossible to ever get tests on it, and you spend the rest of your life wasting time on stupid bugs and slow, fearful development, never knowing what else you are breaking with your “fixes.”

Test-driven development is one reason The Rimm-Kaufman Group’s in-house bidding system blows away their competition. (RKG is my previous employer.) The comprehensive unit-test suite lets you know right away if you’ve broken something. That keeps the code clean and makes it possible to be extremely productive. I remember once when one of my co-workers there implemented a major feature in a very short time. It was also incredibly helpful when sharding the databases (anyone ever done this without a test suite? Would you like to share about how much of your systems broke during sharding? It was almost a non-event at RKG). The people I worked with before I joined RKG looked at me like an alien when I tried to explain that this was possible.

If you’re thinking that your code is not “that kind of code,” that “only certain kinds of code lend themselves to unit tests,” then stop. I’ve heard this before, and you’re wrong. It’s only “untestable” because you didn’t write tests first. Write tests first, and your code — all of it! — will be “that kind of code” that is testable. It’s hard. No one says it’s not; good programming is much harder than sloppy programming. But it’s well worth it.

Converting untested, untestable code into tested code is not so much fun, though. And in my experience you’ll rarely be rewarded for it, and your coworkers will not appreciate you raising the bar for them. Maybe you need a new job. I hear RKG is hiring. Did I mention that their codebase is built from the ground up on unit tests?

* OK, we’re not perfectly disciplined about this, but we’re pretty good about it.

Further Reading:

• A productivity tip for test-driven development
• How to unit-test code that interacts with a database
• Planned change in Maatkit & Aspersa development
• Four companies to sponsor Maatkit development
• Google Test Automation Conference, Day 1

• All the code, except for a tiny bit of “glue” and “setup” code, is in modules.
• Lots more tests for the modules.
• A new sync algorithm (I still haven’t rewritten the top-down and bottom-up, which are designed for network efficiency more than MySQL efficiency, and are very complicated). This algorithm is called “Chunk” and is based on the chunking module I’m re-using from two of the other tools. This allows syncing the table a bit at a time to avoid locking it so much.
• The tool chooses its own parameters, including choosing the sync algorithm automatically by examining indexes.
• Proper exit codes, as well as several other smaller issues requested via bug reports.
• The tool now syncs entire servers. That is, you don’t have to specify a table. It’ll find all the tables and just sync them.
• The tool can sync many servers. You give it five servers, it will treat the first as the source, and sync every table in the source to each of the four remaining servers in turn.
• It can work via replication. It can discover a master’s slaves via SHOW SLAVE HOSTS and sync each slave to the master. You can also point it at a slave and it’ll discover the master, connect to it, and sync the slave to the master.
• It integrates with mk-table-checksum’s results. If you’ve given the –replicate option to mk-table-checksum, the slave’s results are stored in a table. It can read that table and sync anything marked as different. This can be combined with sync-to-master and auto-discover-slaves functionality.
• Lots of other bugs and problems are gone simply because I’m using the modules I wrote for other tools. This includes issues with table parsing, identifier quoting, etc etc. As an aside, I have to roll my own for almost everything, because I can’t rely on things like DBI’s quote_identifier() function — it does not work in earlier versions, which are amazingly common in the real world.

Whew! So what isn’t done yet?

• Bi-directional syncing.
• The Nibble sync algorithm. It will be preferred over Chunk and can be used in more cases.
• Documentation.
• Full support for wide characters. (This is non-trivial in Perl. I need to research it. A partial solution might not be hard, but I’m worried about the versions included in, for example, RHEL 3, which is very widely used.)
• Updating other tools to work right with the changes to shared code.
• Locking and transaction code. The tool will ultimately use FOR UPDATE/LOCK IN SHARE MODE automatically on InnoDB tables instead of locking them, for example.

Here’s a sample of what it can do, using a replication sandbox I set up with Giuseppe’s MySQL Sandbox. The sandbox contains a copy of the Sakila sample database. I’ll just mangle a few films on the slaves:

baron@kanga:~$ cd rsandbox_5_0_45/
baron@kanga:~/rsandbox_5_0_45$ ./s1
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.0.45-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

slave1 [localhost] {msandbox} ((none)) > update sakila.film set title='academy dinosaur2' limit 12;
Query OK, 12 rows affected, 12 warnings (0.07 sec)
Rows matched: 12  Changed: 12  Warnings: 0

slave1 [localhost] {msandbox} ((none)) > Bye
baron@kanga:~/rsandbox_5_0_45$ ./s2
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.0.45-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

slave2 [localhost] {msandbox} ((none)) > update sakila.film set title='academy dinosaur2' limit 1;
Query OK, 1 row affected, 1 warning (0.05 sec)
Rows matched: 1  Changed: 1  Warnings: 0

slave2 [localhost] {msandbox} ((none)) > Bye

OK, now I’ve messed up the first 12 films on one slave, and the first 1 on another. I could just go ahead and sync them right away, but first I’ll do a table checksum to demonstrate that functionality:

baron@kanga:~/rsandbox_5_0_45$ mk-table-checksum --replicate=test.checksum --port=16045 127.0.0.1 -q

And now I’ll tell the sync tool to go fix the differences the checksum revealed:

baron@kanga:~/rsandbox_5_0_45$ mk-table-sync  --replicate=test.checksum h=127.0.0.1,P=16045 -vx
# Syncing P=16046,h=127.0.0.1
# DELETE INSERT UPDATE ALGORITHM DATABASE.TABLE
#      0      0     12 Chunk     sakila.film
#      0      0      0 Chunk     sakila.film_text
# Syncing P=16047,h=127.0.0.1
# DELETE INSERT UPDATE ALGORITHM DATABASE.TABLE
#      0      0      0 Chunk     sakila.film
#      0      0      0 Chunk     sakila.film_text
baron@kanga:~/rsandbox_5_0_45$ 

Pretty easy, huh? Take a look at the output: the first thing it did was fix the 12 films I changed. sakila.film has a trigger that updates sakila.film_text, so that table got changed too. The checksum tool caught this difference, but the differences were gone by the time the sync tool examined them, again due to the trigger. On the second slave, no differences were found at all, because the changes to the first slave were made on the master, automatically fixing the second slave. (This won’t always be the case, but it worked in this example).

While I’d love to continue building the perfect beast, I’m going to have to call it quits around noon today and start cleaning up, writing the documentation, and getting ready to release the code. I’m not sure how much I’ll finish in that time.

By the way, anyone who wants to is welcome to get the code from the Maatkit SVN repository! I never make a big deal out of that because I generally assume people want to run released code, but SVN is there if you want it…

Further Reading:

• Progress on Maatkit bounty, part 2
• Progress on Maatkit bounty
• Progress on Maatkit bounty, part 4
• Maatkit bounty begins tomorrow
• How to sync tables in master-master MySQL replication

Rewriting code so it’s testable is a major focus for me now. Some of these tools have gotten complicated enough that I can’t keep track of all the code. In other words, they’re collapsing under their own weight.

Back in the project’s humble beginnings, it seemed adequate to just copy and paste a few lines here and there; after all, these are just scripts, right? Right. So I’ll just copy a few lines of code that do command-line option parsing and help screens. Hey, it turns out that several of the tools can connect to more than one server, so simple -u, -h and -p options won’t do; so I invent a DSN-like notation that lets the tools connect to an arbitrary number of servers. Copy and paste that code, too. It’s only ten lines — no big deal. Pretty soon I find out that many of the standard Perl modules aren’t available, for a lot of people. And even when they’re available, people have old versions and can’t upgrade, so I can’t rely on basic things like the quote_identifier() function in DBI modules; time to write my own. Well, that’s only a single line! Surely that’s okay to copy and paste.

As Kurt Vonnegut says, “So it goes.” This is the death not only of quality, but of maintainability and extensibility. The Right Answer ™ is to write everything as modules, with proper test suites, and then make the scripts as minimalistic as possible — essentially gluing the modules together with a few lines of harder-to-test code. That’s how I’m used to working, too, but for some reason I can’t explain, it seemed okay not to work that way with this project. That has turned out to be a big mistake, which I’m slowly correcting out of necessity.

But it turns out it’s not that simple, either. I’ve gotten a lot of emails, phone calls from friends, and bug reports about how hard it is to install or update Perl, or get a CPAN module, on many systems. It turns out that a lot of companies are rightfully suspicious about CPAN (I have a tolerate-hate relationship with it myself), and won’t let my consultant friends install or upgrade any module without a lot of red tape. OK, you say, so bundle and distribute the modules the toolkit needs, and they can be installed locally with the toolkit. That sounds nice, but it’s even worse for a variety of reasons. Just to mention one: did you know that it can be a pain in the butt even to set @INC so a module sitting in the same directory with the script will be found by the script? (Please don’t tell me how easy it is, or I’ll let you respond to the next person trying to get it to work on an obscure platform with a Perl installation from the middle ages). Okay, I’ll mention two reasons: some Perl modules have to be compiled and customized just for the operating system you’re installing them on, or they’ll segfault (of all things)! Don’t get me wrong, I don’t think the grass is greener on the other side; no way do I want to try writing these things in C or Java. Perl is about as portable as it gets.

The net result is that I have to do a lot of little tricks to make these things standalone programs, as much as humanly possible. I’m trying to reduce dependencies on external modules, even those that are part of core Perl. I’m re-inventing functionality because it’s not available in all versions. I’m writing modules that can be tested, but I’m not shipping them as separate modules; I’m basically using sed to copy-and-paste the module’s code into the scripts.

Why am I doing all this work?

Because it’s less work than not doing it.

But it is significantly more work than just whacking together some “scripts” and uploading them. That’s why there is a critical mass beyond which it gets harder to grow a project. The solution to this is to find a way to do things differently, work smarter, not harder. The challenge is to switch the fight against the demons of bad code and maintainability so it’s on my terms. In other words, don’t fight against these characteristics of growth; make them work for me. I won’t say I’ve learned that lesson completely, but I’m starting. That’s why I’m automating basically everything about this project (though for some reason I can’t get WWW::Mechanize to stay logged into Sourceforge, so I’m having a hard time automating part of the release process).

I’m also considering ways to provide this toolkit without taking so much out of my own pocket. What started out as me developing tools for my employer, and them graciously agreeing to let me make them available for Sourceforge, has gone far beyond my employer’s needs now. I can’t ask my employer to carry the weight, so it has fallen to me for a while now. That’s okay for some period while I work out how to do it differently, but not indefinitely. Among other things, it cuts into time I want to spend with my wife. Charging for support has definitely crossed my mind, as has some kind of community/enterprise split (such as the one Zmanda does). I don’t want to go there yet — so I’m just asking for a week of sponsored time off work, to begin with.

By the way, the process of replacing copy/pasted code isn’t without its hitches. I just found and fixed a bug in MySQL Table Checksum that I caused by moving the DSN parsing code to a module. And someone else just reported a different bug in another tool, where it turns out the copy/pasted code wasn’t quite identical and I changed the functionality by moving it to the module. Release early, release often. Rely on users to find bugs and report them. So it goes.

Further Reading:

• Does MySQL really have an open-source business model?
• MySQL: Free Software but not Open Source
• Making Maatkit more Open Source one step at a time
• What does an open source sales model look like?
• Measuring open-source success by jobs