Tag Archive for maatkit at Xaprb

Tag Archive for 'maatkit'

Baron Schwartz on a podcast at MySQL Conference and Expo 2008

Published at April 25, 2008 in PostgreSQL and SQL. 0 Comments

I did an interview with Barton George from Sun while I was at the conference last week. Barton has now posted the interview. If you’re quick, you can listen to it before I do.

Topics: everything and anything, including Maatkit and PostgreSQL.

Technorati Tags:Baron Schwartz, Barton George, maatkit, mysqluc08, mysqluc2008, Podcast, Sun

MySQL Conference and Expo 2008, Day Three

Published at April 19, 2008 in SQL. 0 Comments

Here’s a rundown of Thursday (day 3) of the MySQL Conference and Expo. This day’s sessions were much more interesting to me than Wednesday’s, and in fact I wanted to go to several of them in a single time slot a couple of times.

Inside the PBXT Storage Engine

This session was, as it sounds, a look at the internals of PBXT, a transactional storage engine for MySQL that has some interesting design techniques. I had been looking forward to this session for a while, and Paul McCullagh’s nice explanations with clear diagrams were a welcome aid to understanding how PBXT works. Unlike some of the other storage engines, PBXT is being developed in full daylight, with an emphasis on community involvement and input. (Indeed, I may be contributing to it myself, in order to make its monitoring and tuning capabilities second to none).

PBXT has not only a unique design, but a clear vision for differentiating itself from other transactional storage engines. It’s not trying to clone any particular engine; Paul and friends are planning to add some capabilities that will really set it apart from other engines, including high-availability features and blob streaming.

I left this session with a much better understanding of how PBXT balances various demands to satisfy all sorts of different workload characteristics, how it writes data, how it achieves transactional durability, and so on. I think these capabilities, and its performance, can really be assessed only in the real world (of course), but in principle it sounds good. I love knowing how things work!

There were about 30 people in the talk. I wish there had been more, because I think PBXT is going to be an important part of the open ecosystem going forward. However, I feel pretty confident people will take more notice if it starts to get used in the real world. Someone had a video camera there, so you might check out the video when it’s available. Paul’s explanations are really good.

Helping InnoDB Scale on Servers with Many CPU Cores and Disks

This session was Mark Callaghan’s chance to unveil the work he and others have been doing on InnoDB’s scalability issues, which mostly revolve around mutex contention. Mark’s team has completely solved the problems on their workload and benchmarks. In fact, after the changes, InnoDB exhibited significantly better performance even than MyISAM, which began to be limited by the single mutex that synchronizes access to its key cache. (Yes, in fact MyISAM has scalability problems too).

Google’s workload for MySQL, in case you’re wondering, is pretty traditional (i.e. not web-like; more like an “enterprise” application). Heavily I/O-bound, 24/7 critical systems, and so on.

Mark also wore several community t-shirts at various points in the talk, including one of my Maatkit t-shirts. Mark said Maatkit would be perfect if only it were written in Python (Google’s preferred scripting language). Alas, Mark, it’ll stay in Perl. But thanks for the nice compliment anyway.

The room was packed full.

Scaling Heavy Concurrent Writes In Real Time

Dathan Pattishall, formerly the lead architect at Flickr, explained his techniques for scaling Flickr’s write capacity. He talked about how he’d worked to reduce primary key sizes, queued writes for batching, separated different types of data into different types of tables, and more. Dathan has never been afraid to do what he thinks is a good idea, even if it flies in the face of “best practices,” so I was happy to finally hear him talk.

By the way, Dathan pointed out that distributed locking with memcached and add() isn’t a silver bullet. It works ok until memcached evicts your lock due to the LRU policy. He uses MySQL’s built-in GET_LOCK() function for locking.

Dathan’s blog is a good source of information about his sometimes unorthodox approaches to database design.

The Power of Lucene

This was the only one of Frank (Farhan) Mashraqi’s talks I got to attend. This was pretty technical: how Lucene works, how to configure and install it, how to index documents, how to execute searches. If you were wondering how much work and complexity it would be to install and use Lucene, this talk would have been good for you to attend; I’ve never used it myself, but I’m pretty sure Frank covered everything you need to know.

Technorati Tags:Dathan Pattishall, Farhan Mashraqi, GET LOCK, innodb, Lucene, maatkit, Mark Callaghan, memcached, myisam, mysqluc2008, Paul McCullagh, pbxt

MySQL Conference and Expo 2008, Day Two

Published at April 17, 2008 in SQL. 0 Comments

Day two of the conference was a little disappointing, as far as sessions went. There were several time blocks where I simply wasn’t interested in any of the sessions. Instead, I went to the expo hall and tried to pry straight answers out of sly salespeople. Here’s what I attended.

Paying It Forward: Harnessing the MySQL Contributory Resources

This was a talk focused on how MySQL has made it possible for community members to contribute to MySQL. There was quite a bit of talk about IRC channels, mailing lists, and the like. However, the talk gave short shrift to how MySQL plans to become truly open source (in terms of its development model, not its license). I think there was basically nothing to talk about there. I had a good conversation about some of my concerns with the speaker and some others from MySQL right afterwards.

There was basically nobody there — I didn’t count, but I’d say maybe 10 or 12 people. I think this is a telling sign.

Architecture of Maria: A New Storage Engine with a Transactional Design

I was interested in this talk because I’m interested in the tension between Falcon and Maria (and between Falcon and everything, for that matter) but I left and went to the expo hall again after a bit. The talk was good but I’d already seen and/or read it, and the question-and-answer component wasn’t enough to keep me there.

The MySQL Query Cache

This was the second session I gave at the conference, and again it was standing-room-only, with nearly 300 attendees according to the person who was watching the door. The questions were frequent and added a lot to the discussion. Slides will be on the conference website when they post them.

Grazr: Lessons Learned Building a Web 2.0 Application Using MySQL

I was keenly interested in this talk because a) I am a big fan of Patrick Galbraith’s work with many different projects, and b) I had heard a lot about Grazr but didn’t know much about it. However, I missed most of the talk. About ten minutes into it, I got a call I couldn’t refuse: my wife!

However, I did sneak back into the room for the last bit too. And I gave Grazr a try. Unfortunately, I got really confused by it; I tried a bunch of different ways to import my Google Reader’s OPML. I got that to work, but then I couldn’t figure out how to read the feeds in the OPML via Grazr. Then I think I figured that out (I’m not sure) but it didn’t strike me as a very handy way to read my feeds. I’ll try taking another look at it later if I get time. (I’m all ears if there’s a better way to read feeds).

Extending MySQL

This one was mostly for fun. I knew a lot about UDFs already (I’ve created some) and I knew about the pluggable storage engine API. But I didn’t know about pluggable event daemons. Holy cow, what a great way to shoot yourself (or your server) in the foot! All the power of an atomic bomb, with all the safety of SPF 5 sunblock in a nuclear attack. Or something like that. But darn, it sure is nifty. Brian is a great speaker too — very lively.

You know, there’s another way to extend MySQL that most people don’t seem to know about, which Brian didn’t mention. That is procedures (not stored procedures). They are sort of like a post-filter for a result set, and like UDFs they’ve been around forever. I have never heard of anyone writing their own, but there’s an example in the server itself: PROCEDURE ANALYSE.

Expo hall

I went to the expo hall to meet and greet many of the companies that Percona (my employer) is already working with (doing independent benchmarks, performance verification, analysis etc) or will be in the future. I also wanted to grill some of the vendors on their technology. Usually I find them very cagey; they claim X times faster this-or-that, but won’t tell you how, and won’t tell you what their systems don’t do well. I don’t understand why they take this approach; you can’t hide your system’s strong and weak spots. There is no security through obscurity, and shrewd independent observers are going to get to the bottom of it with or without your permission.

So, for instance, I was talking with Tokutek, who claimed to be a drop-in replacement for InnoDB with 200x better performance and apparently no downsides. However, on closer questioning, I did get him to admit that the system has table-level locking. Thus it won’t give any concurrency, so saying it’s a drop-in InnoDB replacement is questionable. And the comparison against InnoDB seemed contrived to create a worst-case situation with bad tuning and a workload so it would perform terribly. An honest comparison tunes both systems to their highest performance and measures them; you can’t tune one system as badly as possible and compare it to the other’s best-case performance. I pressed on further and asked about range scans in some specific cases (they claim they’re great at range queries, and equal to InnoDB on everything else). At last they admitted they can’t perform well on some very common queries such as real-life queries InnoDB performs very well on for me. They said these are “point queries” but that’s not true; you can design indexes to support many different ways to range-query a table in InnoDB and get great performance. So it sounds to me like Tokutek’s storage format is extremely narrowly focused, and there is indeed a trade-off. I will be interested to see how their technology develops, though. It’s not done yet.

In general

There are a lot of Maatkit t-shirts walking around, which makes me happy. If I’d printed 200 of them, I probably could have given them all away. I was wearing a PostgreSQL t-shirt myself. Proudly, I might add. I’m not the only person here who’s interested in PostgreSQL. This morning I met a person from EnterpriseDB.

Yesterday was a bit slow in terms of interesting sessions, but there was a lot going on in the hallways, the expo hall, the meetings over lunch, and so on.

Technorati Tags:Brian Aker, EnterpriseDB, Falcon, Grazr, maatkit, Maria, mysqluc2008, Patrick Galbraith, Percona, query cache, Tokutek

Maatkit t-shirts are here

Published at April 14, 2008 in Maatkit and SQL. 11 Comments

I’m at the MySQL Conference and the t-shirts I created for Maatkit have arrived. Come get yours! They are high-quality, attractive shirts you’ll be proud to wear, and they are a nice rich wine-red color.

Harrison Fisk (co-author of MySQL Clustering) got the first one, because he told me that he recommends Maatkit to MySQL Support customers about twice a week. I made sure to save one for Jay Pipes too, because his luggage got lost so he has nothing to wear. Unfortunately, I didn’t make any Maatkit underwear, sorry Jay. Now I know what to do for next year…

I’ve already given out a whole bunch of them, and at this rate all 50 of them might not last the day!

Technorati Tags:Harrison Fisk, Jay Pipes, maatkit, mysql

New Maatkit release policy

Published at November 18, 2007 in Perl, SQL and Tools. 0 Comments

Download Maatkit

Maatkit (formerly MySQL Toolkit) has for some time been released both as a bundle, and as individual tools. It’s too much work to maintain the individual packages, and I don’t think it really benefits anyone much, if at all. While the tools will still be versioned separately, I’m going to discontinue releases of the individual packages, and just release the one uber-package from now on.

This will also make it easier for me to manage the name change, but that’s just an extra incentive; I’ve been considering this for a while.

By the way, Sourceforge indicated it would take up to a couple of days to finish the project’s rename, but it took only a few minutes. Lots of broken links; I’ve asked for a permanent redirect from the old URLs to the new.

Technorati Tags:maatkit, mysql, release policy, sql

MySQL Toolkit is now Maatkit

Published at November 18, 2007 in Perl, SQL and Tools. 9 Comments

I am so lucky I married an archaeologist.

Choosing a new name for MySQL Toolkit has been a hassle. I wanted to avoid a literal name, such as, um, MySQL Toolkit. Short is good. And so on, and so on. All the while, the Phoenix/Firebird/Firefox naming debacle was in my thoughts. I only want to do this once.

At first I tried not to stray too much from the current name. MyToolkit, eh, it’s okay, but a) it’s taken and b) it reminds me of Microsoft Windows, where everything is “my.” My Documents, My Pirated Music, My… you get the idea. I tossed out various combinations of Xaprb and Toolkit. Xaprb is unique, and it’s not completely unknown anymore, but it’s not that great a name. (For those who don’t know, it’s a total geek-out thing. It’s what you get when you type my first name on a keyboard that’s been remapped to Dvorak.) XAToolkit seems cool at first, but is this thing really about XA transactions? … No.

I tried to think of some mythical figure, such as an Egyptian god(dess). Oooh! Thoth is the god of writing, the scribe, record-keeping, etc… that’s related to databases, right? But it’s actually pronounced ‘toe-th’ so no one would ever find it, and I’d have to correct people at conferences and such (ack!). And anyway, that’s an appropriate name for a database, not a set of tools for augmenting a database. (Seshat got eliminated for the same reasons, though she’s even cooler than Thoth).

On the topic of mythical figures, Sargon, Hammurabi, Ashurbanipal, and Gilgamesh are all wicked cool (and they’re not all mythical), but not good names for the toolkit. (Neither is Engleburt Humperdink, but that’s another blog post).

What to do?

Ask my wife, of course. She is a Near East Archaeologist, among her many areas of expertise. She’s wonderfully clever. I must say, she was initially too clever for the task. She wrote me an email suggesting “IT Toolk (get it?)” I did not get it, and she didn’t reply to my “I give up” email for a while, so I was left to agonize over what I was missing. Is it a name of an Assyrian scribe? A word in some language only she can read…? No. It’s “Toolkit” with the last two letters placed first. That was anti-climactic. But when I started picking her brain, she immediately thought of Ma’at.

Ma’at is not only an Egyptian word, she’s a goddess. (We need more women in this profession!) She is the patron of truth, harmony, and order. She restores things to their proper balance and place. Without her, everything would return to chaos. She wears an ostrich feather, and the heart of a deceased person has to be weighed against the feather when passing to the underworld.

In fact, ma’at isn’t just a name and a word, it’s a concept, as my wife explained to me. This concept doesn’t have an exact parallel in other languages and cultures. You should read about it via the link I just gave — it’s really quite a fascinating bit of Egyptology. I asked my wife to find a good image of weighing the heart of the deceased, and she took some time to describe the scene:

The deceased is in white. He is visible along the top of the image, in front of a dozen or so judges. He’s visible again in the main part of the image. Anubis is leading him by the hand into the presence of Osiris. Osiris is not the figure kneeling under the scale — that’s Anubis again.

The scale of judgment has the deceased’s heart on the left side, in a jar, and the feather of justice (ma’at) on the right side. The Egyptians believed that the heart, not the brain, was the source of one’s personality and identity. The goddess Ma’at is on top of the middle of the scale, supervising the proceedings. She has the feather of justice on her head. Thoth, the god of writing (a personified Ibis), takes down the judgment. If the deceased passes judgment, he will continue to the underworld; otherwise he will be eaten by the “devourer,” who is part lion, part alligator, and sits under the right-hand arm of the scale.

In this scene the judgment was positive and Horus (the son of Osiris — a hawk) leads the deceased towards the canopy where Osiris awaits. Osiris is flanked by two goddesses. One is Isis; Nephthys is probably the other.

I don’t know about you, but I think this is all very interesting; maybe I should become an Egyptologist.

But best of all, ma’at applies to the tools I’ve written, too! Without them, your replication gets out of sync, and you don’t even know it. Fortunately, the tools let you bring things back to the way they should be, restoring order to your universe of data. And so on.

In the end, maybe Maatkit is not the greatest name for various reasons (Ma’at was already taken on Sourceforge, by the way), but it’s so freakin’ cool that I can’t pass it up. You can’t find a perfect name, anyway; if it’s good by one metric, it’s bad by another. Maatkit it is.

I’m going to be changing the toolkit’s name on Sourceforge quite soon. There’s also some other interesting stuff going on, which I’ll write about separately.

(Ma’at is pronounced “mott,” by the way.)

Technorati Tags:Dvorak, Egypt, egyptian god, Maat, maatkit, mysql, mythology, sql, Thoth

Xaprb