To Gentoo or not to Gentoo?
Some people who know I’ve used Gentoo asked me my thoughts on using it for MySQL servers. Here are my opinions and experiences while using Gentoo, both for desktop systems and for servers.
This is long, but hopefully the brain dump is useful to you. I have so many thoughts on this subject that I’m having a hard time really organizing them well, and I’m censoring myself too because I know I represent only a very small, focused viewpoint on Gentoo.
This article is highly subjective, so please take what I say with a spoonful of salt. Your mileage may vary.
Finally, before I begin: I have a lot of reasons for moving away from Gentoo myself, and this article is mostly about why I don’t think Gentoo is a good idea for what I use computers for, but I don’t mean to criticize anyone or say it’s no good in general (in fact I know of some uses where it works great, and lots of places use it). Nothing here is personal. This is simply the train of thought I went through when deciding not to use a source-based, continuous upgrade distribution for my own systems.
The 50,000 foot view
I recently switched away from Gentoo, after using it since 2001 or 2002. I used Red Hat, Slackware and other distributions before that, and my main gripe with them was the unavailability of reasonably recent software; I felt they became stale and forced you to upgrade your entire system to get something other than an ancient version of [insert package name here]. There were also endless dependency conflicts. By contrast, Gentoo let me build a recent version of whatever I wanted, and I felt a lot happier with it.
In recent years though, I’ve become more aware of the pain of a constant, incremental upgrade process, which is what Gentoo gave me. Even though in theory it’s not necessary to constantly keep updated with the most recent version of all installed packages, in practice I find it is. I found if I didn’t keep abreast of changes, I couldn’t catch back up again without a lot of pain.
I also started to question some of the things people assert about Gentoo, such “it is small, light and focused on speed.” I find packages compiled under Gentoo, with the precisely right CFLAGS customized just for my processor and hardware, and with just the right USE flags customized to only install what’s needed, are often much less performant than software compiled for generic i386 architecture with no optimizations. Finally, some of the software I’ve installed under Gentoo seems to be broken.
The following are more details of my experiences.
Upgrading is painful
Upgrading can be an enormous pain:
- First you sync,
- Then you run
emerge -avuD --newuse worldand look at what’s going to be upgraded, - Then you cancel and tweak some USE flags,
- Then you pull your hair out as you still can’t understand what package is trying to install 301MB of dependencies you don’t want,
- Then you use
equeryand friends to try to find out, - Then you pull your hair out because these tools say there is no such package, or can’t find the files for it, or find the package but say nothing at all,
- Finally you get on the forums and figure it out, and get the flags/whatever right. It turns out there’s a new default USE flag for some package that doesn’t show up in the upgrade list, and it wants to install a bunch of “dependencies” it doesn’t need. For example, there are lots of things that want to install multiple versions of QT, including QT itself. If you let it install version 3 for just one program, it will want to simultaneously upgrade itself to 4 and keep the version 3, because that program needs it,
- Then you let it upgrade, which still has to download 194MB (hey! you saved 105MB, be happy!) over your DSL connection and takes 4 hours to compile on a fast 64-bit machine loaded with RAM — why? Why, because of course GCC has to be upgraded from 4.1.1-pre5_r16a to 4.1.1-pre5_r16b, which is probably three lines of source that changed. But as I say later, you don’t dare not upgrade GCC.
Now your system is upgraded, right? Wrong, everything is broken because it isn’t linked against the new glibc/QT/whatever. Time to revdep-rebuild — oh goody, it wants to rebuild your entire system. Another 7 hours of compiling. Cozy up to the back of your computer’s case, it’s cold in the basement and that thing is warm. Who needs a fireplace? Aaahhhhh. Watch a movie.
You won’t have to do this again for another three days, don’t worry.
I’m exaggerating for effect. It isn’t like this every three days. There is something to upgrade every three days, or even every day, but it’s usually small and incremental. However, if you let it slip, it becomes a nightmare, and even when you keep up with it, occasionally you still get this kind of mess. I am absolutely not exaggerating about the size of these. Recent memory contains painful un-repressed experiences with xorg and GCC, which took weeks to solve on certain machines (how much pain you experience totally varies from system to system, depending on what you’ve installed and how it’s configured). And even when this doesn’t happen, it is a royal pain to do this every few days. A constant upgrade process is just way too much work.
Upgrading breaks things
Upgrading can badly break things too frequently for my taste. For example, these are things that used to break on my desktop machine all the time:
- My atheros network card. Every time I upgraded anything to do with networking (such as my kernel, or the drivers), networking broke. I know it has to be this way, but when upgrading a kernel, it’s tough to remember everything else that has to be rebuilt against it. I usually figured it out by rebooting and noticing that the network wouldn’t come up, at which point I didn’t have a working network to fetch the latest madwifi drivers for building (oops, I should have been upgrading constantly; I’d have the .tar.gz files cached if I did).
- Samba. Any idea how many things in a typical Samba-enabled desktop system are linked against Samba? Me neither, I’m trying to forget. Upgrading a kernel means re-compiling all that stuff (if there’s a way to avoid it, I never found it, and that’s what
revdep-rebuildtold me to do — and the only thing that worked for me).
There is no way around this, as far as I know.
Even in the best case, it’s too slow
Downloading and building everything from source may promise “infinite flexibility” (more on this later), but it’s too slow. It doesn’t seem that way at first, but after a while, it gets tedious. Compiling everything is just a waste of time, for most people and most purposes. Why should everyone have to start from source code with every package? If we all compiled all our own software, we’d waste billions of hours of processor time every year. There’s something to be said for doing things once and then enjoying the benefits, and compilation is a good example, in my opinion.
Downloading the source is also usually a larger download than getting pre-compiled packages. It’s a waste of bandwidth.
Gentoo is not optimized for my hardware!
As I said above, the whole “Gentoo is optimized” train of thought doesn’t ring true for me. Here are some experiences I’ve had where the Gentoo “optimized” build ran very badly on my machine:
- Firefox. When I used an official binary package, Firefox ran like a dream. But if I did this, I couldn’t link against it for building the desktop environment, so then that desktop environment wanted to build the Mozilla suite. So one way or another, I ended up either compiling Firefox from source, or Mozilla from source. And Firefox from source ran like a dog. Even scrolling in a text field was slow. I know, I know — I read all the forum entries about locales and blah blah, and it helped, but it still ran badly. One note about desktop environments — I like to use Fluxbox or XFCE, but there are certain things that have at least some dependency on gnome-base. I don’t mind having that installed, except that it in turn depends on things… and that makes for a lot more things to upgrade.
- Any DVD playing software.
- OpenOffice.org
- MySQL server (I think — preliminary results discussed below).
These are the places where I most notice the speed difference. It beats me why packages I compile from source are slower. I bet I’ve spent a hundred hours trying to learn enough to fix these and other speed problems, but I never could.
The only thing I think might have been more responsive on Gentoo, before I switched to a binary distribution on one machine, is GTK+. On this machine now I notice a small lag in bringing up dialog boxes and so forth. (QT has always been lightning quick for me, no matter what).
Gentoo doesn’t have the latest and greatest
While Gentoo usually has more up-to-date versions of packages in its source tree than Red Hat sometimes used to in 5.1 (ah, the good old days), it doesn’t always have updated versions. I can’t think of a good example right now, except for how long it took to get XFCE 4.2 into the tree, but I know I’ve seen it — common desktop apps that are a year and a half out of date or something like that. Sorry I’m getting a bit hand-wavy here, but I’m sure if you browse Portage you’ll find something really stale, which binary distributions provide much more up-to-date. In general, I don’t think Gentoo is any better or worse than a binary distro.
Gentoo builds are sometimes really broken
Other things I’ve written about are painful, but this is worse. Slow is bad, incorrect is worse. I can’t tell you all the programs I’ve seen behave bizarrely or crash after being built from source on Gentoo (and I stress I don’t do anything exotic with USE flags or compiler optimizations). Most programs work fine, it’s true, but some don’t. Here’s a partial list:
- Kexi was completely and bizarrely broken on my Gentoo box. Even though it was supposedly version 1.6.something, dialog boxes popped up saying version 0.8, large sections of functionality simply didn’t exist or didn’t work, and everything crashed constantly. I installed a lower version number on a binary distribution and everything worked like a dream. This was really strange; it was as though the Gentoo build was mixing together source from different trees!
- MySQL Administrator and MySQL Query Browser were similarly busted beyond all recognition. Yet when I downloaded the official builds from MySQL and ran them, everything worked flawlessly. I tried this multiple times on many different machines. Why can’t the Gentoo version — built from the same source, I assume — be made to work? Granted, it was masked in portage, but why should it be if the official builds work great?
- Perl’s MySQL support broke, then unbroke, then broke… I don’t know whether it was MySQL, Perl, or the DBI or DBD libraries (or something else). I couldn’t connect with
AutoCommitdisabled; it would throw an error saying “transactions are not supported.” I found a number of places where other people complained of the same problem, but never found a solution, and recompiling endlessly never fixed it.- Update: this is not Gentoo’s fault, it’s MySQL’s fault (bug 21829). The reason I’ve never seen it elsewhere is because other distributions choose a non-broken MySQL version. In this case I shot myself in the foot by choosing a broken MySQL.
- GRUB mysteriously broke one day after an upgrade. I could never get it to work again. It would start and then have an error, which I forget now, and drop into a command-line. I had to manually type the boot commands at the GRUB command-line every time I booted. I probably spent five or six hours trying to solve this too. Again I saw many other people had the same problem, and none of the tutorials or FAQs fixed it, even when they described the problem exactly.
- The madwifi libraries were flakey at best. Half the time when I booted, the Atheros module wouldn’t find the card because the device didn’t exist. I suspected something was wrong with my hardware, but then I switched distributions. Nary a problem since then.
Experienced developers will likely see this list and see I’m confused about the source of some problem, or “oh, you just need to do X” or something. Why does it take an experienced Gentoo developer to properly configure a Gentoo machine? Have you ever noticed that’s who gives the answers to the sticky problems on the forums? Users shouldn’t have to be C programmers to know how to configure their system. Unfortunately, when you run into these issues you do, as far as I can tell — and that’s a risk.
Infinite flexibility and minimality are myths
Gentoo isn’t infinitely flexible, no matter how many USE flags there are. Here are two examples where you can’t get just some part of the required functionality.
The first is MySQL. You can’t just get the mysql client, or C libraries, which means you have to download and build the entire server if anything on your system needs MySQL support. For example, suppose you’re setting up a web server, which will run Perl web apps that talk to other servers running MySQL. You can’t just build Perl with MySQL support. You build dev-perl/DBD-mysql, and it depends on dev-db/mysql. Look at the ebuild:
DEPEND="dev-lang/perl dev-perl/DBI dev-db/mysql"
If you want anything to do with MySQL, you just installed a MySQL server, even if you wanted your web servers to be “minimal.” Other distributions let you get just the client programs or C libraries.
The next example is Samba. It’s much the same — you can’t just get Samba client libraries and programs, you install a Samba server too.
Even USE flags don’t give you full flexibility. For example, mytop will use ANSI color codes and hi-res time if there’s support for them, but it doesn’t really need them. However, they are listed as dependencies in Portage, so even though they’re optional for mytop, they’re not optional in Gentoo. In theory you could add USE flags for that, but in practice, you don’t get the choice. In theory, Gentoo is infinitely flexible, but in practice it’s not. This should not be a surprise.
I’m sure there are other examples, but I don’t want to get into it too much. I don’t mean to put down the developers’ hard work, but in these cases this just doesn’t do what I want. I want a system that can talk to MySQL or Samba servers, without being one itself, and without having to compile all that extra code.
It’s reinventing the wheel
As a distribution, Gentoo has been and continues to be great. I can’t thank the Gentoo people enough. But I personally don’t want to run any system that I can’t just install and run, with a modicum of knowledge and reasonable installation/configuration time. I’m not a “protect me from complexity because it scares me” person. I’m a “don’t make me do all this work, my time is scarce” person. I do not want to solve the same problems other people are solving, over and over. I want a distribution that solves the problems and gives me the result. If you don’t think Gentoo users reinvent the wheel thousands of times a day, go read the forums!
Package staleness vs. flexibility vs. stable release cycles
Gentoo is all about choice and flexibility. While that is true, as I said infinite flexibility isn’t a reality. What you get instead is the ability to choose exactly which versions of packages you want (theoretically).
For example, if I use a binary distribution with regular “snapshot” releases and only security upgrades, I might be limited to versions 1, 2 and 3 of certain packages. Maybe I want those packages to be versions 2, 2 and 3, but when I upgrade to a newer release of my binary distribution, I can only get versions 2, 3 and 3. Gentoo will likely let me install exactly the versions I want.
That’s the upside of being able to mix and match versions of software. The downside is “infinite flexibility” really means much greater system complexity, when you consider the number of combinations of packages, versions, and interdependencies between them.
Binary or “snapshot” releases are much simpler and the packages can be much better tested with each other. Simple math shows this: if you have N packages in a release snapshot, you have a graph with potentially (N-1)^2 edges. If you have a distribution that lets you install many versions of a given package, say 5 on average (I think this is about right for Gentoo), you have (5(N-1))^2 edges in the graph — 25 times as many. In real numbers, Gentoo currently has 11453 packages in the Portage tree. A hypothetical snapshot release would have about 131 million package inter-relationships, while Gentoo with an average of 5 versions per package would have about 3.27 billion inter-relationships.
In practice, most of these packages aren’t installed and don’t actually interact with each other, so the number of relationships to test and integrate is much smaller, but you can see how the combinatorial explosion makes it much less likely that any given set of packages has really been installed widely and tested together widely.
Combine this with the fact that most binary distro users are using basically the same system, starting from the same installation CD, and you get an idea how much less variability there is in, say, people running the stable version of Ubuntu. A person who installs Gentoo, doesn’t tweak USE flags or other settings at all, syncs her Portage tree, and runs emerge -uD world probably ends up with an entirely unique system, different from any other system anywhere in the world. Just like a snowflake. That’s bad, not good.
Gentoo is like a continuously variable function, whereas a snapshot-based distribution is like a step function.
There’s something else I haven’t added into the mix: USE flags and compiler and CPU settings. I doubt your web browser has enough zeroes to express the combinatorial explosion that causes.
For this reason, I feel far less comfortable with building everything from source and continuously upgrading, than accepting something that’s a few months old but better tested.
This isn’t just theory. Many packages won’t even build unless you get everything just right. For example, a lot of “package won’t compile” forum entries are solved with answers like “you can’t build that with GCC 4.1, you have to upgrade to 4.1.1.” Who knew?
While we’re on the subject…
Certain upgrades can be either difficult or impossible, depending on how much software you have installed. When the most recent “large” GCC upgrade appeared, I tried to put off upgrading my home desktop machine to it because I saw forum threads from people having huge problems. I thought maybe these would get ironed out over time. But then I needed to upgrade or install something myself, and it wouldn’t build with my current GCC version. The only option was to upgrade GCC, I think to 4.1.1.
“Nightmare” doesn’t really capture the experience. Everything broke, and I mean everything. It would have been far easier to just re-install the system from scratch — except that when you install your system, you get whatever GCC is on the CD, so you still have to complete the upgrade. When you go through one of these upgrades, you have to re-compile your entire system, all at once in the proper order. If it breaks partway through — which it will unless you’ve got a really minimal system (e.g. not a desktop machine) — you have to restart, because the dependencies and order of compilation really matters. And there’s no facility for only re-compiling the things that aren’t already re-compiled. After about three or four cycles of emerge --emptytree world, watch it break after eight or ten hours, read the forums and try something, restart only to have the same package break again, it was just too much for me. I switched to Ubuntu.
I’ve had a good bit of experience with other people’s installations too. I know of several servers at various sites that have bumped up against that GCC upgrade cliff. I have only seen it solved by rebuilding the machine from scratch, and upgrading the system before adding non-base-system packages in. I have seen servers remain in the queue a long time for that, as they can’t always be taken down and rebuilt at will. If you try to put it off, every time you try to change anything you’ll likely find its Portage tree is out of sync and refers to packages that don’t exist anymore, so you have to re-sync Portage, and then tons of stuff wants to be upgraded as a dependency. Sometimes I’ve seen machines get so stale they want to upgrade hundreds of packages, but that can’t happen or everything would bust. This is an ugly reality of life with Gentoo, as far as I’m concerned.
If you’re running a desktop machine on Gentoo, there are other painful upgrades too. My main home desktop machine somehow had no problems upgrading xorg to the modular distribution, but on another machine, I basically had to nuke xorg and install it from scratch — and then the modular builds failed, because they were really buggy and didn’t properly list all dependencies, etc (and when there are over 300 modular xorg packages alone to build, this is pretty ugly too). So everything failed and I didn’t have a graphical desktop. I’m okay working in that environment — I spend most of my day at a terminal anyway — but it’s nice to have a web browser. I could not resolve this, and spending days reading forums was simply not an option. I may have had that luxury in the past when I wasn’t really busy on other things (ha, ha), but not anymore. I ended up upgrading that machine to Ubuntu as well.
Configuration management is tough
It’s really tough to get all your machines uniform. There are ways around it, but you still have to expend that effort, and it can be significant. I know of people who set up a build machine that will host binaries for other machines, such as white-box workhorses. This machine is a proxy for the others, and they sync up with it. You can also use g-cpan to manage Perl code in a separate Portage overlay, so you’ll have uniform Perl module versions too. But this is a fair amount of work no matter what, at least with the tools I know about.
That might only be feasible with cheap white-box workhorses. What if you need this same process for really expensive machines? Can you spare a really expensive machine big-iron server just to act as a build host? Even if you do, do all your machines have the same hardware? That’s kinda doubtful. Virtual machines are only a partial solution — a virtual machine isn’t the same as the server it runs on, and installing a virtual machine by definition makes the server different from the others in its “class.”
You might try documenting your install process. I guess you could get machines reasonably close to identical, as long as you have a single Portage tree you’re syncing against. (Whatever you do, don’t sync against the main Gentoo mirrors, or you’re finished before you start. It changes constantly.) I’ve personally had mixed success with this type of process. I don’t think it works as well as it might. There are still too many places to mess up, too many things that introduce variability between machines.
There are ways to deal with all this, but they require extra work, and are basically imposing a “snapshot release cycle” on top of Portage’s stream-of-consciousness continuous change, which is a wheel that’s been invented in almost every other distro out there. I think Gentoo is enough work already.
When you have a problem, where is the problem?
When you’ve compiled everything from source, with your own USE flags, linked against your own libraries etc, you have a version of the software that’s different from anyone else’s anywhere else in the world. And when you have problems with that, you don’t know what the problem is. Is it a bug, or is it just your system?
You can’t always know. Based on my other experiences with things being broken, I am not prepared to say any Gentoo build isn’t subtly buggered.
For example, I’ve seen a business have trouble with MySQL replication. Is it MySQL, is it the way they’ve been doing replication, is it the setup and configuration, or is it that they have a broken MySQL build? I don’t know. They are not executing the same bits as anyone else. I don’t even feel comfortable submitting certain of their bugs as bug reports to MySQL because of this.
I suggest using the official MySQL builds
One site using Gentoo recently decided to download MySQL’s official build and untar it instead of building it from Gentoo’s source tree. Preliminary results indicate this was a wise decision. For one thing, the official build seems to be getting dramatically higher throughput. There were queries that ran in 90+ minutes on Gentoo’s build, and after switching to the official build, ran in 20 minutes (granted, they also went from 5.0.21-log to 5.0.26-standard, but that isn’t the cause of the increased performance!). This is not the only indicator of better performance, as the disk activity increased significantly (by about 2x) on the official build too. That might indicate the CPU is being used much more efficiently. In fact, the official build seems to saturate the I/O, so the disk is now the bottleneck, which is as it should be in that site’s workload.
Regardless of whether you run Gentoo or a binary distribution, I generally advocate running the same bits as other MySQL users. I’ve chatted with dozens of people on IRC who are having trouble with broken builds from third parties, not just Gentoo users. The official builds are tested by thousands of people already, so the bits are much more likely to be good. And if there’s a problem, it makes it dramatically easier to troubleshoot and/or give good bug reports.
This has its downsides too. Getting other things in Portage to link against MySQL libraries, etc etc, without installing the MySQL package from Portage, can be non-trivial. You probably don’t want to install both, or you’ll never know what your programs are linked against.
Conclusion
That’s a heck of a brain dump, but hopefully you can sort through it and glean something from it. I would summarize by saying any source-based, continuous-upgrade distribution is less likely to be well-tested and widely verified, the complexity is much higher, and it is probably much slower and more difficult to install, maintain and upgrade, and keep everything at the same configuration.
If you’re using Gentoo for a MySQL server, I would at a minimum install the official builds, not build from source. I don’t want to say what distribution to use for a MySQL server, but my personal experience has made me want to stay away from Gentoo.
But that’s just my opinion :-)
I love Gentoo and I hate Gentoo for the same reasons. I used it at my last job for small projects (Solaris was the only *acceptable* option for serious applications). We’ve been running kruckenberg.com on it for many years and have experienced much of what you suggest above.
At first we let it lag because it was too much to keep it up do date, and then it got hacked. After rebuilding we started to do a once-a-week emerge process. That turned into a weekly outage.
We’ve been saved from this because we had to move to a shared hosting solution so the Gentoo box has become more of a place to experiment.
In the end I think it’s a god experience for anyone who wants to dig a little deeper into what is involved in running a Linux system. Esp a stage 1 where you do everything by hand. Having to think about how and what gets compiled was an eye opener for me. The emerge is nice in small increments. When the list gets to 100+ packages that need updated it becomes to much of a pain.
I’ve starting using Ubuntu for now – every now and then you need a period of time where your OS works with minimal involvement.
Mike Kruckenberg
22 Nov 06 at 10:35 am
So the question remains, what distribution do you use now ? :)
Arnaud
22 Nov 06 at 10:56 am
I use Ubuntu on my personal machines, and at the moment I’m not running my own database servers for anything. I don’t know what I’d use for a MySQL server. I might give Ubuntu server a try, or Debian; but I’d still install the official MySQL build.
What about you? I’m actually interested to hear what people have good luck with — for servers in general, not just DB servers. Some of my clients use Red Hat and are happy.
Xaprb
22 Nov 06 at 11:13 am
Much of this post is about running Gentoo on the desktop and not MySQL related. I really wish you had made that more clear at the beginning. I had the same issues with Gentoo on the desktop and therefore switched to the best maintained Unix OS, Mac OS X. =)
However, for servers, I have yet to find anything as nice as Gentoo for management. One day of RPM hell rules out any RH solution. Debian is running 1999 code for some daemons (exageration). Ubuntu is focused on the desktop more than server. I guess a BSD might work, but I like Linux.
I do find your comments on MySQL binaries interesting though. We have had an issue with Gentoo built MySQL restarting every couple of weeks. I wonder if we would see the same issue with the binary. hmmm.
Brian Moon
22 Nov 06 at 1:07 pm
I agree with a lot of your article, but I don’t understand why you are upgrading everything all the time for the sake of upgrading it. If it ain’t broke why fix it?
I actually like Gentoo on the server side as I can build exactly which combination of versions of software I require. The thing I’ve never really understood about Gentoo is the ‘emerge world’ option, as it seems to contradict why you would want to use Gentoo in the first place.
If you do require this ‘install updates’ functionality – yup, you’re better off with another distro. I’m just bitter as once upon a time one of my Red Hat updates overwrote my website with a ‘congratulations, you’ve installed apache!’ page.
A good friend of mine is very happy with Ubuntu and it seems to be installing updates all the time without breaking his system. He hasn’t converted me yet though :)
senyahnoj
22 Nov 06 at 1:28 pm
RPM hell is definitely a less enjoyable experience (or was last time I tried) than Ubuntu. For some reason, dependencies are just not handled well. You have version 5.1, want to upgrade to version 5.2, but it depends on something that depends on something… and all the dependencies have explicit version requirements in them — and suddenly you can’t even remember which package you were trying to upgrade anymore. Ubuntu doesn’t seem to get its panties in a wedge about upgrading a package here or there.
Xaprb
22 Nov 06 at 1:46 pm
I run gentoo on a desktop, a laptop and on a server of mine. I agree with most of what you say so I try to sync regularly to avoid most of the pains. There are still a lot of things that break and a lot of packages that get installed when you upgrade. I do not use it on servers at work, our sysadmins run debian.
Arnaud
22 Nov 06 at 5:22 pm
As a gentoo developer, and at one point the only developer maintaining the mysql package, I suspect that your system has deeper troubles.
You CAN get just the mysql client and libraries by building mysql with USE=minimal. There’s similar development for samba, but the maintainer of the samba package seems somewhat AWOL lately.
If there’s one problem that is visible in Gentoo presently, it’s that some previously well established packages are bit-rotting, because their maintainers are too busy. mysql administrator and query browser are a good example of this. I don’t personally use them, so I hadn’t noticed how bad they’d gotten, so I’ll try to get somebody to follow up those.
For your note on QT-3/QT-4 both getting installed, if a package declares that it can be built with both, and you only have qt-3 on your system, you’re going to get qt-4 in an upgrade now, because the system seems it as such. qt-3 is there for packages not supporting qt-4 and if you are certain those are gone, you can remove it.
For your general complaints that Gentoo is not optimized, and every thing else. Have you filed a single bug on these items? In my daily use of firefox, openoffice and mysql (all day, every day since I work from home), I have not seen anything like this.
One thing I have seen with Gentoo, is that if you something even marginally flakey with your hardware, while 3rd-party binaries run perfectly fine, Gentoo-compiled versions trip up very often, even with conservative compile settings – maybe something with libc instead. Why this is, I’ve never figured out – but I have found it to be very good to throughly test hardware before putting it into production use (400+ boxes of Gentoo and counting).
Do some gentoo ebuilds have problems? Yes, most definitely some ebuilds have problems. But go and compare how many packages you have problems with to how many packages you have installed, and you’ll find that 90% of the bugs come from 10% of the packages.
On configuration management, see my recent post to -dev (and in a recent GWN) about my methods of managing large numbers of machines, by having a non-dedicated box that took on a build role (using catalyst and chroots). It’s not perfectly, but it does make things considerably easier. The break-even point (at which maintaining your own snapshot of the portage tree is worthwhile) is unfortunately very high at the moment, and having a way to lower it would aid ‘enterprise’ deployments.
Is Gentoo easy to break? Far too easy unfortunately. And it gets very hard to fix breakages for most users. The GCC wall is an example of this. As a developer, I managed to break at least one system in the GCC2.95 to GCC3 change, and then the GCC3 to GCC4 change as well, mainly while experimenting with a way to do it while NOT breaking my system.
However I think a contributing factor on breaking things, is that most upstream source packages are even more vulnerable to breakage – just look at the number of ebuilds that fix up automake/autoconf breakages just to ensure that packages will build reliably – let along actually work correctly.
For your broken MySQL, did you use FEATURES=test emerge mysql? Did it pass completely? My dayjob has a reasonably sized MySQL setup that is extremely heavily used (Our Alexa traffic ranking is in the top 370 presently) – we’ve had our fair share of MySQL glitches – because of faulty hardware, insufficient hardware (found via analysis into bottlenecks), and upstream source screwups.
I think I’ve covered all of your arguments here, but do mention if I’ve missed some.
Robin H Johnson
23 Nov 06 at 7:47 am
I’ve been using SuSE, Fedora and Ubuntu for desktops and Debian for servers, before switching to Gentoo some years ago.
Main reasons: RPM-based systems seem to be unable to resolve dependencies and if your distribution does not provide an rpm for [somepackage], you’ll have to find some in [anyunofficialrpmvendor] to get it installed (unless you want to mess around with compiling — and especially removing if you decide not to need the software once).
Debian is fine for running servers, unless you need some striking new feature (‘new’ in a Debian way of thinking about time). For example, I had to build a redundant database system, providing failover (that was at the beginning of mysql cluster — don’t know if it can do now, but at that time it could only hold the tables in memory which was not enough. so I had to use postgres+slony). Turns out that slony is somewhere in testing, so I first had to upgrade to testing and the experience that slony needed postgres 8 even though it worked with postgres 7. So do what? Upgrade an existing, productional database by a major version? I figured I’d have to, but believe me, it wasn’t fun (and having postgres 7 AND 8 installed after that messed up things even more).
Ubuntu is a nice distro and I used for quite a long time on my desktops, but even though the Ubuntu team provides more up2date packages than Debian, apt still is a very old system (means I hate searching for a package, just to find it in universe).
So, I switched to Gentoo staying with it since than. Of course, updating takes time, BUT that just depends on your update-cycle. Currently I update my Desktops in daily to weekly cycles and the servers about once a month.
And, of course you pray that gcc/glibc/… wasn’t updated again (especially since portage considers IUSE when you do an –newuse), but that’s an overnight compile after which the world is alright again.
thpani
25 Nov 06 at 11:16 am
Thanks everyone for writing in. There’s a lot to respond to, but I’ll try.
senyahnoj, I didn’t respond to your question about why upgrade all the time. Maybe it does work better to let things be until you need to upgrade them, I’m not sure, but it seemed to me that this tactic just delayed the inevitable, and made it much more painful when it was forced.
Robin, thanks for writing in. I appreciate the developer perspective.
Re: building just the mysql client with
USE=minimal, that’s not documented, and quite counter-intuitive! It’s nice to know that it’s possible, but the only reference to ‘minimal’ I see in /usr/portage/profiles/use.desc is “minimal – Install a very minimal build (disables, for example, plugins, fonts, most drivers, non-critical features).” There’s nothing mysql-specific in /usr/portage/profiles/use.local.desc. I think this goes back to “the developers always have an answer to the sticky problems.” I wouldn’t think “minimal” is the way to avoid building a server, as the USE flag’s description says it just avoids non-critical features and drivers. So I build it with minimal and it a) avoids the server (which some consider critical) and b) installs the drivers? That’s just the reverse of what the USE flag seems to say it should do. If I had to guess, I’d say it would install only the server and omit the client/library stuff. I would appreciate a “clientonly” USE flag, or separate builds for mysql-client and mysql-lib. I’m sure the latter is much more work, and probably not a good idea.On QT3/4, I was playing with Kexi, and couldn’t get rid of QT3 because (as I recall) it explicitly wanted 3, but then 3 wanted to upgrade to 4 as mentioned. It definitely wanted to force both to install. Again, maybe I just don’t know the magic word to make it go away.
I wasn’t aware that what I perceived as non-optimized builds should be filed as bugs. I assumed it was my fault, that I just didn’t know what to do to make them as fast as they should be. Since I’ve been on both sides of the bug-reporting fence, I’d feel guilty filing a bug that’s just me doing something stupid. I say that because I try to put myself in the Gentoo developer’s shoes and see all these bug reports from people who just need help configuring their systems — those aren’t bugs. I always went to the forums, because I thought that was the appropriate place to say “hey, this is slow, what’s wrong?”
It sounds like you’re actually cautioning against using any hardware-specific optimizations, which I thought was one of the main reasons many people gravitate to Gentoo? (Granted, there are always the Gentoo Ricers, whom I can’t take seriously either).
Re “90% of the bugs come from 10% of the packages” — I’d always assume this to be true in general, so you and I are in complete agreement here.
Thanks for mentioning your work with catalyst. For those who are curious, here’s the GWN referenced, and here’s the -dev thread. I’ll have to read up on this more.
I’ll look into getting the ebuild to run the MySQL tests and see if they pass completely, too.
thpani: “that’s an overnight compile after which the world is alright again.” I’d argue that an overnight compile automatically means the world is broken :-) What Robin has mentioned would at least take the compile off the machine in question, so that’d be more workable, but goodness — I don’t want to, and many businesses just can’t — tell all their machines to spend half a day compiling. Especially if the machine is somewhat heavily used and needs to stay responsive. Even more especially since you don’t know in advance whether it’ll even succeed.
I don’t see completely eye-to-eye with everyone who’s written in here, but I assumed this would be the case when I wrote the article. I’m glad I got a chance to hear how other people do things with Gentoo, and some new options when I do use it.
One postscript I want to write to the article: I focused on reasons why I don’t use Gentoo anymore myself. That’s automatically a lopsided article. There were and are lots of things I like about Gentoo, too. I’m not going to rewrite the article to be more balanced, but I just want to mention it here.
Xaprb
26 Nov 06 at 9:52 am
I have been using Gentoo for years now because someone I know also used it, and I thought I could get help from that person if I used the same distribution. I was right, to some extent. Once I got into it and learned some of the ways of the distribution and working with GNU/Linux, I pretty much stayed with it because it was what I knew.
I had tried Mandrake, Redhat, and some others, but it wasn’t until Xubuntu 6.10 that I saw a distro that I really liked and thought I could move over to. I’m still using Gentoo on my desktop, but I’ve started using Xubuntu on my laptop, and I also changed it over on my girlfriend’s computer. The reason? I don’t have time to constantly update stuff (and I don’t know how to optimize very well).
I would say there are two main areas that are awesome about Gentoo though; learning how to install and configure Gentoo forces you to learn a lot of other things, and I’m happy I did that. Second, the forums and people there are really nice, and helpful.
Nathaniel
27 Nov 06 at 12:21 am
Gosh. I just don’t know how I could agree with you more. Admining your gentoo box can be such a pole sized pain in the arse. If you can’t leave it to compile updates over nights, forget it. Did someone mention that the build process is quite likely doomed to fail. Even if you run stable only system.
On the other hand there is something in Gentoo that keeps me sticking with it. Sure, it leaves *a lot* for the user to handle. On the other hand, you google “gentoo <task-to-do>” and most likely you’ll receive a link to Gentoo’s wiki. In a while you’ll have the task done.
There is also something that simply makes your gentoo box feel personal. And I love the feeling.
Jussi
27 Nov 06 at 5:46 pm
THANK YOU FOR YOUR ARTICLE!
I am a very happy Slackware user who is actually completely satisfied with this distro, but purely out of curiousity have always been tempted to try out Gentoo. I was very apprehensive that living in Gentoo _might_ end up being the nightmare you describe and while I have been reading similar comments about Gentoo before, I was thinking maybe the Gentoo fans who dismissed such posts still knew what they were talking about.
Your blog post was the nail in the coffin though… Like you and so many others out there, I’m a “don’t make me do all this work, my time is scarce person” and can barely afford to get sucked into configuration timesinks (something which anyone dabbling in Linux is all too familiar with). While such tweaking does bring with it the control, power and knowledge that again, anyone dabbling in Linux is well aware off, often times it is just too overwhelming.
So why, someone might ask, do I use Slackware which is viewed as “outdated” and does not even have automated dependency checking? The truth is that Slackware does keep pace with the latest packages, figuring dependencies out yourself is kind of fun and *very* educational, and once you are over the learning curve, actually getting things done in Slackware is a real breeze. Another side benefit of the Slackware learning curve is that majority of the stuff you need to learn are not Slackware specific but are actually how all Linux systems work under the hood. (For example, Slackware does not include a firewall front-end, you have to learn to set up iptables yourself, but such knowledge can be applied to ANY distro out there, can’t say the same for the intricate portage system, for example).
Here are my tips for Slackware heaven:
* Don’t be anal, don’t prune by the package, use rough grained Slackware package series installs. This means a lot less headaches. I usually just do a full install. The 3-4 GB it takes up is not such a liability and the truth is that the full install make little difference in the daemons that end up running. A ps -ax will confirm that it’s still mostly the same small set that gives Slackware its reputation for good security. The other daemons will be in a quiescent state, waiting for easy enabling by just chmod a-x’ing their respective rc scripts. You will also now have most of the recent versions of important, widely used libraries already in your system such that if you install something new, the number of unsatisfied dependencies is greatly lessened.
* If you want an optimized binary compile of a package or need something that is not included with Slackware, make your own packages from .tar.gz source. Like a lot of things in Slackware, it was hard to figure out at first, but actually quite easy if you have someone to show you how. I wrote a package
tutorial here. Nowadays, 99.9% of ./configure scripts can be expected to work correctly, so it’s not painful at all to get the vanilla .tar.gz’s and compile from source into your own binary package. Plus there’s always linuxpackages.net if you’re lazy.
* For the seasoned Linux user, a package’s required libraries are easily figured out from its README. I find that even for stuff that’s fairly specialized and not included with Slackware, you usually just have to get just a couple more additional libraries to get things working the way you want.
Andy
28 Nov 06 at 12:09 am
> waiting for easy enabling by just chmod a-x’ing their respective rc scripts.
by the way, that should have been “chmod ug+x’-ing” the respective rc scripts (to enable daemons to run)…
Andy
29 Nov 06 at 5:00 am
Looks like the ‘minimal’ USE flag for MySQL disables some critical functionality, so it’s not just ‘client-only’ after all. I still can’t install a client-only build.
Xaprb
6 Dec 06 at 4:01 pm
from http://bugs.gentoo.org/show_bug.cgi?id=157362
In reply to:
Gentoo developers ALWAYS cautions against using dangerous CFLAGS, they generally bring low speed enhancement and big pain. We simply leave you the ability to do so but then you need to take care of the result.
There are a whole world of things that gentoo make faster, do you need to apply a patch, build with some thing more, or in the case of mysql add a storage engine? You simply copy the original in an overlay, create digests and re-emerge, it’s question of minutes, ever tryed to rebuild apache + php in a whatever binary distro? then you know what I’m saying.
By the way I’ve switched from suse to gentoo because when I needed to install MySQL on a dual opteron 246 it was the only distro that permitted me to do that, others were plain too instable.
Going back on fixing bugs now.
Francesco Riosa
9 Dec 06 at 9:44 pm
I think any OS needs and should be re-installed after a few years. Specially Windows! After installing gentoo soooo many times on different machines from past 6 yrs…the one thing I get is -Instant satisfaction …just because I feel I know what is on my System. I had a G3 ibook 12″ and had every thing working…webconferencing, wireless, cisco vpn, special configs to lengthen battery life, xdmcp, I was so happy that I did not touch my ibook for a year (before I had to sell it :( ). And that was the best experience I ever had with computing. The speed was ‘wayyy’ faster than os X.
As one of the reader earlier posted…why would you want to fix it if it aint broken? To get all the packages you want ‘will’ take several days but once you have things working…updating few things here n there/ maintaining it is a dream… Until you decide after 3 yrs that you want to do a clean install…which you should do…even if your system was perfectly maintained
Stating the above … I spent my whole weekend setting up MythTV , Raid and fresh gentoo installs on my 2 AMDs
Gaurav
22 Jan 07 at 5:29 am
The post is about personal experience with Gentoo. It is a perfect post, since the type of the person, his experience, and his reactions to that experience are crystal clear. Unfortunately, the assumptions the author based his reactions on are incorrect.
Gentoo is a meta-distribution. That is, it is NOT a distribution. When somebody installs Gentoo, emerges some packages, runs some apps and gets the experience, he or she is not dealing with the Gentoo Linux distribution. They are dealing with the distribution they just created for themselves from Gentoo.
I see only 2 issues in the post that are truly Gentoo related:
How to install exactly what I want from MySQL? If Gentoo does not provide exactly what you want, then you are likely the first one who wants that. Just create your own ebuilt in an overlay and submit it to Gentoo. If you cannot, submit a bug. It may be a good idea to split MySQL into client, server, and admin ebuilds, maybe not.
Properly compiled packages are slower than binary packages from the vendor. It is VERY interesting. The binaries have been compiled from different source, or there are some hidden defines, or a much better compiler than GCC has been used. You should have filed a bug or, better, ask MySQL how you should compile their source!
All the rest is just the “I found I need help to figure out how to do … properly” stuff. The author assumes he is using a distribution that should iron out all the issues for him. If it were the case, these complaints would be justified. Unfortunately, he is building a distribution himself and should compare his experience to that of the Red Hat or Mandriva development teams, not their end users. Gentoo allows an individual to compete against large companies and I find that amazing, but there is a price to pay.
The “stable release cycles” complaint deserves separate treatment. Being a meta distribution, Gentoo does not need any release cycles, period. It is actually a request for another distribution, presumably Gentoo based, that will have release cycles.
My point is that there are only 2 reasons to use Gentoo: the need to know what exactly is in the system, and the need and ABILITY to build a distribution that suits you better then any other. Any Gentoo reviewer should have that in mind.
Now, what are my thoughts about “using it for MySQL servers”? Use if you need it. Use if you can. You will get better mileage if you cooperate with other people who want “MySQL servers” and derive your specialized distribution from Gentoo.
Andrei Gerasimenko
24 Jan 07 at 7:42 am
Andrei, well spoken, and thanks very much for your thoughts. I have actually come around to something like this point of view since writing this article. Once I began to think of Gentoo as a meta-distribution, many of the things I mentioned began to make sense. I still don’t use it myself, because — not surprisingly — I don’t want to maintain my own distribution! That comes back to the “I don’t want to solve the problems a distribution solves” point of my article.
I am planning to benchmark MySQL so I can figure out how to properly compile it. All I need to do is set up a non-production machine for that.
Xaprb
24 Jan 07 at 8:17 am
When I started looking around for other distro’s I ran across archlinux (http://archlinux.com). The major selling points for me is its package management application (called Pacman). This little app allows me to start from scratch and configure a new box in a 1/2 hour or less. I can ensure that all my boxes are the same.
Upgrading is a breeze. The system is optimized for !686 hardware and will not run on anything older than a Pentium III – but that’s fine for my needs.
If you looking for a new disto, you should check this one out.
Chris….
Chris
15 Feb 07 at 5:08 pm
i completely agree with the author.
gentoo takes just too much time to maintain and keep up-to-date, especially if you have other things to do. i’m using linux for software development and simply can’t afford to waste hours updating/fixing the system every week.
if something breaks and you need to reinstall everything from scratch, that might take hours if not days – now imagine you have something urgent to do…
alex
alex
4 Nov 07 at 10:58 am
I have to disagree with the author on many points but basically it boils down to choice. I run many, many Linuxs of various flavours but my personal favourite is a penguin from the Falkland Islands.
If MySQL runs like shit for him on Gentoo then there is a reason why! As someone who writes books on the subject, I’d hope he’d have the knowledge to publish bug reports – I’m sure they’d be received well and we’d all benefit.
I don’t know if it is relavent but I use MySQL for mail logging via rsyslog. On one of my customer’s site they are the victim of rather a lot of attempts to relay. This means that rsyslog ends up doing an INSERT/UPDATE on a 20M record database several times per second. The box is a small Dell system with RAID 1 and 2Gb RAM. It does not even sweat over this. I accept that I’m not a proper DBA but this is pretty impressive to me.
Mind you what is really impressive is Exim dealing with it – it has to spawn kids for each connection and do the various DNS look ups before rsyslog gets into the logging. … and there are quite a few connections!
Jon Gerdes
14 May 10 at 8:43 pm
Hi,
your explanation is actually aweasome.
It tried gentoo and recommened it when i used to rule with it.
After day and another day compiling gentoo packages one day someone talked to me about debian.
Gentoo good: strong
bad: 2 much time compiling (even long days doing it)
ps: ruby rulz, sql alone is nothing, it must be backgrounded.
That is why i debian’ize myself
So debian rulz! and gentoo for certainly kind of projects.
alexrio4
20 May 12 at 5:24 am