The unexpected consequences of SELinux
I’ve been working with a client recently who has SELinux on his servers. It has been quite a struggle sometimes.
My colleages tell me that SELinux has a pretty noticeable performance impact. I am not sure if we have benchmarks to support this; at any rate, the client said it’s OK, we’ll take the performance hit.
There have been a few problems (Nagios can’t run because it can’t write to its own pid file, etc etc). Last night there was something new: “too many connections” when trying to connect to MySQL. As a result the site was down, and it wasn’t possible to log in to MySQL and see why. But the oddest thing happened: mysqld would not shut down. It was sitting basically idle, doing absolutely nothing, and wouldn’t stop. There was nothing in any log files to show what might be going on.
So I tried some standard tricks of the trade to see what it was doing, starting with strace. No go. It wouldn’t show me anything. What about gdb?
No, there was an error about something failing, do I want to continue, yes I do, but then I don’t get any information when I ask for stack traces or anything else.
Finally I resorted to kill-dash-nine. Then I was able to restart, log in quickly before the server bogged down, and watch the processlist grow full of unauthenticated connections. Ah hah! It’s a networking problem. The old skip-name-resolve problem reared its ugly head. It was an easy fix.
And why couldn’t I get any information about this? Why did I have to kill the server? Well… here’s the messages from /var/log/messages:
SELinux is preventing strace (mysqld_t) "signal" to <Unknown> (unconfined_t). SELinux is preventing gdb (mysqld_t) "signal" to <Unknown> (unconfined_t).
Sigh. I’m root on this machine, but I can’t do anything with mysqld — not because I’m restricted, but because mysqld is restricted.
setenforce permissive
watch logs for deny messages from selinux
update selinux ruleset
setenforce enforcing
Antonio
16 Oct 08 at 5:00 pm
If this is running on Fedora 9 or 10 you can simply put the mysql daemon in permissive mode
# semanage permissive -a mysqld_t
(semanage permissive -d mysqld_t to remove )
Or you can put the machine in permissive mode
# setenforce 0
(setenforce 1 to put back in enforcing)
Or you can run mysql without using the initrc script so it will not transition to mysqld_t, running mysqld directory from unconfined_t will stay in the current mode.
dan walsh
16 Oct 08 at 5:43 pm
Thanks very much for the suggestions. If I had seen that the issue was selinux *before* I killed and restarted mysql I would have known to do this, but I didn’t think the failures of strace and gdb were selinux — it did not occur to me, since they occasionally fail anyway to give the desired information. I only saw that later when I checked /var/log/messages again.
Xaprb
16 Oct 08 at 7:24 pm
@dan walsh – I suggested putting selinux into permissive mode, tuning the ruleset after starting, excericising, and shutting mysql down, figuring out what rules need to be added, then putting selinux back into enforcing mode. Just putting mysql into permissive mode in perpituity negates the security benefits of selinux WRT mysql. In the case of RHEL or CentOS – the native mysql packages should come with a ruleset designed/written by the mysql developers. Probably not the case with rpms provided directly from mysql. Should be able to borrow the selinux rules and with some slight tweaks port it over though.
Antonio
16 Oct 08 at 7:35 pm
Ye SElinux is very finicky.
Arjen Lentz
16 Oct 08 at 7:44 pm