Secure, easy Cacti graphing without SNMP
Cacti is a great tool for collecting information about systems and graphing it. However, it likes to use SNMP, and SNMP is often not desirable. Instead, I often see the need for a method that is:
- Secure. Use trusted, well-known, encrypted communication. Do not open up new ports.
- Zero install on the monitored system.
- As little installation or modification on the monitoring system as possible.
Over the last several years, I’ve slowly created more and more software to create Cacti graphs via standard POSIX command-line utilities over SSH with key-pair authentication. (I’ve also created similar software for Nagios, but that’s another matter.) The major problem with the work I’ve done is that it’s totally un-publicized.
The system works by passing command-line arguments to a local PHP script like any other Cacti script. This script then executes a remote SSH command, such as ssh somehost uptime and extracts statistics from the result.
The requirements are very simple. On the system to be monitored, a public key must be installed. On the monitoring system, the cacti user account must have a private SSH key that it can read and others can’t. This is standard for any SSH key. The cacti user account must also have the SSH key fingerprint of the monitored system in its known_hosts file.
The work I’ve done thus far is available from Subversion in the mysql-cacti-templates project. This project has the scaffolding for creating Cacti templates easily, so I’m using it.
At this point, the following are available:
- Apache
- Operating system (CPU, memory, load average, etc)
- Memcached
- Nginx
Unfinished work includes network, disk, etc. The techniques to monitor something of which there is a variable number (e.g. there can be many disks, each of which needs its own graph) are a little more complex than simple things like monitoring overall CPU usage. So that’s a work in progress. Once it’s done, it’ll make it really easy to discover and monitor multiples of anything — for example, multiple MySQL servers or memcached servers on a single host — without creating a new host for each resource to monitor.
As with the MySQL templates I created, these templates are comprehensive and have lots of nice properties most templates lack. This is something you get free with my scaffolding. If you’ve ever created templates by hand through the web interface, you should give my work a try. You can turn a three-day project into a few minutes and avoid bugs and other hassles. There are instructions for creating Cacti templates on the project wiki.
BTW, did you get a look on dim_STAT?
http://dimitrik.free.fr – it’s already doing all stuff you described and even more :-) integration of any new stats as add-ons takes usually 5 min. All data are kept in MySQL database. You may analyze a single host as well several hosts on the same time, etc.. Tool has over 10 years maturity :-)
New v.8.3 (coming soon) has also pre-integrated InnoDB and generic MySQL monitoring add-ons.
Rgds,
-Dimitri
Dimitri
26 Apr 09 at 6:07 am
Thanks Baron for your hard work on the mysql-cacti-templates. I used them daily and have also also base new templates base on your work.
Now have two pretty templates to check my application internals.
Ernesto Vargas
26 Apr 09 at 1:18 pm
Baron, your mysql cacti templates are really useful. Establishing an ssh connection seems overkill for systems monitoring — this is for what SNMP was expressly designed. SNMP is quite secure with (more so with v3). Though I find read-only SNMP 2c to be plenty secure.
Theo Schlossnagle
26 Apr 09 at 11:05 pm
Theo, true — but some people want SSH and only SSH open, even internally :-)
Xaprb
27 Apr 09 at 10:27 am
Ernesto, if you want commit access to SVN, just say so!
Xaprb
27 Apr 09 at 10:28 am
Hi Baron,
BTW, did you get a look on dim_STAT?..
It’s doing similar stuff, able to graph several hosts on the same time, based on MySQL and particularly easy in the new stats (add-on) integration – more details are @ http://dimitrik.free.fr
New v.8.3 (coming soon) includes also InnoDB and MySQL generic stats.
Rgds,
-Dimitri
Dimitri
27 Apr 09 at 2:07 pm
Dimitri,
I only just heard about dim_STAT today while reading about your performance research with InnoDB and XtraDB. Your comments got spammed by Akismet, I just retrieved them.
Xaprb
28 Apr 09 at 11:04 pm
Hi Baron,
good to know there may be issues with Akismet – because I’m also using it on my blog and now will be aware about :-))
Rgds,
-Dimitri
Dimitri
29 Apr 09 at 2:56 am
Akismet is generally very good, and only rarely spams something it shouldn’t. Bad Behaviour is the real secret sauce for this blog.
Xaprb
29 Apr 09 at 8:26 am
I had to disable Hyperic recently because it was seriously impacting server performance. I’ve been lazy about finding a replacement, but your blog post finally inspired me to start graphing again.
Ran into a few small issues with the setup (mainly due to never having used Cacti), but I managed to get everything setup in about 2 hours. Thanks Baron!
Gil
29 Apr 09 at 10:57 pm
We’ve seen servers brought to their knees by Hyperic, too. Running tons of commands every second or so, including — of all things — FLUSH TABLES. Gah. I love the quote they have on their homepage, from MySQL themselves of all people.
Xaprb
30 Apr 09 at 12:16 am
What Hyperic has going for it is ease of setup. It is dead simple to implement. Just run an installer, start the client daemon, and go.
I suppose it worked well until our servers got very busy, and eventually all that table locking caused too many queries to show up when running SHOW PROCESSLIST. When I finally realized the problem and disabled Hyperic entirely, our homepage response time dropped by 30%!
Personally, I never found all those db and table level statistics useful anyway.
Gil
30 Apr 09 at 9:11 am
just ditch cacti and put together a gmond plugin
pleeeaase :)
jimb
5 May 09 at 4:11 pm