Monitoring with Artificial Intelligence and Machine Learning

May 26, 2017

Artificial intelligence and machine learning (AI and ML) are so over-hyped today that I usually don’t talk about them. But there are real and valid uses for these technologies in monitoring and performance management. Some companies have already been employing ML and AI with good results for a long time. VividCortex’s own adaptive fault detection uses ML, a fact we don’t generally publicize.

AI and ML aren’t magic, and I think we need a broader understanding of this. And understanding that there are a few types of ML use cases, especially for monitoring, could be useful to a lot of people.

Setting Thresholds With Quantiles

Nov 7, 2015

I was talking with someone the other day about a visualization I remembered seeing some years ago, that could help set a reasonable value for a threshold on a metric. As I’ve written, thresholds are basically a broken way to monitor systems, but if you’re going to use them, I think there are simple things you can do to avoid making threshold values completely arbitrary.

I couldn’t find the place I’d seen the visualization (if you know prior art for the below, please comment!) so I decided to just blog about it. Suppose you start off with a time series:

New O'Reilly Book, Anomaly Detection For Monitoring

Aug 3, 2015

Together with Preetam Jinka, I wrote a book for O’Reilly called Anomaly Detection for Monitoring.

The book contains great examples of anomaly detection used for monitoring. It’s practical and simple, and contains no hype.

Early-Warning Is an Unknown Unknown

Oct 8, 2013

This post originally appeared on O’Reilly Radar. In 2002, US Secretary of State Donald Rumsfeld told a reporter that not only don’t we know everything important, but sometimes we don’t even know what knowledge we lack: There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.

MySQL monitoring meetup tonight!

Mar 21, 2012

Here’s your spammy day-of reminder about tonight’s free MySQL meetup. The topic is MySQL Monitoring Bonanza. There will be beer, pizza, pretty charts and graphs, and friends. Free as in beer (and pizza, did I mention that?)! Note: VividCortex is the startup I founded in 2012. It’s the easiest way to monitor what your servers are doing in production. VividCortex offers MySQL performance monitoring and PostgreSQL performance management among many other features.

Free webinar on monitoring MySQL

Mar 19, 2012

If you follow what Percona is up to, you might have noticed that I’ve created a set of high-quality monitoring and graphing plugins for MySQL and related systems. Currently they support Nagios and Cacti. I’ll give a free webinar on March 28th discussing these, and more broadly, discussing how to monitor MySQL successfully to avoid common problems like spammy alerts about nonexistent problems. VividCortex is the startup I founded in 2012.

Version 1.1.8 of Better Cacti Templates released

Jan 22, 2011

I’ve released version 1.1.8 of the Better Cacti Templates project. This release includes a bunch of bug fixes and several important new graphs. There are graphs for the new response-time statistics exposed in Percona Server, and a new set of graphs for MongoDB. VividCortex is the startup I founded in 2012. It’s the easiest way to monitor what your servers are doing in production and I consider it far superior to Cacti.

A better way to build Cacti templates

May 25, 2010

The traditional way to build Cacti templates is through the Cacti web interface. This is an enormous amount of work, and the result is generally not very consistent or good quality. The process is too error-prone. You can export the templates as XML, but they tend to have problems such as version incompatibilities with other Cacti installations, and it’s hard to adapt them for user preferences such as different graph image sizes and polling intervals.

Version 1.1.7 of Better Cacti Templates released

Apr 20, 2010

I’ve released version 1.1.7 of the Better Cacti Templates project. This release includes a bunch of bug fixes, some new graphs for MySQL, and two new sets of graphs, for Redis graphing and for JMX graphing. VividCortex is the startup I founded in 2012. It’s the easiest way to monitor what your servers are doing in production and I consider it far superior to Cacti. VividCortex offers MySQL performance monitoring and PostgreSQL performance management among many other features.

Version 1.1.6 of Better Cacti Templates released

Jan 10, 2010

I’ve released version 1.1.6 of the Better Cacti Templates project. This release includes a bunch of bug fixes (but not all of them!) and two new sets of graphs. One set is for disk I/O on GNU/Linux, and the other is a new set of templates for OpenVZ. I’m looking for feedback on both of those. This release also has a bunch of code-level features: much better test coverage (hooray!), and a refactored ss_get_by_ssh.

A Review of Cacti Network Monitoring by Kundu and Lavlu

Jan 9, 2010

VividCortex is the startup I founded in 2012. It’s the easiest way to monitor what your servers are doing in production and I consider it far superior to Cacti. VividCortex offers MySQL performance monitoring and PostgreSQL performance management among many other features. Cacti 0.8 Network Monitoring, Dinangkur Kundu and S. M. Ibrahim Lavlu, Packt, 2009. Page count: 110 pages. This is a quite short book that covers some of the breadth but very little of the depth of Cacti.

How To Read Linux Iostat's Output And Interpret System Performance

Jan 9, 2010

The primary tool for inspecting Linux disk performance is iostat. The output includes many important statistics, but they’re difficult for beginners to understand. This article explains what they mean and how they’re calculated.

Layers

Vote for Cacti template enhancements

Dec 30, 2009

If you’d like some improvements to the Cacti templates (MySQL, Apache, Memcached, and more) I’ve been maintaining, please make your voice heard – either write to the mailing list, or post a new issue (or comment on an existing one) on the issue list. I’ve got a bunch of work underway, including a test suite for ss_get_by_ssh.php. VividCortex is the startup I founded in 2012. It’s the easiest way to monitor what your servers are doing in production and I consider it far superior to Cacti.

Version 1.1.5 of improved Cacti templates released

Dec 13, 2009

I’ve released version 1.1.5 of my improved Cacti templates for MySQL and other components of a LAMP application. This is a pure bug-fix release. One of the bug fixes prevents spikes in graphs, but requires you to rebuild your RRD files. There are upgrade instructions on the project wiki for this and all releases. Use the project issue tracker to view and report issues, and use the project mailing list to discuss the templates and scripts.

Version 1.1.4 of improved Cacti templates released

Oct 25, 2009

I’ve released version 1.1.4 of my improved Cacti templates. Unlike the prior release, which was solely bug fixes, this one includes new graphs in the MySQL template. Some of the graphs are of data that’s exposed in standard MySQL versions, but some of it is available only in Percona’s high-performance builds of the MySQL database server. If you don’t have a Percona build, those graphs will just contain nothing, but there is no detrimental effect.

Version 1.1.3 of improved Cacti graphs for MySQL released

Oct 24, 2009

I’ve just released version 1.1.3 of the Cacti templates I wrote for MySQL. This is a bug-fix release only, and affects only ss_get_mysql_stats.php. To upgrade from the previous release, upgrade ss_get_mysql_stats.php. Don’t forget to save and restore your configuration options, if any. (Note that there is a feature to help with this: you can keep configuration options in ss_get_mysql_stats.php.cnf to avoid making them in ss_get_mysql_stats.php.) VividCortex is the startup I founded in 2012.

A tip when upgrading mysql-cacti-templates

Oct 15, 2009

A client recently asked me to fix some Cacti graphs that had broken after upgrading the Cacti templates I wrote for MySQL. The symptoms were weird; I’m not sure I understand fully what happened, but some of the graphs were OK and some had only part of the data they were supposed to. Some graphs would have one data element as usual, and others would be nan (not a number).

Version 1.1.2 of improved Cacti templates released

May 7, 2009

I’ve packaged up and released version 1.1.2 of the Cacti templates I’ve written for MySQL, Apache, memcached, nginx etc. VividCortex is the startup I founded in 2012. It’s the easiest way to monitor what your servers are doing in production and I consider it far superior to Cacti. VividCortex offers MySQL performance monitoring and PostgreSQL performance management among many other features. Anyone who would like to help write documentation (or do anything else, for that matter) is welcomed to participate.

Secure, easy Cacti graphing without SNMP

Apr 25, 2009

Cacti is a great tool for collecting information about systems and graphing it. However, it likes to use SNMP, and SNMP is often not desirable. Instead, I often see the need for a method that is: Secure. Use trusted, well-known, encrypted communication. Do not open up new ports. Zero install on the monitored system. As little installation or modification on the monitoring system as possible. Over the last several years, I’ve slowly created more and more software to create Cacti graphs via standard POSIX command-line utilities over SSH with key-pair authentication.

News on MySQL Cacti Templates

Feb 9, 2009

It’s been a while since I’ve sent any news about the MySQL Cacti Templates project I started a while ago. Here’s what’s new: VividCortex is the startup I founded in 2012. It’s the easiest way to monitor what your servers are doing in production and I consider it far superior to Cacti. VividCortex offers MySQL performance monitoring and PostgreSQL performance management among many other features. First of all, it’s not just MySQL templates anymore.