How to gather statistics at regular intervals
I gather a lot of statistics such as performance data. Sometimes I have multiple things going on a system and I want to be able to align and compare the resulting data from multiple processes later. That means they need to be aligned on time intervals. Here is a naive way to gather stats at intervals:
while sleep 1; do gather-some-stats; done
There are two problems: each iteration will take longer than a second, so there will be drift; and the iterations will not be aligned exactly on the clock ticks, so the data isn’t as easy to correlate with other samples. This becomes a bigger problem when there are many such jobs gathering data at longer intervals such as 15 seconds or 5 minutes, where the lack of correlation between samples can be frustrating.
Here is what I’ve been doing recently. Is there a better way?
INTERVAL=1
while true; do
sleep=$(date +%s.%N | awk "{print $INTERVAL - (\$1 % $INTERVAL)}")
sleep $sleep
gather-some-stats
done
If you don’t care about the possibility of concurrent stats collection processes, you could simply fork the gather-some-stats call and sleep for $INTERVAL. You will perhaps still “drift” over the course of a week or so, but perhaps that’s acceptable.
lamby
18 Mar 11 at 10:32 am
Baron,
I usually use Perl and Time::HiRes for that. The following pattern should do what you want (for 1 second intervals here):
use Time::HiRes qw( ualarm );
$SIG{ALRM} = sub {};
ualarm((1_000_000 – (gettimeofday)[1]), 1_000_000);
while(1)
{
sleep;
print “beep!\n”;
}
This will set up a repeating timer, starting at the edge of the next second, and repeating every 1 second after that. Unhooking SIGALRM from its default abort handler will just cause it to wake up from the indefinite “sleep” call.
Regards,
Jeremy
Jeremy Cole
18 Mar 11 at 10:50 am
watch -n 1 gather-some-stats if you want the first behavior. expr or $(()) at least with bash allows easier math.
Allan Wind
18 Mar 11 at 11:31 am
lamby, I found that even backgrounding (and sleep itself) causes enough drift to matter. Jeremy, yes that’s a good Perl-ism. I remember that we used that in mk-heartbeat originally, but ultimately had to move away from it because of various problems I don’t recall with sleep and alarms and signal handlers… it stops working reliably when things get more complex. Allan, bash doesn’t do floating-point math, only integer math.
Xaprb
18 Mar 11 at 12:59 pm