All measurements are wrong
I had the privilege to meet Neil Gunther and listen to him speak this week at Surge. During his talk, he brought up the point that all measurements are wrong by definition. I thought I knew what he meant, but I was stuck with tunnel vision about floating-point precision and such. I had it all wrong. The real answer is obvious and simple.
The point is that the process of measuring, and therefore the answer that comes out of the measurement process, is imprecise. And further, that we need to treat a measurement as a measurement, not as the true value of whatever it is we tried to measure. So although we may say “the CPU was 70% utilized,” we should really be thinking “the measurements of CPU busy-time totaled 70% of the measurements of elapsed-time.” There’s more, but I won’t repeat his whole talk. You might enjoy his book.
Neil mentioned that this way of thinking isn’t foreign — we learn it in physical sciences. Indeed, I immediately remembered all my chemistry and physics labs, and mechanical engineering classes, and…. But that’s a whole education away now. Somehow between then and now, I educated myself to think that computers manipulate numbers, and the numbers are somehow mathematically pure.
When computers store and retrieve numbers, that’s often imprecise, and that is continually present in my mind — but that’s a whole different matter.
True enough, but context matters. If you are working with real time systems then any instrumentation will contaminate the results. For most applications the cost of calculating time or writing the results to a log are negligible compared to the major processing being done.
Real time systems are the only examples that come to mind. Are there other systems where this would pose a significant problem?
Duane Gran
4 Oct 10 at 9:24 am
Measurement intrusion effect is one factor, and was proposed as an answer to Neil’s question by another audience member. But that wasn’t his point — his point was simpler than that technical detail. Measurements result in a quantity, but it isn’t the real quantity. A simple example is a process’s CPU usage. This is gathered by polling with centisecond granularity. It is not an exact measurement because the process of measuring is inexact, and the same is true of everything else we can measure.
Xaprb
4 Oct 10 at 9:58 am
Another way to accurately describe that would be “The CPU usage is estimated* to be 70%.” Also you could call them “samples” rather than “measurements” so it is clear that they are not absolute. It’s good to be aware of the actual method by which such figures are generated, but you can also obfuscate the meaning by using overly descriptive language in general conversation. I had to read the correct definition twice to get it.
* Measured at centisecond resolution. See …
John
4 Oct 10 at 12:40 pm
Hi Baron,
do you mean “measurements are wrong because of measurement uncertainty”? If you mean the same measurement uncertainty as in science, then really, calling all measurements “wrong” is way over the top. It just means we can be sure up to a degree.
It does provoke an interesting discussion about how we can calibrate measurement instruments, and how we can gauge the amount of uncertainty. (Something i don’t have a clue about w/re computer instrumentation)
Roland Bouman
5 Oct 10 at 3:14 pm
Ever watch House? Your title reminded me of his favorite saying: All patients lie.
Joe Devon
7 Oct 10 at 9:38 pm