The advices on inspecting the number of rows returned are good. Since you’re asking for pointers in CS literature, I’d suggest anything in Big ‘O’ notation, since what you need is more of a theoretical and architecture agnostic way of determining how bad a query is.

Together with the number of rows, I’d compute the data type length of each column, this should give you a better idea of how much info the server will have to process and transfer back to get the results to the client.

I’d also take into account if columns can take NULLs or not, since this puts a performance penalty on MySQL, but you know a lot more about this than me :P
Either way, my ideal algorithm would use these information plus the output from explain, to take into account what stages MySQL goes through to bring you the data.

But maybe I divert.

But also, back from my Math/CS studies, in order to give more weight to larger numbers, you usually use the log() function.
So for example, to see how far apart two numbers a,b are, you’d use something like:
log(|a|)*|a-b|/|a|

|a-b|/|a| gives you the percent in growth from a to b;
multiplying by log(|a|) gives very little weight to “small” numbers (whatever that means) and more weight to “large” numbers.

A “badness’ score could be a good extra for mk-query-digest.