The manager-programmer face-off over NoSQL
A lot of conversations with a few different people I respect (no links, sorry) have coalesced some thoughts about these newly popular “non-relational” datastores. I wanted to point out an aspect I’m not sure is very clear in the hot-topic department. This is about what happens when managers learn that their developers or operations team have installed some new technology in their systems without them knowing it.
Lest anyone think that this happens only in a poorly-managed company, I can attest that it happens everywhere, all the time. Remember Marten Mickos’s favorite story about salespeople asking prospects if they used MySQL, the managers saying absolutely not, and the developers contradicting them?
The moment of discovery is unpleasant for the manager, but everything leading up to it was a joy for the programmer. He decided that he’s annoyed with the MySQL database. SQL is hard anyway — it is such a pain to write queries like “find the most recent article in each category.” Besides, things are already slowing down, and it’s obvious that this system is doomed unless someone takes action. Management is stuck in the mud and unwilling to hear about anything; they are so conservative! The programmer works late one night, or over the weekend, and gets Paprika (imaginary NoSQL database, I call dibs on the name) working. The site’s logs are now in Paprika. It’s harmless: just a proof of concept. It will be good to show the managers that this is a better way to do things, without actually risking any real functionality on the system.
The manager now finds out about it, and is seriously stressed. Developers, estimate how much stress you think this could cause for your manager, and then multiply by about ten. This is the kind of thing that can ruin your manager’s sleep, put stress on his marriage and family, and make him end up wanting to quit or fire you.
How can something so joyful for the developer — productivity at last — be so opposed to the needs of the management and business? Let’s see what the programmer thinks is good about this new technology:
- I finally got to learn something cool! I keep reading on makemysitescale.com how all the cool kids are solving their problems, but if I don’t rebel, I’ll never get to use any of these techniques and technologies!
- I get to do something new! Yay! My ordinary life is so boring!
- I get to put this on my resume!
- I get to be the only one in the company who knows about this. I get to strut a bit during lunch break! I can talk casually about it, like it’s no big deal. They’ll all envy me even more.
- This is excellent job security for me!
Which of these things don’t make the manager stressed out? If you’re a manager, you’re looking at a nightmare. Think about it the manager’s point of view:
- This system is impossible to hire for. There are about a billionth the number of people on the job market who know Paprika, as those who know some SQL.
- Most people’s skills and experience will not transfer to this system at all. I can’t train anyone how to use this.
- Commercial support and consulting are either nonexistent or very limited.
- Documentation is inaccurate and scanty. Every Google result is talking about something that doesn’t exist in the newest releases, because this thing has changed every three months for the last year.
- I don’t know where this project/product is going in the future. The project’s developers seem excited about the openness of the roadmap — I do not share their enthusiasm.
- I can’t figure out how to write a query against this. How do I data-mine my logs? I’ve got to find someone who knows how to write a map-reduce function in Prolog or some wacky thing like that. There is no graphical interface, and I can’t plug my Business Objects or Crystal Reports into it.
- I can’t monitor this system. I can’t measure what it is doing. There are little or no tools for managing it. I don’t know about its performance characteristics, and no one else does either; we can’t capacity plan. How do we do backups and restores? Under what cases can it crash or corrupt its data, and what do we do then? My operations team is crippled.
- I have a rogue developer who thinks he’s got job security because of this, and he’s obviously interested in building his resume. He’d better be, if he knows what’s good for him, but I can’t fire him until I fix this mess. All I can do is hope he stays until I do.
What a disaster from management’s point of view! I recently heard about a case pretty similar to this, except that unfortunately the hot-shot coder did leave the company for a cooler job, and management was twisting in the wind with a system they could not support.
If I were hiring a team of developers, I think I’d interview them with the question “how cool is Paprika?” and mark them down for too-enthusiastic answers.
Hi Baron,
I do appreciate the underlying message, but the points you raise are hardly specific for “NoSQL” (as the title suggests) – it could apply to just about anything new – SQL databases, programming languages, even hardware and architecture. That doesn’t make those points less valid either of course, it’s just that NoSQL is one of the many many things someone could worry about.
While I believe the “rogue developer” story immediately, this is just one perspective. It doesn’t have to be like that, and it doesn’t have to be that black and white.
If people are stuck in some way with whatever solution they have now, there just have got to be some individuals with the right attitude to change things. While it would be even better if they have the skills to make the right decisions for a new architecture or technology, I really believe the attitude matters more. Experimenting with whatever options they dream up is a good thing in my book, in as far it doesn’t disrupt daily operations.
Obviously, it is not acceptable to simply make the changes and inform management later – good management takes responsibility for the people working for them, and trust is required both-ways for this to work.
But sometimes it is necessary to work in a stealth mode. Managers are often too busy to be bothered with every idea that might perhaps improve the situation, so in that case, quietly testing and trying until you’re sure you have an alternative to offer isn’t such a bad idea – they’re not going to be interested in whatever you tried which failed to deliver.
Sometimes, you may not be able to convince someone lest you show them for real. Common sense applies – you can’t just disrupt daily operations with wild new ideas and techniques, but at the same time, if you honestly think, based on your experience and evidence, that you cannot sustain the current situation either, well, then you shouldn’t just resign, but act and try to make things better, even if that means introducing someone that you know will be regarded as disruptive initially.
kind regards,
Roland
Roland Bouman
11 May 10 at 5:27 am
You are so so spot on, it’s haunting.
I have been at the “programmer” position, and now for the past 3 years I have experienced the horror of the manager. Couple in the fact that we are not BAD in technology, we just can’t keep up with the daily duties and managing full departments and then going and learning everything about “Paprika”.
Great post.
Mike
11 May 10 at 8:43 am
Absolutely correct. I’ve been on both sides of the problem repeatedly for 35 years or so, and it never changes. In fact, the modern software business was created as a direct response to an earlier version of this problem, which arose naturally from in-house-written systems. It may be hard to remember, but there was a time when most computing groups wrote or heavily modified their OS components, database systems (or their precursors), etc. Then again, maybe not – think of the LAMP environment 15 years ago.
Ross
Ross Patterson
11 May 10 at 9:25 am
I used to work at a place that was an all-Microsoft shop and had a bad case of NiH syndrome: almost everything was written in-house.
In VB6.
And nobody but me knew regular expressions.
And “unit testing” was their terminology for “use this fake credit card during checkout” — live data, live systems.
The horror… the horror.
Xaprb
11 May 10 at 9:54 am
I’m so going to ask that Paprika question verbatim in the next interview ;)
I do think Roland has a valid point here, although I totally see what you mean, Baron.
Strangely enough, many operations teams at smaller sites I know just don’t have the tendency to look for management systems for the infrastructure they use, so that point is somewhat moot (even though maybe the manager actually believes they do have management software ;))
At the end of the day it probably comes down to how strong the development and operations teams are, i.e. if they are able to write the necessary tools and comprehend the system quickly enough to be able to use it.
To pick up on Ross’s comment, many of the NoSQL products evolved just like that: They were developed as in-house tools and later opened up.
Clearly there’s a need for the top 1% of the market to have something that has the typical properties associated with NoSQL, but I strongly believe that most people don’t actually need it (yet). Quite the contrary, actually.
Kay Röpke
11 May 10 at 9:56 am
Baron,
Thanks for this deep analysis.
It’s curious, though.
Just yesterday, Peter was splitting the user base into two camps, the Boring and the Cool, with Percona on the cool (innovative) side
http://www.mysqlperformanceblog.com/2010/05/10/two-types-of-mysql-users/
and today, you seem to advocate the idea of keeping the users on the Boring side.
Giuseppe Maxia
11 May 10 at 10:59 am
I think Peter will be sad to hear that he was misunderstood again, and he will have to write another blog post :-)
I am not advocating for anything in particular, just pointing out that sometimes what’s good for the goose is bad for the gander. Many of our customers are Boring. They have financial data in their databases. They should be Boring. Many others are Cool. They should do Cool Things. Some of these companies don’t even HAVE managers, just extraordinarily talented developers who crank out new applications every time you turn around. The “let’s write a lot of apps, hope that one in 50 will get popular, and deal with the scaling problems if it does” business model is very different from the manager of an established business, whose viewpoint I took above.
Lots of our customers are using or exploring various non-MySQL technologies, including Redis and Cassandra.
Xaprb
11 May 10 at 11:18 am
“If I were hiring a team of developers, I think I’d interview them with the question “how cool is Paprika?” and mark them down for too-enthusiastic answers.”
I think this is the wrong approach. You want your developers to be pushing the technology envelope on their own time. That’s what will give you the flexibility when you NEED one of those tools, one of the guys can bring it on board and train the rest of the team.
What you don’t want are rogue developers who do their own thing without the support of the company. This is a question of business- IT alignment – developers are there to build and support the business. They should be happy to kick these decisions up to management to decide, constantly proposing ways to increase service or improve revenue, but happy to execute management’s decision, even if it means holding back.
The paprika question would be good, if followed up with ‘Tell me a story about a time you had to go off the beaten path to increase profits.’ Hire the first candidate that says what he did was really cool, but a mistake in retrospect.
Alex Bartlow
11 May 10 at 1:00 pm
So, you’d never hire an innovative developer?
I understand the issues around maintainability, but the position you advocate is wrong-headed.
Tim McCormack
11 May 10 at 4:02 pm
“Many of our customers are Boring.”
Oh, wait. Those are feeding you, aren´t they?.
Sip
11 May 10 at 6:18 pm
Yes. I like Boring customers. They rarely call me at 3AM to fix their servers.
Xaprb
11 May 10 at 6:34 pm
Tim, I think the point is innovative but balanced. Xaprb, I agree, boring customers are worth their weight in gold, while innovative customers are necessary in small doses to keep the spirit alive. ;)
Marcelle Paisley
12 May 10 at 3:53 am
Another factor:
Database groups become so regimented and trouble-ticket focused, that they cease to be useful to development projects. Of course you can’t install a competing SQL product.
So invent a basis of need for NoSQL and get full administration and control over your datastore.
cowardlydragon
12 May 10 at 3:09 pm
At one point in time MySQL was considered the same kind of ‘haunting technology’ and looked at in the same way that you’re debasing NoSQL offerings. You wouldn’t be doing what you are doing now if it weren’t for programmers and managers taking a risk with new tech.
I get that you’re concerned about your consulting services by the threat of having to train your team on some of the NoSQL software being used more and more lately, but bashing new tech for the sake of being new tech sounds like hypocrisy.
mbp_cmplr
10 Jun 10 at 12:38 pm
I understand how it might look, but it’s not about protecting consulting business. I’m not that kind of guy, and I have no illusions about my lack of power to influence people — you can’t fight a trend. It’s about helping people understand choices. Besides, I think some of the new databases are really cool, and even traditional relational databases will benefit from the ideas they are introducing. It’s kind of a Cambrian explosion, similar to what I’ve seen in many other areas of technology. Some of the technologies will thrive, and that’s good. In the meantime a lot of purely psychological factors are driving people’s ostensibly technical decisions and recommendations. This, too, is expected.
Xaprb
10 Jun 10 at 4:43 pm
The whole developer “takes action” starts when management wants him to solve the problem but remain conservative. I believe any organization (big or small) has to have the culture right. The culture has to promote innovation (NiH), exploration (Paprika) to keep the developers happy with coolness. The culture also has to see if finally the technologies used are going to give the managers nightmare. One simple way is to let the developers loose to use Paprika if some basic questions are answered:
– What is the popularity of this software? Is it easy to find people around this technology.
– Who is behind this? Community size? Activity? Companies? Their past behavior?
– What are the monitoring tools available around this?
etc, etc.
And as Roland said, this is true for everything.
I believe I repeated most of the post itself but couldn’t help but speak from developer side as well :)
Parvesh Garg
16 Jun 10 at 2:05 am