“Astounding IT sayings”: the inaugural post

I haven’t given myself the luxury of telling an IT anecdote or two here recently, so it’s about time: here are two, with moral-of-the-story observations for each.  Note: these are true stories.  I may have changed some of the facts, lightly, to make them less identifiable.  They’re also always at least several years in the past, to provide a healthy amount of distance for everyone.

I’d actually like to make this post the first in a recurring motif, a series that I’ll call “Astounding IT Sayings,” for what I hope are obvious reasons.  The saying that I consider to be astounding, in its context, will be highlighted for you below in bold.

There I sat, interviewing for a CTO position.  Of course, these sorts of interviews always go both ways, and I was at least as interested in hearing about the state of the company, its products, its technologies, as they were in seeing whether I’d be a good fit for what they needed.  Without a doubt, the company in question looked solid, and was clearly on the move in their market.  As I talked to various executives and also a number of IT staffers, I heard just a few doubts being expressed about a current major project, but otherwise, everyone I talked to was amazingly upbeat about the technology, the capabilities of the staff, their ability to work together with the business, and the robustness of their processes.

Then came the kicker.  As I spoke to the Director of Technical Operations, I asked him about the company’s software launch processes and how solid they were.  Having been burned any number of times by weaknesses in this area, I was especially interested in hearing how they promoted software from development into test into staging and then finally into production, both for maintenance fixes and for major releases.  I asked, “on a scale of 1 to 10, where 1 is complete chaos and 10 is perfection, where do you think this company is, in terms of the solidity of its release processes?”  Without missing a beat, he confidently answered, “Oh, we’re at an 11.

I didn’t really need to ask a lot of follow-up questions, since doing so would have inserted a pointless element of contention into the interview: the “11” had already revealed to me what I would find out. It’s been my observation that nearly all companies have weaknesses in their release processes: e.g., bug fixes break other things; major releases cause old bugs to resurface; rollbacks aren’t well thought-out or fail entirely, and so on.  I’ve actually never seen a company that was even at what I’d consider an 8 on that hypothetical scale of 1 to 10.  More than anything, though, I was looking for an attitude of continuous improvement as the underlying philosophy, and “we’re at an 11” was the exact opposite of that.

It was a bet-the-company project, one that had to succeed: implement a portal web site that would supplant and ultimately replace all the company’s current e-commerce sites with a more sophisticated, more capable, more comprehensive integrated offering.  We’d been working away on it (all new architecture, new approach, groundbreaking innovation in terms of presentation and business process change) for many months, and the launch date loomed before us.  We weren’t ready, and everyone knew it at heart.  Too many glitches, too many unresolved issues, too few successes in our test environments.  Crisis moment.

A new key executive had just ridden into town, was only a few weeks into the job, and he called a meeting of all the project’s principals.  He kicked off the meeting with this announcement: “I’d like to hear a vote on whether we’re ready to launch.” My jaw dropped. No discussion or even mention of metrics, intermediate milestones, or readiness criteria. Instead, a verbal vote.  On a scale of 1 to 10, with 1 being completely unready.

What ensued was actually what one would tend to expect of a group of middle managers in front of the new and powerful key executive, with no one wanting to come across as negative.  “Oh, I think we’re at a 7,” the first person opined. “I’d say an 8,” piped up his neighbor.  We went around the table, with nearly all the votes falling in the range of 6 to 9.  I was among the last to speak: “We’re at a 1 in my view, completely unready, and here’s why,” I said to a chorus of gasps.  I explained that we actually shouldn’t be voting but should instead be nailing down facts-based criteria on which we’d base our launch decision.  I pointed out that we’d gotten precisely zero transactions successfully through the system end-to-end.

OK, said Mr. Key Exec.  Looks like maybe we’re not ready. Everyone is going to work the weekend.  (This meeting was on a Friday).  We’ll meet again on Monday and reexamine where we are.

And that we did.  Lots of hard work, lots of late night pizza and diet Coke.  We gathered together in the same room around noon on Monday.  To my amazement, this is what I heard: “OK, let’s see where we are.  Why don’t we go around the table again and everyone give me your impression of our launch readiness on a scale of 1 to 10?”  And that’s what we did.  “Oh, I’m up to a 9 now. We really made progress this weekend.” “I’m at an 8, but I was at a 6 on Friday.”  And round and round.  When it came to me, I again quietly restated my “1”, pointing out that despite hard work and progress, we had yet to see a single successful transaction.  I said that judging from experience and based on where we were at this point, my prediction was that we wouldn’t really be ready to launch for another three months.

The website launched about three months later.

The key takeaway should be obvious: no matter what the pressures, don’t ever base a major operations decision on a vote, where such voting would serve to substitute for a sober evaluation of facts-based criteria.  Launch readiness is way too important to leave up to pure gut instinct and mass optimism.  Create your metrics and your criteria up front, collect them, and look at what they tell you.  Then act accordingly.  That way, you’ll end up with results, rather than astounding sayings.

Comments

  1. Rick S says:

    Amusing stories. Thanks for both the entertainment and the “moral of the story” conclusions. Quite interesting and useful.

    You mention that you’ve never seen a company that was even an 8 out of 10 on the release process scale. Can you go into more detail about what criteria you would look at for something like that? Thanks.

  2. Peter Kretzman says:

    Great question, one that really demands a whole follow-up post, which I’ll begin to sketch out.

    Release process has to do with ensuring “reproducible results” as well as mitigating risk (since releasing new software into production is about the riskiest thing you typically do). It covers everything from source code control, to what I call “kitting” (i.e., release packaging), to rollback definition and decision process, to testing round and round through the iterative dev/test/debug/rerelease cycle. And then, deciding when and if you’re ready to launch (and no, not by looking at the calendar and seeing that a specific date has arrived).

    All of that is hard, and human being-dependent, and error-prone, and subject to constant forces of entropy and political pressures: in short, way more about process than it is about pure technology. Hence my statement about most companies not even being at an 8 out of 10.

    More to follow. Thanks for the question!

Trackbacks

  1. […] few months back, I wrote a post on various “Astounding Sayings” that I’ve encountered in my career in information […]

  2. […] has ever gone awry or possibly could. Their own approach is the only one conceivable for success. “This one goes to 11” seems to be their slogan. The presenters seem to think that the more enticingly they portray […]

Speak Your Mind

*