CIO: “We can’t go live in six weeks as you want.  It’s going to take at least three months.”

CEO: “That’s … unacceptable!”

One of the most recurring memes in IT, for me, has to be hearing “we don’t like that estimate”, coming from stakeholders, senior management, etc. Depending on the mood and/or semi-intellectual rigor of the person saying it, the conversation then typically devolves into one or more of the following:

1. identifying and removing any hint of schedule contingency (which is often viewed as padding just to make life easier for IT);
2. mentioning repeatedly the idea of “what if we double the team size to get it done twice as fast” etc.;
3. conducting a scrutiny, one by one, of the bottom-up estimates (”it won’t really take three days to test that feature”);
4. volunteering resources (usually less than qualified) to “help”;
5. insisting on scheduling full-time work for all remaining weekends and holidays between now and the desired launch;
6. making frequent use of the phrase “why don’t you just …”
7. declaring that system delivery must occur by a specific date, no matter what.

A real-world micro-example of estimate pushback: one of my Twitter colleagues tweeted this a few months ago: “Me: it’s six weeks effort. Client: what can you give me in two weeks?” That was obviously tweeted in frustration, and was most likely intended to serve as an example of how incredibly unreasonable the client (or manager, or CEO, whatever) can be.

Yet, at core, and read literally, it’s a perfectly natural question coming from an executive who is looking to explore all the possibilities.  IT people need to learn to anticipate such scrutiny, and have coached and practiced enough with their own team to show that they’ve thought through some alternatives.  That’s collaboration as a key business contributor.  That’s strategy. In fact, showing that kind of maturity is what will take IT beyond being mere order-takers.

So the CIO/CTO’s role as leader here is to inculcate the following mindset in the collective team: don’t assume that these management suggestions are simply ignorant or obstinate. In fact, I’m arguing here that management reactions along these lines are understandable, though admittedly sometimes misguided and excessive.  I’ve been there: I’ve done it myself, frustrated (as an exec) by the estimates the team was telling me.  So now, instead of just getting annoyed at the upper management behaviors listed above, I see both sides.  Even better, I have some recommended behaviors that the IT team can adopt that will help in such situations.

First, and this is key, adopt the mindset that every plan needs to withstand some scrutiny. Expect scrutiny and plan how to deal with it. The answer from IT can’t be “these are our estimates, so you just have to accept them.”  Just as with making any management presentation, you need to anticipate the obvious questions, adjust your plan to accommodate them where possible, and have further answers at the ready.  It’s a back-and-forth, not a one-way. Delivery discussions need to be about ways to provide viable business value given the constraints. It’s a collaborative process, not a confrontational negotiation where one side makes demands and the other side caves in.

Second, spend some time seeing and removing your own (IT’s) flaws.  Specifically, consider flaws such as these, common reasons why estimates become regarded with skepticism once an executive drills down:

• Every task in the schedule, no matter how slight, is assigned at least a day’s worth of work in the plan. IT teams need to doggedly squeeze out the obvious air.
• Easily understandable tasks (ones that the executive has good familiarity with) are being estimated as taking much longer than they have historically. Way before it gets to an executive presentation, the team needs to challenge every number themselves so that this either doesn’t happen, or can be explained with a supportable rationale as to why things are different this time.
• More generally, there appears to be little or no historical basis or metrics used in establishing the estimates. Estimates that are based on facts and data, rather than people’s gut, are almost always more credible.
• Once the executive starts to drill, people cave immediately: “Oh, OK, I guess we could do that in a couple of days, rather than a week.”  Making this kind of admission is throwing raw meat to a hungry bear.  It shouts that your estimates probably had little basis in fact.  Or, that your emphasis on certain must-haves like integration testing isn’t so heartfelt at all.  At the very least, it reveals that you haven’t thrashed through the numbers internally well enough to be able to stand behind them with greater firmness. Believe it or not, and often with all evidence to the contrary, the executive is looking for where there’s strength of conviction, and where there isn’t. Strength of conviction (supported by facts, of course) may ultimately be disagreed with, but it’s usually respected.
• The team and/or project leader shows little or no recognition that there may need to be some kind of interim solution, some sort of early delivery. There’s no “team attitude”, no searching for midway points (i.e., acceptable scope reduction), no exploration of “what ifs”, such as “what if we delivered the system without an administration GUI at first?”  Once again, there’s no substitute for holding internal practice sessions and staging multiple “devil’s advocate” discussions.
• Conversely, there’s no clear “line in the sand” visible, meaning points you stick to no matter what, points that cannot be compromised if the company wants a quality delivery.

The very fact that these reasons for estimate-related skepticism even occur in the first place demonstrates the value-add of such skepticism. In other words, everyone needs to realize that it’s actually OK for management to ask people to dig deeper, to explore every option, because it turns out sometimes they don’t on their own.  It’s OK to be asked, and everyone should expect to be asked.  That’s part of everyone doing their jobs, and, cynicism aside, such active scrutiny is most certainly the job of senior management.

Steve McConnell, in his outstanding book on Software Estimation, writes eloquently on this issue.  He points out that executives typically have certain notable characteristics and well-honed skills.  The first two of these?  “Executives will always ask for what they want.” “Executives will always probe to get what they want if they don’t get it initially.”

Example: a developer once told me that it’d take two weeks to generate a report I needed as CIO. Taken aback, I probed into it with him.  Together we determined that the report I was asking for entailed nothing more than a rudimentary SQL SELECT statement, slightly dressed up. Every time a manager drills down into an estimate, only to discover that there was no “there there”, it feeds his or her unfortunate but lingering suspicion that most estimates are grossly padded. Getting the “win” (beating down the number) spurs them into even more dogged drill-down.

So I’m preaching a several-pronged approach:

• doing one’s homework so that maximum scrutiny has been self-applied long before the estimate reaches the executive;
• sticking to one’s guns to an extent that is both reasonable and facts-based;
• and then actively looking for ways to reach value for all.

One important note, though: there are often bad (but unfortunately common) ways to compromise when discussing project delivery estimates.  Here are a couple of important places not to cave:

• Suddenly deciding that you don’t really need contingency. Until perhaps the last week, it’s usually a huge mistake to assume that the rest of the schedule will go exactly as planned. Contingency in a schedule is not padding!
• Suddenly deciding that you don’t need thorough testing. Forgoing testing is pretty much a guaranteed way to launch an unstable, buggy product that ultimately will cost you much more work (not to mention the customer backlash) than taking the necessary time up front to shake out the problems.

Where’s the CIO/CTO in all of this? Supplying active leadership: educating everyone on the tenets above, coaching, cajoling, reminding, monitoring.  Strong leadership can and does reduce these common “we don’t like that estimate” kinds of proposed “solutions”. In fact, it is the mettle of the mature and successful CIO/CTO that they constantly “upward and sideways” manage their peers and management to get them to understand the cold harsh realities and weigh appropriate and reasonable trade-offs.

Thinking about how to prevent big system project failure has somehow always reminded me of the Will Rogers quote: “Don’t gamble; take all your savings and buy some good stock and hold it till it goes up, then sell it. If it don’t go up, don’t buy it.”

In other words, with big projects, by the time you realize it’s failed, it’s pretty much too late.  Let’s think a bit about the reasons why, and what we can do to change that.

First off, I’ve never seen a big project fail specifically because of technology. Ever. And few IT veterans will disagree with me. Instead, failures nearly always go back to poor communication, murky goals, inadequate management, or mismatched expectations.  People issues, in other words.

So much for that admittedly standard observation. But as the old saying goes, “everyone complains about the weather, but no one does anything about it.” What, then, can we actually do to mitigate project failure that occurs because of these commonplace gaps?

Of course, that’s actually a long-running theme of this blog and several other key blogs that cover similar topics. (see my Blogroll to the right of this post). Various “hot stove lessons” have taught most of us the value (indeed, necessity) of fundamental approaches and tools such as basic project management, stakeholder involvement and communication, executive sponsorship, and the like.  Those approaches provide some degree of early warning and an opportunity to regroup; they often prevent relatively minor glitches from escalating into real problems.

But it’s obvious that projects still can fail, even when they use those techniques. People, after all, are fallible, and simply embracing an approach or methodology doesn’t mean that all the right day-to-day decisions are guaranteed or that every problem is anticipated.  Once again, there are no silver bullets.

One of the problems, as I’ve pointed out before, is that it can actually be surprisingly difficult to tell, even from the inside, how well a project is going.  Project management documents can be appearing reliably,  milestones met, etc.  Everything looks smooth. Yet, it may be that the project is at increasingly large risk of failure, because you can’t address problems you haven’t identified.  This is particularly so because the umbrella concept of “failure” includes those situations where the system simply won’t be adopted and used by the target group, due to various cultural or communication factors that have little or nothing to do with technology or with those interim project milestones.

Moreover, every project has dark moments, times when things aren’t going well. People get good at shrugging those off, sometimes too good.  Since people involved in a project generally want to succeed, they unintentionally start ignoring warning signs, writing those signs off as normal, insignificant, or misleading.

I’ve been involved in any number of huge systems projects, sometimes even “death march” in nature.  In many of them, I’ve seen the following kinds of dangerous “big project psychologies” and behaviors set in:

• Wishful thinking – we’ll be able to launch on time, because we really want to
• Self-congratulation – we’ve been working awfully hard, so we must be making good progress
• Testosterone – nobody’s going to see us fail. We ROCK.
• Doom-and-gloom fatalism – we’ll just keep coming in every day and do our jobs, and what happens, happens.  (See Dilbert, virtually any strip).
• Denial – the project just seems to be going badly right now; things are really OK.
• Gridlock – the project is stuck in a kind of limbo where no one wants to make certain key decisions, perhaps because then they’ll be blamed for the failure
• Moving the goal posts – e.g., we never really intended to include reports in the system. And one week of testing will be fine; we don’t need those two weeks we planned on.

An adroit CIO, not to mention any good project leader, will of course be aware of all of these syndromes, and know when to probe, when to regroup, when to shuffle the deck.  But sometimes it’s the leaders themselves who succumb to those behaviors.  And for people on the project periphery, such as other C-level executives? It’s hard to know whom to listen to on the team, and it’s definitely dangerous to depend on overheard hallway conversations: Mary in the PMO may be a perennial optimist, Joe over in the network group a chronic Eeyore who thinks nothing will ever work, and so on.  There are few, if any, reliable harbingers of looming disaster.

Wouldn’t it be great if there were some kind of codified, external measurement/evaluation tool that could methodically identify the kinds of disconnects that even well-led projects can fall prey to? One that could pinpoint where the true risk areas are as the project evolves, and help people take targeted action ahead of time to address those problem spots?

That’s why I got so excited in a recent conversation with well-known IT failure expert Michael Krigsman, CEO of Asuret, a company that sells “technology-backed services”.  He gave me a look at their forthcoming product, an impressively slick, well-engineered tool that in my view promises to provide exactly that kind of benefit: identifying where and why a project might fail in terms of some of those people/best practices aspects, before it actually does.

In a nutshell, Asuret facilitates a cross-sectional analysis of project participants and stakeholders as the project proceeds. By aggregating the answers to its carefully crafted questions and constructing a number of easy-overview summary charts, the tool then displays astonishingly insightful visual breakdowns that let you pinpoint major disconnects, such as between stakeholder groups and IT, or between actual project-specific and industry-best practices.

Let’s look at an example of what it shows you.  By mapping aggregated analysis results onto charted dimensions of importance and vulnerability, and slicing these charts by department, you can see at a glance in the chart below that there’s a disconnect: e.g., that executives think that the business case for the project has high vulnerability, while the IT participants view it as having low vulnerability.  Early warning sign!  And certainly better (more methodical, more aggregated) than relying solely on what you’ve heard Joe grumbling about in the lunchroom.

In the example, the disconnect looms large: look at the darker circle (representing the participants’ responses to questions regarding the project’s business case) and its different location on the two grids shown below:

Figure 1

This all sounds simple in this brief description, perhaps, but taken as a whole, Asuret’s methodical implementation and targeted, useful results are nothing short of groundbreaking. Perhaps other companies provide a similar product, but I don’t know of any. And frankly, I can’t imagine a better-designed or more perfectly suited product as Asuret to address the issues raised in this post.  I’m really looking forward to hearing more as they deploy and hone their product, because I can think of any number of large projects I’ve been on where this approach would have been revealing and useful.

It’s maybe not the ever-hoped-for holy grail, but it promises to be a small piece of it: an extension of our ability to see things before they happen.  If Will Rogers had been an IT guy, I think he would have been excited too.

Anyone ever tell you that a simpler approach can often work better than a more complex one? Whoever it was, it probably wasn’t a project management software vendor.  But simplicity has its merits, and I’d like to point out a few of these when it comes to resource allocation.

Project management, at its core, is largely about resource allocation, and this gets tricky when you have multiple projects going on, as most organizations do. Almost as much as I’ve seen organizations drop the ball entirely on cross-project resource allocation (essentially, simply pretending that there will be no contention issues), I’ve seen organizations go to the other extreme: they dive into the depths of intense Project Management, in capital letters: taken too far too fast, this approach can spin up to a high level of rigor and overhead, involving often-expensive software packages, precise low-level estimates, diligent collection of actuals, and ornate project calculations of hours burned and hours earned.  At the end, there you stand, like Goethe’s Faust, “no wiser than before.”

Occam’s Razor

Let’s talk first about the general problem of resource allocation, then, and the range of possible solutions.  First off, I’ll note that running a successful IT department, one that delivers predictably and with high quality, depends on far more than just being able to run an individual project successfully.

A major part of overall success in the IT arena comes down to whether (and how) you’re allocating the right (and enough) resources to the right tasks at the right time. That’s obviously true at a project level, but it’s especially true across projects, as you juggle multiple goals, diverse timelines, and random delays anywhere along the line.

Project management is relatively easy, in other words. The hard part is PROJECTS management. As I’ve pointed out before, the most common and glaring problem I see in companies with IT in crisis relates to an unfortunate tendency to estimate and commence projects one by one, with little thought to resource contention issues. That approach almost always leads to chronic overcommitment and late delivery.

So how do you do resource allocation, in a practical sense?  I’m talking about determining contention issues, not task-by-task assignments.  That’s where organizations often drop the ball: as I said, they either ignore the problem almost completely (allocating resources essentially by a “seat of the pants” approach), or they go whole-hog into major package adoption and process overhaul. But the seemingly greater precision and granularity of such packages does not guarantee greater accuracy or efficacy, and such organizations can easily founder as fast and as much as organizations that punt completely.

Surprise: there’s no silver bullet

My stance? There’s no ONE answer for what’s the best practical way to do resource allocation in-the-large. It depends. It depends on your organizational politics. It depends on the maturity of your overall software development lifecycle process. And most of all, it depends on you, and the level of your desire/appetite for attaching a degree of rigor to what is by nature an ever-swirling hodgepodge of resources, tasks, projects, deadlines, dependencies, delays, resets.

The Project Management Book of Knowledge (PMBOK, highly recommended) identifies nine different areas of focus, each of which is fairly elaborate. Correspondingly, project management tools cover a spectrum of functionality. You don’t just start using “project management software”; you need to decide, initially and all along the line, just how deep and how broad you want to go.

In fact, the areas of project management and resource allocation can be integrated with task management, reporting, document management, financial projections, collaboration tools, email notification, management escalation, risk evaluation, etc. The list of potential useful and desirable features stretches long.

This results in a futile quest for a kind of Unified Field Theory of IT and project management, where everything can be fruitfully connected to and integrated with everything else. The irony is that while tools approaching this goal do exist, and are getting better all the time, implementing them successfully can be not only very expensive, but is also a massive and risky enterprise-level project in and of itself, both in terms of initial implementation and in ongoing care and feeding.  And the outcome? Well, an awful lot ends up being “tightly coupled“, which can lead to problems in and of itself.

These different tiers of solutions are for vastly different problems, as orthogonal as bicycles are to trucks. Yet the persistent underlying and increasing belief remains, as you get more and more ornate with the tool you use, that project management can be wholly deterministic, that you’ll get to the point where the data will predict the precise end date, everything will line up, and so on.

It can’t. It won’t. It’s an illusion. Project management, and especially resource allocation, consists of constantly revisiting and revising the plan, not trying to get absolute certainty about every aspect of it at all times.  Rather, the successful IT executive learns to live with ambiguity, uncertainty, approximation, and to work within them. Yet, the spectrum of project management software holds out the (false) promise of eradicating much of that uncertainty.

Tools for resource allocation

Let’s look briefly at the range of project management solutions that you might fruitfully consider, with respect to the all-important aspect of resource allocation across projects.  This list is a cursory overview, of course, and is not intended to be comprehensive in any way. For further information, consult the links in my Lagniappe section at the bottom of this post.

• Seat of the pants. Assign work when people seem to have some available cycles.

Pros: simple, intuitive

Cons: completely intuitive, and usually doesn’t scale across multiple projects. Unreliable.

• Build an “approximating” spreadsheet that lets you get a quick overview of coarse-level resource allocation for your portfolio of projects. See example below.

Pros: quick overview, relatively easy maintenance (e.g., monthly)

Cons: requires spreadsheet expertise, and understanding that these are rough estimates rather than precise allocations.  Limited scale for large organizations; doesn’t directly feed actuals/results into further planning.

• Installed software packages: e.g., MS Project, OpenProj, Primavera.

Pros: fine products, broad functionality, well-known

Cons: Getting a good, easy overview of resource contention is difficult, plus requires constant care and feeding. With this approach, I’ve seen projects that needed to assign resources to do nothing but work the project management software

• Web-based products: e.g., Serena, Projects On Demand, Zoho Projects, Innotas, Project Insight, attask, Daptiv.

Pros: This rapidly evolving market niche is centralized (SaaS) and innovative.

Cons: has same issues as installed packages. On a larger scale, becomes a huge business change management project to implement

• Large packages: CA’s Clarity, Changepoint, etc. (PPM)

Pros: enterprise-class, high level of functionality and integration.

Cons: high implementation cost in dollars, time, and resource commitment, up-front and ongoing

Here’s an example of an “approximating spreadsheet”, where resources are roughly allocated day-by-day to any of nine simultaneous projects (numbered 1-9, and each color-coded).  Note that this is a quick-and-dirty planning/working document, meant to do coarse allocation planning at the start of a month, with actual assignments and microadjustments made as the month progresses. Its principal advantage, beside pure simplicity, is that it quickly forces the planners to confront the ever-present “NUTS” problem, where there are too many projects underway at once for the available resources.

I’ve had great success with the approximating spreadsheet approach in a “bang for the buck” sense, in smaller (less than 100 people in IT) organizations. Once you wrap your brain around it being just an approximation tool, it gets you 80-90% of the way there without having to devote a large portion of your staff’s time to the constant and intense administrivia often necessitated by one of the bigger project management packages. (The example spreadsheet itself is available to download for free in a link in the Lagniappe section at the end of this post.)

But those packages do have their place. In short, though, here’s the key ongoing challenge for project management software adopters (and, of course, vendors): how do you obtain the necessary functionality across the vast spectrum of functionality, without turning everyone into a project data entry monkey? Your project management software needs to be a tool, not a torture device, not a time sink.  My argument is that your key goal, in a cross-project sense, is to roughly allocate your resources so that you avoid major contention, and/or taking on too much.  And you want to focus on this with the minimum amount of overhead you can, because at an extreme, it will suck up virtually all your time.

A few years back, I had an extremely surprising and unpleasant experience as CTO. The director of my Program Management Office ran a weekly status meeting for project stakeholders, where we’d all methodically go through the current project portfolio, in order to communicate on issues, gather necessary feedback, and align everyone’s expectations. I typically attended in order to provide input and executive-level decision participation, but left it to the director to actually run the meeting and present the topics.

Unfortunately, immediately before one of these weekly meetings, that director was given bad news (in a brief hallway conversation, no less) about a major bug that had just been discovered in the software for our highest profile project, which was currently in testing and due to launch in just a couple of weeks.  This project, with its strategic and revenue-enhancing potential, was foremost in the minds of everyone in the company.  Stakes were high, in other words.

Without conferring with me or anyone else, the director announced to the entire group of assembled stakeholders that the project had hit an insurmountable snag in terms of our target launch date, and that it would be delayed potentially by weeks. No further information was available on specifics, and people’s concerned questions went unanswered. You could feel the anxiety spread through the room. I actually had to truncate the meeting abruptly and promise to reassemble the group when we knew more.

My director was quite upset with me. Rather than having my action be understood as a necessary attempt to prevent panic until we could get some real facts, I suddenly realized that I was being accused of wanting to “hide information” from the stakeholders. The director even told me, “Well, I like to be honest and tell it like it is, but that obviously just isn’t what is wanted here.” I was stunned. Here was a person who had the wrong idea of transparency, and astonishingly for their position, a warped idea of what project management is all about.

As most experienced project managers know, it makes utterly no sense to drag stakeholders through the entrails of every back-and-forth readjustment that is discussed and worked through in the course of a project’s lifetime.  To do so lowers stakeholder confidence, unnecessarily in many instances.  In this case, the director somehow wanted to be a kind of Paul Revere, forewarning the stakeholders, but instead came off as Chicken Little. Transparency can’t be an act of individual heroism; it needs to be a concerted, institutionalized, collaborative philosophy and approach.

Moreover, good project management isn’t about making everything go smoothly. There’s probably no project plan in the history of the world that has gone completely smoothly from beginning to end.  Instead, project management is about responding appropriately when it doesn’t, and recalibrating to keep everything still on track. In fact, it should be expected that not everything will go as planned, and your whole team should be prepared to take some bumps and glitches in stride without panicking.

It’s not hiding anything to hold back on discussing, in a public forum, a problem that has just been discovered, the ripple effects of which initially can only be guessed at.  When you promulgate potential bad news prematurely, it puts you into the position of trotting out the absolute worst case, before you really have any idea whether that worst case is likely or even plausible. In fact, it’s a fundamentally selfish act: it takes your own personal panic and extends it to as many people as possible.  And if there’s one thing we know about panic, it’s that it’s contagious.

You need to shoot for judicious and organized transparency. That’s where you plan on regular status communications, and you methodically, soberly assemble the facts in a structured manner before you make those announcements. Most of all, you collaborate internally: before you bleat about a potential crisis to the stakeholders, you bring in the rest of your team to talk through the possibilities and “degrees of freedom” in how you’ll act with respect to that crisis. Otherwise, you may think you’re nobly playing Paul Revere, but instead of imparting critical facts, you’re shouting “The British might be coming!”  And that’s not transparency, that’s mere alarmism.  Remember that especially during the dog days of a stressful IT system development effort, what often seems like a dire setback, upon careful and collaborative examination in the light of day, turns out to be merely a minor speed bump that is easily overcome.

Is it “controlling the message” to be cautious about spreading the alarm? You bet it is. And doing that is in everyone’s interest. Remember my adage: show that you’re in control. Your job as project manager (and, by extension, the job of the Director of the PMO, and the CIO) is to be in control and to demonstrate to the users that you’re in control.  That doesn’t mean that you “hide” anything; it does mean that you deliver bad news broadly only when you’re sure of your ground, have weighed the options for how to cope with that bad news, and are ready to present those options soberly instead of in a flailing manner.

Adhering to a philosophy of transparency is a far cry from shouting every blip or misgiving about the project from the rafters. This is particularly so if you’re in a leadership position. You want your communication to be transparent, yes. But without appropriate structure and caution, transparency can be the height of irresponsibility. In those cases, in fact, you’re just building a slightly more institutionalized rumor mill.

Lagniappe:

• Karen Guglielmo, “What is transparency, and how is it relevant to IT?“
• Phil Windley,  “Tools for IT Transparency“
• Peter Kretzman, “Why the CIO should air the dirty laundry“

Roger Sessions recently published a white paper on IT complexity and its role in IT project failure: “The IT Complexity Crisis: Danger and Opportunity”.  It’s certainly possible to quarrel with bits and pieces of his analysis, and thereby tweak his numbers, but the overall thrust remains undeniable: IT failures are costing the world incredible amounts of real money. Sessions even sums it up under the dire-sounding phrase, “the coming meltdown of IT,” and says, “the out-of-control proliferation of IT failure is a future reality from which no country—or enterprise—is immune.” And he presents “compelling evidence that the proliferation of IT failures is caused by increasing IT complexity.”  He points out that the dollar cost of IT failure in the US alone is close to the cost of the recent US financial meltdown, and cites indications that the rate of failure is also increasing by 15% per year.

Roger’s paper is excellent and thought-provoking, and I recommend it highly. And I do agree with his view that complexity is the chief culprit in IT failure. That said, I think his argument focuses a little too strongly on one cause of complexity (unnecessary overcomplexity of architecture), to the neglect of other important factors.

To be sure, some obvious contributors to IT failure (poor project management, and lack of communication within teams and from business to IT implementers) aren’t dismissed by Sessions, but he sees their contribution to the crisis as relatively small. I don’t, and I’ve used this blog to write about those factors quite a bit.

Most of all, though, I differ with Roger’s focus on streamlining architecture as being the key to reducing system complexity. One could say, in fact, that Roger’s solution is primarily a technical one, where the bugaboos I see are primarily cultural and sociological.  I see not one, but at least three distinct complexity-related burdens, increasingly endemic, and increasingly bringing down IT:

• Overly complex design/architecture
• Taking on too much functionality
• Poor implementation (technical debt in-the-large and in-the-small)

Roger has admirably dealt with the issue of overly complex design/architecture, at least in terms of a viable approach for simplifying up-front architecture, so I’ll focus here on the other two.

Taking on too much functionality

I recently rented an economy car (the least expensive option) on a trip with my son. (Remember, my self-appellation is “Cheap Technology Officer.”) He was stunned and dismayed that the car didn’t have automatic door locks; he didn’t realize that they even made cars without them anymore. Similarly, my 11-year-old daughter has grown up in a TiVo-ized world where live TV can always be paused. When she encounters a TV without a DVR (and thus no pause capability), she regards it as hopelessly primitive. Indeed, as unacceptable.

Similarly, the general standard for functionality and UI design has been raised by extremely functional PC software.  I now expect to be able to double-click on a number in any onscreen report, and thus “drill down” into the transactional details that make up that number. When I can’t, I feel cheated. Equally, I expect everything on an interface to drag and drop; I get frustrated if it doesn’t.

So as an industry, we’ve raised the bar of acceptability, considerably, in software and technology systems over the last couple of decades.  What that means in practical terms, though, is that across the board, our eyes have gotten bigger than our stomachs. We want more, up front, than it often makes sense to build at the start. And our demands are not negotiable, or so it seems. The first few cell phones I had didn’t even have a ring silencer on them; I used to silence the phone by adroitly disconnecting the battery when a call came through at an inopportune time. Today, most people wouldn’t even consider buying a phone that lacks much more elaborate features, such as a camera, that I would have considered as space-age in nature back in the 80s.

So our increase in expectations, alone, has added considerably to the functionality of systems we tend to build. There’s more functionality in and of itself, and usually more interface points to other complex systems. In fact, integration testing—where you connect new code into a working environment where it has to interface correctly with other systems—has become a frequent and major sticking point in launching information technology projects.  In essence, we’ve fallen into the “nuts” dilemma, both in large and small ways.  We want so much, and attempt so much, that we increase our risk of failure considerably.

Technical debt (in the large and in the small)

Any software developer will tell you that their first stab at implementing a given piece of functionality is often (if not usually) much more complex than turns out to be needed. Only after exploring the problem domain, with experiments and backtracking and restarts, do developers usually realize that their code can be pared down, simplified, combined with other modules, etc. This is usually called “refactoring”, and its importance is a relatively recent insight in the software development discipline.

A key insight about refactoring, though, is that it means improving the code without changing its overall results. There’s often no immediately obvious payback to this undertaking: it’s a roof project, in essence. To the extent that refactoring isn’t done (no time, no inclination, no recognition of a simpler approach), the end product is left with vestiges of unnecessary complexity. The greater the time crunch, and the greater the aspiration for the functional depth and breadth of the software to begin with, the more likely it is that these vestiges linger. And one ancillary aspect of the raised bar in expected minimum functionality is that it causes the time crunch to get ever greater.  It’s no longer about delivering just a solution that will work, it’s about making sure that the solution includes (metaphorically speaking) a 5-megapixel camera too.

Couple this unnecessary but accidental complexity with what often amounts to a rush job on design for the sake of meeting schedule (creating “technical debt in-the-large”). An example I’ve used here before: choosing a different core DBMS to implement a given function, simply out of expedience, and failing to take time to modify previous functionality to use that new DBMS. Supporting two DBMSes within the same product represents significant technical debt: every subsequent system change and addition will entail “paying interest” on that debt, which not only increases schedule and manpower costs, but increases risk of failure as well. And that’s just one example; the technical debt cascades, feature upon feature, release upon release. Technical debt, until paid down, can be equated to a invisibly rising substrate of complexity, and it contributes massively to an increasingly wobbly, risky system. And lately, I see more and more organizations “pyramiding” their technical debt, never taking the time and cost to pay it down, with disastrous results. As Hemingway said about going broke, it happens slowly, then all at once.

What now?

Roger’s analysis, to its large credit, outlined an important aspect of complexity and posed a solution (an approach he calls SIP, or Simple Iterative Partitions).  The aspects that I’m presenting (taking on too much functionality, and the pyramiding of technical debt) are, as I’ve said, cultural and sociological within companies. The answer is not nearly as simple or as neat as a specific technical solution (although I am certain that I will get comments on this post, perhaps rightly, from devotees of Agile).

To my mind, it’s engaged, savvy, forceful leadership that alone can address these issues, slow down the demand train, stop the madness. If anything, I think that there is an increasing lack of leadership in IT circles that can suitably recognize and address these factors, as well as educate their peers. And that’s what needs fixing most of all.

Lagniappe:

• Kevin Schlabach, “Agile East 2009 by Thoughtworks”
• Martin Fowler, “Technical Debt“
• Eric Ries, “Embrace technical debt”
• Johan Lindberg, “Painting With Hands and Feet: The IT Complexity Crisis”

“We went live before the system was ready”.  It’s a common excuse/explanation that I hear from IT people when they tell war stories about system launches that failed miserably. Implicit (and sometimes explicit) is the add-on statement: “and we told them so beforehand, too.”

There are obviously many things (and many parts of the org chart) that contribute to a failed launch, but here I’d like to focus on what drives this particular kind of launch-before-readiness, where the views of the rank-and-file are unheard or ignored.

In a nutshell: it’s management pressure. Sometimes that pressure comes from middle management, sometimes from the very top, and often from both.

Why do executives apply the pressure? There are a number of reasons:

• They actually believe it’s the best way to keep the team producing and working hard. They fear that extending the deadline will cause a let-up.
• Their own bonus and/or job is on the line if there’s a delay. They, or their management, regard a launch delay as a failure, just as much as a bad launch. At the very least, a delay means losing face.
• Testosterone (for either gender) sets in: a “refuse to lose” mentality. To some extent, that’s what has gotten them where they are.

Deadlines matter, of course, a lot, and people who are lackadaisical about deadlines simply shouldn’t work in IT. That said, if you bull-headedly insist on adhering to deadlines, to the exclusion of facts-based input, it makes you all the more likely to delude yourself that you’re really ready, just because the launch date has arrived and the pressures are heavy. If the stakes aren’t high, that may even be fine. (But, are the stakes ever not high in these situations?)

Let me tell three anecdotes that illustrate management “refuse to lose”:

• Earlier in my career, I was a middle manager, wrestling with a project that was both controlled by and staffed with a single “Big 5” consulting vendor’s people, with millions of dollars per month of consulting costs being racked up.  The high-profile initial system launch was a week away, and according to the bug tracker, there were still over 100 open major bugs. I sat down with the vendor’s partner-level project head. I don’t remember the precise conversation, of course, but it went something like this from him: “Oh, we’ve already got 39 of those 100 fixed in dev, 44 others are being worked by our best people and will be resolved by tonight, so really, there are only about 15. Four of those we think have workarounds so there are something like 10. I don’t think we’ll be getting many more bugs. We usually resolve bugs in a day or two, so even if each one takes two days, we’ll be able to close those ten and be ready for launch in a week.” I sat in awe (and frustration) at this hand-waving, this magic with math, and at his reluctance to think things through without an agenda. Here, “refuse to lose” had developed blinders, simple math had turned fuzzy and infused with vested interest, and IT best practices (not to mention common sense) had flown out the window.

• I worked as VP of Operations under a dev-oriented CTO. We had a late night launch, and the CTO was there in person, hovering over my sysadmins who were attempting to install the latest release into production with little success. The installation had failed, starting with the very first run of the script. “No, try this: run just those three lines of the installation script we provided you, then type these two commands, then run these other five lines.”  When that didn’t work, he came up with another idea, and then another. The next day, he sent out a company email declaring victory and congratulating his dev team, because the system had come up finally, even though it had run for a total of only two hours before crashing hard. Installation by seat of the pants. Refuse to lose.

• I arrived on the scene as the new CTO of a company in the midst of a complete replatforming project, completely retooling and replacing their sole product. The project had been underway for about nine months already; just before the point of my arrival, the team had been told by senior management that their launch date was going to be in six weeks, no ifs, ands, or buts.  I proceeded to do what I call a quick “temperature check”: I asked to see key indicators of readiness, such as bug counts over time and test case execution reports.

As you’ve surely guessed, I quickly determined that the facts didn’t even remotely support the possibility of a launch in a mere six weeks. About 40% of the code was still to be written; less than half of the test cases had been executed at all, with an even lower percentage having passed.  I marshaled my facts and figures, and went in to tell the CEO of this sorry state.  Judging from my experience, I told him, it looked like it would be a number of months, not weeks, before the product could be successfully launched. I remember him sputtering “that’s unacceptable”, and pressuring me on what I could do to, yes, bring off a launch in six weeks. (Cutting to the end of the story: we launched, successfully, nine months later).  In this case, a mixture of testosterone and probably some job-on-the-line aspects were driving this CEO to ignore facts and look for ways to bull through to success despite obvious counter-indicators.

The tragedy of the space shuttle Challenger has become almost cliché along these lines: a combination of executives feeling pressure and ignoring communication from lower ranks about major problems. “They also disregarded warnings from engineers about the dangers of launching posed by the cold temperatures of that morning and had failed to adequately report these technical concerns to their superiors.”  For whatever reasons, NASA management gave the green light for launch despite the adverse weather conditions. In essence, they “refused to lose”, and by doing that, they lost in a major way.

Takeaways:

• Don’t let your executives turn you, and don’t let them become, riverboat gamblers, where failure is a strong possibility through a premature launch. The company culture must be shaped so that it views a spectacularly failed launch as much more onerous than a delayed launch.
• Don’t let “groupthink” settle in, where your people are afraid to speak up. Any and every member of the team should have a chance to “pull the emergency brake”.  Especially, holding a formal “go/no-go” meeting is essential, where any stakeholder or project participant can voice concerns and point out facts that speak against readiness.
• That said, do be careful of the pessimist amid your ranks, who typically voices dire predictions on every project, and who sooner or later will be proved right. Insist on facts, not “Eeyore-ism”.
• Let your team watchwords for these situations become the following: candor, and facts. Bring both to the meetings, or go home.
• The best way to avoid letting pressure drive a premature launch: establish your launch criteria clearly and openly, with quantitative measurements, way in advance. If you don’t have legitimate facts-based ways to tell if you’re ready (i.e., not just the launch date arriving, and not your testosterone level that morning), well then, you’re not ready.

Without a doubt, “refuse to lose” can be a great attitude, up to a point. No executive, no team, should ever become blasé about missing a target. But “refuse to lose” can’t ever be allowed to become “refuse to face facts”. When it does, you don’t just lose face: you just plain lose.

I’ve done several posts featuring what I call “Peterisms”, which are basically aphorisms I’ve adopted that encapsulate hard-earned IT lessons. Let’s turn it around this time, and talk about two sayings that sound equally folksy-sensible, and that I hear again and again, but which I feel are actually dangerous to apply to information technology work. And, of course, I’ll discuss why that’s so.

• If it ain’t broke, don’t fix it

• A camel is a horse designed by a committee

So, not all aphorisms are created equal. If you’re in IT, I advise that you avoid the attitudes and philosophies contained in these two.

Here’s one series of questions that’s great to ask a candidate for a senior position in IT, be it in project management, development, operations, or whatever: Tell me about a recent project or initiative for which you were responsible. What was the goal, and how did you ensure that the goal was achieved? What were the roadblocks you encountered along the way, and what did you do to address them? What mistakes did you make? What would you do differently if you were faced with a similar project or initiative again?

For this post, I’ll play candidate and outline the kind of full and detailed response I would be looking for as a hiring manager, based on a project from my own experience. Let’s use a relatively simple one: implementing off-site (Software-as-a-Service, known as SaaS) email, replacing a poorly implemented in-house email system. Today, this would loftily be called “going to the cloud.”  (Yes, I’m aware that cloud computing encompasses far more than SaaS, but that’s not my thrust here).

As the incoming CTO at an Internet social networking company several years ago, I quickly observed that an unacceptable percentage of my IT staff’s time was absorbed in administering, troubleshooting, and user handholding related to the in-house email system, a third-party substitute for Microsoft Exchange.  But far more than the high cost factor was at stake: the flakiness of the company’s email and calendaring system had become both a cultural punchline and, frankly, an easy company-wide excuse (a little like “the dog ate my homework”) for people missing meetings and failing to communicate on key initiatives. Undeniably, meetings were indeed disappearing regularly from people’s calendars, and even simple meeting scheduling required using an unintuitive workaround to guarantee that all recipients would get the meeting notice. So, our goal was two-fold: reduce cost and staff distraction significantly, AND (most importantly) give the company a solid, reliable system for internal and external communication.

The mail server package had been implemented several years before my arrival as CTO, apparently without a careful plan or study, and mostly as a gut reaction by a then-VP of IT who simply loathed Microsoft and wanted a non-Microsoft solution. A significant portion of a DBA’s time, not to mention that of the entire help desk staff, was being expended in support of the system, not to mention the high hardware and software maintenance costs. My staff, and just about all users, detested the system. But oddly, roadblocks were present nonetheless:

• staff resistance on changing out a system they regarded as part of their job;
• CFO reluctance to make changes in general, and especially reluctance to go to an external email provider (concern about loss of control).

One lesson here? Things are never so bad that people drop their innate resistance to change!  But I digress.

We countered these roadblocks via several approaches:

• We surveyed all users in the company for their views of the system and tolerance for change in this area, and fed this feedback into the resulting action plan.
• We constructed a thorough cost analysis of the status quo: how much the current solution was costing the company in both hard and soft costs. I used this analysis to obtain executive buy-in, and to specify in detail what the cost and effort would be to fix it.
• We made sure that both junior and senior staff were involved in the methodical evaluation of alternative approaches, including the alternative of keeping the status quo. This helped achieve staff buy-in.
• We brought our “short list” vendors in for week-long user proof-of-concept trial periods, getting both staff and user feedback, thus defusing a lot of the “loss of control” type of uncertainty regarding the change.
• We negotiated hard with our “short list” vendors to obtain a good deal for the company.
• We worked hand-in-hand with our excellent legal staff to make sure we were well protected in terms of vendor SLAs, security, etc.
• We constructed and followed a careful user communication plan, keeping everyone informed of the rollout schedule via FAQ-style emails.

The rollout of the chosen solution went fairly smoothly.  We failed to identify two key risk areas or areas of concern clearly enough, though:

1. Users were much more concerned about calendar migration (old appointments and their attachments) than we had anticipated, and somehow this didn’t come out in surveys etc. before the implementation; we responded not by migrating the old items (expensive and difficult), but by keeping the old system running for a period of time after system rollout of the new solution, and that seemed to be satisfactory.
2. Users had great difficulty adapting to a 250MB email space limitation, since they had effectively had “no limit” (albeit poor performance and terrible stability) under the old system.  Even though we had discussed that limit in advance, done benchmarking across similar companies about their space limits, and gotten clear user buy-in, it still might have been better to consider a higher limit (and hence more cost) as we sought expenditure approval.

Now, just three years or so later, I’d do several things differently:

1. obviously, anticipate and avoid the mistakes itemized above;
2. consider using Google’s business email offering (we ruled it out since it was brand-new as of the time of the implementation, but it offers a much greater storage limit than any other provider I’m aware of); and finally,
3. implement sooner rather than later. Why sooner? The implementation and system was such a success that despite the bumps and concerns I just named, there was no more constant hallway conversation about the email system. It had ceased to become a concern, and also ceased to become an excuse.

And bottom line for the company, besides those productivity benefits? The total cost of running our (now very stable) mail system had dropped to the equivalent of just one FTE, about a third of what it had been.  Happier staff, happier users, and much lower costs: all projects should be so fortunate.

Lagniappe:

• Maria Spínola, “An Essential Guide to Possibilities and Risks of Cloud Computing“
• Michael Coté, “Don’t confuse SaaS with Cloud Computing“

OK, let’s assume you’ve gotten great at picking the right projects to do to benefit the company. How do you know you can actually accomplish the ones you’ve picked with the resources you have? If you’re like most companies I’ve seen, it’s done on a wing and a prayer. But there’s a better way.

Last time, I wrote about ways to pick projects that satisfy the company’s “SHOULD do” dimension: ones that are strategic, financially beneficial, risk mitigating, or legally mandated, for example. I set out practical guidelines for the process of selection in that dimension, to ensure as level a playing field as is possible.  And I left it for this follow-on post to discuss the other dimension, the “CAN do” dimension, which needs to calibrate the list of chosen projects to what can actually be accomplished by the available resources.

Both dimensions inform each other, of course; they’re not independent or sequential. In other words, the company might be able to tackle five smaller projects rather than one huge project, and figuring out if it’s a good idea to actually make that trade-off will be based on executive judgment of the benefits in both scenarios.

In “practical CIO” terms, though, the SHOULD do dimension has the rather unfortunate trait of being a challenge to automate or model, especially fuzzier aspects of proposed projects such as risk or strategy or market positioning. In fact, I’ll go out on a limb and say that overemphasizing this dimension may even be ultimately pointless in terms of the eventual project selection. Bear with me here; I’m certainly not dismissing the value of ROI estimates etc.  Project managers of one persuasion will claim that a deterministic modeling for all such factors is possible — one recent blog informs us that “One can use either Benefit Measurement Methods (Comparative approach) or Constrained Optimization Methods (Mathematical approach) or both to arrive [at a] conclusion on project selection.”

I agree it is both possible and even desirable in the selection process to have a full and quantified assessment of factors affecting the “SHOULD do” dimension.  But here’s my practical experience, again: in the end, the findings of such a model are often ignored, almost arbitrarily.  In other words, in many companies of all sizes, the fact of the matter is that politics (and gut) trump spreadsheets: project selection often strays away from sheer issues of ROI or weighted risk calculations and into the realm of a CEO (or other senior executive) simply determining a strategic mandate.

Hence, I recommend not obsessing on honing a deterministic model with intricate weighting and scoring schemes, etc.  Instead, as I wrote last time, recognize that there will still be a judgment-based component to project selection, and focus on standardizing the input information to the “SHOULD do” phase to ensure that judgment is maximally informed.  Recognize that biases and politics will play some role in project selection, and seek actively at all times to mitigate the effects of such bias. A CEO who really wants to initiate a given project will often do so no matter what the model says; nonetheless, the inputs are still vital.

On the other hand, the “CAN do” dimension aligns the list of projects with what can be accomplished by available resources (I call this “factory capacity“). In essence, it’s meant to provide a way to overcome the natural corporate tendency to take on too much, to “stuff eight pounds of manure into the five-pound bag.“  This is one case where you can’t afford to have politics trump spreadsheets. Having determined, through whatever arduous and (no doubt) contentious process, the “vital few” projects of the “SHOULD do” dimension, it is absolutely critical that you achieve success with them. A key way to fail is to take on more than you have resources to do; doing so casts into doubt the success of all projects in the portfolio.

So here’s one relatively quantitative way to proceed.  In a nutshell, evaluate whether you’ve “right-sized” your selected project portfolio to your factory capacity by making some over-the-thumb squinting-style estimates for each project, simply classifying each into “T-shirt” size categories.  You then link these with general assumptions about the number of hours required for work on a given project in each category. Linked in the Lagniappe section is a sample template spreadsheet, containing three potential templates of varying complexity that you can adopt for such estimating. The goal? To make sure that the projects you’ve picked will be achievable by the resources you’ve got available, based on the assumptions you’ve made about how big projects tend to be.  Later, actual analysis and scoping will allow a more precise resource commitment.

Again, you can make this process very simple or very complex, depending on your needs, inclinations, and patience. Let’s start with a very simple model: three categories of project (small, medium, large). You and your staff should know best, from past experience, how many categories make sense to have, and you should be able to gauge, overall, roughly how many total hours of work are required on average for each size.  Include in your scope of consideration all core work of the through-launch development lifecycle: analysis, requirements, design, development, QA, and implementation.  These hours will be different for each organization, needless to say.

Also up front, you’ll need to enter your total headcount available for the given period (in my experience, a calendar quarter works well for this planning exercise), in the same work areas.  Don’t make the typical error of assuming that each person will be available for forty hours a week (or more): the model includes a productivity factor, currently set at 80%, to account for meetings, sick days, administrative duties, and the like. Remember, this exercise entails rough estimates, and they won’t give you a precise answer or actually do resource allocation.

Then, do a “speed chess” exercise where you estimate, project by project, whether it’s a S, M, or L.  This tends to go pretty quickly. Then the fun begins: along with senior stakeholders, you can play “what if” games: a spreadsheet is a great way to include or exclude projects (usually based on your “SHOULD do” evaluated criteria), trying to make as much as possible fit into the bag.  In this model, placing an “X” in the “Include in Final Total” column for a given project uses the estimate of hours for that project in calculating the total commitment (with all included projects), and judging whether or not you’re over capacity.

Here’s a look at an example of the most simple model:

As you can see, the included projects create a total of estimated hours (based on the T-shirt sizing), which is compared to the total hours available from the current headcount.  In the above example, the total hours is slightly over what will be available from the current headcount, meaning that you should look for a different mix of projects that will be achievable.

Additional sample templates are included for greater complexity: e.g., more categories (1-5 in the most complex template), or providing for component T-shirt sizing, recognizing that some projects involve little development but intense QA, for example.  Picking any of these is vastly better than just shrugging and thinking you can do it by instinct.  And it helps all participants understand the undeniable (but often-ignored tenet) that projects really do need to be narrowed to the “vital few.”

Lagniappe:

• Project Estimation sample templates (Excel spreadsheet)