I recently had the privilege of being interviewed, along with other experienced senior technology executives, by CIO magazine for my thoughts on communication mistakes still made by CIOs. Some great ideas came out in the article, but when it comes to communication (see tip #1 below), there’s always more to say. So here goes.

• Communication can always be worked on and improved. I was at one company where we did a semiannual employee satisfaction survey. Even better, the company was admirably dogged about implementing specific measures to address areas of dissatisfaction that emerged from the survey results. But in every single survey, the number one vote-getter was the need to improve intracompany communications, no matter what initiatives were spawned to improve them. Communication is an ongoing challenge and necessity.

• Don’t just communicate upwards. Communication with your team is every bit as vital as your communication to management and to peers. You need your team to have a clear and common understanding of the goals, as well as getting their active contribution and buy-in on the necessary approach to getting there. You can’t go it alone.
• Hierarchy is actually helpful; use it actively. Do your best to avoid providing explicit nuts-and-bolts-level direction to staff members other than your direct reports. Jumping in and telling people at all levels what to do confuses people mightily, undermines the authority of your direct reports, and increases the likelihood of mixed signals. And it’s just plain inefficient. Early in my career, I was a director responsible for a major week-long software rollout in one of my company’s regional offices. The project manager of the initiative was one of my direct reports. But then, the stakes were high enough that the CIO decided to fly down, and of course, the regional VP of IT was there as well. So for a week, we had four management-level people all sitting in every meeting, each trying to actively demonstrate they were in charge and look good to the others. Or, there’s out-and-out, from-the-top, intentional bypassing of the chain of command: Amazingly, I’ve seen cases of a CEO setting up a task force without involving or even informing the people who managed the members of the task force. Talk about undermining!
•  Be clear about what’s actually a directive and what’s just an exploration. Dawn Lepore, CEO of Drugstore.com, recently spoke in an interview about how she couches her direction to her staff as to whether it’s a “a light bulb or a gun”. “A light bulb means this is just an idea I had, so think about it, see if you think it’s a good one. Either follow up or don’t, but it’s just an idea. A gun is, I want you to do this.” Surprisingly, people often can’t tell the difference on their own. As a leader, you typically want most meetings, particularly at a senior level, to feature open brainstorming, a free exchange of out-of-the-box ideas, but that means it’s all the more important to carefully identify what emerges as actionable initiatives and what things continue to be simply thrown around as possibilities.
• Put it in writing. Supplement the directions you give in meetings and one-on-ones with a clear and succinct written summary. Doing this both clarifies and makes an unambiguous record of what you’ve asked for, and accurately communicates it more broadly than just to the people who happened to be in the meeting. It’s a way of demonstrating that you’re declaring yourself accountable. Doing so is vastly preferable to having your people run around claiming “The CIO wants this.” (And with some, once they discover that such a line grants them all sorts of vicarious power, they’ll tend to overuse it, and at inappropriate points). I’ve seen a lot of ineffective leaders follow some kind of old-school “just tell them” mentality, and they cause significant confusion in the ranks by failing to put things in writing. And it’s no wonder that with this approach, which evolves into a kind of “Telephone” game, they’re often disappointed in what they get.

Oral communication is often vague and ambiguous. Written communication is on the record. Strive to go on the record as much as possible.

• Be utterly clear about who owns the execution of the initiative, and who will be making specific lower-level related decisions. Most notably, avoid having people think that everything needs to be cleared with you at all junctures. For most initiatives, your role is to set the overall goal, not to shepherd day-to-day the specific process of getting there. Lack of clarity here is common, even when everything else is done right.

Common sense, all these tips, no? The thing about common sense, though, is that it’s often frightfully uncommon.

Most senior technology executives have a good sense of the huge value that comes from comparing notes and impressions with one’s peers about industry trends, techniques, project approaches, even vendors. Networking, appropriately handled, can enable you to find out all sorts of “lessons learned” without having to go through the pain of learning them the hard way.

But as with most things, there are effective and less effective ways of going about that sort of networking. For a long time, I looked to industry conferences to provide this sort of connection and exposure to a wider and wiser set of peers. But despite a few positive experiences, I’ve changed my mind in general about the utility of conferences.

Aside from technical exposition and tutorials, most industry conference sessions revolve around case studies. And oh, what cases they are, according to the presenters. Quite typically, everything is golden, nothing has ever gone awry or possibly could. Their own approach is the only one conceivable for success. “This one goes to 11” seems to be their slogan. The presenters seem to think that the more enticingly they portray their project and approach, the greater value they’ll provide to their audience.

And as for dialog? I’ve found, through painful experience, that the kinds of questions typically asked at conference sessions are most often designed to make the questioner himself look really smart. They’re often not even real questions, more lengthy expositions. We’ve all encountered that sinking “captive audience” feeling you get when you realize that such a questioner, a zealot of one flavor or another, has commandeered the audience microphone and is out there grinding his or her axe, and that someone is going to have to deal with it. Awkwardness pervades the session until a moderator or a fellow participant finally speaks up and shuts it down. That’s not dialog. That’s not networking. Even hallway conversations at such conferences can be filled with similar self-aggrandizing.

But this syndrome is actually deeper than something you just see at conferences. I would submit that it is unfortunately characteristic (admittedly in the extreme case) of what often plagues IT spokespeople in general, as they present before senior management or their board. We all want to be valued and highly regarded, and somehow, many of us decide that the best way to achieve that is to tout the bright side of our coins and leave any dark sides unmentioned or glossed over.

Let’s look at an example, in the form of a recent article about the fabulous success of Agile approach at eBay. Read the article; you won’t find a single wisp of a thought about any downsides, any blemishes, that occurred along the way. It was all golden, apparently. You see only references on how “to out-think and out-execute the competition”; or, “deliver useful information in days instead of months.” Can you say “silver bullet”?

Presenting any complex endeavor in nothing but glowing terms is to willfully forge an illusion, of course, not just at a conference or in an article, but at a board meeting as well. Somehow, the “case study” style of presentation tends to feature just that sort of dubious strutting, an implicit declaration that “everything went great.” Yet recognize that the people you’re presenting to are accustomed to piercing through that sort of bullpucky every day from vendors they deal with; you’ve lost your audience as soon as they figure out that you’re of similar ilk. It’s a similar deaf ear to the ones that teenagers turn when their parent tells them again and again about trudging ten miles through the snow to get to school, uphill, both ways. It’s a fish story.

The broader issue of how to present to one’s own management is something I’ve discussed elsewhere. But on the specific issue of finding a better solution to accomplish most of the goals that people hope to achieve by attending conferences? Local peer groups with regular meetings. Well-facilitated and attended peer groups, in my experience, can provide benefits that most conferences can’t or don’t: far greater continuity, candor, dialog, and, yes, screening. Most are participation by invitation only. And in such a setting, people stop having their principal goal being to impress others, and turn to looking to glean and impart useful information and tips above all. The best of the ones I’ve participated in adopt a tenet of “what’s said in the room stays in the room,” which greatly encourages candor and maximizes the true practical utility of the discussion.

Case studies and anecdotes are interesting, but my point is that conferences typically consist of lots of these, all golden, implausibly strung together back-to-back. And as I’m fond of saying, the plural of anecdote is not data. Many conference presentations are, at least in some sense, nothing but elaborate sales jobs. And if you really went to the conference for networking and learning, you want dialog, not sales.

Thinking about how to prevent big system project failure has somehow always reminded me of the Will Rogers quote: “Don’t gamble; take all your savings and buy some good stock and hold it till it goes up, then sell it. If it don’t go up, don’t buy it.”

In other words, with big projects, by the time you realize it’s failed, it’s pretty much too late.  Let’s think a bit about the reasons why, and what we can do to change that.

First off, I’ve never seen a big project fail specifically because of technology. Ever. And few IT veterans will disagree with me. Instead, failures nearly always go back to poor communication, murky goals, inadequate management, or mismatched expectations.  People issues, in other words.

So much for that admittedly standard observation. But as the old saying goes, “everyone complains about the weather, but no one does anything about it.” What, then, can we actually do to mitigate project failure that occurs because of these commonplace gaps?

Of course, that’s actually a long-running theme of this blog and several other key blogs that cover similar topics. (see my Blogroll to the right of this post). Various “hot stove lessons” have taught most of us the value (indeed, necessity) of fundamental approaches and tools such as basic project management, stakeholder involvement and communication, executive sponsorship, and the like.  Those approaches provide some degree of early warning and an opportunity to regroup; they often prevent relatively minor glitches from escalating into real problems.

But it’s obvious that projects still can fail, even when they use those techniques. People, after all, are fallible, and simply embracing an approach or methodology doesn’t mean that all the right day-to-day decisions are guaranteed or that every problem is anticipated.  Once again, there are no silver bullets.

One of the problems, as I’ve pointed out before, is that it can actually be surprisingly difficult to tell, even from the inside, how well a project is going.  Project management documents can be appearing reliably,  milestones met, etc.  Everything looks smooth. Yet, it may be that the project is at increasingly large risk of failure, because you can’t address problems you haven’t identified.  This is particularly so because the umbrella concept of “failure” includes those situations where the system simply won’t be adopted and used by the target group, due to various cultural or communication factors that have little or nothing to do with technology or with those interim project milestones.

Moreover, every project has dark moments, times when things aren’t going well. People get good at shrugging those off, sometimes too good.  Since people involved in a project generally want to succeed, they unintentionally start ignoring warning signs, writing those signs off as normal, insignificant, or misleading.

I’ve been involved in any number of huge systems projects, sometimes even “death march” in nature.  In many of them, I’ve seen the following kinds of dangerous “big project psychologies” and behaviors set in:

• Wishful thinking – we’ll be able to launch on time, because we really want to
• Self-congratulation – we’ve been working awfully hard, so we must be making good progress
• Testosterone – nobody’s going to see us fail. We ROCK.
• Doom-and-gloom fatalism – we’ll just keep coming in every day and do our jobs, and what happens, happens.  (See Dilbert, virtually any strip).
• Denial – the project just seems to be going badly right now; things are really OK.
• Gridlock – the project is stuck in a kind of limbo where no one wants to make certain key decisions, perhaps because then they’ll be blamed for the failure
• Moving the goal posts – e.g., we never really intended to include reports in the system. And one week of testing will be fine; we don’t need those two weeks we planned on.

An adroit CIO, not to mention any good project leader, will of course be aware of all of these syndromes, and know when to probe, when to regroup, when to shuffle the deck.  But sometimes it’s the leaders themselves who succumb to those behaviors.  And for people on the project periphery, such as other C-level executives? It’s hard to know whom to listen to on the team, and it’s definitely dangerous to depend on overheard hallway conversations: Mary in the PMO may be a perennial optimist, Joe over in the network group a chronic Eeyore who thinks nothing will ever work, and so on.  There are few, if any, reliable harbingers of looming disaster.

Wouldn’t it be great if there were some kind of codified, external measurement/evaluation tool that could methodically identify the kinds of disconnects that even well-led projects can fall prey to? One that could pinpoint where the true risk areas are as the project evolves, and help people take targeted action ahead of time to address those problem spots?

That’s why I got so excited in a recent conversation with well-known IT failure expert Michael Krigsman, CEO of Asuret, a company that sells “technology-backed services”.  He gave me a look at their forthcoming product, an impressively slick, well-engineered tool that in my view promises to provide exactly that kind of benefit: identifying where and why a project might fail in terms of some of those people/best practices aspects, before it actually does.

In a nutshell, Asuret facilitates a cross-sectional analysis of project participants and stakeholders as the project proceeds. By aggregating the answers to its carefully crafted questions and constructing a number of easy-overview summary charts, the tool then displays astonishingly insightful visual breakdowns that let you pinpoint major disconnects, such as between stakeholder groups and IT, or between actual project-specific and industry-best practices.

Let’s look at an example of what it shows you.  By mapping aggregated analysis results onto charted dimensions of importance and vulnerability, and slicing these charts by department, you can see at a glance in the chart below that there’s a disconnect: e.g., that executives think that the business case for the project has high vulnerability, while the IT participants view it as having low vulnerability.  Early warning sign!  And certainly better (more methodical, more aggregated) than relying solely on what you’ve heard Joe grumbling about in the lunchroom.

In the example, the disconnect looms large: look at the darker circle (representing the participants’ responses to questions regarding the project’s business case) and its different location on the two grids shown below:

Figure 1

This all sounds simple in this brief description, perhaps, but taken as a whole, Asuret’s methodical implementation and targeted, useful results are nothing short of groundbreaking. Perhaps other companies provide a similar product, but I don’t know of any. And frankly, I can’t imagine a better-designed or more perfectly suited product as Asuret to address the issues raised in this post.  I’m really looking forward to hearing more as they deploy and hone their product, because I can think of any number of large projects I’ve been on where this approach would have been revealing and useful.

It’s maybe not the ever-hoped-for holy grail, but it promises to be a small piece of it: an extension of our ability to see things before they happen.  If Will Rogers had been an IT guy, I think he would have been excited too.

How do we (IT executives) get away from being typecast as technologists, unconsulted on core business issues and approaches? Face it, that’s a common situation and dilemma that we all encounter, early and often, and it’s the grist for a constant mill of articles and blog posts and books on business/IT alignment.

Lately, though, a part of that mill has started insisting that focus on technology should be avoided altogether by what they usually cast as the “next generation” of CIO.  So I’m going to (again) be a bit of a contrarian here: it’s possible for the pendulum to swing too far in the wrong direction. I think that we can at times go overboard in our desire to avoid being seen as the geek with the pocket protector.  Examples: some preach outright denial that there might be such a perception problem: don’t even think of using the terms “IT” and “business”, they urge, and they recommend against ever discussing “alignment” as a goal.  Stop referring to the “business” as something separate, they recommend; IT is just as much part of the business as anything else! Similarly, their advice is “avoid discussing the technology itself.” As if a mere shift in language could solve the perception problem and automatically propel the CIO into the inner circle of decision-makers.

Here’s the gist of how I see it, though: in many (I daresay most) companies, the path of IT from high priesthood to strategic key playerdom has not really been fully traversed: in other words, greater alignment IS still needed of IT with “the business.”

In many companies, there’s still a trust problem. IT is arcane. IT can be as resented as a roadblock as the in-house corporate counsel types who veto certain kinds of corner-cutting.  Of course we’re “part of the business” (and increasingly so), but that doesn’t mean it’s wrong to continue to focus on breaking down the walls that linger.  The walls, in many/most places, are a reality, and it’s going to take steady, relentless, active work, not just a shift in language, to scale them. What especially will not help our effort to do that is if we embrace this odd trend of sneering at technology as a lesser-order concern—basically, positing that we should let someone else do that while we do the important (strategic) stuff.

In other words, I differ (vehemently) with those folks who now look down on such things as the need to align IT with the business, or who don’t recognize the benefits that come from regarding people as “internal customers”, or who recommend even avoiding using the terms “IT” and “business”. While the basic spirit behind those sentiments is of course valid—again, it’s undeniable that IT is part of the business, as much as any other department—it is in fact IT that has a problem in how it’s perceived by other elements of the company, going back to the high priesthood. Let’s be frank: IT folks have often been snooty, vague, cagey even, about what we do, why we do it, and how we’re meeting the company’s business needs as a whole. Deprecating the important progress and mentality shift that’s occurred in the last few years (that is, the high priesthood evolving to a service mentality) is exactly the wrong approach.

And somehow, inexplicably, the deprecators are getting louder, despite years of research and general agreement on the value of approaches such as IT governance, the ongoing need for IT/business alignment, etc.

Taken to an extreme, insisting loudly that IT is part of the business can extend to feeling that IT knows just as much about what’s good for certain business processes (say, manufacturing or shipping or collections or whatever) as the people executing those functions on a daily basis.  From there, it’s just a small but perilous step back into the old sort of IT arrogance, where systems were designed and deployed with little or no feedback from the actual users.  I exaggerate? Quite possibly, but I’ve seen it happen.

IT needs to both know its place and insist on its place, not one or the other. Here’s another example of a truly ham-fisted (even though well-intentioned) attempt to align IT more closely with business: I took over as CTO at a high-profile internet company, in the midst of deep, bet-the-company changes to its main product, a consumer-facing web site. I learned in my first few weeks that my predecessor had initiated a robot project.  His idea was apparently that he’d program this robot to walk the halls of the company, greeting people and thus making them feel more comfortable with technology. As Dave Barry is fond of saying, I am not making this up.  Needless to say, this project swiftly became a laughingstock across the company, representative of IT embracing technology for technology’s sake, completely OUT of alignment with business needs. Meanwhile, systems were crashing, projects delayed, etc.  This CTO somehow thought he was being strategic and business-oriented, but he actually was just being plain clueless, and more tech-focused than ever.

With IT’s background and history, the push towards IT/business alignment is ongoing and needs to be continual, rather than dismissed as something we’ve now evolved beyond. Even more, alignment mustn’t be deprecated as somehow beneath the higher goal of strategy. It’s not an either/or, and it most certainly doesn’t happen through mere declaration. And openly striving for alignment with our peers is nothing that anyone should present as a taboo or a sign of weakness. Additionally, at the same time that we serve as service providers to the company, we can and must be strategic in our outlook. This shouldn’t be either a surprise or controversial; in fact, excellence in tactical delivery is the ante that gets you to the strategic table.

It’s taken decades to get where we are—and I’d say (given, for example, the anecdote I cite above) that most of us are not really there yet anyway. Let’s not risk reinstating the “ivory tower” of strategy, devoid of the reality that comes from grappling with the actual pragmatics of implementation.

So, here are MY three tips on how to help convert your CEO’s understanding of where IT fits into the bigger picture. You’ll notice the emphasis of these points is quite different from the similar list in the piece I’m responding to here: in particular, my points stress a healthy balance between strategy and tactics, technology and business.

• Make sure there’s a rock-solid prioritization mechanism for all projects, where the priorities are clearly set, the tradeoffs carefully examined, and tough choices explicitly made and regularly revisited by the group of senior management, including the senior technology executive, who are jointly responsible for the business success of the company.  Focus on IT governance, Project Portfolio Management, etc.

• Speak up visibly and strongly to ensure that “roof projects” are given appropriate attention and priority. Build for the day after tomorrow rather than just for tomorrow. You in IT are a key advocate for this sort of basic corporate responsibility; you have and need to exercise your equal seat at the table.  And don’t let a misguidedly monomaniacal focus on strategy derail your obligation to keep the trains running on time. Nothing undermines the case for IT to be viewed as having a greater strategic role more than if the company is experiencing ongoing crises in basic delivery and systems.

• Be the straw that stirs the drink. IT, given a leader with appropriate vision and perspective, can and should be a natural cross-functional leader in the enterprise, both initiating business innovation AND espousing tactical responsibility.  They go hand in hand.

Let me end by quoting my colleague Steve Romero, who wrote, “Recognize that the “achievement” of alignment does not mean you’re done. This is where the “journey” analogy comes in. Think of IT-Business Alignment as “getting your head above water.” Once you reach the surface, you are not done. You need to keep treading water or you sink again. IT-Business Alignment is something that must be maintained as opposed to achieved.”

Lagniappe:

• Michael Krigsman, “IT failure? Blame your CEO.”  February 16, 2010
• Steve Romero, “IT-Business Alignment is Not a Meaningless Catchphrase“, October 28, 2009
• Michael Pattison, “Is IT – Business Alignment Meaningless?“, November 6, 2009
• Bob Evans, “Global CIO: Suicide Strategy For CIOs: Aligning IT With The Business“, October 13, 2009
• Fred Cummins, “Is Business-IT Alignment Suicide for the CIO?“, October 21, 2009
• Mary Nugent, “The Four Phases of IT/Business Alignment“, December 10, 2004
• Paul A. Strassman, “What is Alignment?  Alignment is The Delivery of the Required Results“, Cutter IT Journal, August 1998

Anyone ever tell you that a simpler approach can often work better than a more complex one? Whoever it was, it probably wasn’t a project management software vendor.  But simplicity has its merits, and I’d like to point out a few of these when it comes to resource allocation.

Project management, at its core, is largely about resource allocation, and this gets tricky when you have multiple projects going on, as most organizations do. Almost as much as I’ve seen organizations drop the ball entirely on cross-project resource allocation (essentially, simply pretending that there will be no contention issues), I’ve seen organizations go to the other extreme: they dive into the depths of intense Project Management, in capital letters: taken too far too fast, this approach can spin up to a high level of rigor and overhead, involving often-expensive software packages, precise low-level estimates, diligent collection of actuals, and ornate project calculations of hours burned and hours earned.  At the end, there you stand, like Goethe’s Faust, “no wiser than before.”

Occam’s Razor

Let’s talk first about the general problem of resource allocation, then, and the range of possible solutions.  First off, I’ll note that running a successful IT department, one that delivers predictably and with high quality, depends on far more than just being able to run an individual project successfully.

A major part of overall success in the IT arena comes down to whether (and how) you’re allocating the right (and enough) resources to the right tasks at the right time. That’s obviously true at a project level, but it’s especially true across projects, as you juggle multiple goals, diverse timelines, and random delays anywhere along the line.

Project management is relatively easy, in other words. The hard part is PROJECTS management. As I’ve pointed out before, the most common and glaring problem I see in companies with IT in crisis relates to an unfortunate tendency to estimate and commence projects one by one, with little thought to resource contention issues. That approach almost always leads to chronic overcommitment and late delivery.

So how do you do resource allocation, in a practical sense?  I’m talking about determining contention issues, not task-by-task assignments.  That’s where organizations often drop the ball: as I said, they either ignore the problem almost completely (allocating resources essentially by a “seat of the pants” approach), or they go whole-hog into major package adoption and process overhaul. But the seemingly greater precision and granularity of such packages does not guarantee greater accuracy or efficacy, and such organizations can easily founder as fast and as much as organizations that punt completely.

Surprise: there’s no silver bullet

My stance? There’s no ONE answer for what’s the best practical way to do resource allocation in-the-large. It depends. It depends on your organizational politics. It depends on the maturity of your overall software development lifecycle process. And most of all, it depends on you, and the level of your desire/appetite for attaching a degree of rigor to what is by nature an ever-swirling hodgepodge of resources, tasks, projects, deadlines, dependencies, delays, resets.

The Project Management Book of Knowledge (PMBOK, highly recommended) identifies nine different areas of focus, each of which is fairly elaborate. Correspondingly, project management tools cover a spectrum of functionality. You don’t just start using “project management software”; you need to decide, initially and all along the line, just how deep and how broad you want to go.

In fact, the areas of project management and resource allocation can be integrated with task management, reporting, document management, financial projections, collaboration tools, email notification, management escalation, risk evaluation, etc. The list of potential useful and desirable features stretches long.

This results in a futile quest for a kind of Unified Field Theory of IT and project management, where everything can be fruitfully connected to and integrated with everything else. The irony is that while tools approaching this goal do exist, and are getting better all the time, implementing them successfully can be not only very expensive, but is also a massive and risky enterprise-level project in and of itself, both in terms of initial implementation and in ongoing care and feeding.  And the outcome? Well, an awful lot ends up being “tightly coupled“, which can lead to problems in and of itself.

These different tiers of solutions are for vastly different problems, as orthogonal as bicycles are to trucks. Yet the persistent underlying and increasing belief remains, as you get more and more ornate with the tool you use, that project management can be wholly deterministic, that you’ll get to the point where the data will predict the precise end date, everything will line up, and so on.

It can’t. It won’t. It’s an illusion. Project management, and especially resource allocation, consists of constantly revisiting and revising the plan, not trying to get absolute certainty about every aspect of it at all times.  Rather, the successful IT executive learns to live with ambiguity, uncertainty, approximation, and to work within them. Yet, the spectrum of project management software holds out the (false) promise of eradicating much of that uncertainty.

Tools for resource allocation

Let’s look briefly at the range of project management solutions that you might fruitfully consider, with respect to the all-important aspect of resource allocation across projects.  This list is a cursory overview, of course, and is not intended to be comprehensive in any way. For further information, consult the links in my Lagniappe section at the bottom of this post.

• Seat of the pants. Assign work when people seem to have some available cycles.

Pros: simple, intuitive

Cons: completely intuitive, and usually doesn’t scale across multiple projects. Unreliable.

• Build an “approximating” spreadsheet that lets you get a quick overview of coarse-level resource allocation for your portfolio of projects. See example below.

Pros: quick overview, relatively easy maintenance (e.g., monthly)

Cons: requires spreadsheet expertise, and understanding that these are rough estimates rather than precise allocations.  Limited scale for large organizations; doesn’t directly feed actuals/results into further planning.

• Installed software packages: e.g., MS Project, OpenProj, Primavera.

Pros: fine products, broad functionality, well-known

Cons: Getting a good, easy overview of resource contention is difficult, plus requires constant care and feeding. With this approach, I’ve seen projects that needed to assign resources to do nothing but work the project management software

• Web-based products: e.g., Serena, Projects On Demand, Zoho Projects, Innotas, Project Insight, attask, Daptiv.

Pros: This rapidly evolving market niche is centralized (SaaS) and innovative.

Cons: has same issues as installed packages. On a larger scale, becomes a huge business change management project to implement

• Large packages: CA’s Clarity, Changepoint, etc. (PPM)

Pros: enterprise-class, high level of functionality and integration.

Cons: high implementation cost in dollars, time, and resource commitment, up-front and ongoing

Here’s an example of an “approximating spreadsheet”, where resources are roughly allocated day-by-day to any of nine simultaneous projects (numbered 1-9, and each color-coded).  Note that this is a quick-and-dirty planning/working document, meant to do coarse allocation planning at the start of a month, with actual assignments and microadjustments made as the month progresses. Its principal advantage, beside pure simplicity, is that it quickly forces the planners to confront the ever-present “NUTS” problem, where there are too many projects underway at once for the available resources.

I’ve had great success with the approximating spreadsheet approach in a “bang for the buck” sense, in smaller (less than 100 people in IT) organizations. Once you wrap your brain around it being just an approximation tool, it gets you 80-90% of the way there without having to devote a large portion of your staff’s time to the constant and intense administrivia often necessitated by one of the bigger project management packages.

But those packages do have their place. In short, though, here’s the key ongoing challenge for project management software adopters (and, of course, vendors): how do you obtain the necessary functionality across the vast spectrum of functionality, without turning everyone into a project data entry monkey? Your project management software needs to be a tool, not a torture device, not a time sink.  My argument is that your key goal, in a cross-project sense, is to roughly allocate your resources so that you avoid major contention, and/or taking on too much.  And you want to focus on this with the minimum amount of overhead you can, because at an extreme, it will suck up virtually all your time.

What is it that sets apart a top-notch IT executive from others of his calling? To my mind, one mark of today’s true professional, especially at the senior executive level, is to be deeply familiar with the seminal books in his or her field. The dilemma for an IT professional, though, comes from the ongoing and increasing flood of books to choose from, and trying to figure out how to walk the fine line between focus on the intensely tactical and focus on higher-level concepts and ideas.

The tactical books do have their place on your shelf, actually, and it would be a mistake to ignore them simply because you’ve moved beyond daily application of your development, configuration, and technical trouble-shooting skills: judicious selection and absorbing of nuts-and-bolts techniques and new approaches will keep your insight into technology and its possibilities fresh.

I started in IT as a developer, and I remain fascinated by the endless possibilities and techniques of the world of software. In the last decade or two, though, I’ve become even more intrigued by a metalayer above the more tactical concerns. True to my ongoing insistence that the biggest challenges in IT aren’t purely technical, I am ever more convinced that the greatest difficulties are presented by “psychology of IT” issues: the human factors in how software and systems are conceived, built, tested, deployed, maintained, and eventually decommissioned.  Here are just a smattering of the eternal, non-technical questions that go far beyond the computer language du jour or the latest hot methodology:

• How do teams actually create and complete information technology projects? What works, what fails, and why?
• Why are some software developers ten times as productive as others?
• Why do some software teams gel and others don’t?
• Why do small companies with very few resources often beat out large, well-funded efforts in the marketplace?
• How technical should managers be?

So starting with this post, let’s embark on a multi-part survey of the groundbreaking, timeless books on such issues. I’m going to pick what I consider to be the top three books from each decade, starting with the 70s.  Each of them deserves not only a place on your bookshelf, but to be read and reread every few years. And contrary to what one might think, their insights remain not only valid after all these years, but have become all the stronger by having been confirmed by the history of the industry since their publication.

• Weinberg, The Psychology of Computer Programming (1971)

Weinberg opens with, “This book has only one major purpose—to trigger the beginning of a new field of study: computer programming as a human activity…. If our experiences are any indication, each [software developer] could be functioning more efficiently, if he and his manager can learn to look upon [him] as a human being, rather than as another one of the machines.”  This book was especially groundbreaking by addressing software development as both an individual and a team effort. Remarkably readable and full of anecdotal examples, it covers virtually every human aspect of the software development “sausage factory”: team formation, individual contributions, goal setting, leadership, estimating.  Weinberg’s “Silver Anniversary” edition of this book contains annotations and commentary for each chapter, reflecting on similarities and differences to today’s environments.

• Brooks, The Mythical Man-Month (1975)

I’ve said it before: if there’s one single book that every IT professional should read, it’s this one. Brooks’ foreword presents it as a “belated answer to Tom Watson’s probing questions as to why programming is hard to manage.”  Brooks’ writing brims with key universal concepts that can be unfortunately non-intuitive to people outside the field, but which ring familiar to any long-time IT practitioner. The titular essay introduces the concept of the “mythical man-month”, his adage that “adding manpower to a late software project makes it even later.” But there’s much more, such as “plan to throw one away”, where Brooks points out that the “first system built is barely usable; … there is no alternative but to start again.” I can’t say enough about this book. As with the Weinberg book, the anniversary edition now being sold includes Brooks’ later (1986), equally seminal essay, “No Silver Bullets” (see my post on this: “No Silver Bullets. Really!“), as well as some added chapters that revisit the assumptions of the first edition.

• Weizenbaum, Computer Power and Human Reason (1976)

Ultimately, this is a book about the limits of computers and software. Weizenbaum, a noted computer scientist in the early days of Artificial Intelligence research, wrote a program called ELIZA, featuring a script that “enabled it to parody the responses of a nondirective psychotherapist in an initial psychiatric interview.”  In other words, his test subjects would converse with a software program that emulated a therapist. Weizenbaum became horrified by how many people forgot that this was “just” a machine they were talking to, and that the dialog was really just an illusion. Many insisted, to his amazement and despite his explanations, that the computer actually understood them.

IT professionals today can be just as swept away, to a fault, with the potential ultimate power of software and systems as Weizenbaum describes.  I’m inspired and reinvigorated every time I read his sobering, methodical discussion of the nature of programming, the limits of its scope, and the need to consider the social implications of technical projects.

The next time, I’ll be on to the key books of the 80s. I’ve already picked the books I tentatively plan to write about, but I welcome your suggestions.

Lagniappe:

Here are a couple of books that didn’t quite fit in my theme and chosen time frame, but which are still worthy of mention:

• Thomas Kuhn, The Structure of Scientific Revolutions (1962)

Nobel Prize-winning physicist Steven Weinberg wrote that this book “has had a wider influence than any other book on the history of science.”  This book is not about information technology directly, but its influence has been monumental across all scientific disciplines, and it is a book that any technologist should know well.

• Ted Nelson, Computer Lib/Dream Machines (1974)

This was a self-published book in the early 70s, by influential industry visionary Ted Nelson, the man who coined the term “hypertext.” It’s probably different from just about any book you’ve ever read.

Fred Brooks wrote a seminal essay in 1986, “No Silver Bullet — Essence and Accidents of Software Engineering“, a model of clear and cogent thinking that I consider to be required regular reading for anyone involved in information technology.  Despite the essay’s brilliance, and despite its wide promulgation and deserved fame, the phenomenon it describes seems to have only broadened in the last twenty-three years.  Brooks argues as follows (with bolding added):

“The familiar software project, at least as seen by the nontechnical manager, has something of this character; it is usually innocent and straightforward, but is capable of becoming a monster of missed schedules, blown budgets, and flawed products. So we hear desperate cries for a silver bullet—something to make software costs drop as rapidly as computer hardware costs do.

…

“There is no single development, in either technology or in management technique, that by itself promises even one order of- magnitude improvement in productivity, in reliability, in simplicity.”

So this basic tenet has been convincingly articulated by a leading IT thinker for almost a quarter century. Yet, the trend continues: new technologies pop up every couple of years and the hype cycle begins. Evidently, hope springs eternal.

IT and an inclination towards zealotry have always tended to have a high correlation, but one thing I find interesting about the incessant quest for a “silver bullet” solution is that the impetus for that quest often stems from non-IT senior management. They long for something, anything, which will take this risky, recalcitrant technology beast and tame it into predictability.  As a result, we see notable IT thinkers respond to that pressure, often by championing one or more specific practices, methodologies, approaches, or technologies as “the answer.”  Many sorts of silver bullets have come and gone in the decades since Brooks’ essay. The one constant seems to be that there are always a few memes out there being touted as cure-alls. Why haven’t we learned?

Let me gingerly pose the empirical evidence of this “quest for the cure-all” meme that is provided by my Twitter stream every day.  I see a number of IT tweeters who have doggedly taken up the flag of a particular movement or approach; in some extreme cases, virtually every tweet and comment from those people views all conceivable issues through that one lens.

Let’s list some recent candidates for the ongoing “silver bullet glomming” that I see, with some sample quoted hype for each one.  My Lagniappe section at the end of this post, for your amusement, will provide additional links to sites decrying the hype in each case.

Examples:

• Agile software development

“Agile Development Can Lead to Better Results and Technology-Business Alignment”

• Cloud computing

“Cloud computing heralds an evolution of business that is no less influential than e-business, according to Gartner Inc.”

• Project Portfolio Management

“Multiplying ROI at Warp Speed”

• IT Governance

“Firms with superior IT governance have more than 25% higher profits than firms with poor governance given the same strategic objectives.”

• Business alignment above all: remove technology from purview of CIO

“There are no IT projects”

“Only once we stop having IT projects and start having business projects will we see the full value of project management.”

• Eliminate complexity: a simpler architecture is the key

“The proliferation of IT failures is caused by increasing IT complexity. And this is good news, because complexity is a solvable problem.”

• Information Technology Infrastructure Library (ITIL)

“From small organizations to multinational enterprises and anything in between, this best practice framework has helped many improve efficiencies and bottom line figures, putting IT back in business!”

• Ruby on Rails (or, .NET, or J2EE, or Smalltalk, etc.)

“Ruby on Rails is a breakthrough in lowering the barriers of entry to programming. Powerful web applications that formerly might have taken weeks or months to develop can be produced in a matter of days.”

• Design Thinking

“It’s a technique that designers and executives alike hope may help to provide a solution to some of the world’s serious challenges.”

• Service-Oriented Architecture (SOA)

“SOA provides benefits in four basic categories: reducing integration expense, increasing asset reuse, increasing business agility, and reduction of business risk.”

• Web Services

“Experts and visionaries believe that the benefits of XML Web services will be instrumental in propelling explosive business growth over the next few years.”

It’s important that I note here that none of these theories, approaches, or technologies is necessarily a bad idea (although I think that each does tend to land at quite different points on the spectrum of goodness). There are some, in fact, for which I myself have major enthusiasm, because I think they promise to relieve at least portions of the IT delivery crisis.

The key takeaway, I’d argue along with Brooks, is that there’s no one technique, approach, methodology, philosophy, or magic that will wondrously make systems work easy and reliable.  No matter what the tools and techniques used, it’s hard work. It’s rife with opportunity for failure. It requires leadership, detail-oriented personnel, skillful practitioners.  Brooks took pains to say,

“Skepticism is not pessimism, however. Although we see no startling breakthroughs–and indeed, I believe such to be inconsistent with the nature of software–many encouraging innovations are under way. A disciplined, consistent effort to develop, propagate, and exploit these innovations should indeed yield an order-of-magnitude improvement. There is no royal road, but there is a road.”

So, we all have our silver bullet, I suppose: mine, along with Brooks’, consists of maintaining balance, practicality, and a broad embracing of multiple approaches, rather than one panacea.

Lagniappe:

• Peter Merholz, “Why Design Thinking Won’t Save You”
• Jerry Iacouzzi , “Debunking the myth of SOA”
• Matthew Huntbach, “What’s Wrong With Ruby?”
• Rob England, “The Emperor has no clothes. Where is the evidence for ITIL?”
• Colin Beveridge, “Biggest IT Myth of all time?”
• Steve Yegge, “Good Agile, Bad Agile”
• Cath Everett, “Five cloud computing myths exploded”

“Don’t write about that,” I’ve been told by several colleagues, when I’ve mentioned that I was working on a post about how best, as the senior technology executive, to negotiate with vendors.  “You’ll give away all your tricks!” they’ve said.

Well, actually, no.  Here’s the main trick: this particular CIO doesn’t have any “tricks”, if by tricks you mean ways to outfox the opposition, or anything else that is best kept secret.  In fact, I’m not a natural avid negotiator. I’m not one of those people who looks forward to buying a new car because of the thrill of haggling with the salesperson.  But I’ve learned over the years how negotiations can best be structured for the optimal outcome.  Like cryptography, where greater obscurity isn’t equivalent to greater security, successful negotiation isn’t dependent on tricks or subterfuge.

I’m quite content to tell any vendor or salesman how I go about negotiating, because doing so doesn’t provide them any kind of advantage.  If anything, it’s beneficial to me and my company that all parties in the negotiation understand clearly the basic principles and approach that I’m using; it cuts a lot of the normal gamesmanship out of the equation.

What I’m about to tell you about negotiating actually strikes me as incredibly obvious. Yet, time and again, I’ve come into a new CTO/CIO role and been astounded at the poor technology deals that were cut by the previous executive: inadequate terms, failure to protect both parties instead of just the vendor, unnecessary and undesirable contractual lock-in.  What happened in most of these cases, as far as I can usually determine, is that the executive in charge got swept away by emotional attachment to a specific solution, and tipped his or her hand. Game over: at that point they were putty in the vendor’s hands. I’ve even seen cases where contracts have completely put the vendor in the driver’s seat of actual implementation, including spending authority, to an unconscionable degree.

Make no mistake: vendors are typically much more experienced and skilled in negotiations than most of the technology executives they deal with.  Achieving excellence in negotiation should therefore be seen as a critical part of the technology executive’s skill portfolio.  But, like finance skills, it’s a skill more honored in the breach than the observance.  Your ability to negotiate well can not only protect your company’s long-term interests, but can actually recoup far more than the equivalent of your annual salary to the bottom line. It’s not uncommon for technology executives at even relatively small companies to negotiate deals worth hundreds of thousands or even millions of dollars of capital and operating expense.

So let’s cut to the “secret”. It boils down to this simple principle: give yourself choices. Identify two, preferably three, viable, affordable, achieveable ways to solve your problem or need.  Once you’ve successfully established that viable short list, the power is all in your hands.  If you haven’t fully done that, you’re not really ready to negotiate, so you shouldn’t even start. Power in negotiation comes from not being wedded to a particular solution. Another way of expressing this idea is “always be truly ready and willing to walk away.”

Pay attention here: I’m about to say two apparently contradictory things, both of which are true and key:

1. You need to regard all of the choices on your short list as fundamentally on equal footing, each with its own advantages and disadvantages, but on balance having roughly the same merit as a solution.  Don’t ever go in having already decided which one you really want!
2. Yet, you need to remember that all of the choices are also fundamentally different in their details. For example, one might have 30% greater functionality or come from a more established and stable company than the others, but recognize that this advantage probably comes with a cost premium. The market is generally pretty efficient that way.

Your goal in the negotiation process is to see where you can find “wiggle room” on each alternative’s components (price, terms, features) that will ultimately make one of those choices outshine the other, on balance.  In other words, you are looking to unsettle the equilibrium you’ve achieved in #1, so that one choice comes out on top. And here’s another key: let each vendor know, and remind them frequently, that this is what you’re doing. Usually, there’s no need (or benefit) in letting the participating vendors know exactly what your list of alternatives consists of, but it will suffice that they know that you know that you have choices.

Miscellaneous tips and observations:

• Negotiation, in general, is not simply about money, although that’s certainly a key dimension that can make one of your viable alternatives rise above the others.  Any facet of the deal should be open to scrutiny and discussion: cost, terms, rights to upgrade, SLAs, warranty, etc. You don’t always automatically choose the least expensive option, nor do you always choose the most featureful.

• Don’t ever tell vendors (and believe me, they’ll ask again and again) what your budget is for this project. They don’t tell you what deals they’ve cut when they’ve sold their product to other customers, do they?  Again, the fact that you are actively pursuing other alternatives needs to speak the loudest.

• Remember, though, that your goal is not to hammer down the price so far that the vendor won’t make any money on the deal. You can do that, at times, but it ultimately means that you’ll get poorer service and less attention when you need it.

• Where possible, involve your legal counsel in the negotiations; don’t just bring them in at the end. The best negotiations for me, in terms of viable outcome, were where I had a tight bond and common approach with my legal counsel.

In the end, it’s all about getting a viable solution for your need, at a reasonable price and beneficial terms for all. It’s not really about “winning”; it’s about choosing. Give yourself choices, and in the end, you (and your organization) do win.

Lagniappe:

• Ward Keever, “Ten Keys to Successful Negotiations”
• Martin Ewing, “5 Can’t-Miss Vendor Negotiation Tips”
• Michael Krigsman, “Negotiation Tips for CIOs”
• Peter Kretzman, “More tips for dealing with IT vendors“

A few years back, I had an extremely surprising and unpleasant experience as CTO. The director of my Program Management Office ran a weekly status meeting for project stakeholders, where we’d all methodically go through the current project portfolio, in order to communicate on issues, gather necessary feedback, and align everyone’s expectations. I typically attended in order to provide input and executive-level decision participation, but left it to the director to actually run the meeting and present the topics.

Unfortunately, immediately before one of these weekly meetings, that director was given bad news (in a brief hallway conversation, no less) about a major bug that had just been discovered in the software for our highest profile project, which was currently in testing and due to launch in just a couple of weeks.  This project, with its strategic and revenue-enhancing potential, was foremost in the minds of everyone in the company.  Stakes were high, in other words.

Without conferring with me or anyone else, the director announced to the entire group of assembled stakeholders that the project had hit an insurmountable snag in terms of our target launch date, and that it would be delayed potentially by weeks. No further information was available on specifics, and people’s concerned questions went unanswered. You could feel the anxiety spread through the room. I actually had to truncate the meeting abruptly and promise to reassemble the group when we knew more.

My director was quite upset with me. Rather than having my action be understood as a necessary attempt to prevent panic until we could get some real facts, I suddenly realized that I was being accused of wanting to “hide information” from the stakeholders. The director even told me, “Well, I like to be honest and tell it like it is, but that obviously just isn’t what is wanted here.” I was stunned. Here was a person who had the wrong idea of transparency, and astonishingly for their position, a warped idea of what project management is all about.

As most experienced project managers know, it makes utterly no sense to drag stakeholders through the entrails of every back-and-forth readjustment that is discussed and worked through in the course of a project’s lifetime.  To do so lowers stakeholder confidence, unnecessarily in many instances.  In this case, the director somehow wanted to be a kind of Paul Revere, forewarning the stakeholders, but instead came off as Chicken Little. Transparency can’t be an act of individual heroism; it needs to be a concerted, institutionalized, collaborative philosophy and approach.

Moreover, good project management isn’t about making everything go smoothly. There’s probably no project plan in the history of the world that has gone completely smoothly from beginning to end.  Instead, project management is about responding appropriately when it doesn’t, and recalibrating to keep everything still on track. In fact, it should be expected that not everything will go as planned, and your whole team should be prepared to take some bumps and glitches in stride without panicking.

It’s not hiding anything to hold back on discussing, in a public forum, a problem that has just been discovered, the ripple effects of which initially can only be guessed at.  When you promulgate potential bad news prematurely, it puts you into the position of trotting out the absolute worst case, before you really have any idea whether that worst case is likely or even plausible. In fact, it’s a fundamentally selfish act: it takes your own personal panic and extends it to as many people as possible.  And if there’s one thing we know about panic, it’s that it’s contagious.

You need to shoot for judicious and organized transparency. That’s where you plan on regular status communications, and you methodically, soberly assemble the facts in a structured manner before you make those announcements. Most of all, you collaborate internally: before you bleat about a potential crisis to the stakeholders, you bring in the rest of your team to talk through the possibilities and “degrees of freedom” in how you’ll act with respect to that crisis. Otherwise, you may think you’re nobly playing Paul Revere, but instead of imparting critical facts, you’re shouting “The British might be coming!”  And that’s not transparency, that’s mere alarmism.  Remember that especially during the dog days of a stressful IT system development effort, what often seems like a dire setback, upon careful and collaborative examination in the light of day, turns out to be merely a minor speed bump that is easily overcome.

Is it “controlling the message” to be cautious about spreading the alarm? You bet it is. And doing that is in everyone’s interest. Remember my adage: show that you’re in control. Your job as project manager (and, by extension, the job of the Director of the PMO, and the CIO) is to be in control and to demonstrate to the users that you’re in control.  That doesn’t mean that you “hide” anything; it does mean that you deliver bad news broadly only when you’re sure of your ground, have weighed the options for how to cope with that bad news, and are ready to present those options soberly instead of in a flailing manner.

Adhering to a philosophy of transparency is a far cry from shouting every blip or misgiving about the project from the rafters. This is particularly so if you’re in a leadership position. You want your communication to be transparent, yes. But without appropriate structure and caution, transparency can be the height of irresponsibility. In those cases, in fact, you’re just building a slightly more institutionalized rumor mill.

Lagniappe:

• Karen Guglielmo, “What is transparency, and how is it relevant to IT?“
• Phil Windley,  “Tools for IT Transparency“
• Peter Kretzman, “Why the CIO should air the dirty laundry“

Roger Sessions recently published a white paper on IT complexity and its role in IT project failure: “The IT Complexity Crisis: Danger and Opportunity”.  It’s certainly possible to quarrel with bits and pieces of his analysis, and thereby tweak his numbers, but the overall thrust remains undeniable: IT failures are costing the world incredible amounts of real money. Sessions even sums it up under the dire-sounding phrase, “the coming meltdown of IT,” and says, “the out-of-control proliferation of IT failure is a future reality from which no country—or enterprise—is immune.” And he presents “compelling evidence that the proliferation of IT failures is caused by increasing IT complexity.”  He points out that the dollar cost of IT failure in the US alone is close to the cost of the recent US financial meltdown, and cites indications that the rate of failure is also increasing by 15% per year.

Roger’s paper is excellent and thought-provoking, and I recommend it highly. And I do agree with his view that complexity is the chief culprit in IT failure. That said, I think his argument focuses a little too strongly on one cause of complexity (unnecessary overcomplexity of architecture), to the neglect of other important factors.

To be sure, some obvious contributors to IT failure (poor project management, and lack of communication within teams and from business to IT implementers) aren’t dismissed by Sessions, but he sees their contribution to the crisis as relatively small. I don’t, and I’ve used this blog to write about those factors quite a bit.

Most of all, though, I differ with Roger’s focus on streamlining architecture as being the key to reducing system complexity. One could say, in fact, that Roger’s solution is primarily a technical one, where the bugaboos I see are primarily cultural and sociological.  I see not one, but at least three distinct complexity-related burdens, increasingly endemic, and increasingly bringing down IT:

• Overly complex design/architecture
• Taking on too much functionality
• Poor implementation (technical debt in-the-large and in-the-small)

Roger has admirably dealt with the issue of overly complex design/architecture, at least in terms of a viable approach for simplifying up-front architecture, so I’ll focus here on the other two.

Taking on too much functionality

I recently rented an economy car (the least expensive option) on a trip with my son. (Remember, my self-appellation is “Cheap Technology Officer.”) He was stunned and dismayed that the car didn’t have automatic door locks; he didn’t realize that they even made cars without them anymore. Similarly, my 11-year-old daughter has grown up in a TiVo-ized world where live TV can always be paused. When she encounters a TV without a DVR (and thus no pause capability), she regards it as hopelessly primitive. Indeed, as unacceptable.

Similarly, the general standard for functionality and UI design has been raised by extremely functional PC software.  I now expect to be able to double-click on a number in any onscreen report, and thus “drill down” into the transactional details that make up that number. When I can’t, I feel cheated. Equally, I expect everything on an interface to drag and drop; I get frustrated if it doesn’t.

So as an industry, we’ve raised the bar of acceptability, considerably, in software and technology systems over the last couple of decades.  What that means in practical terms, though, is that across the board, our eyes have gotten bigger than our stomachs. We want more, up front, than it often makes sense to build at the start. And our demands are not negotiable, or so it seems. The first few cell phones I had didn’t even have a ring silencer on them; I used to silence the phone by adroitly disconnecting the battery when a call came through at an inopportune time. Today, most people wouldn’t even consider buying a phone that lacks much more elaborate features, such as a camera, that I would have considered as space-age in nature back in the 80s.

So our increase in expectations, alone, has added considerably to the functionality of systems we tend to build. There’s more functionality in and of itself, and usually more interface points to other complex systems. In fact, integration testing—where you connect new code into a working environment where it has to interface correctly with other systems—has become a frequent and major sticking point in launching information technology projects.  In essence, we’ve fallen into the “nuts” dilemma, both in large and small ways.  We want so much, and attempt so much, that we increase our risk of failure considerably.

Technical debt (in the large and in the small)

Any software developer will tell you that their first stab at implementing a given piece of functionality is often (if not usually) much more complex than turns out to be needed. Only after exploring the problem domain, with experiments and backtracking and restarts, do developers usually realize that their code can be pared down, simplified, combined with other modules, etc. This is usually called “refactoring”, and its importance is a relatively recent insight in the software development discipline.

A key insight about refactoring, though, is that it means improving the code without changing its overall results. There’s often no immediately obvious payback to this undertaking: it’s a roof project, in essence. To the extent that refactoring isn’t done (no time, no inclination, no recognition of a simpler approach), the end product is left with vestiges of unnecessary complexity. The greater the time crunch, and the greater the aspiration for the functional depth and breadth of the software to begin with, the more likely it is that these vestiges linger. And one ancillary aspect of the raised bar in expected minimum functionality is that it causes the time crunch to get ever greater.  It’s no longer about delivering just a solution that will work, it’s about making sure that the solution includes (metaphorically speaking) a 5-megapixel camera too.

Couple this unnecessary but accidental complexity with what often amounts to a rush job on design for the sake of meeting schedule (creating “technical debt in-the-large”). An example I’ve used here before: choosing a different core DBMS to implement a given function, simply out of expedience, and failing to take time to modify previous functionality to use that new DBMS. Supporting two DBMSes within the same product represents significant technical debt: every subsequent system change and addition will entail “paying interest” on that debt, which not only increases schedule and manpower costs, but increases risk of failure as well. And that’s just one example; the technical debt cascades, feature upon feature, release upon release. Technical debt, until paid down, can be equated to a invisibly rising substrate of complexity, and it contributes massively to an increasingly wobbly, risky system. And lately, I see more and more organizations “pyramiding” their technical debt, never taking the time and cost to pay it down, with disastrous results. As Hemingway said about going broke, it happens slowly, then all at once.

What now?

Roger’s analysis, to its large credit, outlined an important aspect of complexity and posed a solution (an approach he calls SIP, or Simple Iterative Partitions).  The aspects that I’m presenting (taking on too much functionality, and the pyramiding of technical debt) are, as I’ve said, cultural and sociological within companies. The answer is not nearly as simple or as neat as a specific technical solution (although I am certain that I will get comments on this post, perhaps rightly, from devotees of Agile).

To my mind, it’s engaged, savvy, forceful leadership that alone can address these issues, slow down the demand train, stop the madness. If anything, I think that there is an increasing lack of leadership in IT circles that can suitably recognize and address these factors, as well as educate their peers. And that’s what needs fixing most of all.

Lagniappe:

• Kevin Schlabach, “Agile East 2009 by Thoughtworks”
• Martin Fowler, “Technical Debt“
• Eric Ries, “Embrace technical debt”
• Johan Lindberg, “Painting With Hands and Feet: The IT Complexity Crisis”