“Don’t write about that,” I’ve been told by several colleagues, when I’ve mentioned that I was working on a post about how best, as the senior technology executive, to negotiate with vendors.  “You’ll give away all your tricks!” they’ve said.

Well, actually, no.  Here’s the main trick: this particular CIO doesn’t have any “tricks”, if by tricks you mean ways to outfox the opposition, or anything else that is best kept secret.  In fact, I’m not a natural avid negotiator. I’m not one of those people who looks forward to buying a new car because of the thrill of haggling with the salesperson.  But I’ve learned over the years how negotiations can best be structured for the optimal outcome.  Like cryptography, where greater obscurity isn’t equivalent to greater security, successful negotiation isn’t dependent on tricks or subterfuge.

I’m quite content to tell any vendor or salesman how I go about negotiating, because doing so doesn’t provide them any kind of advantage.  If anything, it’s beneficial to me and my company that all parties in the negotiation understand clearly the basic principles and approach that I’m using; it cuts a lot of the normal gamesmanship out of the equation.

What I’m about to tell you about negotiating actually strikes me as incredibly obvious. Yet, time and again, I’ve come into a new CTO/CIO role and been astounded at the poor technology deals that were cut by the previous executive: inadequate terms, failure to protect both parties instead of just the vendor, unnecessary and undesirable contractual lock-in.  What happened in most of these cases, as far as I can usually determine, is that the executive in charge got swept away by emotional attachment to a specific solution, and tipped his or her hand. Game over: at that point they were putty in the vendor’s hands. I’ve even seen cases where contracts have completely put the vendor in the driver’s seat of actual implementation, including spending authority, to an unconscionable degree.

Make no mistake: vendors are typically much more experienced and skilled in negotiations than most of the technology executives they deal with.  Achieving excellence in negotiation should therefore be seen as a critical part of the technology executive’s skill portfolio.  But, like finance skills, it’s a skill more honored in the breach than the observance.  Your ability to negotiate well can not only protect your company’s long-term interests, but can actually recoup far more than the equivalent of your annual salary to the bottom line. It’s not uncommon for technology executives at even relatively small companies to negotiate deals worth hundreds of thousands or even millions of dollars of capital and operating expense.

So let’s cut to the “secret”. It boils down to this simple principle: give yourself choices. Identify two, preferably three, viable, affordable, achieveable ways to solve your problem or need.  Once you’ve successfully established that viable short list, the power is all in your hands.  If you haven’t fully done that, you’re not really ready to negotiate, so you shouldn’t even start. Power in negotiation comes from not being wedded to a particular solution. Another way of expressing this idea is “always be truly ready and willing to walk away.”

Pay attention here: I’m about to say two apparently contradictory things, both of which are true and key:

1. You need to regard all of the choices on your short list as fundamentally on equal footing, each with its own advantages and disadvantages, but on balance having roughly the same merit as a solution.  Don’t ever go in having already decided which one you really want!
2. Yet, you need to remember that all of the choices are also fundamentally different in their details. For example, one might have 30% greater functionality or come from a more established and stable company than the others, but recognize that this advantage probably comes with a cost premium. The market is generally pretty efficient that way.

Your goal in the negotiation process is to see where you can find “wiggle room” on each alternative’s components (price, terms, features) that will ultimately make one of those choices outshine the other, on balance.  In other words, you are looking to unsettle the equilibrium you’ve achieved in #1, so that one choice comes out on top. And here’s another key: let each vendor know, and remind them frequently, that this is what you’re doing. Usually, there’s no need (or benefit) in letting the participating vendors know exactly what your list of alternatives consists of, but it will suffice that they know that you know that you have choices.

Miscellaneous tips and observations:

• Negotiation, in general, is not simply about money, although that’s certainly a key dimension that can make one of your viable alternatives rise above the others.  Any facet of the deal should be open to scrutiny and discussion: cost, terms, rights to upgrade, SLAs, warranty, etc. You don’t always automatically choose the least expensive option, nor do you always choose the most featureful.

• Don’t ever tell vendors (and believe me, they’ll ask again and again) what your budget is for this project. They don’t tell you what deals they’ve cut when they’ve sold their product to other customers, do they?  Again, the fact that you are actively pursuing other alternatives needs to speak the loudest.

• Remember, though, that your goal is not to hammer down the price so far that the vendor won’t make any money on the deal. You can do that, at times, but it ultimately means that you’ll get poorer service and less attention when you need it.

• Where possible, involve your legal counsel in the negotiations; don’t just bring them in at the end. The best negotiations for me, in terms of viable outcome, were where I had a tight bond and common approach with my legal counsel.

In the end, it’s all about getting a viable solution for your need, at a reasonable price and beneficial terms for all. It’s not really about “winning”; it’s about choosing. Give yourself choices, and in the end, you (and your organization) do win.

Lagniappe:

• Ward Keever, “Ten Keys to Successful Negotiations”
• Martin Ewing, “5 Can’t-Miss Vendor Negotiation Tips”
• Michael Krigsman, “Negotiation Tips for CIOs”
• Peter Kretzman, “More tips for dealing with IT vendors“

Roger Sessions recently published a white paper on IT complexity and its role in IT project failure: “The IT Complexity Crisis: Danger and Opportunity”.  It’s certainly possible to quarrel with bits and pieces of his analysis, and thereby tweak his numbers, but the overall thrust remains undeniable: IT failures are costing the world incredible amounts of real money. Sessions even sums it up under the dire-sounding phrase, “the coming meltdown of IT,” and says, “the out-of-control proliferation of IT failure is a future reality from which no country—or enterprise—is immune.” And he presents “compelling evidence that the proliferation of IT failures is caused by increasing IT complexity.”  He points out that the dollar cost of IT failure in the US alone is close to the cost of the recent US financial meltdown, and cites indications that the rate of failure is also increasing by 15% per year.

Roger’s paper is excellent and thought-provoking, and I recommend it highly. And I do agree with his view that complexity is the chief culprit in IT failure. That said, I think his argument focuses a little too strongly on one cause of complexity (unnecessary overcomplexity of architecture), to the neglect of other important factors.

To be sure, some obvious contributors to IT failure (poor project management, and lack of communication within teams and from business to IT implementers) aren’t dismissed by Sessions, but he sees their contribution to the crisis as relatively small. I don’t, and I’ve used this blog to write about those factors quite a bit.

Most of all, though, I differ with Roger’s focus on streamlining architecture as being the key to reducing system complexity. One could say, in fact, that Roger’s solution is primarily a technical one, where the bugaboos I see are primarily cultural and sociological.  I see not one, but at least three distinct complexity-related burdens, increasingly endemic, and increasingly bringing down IT:

• Overly complex design/architecture
• Taking on too much functionality
• Poor implementation (technical debt in-the-large and in-the-small)

Roger has admirably dealt with the issue of overly complex design/architecture, at least in terms of a viable approach for simplifying up-front architecture, so I’ll focus here on the other two.

Taking on too much functionality

I recently rented an economy car (the least expensive option) on a trip with my son. (Remember, my self-appellation is “Cheap Technology Officer.”) He was stunned and dismayed that the car didn’t have automatic door locks; he didn’t realize that they even made cars without them anymore. Similarly, my 11-year-old daughter has grown up in a TiVo-ized world where live TV can always be paused. When she encounters a TV without a DVR (and thus no pause capability), she regards it as hopelessly primitive. Indeed, as unacceptable.

Similarly, the general standard for functionality and UI design has been raised by extremely functional PC software.  I now expect to be able to double-click on a number in any onscreen report, and thus “drill down” into the transactional details that make up that number. When I can’t, I feel cheated. Equally, I expect everything on an interface to drag and drop; I get frustrated if it doesn’t.

So as an industry, we’ve raised the bar of acceptability, considerably, in software and technology systems over the last couple of decades.  What that means in practical terms, though, is that across the board, our eyes have gotten bigger than our stomachs. We want more, up front, than it often makes sense to build at the start. And our demands are not negotiable, or so it seems. The first few cell phones I had didn’t even have a ring silencer on them; I used to silence the phone by adroitly disconnecting the battery when a call came through at an inopportune time. Today, most people wouldn’t even consider buying a phone that lacks much more elaborate features, such as a camera, that I would have considered as space-age in nature back in the 80s.

So our increase in expectations, alone, has added considerably to the functionality of systems we tend to build. There’s more functionality in and of itself, and usually more interface points to other complex systems. In fact, integration testing—where you connect new code into a working environment where it has to interface correctly with other systems—has become a frequent and major sticking point in launching information technology projects.  In essence, we’ve fallen into the “nuts” dilemma, both in large and small ways.  We want so much, and attempt so much, that we increase our risk of failure considerably.

Technical debt (in the large and in the small)

Any software developer will tell you that their first stab at implementing a given piece of functionality is often (if not usually) much more complex than turns out to be needed. Only after exploring the problem domain, with experiments and backtracking and restarts, do developers usually realize that their code can be pared down, simplified, combined with other modules, etc. This is usually called “refactoring”, and its importance is a relatively recent insight in the software development discipline.

A key insight about refactoring, though, is that it means improving the code without changing its overall results. There’s often no immediately obvious payback to this undertaking: it’s a roof project, in essence. To the extent that refactoring isn’t done (no time, no inclination, no recognition of a simpler approach), the end product is left with vestiges of unnecessary complexity. The greater the time crunch, and the greater the aspiration for the functional depth and breadth of the software to begin with, the more likely it is that these vestiges linger. And one ancillary aspect of the raised bar in expected minimum functionality is that it causes the time crunch to get ever greater.  It’s no longer about delivering just a solution that will work, it’s about making sure that the solution includes (metaphorically speaking) a 5-megapixel camera too.

Couple this unnecessary but accidental complexity with what often amounts to a rush job on design for the sake of meeting schedule (creating “technical debt in-the-large”). An example I’ve used here before: choosing a different core DBMS to implement a given function, simply out of expedience, and failing to take time to modify previous functionality to use that new DBMS. Supporting two DBMSes within the same product represents significant technical debt: every subsequent system change and addition will entail “paying interest” on that debt, which not only increases schedule and manpower costs, but increases risk of failure as well. And that’s just one example; the technical debt cascades, feature upon feature, release upon release. Technical debt, until paid down, can be equated to a invisibly rising substrate of complexity, and it contributes massively to an increasingly wobbly, risky system. And lately, I see more and more organizations “pyramiding” their technical debt, never taking the time and cost to pay it down, with disastrous results. As Hemingway said about going broke, it happens slowly, then all at once.

What now?

Roger’s analysis, to its large credit, outlined an important aspect of complexity and posed a solution (an approach he calls SIP, or Simple Iterative Partitions).  The aspects that I’m presenting (taking on too much functionality, and the pyramiding of technical debt) are, as I’ve said, cultural and sociological within companies. The answer is not nearly as simple or as neat as a specific technical solution (although I am certain that I will get comments on this post, perhaps rightly, from devotees of Agile).

To my mind, it’s engaged, savvy, forceful leadership that alone can address these issues, slow down the demand train, stop the madness. If anything, I think that there is an increasing lack of leadership in IT circles that can suitably recognize and address these factors, as well as educate their peers. And that’s what needs fixing most of all.

Lagniappe:

• Kevin Schlabach, “Agile East 2009 by Thoughtworks”
• Martin Fowler, “Technical Debt“
• Eric Ries, “Embrace technical debt”
• Johan Lindberg, “Painting With Hands and Feet: The IT Complexity Crisis”

Just a quick, personal post this time: I was recently interviewed by CIO Magazine on the topic of “How CIOs Can Stay Tech-Savvy“.  Since (as is normal) only a portion of my conversation with the reporter actually made it into the article, I thought I’d expand briefly on the topic here.

My remarks were two-fold, consistent with what I’ve written before on this all-important topic:

• It’s critical for the IT executive to “keep his or her hand in” by doing some hands-on work and experimentation with new technologies
• Your purpose in doing this hands-on work is not to become a viable technical resource in the area, but rather to get some deeper understanding than you’d obtain by just reading an article or two.

As mentioned in the article, I estimate that I spend 5-10 hours a month doing this kind of hands-on dabbling, sometimes with more success than others.  Let’s look at the kinds of things I do, large and small:

• Obviously, I administer my home network (four machines running three different operating systems, plus other home networking devices) and provide advice to neighbors and friends.
• I administer my blog, including configuration, changing WordPress templates, and even custom-coding PHP callbacks at times.
• I also actively seek out “early adopter” opportunities with new technologies, or technologies that are simply new to me.  I currently have four virtual machines that are launchable on my Mac: Ubuntu, Fedora 11, Windows 7, and Windows Vista.
• I have an ongoing Javascript dev project I work on that analyzes my iTunes music library and helps me identify gaps in metadata and lyrics, so that these can be corrected. That Javascript also dumps all the lyrics in my music library out into XML, to get them out of the proprietary world of iTunes.
• At the beginning of each year, I list out the technologies I’d like to delve into more deeply that year, in terms of reading and experimenting.  This list is based purely on what has intrigued me as I’ve scanned blogs, feeds, and Twitter. For 2009, my list included Amazon EC2 and S3, Ruby, Heroku, and CouchDB.  I’ve not gone as far as I’d hoped with a couple of these, but hey, 2010 will have a list too.
• In a given year, I might do some coding in Javascript, Perl, PHP, and Ruby. Admittedly, I usually need to look quite a bit of stuff up, but that’s mostly a factor of doing this only an hour or two a week.

As I emphasized in my remarks for the article, the point here is not to become a player on the field. I’ll never be as skilled in any of these technologies as the people I’d hope to hire with that expertise, should the need arise.  And that’s a good thing: the temptation is always there, particularly for someone who rose up through the developer ranks, to micromanage.  But at the senior executive level, it’s far more important that you stay focused on process improvement and strategy than on nuts-and-bolts techniques. Any of the experimenting I describe above should be viewed as self-education and a hobby, not a serious endeavor.

But you can bet that my self-education practice lends me a deeper insight into any of these technologies than if I’d sat back and simply read magazine articles on them. And oddly, I’m one of the few senior IT executives I know who still do this sort of thing. Granted, it will always feel to me like it’s too little, but not doing it at all is, well, not an option.

Given the recognized ascendancy of business/IT alignment and business change management as a vital purview of the effective CIO/CTO, should senior technology executives decrease their emphasis on technology, and focus primarily on corporate strategy and change? Should the company just have one role (call it a CTO, perhaps) do all that technical stuff, and move the CIO role into that of predominantly business strategist?

Let me raise my hand for the Nays. That would be the pendulum swinging way too far in the opposite direction. The problem of business/IT alignment won’t be solved by ghetto-izing technology concerns, and/or pretending that an executive is really only part of the senior team if she/he has a mostly strategic orientation and little responsibility for technology. That’s called a backlash, and it’s bound to lead to trouble. Here’s why.

It’s long been pretty close to a received truth that a key success factor for IT is to forge deep alignment with the business.  And that’s been an elusive goal. Indeed, one CIO Magazine article from way back in 2004 referred to business/IT alignment as a “Holy Grail”.  More recently, a Forrester survey discovered that “only 15 percent of IT leaders declared themselves to be fully aligned with the business.”

I’ve posted myself, early and often, on myriad ways to push effectively for IT/business alignment. So I’m anything but a blind proponent for a technology-centric CIO/CTO.  But now let’s talk about the backlash I’m seeing, and use it to reemphasize my point from the last time I wrote directly about “CTO vs. CIO” on this blog: it’s not the title per se that matters, but that a company have a single, key, senior information technology executive who is tasked with shepherding information systems strategy, decisions, and ongoing projects company-wide. Note, however, that as I stated last time, “The important part is to recognize two conflicting truths: technology is all-important in many leading and bleeding-edge companies today; technology itself, however, cannot be the sole, or even the main, focus and purview of the senior technology executive.”  Elsewhere, I wrote that “It is absolutely critical that the CIO/CTO be the chief voice at the senior management table, when it comes to educating and advocating the judicious undertaking of “roof projects” when necessary.”

What I’m seeing recently, as I wade through countless magazine articles and books on IT management matters (let alone hundreds of tweets from colleagues on Twitter), is a movement towards deprecating the need for strong technology leadership at an executive level. If past IT executives have been overly focused on technology and not enough on business value, then the answer must be (per this backlash) that the IT executive should move away from focusing on technology, put that in the hands of others, and move instead toward business, innovation, and shepherding corporate change.  Specific examples of the backlash:

• “There’s no “T” in CIO”. In Chris Potts’ loosely fictionalized (and excellent) narrative, “The CIO as a Corporate Strategist,” he has a CEO reflect as follows: “Maybe it’s the constant reference to technology that’s getting in the way of understanding what the CIO role is really all about. After all, CIO doesn’t even have the letter T in it.  What if we took technology away from the CIO and focused him uniquely on Business Leadership? What then could the CIO role do for us?”  Elsewhere in the same article, Chris writes that “in-house IT management has increasingly become about sourcing and supplier management. Maybe one day, [the CIO] speculated, the company’s sourcing people should do all of that instead.”

• “‘There are no IT projects, only business projects,’ is the frequent imperative of many CIOs and IT leaders.”  A number of us had a raging debate on Twitter about this. While it’s certainly true that all projects must have business justification (e.g., revenue enhancement, strategic impact, cost saving, legal imperatives), there will of course always be projects that have little or no direct, short-term impact on the business stakeholders of the company, yet are critical to do.  See my recent post on “roof projects”.

The backlashers (to coin a word) are right in their emphasis on business strategy and change, and yet they are wrong: they are throwing the baby out with the bathwater. My experience has taught me that technology management cannot be dismissed as purely a lower-level activity, with the senior executives in the company able to take a hands-off approach while they evolve their strategy.  In fact, the problem is that too often, technology management has been placed in the hands of people who had only a lower-level approach and perspective.  Doing that has resulted in exactly what you’d expect: a lower-level perspective. Hence the stereotype of the unresponsive, uncommunicative, difficult IT organization, often working on matters that aren’t congruent with the business needs of the company.  Or, conversely, IT has been “managed” by people who weren’t at all technology-oriented, who have then proved ineffective, open to vendor manipulation, staff disrespect, and a steadily increasing “herding cats” mentality on technology matters.

The answer is not to reject IT as an important purview of a senior-level executive (call that person a CIO or a CTO, as you wish) in a company.  The issue is getting a senior executive who will exercise the right balance between technology and business strategy. Downgrading “pure tech” matters to a non-senior executive leads to the same old rut: tech decisions made poorly, with tunnel vision.  Instead, the CIO needs to straddle a midpoint on the business/technology spectrum, not swing to one end or the other. You can’t have one person (say, the CIO) responsible for strategy and still another (say, the CTO) responsible for technology. It turns out that you need the combination: a senior executive who is part of the strategic definition for the company, and who can ensure that the day-to-day decisions in information technology will be made accordingly. In other words, companies need to recognize that business projects can fail equally through technology tunnel vision or through too little attention to core technology matters by executives who spend their time elsewhere on matters they deem more “strategic”.

In fact, if there is no (or the wrong kind of) executive in charge of technology, one sees effects such as the following:

• technical and applications architecture tends to grow haphazardly, becoming increasingly inflexible and unwieldy;
• no metrics are gathered, much less used for continuous improvement;
• open season reigns for vendors, who then deal primarily with lower-level buyers who often lack the big picture financially and strategically;
• the “dev guru” phenomenon appears, where the company is dependent on one or two individuals and there’s insufficient cross-training;
• no delivery commitments are made—or commitments are made with no factual basis;
• silos appear in Ops, QA, Dev, PM, often at cross-purposes with each other;
• Multiple points of entry into IT abound for business folks. What gets worked on depends on personality, not corporate exigency;
• little process improvement is considered or exercised;
• “IT sourcing” groups emerge that become sheer order takers for stakeholders who’ve been swayed by a vendor demo.

For further examples, consider the points I made in a recent post, “Canaries in the coal mine: Why your IT department may be in worse shape than you think.”  To avoid these and other examples of IT failure, companies need to place at the helm of information technology an active, savvy executive, serving as a peer of the senior executives in the company, and they must look to that individual for leadership, guidance, and day-to-day influence.  Should that person have the title of CIO or CTO? That’s not the right question, because it doesn’t matter.

As I’ve written here before, I strongly advocate thinking of IT in general as a service organization to the rest of the business.

Any service organization needs one or more forms of “feedback loop” to be able to gauge whether it is successfully accomplishing its mission.  However, I’ve observed relatively few IT organizations that actively seek to implement such feedback loops on a regular basis.  At best, the IT executive does it informally by consulting with his peers at the executive table.  But with any such anecdotal feedback, the information gathered that way tends to be fleeting and unreliable, and it is especially influenced by strong personalities and emotions during crisis situations.

Here’s a better, and simple, suggestion, one that I’ve implemented to varying degrees at several firms with a good amount of success: Survey your constituents regularly and then publish the results.

Sounds daunting?  I promise it really isn’t, not in this day and age of easy-to-use web-based surveys.  With less than an hour of work, you can design and initiate a survey using a free service like Zoomerang or SurveyMonkey, and easily gather high-quality results (reports and statistics) in just a few days that can help you gauge (and present) how you’re doing.  Here’s how.

You actually need to think about three broad groups of constituents whose pulse you want to take regularly (at least once a year, preferably more):

• internal customers (your help desk customers)
• peer executives and key business stakeholders
• your IT staff

I’ll focus here primarily on the surveying of key business stakeholders.  For the first category, you need to get a trouble ticketing system that incorporates user surveys after each ticket closure.  For the third category, well, that deserves a whole other post to discuss, since in some ways, it’s the most complicated and pitfall-riddled area of all.

For stakeholder feedback, most importantly, you need to start off (within the first few weeks of engagement in your role) by getting a general “baseline” understanding of current satisfaction, on the part of internal key business stakeholders, with IT in general.  Of course, don’t let a mere survey ever substitute for actually getting out and talking to as many people as possible about how good the service has been from IT.  When you send the survey out, emphasize to everyone that their input is anonymous, unless they choose to identify themselves in their comments. All you will know is whether or not any given individual has completed the survey.

Keep the survey as simple as possible, and you’ll get more responses.  You’ll want to balance free form input (usually just one broad question) with questions with quantitative answers, ones that will let you accumulate metrics over time.  And you’ll need to be prepared for some harshly stated criticisms, so get used to that idea.

Here are some sample questions I’ve asked in such surveys:

1. Compared to other companies you’ve worked at, how would you characterize the overall level of service of IT at this company?
Choices:

Not good
Could be better
OK
Reasonably good
Quite good
Couldn’t be better

2. Of the following areas of responsibility of the IT department, which ones would you say most need improvement? (pick all that apply)
Choices:

Communication on progress
Cohesive strategy and direction
Internal systems development
Overall service ethic
Business reporting
Alignment with business needs
Responsiveness to emergencies
Internal help desk (PC hardware / software issue handling)
Project management
Other (please specify)

3. Of the following areas of responsibility of the IT department, which ones are currently being handled especially well, in your view? (pick all that apply)

(same list as above)

4. Compared to six months ago (assuming you were here at the company then), would you say that the IT department’s overall service to the company is…

Better  /  Worse  /  About the same  / Not applicable

5. Please give the IT department an overall rating on a scale of 1-10 in each of the following areas, with 10 being perfection.
Choices:

Service
Efficiency
Timeliness
Business orientation
Attitude
General competence

6. What other general comments do you have regarding IT, interaction with your team and department, etc.?

(free form entry field)

Send out the link to the survey in an email, explaining your purpose and goals in doing so.  Give your stakeholders (hopefully at least 20 people across the business; don’t arbitrarily limit it!) at least a week to respond, and nudge them every few days with reports on how many have filled out the survey so far.

When the survey finally closes, thicken your skin a bit (OK, maybe a lot) and take a good amount of time to look carefully at the results, and to compose a report to the people who responded.  By doing so, you’ll be showing that you take the feedback seriously, and have focused on specific action items as a result.  Then, you’ll want to send out the survey again in no more than six months; your subsequent report there can talk about the specific actions that you took in response to the last survey.

The good will generated by this service-oriented approach should not be underestimated.  Face it: it’s the right thing to do, and people will instantly recognize that.  As Mark Twain famously said, “Always do right. This will gratify some people and astonish the rest.”

Here’s a disquieting little secret that few of us ever really acknowledge, maybe because it’s rather painful and also an unavoidable part of the fabric of our existence in IT. I don’t know how to say it more eloquently (or less bluntly), so here goes: being in information technology is hard. In our day-to-day dealings with stakeholders, with end users, with management, even within our own ranks, it’s common to hear some pretty discouraging and recurring things, voiced either explicitly or implicitly. For example,

• “what have you (IT) done for us lately?”;
• “what do you (IT as a whole) do all day?”;
• “we’ve been asking for that system for years now and not gotten it”;
• “how can that be so hard? Why can’t you just …”;
• “at my last company we did that in just [names an absurdly short amount of time] and it worked really well.”

A major role of IT management, in my view, consists of steadfastly, repeatedly, and quietly combating these discouraging statements, of raising people’s consciousness of complexities and risk and levels of effort, of building ongoing trust, of establishing a clear and public record of excellence and collaboration. And it necessitates frequently reminding our staff not to get disheartened when we hear indications that we haven’t fully succeeded in that.

It’s normal that business expectations are high. Yet, those expectations are significantly and often unduly heightened by one of today’s realities: lots of non-IT people do “IT in the small”, working with spreadsheets, surfing countless applications on the web, even administering their home networks. But of course, this IT-like work “in the small” comes not even close to having the issues and complexity of enterprise-level systems. We in IT know this, but it’s anything but obvious to others. In essence, we’re in the position of constantly and actively having to fight some very basic resulting assumptions and myths about what we do. Those myths are often held widely throughout a company, including by senior management and boards of directors.

Here are just a few of the myths we combat:

• Software is like a toaster: you get it (or make it), plug it in, and you’re ready to roll.
• Maintenance shouldn’t take much of anyone’s time.
• Most changes to systems (e.g., to accommodate new business rules) are simple.
• Stakeholders’ role is to say what they need; IT simply has to deliver (all of it). Collaboration and careful prioritization isn’t all that necessary and boy, it consumes a lot of time.
• We should keep all data forever; disk space is cheap.
• Industry best practices don’t matter, especially if it means things will take longer. We can ignore best practices and we’ll be successful anyway.

I could (and probably will in future posts) write in detail on each of the above myths, but for now I’ll tell just one illustrative anecdote that demonstrates the kind of world view I’m discussing here.

Quite some years ago, I worked for a company that was bought by a much larger entity. At the time, we were pushing hard to finish the construction and deployment of a revolutionary (for its time) CRM-like system for use in the company’s call centers. One of our stakeholder meetings featured the following statement from a newly hired VP of Marketing: “Well, we just got bought by company X–they have call centers. Hey, why don’t we just snag some of their code?”

Everyone needs to be wary of any systems-related statement that includes words like “just” and “only”. Of course, the notion of reusing other companies’ software, when possible, isn’t something to dismiss out of hand (hence the subsequent rise of ERP packages). But the word “just”, combined with the artful word “snag,” revealed an amazingly dismissive world view about the complexities of building and integrating systems. We had a long discussion with that executive about how the word “snag” pretty much didn’t apply to the level of systems we were discussing.

So again, it’s hard being in IT, and it’s hard being the leader of IT. You need a large dollop of resilience, optimism, and stamina as your psychological armor. It’s definitely possible (indeed, necessary) to break these myths down, but it takes coming in fresh every day, ready to work hard and ready to face the same kinds of misunderstandings of the basics.

As is my frequent practice, I’m going to follow up on this post with another that talks about the flip side: the ways that IT itself can unfortunately perpetuate some of the myths I’ve been discussing.

Here’s a post of a type I rarely do: a reaction to an item recently posted to the Internet. Specifically, a day or two ago, Steve McConnell’s firm Construx, Inc. released their update of McConnell’s list of classic software development mistakes. This survey and its results is worth everyone’s time to read (and ponder) carefully. Their summary of the paper is as follows:

“Classic mistakes are ineffective software development practices that have been chosen so often, by so many projects, with such predictable results that they deserve to be called classic mistakes. Steve McConnell first introduced this concept in Rapid Development in 1996. Construx recently updated McConnell’s original list of classic mistakes and then conducted a survey to assess the prevalence and impact of these mistakes. This white paper shares survey results–both expected and surprising–and analyzes the survey findings.”

The Construx survey results document pretty much speaks for itself, and provides interesting detail that I won’t repeat here. Hence, this is a bit of a “piling on” kind of post, in part because I respect McConnell’s work in the extreme, but also because I want to add some “color” to what he intentionally presents with (relatively speaking) a “just the findings” deadpan dryness.
Here are the top 6 “Worst Mistakes” outlined in the document, taking into account both their frequency of occurrence and the severity of their impact when they do occur:

1. Unrealistic expectations
2. Overly optimistic schedules
3. Shortchanged quality assurance
4. Wishful thinking
5. Confusing estimates with targets
6. Excessive multi-tasking

Needless to say, this list struck me as eerily accurate to what we all experience in IT circles. The thing I find the most astounding about this list is how we keep making the same mistakes over and over again. Despite all the project management lore we’ve all absorbed, despite all the “lessons learned” meetings we’ve all sat through, this list hasn’t (and maybe won’t) change. Why? Here are just a few contributing reasons:

• The systems that stakeholders see and regard as benchmarks (i.e., for functional capability) have evolved in most cases over many years, in terms of functionality, robustness, and elegance. But those systems have set the standard, and set generic expectations. Things seem simple to do once you’ve seen them done, and it’s oh-so-easy to forget how much time and resources it took to get them to “done” status.
• There’s a rising belief in management that schedules (and people’s work intensity) will naturally adapt to fit whatever time is allotted, so why not set a purposely short time to start with, “just to keep the pressure up.”
• Market pressures keep increasing, meaning we all are constantly challenged to get more done, faster, and with fewer resources.
• Stakeholders won’t naturally take a long-term view: they tend to minimize the often extreme down-the-road headaches that result from the cutting of corners necessitated by the rush, rush, rush mentality. They’ll drive the car without ever changing the oil.

But in a nutshell, struggling proactively against the setting of unrealistic expectations and overly optimistic schedules is the brunt of a CTO/CIO’s job. When unrealistic expectations are allowed to persist and prevail anyway, it actually means we’ve failed: failed to educate, failed to take a sufficient stand. It all comes down to leadership. It’s part of the territory. Whether we like it or not.

Actually, more often than not, I think that stakeholders secretly expect to have their expectations reined in. My shortcut slogan for doing this is, “I can’t be in New York in an hour.” (Note that I’m based in Seattle). If you somehow have the expectation (hope, desire, etc.) that we’ll be able to be in New York in an hour, I can’t and won’t sit by and wait for the hour to elapse before I let you in on the disappointment. Taking that kind of shatter-expectations-up-front approach requires spine, verve, fearlessness. It takes the kind of self-confidence that accepts the risk that one will be unfairly labeled as a “no can do” kind of guy/gal.

But that’s why you’re the leader.

As I discussed last time, everything you add to your environment (hardware, software) costs money in recurring fees. Part of the job of the CTO/CIO is to sign dozens of invoices, each and every week, that approve payment for the various elements in your infrastructure that have come up for renewal. And hey, we’re all busy. Anyone who’s been pestered by the company’s usually indefatigable accounts payable department knows the perils of not having signed off an invoice in a timely manner: in other words, you can’t afford to let them languish. Problem is, it’s relatively easy to fall into a “rubber stamp” mode, scribble a quick signature, and move on to the rest of your busy day.

Multiply that by dozens of invoices a week, 52 weeks a year. A few thousand dollars here, a few thousand there: as the saying goes, pretty soon you’re talking about real money. Careful scrutiny of each and every expense takes time and effort, but my view is that it’s one of the major responsibilities of the job. The amount of company expenditures that flow through IT places an important onus on you, the head of technology, to constantly be thinning the carrots, so to speak.

And, especially if you’re new to the company and this job, chances are pretty good that you’re inheriting an essentially unweeded garden. At least, that’s been my experience. Your first year will be one of constant discovery, to put it charitably, as invoices land on your desk and you steadfastly resist the Rubber Stamp Syndrome.

Here are a few real-life examples. I’ll leave the names of the vendors out of these little anecdotes, although your guesses will probably hit pretty near the mark.

When I had just taken over as CIO at one company, a portion of our database licenses came up for renewal. As I pored over the perhaps intentionally inscrutable invoice, I discovered a couple of items that seemed questionable. It turned out that as part of a database software deal several years before, this vendor had included, with the consent of an apparently very optimistic but certainly very gullible head of IT, a few “free” licenses for one of its applications packages. Free, hmmm? Well, free except for the ~20% annual maintenance fee, which had been charged, and paid, for three consecutive years, with no implementation of the package and no plans to implement it. The Rubber Stamp Syndrome had struck. You can bet two things: one, we immediately stopped paying the maintenance fee; and two, we had a pretty unpleasant meeting with the very defensive, “hand caught in the cookie jar” vendor.

Similarly, at another job, I learned that various additional “modules” had been sold to the company years before as part of the company’s ERP solution. Vendors seem to sense when a buyer is susceptible to the “such a deal” sales pitch that I also refer to as “buy five of an item, shrink-wrapped, at Costco when you really only need one.” With software, it’s because those deals have incredible legs, with year after year of Rubber Stamp renewals. In this case, we had never used several of the modules in question, and had no plans to, but yet had been paying thousands of dollars in maintenance every year on those licenses.

Finally, to give a smaller example from a dollar impact point of view: I discovered that my infrastructure staff needed to make regular purchases of back-up tapes. We had been using the same vendor for this for several years, paying about $65 per tape. “Have we compared prices elsewhere?” I asked. Um, no, these are really low-cost items, I was told, and besides, this vendor gives us great service. Well, when you’re buying 50 or 100 items at a time, and can get the same product for $30 per tape cheaper (as it turned out), it’s definitely worth switching vendors.

In a nutshell: it behooves you to ask the following standard and basic questions regarding “legacy” renewals of maintenance and support:

• Are we happy with this vendor and its product(s) in general?

• Are we right-sized in terms of the number of licenses, now and for the foreseeable future?

• Do we really need and use this support? What are our other options? Is per incident support available? You don’t buy the extended warranty (I hope) on every major appliance you purchase; don’t buy it on absolutely every element in your infrastructure either. Don’t buy 24×7 support if a lesser level will suffice. That’s obvious, you say, and perhaps it is: except, of course, if people fall prey to the Rubber Stamp Syndrome.

If you find yourself becoming a rubber stamp, it’s time to get re-energized. Remember, the C in CTO or CIO needs to stand for Cheap.

This time on a more personal note: I’ve been reflecting lately about the various specific skills that helped propel me in my career, and how I picked those up. These are mostly metaskills, rather than specific technical capabilities. A number of technologies that I spent a long time becoming expert in are not listed, for example, in the interest of emphasizing the broader lessons, the mindsets, the “core understandings” that have molded my outlook. Are these skills applicable to you and to your path? Only you can be the judge. I offer them up simply as a catalog of things that I feel have boosted my career.

• Writing. The ability to express one’s thoughts and plans in clear, logical, well-formed language is, I feel, the single most valuable skill to bring to the workplace, particularly in an executive role. Writing is not easy, and the result is by no means always perfect. But this skill is definitely top of the list.

• Typing. Yes, believe it or not. The skill that I first learned during the summer after sixth grade, following along on my own in the typing course my older sister was taking, has been a surprising boon to me. Later, I honed this skill by necessity, as I jockeyed for mere 15-minute slots of computer time in our school’s computer lab, and I had to maximize my productivity during that time. The result is that I can type far faster than I can write by hand, meaning that thoughts flow out and get captured (and adjusted) far more readily.
• Assembly language. Bare-metal programming. There’s nothing quite so instructive. I would have little hope of understanding things like network protocols, caching algorithms, and software in general, as deeply as I do, if I had not spent quite so many hours with the PDP-8 and NOVA instruction sets, way way way before the advent of the personal computer.
• Public speaking. In another life, I was a teacher for several years, but it didn’t come easily. I had to get over the all-consuming sense of nausea that would hit me before (and after) I taught. After years of teaching, though, this “stage fright” phenomenon simply disappeared. Executives need to be able to present. If you don’t feel comfortable talking to groups, I’d recommend that you hone this skill.
• SQL. I’m not putting specific computer languages on this list, although each language tends to expand one’s mind in a slightly different direction from before. SQL, though, is a non-procedural, data-driven, very different way of thinking, and I believe that it is so crucial to today’s systems that anyone in IT should understand the basics of how it works.
• Unix piping. It’s not the syntax, it’s not the specific OS, it’s the mindset that matters here: simple, small, single-purpose, independent tools, chained together, creating magic.
• Also-rans: source code control; object-oriented programming, and (yes, a specific language, the best “Swiss Army chainsaw” around) Perl.

So what skills do I wish I had that I haven’t been able to acquire? A couple of key ones come to mind:

• Visual thinking. The old cliche about a picture being worth a thousand words? That just doesn’t work for me, although I certainly recognize that I’m in the minority on this one. Give me quality prose over often incomprehensible diagrams any day. But when diagrams and pictures do work, they elicit a kind of startling clarity and focus that I wish came to me more naturally. See the Lagniappe section for a couple of references.

• Glib persuasiveness. I’m not great at “selling” in situations where I can’t feel a deep personal conviction. I’ve met people over the years whose powers of persuasion stunned me. Without any tinge of prevarication, these people were able to “spin” difficult situations or product aspects into total non-issues. Some call it “selling ice to Eskimos”. Me? At the extreme, it’s kind of like the old joke that HP would tend to market sushi as “cold dead fish”. Some days I feel that I could have worked for HP.

Lagniappe:

• Edward Tufte, The Visual Display of Quantitative Information, 2nd edition. Aside from the content itself, this is also among the most beautiful physical books I’ve ever seen.
• Malcolm Craig, Thinking Visually: Business Applications of 14 Core Diagrams.

In honor of the season, I thought I’d share a few recurring nightmares, ones that unfortunately don’t seem to confine themselves to the fall time frame. All of these are chronic worries that have truly kept me up at night; most of them stem from actual real-life situations I’ve encountered.

1. Your CEO calls you and asks you why the web site is down… and you didn’t know it was!

When the company’s web site (or any other mission-critical system) is down, escalation mechanisms need to inform you and inform you fast. Of course, the site should rarely / never be down other than for scheduled maintenance, so putting yourself in that notification loop (subject to calls in the middle of the night) shouldn’t be too common and painful. If you’re not informed of these situations, I’d argue that either your team isn’t sufficiently on top of detecting them, or they’re “sparing you the pain” of being told. In truth, the pain has to be spread around. The onus of notifying management is one mighty incentive to make sure that the need to do so arises as seldom as possible. Don’t tolerate being part of (much less at the helm of) an organization that purposely or through omission sweeps things under the rug.

2. Sudden night sweats: “I’ve spent too much on infrastructure!” Or, just as bad, “I’ve spent too little on infrastructure!”

Do you know for sure that you’re not in one or the other of these situations? Are you really prepared to scale to a degree that reflects with your growth? Do you have metrics and processes that support a methodical capacity plan in general? In my view, the “C” in CTO should stand for “Cheap”, because it’s remarkably easy to be the Expensive Technology Officer, rather than the Cheap Technology Officer. But in truth, the C should also stand for “Commensurate”: i.e., you want to build for your anticipated growth, but not be too far ahead of that curve. Anything else (too little, too much) is the making of nightmares.

3. Your software architecture looks fine now but won’t scale.

This scenario covers the possibility that one day you’ll cross some unknown threshold (data volume, for example) where response time or other key metrics go massively and suddenly south. How do you mitigate this nightmare? Well, the paranoid among us will never lose it entirely, but the best (perhaps the only) approach you can take is to conduct intense design reviews, evaluating for potential performance and throughput bottlenecks that could crop up. Another approach (often ignored) is to plan proactively for data “deadwooding” operations. Example: a high-volume web site that has people register should periodically eliminate those registrants who never logged in again within (say) a year of registering. Business users will often clamor for retaining everything, but there’s a real (and looming nightmare) cost of doing so.

4. The system goes down and you … can’t…. get…. it…. back…. up.

Today’s systems are frightfully complex, with lots of interconnecting moving parts. I’ve had times, while sitting through a middle-of-the-night crisis conference call discussing a badly non-functional system, where this kind of desperation started to set in. No matter what we tried, the system wouldn’t respond. At times like this, you start to wonder if it ever will. Mitigation? Remember that it always does, one way or another. It may not help the nightmare. This one’s a doozy.

I commiserate with all of you who may have experienced these and others. Sominex only goes so far to soften the impact of these nightmares. Rather, careful and methodical risk mitigation planning beats pharmaceuticals every time, in my experience.

Lagniappe:

• Paul Chard, 10 IT Nightmares and How to Avoid Them.
• Deloitte and Touche, The Risk Intelligent CIO: Becoming a Front-Line IT Leader in a Risky World