What is it that sets apart a top-notch IT executive from others of his calling? To my mind, one mark of today’s true professional, especially at the senior executive level, is to be deeply familiar with the seminal books in his or her field. The dilemma for an IT professional, though, comes from the ongoing and increasing flood of books to choose from, and trying to figure out how to walk the fine line between focus on the intensely tactical and focus on higher-level concepts and ideas.

The tactical books do have their place on your shelf, actually, and it would be a mistake to ignore them simply because you’ve moved beyond daily application of your development, configuration, and technical trouble-shooting skills: judicious selection and absorbing of nuts-and-bolts techniques and new approaches will keep your insight into technology and its possibilities fresh.

I started in IT as a developer, and I remain fascinated by the endless possibilities and techniques of the world of software. In the last decade or two, though, I’ve become even more intrigued by a metalayer above the more tactical concerns. True to my ongoing insistence that the biggest challenges in IT aren’t purely technical, I am ever more convinced that the greatest difficulties are presented by “psychology of IT” issues: the human factors in how software and systems are conceived, built, tested, deployed, maintained, and eventually decommissioned.  Here are just a smattering of the eternal, non-technical questions that go far beyond the computer language du jour or the latest hot methodology:

• How do teams actually create and complete information technology projects? What works, what fails, and why?
• Why are some software developers ten times as productive as others?
• Why do some software teams gel and others don’t?
• Why do small companies with very few resources often beat out large, well-funded efforts in the marketplace?
• How technical should managers be?

So starting with this post, let’s embark on a multi-part survey of the groundbreaking, timeless books on such issues. I’m going to pick what I consider to be the top three books from each decade, starting with the 70s.  Each of them deserves not only a place on your bookshelf, but to be read and reread every few years. And contrary to what one might think, their insights remain not only valid after all these years, but have become all the stronger by having been confirmed by the history of the industry since their publication.

• Weinberg, The Psychology of Computer Programming (1971)

Weinberg opens with, “This book has only one major purpose—to trigger the beginning of a new field of study: computer programming as a human activity…. If our experiences are any indication, each [software developer] could be functioning more efficiently, if he and his manager can learn to look upon [him] as a human being, rather than as another one of the machines.”  This book was especially groundbreaking by addressing software development as both an individual and a team effort. Remarkably readable and full of anecdotal examples, it covers virtually every human aspect of the software development “sausage factory”: team formation, individual contributions, goal setting, leadership, estimating.  Weinberg’s “Silver Anniversary” edition of this book contains annotations and commentary for each chapter, reflecting on similarities and differences to today’s environments.

• Brooks, The Mythical Man-Month (1975)

I’ve said it before: if there’s one single book that every IT professional should read, it’s this one. Brooks’ foreword presents it as a “belated answer to Tom Watson’s probing questions as to why programming is hard to manage.”  Brooks’ writing brims with key universal concepts that can be unfortunately non-intuitive to people outside the field, but which ring familiar to any long-time IT practitioner. The titular essay introduces the concept of the “mythical man-month”, his adage that “adding manpower to a late software project makes it even later.” But there’s much more, such as “plan to throw one away”, where Brooks points out that the “first system built is barely usable; … there is no alternative but to start again.” I can’t say enough about this book. As with the Weinberg book, the anniversary edition now being sold includes Brooks’ later (1986), equally seminal essay, “No Silver Bullets” (see my post on this: “No Silver Bullets. Really!“), as well as some added chapters that revisit the assumptions of the first edition.

• Weizenbaum, Computer Power and Human Reason (1976)

Ultimately, this is a book about the limits of computers and software. Weizenbaum, a noted computer scientist in the early days of Artificial Intelligence research, wrote a program called ELIZA, featuring a script that “enabled it to parody the responses of a nondirective psychotherapist in an initial psychiatric interview.”  In other words, his test subjects would converse with a software program that emulated a therapist. Weizenbaum became horrified by how many people forgot that this was “just” a machine they were talking to, and that the dialog was really just an illusion. Many insisted, to his amazement and despite his explanations, that the computer actually understood them.

IT professionals today can be just as swept away, to a fault, with the potential ultimate power of software and systems as Weizenbaum describes.  I’m inspired and reinvigorated every time I read his sobering, methodical discussion of the nature of programming, the limits of its scope, and the need to consider the social implications of technical projects.

The next time, I’ll be on to the key books of the 80s. I’ve already picked the books I tentatively plan to write about, but I welcome your suggestions.

Lagniappe:

Here are a couple of books that didn’t quite fit in my theme and chosen time frame, but which are still worthy of mention:

• Thomas Kuhn, The Structure of Scientific Revolutions (1962)

Nobel Prize-winning physicist Steven Weinberg wrote that this book “has had a wider influence than any other book on the history of science.”  This book is not about information technology directly, but its influence has been monumental across all scientific disciplines, and it is a book that any technologist should know well.

• Ted Nelson, Computer Lib/Dream Machines (1974)

This was a self-published book in the early 70s, by influential industry visionary Ted Nelson, the man who coined the term “hypertext.” It’s probably different from just about any book you’ve ever read.

Fred Brooks wrote a seminal essay in 1986, “No Silver Bullet — Essence and Accidents of Software Engineering“, a model of clear and cogent thinking that I consider to be required regular reading for anyone involved in information technology.  Despite the essay’s brilliance, and despite its wide promulgation and deserved fame, the phenomenon it describes seems to have only broadened in the last twenty-three years.  Brooks argues as follows (with bolding added):

“The familiar software project, at least as seen by the nontechnical manager, has something of this character; it is usually innocent and straightforward, but is capable of becoming a monster of missed schedules, blown budgets, and flawed products. So we hear desperate cries for a silver bullet—something to make software costs drop as rapidly as computer hardware costs do.

…

“There is no single development, in either technology or in management technique, that by itself promises even one order of- magnitude improvement in productivity, in reliability, in simplicity.”

So this basic tenet has been convincingly articulated by a leading IT thinker for almost a quarter century. Yet, the trend continues: new technologies pop up every couple of years and the hype cycle begins. Evidently, hope springs eternal.

IT and an inclination towards zealotry have always tended to have a high correlation, but one thing I find interesting about the incessant quest for a “silver bullet” solution is that the impetus for that quest often stems from non-IT senior management. They long for something, anything, which will take this risky, recalcitrant technology beast and tame it into predictability.  As a result, we see notable IT thinkers respond to that pressure, often by championing one or more specific practices, methodologies, approaches, or technologies as “the answer.”  Many sorts of silver bullets have come and gone in the decades since Brooks’ essay. The one constant seems to be that there are always a few memes out there being touted as cure-alls. Why haven’t we learned?

Let me gingerly pose the empirical evidence of this “quest for the cure-all” meme that is provided by my Twitter stream every day.  I see a number of IT tweeters who have doggedly taken up the flag of a particular movement or approach; in some extreme cases, virtually every tweet and comment from those people views all conceivable issues through that one lens.

Let’s list some recent candidates for the ongoing “silver bullet glomming” that I see, with some sample quoted hype for each one.  My Lagniappe section at the end of this post, for your amusement, will provide additional links to sites decrying the hype in each case.

Examples:

• Agile software development

“Agile Development Can Lead to Better Results and Technology-Business Alignment”

• Cloud computing

“Cloud computing heralds an evolution of business that is no less influential than e-business, according to Gartner Inc.”

• Project Portfolio Management

“Multiplying ROI at Warp Speed”

• IT Governance

“Firms with superior IT governance have more than 25% higher profits than firms with poor governance given the same strategic objectives.”

• Business alignment above all: remove technology from purview of CIO

“There are no IT projects”

“Only once we stop having IT projects and start having business projects will we see the full value of project management.”

• Eliminate complexity: a simpler architecture is the key

“The proliferation of IT failures is caused by increasing IT complexity. And this is good news, because complexity is a solvable problem.”

• Information Technology Infrastructure Library (ITIL)

“From small organizations to multinational enterprises and anything in between, this best practice framework has helped many improve efficiencies and bottom line figures, putting IT back in business!”

• Ruby on Rails (or, .NET, or J2EE, or Smalltalk, etc.)

“Ruby on Rails is a breakthrough in lowering the barriers of entry to programming. Powerful web applications that formerly might have taken weeks or months to develop can be produced in a matter of days.”

• Design Thinking

“It’s a technique that designers and executives alike hope may help to provide a solution to some of the world’s serious challenges.”

• Service-Oriented Architecture (SOA)

“SOA provides benefits in four basic categories: reducing integration expense, increasing asset reuse, increasing business agility, and reduction of business risk.”

• Web Services

“Experts and visionaries believe that the benefits of XML Web services will be instrumental in propelling explosive business growth over the next few years.”

It’s important that I note here that none of these theories, approaches, or technologies is necessarily a bad idea (although I think that each does tend to land at quite different points on the spectrum of goodness). There are some, in fact, for which I myself have major enthusiasm, because I think they promise to relieve at least portions of the IT delivery crisis.

The key takeaway, I’d argue along with Brooks, is that there’s no one technique, approach, methodology, philosophy, or magic that will wondrously make systems work easy and reliable.  No matter what the tools and techniques used, it’s hard work. It’s rife with opportunity for failure. It requires leadership, detail-oriented personnel, skillful practitioners.  Brooks took pains to say,

“Skepticism is not pessimism, however. Although we see no startling breakthroughs–and indeed, I believe such to be inconsistent with the nature of software–many encouraging innovations are under way. A disciplined, consistent effort to develop, propagate, and exploit these innovations should indeed yield an order-of-magnitude improvement. There is no royal road, but there is a road.”

So, we all have our silver bullet, I suppose: mine, along with Brooks’, consists of maintaining balance, practicality, and a broad embracing of multiple approaches, rather than one panacea.

Lagniappe:

• Peter Merholz, “Why Design Thinking Won’t Save You”
• Jerry Iacouzzi , “Debunking the myth of SOA”
• Matthew Huntbach, “What’s Wrong With Ruby?”
• Rob England, “The Emperor has no clothes. Where is the evidence for ITIL?”
• Colin Beveridge, “Biggest IT Myth of all time?”
• Steve Yegge, “Good Agile, Bad Agile”
• Cath Everett, “Five cloud computing myths exploded”

“Don’t write about that,” I’ve been told by several colleagues, when I’ve mentioned that I was working on a post about how best, as the senior technology executive, to negotiate with vendors.  “You’ll give away all your tricks!” they’ve said.

Well, actually, no.  Here’s the main trick: this particular CIO doesn’t have any “tricks”, if by tricks you mean ways to outfox the opposition, or anything else that is best kept secret.  In fact, I’m not a natural avid negotiator. I’m not one of those people who looks forward to buying a new car because of the thrill of haggling with the salesperson.  But I’ve learned over the years how negotiations can best be structured for the optimal outcome.  Like cryptography, where greater obscurity isn’t equivalent to greater security, successful negotiation isn’t dependent on tricks or subterfuge.

I’m quite content to tell any vendor or salesman how I go about negotiating, because doing so doesn’t provide them any kind of advantage.  If anything, it’s beneficial to me and my company that all parties in the negotiation understand clearly the basic principles and approach that I’m using; it cuts a lot of the normal gamesmanship out of the equation.

What I’m about to tell you about negotiating actually strikes me as incredibly obvious. Yet, time and again, I’ve come into a new CTO/CIO role and been astounded at the poor technology deals that were cut by the previous executive: inadequate terms, failure to protect both parties instead of just the vendor, unnecessary and undesirable contractual lock-in.  What happened in most of these cases, as far as I can usually determine, is that the executive in charge got swept away by emotional attachment to a specific solution, and tipped his or her hand. Game over: at that point they were putty in the vendor’s hands. I’ve even seen cases where contracts have completely put the vendor in the driver’s seat of actual implementation, including spending authority, to an unconscionable degree.

Make no mistake: vendors are typically much more experienced and skilled in negotiations than most of the technology executives they deal with.  Achieving excellence in negotiation should therefore be seen as a critical part of the technology executive’s skill portfolio.  But, like finance skills, it’s a skill more honored in the breach than the observance.  Your ability to negotiate well can not only protect your company’s long-term interests, but can actually recoup far more than the equivalent of your annual salary to the bottom line. It’s not uncommon for technology executives at even relatively small companies to negotiate deals worth hundreds of thousands or even millions of dollars of capital and operating expense.

So let’s cut to the “secret”. It boils down to this simple principle: give yourself choices. Identify two, preferably three, viable, affordable, achieveable ways to solve your problem or need.  Once you’ve successfully established that viable short list, the power is all in your hands.  If you haven’t fully done that, you’re not really ready to negotiate, so you shouldn’t even start. Power in negotiation comes from not being wedded to a particular solution. Another way of expressing this idea is “always be truly ready and willing to walk away.”

Pay attention here: I’m about to say two apparently contradictory things, both of which are true and key:

1. You need to regard all of the choices on your short list as fundamentally on equal footing, each with its own advantages and disadvantages, but on balance having roughly the same merit as a solution.  Don’t ever go in having already decided which one you really want!
2. Yet, you need to remember that all of the choices are also fundamentally different in their details. For example, one might have 30% greater functionality or come from a more established and stable company than the others, but recognize that this advantage probably comes with a cost premium. The market is generally pretty efficient that way.

Your goal in the negotiation process is to see where you can find “wiggle room” on each alternative’s components (price, terms, features) that will ultimately make one of those choices outshine the other, on balance.  In other words, you are looking to unsettle the equilibrium you’ve achieved in #1, so that one choice comes out on top. And here’s another key: let each vendor know, and remind them frequently, that this is what you’re doing. Usually, there’s no need (or benefit) in letting the participating vendors know exactly what your list of alternatives consists of, but it will suffice that they know that you know that you have choices.

Miscellaneous tips and observations:

• Negotiation, in general, is not simply about money, although that’s certainly a key dimension that can make one of your viable alternatives rise above the others.  Any facet of the deal should be open to scrutiny and discussion: cost, terms, rights to upgrade, SLAs, warranty, etc. You don’t always automatically choose the least expensive option, nor do you always choose the most featureful.

• Don’t ever tell vendors (and believe me, they’ll ask again and again) what your budget is for this project. They don’t tell you what deals they’ve cut when they’ve sold their product to other customers, do they?  Again, the fact that you are actively pursuing other alternatives needs to speak the loudest.

• Remember, though, that your goal is not to hammer down the price so far that the vendor won’t make any money on the deal. You can do that, at times, but it ultimately means that you’ll get poorer service and less attention when you need it.

• Where possible, involve your legal counsel in the negotiations; don’t just bring them in at the end. The best negotiations for me, in terms of viable outcome, were where I had a tight bond and common approach with my legal counsel.

In the end, it’s all about getting a viable solution for your need, at a reasonable price and beneficial terms for all. It’s not really about “winning”; it’s about choosing. Give yourself choices, and in the end, you (and your organization) do win.

Lagniappe:

• Ward Keever, “Ten Keys to Successful Negotiations”
• Martin Ewing, “5 Can’t-Miss Vendor Negotiation Tips”
• Michael Krigsman, “Negotiation Tips for CIOs”
• Peter Kretzman, “More tips for dealing with IT vendors“

A few years back, I had an extremely surprising and unpleasant experience as CTO. The director of my Program Management Office ran a weekly status meeting for project stakeholders, where we’d all methodically go through the current project portfolio, in order to communicate on issues, gather necessary feedback, and align everyone’s expectations. I typically attended in order to provide input and executive-level decision participation, but left it to the director to actually run the meeting and present the topics.

Unfortunately, immediately before one of these weekly meetings, that director was given bad news (in a brief hallway conversation, no less) about a major bug that had just been discovered in the software for our highest profile project, which was currently in testing and due to launch in just a couple of weeks.  This project, with its strategic and revenue-enhancing potential, was foremost in the minds of everyone in the company.  Stakes were high, in other words.

Without conferring with me or anyone else, the director announced to the entire group of assembled stakeholders that the project had hit an insurmountable snag in terms of our target launch date, and that it would be delayed potentially by weeks. No further information was available on specifics, and people’s concerned questions went unanswered. You could feel the anxiety spread through the room. I actually had to truncate the meeting abruptly and promise to reassemble the group when we knew more.

My director was quite upset with me. Rather than having my action be understood as a necessary attempt to prevent panic until we could get some real facts, I suddenly realized that I was being accused of wanting to “hide information” from the stakeholders. The director even told me, “Well, I like to be honest and tell it like it is, but that obviously just isn’t what is wanted here.” I was stunned. Here was a person who had the wrong idea of transparency, and astonishingly for their position, a warped idea of what project management is all about.

As most experienced project managers know, it makes utterly no sense to drag stakeholders through the entrails of every back-and-forth readjustment that is discussed and worked through in the course of a project’s lifetime.  To do so lowers stakeholder confidence, unnecessarily in many instances.  In this case, the director somehow wanted to be a kind of Paul Revere, forewarning the stakeholders, but instead came off as Chicken Little. Transparency can’t be an act of individual heroism; it needs to be a concerted, institutionalized, collaborative philosophy and approach.

Moreover, good project management isn’t about making everything go smoothly. There’s probably no project plan in the history of the world that has gone completely smoothly from beginning to end.  Instead, project management is about responding appropriately when it doesn’t, and recalibrating to keep everything still on track. In fact, it should be expected that not everything will go as planned, and your whole team should be prepared to take some bumps and glitches in stride without panicking.

It’s not hiding anything to hold back on discussing, in a public forum, a problem that has just been discovered, the ripple effects of which initially can only be guessed at.  When you promulgate potential bad news prematurely, it puts you into the position of trotting out the absolute worst case, before you really have any idea whether that worst case is likely or even plausible. In fact, it’s a fundamentally selfish act: it takes your own personal panic and extends it to as many people as possible.  And if there’s one thing we know about panic, it’s that it’s contagious.

You need to shoot for judicious and organized transparency. That’s where you plan on regular status communications, and you methodically, soberly assemble the facts in a structured manner before you make those announcements. Most of all, you collaborate internally: before you bleat about a potential crisis to the stakeholders, you bring in the rest of your team to talk through the possibilities and “degrees of freedom” in how you’ll act with respect to that crisis. Otherwise, you may think you’re nobly playing Paul Revere, but instead of imparting critical facts, you’re shouting “The British might be coming!”  And that’s not transparency, that’s mere alarmism.  Remember that especially during the dog days of a stressful IT system development effort, what often seems like a dire setback, upon careful and collaborative examination in the light of day, turns out to be merely a minor speed bump that is easily overcome.

Is it “controlling the message” to be cautious about spreading the alarm? You bet it is. And doing that is in everyone’s interest. Remember my adage: show that you’re in control. Your job as project manager (and, by extension, the job of the Director of the PMO, and the CIO) is to be in control and to demonstrate to the users that you’re in control.  That doesn’t mean that you “hide” anything; it does mean that you deliver bad news broadly only when you’re sure of your ground, have weighed the options for how to cope with that bad news, and are ready to present those options soberly instead of in a flailing manner.

Adhering to a philosophy of transparency is a far cry from shouting every blip or misgiving about the project from the rafters. This is particularly so if you’re in a leadership position. You want your communication to be transparent, yes. But without appropriate structure and caution, transparency can be the height of irresponsibility. In those cases, in fact, you’re just building a slightly more institutionalized rumor mill.

Lagniappe:

• Karen Guglielmo, “What is transparency, and how is it relevant to IT?“
• Phil Windley,  “Tools for IT Transparency“
• Peter Kretzman, “Why the CIO should air the dirty laundry“

Roger Sessions recently published a white paper on IT complexity and its role in IT project failure: “The IT Complexity Crisis: Danger and Opportunity”.  It’s certainly possible to quarrel with bits and pieces of his analysis, and thereby tweak his numbers, but the overall thrust remains undeniable: IT failures are costing the world incredible amounts of real money. Sessions even sums it up under the dire-sounding phrase, “the coming meltdown of IT,” and says, “the out-of-control proliferation of IT failure is a future reality from which no country—or enterprise—is immune.” And he presents “compelling evidence that the proliferation of IT failures is caused by increasing IT complexity.”  He points out that the dollar cost of IT failure in the US alone is close to the cost of the recent US financial meltdown, and cites indications that the rate of failure is also increasing by 15% per year.

Roger’s paper is excellent and thought-provoking, and I recommend it highly. And I do agree with his view that complexity is the chief culprit in IT failure. That said, I think his argument focuses a little too strongly on one cause of complexity (unnecessary overcomplexity of architecture), to the neglect of other important factors.

To be sure, some obvious contributors to IT failure (poor project management, and lack of communication within teams and from business to IT implementers) aren’t dismissed by Sessions, but he sees their contribution to the crisis as relatively small. I don’t, and I’ve used this blog to write about those factors quite a bit.

Most of all, though, I differ with Roger’s focus on streamlining architecture as being the key to reducing system complexity. One could say, in fact, that Roger’s solution is primarily a technical one, where the bugaboos I see are primarily cultural and sociological.  I see not one, but at least three distinct complexity-related burdens, increasingly endemic, and increasingly bringing down IT:

• Overly complex design/architecture
• Taking on too much functionality
• Poor implementation (technical debt in-the-large and in-the-small)

Roger has admirably dealt with the issue of overly complex design/architecture, at least in terms of a viable approach for simplifying up-front architecture, so I’ll focus here on the other two.

Taking on too much functionality

I recently rented an economy car (the least expensive option) on a trip with my son. (Remember, my self-appellation is “Cheap Technology Officer.”) He was stunned and dismayed that the car didn’t have automatic door locks; he didn’t realize that they even made cars without them anymore. Similarly, my 11-year-old daughter has grown up in a TiVo-ized world where live TV can always be paused. When she encounters a TV without a DVR (and thus no pause capability), she regards it as hopelessly primitive. Indeed, as unacceptable.

Similarly, the general standard for functionality and UI design has been raised by extremely functional PC software.  I now expect to be able to double-click on a number in any onscreen report, and thus “drill down” into the transactional details that make up that number. When I can’t, I feel cheated. Equally, I expect everything on an interface to drag and drop; I get frustrated if it doesn’t.

So as an industry, we’ve raised the bar of acceptability, considerably, in software and technology systems over the last couple of decades.  What that means in practical terms, though, is that across the board, our eyes have gotten bigger than our stomachs. We want more, up front, than it often makes sense to build at the start. And our demands are not negotiable, or so it seems. The first few cell phones I had didn’t even have a ring silencer on them; I used to silence the phone by adroitly disconnecting the battery when a call came through at an inopportune time. Today, most people wouldn’t even consider buying a phone that lacks much more elaborate features, such as a camera, that I would have considered as space-age in nature back in the 80s.

So our increase in expectations, alone, has added considerably to the functionality of systems we tend to build. There’s more functionality in and of itself, and usually more interface points to other complex systems. In fact, integration testing—where you connect new code into a working environment where it has to interface correctly with other systems—has become a frequent and major sticking point in launching information technology projects.  In essence, we’ve fallen into the “nuts” dilemma, both in large and small ways.  We want so much, and attempt so much, that we increase our risk of failure considerably.

Technical debt (in the large and in the small)

Any software developer will tell you that their first stab at implementing a given piece of functionality is often (if not usually) much more complex than turns out to be needed. Only after exploring the problem domain, with experiments and backtracking and restarts, do developers usually realize that their code can be pared down, simplified, combined with other modules, etc. This is usually called “refactoring”, and its importance is a relatively recent insight in the software development discipline.

A key insight about refactoring, though, is that it means improving the code without changing its overall results. There’s often no immediately obvious payback to this undertaking: it’s a roof project, in essence. To the extent that refactoring isn’t done (no time, no inclination, no recognition of a simpler approach), the end product is left with vestiges of unnecessary complexity. The greater the time crunch, and the greater the aspiration for the functional depth and breadth of the software to begin with, the more likely it is that these vestiges linger. And one ancillary aspect of the raised bar in expected minimum functionality is that it causes the time crunch to get ever greater.  It’s no longer about delivering just a solution that will work, it’s about making sure that the solution includes (metaphorically speaking) a 5-megapixel camera too.

Couple this unnecessary but accidental complexity with what often amounts to a rush job on design for the sake of meeting schedule (creating “technical debt in-the-large”). An example I’ve used here before: choosing a different core DBMS to implement a given function, simply out of expedience, and failing to take time to modify previous functionality to use that new DBMS. Supporting two DBMSes within the same product represents significant technical debt: every subsequent system change and addition will entail “paying interest” on that debt, which not only increases schedule and manpower costs, but increases risk of failure as well. And that’s just one example; the technical debt cascades, feature upon feature, release upon release. Technical debt, until paid down, can be equated to a invisibly rising substrate of complexity, and it contributes massively to an increasingly wobbly, risky system. And lately, I see more and more organizations “pyramiding” their technical debt, never taking the time and cost to pay it down, with disastrous results. As Hemingway said about going broke, it happens slowly, then all at once.

What now?

Roger’s analysis, to its large credit, outlined an important aspect of complexity and posed a solution (an approach he calls SIP, or Simple Iterative Partitions).  The aspects that I’m presenting (taking on too much functionality, and the pyramiding of technical debt) are, as I’ve said, cultural and sociological within companies. The answer is not nearly as simple or as neat as a specific technical solution (although I am certain that I will get comments on this post, perhaps rightly, from devotees of Agile).

To my mind, it’s engaged, savvy, forceful leadership that alone can address these issues, slow down the demand train, stop the madness. If anything, I think that there is an increasing lack of leadership in IT circles that can suitably recognize and address these factors, as well as educate their peers. And that’s what needs fixing most of all.

Lagniappe:

• Kevin Schlabach, “Agile East 2009 by Thoughtworks”
• Martin Fowler, “Technical Debt“
• Eric Ries, “Embrace technical debt”
• Johan Lindberg, “Painting With Hands and Feet: The IT Complexity Crisis”

Just a quick, personal post this time: I was recently interviewed by CIO Magazine on the topic of “How CIOs Can Stay Tech-Savvy“.  Since (as is normal) only a portion of my conversation with the reporter actually made it into the article, I thought I’d expand briefly on the topic here.

My remarks were two-fold, consistent with what I’ve written before on this all-important topic:

• It’s critical for the IT executive to “keep his or her hand in” by doing some hands-on work and experimentation with new technologies
• Your purpose in doing this hands-on work is not to become a viable technical resource in the area, but rather to get some deeper understanding than you’d obtain by just reading an article or two.

As mentioned in the article, I estimate that I spend 5-10 hours a month doing this kind of hands-on dabbling, sometimes with more success than others.  Let’s look at the kinds of things I do, large and small:

• Obviously, I administer my home network (four machines running three different operating systems, plus other home networking devices) and provide advice to neighbors and friends.
• I administer my blog, including configuration, changing Wordpress templates, and even custom-coding PHP callbacks at times.
• I also actively seek out “early adopter” opportunities with new technologies, or technologies that are simply new to me.  I currently have four virtual machines that are launchable on my Mac: Ubuntu, Fedora 11, Windows 7, and Windows Vista.
• I have an ongoing Javascript dev project I work on that analyzes my iTunes music library and helps me identify gaps in metadata and lyrics, so that these can be corrected. That Javascript also dumps all the lyrics in my music library out into XML, to get them out of the proprietary world of iTunes.
• At the beginning of each year, I list out the technologies I’d like to delve into more deeply that year, in terms of reading and experimenting.  This list is based purely on what has intrigued me as I’ve scanned blogs, feeds, and Twitter. For 2009, my list included Amazon EC2 and S3, Ruby, Heroku, and CouchDB.  I’ve not gone as far as I’d hoped with a couple of these, but hey, 2010 will have a list too.
• In a given year, I might do some coding in Javascript, Perl, PHP, and Ruby. Admittedly, I usually need to look quite a bit of stuff up, but that’s mostly a factor of doing this only an hour or two a week.

As I emphasized in my remarks for the article, the point here is not to become a player on the field. I’ll never be as skilled in any of these technologies as the people I’d hope to hire with that expertise, should the need arise.  And that’s a good thing: the temptation is always there, particularly for someone who rose up through the developer ranks, to micromanage.  But at the senior executive level, it’s far more important that you stay focused on process improvement and strategy than on nuts-and-bolts techniques. Any of the experimenting I describe above should be viewed as self-education and a hobby, not a serious endeavor.

But you can bet that my self-education practice lends me a deeper insight into any of these technologies than if I’d sat back and simply read magazine articles on them. And oddly, I’m one of the few senior IT executives I know who still do this sort of thing. Granted, it will always feel to me like it’s too little, but not doing it at all is, well, not an option.

“We went live before the system was ready”.  It’s a common excuse/explanation that I hear from IT people when they tell war stories about system launches that failed miserably. Implicit (and sometimes explicit) is the add-on statement: “and we told them so beforehand, too.”

There are obviously many things (and many parts of the org chart) that contribute to a failed launch, but here I’d like to focus on what drives this particular kind of launch-before-readiness, where the views of the rank-and-file are unheard or ignored.

In a nutshell: it’s management pressure. Sometimes that pressure comes from middle management, sometimes from the very top, and often from both.

Why do executives apply the pressure? There are a number of reasons:

• They actually believe it’s the best way to keep the team producing and working hard. They fear that extending the deadline will cause a let-up.
• Their own bonus and/or job is on the line if there’s a delay. They, or their management, regard a launch delay as a failure, just as much as a bad launch. At the very least, a delay means losing face.
• Testosterone (for either gender) sets in: a “refuse to lose” mentality. To some extent, that’s what has gotten them where they are.

Deadlines matter, of course, a lot, and people who are lackadaisical about deadlines simply shouldn’t work in IT. That said, if you bull-headedly insist on adhering to deadlines, to the exclusion of facts-based input, it makes you all the more likely to delude yourself that you’re really ready, just because the launch date has arrived and the pressures are heavy. If the stakes aren’t high, that may even be fine. (But, are the stakes ever not high in these situations?)

Let me tell three anecdotes that illustrate management “refuse to lose”:

• Earlier in my career, I was a middle manager, wrestling with a project that was both controlled by and staffed with a single “Big 5” consulting vendor’s people, with millions of dollars per month of consulting costs being racked up.  The high-profile initial system launch was a week away, and according to the bug tracker, there were still over 100 open major bugs. I sat down with the vendor’s partner-level project head. I don’t remember the precise conversation, of course, but it went something like this from him: “Oh, we’ve already got 39 of those 100 fixed in dev, 44 others are being worked by our best people and will be resolved by tonight, so really, there are only about 15. Four of those we think have workarounds so there are something like 10. I don’t think we’ll be getting many more bugs. We usually resolve bugs in a day or two, so even if each one takes two days, we’ll be able to close those ten and be ready for launch in a week.” I sat in awe (and frustration) at this hand-waving, this magic with math, and at his reluctance to think things through without an agenda. Here, “refuse to lose” had developed blinders, simple math had turned fuzzy and infused with vested interest, and IT best practices (not to mention common sense) had flown out the window.

• I worked as VP of Operations under a dev-oriented CTO. We had a late night launch, and the CTO was there in person, hovering over my sysadmins who were attempting to install the latest release into production with little success. The installation had failed, starting with the very first run of the script. “No, try this: run just those three lines of the installation script we provided you, then type these two commands, then run these other five lines.”  When that didn’t work, he came up with another idea, and then another. The next day, he sent out a company email declaring victory and congratulating his dev team, because the system had come up finally, even though it had run for a total of only two hours before crashing hard. Installation by seat of the pants. Refuse to lose.

• I arrived on the scene as the new CTO of a company in the midst of a complete replatforming project, completely retooling and replacing their sole product. The project had been underway for about nine months already; just before the point of my arrival, the team had been told by senior management that their launch date was going to be in six weeks, no ifs, ands, or buts.  I proceeded to do what I call a quick “temperature check”: I asked to see key indicators of readiness, such as bug counts over time and test case execution reports.

As you’ve surely guessed, I quickly determined that the facts didn’t even remotely support the possibility of a launch in a mere six weeks. About 40% of the code was still to be written; less than half of the test cases had been executed at all, with an even lower percentage having passed.  I marshaled my facts and figures, and went in to tell the CEO of this sorry state.  Judging from my experience, I told him, it looked like it would be a number of months, not weeks, before the product could be successfully launched. I remember him sputtering “that’s unacceptable”, and pressuring me on what I could do to, yes, bring off a launch in six weeks. (Cutting to the end of the story: we launched, successfully, nine months later).  In this case, a mixture of testosterone and probably some job-on-the-line aspects were driving this CEO to ignore facts and look for ways to bull through to success despite obvious counter-indicators.

The tragedy of the space shuttle Challenger has become almost cliché along these lines: a combination of executives feeling pressure and ignoring communication from lower ranks about major problems. “They also disregarded warnings from engineers about the dangers of launching posed by the cold temperatures of that morning and had failed to adequately report these technical concerns to their superiors.”  For whatever reasons, NASA management gave the green light for launch despite the adverse weather conditions. In essence, they “refused to lose”, and by doing that, they lost in a major way.

Takeaways:

• Don’t let your executives turn you, and don’t let them become, riverboat gamblers, where failure is a strong possibility through a premature launch. The company culture must be shaped so that it views a spectacularly failed launch as much more onerous than a delayed launch.
• Don’t let “groupthink” settle in, where your people are afraid to speak up. Any and every member of the team should have a chance to “pull the emergency brake”.  Especially, holding a formal “go/no-go” meeting is essential, where any stakeholder or project participant can voice concerns and point out facts that speak against readiness.
• That said, do be careful of the pessimist amid your ranks, who typically voices dire predictions on every project, and who sooner or later will be proved right. Insist on facts, not “Eeyore-ism”.
• Let your team watchwords for these situations become the following: candor, and facts. Bring both to the meetings, or go home.
• The best way to avoid letting pressure drive a premature launch: establish your launch criteria clearly and openly, with quantitative measurements, way in advance. If you don’t have legitimate facts-based ways to tell if you’re ready (i.e., not just the launch date arriving, and not your testosterone level that morning), well then, you’re not ready.

Without a doubt, “refuse to lose” can be a great attitude, up to a point. No executive, no team, should ever become blasé about missing a target. But “refuse to lose” can’t ever be allowed to become “refuse to face facts”. When it does, you don’t just lose face: you just plain lose.

I’ve done several posts featuring what I call “Peterisms”, which are basically aphorisms I’ve adopted that encapsulate hard-earned IT lessons. Let’s turn it around this time, and talk about two sayings that sound equally folksy-sensible, and that I hear again and again, but which I feel are actually dangerous to apply to information technology work. And, of course, I’ll discuss why that’s so.

• If it ain’t broke, don’t fix it

• A camel is a horse designed by a committee

So, not all aphorisms are created equal. If you’re in IT, I advise that you avoid the attitudes and philosophies contained in these two.

Consider these statements:

• Baseball is a game where the pitcher throws to the catcher.
• An iPhone is a device that lets you call anywhere in the world.
• The Grand Canyon is a tourist attraction in Arizona

You’ll have noticed that these statements aren’t wrong, per se. But they still take you aback, don’t they? They miss the point, miss the magic, neglect the important differentiators. By explaining too little, defining the subject too narrowly, they explain nothing that’s really useful.

Here’s another:

• Cloud computing is where you have a lot of intelligence in the network and it’s available from wherever you need to get to it

A distressing portion of mainstream media covering cloud computing has decided that the best way to explain the phenomenon is first to make hand-waving general statements such as the above example from BusinessWeek, and then cite a few consumer-understandable examples such as in this piece from NPR: “Do you have a Yahoo e-mail account? Maybe a Gmail account? Do you put up pictures on Flickr? Perhaps you’ve started keeping your schedule online. If so, then you are using cloud computing — that’s what tech companies call it when people work and store information on the Internet.”

Flickr, Gmail, and Facebook are great services, but declaring that they represent the burgeoning trend of cloud computing is as incomplete and unsatisfying as explaining the Grand Canyon as just a tourist attraction in Arizona.

The problem here, and the reason that so many of these mainstream articles get it so wrong, is they’re trying to explain cloud computing as a consumer-oriented phenomenon, and it’s basically not. Not the exciting or “new” part, anyway. Even technology vendors drift into this as they try to tout their cloud offerings: witness a recent TV commercial from IBM entitled “My Cloud: Virtual Servers on the Horizon”, a commercial which would work just as well if it were titled “the incredible power of the Internet”, or even, “aren’t computers cool?” Similarly, that cloud computing “definition” from BusinessWeek is, quite frankly, nonsensical in its broadness: it not only completely misses the point of what makes cloud computing relevant and compelling as a game-changer, it even fails to distinguish it from the last 15+ years of the Internet in general.

Mainstream media drifts into this oversimplification in part because they’re leery of delving into technical arcania (virtualization, scalable architectures, APIs) that many of their readers can’t relate to. Yet, there’s actually no need, when you try to explain its real impact, to make cloud computing sound geeky and complicated; it’s not, at least at core. I’m going to trot out some analogies here; like most analogies, they’ll necessarily gloss over some important complexities, and will only go so far before they break down. Nonetheless, they should give you a better idea of what’s different about this trend, in a way that talking about storing your photos on Flickr won’t.

Cloud computing is simply a way for a company to use someone else’s computing resources (servers, software, etc.), on demand, to fill its need, rather than buy and manage and maintain those resources itself. Instead of bulking up its own data center, the company uses as little or as much of someone else’s as its immediate needs dictate, on a pay-as-you-go basis.

Does that still sound complicated? Okay, think ZIPcar. Rather than own your own car, (purchase it, license it, insure it, maintain it, fuel it, pay to park it, etc.), you can choose to use a ZIPcar that’s parked near where you live. You reserve it, walk up to it, and drive it away as if it were your own. You pay an hourly or daily fee, to be sure, but perhaps you don’t need “fulltime, anytime” access to a car, and it works out to be both easier and cheaper to get one when you need it, and not worry about all the ancillary details. Are there downsides? Sure, and these will vary depending on your situation. This solution may be great and cost-effective for you, but not work at all well for your neighbor, who has different needs.

In the not-too-distant past, ZIPcar wasn’t available to you as an option. Neither was cloud computing. If you started a company that provided an online product or needed internal systems, you bought servers. And electricity and cooling. And storage. And software. And you hired people to set that all up for you, and keep it all operating smoothly. And you tried to anticipate your demand, and to make sure you had just the right amount of capacity (not too much, not too little) for your customers or users. Almost always, that meant you bought ahead of the curve, and you sat on (and paid for) your excess capacity while the demand increased.

Enter the cloud. Now, depending on your company’s situation and needs, you don’t need to sink in resources and funds (and risk) up front. Reserve your server, and (metaphorically) walk up to it and drive it away as if it’s yours. Let it go when you’re done, and poof, it’s effectively gone; no more overhead. Think about the sheer power that possibility represents. Think of the reduction in logistics and interdependencies. Think about how much less risk the company has incurred, if your plans happen to change. The potential independence, enablement, and empowerment that the cloud brings, particularly for new and small businesses, is as close as anything I’ve witnessed to the way the industry was shaken and shaped by the advent of the PC, starting in the early 80s. And that “disruptive technology” nature of the cloud, astonishingly, is what’s being missed by the kinds of articles in BusinessWeek and NPR that I’ve cited, not to mention by the myriad old-timers who like to sneer loftily that nothing here is new.

Lest I be accused of being starry-eyed about the cloud (to mix some firmamental metaphors), let me make sure I acknowledge here how early this technology is, how key aspects are still being worked out, and that it’s not a panacea. And, like ZIPcar, it’s not for any and every situation. But none of those caveats detracts from the cloud’s potential and the ground-shaking nature of the phenomenon.

It’s all about cost, flexibility, time to market, and risk mitigation, basically, for businesses. Just a couple of quick examples: eHarmony recently did a project where they took a monthly expense of $5K down to $1.5K with cloud computing. And here’s another study of a startup using cloud approaches and reaping a lot of benefits. As a CTO/CIO, I can personally attest to how much time and heartache goes into planning capital investments and attempting to right-size infrastructure; anything that can simplify and streamline that thorniness is welcome indeed.

Remember, everyone wants IT to be less costly and more flexible, to focus on business needs more than on technical minutiae, and to be able to turn on a dime to meet new needs. Cloud computing will be key to fulfilling those desires. In fact, I believe it will be revolutionary to the industry over the coming years.

Lagniappe:

• Maria Spinola, “An Essential Guide to Possibilities and Risks of Cloud Computing — A Pragmatic, Effective and Hype-Free Approach For Strategic Enterprise Decision Making“
• Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google
• Lori MacVittie, “Cloud Computing: The Last Definition You’ll Ever Need“
• Cath Everett, “Five cloud computing myths exploded“
• Jeffrey Burt, “Gartner Predicts Rise of ‘Cloud Service Brokerages’”, July 9, 2009.
• Brenda Michelson, “Cloud Computing Picks for Business Analysts“

Are you among those who believe that if you don’t follow someone back on Twitter, you’re being snobby and arrogant?  Then this post is meant for you. My purpose here, quite candidly, is to persuade you that reflexively following someone back is not only a habit which encourages spam, but is in fact a major contributor to making Twitter a thriving spam platform.

For those who reflexively follow, in other words, I ask you to consider the ramifications of your behavior to the greater community, especially when multiplied by the thousands or millions of Twitterers who may behave likewise. Basically, you’re helping the spammers win.

First, let’s think about this: why does anyone follow anyone else on Twitter?  Three main reasons come to mind:

1. The follower believes that the person he’s following has interesting things to say, and wants to read those interesting things;
2. The follower is hoping that the person he’s following will follow him back, for one or more of the following reasons:

1. so that the follower’s count will increase
2. so that the follower’s messages will then have broader distribution/marketing power
3. so that the follower can send Direct Messages (DMs) to that person for even greater exposure and attention.

3. The follower is reciprocating being followed, out of politeness, sense of obligation, or idealism. Often, there’s a belief that following back will “strengthen the relationship.” You can “transform them into a fan with your valuable tweets“! Or so goes the claim.

I’m arguing here that the first of these behaviors is useful, the second describes spammers and marketers above all, and the third is a well-intended but unfortunate fulfillment of the spammer’s hope, one which encourages their continued activity.

Reciprocal following by rote in many cases does little to further a relationship.  Remember that they call Twitter “asymmetrical”.  Let’s use myself as an example.  I tweet about a fairly narrow range of topics, basically: IT management, cloud computing, and sometimes interesting or amusing industry or sociological matters.  I happen to have a broad list of other interests that I myself don’t typically tweet about but helps me pick whom I follow: literature, languages, music, politics, travel, theater, to name a few. I follow people solely for the first of my reasons listed above: because I want to read what they have to say. Again, it’s asymmetric: my reading interests are not the same as what I tweet about. Anyone who follows me simply because I followed them (i.e., behavior #3) may be taken aback by how uninteresting my tweets are to them.

Let’s look more closely at what happens when I’m followed by people who are unlikely to be interested in the content of my tweets.  If you watch closely, you’ll notice that you are often followed by entities (read: marketers and spammers) because of a keyword contained in something you’ve tweeted.

Just to give some examples: recently, I’ve been followed by:

• @HerbiesHeadshop, because I happened to use the phrase “down in the weeds” in a tweet;
• @BuilderPal, because I wrote and tweeted about what I call “roof projects” in the context of information technology;
• @proxyserver, because I happened to use the term “proxy server” in a tweet.

I’m not interested in any of the services these people provide, so I have no reason to follow them back. But more important: I’d argue that none of the above entities is remotely interested in my tweets. As a matter of fact, I am fairly certain that none of the above entities, or actually anyone who follows people as a result of keywords harvested from the Twitstream, is even reading my tweets, let alone anybody else’s. They just want to be followed back, so that I will then be more likely to read their tweets. They are using Twitter as a means to one end: marketing their services. Advertising. Exposure. Page views. And so they target (however clumsily) people they believe are interested in the products they provide. It’s kind of a new (and cheap) way of generating a targeted email address list.

Let me say it again: people who follow you using behavior #2 usually have NO interest in your tweets and in most cases aren’t reading anyone’s tweets at all. You are just a means to an end.  It’s not about you, it’s about them. If you’re laboring under the illusion that following someone who follows you is a way of strengthening the relationship, you should recognize that that strength is quite often going to be one-way only.

I could, of course, simply ignore any of these follows; most of them will eventually unfollow me anyway, not because of the content of my tweets (remember, they’re not even looking at these), but simply because I didn’t bite: I didn’t follow them back. I didn’t help them meet the sole objective they had in following me in the first place.

How do the spammers win? Spammers really only have an audience on Twitter if people follow them. More followers mean higher positions in search results for their pages and products. Most notably, following a spammer gives them the ability to Direct Message you, which increases the likelihood you’ll see and read their message and click on their links. And let’s be clear: the only reason people follow spammers is this strange perceived obligation that’s arisen on Twitter: follow everyone back who follows you; “it’s only polite,” after all, or it’s “snobby and arrogant” not to. When you succumb to that perceived obligation, the spammers win. If no one followed back, the spammers would go away.

It’s easy to turn behavior #3 (reflexive following) into behavior #1 (following because you’re interested in the person’s tweets): before you follow someone back, simply review the last 10 tweets the person has sent, which turn out to be astonishingly predictive of whether they’re a spammer.

You’re obviously welcome to use Twitter as you please and follow people for any or no reason. But consider the points I’ve made, in terms of the impact on the Twitter community of your actions.  And if you follow reflexively, don’t ever complain about getting spam DMs, because it’s your behavior that got you into that position.

Use Twitter for education, conversation, and interaction. It’s really the only thing it’s there for (other than providing rife subject matter for TechCrunch articles). Unless, of course, you’re a spammer.

Lagniappe:

1. Atherton Bartelby, “FOLLOW FAIL: The Top 10 Reasons I Will Not Follow You in Return on Twitter”

2. Skellie, “How to Follow Everyone Back on Twitter Without Ruining Your Experience”

3. Roger Hjulstrom, “Don’t let anyone tell you how to act on Twitter”

4. Twitter, “Turning Up The Heat On Spam”

5. “How spammers use Twitter”

6. Stop Twitter Spam blog