Thank-you, this has been driving me crazy for years with Microsoft office. Since XP from a business productive prospective to the average business user Microsoft issue has not been opearting system, it has been office. Feture such spell check, gramer check, print preview in Excel are all things which have remained unchanged for at least a decade and need improvements. I do not need new development tools. Make what I use the most better.

Overall a good artical.

Note that the above is only possible with a modifiable and extensible design. The design follows from the architecture, so a modifiable and extensible architecture is crucial.

Let me toss out a genuinely radical concept. If we admit that much of the problem is complexity, maybe it’s (past) time to admit that the systems we are trying to build are simply too complex to develop using human methods.

We accept this prinicple in modern cpu design, which has grown too complex to be efficiently design and verified by humans. Instead, we use heuristics.

Our systems have similarly grown too complex for us to manage by human minds alone, whether we are talking military/aerospace, transportation, energy, or information technology.

There are methods and tools available providing such technology for systems and software engineering. These technologies involve formal methods, however, and have therefore been avoided by the systems and software engineering industry.

Companies like Praxis High Integrity Systems use such technology integrally in their process. They are a very notable exception to prevalent industry practice with consistent deliveries on schedule, under budget, and having the lowest defect rates in the industry. They even have a zero-defect project delivery in their portfolio.

The hurdles to adopting such technologies in the rest of the industry are cultural rather than technological. I often wonder what it would take to overcome these hurdles.

it must also be true that if you can break up a big complex project up into smaller simple projects that can be managed autonomously (as I claim can be done with SIP), then you are going to be more likely to succeed. So whether you say the primary problem is complexity or project management, the bottom line is that you must deal with complexity before you can effectively deal with project management.

The key realization for me is that you repeatedly use the phrase “deal with complexity” mainly in the architectural sense, essentially as an equivalent “code phrase” (for you) for the SIP approach. I certainly am inclined to applaud what I understand so far about SIP as a philosophy, but I disagree that it is as all-solving and all-resolving as you tend to imply (despite the caveats you’ve made and which I’ve heard) by your concerted focus on that approach. And I also differ with your preceding assumptions in subtle ways: for example, only certain kinds of projects can be (or are) practically broken up into autonomous smaller simple projects, and there is (as you have acknowledged in your white paper) risk and overhead involved in doing so (multiple projects requiring a kind of metamanagement if nothing else). In addition, I’m not convinced that breaking a large project into smaller ones indeed brings, prima facie, significantly lower risk. There may be a “tipping point”, up to which suboptimal architectures can be absorbed just fine by a given organization (due to size, people’s skills, etc.).

Finally, there are many many more reasons, both technical and non-technical, for failure of a given system beyond the mere architecture-in-the-large of that system. SIP, while laudable, may be (and I’m being overly dramatic here to make a point) akin to strengthening the front door of a house against burglary, even while burglaries all around the neighborhood are happening through second-story windows. Technical reasons for system failure could include performance issues, or inadequate interaction with other systems. (Better partitioning of subsystems is by no means guaranteed to address such thorny integration issues, and I’d argue it doesn’t necessarily substantially reduce their risk). Non-technical reasons for IT system failure, particularly given the typically loose definition of such failure (“cancelled prior to completion or delivered and never used”), might be purely political: a new CEO comes in who wants to sweep clean, or requirements gathering was botched such that the delivered system was wholly inadequate to the users’ needs. None of those kinds of failure has anything to do with the optimal or suboptimal partitioning of what was actually built.

I hope this hasn’t been too long-winded, but I needed to attempt to clarify where you and I differ. In sum, that difference is on what I see as your (laudable, defensible, groundbreaking even) approach to addressing a major cause of IT system complexity. You tend to view that approach, which is largely a technical one, as the fundamental lever which will make the difference in the necessary struggle against complexity overall, while I insist more strongly that the complexity problem is manifold and requires simultaneous efforts on a variety of fronts.

At its worst, I’ve seen IT become little more than “order takers” for the enterprise — relegated to asking questions that are essentially equivalent to, “oh, do you want fries with that?” and obediently scribbling down the answers. That approach of course seems cooperative and agreeable, but in truth, treating requirements gathering that way is actually a form of neglect of one’s responsibilities to the greater good of the enterprise. Ironically, it often leads to long-term failure rather than success. Don’t let this happen. Instead, IT people need to be there at every juncture, going full throttle, to challenge and to help mold requirements towards greater viability and cost-effectiveness.

Thanks for commenting!

The key is to start small and then add incrementally over time. At each stage really testing the new bits so they work and adding more and more value.

Too often the requirements are for an iPhone, IT is an order taker not part of the organisation, accepts them rather than challenges. The result disaster.

I think the first point on which we seem to disagree is on the relative contribution to IT failure of issues like poor project management, lack of communications, etc. You say that I see their contributions as relatively small. But I don’t see these as small. I agree that these are important issues. It’s just that I believe that can’t be dealt with effectively until complexity is dealt with effectively.

Let’s look at project management as a representative member of the set. I think that you would agree with the following points:

- The more complex the project is, the more difficult it is to manage.
- The more difficult a project is to manage, the more likely it is to be poorly managed.
- The more poorly a project is managed, the more likely it is to fail.

Now it seems that if you agree with all of those points (I’m assuming that you do), then you must also agree with the inverse points:

- The simpler the project, the easier it is to manage.
- The easier it is to manage a project, the more likely it is to be well managed.
- The better a project is managed, the more likely it is to succeed.

So based on these presumed agreements, it must also be true that if you can break up a big complex project up into smaller simple projects that can be managed autonomously (as I claim can be done with SIP), then you are going to be more likely to succeed. So whether you say the primary problem is complexity or project management, the bottom line is that you must deal with complexity before you can effectively deal with project management.

Now it is certainly true that you need good project management skills. If you can’t manage a project, that you can’t manage a simple project. (See my earlier post for my definition of “simple” project.) So dealing with complexity does not mean you don’t need good project managers. My point is that if you don’t deal with complexity, then you are likely to fail regardless of your project management skills.

My analysis of “taking on too much functionality” (which, again, I fully agree with you about) is similar. When the project is too complex, you can’t tell whether or not you have too much functionality. It’s too complex to figure out.

Now as to your point that complexity issues are primarily cultural and sociological, especially around the area of technical debt, we largely agree here as well. I hadn’t thought much about the metaphor of technical debt until you brought it to my attention, but I think it is a great metaphor.

From my perspective, one of the largest components of technical debt is not addressing complexity at the start of the project. Part of the reason people don’t do so is that they don’t know how to do so. But the bigger part of the problem is exactly the issue you mention, lack of leadership.

In my experience, far too many IT leaders are much more interested in blame avoidance than risk avoidance. So they continue to do things the way they always have, despite all of the evidence that the way they have always done things doesn’t work.

So I’m not sure that we disagree on anything. With a little time together, I think we could pull our respective ideas together into a nice neat package.

These are great discussions!

But that’s my point. I’m a practical CIO; I’ve gone in, both as an employee and as a consultant, to quite a few “turnaround” situations (IT departments and projects in serious crisis), and I can attest that the number one thing I hear from the business stakeholders about the delivery to date is “they [IT] don’t even ask us what we need! And then they just give us something that doesn’t do what we need it to!” We all know that a key IT focus over the last decade has been the need for increased business/IT alignment. There’s room for lots of disagreement on the hows and whats involved there, but the answer is simply not “we’ll figure out what you need and give it to you. Trust us.” If any one attitude/approach is a veritable recipe for IT failure, it’d be that.

In short, it’s lofty, arrogant, and usually plain dead WRONG to say “we’ll redefine what you want.” (On the other hand, you can work with stakeholders to achieve a mutual redefinition of what’s needed. That, astonishingly, is called requirements gathering, as David pointed out.) Of course you can point at examples where a unilateral redefinition approach has worked (although I don’t think Twitter is a particularly good one, with its issues in every conceivable area –performance, UI design, functionality, business model. Apple (iPod, iPhone) is perhaps a better thing to point to. But mainstream business systems? Seldom.

But I think we’ve been sidetracked, as this is a fringe issue, as I’ve stated above. Roger, your reply earlier that related directly to my post has again focused on architecture (application partitioning, etc.), as if that’s perhaps not the only thing, but certainly the main thing in your eyes when it comes to complexity. My blog post argued that it’s not. Do you continue to disagree? I was hoping, I suppose, to have “moved the needle.”

1. We have to consider the ‘continuity’ of this stuff. We tend to treat (mainly because of funding) such efforts as 1-offs and do not create persistent artifacts and architectures that puts what’s being done into a larger context (artifacts noted here http://www.fastforwardblog.com/2008/07/07/transparent-and-explicit/) Every initiative starts from scratch. There is no shared learning.

2. Design Thinking helps us to see that we start with leveraging some of these artifacts as a means by which to challenge assumptions (there are often many critical and fundamental assumptions that have been held for so long in companies that everyone believes that they’re true). Bringing these assumptions out to be challenged is critical (the visual artifacts and discussions around them help). And as we uncover facts to challenge assumptions we need to create a persistent sharable collection of these findings for a continuous business context — to inform new efforts http://www.fastforwardblog.com/2009/07/31/the-context-of-intent/

3. The whole argument of requirements/testing having been long established is of no consequence when it becomes clear that they help cement reliability but do nothing to balance it with validity http://www.fastforwardblog.com/2009/08/07/reliability-vs-validity/. Yes, if we’re building rockets, reliability is of greater significance, but most of what’s happening in business today requires a greater focus on validity (which is where the challenge of assumptions becomes so critical). The current methods and models are still on a path to lock down reliability. This is fundamentally flawed for most of the business problems we are addressing today (as Roger noted, not to say that there aren’t scenarios that still require it).

4. The SDLC is fundamentally flawed. The ‘design’ phase is a sub-design phase. There’s a larger architectural design that should feed individual initiatives, similar to the model engaged by commercial construction. Individual projects should be trades responding to a general contractor as part of a larger initiative with defined blueprints and specifications )http://www.fastforwardblog.com/2007/09/20/crossing-the-chasm/). Any ‘requirements’ that the sub-contractor want to provide are for their own purpose of fulfilling their response to the larger initiative (the business at large).

IT fails at understanding true architecture. They even behave as sub-contractors — seeing only their piece of the overall construction effort (e.g. landscaping is the center of the project and everything else revolves around it).