When I first started writing code in the early 90′s, the software industry was in one of its constant phases of reinvention. At that time, the game changers were object oriented programming (OOP), and (to a lesser extent) what we now call model-driven development (MDD). OOP is just a fancy way of describing programming languages that let programmers model their software systems in terms of objects like Person, Employee, Customer, Order, etc, instead of the somewhat less expressive languages of the past which focused on procedure and algorithm over entity. MDD is just a set of agreed-upon graphical notations for describing various aspects of a software system.

Anyway, the emphasis among professional developers at that time was on a highly structured, phased approach to building software. The first phase involved talking to customers and other stakeholders to determine what the system should do (the so-called ‘requirements gathering’ phase). Next came the design phase, when architects (who may or may not have actually written code, but almost certainly were assholes) would write documents and build graphical models of the solution as they envisioned it based on the requirements documents. During the development phase, programmers (who may or may not have been involved in the design phase) implemented the spec from the architects. In the testing phase (which may or may not involve any actual users), testers tested the software to see that it implemented the requirements correctly, and developers fixed any bugs the testers found. After that came deployment, when admins rolled out the software to actual users, who in all likelihood would be seeing the new system for the first time at that point and would hate it. Last the project ends in maintenance mode, where all the requirements analysts, architects, developers and testers with any ambition or skill had long-since left the project leaving only the dregs and those without knowledge of the system to keep it running.

As a kid learning to write software, I didn’t know enough to question this model, but as I grew more experienced I begin to find it woefully inadequate. Requirements analysts cannot hope to capture all the details of the system as the user representatives articulate them. User reps cannot hope to think clearly and far enough ahead to articulate all their needs unambiguously to the analysts. Inevitably corner-cases or users with odd needs will be left out of the discussion. Architects are not likely to build a system that can accommodate these unspoken needs, and often struggle just to implement the requirements as given. Developers have trouble keeping faithful to the design, particularly when the design doesn’t anticipate a conflict or gotcha, or is based on the world within the architect’s reality distortion field. Testers can’t understand the requirements from the user’s perspective, so they inevitably end up dutifully conforming to the letter of the requirements docs. Admins won’t know the system and may have knowledge of the deployment environment that would’ve been handy back in the design phase. Maintenance is shit-work nobody wants to do, so it’s often done poorly. Fred Glass articulates this much more clearly in his book, Facts and Fallacies of Software Development, but now I digress.

I’ve come to see this as but one of many examples in nature and human activity wherein complex, ordered, top-down systems fail to scale. Anyone who’s hit a particularly annoying deficiency in Windows or Visual Studio knows what I’m talking about. Whatever underlying phenomenon causes complexity to explode and exceed the ability of host organizations to understand it has the same influence on markets and societies as it does on software.

By contrast, the growing success of the Agile movement, which focuses on small groups, tight integration with the customer, and frequent, short iterations of the requirements/design/develop/test/deploy cycle over the life of the project, demonstrates the opposite approach to software, somewhat akin to laissez fair economic policy: small groups of motivated actors converging on a right answer iteratively, constantly responding to inevitable changes that couldn’t be foreseen at the outset of the project and thus are not. I’m of course horribly simplifying agile methods, but that’s the general idea.

One of the most striking features of agile teams that is hard to accept if you’re more comfortable with the top-down rigor of heavy process is the almost casual approach to possible changes. Since each iteration is short, and there’s a tight feedback loop from end-users to developers, agile teams don’t waste time trying to cover all their bases, design for possible future contingencies, or building elaborate frameworks to accommodate every imaginable need, because they know they can respond _if_ any of those things come up much more effectively at the time they face those issues, and thus don’t dwell on them from afar.

When I read this from paper Cato about spontaneous order arising from anarchy, I immediately thought of agile methods. Some of the same emergent behaviors appear in the resulting software, just as they do in human societies. It’s yet another example where ‘common sense’ is actually wrong.

A further wrinkle in the IT development experience (as in all multi-human activities) is along an orthogonal axis in humanspace: personality. Sometimes the ‘detrimenal’ and ‘useless’ devs are sociable, pleasant, likable as people, perhaps even ingratiating. Similarly, the ‘good’ and ‘heoric’ devs are occassionally insufferable assholes, or more likely emotionally stunted borderline-autistic introverts (like, uh, me). I personally have fired a borderline plugger/good developer due to a toxic personality, and I’ve knowm a couple pluggers who really needed firing, but were such decent people that the economically obvious termination decision was not so clearcut.

For those on the top of the pile, working with the mainstream masses can be quite trying at times, as Bill relates. I suspect this is a problem at even the most elite of organizations; even with an elite population, some are more elite than others. The answer is clear, and I’ve known it all along: humans are not worth the hassle.

I had this article in my unpublished queue for a couple weeks, and have only now come back to it. I read Bill’s piece on the development skills continuum, which reminded me of the epiphany I describe in this post.

One of Bill’s assertions is:

a successful company could be formed merely by avoiding the above groups [mediocre devs] and only allowing in good developers (or better)

I have long held this opinion as well. However, my work on federal government IT projects at BearingPoint has brought about an epiphany of sorts, which leads me to question this belief.

I acknowledge that an operating system, a search engine, even an office productivity suite or a web browser, cannot be reliably and efficiently developed by the mediocre half of Bill’s skill continuum, the ‘detrimenal’ and ‘useless’ classes. The nature of the work is substantively a technical challenge, which must be met by technically competent people (‘pluggers’ at minimum, with at least some ‘good’ or ‘heroic’ devs thrown in for force multiplication).

However, the custom IT development work which I’ve been engaged in for the last couple of years it unlike operating system development, search engine development, or productivity suite development. The technologies are usually well-understood, the specifications seldom present significant technical challenges, and the quality of the resulting system is less important (although not totally irrelevant; an ususable system might still be refused).

My conclusion after working on a typical IT project with a team that spans almost the entire skill continuum (I don’t think we had any heroes) is that ‘good’ developers can do work alot faster and much better than ‘pluggers’, as would be expected. However, given strong manager and a realistic schedule (that is, one that reflects the weakness of one’s team), pluggers are only slightly less likely to product an acceptable system.

I attribute this conclusion (albeit empirical and qualitative) to a few observations:

• The clients for whom these IT projects are performed usually lack the sophistication required to discern between ‘plugger’ and ‘good’ work products. While I suspect a client might favor the work of a ‘good’ team when compared side-by-side with the ‘plugger’ output, the client is nonetheless unlikely to refuse the ‘plugger’ deliverable because the code stinks, test coverage is practically negative, and it takes two days to set up a machine capable of building it (quality issues may drag down the ‘plugger’ deliverable, but you’d be amazed at the high shit tolerance in large IT projects).
• Non-technical risks dominate the technical ones, reducing net project impact a ‘good’ dev can make. Wishful schedules, missing, ambiguous, and constantly changing requirements, client-side political wrangling, and dependencies on other teams present significant risks which even a ‘heroic’ dev would be hard-pressed to eliminate.
• IT and large consulting firms tend to be staffed by ‘plugger’s and ‘useless’ devs (not surprising considering the likely statistical distribution of tech skills among likely candidates). Thus, their culture and processes are designed for (and probably by) such skill groups. This further impairs the ability of ‘good’ and ‘heroic’ devs to leverage their superior abilities. This can manifest in a number of ways, such as:
• Inadequate equipment and tools, selected by PHBs, bean-counters, and cable monkeys
• Minimal incentive and stimulation to achieve
• Work assignments typically lacking the challenge which intelligent minds require for stimulation and maximal performance
• The client environment is frequently plagued by ‘bullshit’ constraints, which cause frustration, disillusionment, and cynicism among technical minds inclined to roam a bit more freely

All that said, I’ve spent alot of my career in commercial software development, and I can’t say any of the above factors are completely absent. I do feel, however, that they are less pronounced, and the economic value of the technical endeavor is higher, leading to a higher probability that a ‘good’ or ‘heroic’ dev is the difference between likely success and likely failure.

I wonder what it’s like to work at Google…