Programming and software development, medium-rare

In a past life as a "software engineer" on contract, a favorite analogy of coworkers was to compare software development to construction (perhaps influenced by the early 2000's housing boom?). Projects were like houses, plans were needed to properly architect the end result, and schedules were laid down to ensure that the "foundations" were done before we started "framing" the building and "furnishing" the details. Conceptualizing software development in such a way is common, and there's a long history of people involved in what has casually been called "software engineering" thinking that what they do is, or should be, related to "old-world" engineering.

This is largely both nonsense and decidedly unnerving.

I'm not like my grandfathers

Both of my grandfathers were involved in engineering; knowing something of what they did makes me even more sure that what I do is not related to engineering.

One was an electrician, more a tradesman than an engineer, who worked in the construction of large commercial buildings in downtown Hartford and Denver. Each day, he would look at the blueprints painstakingly drafted by the project's architects and engineers, and go about making those plans a reality – installing high-voltage switching equipment, stringing miles of Romex (or, whatever they used back then), and doing all of it in hazardous conditions.

My other grandfather was, as far as I remember, something of a process engineer at a subsidiary of Olin, helping to design the manufacturing processes that would heat, pound, roll, cut, stamp, and test thousands of varieties of copper and stainless steel foils, strips, and other formulations for later inclusion in all sorts of products, both industrial- and consumer-related.

These men's careers were very different, but they were involved in what are clearly engineering tasks.

An art's constraints are what define the art

There's a lot that separates my discipline from my grandfathers', but I think the most significant is that, as someone who builds software, I have far more discretion in how I achieve my end results than they had in their work. The degree to which this is the case cannot be overstated, but I'm at a loss for words as to how to concisely characterize it. Materials in the real world behave in ways that are, in the modern age anyway, understood: electricity and copper and steel and wood have known physical characteristics and respond in known ways to all of the forces that might be applied to them.1

In contrast, the world of software has so many degrees of freedom in so many vectors, the possibilities are functionally limitless. This is both a blessing and a curse, as it means that the programmer is something of a god within her domain, free to redefine the its fundamental laws at will. Given this context, it's no wonder that software projects fail at an astounding rate: we simply have nothing akin to the known natural constraints that are conveniently provided for our real-world engineer friends, so we have no option but to discover those constraints as we go along.

The software community's response to this has been to erect artificial constraints in an effort to make it possible to get things done without simply going insane: machine memory models with defined semantics, managed allocation, garbage collection, object models, concurrency models, type systems, static analysis, frameworks, best practices, software development methodologies for all stripes and inclinations. This is natural, and good, and the only way we could possibly make sense of this thing called software given how young it is.

Yet, even after all this edifice, ask 100 software developers how to build a website, and you will get at least 500 answers – and the people that respond with a hesitant "It depends" are likely the most clueful of the group. If my grandfather had responded "It depends" to a question about how to produce a 1-ton spool of 2mm-thick copper strip, he'd have been fired on the spot.2

Confessions and snotty ego trips

Interlude: If Top Chefs were programmers

Susur's work is remarkably intricate and exhibits a delicate precision unmatched by his peers. He prefers the crystalline perfection of a perfectly-balanced type system, and so chooses Haskell for most of his work. Living up to his "Obi-Wan" moniker, Jonathan makes the most familiar t hings amazing, and people are never sure how. His secret is that he usually works in some lisp or scheme, which he then uses to generate whatever comforting form his customers prefer, often Java or C#. Marcus likes to surprise people with the most esoteric things possible: his pièce de résistance is written using his own prolog variant, implemented in Factor. People love it either for the ballsiness of it all, or because his stuff works so well they don't notice. Rick is a madman, insisting on using C++ for everything, even web applications.

Okay, so software development isn't engineering. It's probably safe to say it's a craft, though (in contrast to pure art like painting or sculpting, but that's a different blog post). In particular, its similarities to cooking are legion, something that I noticed while watching one of the few bits of television I bother with (and a guilty pleasure), Top Chef (though I redeem myself oh-so-slightly by preferring the Masters incarnation by a mile). blush

Watching this show with an eye towards the methods and attitudes of the chefs is like watching a group of programmers struggle to build quality software. Functionally no constraints in process, methodology, and materials? Check. Infinitely many ways to get the job done depending on the skill of the craftsman, the tastes of the customers, and the specifics of the materials? Check. Identiying the best craftsmen is difficult, and most of those at the top of their field have a murky combination of ill-defined qualities? Check. A wide disparity between the effectiveness of individuals in the field? Check. At the edges, people experiment with hacks that sometimes come to be part of everyone's repertoire? Check. Technical capability is not a necessary requirement for success, due to fluctuating trends (less charitably called "fashion") and a variety of potentially-compensating personal characteristics? Check. Less mature (and often less capable) members of the community are given to snotty ego trips, bad attitudes, and frequent tantrums? Check.

Given all of the similarities, I think the fact that it's difficult to assess the quality of individuals in either field is the most striking (and perhaps a key indicator that a field has not (or intrinsically cannot) internalized a certain minimum degree of rigor in its methods). It's telling that TopCoder predated Top Chef by a decade. Great hackers and those guys that can hit the high notes are few and far between, mixed in with scads of cooks that come to work stoned and "programmers" that put HTML on their resumés that still manage to get hired, somewhere. These are fields that are far from science, far from engineering, and well within the lush, rolling hills of craft.

Enough already. So what?

First, words matter and it's important to call a spade a spade. Thoughtlessly (or disingenuously) call software development "engineering", and people walk off with all sorts of notions that are inappropriate. This is particularly damning with customers and other nontechnical "civilians".

Second, craft, as revered a concept as it is, is not desirable when businesses, livelihoods, and actual lives are at stake. I've never worked on aeronautics software or the like, but despite its codified standards, its rigorous testing protocols, and access to millions and billions of dollars in resources, we've crashed satellites into planets because of something as triflingly simple as conversion between metric and standard measures. Similar examples are everywhere. We need software development to be built with an engineering discipline, because everything we do depends upon it.

Of course, I have no tidy answers for that challenge. I think that if we can pull ourselves out of this primordial ooze of twiddling bits and get to a point where we describe how to do things relevant to the domains we're working in, then I think there's a chance for the species to survive. Ideally, domain experts should be doing the bulk of the "programming" work, given that communication between such experts and programmers is probably the source of at least half, if not the vast majority of software design flaws. This is an old chestnut, dating back to the 4GL programming languages (now 5GL?), declarative programming, expert systems, and so on.

Programmers' work will be done when we've put ourselves out of a job

Roughly, I think the point is to solve for X:

blueprint:building :: X:code 3

One strain of real-life examples – such as Pipes, DabbleDB, and WuFoo – allow people to collect and process data without touching a line of code. The best specimens of these kinds of systems are often labeled with the "tool" slur by programmers, but the fact is such things allow for greater aggregate productivity and less possibility for error than any similar purpose-built application. Meta-software that can deliver the same for any given domain is the future, and the result will be that programmers, as they are known today, should cease to exist in commercial settings.

The craft of programming will survive though, and continue to be the source of innovation. After all, there's no shortage of demand for great chefs, even with all these McDonald's and Olive Gardens dotting the landscape.

1 There are obviously still unknowns in the physical world, but those are irrelevant insofar as we're contrasting engineering and software development within the scope of commercial projects. Research matters are an entirely separate issue in both camps. 2 Another good question to ask: Can you estimate time and materials for projects...and be correct within a 100% margin of error? If so, congratulations, yours might be an engineering discipline. 3 From an informal, but very high quality discussion on this topic in #clojure IRC: