Scott Rosenberg: What Makes Software So Hard

By Edward Cone

Scott Rosenberg: What Makes Software So Hard

Scott Rosenberg has written an important and entertaining book about the way software projects work—or don't. Dreaming In Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software (Crown, 2007), chronicles an open-source effort to build a better personal information manager. Published this month, the book also delves into the history and culture of software development in an attempt to answer a fundamental question: Why is software so hard?

Rosenberg is a founding editor of Salon, the seminal online magazine. While writing the book he had close access to the open-source project, run by the nonprofit Open Source Applications Foundation, and its principals, including Mitch Kapor, the head of OSAF and a software-industry legend. Following in the footsteps of Tracy Kidder's classic book about computer design, The Soul of a New Machine, Rosenberg's book limns the deadline-driven drama of the job and the personalities of the people involved. "I was trying to capture the applied creativity of the process—but I also wanted to tell a good story," he says.

See Also: 5 Software Development Mistakes to Avoid

Kapor's team faced a series of delays and disappointments as they worked to make their product, named Chandler, into something useful and relevant (the project is ongoing). Rosenberg weaves that tale through a much larger saga about software projects and their discontents. Along the way he introduces generations of programming languages and programmers, gives plain-English explanations of coding arcana, and discusses some of the challenges facing developers in the future. Rosenberg spoke recently with Senior Writer Edward Cone about software development. An edited version of his remarks follows.

CIO Insight: Your narrative of the Chandler project weaves through some much larger issues concerning software development. Did you set out to work on such a large canvas?

Rosenberg: I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem.

Clearly, people have succeeded in creating amazing software systems that do work, and do serve people's needs. In many cases they are very large, from the scale of the Internet to the scale of enterprise platforms. So it's doable. But when you pick up the rock and look underneath the surface of every one of those systems, you find stories of delay and difficulty and problems. And it seems the software industry tends not to be introspective. There isn't a lot of history; there isn't a lot of examination of past mistakes. In the physical world, the examination of past mistakes and failures, and the application of principles based on what you have learned, is the classic technique of engineering. There is a thread of that kind of post-project review running through the software field, but overall it doesn't seem to be as closely studied or as widely known as it probably should be.

That's one of the things I tried to do in the book, to unearth that history. It's an exploration of this stuff, most definitely not a set of recommendations. I hope people will come away from the book with a deeper understanding of what happens in the making of a piece of software. I felt that in so many books about making technology, you'd get to the point where people actually start creating the software and then it would be kind of like the sex scene in an old movie: They would just skip it, cut to the next morning, cut to the marketing team getting ready to ship the product. It was like people would avert their eyes from the actual act of making software. Maybe they were afraid readers would be bored, or didn't understand software.

Next page: Improvement Over Time

1 | 2 | 3

Improvement Over Time

CIO Insight: It's depressing that some things never seem to change. Your discussion of Frederick Brooks' 1975 book, The Mythical Man-Month, explains how adding programmers can actually slow down a project. Brooks wrote the book about a huge IBM project gone bad in the 1960s, and yet it's still relevant today.

It is an amazing book. I wish every developer and manager would just sit down and read the first few chapters, and then get back to work. Brooks talks about the original software disaster, with the biggest company of its time betting on a system that would change everything, and the entire software component of the project was a year late. Now, the flip side is that the IBM 360 did become the foundation of corporate computing for the next quarter-century, so the delay in creating its operating system didn't ultimately sink the project, or IBM. But it was an early indication that this whole undertaking of writing large software projects was not a simple matter. And Brooks, an extremely thoughtful man, made it into a great read.

CIO Insight: So where's the improvement over time? Are we just being impatient with a branch of knowledge that is still fairly new? Or is there something inherent to software development that makes it so weird and vexing?

There is an optimistic side to this. I tried to include it in the book by interviewing smart, articulate people with different points of view. You get one perspective that says, hey, we now have a computer on every desk that does things that were unimaginable 20 years ago, and they're all connected in this network that gives us instant answers and instant connections. These are miraculous things. And then you find other people who say, you know what? We're still writing code basically by picking out characters one at a time, we still have programs that are laid low when a single bug creeps in, we still have projects that take ten times longer than they should, we need to rethink everything from the ground up.

I don't have an answer between them. My personal temperament is more towards the optimistic. In the end, what you've got is this industry that's been conditioned by Moore's Law, and by its own fantastic financial success, to assume that the curve is always an upward curve, that everything gets better at an exponential pace. That's the experience of the technology industry. You have that smacking up against the reality of human experience, of creativity, of people working in teams. We have these basic human factors, psychology, the limits of the conceptual capacity of the human brain—things that do not move at an exponential pace. They simply don't. They tend to move linearly, if they are improving at all. People in the technology industry are loath to accept that.

CIO Insight: You talk about "the physics of software." What's that all about, and could it help solve some of these seemingly intractable problems?

It's something we don't have now, that some people are yearning for. The idea is that if we're going to turn the creation of software into a true science, we need to first have principles. We need to know the fundamentals and formulas by which software behaves. What are the laws and principles we can count on in creating it? The problem is that software is not a thing, not a preexisting phenomenon of the universe; it's a product of the human imagination. One of the ideas that stayed with me most strongly is that computer science today accepts the file system as if it were some law of nature, and it's not. Every aspect of software is a human construct. The fact that we have created this pile of abstractions, one on top of another, and that it works most of the time—and produces amazing results—doesn't mean the whole enterprise is a fait accompli and must always be that way. So, personally, I don't believe there is such a thing, or could be such a thing, as the physics of software. But accepting that there couldn't be such a thing has implications that are disturbing to people in the software field.

CIO Insight: Many problems with big projects appear to stem from expectations: The bosses aren't realistic, the programmers overpromise and overreach. It seems like a management problem and a culture problem, not just a software problem.

It's all bound up together. Some of it is rooted in the industry's own history and rhetoric, the mythology of forced marches and death marches and Netscape Time—all these things that suggest that if only you push hard enough, you can do it faster. Managers get into saying: "I want it yesterday, the competition had it yesterday, why can't I have it yesterday?"

Another aspect is that, because software is so insubstantial and a lot of managers who are not programmers themselves don't really understand anything at all about it, they don't have the frame of reference that people have when they're building a bridge or a building. When someone is building a skyscraper, there is a gut-level understanding that you have to dig the hole before you can pour the concrete, then you can put up the beams. There's a natural sequence of dependencies in the physical world that you can imagine very easily, and you can see it's going to take time. Software is basically entirely abstract, except for the screens, and the screens are what business people always end up focusing on. The insubstantiality of the product promotes the idea that, hey, why should it take so long? There's nothing there.

Next page: Shifting Ground

1 | 2 | 3

Shifting Ground

CIO Insight: The Chandler project was launched a few years back, when peer-to-peer networks seemed like a good Web alternative. Reading your book with the benefit of hindsight, I thought, uh-oh, this isn't going to end well: These guys were developing at a point of technological change. But your book says that's always going to be the case.

The ground is always shifting. If you try to follow the changes too closely, you will get into trouble, because things keep changing. If you have a project that takes a few years, and you change every six months, you'll get even deeper into a hole. So if you're going to be ambitious and make a big bet, how do you protect against that? The agile movement says to keep horizons so short that you can always change course, and I think there's a lot of good sense in that. We may know that our ultimate destination is that far-off mountain in the distance, but we're going to keep our heads down and just get through the next hundred yards, see that it's working, and go from there. Break things up into small bites. Any opportunity you have to do that should be seized.

CIO Insight: So you endorse an iterative process of release, test and improve, rather than going for a grand-slam introduction of something new and huge and finished?

If we try to design a system that is fully featured and complex and does everything everyone wants it to, we'll never have any system. You can take that same argument and support it with the history of the Internet itself. There are all kinds of amazing and wonderful schemes people had for hypertext environments, and they were very cool and exciting—and then the one that got widely adopted was this very simple HTTP thing. A lot of people thought HTTP was too crude and didn't fulfill the highest aspirations of the software field. And yet here it is, it worked. People understood it, they used it, and it was widely adopted too quickly for the perfectionists to kill it in the cradle. That's a good thing. The fact that the system in its initial deployment did not live up to ideals of perfection mattered far less than the perfectionists said it would.

CIO Insight: In your book's narrative, Kapor's team seemed to dither a lot. They spent a long time doing stuff that I would think you'd do before you really started. It didn't always seem like an engineering project to me.

It was a matter of ambition colliding with practical reality. One reason the whole engineering approach has proven so difficult for the software field is also one of the unique things about software: Once a particular problem is solved, it's almost infinitely cheaper to use the existing solution. There's no cost to make additional copies, and the cost of creating a new piece of software is huge, so once there's a dominant product, there is no reason to write your own version. Even at Microsoft prices, it's cheaper to use Word than to write a new word processor. The only real reason that a team gets together and writes a new piece of software is that they want to do something new, something another product doesn't do, whether it's to add a feature, or to be compatible in a different way with another system.

In the case of Chandler, they had a number of really interesting and ambitious ideas about flexible data, and having a system with an almost spreadsheet-like fluidity that applied to e-mail and calendars and other kinds of personal information. That was an extremely ambitious goal. What you saw there at the beginning of the project was not dilettantism, but people whose job it is to solve hard problems tackling problems that were just too big.

CIO Insight: So is your book a cautionary tale? Where will Chandler end up?

Nobody knows. There are few different scenarios. One is that it will be a respectable footnote in the history of open-source software, but it won't really take root, because so much has happened in the ensuing years. Another possibility is that some small piece of it will turn out to be incredibly useful in some other product. That's one of the things that happens in open source that make it so flexible and gives it a lot of staying power. They're doing a lot of interesting work at the Open Source Applications Foundation. Maybe an underground development will emerge; maybe Chandler will be like a rock band that was respected even though nobody bought the records, and then, years later, 50 bands show its influence.

Mitch [Kapor] is determined and persistent. They plan to ship the calendar in the first part of 2007. It could be that, in a few years, as everyone's coming down off the Web 2.0 high, people will discover it's an interesting, flexible product. It happens: Look at Firefox. Mozilla was written off, Internet Explorer had won the war, it was done, finished, end of story. And now look at it.

1 | 2 | 3

This article was originally published on 01-05-2007