'A train wreck in slow motion" is how Sen. Patrick Leahy described an FBI project that wasted $170 million in an unsuccessful attempt to modernize the bureau's computer software. McDonald's spent the same amount trying to install a real-time network for its outlets before finally admitting failure. The IRS spent even more in a botched attempt to upgrade its software and computer systems. The saddest part about such tales is that they are so common. Failed, late and over-budget software projects are so routine that they aren't news.

A 1995 survey revealed that technology managers rated 31 percent of the software projects undertaken as "total failures." Nine years later, a similar survey showed that total failures had been almost halved to 18 percent -- however the same survey found that only 29 percent of the projects were rated as successes.

In his revealing book "Dreaming in Code," Scott Rosenberg, a co-founder of the online magazine Salon, asks: Why is writing code for a computer so difficult to do right and on time? To answer the question, he hooks up with a software team headed by Silicon Valley legend Mitch Kapor to observe day by day how Kapor and his group develop software and introduce it to the marketplace.

First some background. Software is, at bottom, a string of abstract symbols that tell the computer what to do. One error or bug -- one [ where a ( needs to be -- in a million lines of code can bring an entire system down. Finding those bugs is like trying to locate the bad bulb on a 50-mile-long string of Christmas lights. Consequently, bug fixing is demanding, time-consuming work that can bring a project it to a standstill.

Another common reason for software failures is mushy or changing specifications. To create a usable system, programmers must know from the outset what the final software is supposed to do. Many projects have foundered because customers demanded changes after work was under way or, worse yet, nearing completion.

Choosing Kapor's project enabled Rosenberg to find another reason software projects fall behind schedule and sometimes fail: Without the harsh discipline of the marketplace, programmers tend to spend too much time wallowing in software sinkholes.

Mitch Kapor created one of the first "killer apps" in the 1980s. His spreadsheet program Lotus 1-2-3 was so desirable that customers bought PCs just to get the software. Kapor walked away from the company he founded in 1986 with $100 million in his jeans pocket. He was, he said, "extricating myself from my own success."

Before he left, he began work on a software package called Agenda. Agenda was a PIM, a personal information manager, aimed at keeping busy people organized. Calendars and address books can handle some items, but Kapor wanted a system that would organize the whole barrage of details daily life throws at us.

Agenda was a modest success for Lotus, but not the world-changing software that Kapor hoped it would be. So, after trying other careers from MIT professor to venture capitalist, Kapor returned to Agenda. He wanted to completely rewrite the program, make it more elegant, more useful. He named his software-to-be Chandler, not after the character on "Friends" but for the mystery writer Raymond Chandler.

Kapor assembled an outstanding team of software veterans: computer scientists from Stanford, software designers and programmers from Microsoft, Netscape and Apple. However, two peculiarities would slow down Kapor's dream team.

First, unlike most projects, which begin with a list of desired features, Chandler started as a vision of how the innards of a sophisticated PIM should work. But no one specified exactly what Chandler was supposed to do. What reports were to be issued? What screens would be needed? Second, the company that Kapor founded to develop Chandler was a nonprofit funded almost entirely by Kapor himself. Without investors or customers breathing down their necks, the team never had any hard-and-fast deadlines. What were the penalties for being late and what were the rewards for succeeding?

Next page >