Why do software projects fail? Computer scientists and software practitioners alike have wrestled with this question ever since software has entered the corporate space. There are supposedly many factors leading to the “failure” of a software project (however failure is defined) but the one constant I’ve witnessed in many projects I was on is that Conway’s law is always true. No single corporate entity has ever been able to detach its software engineering or software operations teams from the rest of the organization in a productive way.
Companies tend to apply the same processes and principles they already have in place to their software creation and maintenance endeavors. A company that is very good at producing cars will take their successful car-making processes and force them on their software engineering teams. The same goes for any other manufacturer, construction, utility, healthcare, education, trading or finance company. And it Does. Not. Work.
You cannot successfully build software the same way you build cars, maintain power grids or produce industrial-grade machines. All the metaphors out there like “building software is like building a house” are wrong! There is no single metaphor for the software development process. You can’t plan it 3 years ahead. You can’t make up a blueprint and go building. You’re not building a single thing. You’re not even building something that will not change in a significant way 1 or 2 years from now. Software development needs to be rapid, fast-paced with short feedback cycles to be successful. Bill Gates and Paul Allen built the first BASIC interpreter for the Altair within weeks! Let that be your benchmark and not how you build your great cars or machines or space rockets.
That’s why heavyweight processes have been largely replaced by lightweight methodologies for software engineering in the 1990s when XP, Scrum, FDD and later the Agile Manifesto have been introduced. This is now more than 20 years ago! So why do many software projects still fail? The methodologies are not to blame. It’s the organizational structures that they are pressured into. For an agile software development process to be applied successfully it needs to be implemented in a way that allows it to breathe, to blossom. Software engineers need to be given creative space. Building software never is like building a physical machine from a blueprint. There are no blueprints for software. Each piece of software is different from anything that has been built before.
That’s what agile is all about. You can’t lock developers into a corporate chamber and expect anything meaningful to come out of there. And even in cases where a company tries to detach their software development from their other business as much as possible, processes and culture will not stop leaking into it. Volkswagen’s Cariad is the most recent example of how this doesn’t work.
So if Conway’s law does proof to be true in almost every case, what can companies do about that? They obviously can’t (and shouldn’t) change the processes they’ve successfully applied to their other businesses. Should they stop producing, maintaining and operating software at all and leave it to those who are specialized software practitioners? Should they stop trying to produce any software and always opt for buying existing software products?
People who know me also know I strongly believe that every big company out there needs to admit to the fact that software is and will continue to be a significant part of your business. Software has long passed its status as a mere utility for the “actual” business. It IS your business. Look at cars again: Those companies that have made software a central and integral part of their manufacturing process are successful (Tesla, Xiaomi, BYD and others). Those that have not, aren’t (any traditional car maker). In space technology you can even see how the recognition of Conway’s law can lead to the reverse effect, i.e. agile methodologies being applied to the manufacturing process by SpaceX, in effect making them the leading space technology company in the world.
Marc Andreessen coined the phrase software is eating the world in 2011. Now, 13 years later, many big corporations out there have still not grasped the extent to which this phrase affects them and anyone else, too. Stop treating software as a utility. Start making it the defining part of your business, adding value to your baseline products, be it cars, rockets, supermarkets or utilities. This is the foundation upon which companies will build a successful future for their businesses.