Waterfall, Agile, and Death Marches

I’ve grown weary of internet debates about software development methodology, argued by people who don’t really understand the vocabulary they use. Allow me to provide a brief lesson on three of the most commonly misused terms.

The Origins of the Epithet “Waterfall”

A little history: Winston Royce is attributed with defining the methodology now called “waterfall” in his 1970 paper, Managing the Development of Large Software Systems although that term never appears in the paper. Royce was offering his opinion on what worked and what didn’t work after nine years of software project experience. While software for internal use could get by with just analysis and coding, paying customers needed a more rigorous approach. However, Royce argued against using a purely sequential process for developing software. In his paper, he critiqued several diagrams, beginning with the one below.

Royce noted that the lack of feedback between the sequential activities was a fatal flaw, but that the feedback flows had to be managed in order to avoid runaway scope. He explored several other diagrams in his paper, which became progressively more complicated until they resembled a spaghetti-and-meatballs dinner for sixty or so. Plainly, feedback was necessary but it was possible to become paralyzed.

The earliest use of waterfall in referring to a software development model may have been in a 1976 paper on software requirements by Bell and Thayer, which referred to Royce’s paper. During the 1980’s, there were several “structured” software project management methodologies marketed by various consulting firms that looked much like the more complex diagrams in Royce’s paper, but no one has actually used any of those methodologies for decades. None of them called their methodologies waterfall, either.

Time Marches On

Technology has changed dramatically in the last half-century. Where the primary cost constraint in 1970 was computing time, it is now knowledge-worker time. Where the primary technical challenges were data storage and code efficiency (does anyone remember dates with two-digit years?), storage and retrieval technology now provides terabytes for the price of a good dinner, free relational databases, structured and ad hoc queries that return results in milliseconds, and a global network with connections to tens of billions of processors. As a result, the nature of software products has changed from calculations of orbital mechanics and industrial controls, written by and for large corporations and government agencies to (overly) personal communications and videos of cats, paid for with marketing messages.

In Royce’s era, programmers and users submitted Hollerith cards and (maybe) a day later scanned the results on 132-column green-bar paper. Today, users have the expectation of point, click, fill-it-in-for-me, and tell-me-what-to-correct. Where once coders spent most of their time writing code to detect and handle logic breaks and invalid inputs, they now focus on the presentation layer. No one optimizes for memory usage, and execution speed only becomes a concern when responses are returned in more than two seconds. There are standards for virtually everything, reducing the effects of obsolescence, and a global communication network that facilitates collaboration across and between continents and cultures.

Software development today bears as little resemblance to 1970 as modern agriculture does to Mesolithic-era farming in the Fertile Crescent twelve millennia ago. But there are still farmers and there are still programmers, and the micro-economics of production still apply. And the same seven activities identified in Royce’s diagram are still needed, even if we organize and execute them differently.

Agile is Not a Performance Characteristic

Most modern software development organizations will tell you that they use agile methods. The term refers to the Agile Manifesto, composed by seventeen thought leaders at a conference on lightweight software development methods in 2001. While there are several software development methodologies marketed as “agile,” in most organizations “agile” is a synonym for Scrum. Other common “agile” methods include Kanban, XP, and DSDM. Each of them addresses the seven basic activities, although the level of detail varies widely.

Note that “agile” in this context is a label, rather than a performance characteristic. While many organizations use these methods (or combinations of them) effectively to produce reliable, maintainable software that creates real value, many others flounder. An entire industry of consultants and trainers offer their services to those who want to succeed with agile methods. Many of these experts (and their students) disparage what they assert are “non-agile” approaches with the generic epithet “waterfall.” That doesn’t make them so.

Complexity Requires an Even More Rigorous Approach

As noted, the Agile Manifesto arose from a conference on lightweight methods. While that certainly describes much of modern software development, many software projects require compliance with a variety of contractual, legal, and regulatory requirements, in addition to functional and technical requirements. These complex projects tend to include a lot of requirements elicitation, analysis, and design tasks that must be finalized by people who are not coders before decision makers will give their approval to proceed with actual coding. Because of these controls and process requirements, numerous parallel workflows by people who are not coders, critical decisions that often have nothing to do with code or coders, and the higher cost and longer duration of complex projects, they are typically managed using predictive methods—in other words, estimating task duration in advance, in order to schedule activities so that knowledge workers who are not coders can manage their workflows.

Complex projects include most government-funded projects, most projects for heavily-regulated businesses, and nearly all products in the medical devices or aerospace industries. Agile methods are used for some project activities, in parallel or in series with other activities using other approaches, when it makes sense to do so. People who don’t understand (and don’t care to understand) anything about these external controls and processes often dismiss them as command and control or waterfall or some other epithet. The rest of us understand that you have to skin a rabbit before putting it over the fire—see “Payne, Podrick” for an illustrative misuse case.

Death Marches Aren’t About Methodology

A project should be approved based on a specific business case, with parameters such as a scope, delivery schedule, budget, a staffing model, and performance and quality goals that are realistic. Ed Yourdon defined a death march project as, “One whose ‘project parameters’ exceed the norm by at least 50 percent.”

Acme Tornado KitThe business case should be updated whenever the business environment changes or assumptions prove to be invalid. An oversight committee should meet periodically to assess progress and risks, along with the probability of achieving the goals in the business case. This committee should be empowered to adjust any of the aforementioned parameters, and even to cancel the project outright if it seems necessary. Thus, death marches are not a failure of methodology but a failure of project governance.

Although there are still many organizations that will continue death march projects even in the face of evidence that the project will fail to achieve the intended business goals, few of them are profitable and essentially all of them have other bad management habits. In the words of Eric Bogle, “… year after year, their numbers get fewer. Someday no one will march there at all.”

Decision-Making Under the Influence: SME, HiPPO, and BOGSAT

The most significant driver of cost and schedule variance in the execution of any project is indecision. While most projects can absorb a few bad decisions or even course-correct without a hitch, delaying a decision almost invariably creates damage. Agile practitioners will typically defer decisions until required to move forward so that the Decider has as much information as possible, but a lack of information isn’t always—or even usually—the problem. Sometimes the Decider just doesn’t feel empowered, and sometimes they are having trouble reconciling input from different Influencers. And there are several common types of Influencers.

SME: The Subject Matter Expert

The SME understands the business process, constraints, data records, information security concerns, and so on. In a lot of cases, the SME is the person who owns—or will own—the results of the decision, but doesn’t have the authority to make the decision. So the Decider—rightly—looks to the SME for input, trusting that the expert opinion will not be influenced by the impact of the decision. Usually, it isn’t. Usually.

The wise Decider will usually align with the SME, unless something else comes up. Usually, this takes the form of another Influencer.

HiPPO: Highest Paid Person’s Opinion

Inviting a very senior person to weigh in on a decision they aren’t being asked to make is like prospecting for uncertainty and doubt. Numerous studies have verified the existence of the HiPPO effect—junior team members defer to the expressed opinions of the Highest Paid Person, because they assume that they are paid so much due to their greater knowledge. And the Highest Paid Person may actually have greater knowledge in some area—but that doesn’t mean their Kung Fu is the best for this particular decision.

An even worse situation can occur when the Highest Paid Person asks a clarifying question, which nearly always raises doubt in the minds of the Decider and SME. I once saw a SVP of Human Resources stop a presentation cold by asking whether a bullet point referred to something that was done, or that needed to be done. I had to help the stunned Decider find her car.

“It’s the Heisenberg Principle…me asking the question changes the answer.” – Barack Obama

BOGSAT: Bunch of Guys Sitting Around a Table

It’s good to network, but parents worry about peer pressure for a reason. I once worked for a guy who used the term “strap-hangers” for those he perceived were just along for the ride. As he put it: if they didn’t have any skin in the game, their opinions shouldn’t matter. I used to think that was a bit extreme, but as I’ve come to understand the dynamics of making choices under conditions of uncertainty, it’s become plain that not all points of view are valuable. I had a client defer a decision until she could reach out to confirm whether the passing comment from a friend in another company was relevant. It wasn’t.

Uncertainty, Assumptions, Influence, and Decisiveness

Uncertainty is a given—we are rarely asked to make decisions with all the facts. Usually, at least some things are unknown or even unknowable. So we have to make assumptions in order to close the gaps. Those assumptions represent risks, which have to be managed. We seek out the opinions and counsel of those we trust and respect in an effort to assess the risks and proceed with a decision, and in many cases that additional input makes the issue and choices more clear. But reconciling different opinions and different points of view should not take precedence over making a timely decision.

Managing Transitions Between Outsourcing Vendors

With apologies to Sir Walter Scott: Oh, what a tangled contract we write, when first we practice to outsource. Having managed outsourcing projects on behalf of both the customer and the third-party administrator, and managed transitions from one outsourcing firm to another on behalf of several clients, I have a lot of anecdotal evidence that outsourcing generally works best on a spreadsheet—in practice, results tend to be rather variable. But because most business decisions are driven by spreadsheets, businesses keep outsourcing.

That’s not to say that migrating to a better outsourcing relationship isn’t a net positive—only that changing who does what across organizational boundaries is stressful for all concerned, and that increased stress level and instability tends to last longer than most decision-makers expect. And when transitioning from one third party to another, you get something like a long, drawn-out divorce where the spurned spouse is required to help tailor her wardrobe to the mistress, who arrives with an extensive list of demands.

Transition Projects and VUCA

VUCA is an acronym originally coined by the US Army War College to describe a state of increased volatility, uncertainty, complexity, and ambiguity. Transition projects have it in spades:

  • Volatility: Disrupting the current state produces lots of process change points, and most are not under the control of the client.
  • Uncertainty: There are a lot of opportunities for all parties to be surprised, from the customer decision-makers to the employees of the departing administrator. Do not underestimate the emotional impact of being asked to help a stranger take over your job—some folks resign the same day they get the news.
  • Complexity: Since the only contracts are between the customer and each of the two rival firms, the customer must act as intermediary in all things. Issue management quickly becomes a significant challenge, since each party wants to avoid blame. Cause and effect can become blurred, and even disconnected.
  • Ambiguity: There is so much potential for miscommunication and differences in terminology that even simple discussions require extreme care. Insist on a common set of terms and acronyms and what they mean, or you’ll be discovering gaps nearly every day.

Over the years, I’ve adopted a few techniques for facilitating projects built on a team of rivals, and I’ll share a few of them here.

The Central Clearing House

While every firm has a preferred collaboration and file sharing tool, it is vital for the common client to provide the tools for secure file sharing and provide equal access to both incumbent and replacement. The client must manage the content and organization, no matter how eloquently the replacement firm pitches their approach. All fingers should point to the client, or the potential for blamestorming becomes unmanageable.

The Cheshire Cat

Since the rivals have no contractual relationship, the client (or their representative) must chair every joint meeting. Over time, the individuals on each side develop relationships with their counterparts, and collaboration becomes natural. As this happens, the potential for emotional conflict gradually diminishes, and the client can gradually be less visible, although still present and ready to step in.

The Captain’s Table

Dave GordonOnce the transition is under way, the project leaders for all three organizations need to collaborate on resolving issues, following up on requests, and otherwise executing on their joint plan. This joint meeting should be scheduled as often as necessary, whether weekly or daily, and chaired by the client’s project manager. Avoid having more than one person from each organization, so internal politics don’t bleed over into the project.

An experienced project manager is used to leveraging influence in the absence of direct authority. What most of us are not used to is influencing people who are about to lose their jobs, when we want them to work with their replacements. It isn’t just about the need for emotional intelligence, but the need to preserve the dignity of the employees of the departing incumbent. I’ve seen some folks take the opportunity to move on to much better circumstances, and I’ve seen some fall into deep depression. You might not get the chance to influence that outcome, but be sensitive to the fact that these people are not just numbers on a spreadsheet.