Waterfall, Agile, and Death Marches

I’ve grown weary of internet debates about software development methodology, argued by people who don’t really understand the vocabulary they use. Allow me to provide a brief lesson on three of the most commonly misused terms.

The Origins of the Epithet “Waterfall”

A little history: Winston Royce is attributed with defining the methodology now called “waterfall” in his 1970 paper, Managing the Development of Large Software Systems although that term never appears in the paper. Royce was offering his opinion on what worked and what didn’t work after nine years of software project experience. While software for internal use could get by with just analysis and coding, paying customers needed a more rigorous approach. However, Royce argued against using a purely sequential process for developing software. In his paper, he critiqued several diagrams, beginning with the one below.

Royce noted that the lack of feedback between the sequential activities was a fatal flaw, but that the feedback flows had to be managed in order to avoid runaway scope. He explored several other diagrams in his paper, which became progressively more complicated until they resembled a spaghetti-and-meatballs dinner for sixty or so. Plainly, feedback was necessary but it was possible to become paralyzed.

The earliest use of waterfall in referring to a software development model may have been in a 1976 paper on software requirements by Bell and Thayer, which referred to Royce’s paper. During the 1980’s, there were several “structured” software project management methodologies marketed by various consulting firms that looked much like the more complex diagrams in Royce’s paper, but no one has actually used any of those methodologies for decades. None of them called their methodologies waterfall, either.

Time Marches On

Technology has changed dramatically in the last half-century. Where the primary cost constraint in 1970 was computing time, it is now knowledge-worker time. Where the primary technical challenges were data storage and code efficiency (does anyone remember dates with two-digit years?), storage and retrieval technology now provides terabytes for the price of a good dinner, free relational databases, structured and ad hoc queries that return results in milliseconds, and a global network with connections to tens of billions of processors. As a result, the nature of software products has changed from calculations of orbital mechanics and industrial controls, written by and for large corporations and government agencies to (overly) personal communications and videos of cats, paid for with marketing messages.

In Royce’s era, programmers and users submitted Hollerith cards and (maybe) a day later scanned the results on 132-column green-bar paper. Today, users have the expectation of point, click, fill-it-in-for-me, and tell-me-what-to-correct. Where once coders spent most of their time writing code to detect and handle logic breaks and invalid inputs, they now focus on the presentation layer. No one optimizes for memory usage, and execution speed only becomes a concern when responses are returned in more than two seconds. There are standards for virtually everything, reducing the effects of obsolescence, and a global communication network that facilitates collaboration across and between continents and cultures.

Software development today bears as little resemblance to 1970 as modern agriculture does to Mesolithic-era farming in the Fertile Crescent twelve millennia ago. But there are still farmers and there are still programmers, and the micro-economics of production still apply. And the same seven activities identified in Royce’s diagram are still needed, even if we organize and execute them differently.

Agile is Not a Performance Characteristic

Most modern software development organizations will tell you that they use agile methods. The term refers to the Agile Manifesto, composed by seventeen thought leaders at a conference on lightweight software development methods in 2001. While there are several software development methodologies marketed as “agile,” in most organizations “agile” is a synonym for Scrum. Other common “agile” methods include Kanban, XP, and DSDM. Each of them addresses the seven basic activities, although the level of detail varies widely.

Note that “agile” in this context is a label, rather than a performance characteristic. While many organizations use these methods (or combinations of them) effectively to produce reliable, maintainable software that creates real value, many others flounder. An entire industry of consultants and trainers offer their services to those who want to succeed with agile methods. Many of these experts (and their students) disparage what they assert are “non-agile” approaches with the generic epithet “waterfall.” That doesn’t make them so.

Complexity Requires an Even More Rigorous Approach

As noted, the Agile Manifesto arose from a conference on lightweight methods. While that certainly describes much of modern software development, many software projects require compliance with a variety of contractual, legal, and regulatory requirements, in addition to functional and technical requirements. These complex projects tend to include a lot of requirements elicitation, analysis, and design tasks that must be finalized by people who are not coders before decision makers will give their approval to proceed with actual coding. Because of these controls and process requirements, numerous parallel workflows by people who are not coders, critical decisions that often have nothing to do with code or coders, and the higher cost and longer duration of complex projects, they are typically managed using predictive methods—in other words, estimating task duration in advance, in order to schedule activities so that knowledge workers who are not coders can manage their workflows.

Complex projects include most government-funded projects, most projects for heavily-regulated businesses, and nearly all products in the medical devices or aerospace industries. Agile methods are used for some project activities, in parallel or in series with other activities using other approaches, when it makes sense to do so. People who don’t understand (and don’t care to understand) anything about these external controls and processes often dismiss them as command and control or waterfall or some other epithet. The rest of us understand that you have to skin a rabbit before putting it over the fire—see “Payne, Podrick” for an illustrative misuse case.

Death Marches Aren’t About Methodology

A project should be approved based on a specific business case, with parameters such as a scope, delivery schedule, budget, a staffing model, and performance and quality goals that are realistic. Ed Yourdon defined a death march project as, “One whose ‘project parameters’ exceed the norm by at least 50 percent.”

Acme Tornado KitThe business case should be updated whenever the business environment changes or assumptions prove to be invalid. An oversight committee should meet periodically to assess progress and risks, along with the probability of achieving the goals in the business case. This committee should be empowered to adjust any of the aforementioned parameters, and even to cancel the project outright if it seems necessary. Thus, death marches are not a failure of methodology but a failure of project governance.

Although there are still many organizations that will continue death march projects even in the face of evidence that the project will fail to achieve the intended business goals, few of them are profitable and essentially all of them have other bad management habits. In the words of Eric Bogle, “… year after year, their numbers get fewer. Someday no one will march there at all.”

New PM Articles for the Week of December 16 – 22

New project management articles published on the web during the week of December 16 – 22. And this week’s video: Mike Clayton updates a holiday favorite with the twelve project management days of Christmas. No singing and no birds in fruit trees; just short reminders of key themes in project management. Six minutes, safe for work.

Ethics, Business Acumen and Strategy

  • Bas Kohnke points out four trends in the gig economy, people enablement, chatbots, and AI that will further change the workplace in 2020. 4 minutes to read.
  • Rashmi Sharma describes seven tech trends about to transform business in the coming decade. 6 minutes to read.
  • Kimberley Botwright reports on the furious growth of cross-border data transactions, and the growing body of regulations restricting it. Compliance will be one of the challenges of the 20’s. 4 minutes to read.

Managing Projects

  • Harold Kerzner identifies coming changes to the practice of project management. This will be the decade that project management becomes a strategic skill set. 6 minutes to read.
  • John Goodpasture shares an outline and general guidelines for writing an RFP. 4 minutes to read.
  • Brad Egeland describes what a project manager should be able to expect from the customer, the project team, and from senior management. 9 minutes to read all three.
  • Stephen Biddle tells us how to write a report. And just as important: how to not write a report. 3 minutes to read.
  • Jason Westland gives us an overview of IT governance, including definitions, frameworks, and planning. 6 minutes to read.
  • David Binny looks at the intricacies of the making a business case for a move to S/4HANA, before SAP ends support for ECC. Alternatives abound! 6 minutes to read.

Managing Software Development

  • Stefan Wolpers curates his weekly list of Agile content, from innovation failures to better standups to actively doing nothing. 7 outbound links, 3 minutes to read.
  • Johanna Rothman expands a recent conversation with a project manager to illustrate the power of role titles. 6 minutes to read.
  • Louis-Philippe Carignan observes the impact of DevOps on Agile practices; specifically, the growing value of Kanban metrics as a substitute for story points. 6 minutes to read.
  • Bob Reselman outlines a strategy for testing in an event-driven application architecture. 7 minutes to read.
  • Chris Fox rants, “Testing is essential, but it’s secondary. Sorry.” 6 minutes to read.
  • Nels Hoenig shares an anecdote that underlines the value of validating data before proceeding to analysis. 5 minutes to read.

Applied Leadership

  • Art Petty analyzes the leader’s role in executing on strategy. 5 minutes to read.
  • Amy Jen Su examines the components of trust between manager and team to help us understand if we really trust each other. 10 minutes to read.
  • John Millen uses Jeff Bezos, Warren Buffet, and Elon Musk as examples of great communicators. 5 minutes to read.

Cybersecurity and Data Protection

  • Curtis Franklin explains how to approach API security. 5 minutes to read.
  • Edlyn Levine and Algirde Pipikaite assess the potential of cyberattacks that directly target the hardware in our infrastructure. 4 minutes to read.
  • Colin Jones updates us on developments in mobile two-factor authentication and security keys, in the wake of SIM-swapping attacks. 6 minutes to read.
  • Steven Melendez reports that ransomware attackers are now publishing stolen data from victims who don’t pay. Yes, this means they are leveraging GDPR fines. 2 minutes to read.

Pot Pourri

  • Leigh Espy primes the pump with 35 conversation-starter questions for social and networking events. Not just for the holidays! 5 minutes to read.
  • Harry McCracken extols the iPad as his personal gadget of the decade. 3 minutes to read.

Enjoy!

New PM Articles for the Week of November 18 – 24

New project management articles published on the web during the week of November 18 – 24. And this week’s video: a short video explaining the notion of thinking fast and slow, as described by Daniel Kahneman in his book of the same name. Highly recommended, if you haven’t read it. 5 minutes, safe for work.

Ethics, Business Acumen and Strategy

  • Greg Satell warns against relying on surveys and statistics to develop business strategy. “Assume that [your strategy] is wrong in at least some aspects.” 5 minutes to read.
  • Alexandra Ossola notes the various biases and intellectual limitations that make humans so bad at predicting the future and then briefly suggests ways around them. 5 minutes to read.
  • Eddie Yoon and colleagues explain the difference between a First Mover and a Category Creator. It’s all about flywheels. 5 minutes to read.

Managing Projects

  • Elizabeth Harrin arms us with a meeting agenda template, ten tips, and two infographics to help prepare for that next project meeting. 6 minutes to read.
  • Mike Clayton talks us into pursuing certification of our project management skills and experience and gives us some tips on how to proceed. Video, 10 minutes, safe for work.
  • Hannah Cohen gives us an advanced tutorial on creating a project timeline. 8 minutes to read.
  • Geert-Hinke Addink describes the pre-mortem and the role it can play in identifying many of the risks in a proposal or new initiative. 2 minutes to read.
  • Dmitriy Nizhebetskiy compares the project manager career path with the Scrum master career path. 10 minutes to read, plus two short videos, 9 and 7 minutes, safe for work.
  • Jaclyn Westlake shows how to write a great project manager resume, including an example. 8 minutes to read.

Managing Software Development

  • Stefan Wolpers curates his weekly list of Agile content, from tough conversations and conflict to “stances” of the product owner role to the benefits of transparency. 7 outbound links, 3 minutes to read.
  • Johannes Geske explains whether you should attend the Professional Scrum Foundations course or the Professional Scrum Master course.
  • Angie Jones analyzed which programming languages are most commonly used for UI test automation. 3 minutes to read.
  • John Yorke shares his slide deck from the Agile Product Ownership meetup for a discussion of the discovery phase of a product. 13 slides, including intro and outro.
  • Vadim lists ten things to keep in mind when developing and maintaining a product road map. 6 minutes to read.
  • Salomé Mortazavi describes outcome-based product roadmaps, driven by a vision and a strategy for achieving it. 5 minutes to read.

Applied Leadership

  • Quint Studer suggests a number of specific actions to take that will make it OK for your team to speak the truth to power and take risks of honesty. 7 minutes to read.
  • Art Petty extols the combination of emotional intelligence and situational awareness and offers some thoughts on strengthening your SA skills. 4 minutes to read.
  • Johanna Rothman notes that a major component of culture is what behaviors we choose to reward and punish. 3 minutes to read.

Cybersecurity and Data Protection

  • Richard LeCount gives us the whys and wherefores of an internal cyber security audit. 9minutes to read.
  • Adrien Ogée and Parker Crockford update us on the growing trend of using authentication alternatives to passwords. 4 minutes to read.
  • Alex Humphrey encourages embedding better security practices throughout the DevOps cycle. 4 minutes to read.
  • Brian Wallace shows the history of spam and phishing via timeline. This goes back further than you think. 7 minutes to read.

Pot Pourri

  • Leigh Espy describes several activities that will boost your creative thinking skills. 11 minutes to read.
  • Deepak Vohra reminds us that raw data must be rationalized before it can be analyzed. 3 minutes to read.
  • Rob Marvin points out some of the tech innovations of the closing decade that were so successful, we can’t remember life before them. 13 minutes to read.

Enjoy!