Waterfall, Agile, and Death Marches

I’ve grown weary of internet debates about software development methodology, argued by people who don’t really understand the vocabulary they use. Allow me to provide a brief lesson on three of the most commonly misused terms.

The Origins of the Epithet “Waterfall”

A little history: Winston Royce is attributed with defining the methodology now called “waterfall” in his 1970 paper, Managing the Development of Large Software Systems although that term never appears in the paper. Royce was offering his opinion on what worked and what didn’t work after nine years of software project experience. While software for internal use could get by with just analysis and coding, paying customers needed a more rigorous approach. However, Royce argued against using a purely sequential process for developing software. In his paper, he critiqued several diagrams, beginning with the one below.

Royce noted that the lack of feedback between the sequential activities was a fatal flaw, but that the feedback flows had to be managed in order to avoid runaway scope. He explored several other diagrams in his paper, which became progressively more complicated until they resembled a spaghetti-and-meatballs dinner for sixty or so. Plainly, feedback was necessary but it was possible to become paralyzed.

The earliest use of waterfall in referring to a software development model may have been in a 1976 paper on software requirements by Bell and Thayer, which referred to Royce’s paper. During the 1980’s, there were several “structured” software project management methodologies marketed by various consulting firms that looked much like the more complex diagrams in Royce’s paper, but no one has actually used any of those methodologies for decades. None of them called their methodologies waterfall, either.

Time Marches On

Technology has changed dramatically in the last half-century. Where the primary cost constraint in 1970 was computing time, it is now knowledge-worker time. Where the primary technical challenges were data storage and code efficiency (does anyone remember dates with two-digit years?), storage and retrieval technology now provides terabytes for the price of a good dinner, free relational databases, structured and ad hoc queries that return results in milliseconds, and a global network with connections to tens of billions of processors. As a result, the nature of software products has changed from calculations of orbital mechanics and industrial controls, written by and for large corporations and government agencies to (overly) personal communications and videos of cats, paid for with marketing messages.

In Royce’s era, programmers and users submitted Hollerith cards and (maybe) a day later scanned the results on 132-column green-bar paper. Today, users have the expectation of point, click, fill-it-in-for-me, and tell-me-what-to-correct. Where once coders spent most of their time writing code to detect and handle logic breaks and invalid inputs, they now focus on the presentation layer. No one optimizes for memory usage, and execution speed only becomes a concern when responses are returned in more than two seconds. There are standards for virtually everything, reducing the effects of obsolescence, and a global communication network that facilitates collaboration across and between continents and cultures.

Software development today bears as little resemblance to 1970 as modern agriculture does to Mesolithic-era farming in the Fertile Crescent twelve millennia ago. But there are still farmers and there are still programmers, and the micro-economics of production still apply. And the same seven activities identified in Royce’s diagram are still needed, even if we organize and execute them differently.

Agile is Not a Performance Characteristic

Most modern software development organizations will tell you that they use agile methods. The term refers to the Agile Manifesto, composed by seventeen thought leaders at a conference on lightweight software development methods in 2001. While there are several software development methodologies marketed as “agile,” in most organizations “agile” is a synonym for Scrum. Other common “agile” methods include Kanban, XP, and DSDM. Each of them addresses the seven basic activities, although the level of detail varies widely.

Note that “agile” in this context is a label, rather than a performance characteristic. While many organizations use these methods (or combinations of them) effectively to produce reliable, maintainable software that creates real value, many others flounder. An entire industry of consultants and trainers offer their services to those who want to succeed with agile methods. Many of these experts (and their students) disparage what they assert are “non-agile” approaches with the generic epithet “waterfall.” That doesn’t make them so.

Complexity Requires an Even More Rigorous Approach

As noted, the Agile Manifesto arose from a conference on lightweight methods. While that certainly describes much of modern software development, many software projects require compliance with a variety of contractual, legal, and regulatory requirements, in addition to functional and technical requirements. These complex projects tend to include a lot of requirements elicitation, analysis, and design tasks that must be finalized by people who are not coders before decision makers will give their approval to proceed with actual coding. Because of these controls and process requirements, numerous parallel workflows by people who are not coders, critical decisions that often have nothing to do with code or coders, and the higher cost and longer duration of complex projects, they are typically managed using predictive methods—in other words, estimating task duration in advance, in order to schedule activities so that knowledge workers who are not coders can manage their workflows.

Complex projects include most government-funded projects, most projects for heavily-regulated businesses, and nearly all products in the medical devices or aerospace industries. Agile methods are used for some project activities, in parallel or in series with other activities using other approaches, when it makes sense to do so. People who don’t understand (and don’t care to understand) anything about these external controls and processes often dismiss them as command and control or waterfall or some other epithet. The rest of us understand that you have to skin a rabbit before putting it over the fire—see “Payne, Podrick” for an illustrative misuse case.

Death Marches Aren’t About Methodology

A project should be approved based on a specific business case, with parameters such as a scope, delivery schedule, budget, a staffing model, and performance and quality goals that are realistic. Ed Yourdon defined a death march project as, “One whose ‘project parameters’ exceed the norm by at least 50 percent.”

Acme Tornado KitThe business case should be updated whenever the business environment changes or assumptions prove to be invalid. An oversight committee should meet periodically to assess progress and risks, along with the probability of achieving the goals in the business case. This committee should be empowered to adjust any of the aforementioned parameters, and even to cancel the project outright if it seems necessary. Thus, death marches are not a failure of methodology but a failure of project governance.

Although there are still many organizations that will continue death march projects even in the face of evidence that the project will fail to achieve the intended business goals, few of them are profitable and essentially all of them have other bad management habits. In the words of Eric Bogle, “… year after year, their numbers get fewer. Someday no one will march there at all.”

Managing Globally Distributed Project Teams

I started managing projects that included team members or customers outside the US in the mid-90’s. In the beginning, it was one other country. Then two, and so on. As I progressed in my career, working with globally distributed project teams became my norm. A typical project would include people spread across five to thirty countries, three to five continents, and from three to seven time zones. As you would expect, it’s very different from managing a few folks clustered together in a cube farm. Preparations must begin before the first team meeting.

Working Calendars

It is important to be cognizant of the non-working days for the people in your team. Set up the holidays for each country in your project planning system—here is a list of commonly observed national and religious holidays in several countries for 2020, and here are instructions for updating the working calendar in Microsoft Project. In addition, ask your team members to record their planned vacation dates in a shared location—I usually just use an Excel spreadsheet, to keep the technical overhead down. Also, find a culturally sensitive way to inquire about maternity leave!

Time Zones

One of the biggest problems with working across oceans is the impact of time zones and the international date line on available windows for team meetings. Even if the organization adjusts working hours to get some alignment, it can be a burden for those who are always either getting up early or staying up late. Try to schedule meetings in a way that shares the burden.

Also recognize that not everyone observes daylight savings time, and those that do, don’t all change their clocks on the same day—Europe and North America are a week apart. And the Northern and Southern hemispheres are on completely different schedules. Here is an excellent resource for finding the current time and time zones of most of the large cities in the world, and here is their daylight savings time page.

Visibility into Workload Conflicts

Most globally distributed, cross-functional project teams include some number of people who have additional work responsibilities. The project will always be in competition with that other work, and you won’t necessarily know when priorities change. To avoid delays, maintain contact with your team member’s manager, or a proxy—someone who can act as a remote source of information and as a person of influence, should that be necessary. Not all cultures will openly discuss doubts and conflicts, especially with a distant colleague. It is vital to have a way to identify and resolve conflicts, and getting a periodic pulse check from someone on site can make a huge difference.

A Common, Bland English

For most global organizations, English is the common language. That doesn’t mean everyone speaks or understands it fluently, and it certainly doesn’t mean that everyone is familiar with all the local idioms, slang, and cultural references. When on the phone, speak deliberately (but not too slowly), as it can be difficult to parse out similar-sounding words. Work to avoid misunderstanding by keeping your spoken communications as jargon-free and non-colloquial as possible. And try to take the edge off your regional accent – I work at sounding as much as possible like a “generic American,” without the drawl. I tried to raise this with a colleague from Houston a few years ago, who replied, “What accent?” Note that British, Australian, Canadian, Scottish, Irish, and New Zealand accents and idioms are just as real and just as hard on the ears as Indian or Texas English. Speak to be understood by your audience.

There are many resources available online that can help you build your Cultural Intelligence, and even if you aren’t managing global teams right now, you almost certainly will be before long. I learn more about how cultural differences impact work and communication with every project, but I generally find that if I assume people are doing their best until I have reason to doubt it, my life is a lot happier.

Status Reports: News and Information

Dennis the NewsboyI’m sure you’ve heard the old saw that dog-bites-man is not news, but man-bites-dog is news. While the news can be a source of entertainment, especially when some man bites a dog, such incidents are anecdotal at best—a statistic with a sample size of one. We should also consider that tabulated instances of dog-bites-man and related details can be statistically analyzed based on characteristics like breed, time of year, distance from the animal’s home, behavior of the man immediately prior to the incident, and so on. We call this information.

I don’t mean to imply that news has less merit than information. I simply mean that we should distinguish between the two types of content. And you should include both types in project status reports. Effective decision makers stay current on the news but base their decisions on information.

Project News

WhiteboardProjects are like novels, described by my high school English teacher, Ray Rockwell, as “One damned thing after another.” That said, some activities are recurring, or spread over a long period of time, while other activities and events are one-off, or are notable as start or end-points of longer duration events. This is where we separate the news items from the data points.

Project reporting periods vary, based on the projects and the needs of the stakeholders, but let’s assume you are reporting each week. Some of the news items you might want to cover in your status report include:

• Milestones achieved or missed
• Delayed events or actions that were finally completed
• Noteworthy achievements by team members or the team overall (kudos)
• Noteworthy misses or failures by the team, and what was learned

I could go on, but you get the idea. These are events, both planned and unexpected, at a point in time. Timing is important, as old news is no news, and while a few news items are worth interrupting scheduled programming, most are not.

Project Information

ExcelGanttChart4Projects tend to generate a lot of data that can and should be tracked over time. For example: it can be helpful to understand how risk exposure has evolved over the course of a project. If the project team is continually updating the risk register and the qualitative and quantitative assessment scores, and you have an agreed way to aggregate all of those risk scores, a graph of the cumulative risk exposure can show the trend. If you score open issues, that can be a second line on the chart. Add those to a line with a burn-down chart of planned work and insert major milestones, and you have a picture that tells a compelling story.

During the test stages of a project, graphs can your stakeholders tell a lot about your increasing confidence in the quality of the product. I saw another team graph knowledge capture and transfer as a burn-down chart, to the delight of the folks who would assume support responsibilities after the move to production. Think about your stakeholders and the sort of information (as opposed to news) that they will focus on. Not everyone will care about labor utilization trends or cumulative spend but if your audience wants it, track it. Chance favors the prepared, and management favors the proactive.

Project Scorecards

Most of us have adopted a Red-Amber-Green score-at-a-glance for quick and easy summary of the detailed message. The key here is to make these broad assessments the result of an actual score, decided in advance of the project. I’ve written before about making these indicators rigorous, but let’s be clear about their value to the consumers of your reports: they should call attention to something to be investigated elsewhere in your report. If you have one overall RAG stoplight and six detail-level stoplights, ensure your project news and information is organized in a way that lets them find the details quickly. Don’t force your CFO to read the whole status report in order to find the sentence that says consultant labor spend is running 10% ahead of plan. Highlight the bad news!

Communication Leads to Influence

We prepare project status reports because we want to communicate with our stakeholders, make them aware of progress, road blocks, and speed bumps, and influence them to act on the things that require their action. A good project status report doesn’t “spook the herd,” but it does let management know what to expect. And the key to becoming a positive influence is by managing expectations.