When to use agile methodologies

Among the strangest claims I’ve heard about agile, one is that agile methodologies are no good for complex projects.

The odd thing about that claim is that agile methodologies seem to be, according to the experience of many users, the ones that deal best with complexity.

There are two possible reasons that could explain that kind of claim. The first is that people tend to confuse complex with complicated. The second is the inability to deal with what is complex.

Complex versus complicated

We’ve already talked about this topic several times on this blog. In the article “Lo simple, lo complejo, lo complicado” (“The simple, the complex, the complicated”) I say the following:

When we say something is complex we mean to suggest that it is the inevitable result of combining the elements, and that this doesn’t imply a flaw or a failure, as when we say “a recipe is complex”.

On the other hand, we apply complicated to whatever is very difficult to understand, solve or explain, for example “a complicated court case”.

The complicated is cumbersome and wears you down, because it demands a lot of attention to detail; nothing can be neglected, since a mistake could be disastrous. The complicated is the land of the micromanagers, they’re happy there. But to tackle something complicated it’s important to have a detailed plan, a list of everything that has to be built. A list with all the details, a checklist, is essential in a context like this.

Detailed planning, controls and review: that is what a complicated process or project requires.

Divide and conquer is a suitable strategy in this case. We can split a complicated project into several parts and attack them one by one. This is what we call incremental, or waterfall, development.

Complexity is evolution

The problem with the complex is that it evolves, and sometimes even changes unexpectedly. Complexity is an emergent property of systems. Complexity arises from the combination of elements.

For complex projects it’s hard to have a detailed plan, and maybe it’s not worth writing one, because things can change abruptly.

Complexity has been widely studied by physicists, who know that a small change in the initial conditions can completely alter a model’s prediction.

The idea that we can control a complex project at a high level, from a remote and isolated control tower, is fundamentally wrong.

The key to managing a complex project is to sense and respond.

The right way to steer a complex project is the experimental approach.

The Stacey matrix

The Stacey matrix is a tool built on the work of Ralph Douglas Stacey. This organizational behavior theorist was one of the pioneers in applying complexity theory, from the natural sciences, to the realm of organizations and their management.

In 2011 Stacey proposed a scheme, a map or model for decision makers to guide their strategies along two dimensions: the degree of agreement and the degree of certainty.

In a diagram, Stacey puts on the X axis the degree of certainty the organization has about a problem. On the Y axis, the degree of agreement about it.

Stacey’s diagram lets us map the space of decisions according to the degree of certainty and agreement at which we can use rational approaches.

When certainty is higher but the degree of agreement is lower, more political decisions are required. If there is agreement but little certainty, then decisions call for more expert judgment.

In the upper right quadrant we have chaos, anarchy, where it’s impossible to make good decisions.

The diagram above is taken from Stacey’s work “Strategic management and organisational dynamics: the challenge of complexity”¹.

The regions of the Stacey map

To understand Stacey’s idea a little better, let’s look at this diagram:

Region 1 is that of rational, technical decision making, and it’s the one the management literature covers the most. Here we take data from the past and use it to predict the future. It’s the world of plans, with courses of action and identified outcomes that can be monitored. We want everything to be repeatable, to ensure efficiency and effectiveness.

Region 2 is that of political decisions. Here there is certainty about the outcomes, but a high degree of disagreement about which outcomes are desirable. There are no plans and no shared mission. Here politics matters a great deal: building coalitions, negotiating and making compromises, in order to give the organization a direction and an agenda.

Region 3 is that of judgmental decision making. Some issues have a high level of agreement, but there isn’t much certainty about the causes and effects behind the expected outcomes. In this case monitoring against a predetermined plan won’t work. What matters more here is having a shared vision and mission, rather than a precise plan. Comparisons are made against the vision and the mission, rather than against the plan.

Region 4 is chaos. A high level of disagreement and uncertainty. This often leads to anarchy. The traditional methods of vision, negotiation and planning are insufficient in these contexts. The usual strategy is to avoid conflict, but in the long run that attitude turns out to be disastrous. You don’t want your organization to fall into this zone.

Region 5 is that of complexity. In this zone traditional management methods aren’t very effective, but it’s the zone of the greatest creativity, of innovation, and of breaking with the past to create new ways of operating.

The problem with today’s managers and administrators

The bad news is that today’s managers, executives and administrators have been trained to handle areas 1, 2 and 3. Those are the traditional models. The techniques needed to manage regions 4 and 5 come across as too “soft”, lacking in predictability, with high insecurity and uncertainty, but reality happens mostly in those zones.

If you find yourself in zones 1, 2 and 3 there are well-known techniques you’re better off applying. The problem is that many executives want to manage everything that sits in the other regions as if they were in zone 1.

Agile methodologies and the Stacey matrix

In the world of agile methodologies a version of the Stacey matrix has been adopted that looks more or less like this²:

What this version of the matrix tells us is that in complex scenarios it’s a good idea to use empirical processes, like Scrum, for example. What’s more, in chaotic scenarios we have room to use triage, or case-by-case evaluation; that’s called Kanban, where we can keep prioritizing what we have to do until we get it under control.

For the simple, it’s enough to have a list of what needs to be done and execute it. Or we can use traditional techniques, like Waterfall. But nothing stops you from using agile methodologies in those areas, although they probably won’t add much. The value of agile shows up in complexity, as uncertainty grows and the degree of agreement dissolves.

So the claim that agile methodologies are no good for complex projects is not true. It’s an absolute falsehood, and whoever says it is simply trying to drag everything into their comfort zone, because they don’t understand or don’t know something as basic as the Stacey matrix.