The end of agile (XII)

(You can listen to the soundtrack of this article at this link)

“A good drummer listens as much as he plays.”
— Indian proverb

As I told you before, in the second half of the nineties my friends and I went back to rehearsing as a band. At that time I invested in a drum kit and every so often bought an issue of Modern Drummer magazine. Unlike today, when you can find thousands of tutorials on YouTube, back then you turned to written material to learn new techniques or improve as an amateur musician.

It was in one of those issues that I read an interview with Neil Peart, the drummer of Rush, who told how he had gone back to studying with the master teacher Freddie Gruber.

Gruber’s story is an eventful one. He could have been one of the great jazz drummers, but his heroin addiction took him off the road to success. With the help of his friends, among them the great Buddy Rich, Freddie was able to recover and find his true calling in teaching.

Gruber taught every aspect of playing the drums, but his real gift, in the words of Neil Peart, his most famous disciple, was “an unparalleled understanding of the physical ‘dance’ involved in playing the instrument, the ergonomic relationship between the drummer and the drums” (take note of this because it’s important).

At the end of 2015, as I recounted in part 11, we were in crisis. That same year, at StartechConf, I gave a talk I titled “El viaje del agente de cambio” (The change agent’s journey), where I reflected on my attempt to change as a person and to make changes in my organization. But I had been guilty of arrogance, trying to drive the change through sheer will and without involving my colleagues and my closest collaborators very much. To some extent what I said in that talk was true only in theory; it hadn’t been effective. So I had to own the crisis and I had to change.

Start taking on board the things Miguel, Carolina and Elizabeth had told me over and over. Accept, with humility, that I still had a lot to learn.

So, just as Neil Peart did, who went off to learn in order to improve his technique while being the best drummer in the world, I went off to study too.

Unlearning in order to learn

“If the future's looking dark,
We're the ones who have to shine.
If there's no one in control,
We're the ones who draw the line.
Though we live in trying times,
We're the ones who have to try.
And we know that time has wings,
So we're the ones who have to fly.”
― Neil Peart

As it happened, since we hadn’t coordinated it, my two lieutenants decided to do the same, and the interesting thing is that we studied complementary subjects. Carolina decided to train as an organizational coach, and Elizabeth to study for a diploma in software quality management.

For my part I decided to close a personal chapter. When I was at university my plan was to graduate with an engineering degree and a master’s. But I hadn’t completed either one. A thousand things happened in my life that made me postpone my studies, and so, without my noticing, more than twenty years had gone by. This never meant my professional ability was questioned when I looked for work. In the late eighties and early nineties many of us computer engineers entered the industry in this situation; our skills were in demand and the job offers were good.

I decided I would do the Master’s in Information Technology at my alma mater, the Universidad de Chile. A program offered for industry professionals. My older children were already leaving university, so I could invest in my own education. I also got some support from my employer for this, so all I had to do was put in the work to see these studies through.

I remember that when the Universidad de Chile’s computer science department celebrated the 40th anniversary of its founding, I was invited to a cocktail party at the faculty. That night I talked to several friends and acquaintances who were academics, or taught in this or other extension and continuing education programs. At one point one of them, who later became one of my teachers, asked me what I was doing this for. After all, I already had all the knowledge taught in the program. But I wanted to close the chapter, to have at least one degree, and also to teach my children that everything must be closed and finished properly, even if it takes time.

While doing the enrollment paperwork for the master’s program I discovered that my university was generous with its former students and let them complete their studies, which in my case, for the civil engineering degree, meant one last-semester course and the thesis project, so I seized the opportunity. I studied for two years to get my engineering degree and my master’s.

The Universidad de Chile program requires taking a couple of diploma programs and investing a number of hours in preparing the thesis. But I rode the momentum and signed up for three diplomas: “IT Project Management”, “Software Engineering” and “Data Science and Engineering”, the last one the hardest, but the one where I learned many things that were new to me.

Putting the team back together

In 2016 I took some leadership and negotiation courses. That’s where I began reflecting on the way I related to my team. Around that time we also diagnosed what had happened to us and decided to work on improving the team’s morale.

We suffered another couple of losses, Natalia and José, who had worked on a tough project in late 2015 and early 2016. They really had fallen into burnout, one of the ills of our time.

We started bringing in new people, and more of them, to join the team; that’s when Gerardo and Fabián arrived. Foreign members joined our team too, like Pedro and Johan. Immigration was beginning to show up as a phenomenon in our country at that time, so apart from the gender diversity we were trying to maintain, we had to deal with the cultural diversity that comes with bringing in people of different nationalities.

The most important changes happened in Elizabeth’s area. With the departure of Gustavo and Contardo (see part 11), that team was at risk of collapsing. We made many changes and managed to improve that group’s productivity in spite of so much change.

One of the first things we did was talk to Gabriel Ovalle, who was head of systems at the time. We analyzed all the incidents that generated requests from the processes area and turned into data-operations work, which ate up a lot of the team’s time. We discovered that many requests had a historical origin: users made them because there had been a time when the systems had no features to solve those problems, but the truth is they had since been solved. The problem was that people didn’t know those features existed, or the knowledge got lost with staff turnover. We fixed that: we trained people, built some new features, or fixed bugs. We also cut several incidents by automating some tasks. And we left a pile of data-operations work with the DBA team that reported to Gabriel.

We put the emphasis on continuous integration. That’s when Jonathan Peña y Lillo, who was a quality analyst until then, picked up the baton from the work Gustavo had started and put a continuous integration process in place for good, so that our developers would stop handing over binaries to install. Now at last we had a cycle in which the artifacts to be delivered to production were generated automatically from the source code repository.

I remember the day they showed me the first continuous integration pipeline set up in Bamboo. I was truly moved: what I had seen at Borland in 1998, and in Boston in 2011, was finally taking shape in my area. I felt proud of them.

At the end of 2015 Claudio Araya had joined as a systems engineer, in charge of maintaining the infrastructure of our development and QA environments. But beyond technical skills, what we were looking for at that time was people eager to learn, with a positive attitude, cheerful people who would change the team’s morale, who would push it upward. In that sense Claudio is, and continues to be, an important pillar in keeping up the team’s attitude. One of the positive leaders of my unit.

In 2016 the “Mujeres Programadoras” (Women Programmers) program of the Kodea foundation also appeared. We gave two of the students in the program, Natalia and Camila, the chance to do their internship with our team. In the end we decided to bring Camila Donoso onto the QA team. She was one of the pioneers of an important change we made to the quality control process: bringing in programming. That’s how we started creating automated tests, using Selenium and Python in our process. But other techniques were brought in too, which changed the way testing was done. Gradually Elizabeth’s team was beginning to bring DevOps into our organization.

Rhythm and ritual

“Rhythm is the soul of life. The whole universe revolves in rhythm. Every thing and every human action revolves in rhythm.”
–― Baba Tunji

It was in the software engineering class that I met Daniel Perovich, who would later become my thesis advisor. I remember that in his introductory class to the course he wrote two words on the board:

Rhythm and Ritual

With those two words you could sum up all software development methodologies, and in particular the so-called agile methodologies.

And who is in charge of those two things in a rock band?

The project manager, for example, is like the entire rhythm section of a rock band, in charge of keeping the rhythm, but also of making sure the rituals the software development process calls for get performed. And so on.

I think the metaphor explains itself. But I wasn’t aware of this; I hadn’t made that distinction until that afternoon when I sat in on that class. I thought I knew about software engineering, yet something so essential was escaping me.

It’s by studying, by analyzing things, by taking the time to dissect them and extract their details, that we can truly understand what is going on and make them part of who we are.

Rhythm and Ritual, and what better example than SCRUM to reflect this.

Agile processes

“When Mr. Ludwig invented the bass-drum pedal, that's what made the drum set possible.”
-― Neil Peart

I’m not a big fan of Scrum, but it’s an excellent way to introduce agile into a structured environment, with lots of rules, with regulatory obligations, where your processes are audited. It also works like the training wheels you put on bicycles so kids can learn to ride.

Scrum lets you build the habit of continuous delivery, of the time box, which fixes the time, and you learn to control the scope of what gets delivered. It lets you introduce some agile terms. Even Spotify, seen as one of the leading references in the use of agile methodologies, started with Scrum.

It was in the second half of 2017 that we started officially adopting Scrum in some teams. Before that change we had a process we could more or less describe as iterative and incremental.

But more than the process, what was going on was that we had a serious problem with quality and delivery.

Quality

“I don't think there's such a thing as a 'best' drummer”
― Mike Portnoy

This Geek & Poke cartoon reflects a lot of what happens with quality in the software development process:

Once we were literally asked to do this on a project. The vendor complained that our team was finding too many bugs.

And it really does happen. It also happens that the quality control process seems never to stop. That was when we flipped the switch.

Traditionally the development process involves a stage we’ll call construction, followed by a quality control stage. And that’s where the iteration starts, which can become infinite.

In our case we tried a metric that measured the number of iterations between development and QA on each delivery.

Essentially our process looked like this:

There were many iterations between development and QA, and projects dragged on until we cut them off and made the decision to go to production. That, of course, had to change.

The key is to stop thinking of two independent units. The development team and quality control must work together. We also had to make it clear that quality is not the QA area’s responsibility; quality is a concern of the whole team.

We defined that everyone was a developer; there would no longer be a distinction between those who programmed and those who ran tests. We also asked our testers to write automated tests, so they were quite literally developing too. For me there is no longer any distinction between those roles: they’re all devs.

Another important element was defining the right metrics. The most obvious: the number of defects we ship to production. We started measuring what percentage of defects we introduced with each production release. It’s painful to measure this, but important. A standard measure in DevOps is the change failure rate: you count how many failures you had when making a change to the system. Another measure is to compare how many defects you introduce against the amount of functionality you add with each production release. On the first we had a figure of 16%; on the second we were above 30%. The first figure isn’t bad; the second one is. But you get a better picture when you complement it with other interesting metrics.

Now, thanks to Scrum, delivery cycles were fixed: 2, 3 or 4 weeks. We knew the number of new features, improvements and fixes that would be implemented in each cycle. That made it easier to put in place two metrics that are essential for improving.

The first of these metrics is Defect Removal Efficiency (EED, by its Spanish initials). The classic definition, described in Pressman’s software engineering book (fifth edition), defines it as follows:

EED = E / (E+D) 

In this equation the value E is the number of defects found in a previous delivery, and D is the defects found in the current delivery. Ideally this value is 1, since that means you’re removing all the defects. We use a modified version of the formula Pressman presents (a more demanding metric, in fact). But you can start with this simple calculation to begin measuring improvement in the quality of the work.

The problem was that our EED values were close to 0.5. Our way of measuring EED gives us more information, which let us deduce that what was happening was that we were introducing as many defects as we were removing in each iteration. Another way to look at it is that you spend 50% of your time fixing earlier mistakes.

Another complementary metric is the one we call Delivered Value (VE). A ratio similar to EED that lets us see how much of what we ship to production is new features and improvements, as opposed to defect fixes.

Crude as they may seem, these simple metrics allowed us to improve. We grew, we got better, and we changed the way we did things. We were bringing agile in our own way. And it went well for us. We turned around the results of the workplace climate surveys, we were even among the best rated, and we sustained those results over time. The spirit of the team changed.

Our transformation journey

“It´s nice when somebody says that you're their 'favorite' drummer” –― Mike Portnoy

We told all of this to our country’s agile community, at a Meetup we organized. A presentation we built on the skeleton of my 2015 talk, “the change agent’s journey”.

The big difference is that now it was much more real, and it carried my team’s vision; it was no longer my selfish vision, it was a more inclusive and sincere view. But the best part is that this story was presented by my closest collaborators, Carolina and Elizabeth, without whom I couldn’t have achieved any change at all.

That Meetup was special. Everything was organized and hosted by the whole team, and it marked an important milestone for us: we were stepping outside to tell our experience and being listened to with interest and respect.

All of this was captured in a video recorded by Miguel González, which I’ll leave here for you to watch; it describes better the changes we made, our experiences, pains and lessons learned.

With this we’re getting close to the end of this series. The end of an era was approaching, though we didn’t suspect it. Bigger changes and challenges were coming, but we were better prepared to face them. We still had problems to solve, and not minor ones. But we had changed, and that’s a good thing.

Notes:

The video is at this link: https://www.youtube.com/watch?v=i0GpxfyZjr4&feature=youtu.be

The first part of this series is at this link.