The end of agile (VIII)

Part 8: Heavy metal

(To listen to the soundtrack of this post click here)

Thunder, thunder, thunder, thunder
I was caught
In the middle of a railroad track
— AC/DC

I remember my wife once asked me to talk to our son about the music he was listening to. Apparently she had caught Matías watching a Slipknot video, and well, I doubt the imagery was pleasant, and it must have left her a bit rattled.

Since I’m a firm believer in the relevance of Newton’s third law

\[1\]

to raising children, I spent time with Matías getting to know his musical tastes and talking about different metal bands. We listened to some classics by Iron Maiden and Black Sabbath, went through funk metal, Smashing Pumpkins, Metallica, of course, and checked out some contemporary bands that he liked, such as Korn, System of a Down, and of course Slipknot. The truth is that it’s very hard not to admire Joey Jordison’s technique, but I couldn’t tell my wife that ;-)

Slipknot

I think listening to aggro music isn’t necessarily a bad thing. Teenagers have inner struggles and real problems: conflicts at school with other kids their age, fights, bullying, even parents who don’t understand them, and they channel their anger, in a way, into the music they listen to. I think that rather than criticizing their clothes or their taste in music, we should stand by them and listen to them more. Something we find hard to learn, of course. We forget that we went through that very same stage on our own way to maturity. But that’s not what this post is about, so let’s pick up the story where we left it.

Networks

We fought him hard we fought him well
Out on the plains we gave him hell
– Iron Maiden

In August 2007 I was out of work. I started looking for a job from the very first moment. I sent emails to everyone I knew. For years I’d had a business relationship with Don Enrique Rodriguez, my first mentor. I met him when I was studying engineering; it was the first job I ever had. In those years I built a series of sales systems for his hardware store in Buin, a town south of Santiago. I spent several summers working with him, refining the system and maintaining it. It was a steady income for several years, even while I was building my professional career. Later he acquired more stores and used the same software. He went on to become president of ChileMat, a nationwide network of hardware stores, and one of the initiatives he pushed was the technological upgrade of the stores in the new chain. A team of programmers was put together, and they started building software for the chain and for the stores, based on the systems I had written in the late eighties. I advised them on this for several years, trained the programmers, helped with the design and architecture. Many Saturdays I traveled to Buin to work with Enrique on these projects. It was on one of those trips that I had the accident I told you about in part seven.

After resigning from BioKey I talked to Enrique and asked for his support. Alongside that I started exploring different ideas. One was to start my own company and devote myself entirely to building software, trusting that Enrique could be my main client. I also talked to other friends, had several interviews and a couple of offers.

Something I always recommend to young professionals is to work on building their networks. Spending time with other professionals is essential, and it’s in moments of crisis that it shows the most. As Enrique says, “it’s better to have friends than money”, and he is absolutely right.

I was out of work for exactly twenty days. In the end I talked to Ubaldo; you may remember him because in the second part of this series I wrote about our trip to Borland.

He and his partner were interested in my experience in biometrics, since some important public tenders were coming up where that knowledge would be valuable. He also helped by putting me in touch with a lawyer, who helped me negotiate the terms of my exit from BioKey and gave me legal advice on the matter. Another thing Ubaldo told me was that he wanted more gray hair on his team. So I decided to accept his offer.

Heavy Metal Poisoning

That heavy metal (heavy metal) is poisoning
It's a music wasteland, that destroys the young (yeah!)
– Styx

Some have defined heavy metal as one of the major subspecies of hard rock, with less syncopation, less blues, more showmanship and, of course, more brute force

\[2\]

. Some say the roots of the genre lie in the music of the holy trinity: Led Zeppelin, Black Sabbath and Deep Purple. But its peak, the peak of what we’ll call classic heavy metal, probably came in the second half of the seventies and the early eighties, with bands like Judas Priest, Motörhead, Iron Maiden, Def Leppard, Van Halen and AC/DC.

Heavy metal spawned an enormous number of subgenres, such as Death Metal, Thrash Metal, Black Metal, Power Metal, Glam Metal, Aggro Metal (where some people put Slipknot), and so on.

Ubaldo is a big metal fan, and a fan of its more alternative genres, and to this day we keep up “friendly” arguments about the worth of the music he listens to. Although I haven’t managed to convince him to improve his taste in music, there is some overlap in what we like, mainly in the more classic heavy metal.

An intense year

Run to the hills
Run for your lives
— Iron Maiden

I started working at EXE in September 2007. I spent one year at that company and the truth is it was pretty intense. At the time the operations manager was Eric, a friend of many years, who brought even more gray hair than I did :D. In those months we shared many lunches talking about metamathematics, philosophy and science fiction. I’m very fond of him: a teacher above all else, but a great friend and an even better adviser.

I did some new things while working there. I took part in an ISO certification for the first time. Also in an IT audit of a large concessionaire company, a job where I learned about technologies I hadn’t imagined, and especially about how to organize an audit, something that would be very useful to me years later. I helped organize the team in charge of developing one of the company’s own products. I contributed a few lines of code to that product. And I took part in several other projects too. I was also on a team assembled across several companies to bid on a big nationwide tender. As you can see, it was a busy year.

In companies that build software projects there is always a lot to do. The problem, in my opinion, is the fierce competition and the uncomfortable reality that the client doesn’t always know what they want, and has a budget that usually isn’t aligned with their real need.

The problem with estimating

Estimating how much something will cost to build, in time and effort, is an art. And it’s the main reason companies that build software succeed or fail.

There are studies in software engineering that say that at the start of a project’s life cycle, before requirements gathering, we estimate with an uncertainty of a factor of four. This means that once the project has been carried out, the actual effort or real scope of the project can be up to 4 times, or 1/4 of, the first estimate.

This is what we call estimation uncertainty. In general this uncertainty shrinks over the course of the project, although that is not guaranteed.

It shows up in this diagram:

This is what is known as the cone of uncertainty.

The vertical axis reflects the degree of overestimation or underestimation of the project’s scope. The horizontal axis shows the stages of the project. You can see that a detailed design of the specifications is the only way to reduce the uncertainty. But the big problem is that we are rarely given time to do that detailed design. We have to estimate based on intuition or expert judgment. And on the left side of the diagram we see three phenomena at work:

One: executives underestimate a project’s time and effort.

This is known as Parkinson’s squeeze, after the so-called Parkinson’s Law, which says that “work expands so as to fill the time available for its completion”. So executives apply the opposite, or Parkinson’s squeeze: they try to cut the time in order to increase the productivity of their teams or their vendors.

Parkinson’s Law happens because of our human nature. If you’re given all the time in the world to do a job, you’ll tend to relax a bit, since there’s no urgency. It can also happen that you start making the task bigger than it needs to be, adding complexity in order to fill all the time available.

On the other hand, if you’re given less time to do the task, you’ll approach it with a greater sense of urgency. You’ll cut the work down to the essentials and focus all your energy on getting it done within the deadline you were given. That is Parkinson’s squeeze, and executives will tend to use it.

Two: engineers underestimate too.

The people who carry out the task also underestimate their effort. But unlike executives, who have a clear motive, what happens with engineers is overconfidence. Here we are in the presence of Hofstadter’s Law, which says that “It always takes longer than you expect, even when you take into account Hofstadter’s law”.

There is a related phenomenon, known as the Planning Fallacy, observed by Kahneman and Tversky in 1979. What’s at work here is what’s known as optimism bias, which makes us believe we’re unlikely to experience a negative event. Engineers tend to dismiss, or ignore, other phenomena outside the project that can put its execution at risk. From the possibility of getting sick to the fact that they’re using a software library that isn’t compatible with the client’s production environment. Overhead tends to be ignored by engineers.

Three: project managers estimate better

This depends on the project managers’ experience, of course. Over time, project managers get to know a wide variety of impediments that come up in projects. A good manager is one who is very aware of the project’s risks. In the end, managing a project is managing its risks. That, combined with historical records and statistics, can lead to better estimates.

The key, in my opinion, is that you need good project managers on your team, the kind who know how to question the estimates of both the executives and the developers.

Many software development companies in Chile, in my experience, focus on hiring good programmers but neglect the selection of project managers. I also think project managers are very poorly trained. In my opinion, many are just order takers, and don’t take care of what really matters, which is making things happen. They don’t anticipate risks, don’t motivate teams, don’t report or escalate problems, and don’t convey a sense of urgency either. Some just devote themselves to pleasing executives, clients and developers; they don’t want to look bad to anyone. They don’t demand much of themselves, or of anyone else. On top of that they have poor negotiating skills. They end up agreeing to things without consulting their teams, and a long et cetera that I don’t want to get into because otherwise I’ll get angry.

So here’s a piece of advice for my friends who own software development companies: please, choose good project managers. Train good project managers, hire the best project managers. Be aware that you will be victims of Parkinson’s Law, of Hofstadter’s Law, or you will fall into the planning fallacy. The best antidote I’ve seen for that is having good project managers.

Unexpected encounters

I think of all the education that I missed
But then my homework was never quite like this
– Van Halen

That working year was very intense, as I said. I did almost everything, including acting as a sales rep. It was on a visit to a potential client, a company with offices in the building of the Cámara Chilena de la Construcción (the Chilean Chamber of Construction), that I ran into Jorge Rojas, my former boss at the CCS. He was working at Habitat at the time. We talked for a while; I told him where I was, and he told me about the challenges of his new job.

The following week I had to go back to the same building, this time to give a demo to that client, a company I barely knew, whose name was Previred. In the elevator I ran into Esteban, another former colleague from the CCS. He said hello and told me he’d just been wanting to talk to me, handed me his card and told me to call him. But I’ll tell that story in the next part.

The fact is that those encounters opened the door to my current job, which I’ll talk about in the last chapters of this series. In September 2008 I was getting ready to start over. I talked to Ubaldo, and we parted on good terms; I thanked him for his support and for the opportunity, and he thanked me for our work together and wished me every success. I would be wearing a suit and tie again, but the future looked challenging, and it really was...

Notes