CTO day 4: Team building with "I dare you" challenge

For a change of scenery and pace.

To do something different and fun.

To push the remote team to work together and gel.

To do some important but not urgent platform work...

I dared the team to enlist for an unknown challenge that they will know nothing about until the very last minute.

I dare you ...

The enlisting email:

Hi all,

The Dev brown-bag session of Feb 20th has been postponed.

Instead of finding a replacement, I am going to dare the whole Dev team to start and finish a work related challenge within a working day.

This is not a hackathon. You will be expected to work 8 hours, and it is actual work.

But you will need to work as a team, so if you decide to sign up for the challenge, you will need to make yourself available to the team 8am to
5pm Amsterdam time on Feb 20th, with a 12pm-1pm lunch break. Your colleges will expect that you accommodate your personal live chores
around the team schedule.

The schedule will be:

• 0800: Everybody is ready in the "war room", coffee in hand.
• 0801-0830: Dan dares you to ... I will explain the challenge here and then disappear.
• 0831-1640: Team argues about approach.
• 1641-1644: Team rushes to finish something.
• 1645-1700: Dan shows up again. We celebrate/cry/curse together.

I happen to be in Amsterdam that day, so I will jump in and out, and generally not available.

The activity is voluntary, but if you sign up for it, you are making a commitment to the team. Note that the challenge is not known until the day. Do not ask me beforehand "Would my skill set be useful? Would this be the best use of my time?". I dare you to help the team achieve the challenge in the best way you can.

If you are willing to participate please reply with "How do you dare!".

As always, open for questions/suggestions/comments or other pledges.

Cheers,

Dan

People

Out of 13 people in the Dev team, seven replied with:

Neither UX nor the Product Managers enlisted. Why not?

Seven people joined, two dropped the day before, and when the challenge was revealed, one silently disappeared.

Four brave souls to face the challenge...

Challenge

The proposed challenge was to move one of our existing applications to blue/green deployments.

We already had the blue/green experience with another application, and we loved it. This will be an opportunity to spread the love to other teams.

The main piece of work was changing how the application stored user uploaded images. We were using a Kubernetes PersistentDisk, which at that point in Google Kubernetes Engine meant that we could not have more than one instance of the application running at a given time.

The app was build on Django which has a nice File storage abstraction with a Google Cloud Storage implementation implementation. So in theory, the work was to change the Django configuration and migrate the existing user images to Google Cloud Storage.

As this was the most risky part, the previous week I did a proof of concept to make sure that it actually worked.

And it did. Hooray for abstractions!

The other big pieces of work were:

1. Create a migration plan and script for the files.
2. Copy and paste a bunch of Kubernetes yaml from the other blue/green application. A piece of cake.

The result

The day was an astounding success, if the objective was to experiment all the dysfunctions of a software project within one day:

1. The work was poorly estimated:
   • We finished the work afterwards, and it was way more work than a team of four can handle in a day.
2. The work was estimated by the "Product Owner" (myself), not the team executing it.
3. Jump to doing, not enough planning:
   • The team had to restart once because there was no initial plan and all was too confusing.
   • I have the feeling that a product manager would have pushed for a proper plan before starting.
4. One person was late to the kickoff meeting:
   • Required the rest of the team to explain the job again, wasting valuable time.
   • The late person got the requirements second hand, which ended up causing ...
5. Unclear requirements (and strong personalities):
   • The person that was late single-handedly transformed the objective of the challenge from "lets do something productive in one day" to "lets learn about mob programming and Python. There is no need to deliver."
   • This one still baffles me, but I wrote the requirements, so maybe is what Product Owners feel every day.
6. Devs not asking for clarification, just assume what the client wants:
   • Nobody asked the Product Owner about the unclear requirements.
7. Adding people to a late project ...:
   • I am happy that several people that has not signed up for the challenge jumped during the day to see if they could help, but given that none stayed, they were just a distraction that delayed the team further.
8. Leaky abstractions and unexpected technical issues:
   • Django's File storage abstraction happens to be a leaky abstraction, which means that the Google Cloud Storage worked slightly differently than the Local Storage implementation.
   • The proof of concept tested the file upload code path that worked in Google Cloud Storage.
   • The team stumbled on the file upload code path that did not work.
   • The team completely derailed here:
     • It spent several hours mob debugging a piece of Python code.
     • Did I mention that out of four, there was just one Python developer?
   • I have the feeling that a product manager would have cut losses and moved to team past this hurdle.
   • Why the team did not ask the Product Owner about the proof of concept?
9. Improvement work not important enough:
   • Not caused by the challenge, but the shortcomings of our development workflow became painfully obvious, specially in two ways:
     • No easy way of creating a new test environment to experiment safely.
     • Builds taking 20 minutes are not fine if you have 8 hours to do the work.
10. Siloed knowledge:
    • Copy and pasting a bunch of Kubernetes yaml files from an existing blue/green application is a piece of cake ... if you have designed the original solution, implemented it, and debugged it in production.
    • Tweaking a Helm chart to deploy a custom-built version of the application is a piece of cake ... if you have designed the original Helm chart, implemented it, and debugged it in production.
    • Reminder for all of us: programming is easy, if you have been doing it for 20 years.

Was it really that bad?

Well, at least there was no scope creep!

The fix for the leaky abstraction was useful when we later implemented the work.

Three of the participants said that they would do it again. I did not need to ask the forth one:

Several hours of mob debugging on an unknown language and platform can do this to anybody.

Doing it again?

Not how it is explained here.

First, I would not mix people from different teams. This can be a great tool to debug a dysfunctional team, as the team can see within a day what their pain points are, and the feedback loop is fast (daily).

If it hurts, do it more frequently, and bring the pain forward.”
Jez Humble, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation

Second, our teams are remote and in different time zones and/or schedules. Everybody working a whole day together is an exception, not the norm. Collocated teams may be required.

One year later, I have not dared to run it again.

Lessons

As usual, if you have time to reflect, you can find plenty of lessons:

1. There is nothing new under the sun. John Cutler blogged about One Day Sprints in 2017.
2. The experiment made obvious the value of cross-functional teams, an on-site customer should be part of that team.
3. Work in progress of one is very difficult.
4. Communication is messy, repeating things twice is not enough.
5. Short feedback loops are precious.

But the most important lesson (or should I say reminder?) is that you cannot throw a bunch of people into a project and call them a team.