DEV Community

Cover image for DevOps at scale: a journey where management killed innovation
Λ\: Clément DUFFAU for Stack Labs

Posted on • Updated on

DevOps at scale: a journey where management killed innovation

DISCLAIMER: This article is not trying to promote DevOps at scale or exploring generic solutions. This article only aims to give feedback on the topic of an organization failing to scale and propose opinionated reasons for these failures.

When I arrived in a big safety critical company, the mission was very clear: Analyze and propose solutions to change
the way the testing team was approaching the DevOps philosophy of work.

The department management seemed very clear on what they expected from pushing this team to a DevOps culture meant for the rest of the department.

This article describes my journey on how these initial expectations and resistance to changes kill innovative initiatives.

Context: A hybrid development organization reducing development team flexibility

The company needs to integrate iteratively legacy and cloud native systems, but the Agile culture is not adopted at each company level even though these levels need to cooperate and communicate.

Danger zone

Company hierarchy

The department where I conducted this mission is the first link in the chain with about 40 people in it. Their project management is multi-level, from the final client to the system engineering department. There are several requirements refinement and they are splitted at the end of the process to multiple departments which develop and deliver a part of the final system to a system department that integrates each module and ensures final client needs fullfilment.

All in all, it breaks down to a classic waterfall organization with around 400 contributors building complex systems from defining client specification to operating the product.

Waterfall x SaFe

However, each development department is pushed by its top management to rationalize its development regarding each final client. As a consequence, a software product line approach is used to drive the development department.

Software product line paradigm

Software product line has emerged from the automotive industry where whitin a single factory production line, several car variants could be produced. From the beginning of a project, each variant induced decisions made on the product design, the supply chain and the production line workflow. Thanks to this approach, the manufacturer can propose multiple products to its clients but manages them with efficiency from development to production.

For a compnay which wants to capatilize software around a product line, this strategy implies that the final client and the company project management for this client has a reduced impact on development for their use-case. The development department needs to have a multi-view on the differents clients and needs to maintain consistant software product line to be able to ensure software profitability in the time.

This paradigm also put innovation from the client side but also in the department himself. The latter is the only actor in the company that has a complete visibility on what can be done to accelerate software development. The department manager needs to defend to company procurement that internal budget need to be sanctuarized to let them shape the software product line for the future.

If the product line is not managed as described above, a common issue arises: not a single project leader wants to pay extra costs to maintain the software product line but asks to reuse features from others projects for free. Guess what? My mission takes place in the middle of this mess!

Agile vs agile

In the company where I conducted this mission, Agile seemed to the management to be the silver bullet to manage the software product line. However, they were more at ease with V cycle project management. Therefore, a tradeoff was found with a SaFE methodology to give a 3 month increment view to system project management but manage internaly with 4 iterations of 3 weeks inside the development and test teams.

Let's introduce another issue: the difference bewteen an Agile organization and an agile organization. An upper case letter changes everything here.

An Agile organization uses Agile methodologies to structure its work to achieve the 4 pillars in the Agile manifesto.

For example, the "Respond to change" pillar consists in finding a structured way to organize work and scoping it at a given period of time to be able to quickly develop and iterate during sprints. An agile organization thinks that it is using Agile methodologies to "respond to change" and gives a super-power to its clients procurement. This ends up in putting backlog features into the current sprint at any time and without sprint scope refining.

Guess what? In the company I dealt with, the concept of agile rather than Agile was that system project management staff found benefits for them.

Deal with it, "we are Aagile!"

Do the opposite

What I saw in each development team was terrible. They struggled to do Agile well but were completely unsupported by their management to do it the proper way.

All the different slow down periods initially scoped in their sprints to handle a team breakout, do some technical watch or find solution to internal daily issues collapsed on top of the features stack. The features not initialiy scoped for the sprint but that had to be done anyway due to management pushing for agility rather than Agile were common.

From here, let's look at the scene with DevOps eyes. Between the clear management envision about DevOps and teams that have no time to test, learn, push new tools and methodologies, the gap was huge.

Technical issues: DevOps can not fill a software crasftmanship culture absence

Archeology

Firstly, I began the technical part of this mission analyzing at the department level:

  • the way development team worked within and with other teams
  • what kind of tools they used to manage CI/CD
  • the way they delivered versions to the test team
  • how the latter used these delivers
  • the feedback loop from the testing team to the development teams
  • how clients use these deliveries.

In simple terms, they lived in the 00's where Clearcase was the most common versioning tool in the department. Some teams swicthed to Gitlab to have a GUI and git server but sadly not for Gitlab features such as CI/CD... Some teams used Jenkins to build and sometimes test their features. The build artifacts repository was a simple NFS shared mount volume without snapshot management.

When thinking about DevOps, we think about GitOps, Infrastructure as Code, automation on build, test, deploy and tools such as Terraform, Ansible... At the beginning of this mission, they only had a home made Infrastructure as Code tool based on Cloud-init and integrated Neutron/Nova/Cinder API to be able to deploy VMs on Openstack. We were far away from thinking about DevOps at scale although management thinked having this level of maturity.

These different initiatives were taken individually by each development and testing teams with collaboration between them to centralize the tools and methodology.
For example, they didn't have any Git flow convention, no preferred CI tools, no internal standard delivery process, no share of test coverage. The teams wanted to do their job well, but no strategy had been defined at the organization level to rationalize the initiative and sponsor them to embrace the DevOps at the scale approach within the department.

No, Thanks

At this point, I understood that my role was not to help a specific team on adopting DevOps as pointed by the management but smoothly push the management and development teams to understand that DevOps is not a "we ask, we have" or a lonely work handled by one single person, but a maturity to reach by investing in software craftmanship.

Organization issues: Risk management and resistance to change forgotten

All right, instead of giving some specific advice to a single team, let's propose a complete DevOps pipeline at the organization level to reach the initial target: DevOps at scale!

Management does not support innovation risk

This indecent proposal was stopped by a non-technical aspect!

The department management expected Return on Invest (RoI) on things such as automated system deployment, automated testing at the software product line level, automated test report generation and requirement traceability, ...

(WARNING: salty words here)
From my point of view, in 2020, if we need to demonstrate RoI to people who mandate you for DevOps solutions, they surf on buzz words such as DevOps but don't understand why this is a "must have" to industralize their software product line.
(end of WARNING)

This is pointing also that no one in the department defends to the top managemenet the need to embrace DevOps to scale the business. And yes, we came back to the original Agile organization point where the team does not have time to take a step back to push new ideas because they are drowning in the features backlog.

A lot of developers and team leaders completely stopped to be in a continuous improvment state of mind. Things were delivered at the right time and did not always meet the expectations but the ship is still in the sea!

Technical staffs resign or resist to changes

During the mission, I demonstrated the benefits for developers with presentation, tutorial, return of experience from people adopting DevOps but very few were convinced to go and defend the approach inside the department.

On a coffee break, a team leader said to me "Clément, I am not evaluated on improving the team, only delivering on time! Why should I take the risk of innovation while breaking my objectives? When I argue that the team workload is too heavy, they organize big meetings to explain how their requests should be possible. They do not want to resolve the underlying problems for the future, they want to resolve the visible problems right NOW".

I think that is the most terrible sentence that I have heard in my career. How could someone from software development give up on what they think about the job and the future of it to stay and die with old school practices! The problem was not really him but the way the management evaluates his success and interferes with his role.

Trying to solve problem only focusing on monthly backlog and current team velocity rather that finding solutions in the way of working with dedicated tools and an up-to-date process is a scrappy patch on the underlying issue of not evolving with the rest of the software industry.

I can not picture industry big players from our century such as Google, Tesla or Apple evaluating team leaders on "staying in the past" with objectives only focused on what matters today. I can not even imagine a software engineer looking at cloud technologies or DevOps methodologies in others companies and stands on "it does not matter" for his company and even for himself!

The cherry on the cake was when another team leader said to me in a DevOps tools demonstration "I don't believe in automation"! It was not a "I don't believe in the automation you propose" or a "I need to analyze and test automation tools to be confortable with". It was a sentence like you can have with your family with a "I don't believe in God". How can you argue on a belief ... This is impossible, this is a individual tenet where rational or irrational argument can not change their mind ...

That was the end of this mission for me. When technical challenges are not what matters, run for your life! How can I help technical people when the management kills their taste for innovation...

Run for your life

Conclusion: DevOps is as much of a technical process than an organizational one

Software development process evolves and stackholders jobs too

DevOps breaks the big company silos and this implies not only changes inside development teams but a global shift on interactions between silos. The interactions were mainly based on project management leadership before Agile and even more before the DevOps philosophy, but these new methodologies empower the technical people to do management as well.

Nowadays, software development is not based anymore on established management process but more based on technical ones with shared tools, common initiatives to fill the gap between dev and ops for example.

The management also needs to reverse their way of working and not only control a process execution but trust and sponsor the teams on the way they identify ways to work more efficiently. Knowing technical buzzwords does not mean this is only a technical job!

What are the perspectives for the company where my mission took place?

At the end of the mission, the company created a dedicated DevOps team to standardize practices for the department to break the existing silos in the company. Their goal was to put together people trying to lead DevOps in their teams to federate the DevOps initiative.

I am not sure that just bringing together tech skills in this team will be sufficient. They will do the job, implementing further the methodology and the tools to design a scaled DevOps approach but how will they push those back inside the development teams? How will they convince the management staff that DevOps is not a one-shot project but a daily purpose to everyone in the organization?

My personal conclusions

Personally, as a DevOps architect and an external point of view, I will remember that DevOps is seen too often as a technical thing. Deploying a Git server somehere, developing a Gitlab CI pipeline here and managing production deployment using ArgoCD elsewhere is only the starting point.

After that, the question of defining the development process to identify how DevOps can serve as the foundation to software development and delivery is crucial. This question can not be answered only by the management staff nor by the developers. It can be tackling by working together and cutting the old organization role boundaries to define new ways of working with the same goals.

Personally, as an individual, I will not try anymore to convince and work with a colleague pronouncing "I don't believe in [PUT_TECHNOLOGY_THAT_MATTERS_FOR_YOU]". This is too much time consumed with someone you won't be properly able to argue with.

For further dicussions and comments, I am very interested in the readers feedbacks on their experience to promote DevOps at scale and how they failed or succeeded to it!

Discussion (0)