Fixing a broken software development process through better technology

Introduction

The software development process keeps evolving to keep up with the demands of the enterprise. As software is (famously) eating the world, more and more organizations make more software, have more frequent changes, requirements and integrations, and must manage a global workforce of developers. Software is now discussed in boardrooms as a strategic asset that drives value creation. In addition, work-from-home is the norm, at least for now, and the trend might stick for the foreseeable future. All this is straining the traditional approach to managing software projects. The assumption that a small team, co-located in an office, can easily communicate, plan and execute is no longer valid. A new approach is needed.

A collaboration gap

The typical planning process for software development teams is structured around a sprint. It starts with (a) the allocation of tasks, followed by (b) the execution of those tasks, such as developing a feature, and it ends with a (c) code review triggered by a pull request. During phase (b), developers typically don’t collaborate much. By the time a code review begins, often after a week or two of coding, the amount of work required to rework a suboptimal solution often leads to compromises and the accumulation of technical debt.

The code review process, which typically involves the developer and one or more reviewers, is also meant to share knowledge about the codebase. While ephemeral knowledge is exchanged among the immediate participants, that knowledge is not captured in a digestible form and is not widely available to other developers in the organization. That leads to a knowledge gap that manifests itself clearly when key developers leave the organization.

While developers spend most of their time in the Integrated Development Environment or IDE, IDEs have until now not served as collaborative tools and do not include the essential communication, project management and knowledge sharing components that reduce collaborative friction. Therefore, developers must jump from their IDE, to Slack, to Jira, to GitHub.com, to email, many many times per hour, in order to collaborate. This context switching is highly inefficient, disruptive, and going forward, unnecessary.

Shifting Left in Developer Collaboration

In the last decade, the Shift Left Principle has taken over the software operations and quality assurance process under the newly coined term DevOps. As defined by AWS “DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes.”

Still, the “Dev” half of the DevOps equation remains less automated, less collaborative and less efficient than the “Ops” side. According to Devopedia “The principle of Shift Left is to take a task that's traditionally done at a later stage of the process and perform that task at earlier stages.” The answer we are looking for in addressing the deficiencies in developer collaboration might be a combination of shifting left from a cultural perspective while adopting practices and tools that integrate the essential tools into a collaborative environment.

The IDE becomes the Hub

While the software development process has always involved many participants, the IDE has generally been seen as a standalone, single player tool. In order to stay in the zone, developers shunned distractions, and the IDE was intentionally kept isolated from the communication and collaboration aspects of the software development workflow.

GitHub, Jira and Slack are the most widely used tools among developers, according to the 2020 Stack Overflow Survey. All three are part of the collaboration required to produce high quality software on time. But none of them were integrated with the IDE as a natural environment when code is written and all appears in context. GitHub is mostly used in a web browser, causing regular context switches between the IDE, the command line and back to the browser. Collaborating in Slack requires developers to copy and paste code, a tedious and infrequent task, and then to explain to the recipient the context that was just lost through the cut and paste operation. Over the last 20+ years we've seen incredible innovation in the browser, as web technologies make it easier to deliver robust and compelling user experiences. So perhaps it's no surprise that Jira, Github, and Slack are all web-based.

Starting with the Atom editor 6 years ago, and more recently popularized by VS Code, editors can now be extended with web technologies. This opens up the possibility of bringing these powerful capabilities directly into the IDE. Since context switching represents 70% of the steps involved in the unintegrated developer workflow, the same work can be accomplished with ⅓ the effort in an integrated environment where pull requests, issue tickets and messages are associated with the code itself.

From Transparency to Visibility

Many organizations are striving for transparency as a matter of cultural adaptation. When transparency is implemented, better alignment is possible. Communication becomes more streamlined and overlapping departmental and personal responsibilities are better clarified and defined. In order to become more transparent organizations must deploy technologies that increase visibility.

Examples abound: Many teams share calendars in order to coordinate schedules and meetings, many of us set our status on Slack or Microsoft Teams to show our availability, and in this year of Working From Home many have chosen to keep a Zoom call open among team members to be able to casually discuss ad hoc issues. Still, software developers keep working away in their IDEs after grabbing a ticket and emerge from their focused sprint with an almost finished product. If the individual IDEs were connected to each other in a team setting, increased visibility could be possible. Team members could see what the others are working on, team leaders could help out someone who is blocked, and software quality would improve on the first pass.

A New Workflow is Desirable

It is both possible and desirable to integrate the essential tools developers use every day in order to boost collaboration, timely communication and reduce errors and technical debt. The technology is available. The most important step now is to redefine what’s acceptable from a cultural perspective so that developers can feel free to share imperfect work early that is made dramatically better by the contributions of their peers, thus fostering shared knowledge.

We must step forward in the direction of a new modern workflow where integrations are the norm, code reviews happen sooner and more often, and knowledge is shared effortlessly through technology.