Spacelift team for Spacelift

Posted on Oct 8 • Originally published at spacelift.io

What is an Internal Developer Platform (IDP)?

As the complexity of modern software development and cloud computing technologies increases, organizations aim to streamline their workflows and enable their developers to be as productive as possible. The need to balance agility in software development and infrastructure management at scale has given rise to platform engineering and the internal developer platform (IDP) concept.

This post will explore an IDP, its key components, its benefits, and why it might be the next big thing in your development ecosystem.

What is an internal developer platform?

An internal developer platform (IDP) is a self-service layer that sits on top of an organization's infrastructure and development tools and abstracts many of these components' underlying complexity. An ID comprises integrated tools and services designed to enable development teams to build, deploy, and manage applications more efficiently.

Essentially, an IDP provides a unified interface for developers to interact with the entire software lifecycle, from code to production, while automating many routine tasks, enforcing best practices, and ensuring a consistent experience across an organization and its different environments.

Internal developer platforms are built to improve developer productivity, enhance collaboration, and ensure consistency across development environments. They offer a self-service portal where developers can access resources, tools, and services required for daily tasks without relying heavily on operations teams. Ultimately, the goal is to create "golden paths" for developers, helping them achieve tasks following best practices, automating processes, and ensuring quality.

How does the internal developer platform work? The main components of an IDP

In this section, we will discuss the key components and characteristics of internal developer platforms.

1. Self-service developer portal

An integral part of an IDP is a user-friendly interface that enables developer self-service and allows them to provision resources, access tools, and manage their applications with minimal friction. A developer portal is where developers manage deployments, environments, and application configuration.

This user interface usually exposes a service catalog, a detailed view of all the available services within the development environments. Service catalogs help with standardization by providing templates and abstractions on top of applications and infrastructure.

2. Automated workflows and integration with existing systems

IDPs leverage automation to streamline code testing, building, deployment, and monitoring processes. Adopting an IDP ensures consistency and reduces the risk of human error.

IDPs don't exist in isolation and must integrate with various existing tooling and systems. Working on integrations and automation becomes important when IDPs need to be created on top of an existing setup.

3. Infrastructure management

An IDP integrates with existing Continuous Integration (CI) pipelines on the one side and cloud or on-premise infrastructure on the other side. IDPs reduce manual effort and minimize errors in managing the underlying application infrastructures.

Part of the solution is an infrastructure as code (IaC) tool such as Terraform, OpenTofu, Pulumi, or Crossplane. The IDP provides templates for developers to create high-level definitions of the infrastructure they need, and the infrastructure orchestration behind the scenes is responsible for producing the IaC manifests and deploying the necessary changes.

4. Environment management and application configuration

IDPs promote standardized development, testing, and production environments, ensuring consistency and reducing compatibility issues across teams and projects. Even more, it makes provisioning new environments quicker and removes bottlenecks with self-service and on-demand models.

5. Deployment management and monitoring

An integral part of such a solution is connecting with continuous delivery (CD) pipelines to automate testing and deployment processes. Automated new code deployments are moved from testing to staging environments automatically, and the necessary tests are performed. Since IDPs integrate with other systems, triggering other workflows or communication with other systems is common during and after deployments.

Lastly, IDPs offer a unified monitoring view and provide debugging information, such as logs for deployments and applications, in case of errors. When failures are encountered during deployments, the IDP usually provides options for automated rollbacks and mitigations.

Who uses the IDP?

Typically, the main users of an internal developer platform are platform engineers and developers. Depending on your organization, other kinds of users might be involved in either building or using the platform.

Platform engineers - Platform engineers are responsible for building, configuring, and maintaining the IDP. They own the platform and develop and manage it as their product. They define templates, write abstractions, define golden paths, and set guardrails and sane defaults.
Application developers - Application developers typically write the code in their integrated developer environment (IDE) and follow git-push workflows to integrate their code. IDPs integrate into code repositories and CI/CD pipelines but add further automation. The developer experience is enhanced through an IDP since they can request resources, spin up prepared environments, roll back, and deploy all from one place.
Specialized teams - Specialized teams, such as security experts, are involved in building a robust platform according to requirements. These teams actively collaborate with the platform engineering teams to incorporate best practices, offer guidance, and integrate their specific tooling into the platform ecosystem.

Benefits of an IDP

IDPs offer numerous benefits that enhance productivity, collaboration, and overall efficiency within organizations. Here are the key advantages of implementing an IDP:

1. Increased developer productivity

By automating repetitive tasks and providing self-service tools and capabilities, IDPs enable developers to focus on writing code and delivering value and less on managing infrastructure and processes. IDPs allow developers to own the application's lifecycle end to end, as they are now responsible for development, configuration, deployment, monitoring, and rollback without having to be experts in each domain.

2. Improved consistency, reliability, and compliance

IDPs promote adherence to organizational standards, coding repository guidelines, documentation styles, and security policies, ensuring consistency. Standardized environments and reusable components enhance reliability and compliance against frameworks and policies, making them easier to track.

3. Enhanced collaboration and knowledge sharing

IDPs foster collaboration among development teams by providing a centralized platform for sharing resources, code, docs, and best practices. This centralized place of tools and services fosters a sense of shared responsibility across an organization. Usually, gamification elements are added to IDPs that promote knowledge sharing and a culture of continuous improvement.

💡 You might also like:

Successful IDP implementations and tools

In this section, we will look at some examples of organizations that built successful platforms and some off-the-shelf solutions you buy or adopt to help you get there.

Example 1: Spotify

Known for its Backstage, Spotify has created and open-sourced a developer portal solution that streamlines development workflows and centralizes various development tools. That tool was initially built internally to meet Spotify's own needs in terms of a platform and was later shared with the rest of the community.

Example 2: Airbnb

Another successful example is Airbnb's Airbnb's multilayered, data-driven approach to optimizing developer satisfaction and productivity. Airbnb built a modern platform that manages environments at scale, enables remote on-demand developer environments, provides a unified build process, and incorporates developers' IDEs and cloud assets, aiming at a better developer experience.

Example 3: Netflix

Netflix is another well-known example of unifying engineering and development experience with its platform. They achieved this by building a federated platform console as the single place for all the tools engineers need to develop and deploy software at scale. It combines developers' services and tools into a single, easy-to-use interface.

Off-the-shelf IDP solutions

Next, let's review a few tools and solutions that can assist you build a robust end-to-end internal developer platform.

1. Spacelift

Spacelift is a sophisticated orchestration platform for your IaC tools including OpenTofu, Terraform, Terragrunt, CloudFormation, Pulumi, Kubernetes, and Ansible. With our solution, you can fully automate your IaC processes and set up safe self-service developer access - critical objectives for platform engineers.

Spacelift offers everything you need to create successful internal platforms. Your jobs run as isolated stacks that feature support for complex dependencies, configurable blueprint templates, and automated drift detection and remediation. Spacelift also offers a robust RBAC system and policy-based access control to prevent developers from applying unsafe actions to your resources.

2. Humanitec

Humanitec is one of the leading solutions enabling platform engineers to create IDPs for enterprises. Their primary products, the Humanitec Platform Orchestrator, Score, and Humanitec Portal, are utilized by mid-sized and large engineering organizations.

The Platform Orchestrator dynamically generates application and infrastructure configurations with each new deployment, promoting standardization throughout the software delivery lifecycle. Meanwhile, the Humanitec Portal offers a unified view of the entire organization.

3. Qovery

Qovery is an IDP solution that streamlines the path to production for developers by reducing noise and providing paved paths. It supports testing and ephemeral environments, driving actions to enhance software quality. Qovery focuses on delivering a platform to developers quickly.

Platform Engineering teams can customize and build on Qovery to create their ideal workflows, maintaining control and auditing developer activities. Qovery operates on Kubernetes and offers a comprehensive feature set.

4. Mia Platform

Mia Platform is a comprehensive toolkit combining a few products for platform design and development. This internal developer platform toolkit consolidates tools for developers, enhancing software templating, improving the developer experience, preparing optimal workflows, streamlining software delivery, and facilitating internal and external knowledge sharing.

5. Backstage

Backstage is an open-source framework designed to help organizations create developer portals for managing software development. Initially developed by Spotify as an internal tool to oversee over 14,000 software components, Backstage was open-sourced in 2020, donated to Cloud Native Computing Foundation (CNFC), and has since been adopted by thousands of companies. Key features of Backstage include; software catalog, software templates, tech docs, plugins.

6. Port

Port is a platform for building no-code, comprehensive internal developer portals. Port's software catalog encompasses microservices, resources, and custom assets and supports any data model featuring in-context maturity scorecards. Its portals facilitate any developer's self-service action and workflow automation. Blueprints, or custom entity definitions, are central to Port.

The software catalog is a single source of truth, always current and usable in workflows, automation, and CI/CD processes. Core features of Port include a software and infrastructure catalog, self-service capabilities, and governance.

7. Kratix

Kratix is an open-source framework designed to enable platform engineers to create flexible and customized internal platforms. It establishes a contract between application and platform teams. Leveraging the GitOps workflow and Kubernetes-native constructs,

Kratix allows platform teams to provide the self-service capabilities developers require while maintaining an up-to-date, secure, and relevant platform. Kratix prioritizes the platform engineer's experience, enabling platform teams to develop an API-driven, bespoke platform that meets specific organizational needs and can adapt as those needs evolve.

What is the difference between DevOps Platforms and IDP?

Now that we have a good understanding of what an IDP is and its main components, let's compare it to DevOps Platforms.

Platform engineers develop internal developer platforms and enable golden paths for developers. Platform engineering teams work towards building a product that satisfies the needs of developers in large enterprise setups. These IDPs typically require extensive configuration, are flexible, and can integrate with a variety of systems. With this high degree of customizability, IDPs require a high level of maintenance, a lot of engineering effort, and a product mindset to provide value to an organization.

On the other hand, end-to-end DevOps platform solutions typically also include all the infrastructure out of the box in a simplified or abstracted manner (e.g., Heroku). Although such DevOps Platforms can greatly help get started and iterate quickly, they often don't provide enough customizability options that an enterprise or a sophisticated setup requires. They are a great option when using off-the-shelf infrastructure, and tooling is an option but might become problematic at scale in terms of cost, optimization, and customizability.

While both DevOps platforms and internal developer platforms aim to improve software development and delivery, they do so in different ways. DevOps platforms provide comprehensive tools for the entire DevOps lifecycle, while IDPs focus on simplifying and streamlining the developer's experience by abstracting away the underlying infrastructure complexities.

Challenges and considerations when building an IDP

In this section, we will discuss some considerations and challenges that you should consider before starting to build or design an IDP for your organization.

Scale and size of your organization

The value of a standardized and federated platform becomes more important as the size of an organization grows. With hundreds of developers using such a system, every small efficiency and time gain compounds the overall usefulness of the platform. Although there isn't a golden rule regarding an organization's size and the need for an IDP, typically, IDPs are implemented mainly through enterprises with hundreds of engineers. If you have a small set of developers, the engineering effort to build an IDP might not be worth it.

Microservices & complex application ecosystem

Another thing to consider is the complexity of your applications and infrastructure ecosystem. If you are only developing monolithic applications with well-defined and easily reasoned components, an IDP might not provide much value. On the other hand, if you operate a complex, distributed microservices architecture with multiple components, standardizing based on an IDP might be a good solution.

Developer productivity considerations

The main objective of such a platform is to make the developers' lives easier and their workflows more efficient. An IDP is probably a good fit if your developers are blocked in their work by dependencies on other colleagues or fragmented user experiences and scattered tools slow them down. Alternatively, if your developers are happy with their current workflows, are empowered, and are efficient with their current setup, an IDP might be a waste of time.

Advanced data privacy or security requirements

If your organization has advanced data privacy or security requirements, an IDP could be an effective way to set up guardrails, enforce security rules and policies, and monitor compliance adherence.

Multicloud & hybridcloud

If your organization is using or looking into multicloud or hybrid cloud setups, an IDP could become the abstraction layer and the one-stop-shop for your developers to operate and manage applications that span cloud providers and on-premise infrastructure. On the other hand, if your infrastructure is highly homogenous, leveraging an out-of-the-box cloud providers' platform offer could be sufficient.

Best practices for implementing an IDP

Here, you will find best practices to keep in mind while scoping, preparing, and building an IDP for your organization.

1. Define clear objectives, scope, and success criteria

To validate the success and evolution of your platform, it is critical to define and document the challenges you are trying to solve. Test your hypotheses and define clear metrics, objectives, and scope for your IDP.

Key performance indicators (KPIs) related to the IDP's efficiency and success include deployment frequency, mean time to recovery, lead time, active users, platform uptime, time to build and deploy an app from scratch, and security incidents. Defining and following such metrics will enable continuous improvement and course correction efforts.

2. Start small and iterate

As discussed, IDPs are usually complex setups that involve coordination and communication between various systems. Therefore, it is paramount to start with a minimal viable platform (MVP), validate your hypothesis that this is a useful endeavor for your organization, and incrementally add features and capabilities based on feedback and evolving needs.

Remember that building and maintaining an IDP requires a significant investment of time and effort, especially when IDPs support complex workflows and use cases.

3. Treat IDP as a product and involve the developers

Most successful IDP implementations follow the platform-as-a-product approach. Since this tool is mainly targeted at developers and engineers, continuously incorporating their feedback and having product management processes in place is the only way to ensure long-term success.

4. Start by automating manual processes

As your IDP hides a lot of the underlying complexity from end users and orchestrates different systems masterfully, automating manual and cumbersome processes is a prerequisite.

Before starting to build your IDP, gather all your current manual and laborious processes and build systems to automate them. This will allow your IDP to operate on top of these automations, usually via APIs.

5. Select appropriate tooling

In order to build and customize an IDP according to your organizational and business needs, you can either build customer components, buy specific tools and solutions, or adopt open-source software and customize it accordingly.

Typically, platform engineering teams leverage a combination of all these to craft a solution that makes sense for them. Areas of tooling and solutions that you should investigate are platform orchestrators, developer portals, CI, monitoring, security, infrastructure as code, and deployment operators.

6. Don't sleep on security and governance

It's tempting to focus mostly on developer productivity gains and all the fancy features, but providing self-service access to infrastructure and services requires robust security measures to prevent misuse and breaches. The platform team, in collaboration with security teams, is responsible for defining specific roles, guardrails, guidelines, and policies to ensure proper oversight and management of the IDP.

See: What Makes Spacelift Secure?

7. Focus on documentation and upskilling your teams

Adopting and using such a solution will be a new experience for your end users, and investing time in proper documentation and upskilling your teams will become crucial to a platform's success.

Successful IDP implementations emphasize user onboarding and ensuring they are supported throughout their journey using the platform. Having mechanisms to provide feedback and enabling end users to contribute back to the platform with knowledge sharing and contribution incentives keeps user satisfaction and productivity high.

How can Spacelift help you build your Internal Developer Platform?

Spacelift lets you manage your IaC at scale by implementing robust CI/CD across cloud providers for your infrastructure, enabling developer autonomy. As Spacelift supports various IaC tools, such as Terraform, OpenTofu, Ansilbe, and Pulumi, you can standardize your infrastructure management for your IDP for multi-iac workflows. Even more, Spacelift provides an overview and clear visibility into your infrastructure resources and allows the enforcement of policies and guardrails.

You can use Spacelift as the foundation layer of your IDP by creating different stacks to fulfill your development functions. Spacelift stacks encapsulate your source code, infrastructure state, and deployment configuration. Stacks can be queued, triggered, canceled, and inspected within the Spacelift UI, allowing you to check the health of your infrastructure at a glance.

Use stack dependencies to easily configure your complex infrastructure needs and share outputs between dependent stacks. Other components, such as Blueprints, offer more options to simplify self-service provisioning operations.

If you want to use a product that greatly enhances the lives of your platform team members, create a free account with Spacelift today, or book a demo with one of our engineers.

Key points

In this blog post, we have explored the platform engineering concept of internal developer platforms and their usage. We reviewed the main components of an IDP, the key personas involved in either building or leveraging the platform, and successful platform implementations. We looked into specific tools to help you develop your own platforms to improve the software development process while discussing challenges, considerations, and best practices to get you there.

Written by Ioannis Moustakis