Comparing CI/CD Tools

This article was originally published on the Shipyard Blog

Continuous integration (CI) pipelines, when used as intended, are something you’re triggering and running multiple times per day. Your org’s pipelines can contain hundreds (if not thousands) of lines of code. It won’t come as a surprise that you have a lot at stake when choosing a CI provider — migrating CI providers is expensive and all around messy. With the right research, you can get it right the first time.

What should I look for in a CI tool?

Every tool in this article is obviously great in its own regard (after all, that’s why they’re featured!), but picking the right one for your org takes a little more than just spinning the wheel and choosing whichever one it lands on. Here are the a few things that could make or break your decision.

Hosting options

This is arguably the most important consideration when it comes to choosing a tool. You’ll typically see tools fall into three main hosting models: self-hosted, cloud-hosted, and hybrid. If you’re starting from scratch, cloud-hosted runners are usually managed by your CI provider and require dramatically less maintenance. Hybrid or self-hosted are best for orgs who have a bit more time and effort to invest into this, because when done right, these options will be more cost-effective and secure. They also enable you to get a closer approximation of your infrastructure than a managed (cloud) option.

Source code management provider

The major source code management (SCM) providers all offer their own integrated CI platforms. There’s definitely a benefit here: using the same interface for source control and continuous integration not only makes logical sense (especially since CI events should occur on particular pushes/code changes) but also reduces context switching during the inner loop, where maintaining flow really matters.

Our Favorite CI Tools

Every engineering team has a strong opinion and an obvious favorite when it comes to CI tools. We studied the features, developer experience, and use cases for the most appreciated tools in the industry. This is intended to serve as a high-level guide for narrowing down your CI provider search — the best next step is reading the docs and testing out each offering’s free tier.

CircleCI

CircleCI is a flexible, SCM-agnostic CI provider. In 2024, it’s hugely popular for enterprise orgs for its flexible hosting/compute options and high-quality integrations. CircleCI is considerably stable and reliable. Performance-wise, it’s on par with GitHub Actions and GitLab CI. Users have a huge selection when it comes to pipeline modules, called “orbs”, and have means to create and maintain their own.

Who it’s for: Enterprises that want a well-supported, customizable CI solution decoupled from their SCM.

Hosting model: CircleCI offers self and cloud-hosted runners. It uses usage-based billing.

Pipeline format: Workflows are YAML-based, and users can publish and maintain their own orbs or choose from 3,500 existing orbs.

GitHub Actions

As far as CI tools go, GitHub Actions is the easiest to get started with, especially if you’re already using GitHub. The GitHub Actions developer experience is exceptional: there are countless existing sample workflows, and thousands of official and unofficial Actions that you can piece together to build your workflow(s). It’s entirely accessible from your repository’s dashboard, so you can track CI runs right after making major code changes. GitHub also makes it easy to write, host, publish, and maintain your own custom Actions.

Who it’s for: Teams who use GitHub for source control and want an easy solution with a wide selection of pre-written Actions.

Hosting model: GitHub Actions uses hybrid and cloud-hosted runners and is billed by usage.

Pipeline format: Workflows are written in YAML, using reusable Actions.

Jenkins

Jenkins has been the industry-standard in CI for over a decade. It’s open source, and categorized as an “automation server”, which suits it to use cases beyond CI alone. Jenkins takes more setup and configuration than most alternatives, but since it’s been around for so long, it has an impressively strong community and external knowledge base to solve any and all intricacies. Unfortunately, with so many great CI providers emerging, Jenkins is falling out of favor, and orgs are migrating off of it.

Who it’s for: Large enterprise teams with in-house Jenkins experts.

Hosting model: Jenkins is typically self-hosted, but there are third-party providers out there for Jenkins cloud hosting.

Pipeline format: Workflows are defined with Apache Groovy in a Jenkinsfile. Jenkins offers over 1,900 community plugins.

Buildkite

Buildkite is the CI provider that the Ubers, Doordashes, and Pinterests of the world use to push, build, and test code. It was initially designed as a cloud-agnostic, self-hosted tool for companies at scale. With its hybrid model, your organization will host the agent, and you’ll have access to a cloud-based dashboard. A standout feature of Buildkite is its dynamic pipelines, which enable you to generate custom pipeline specs conditionally during your pipeline’s execution.

Who it’s for: Enterprises who have outgrown CircleCI, GitHub Actions, and GitLab CI and want a CI provider that can scale with them.

Hosting model: Agents are self-hosted with no usage limits. Dashboard lives on the Buildkite cloud. Billing is per-seat.

Pipeline format: Traditional YAML workflows. Support for dynamic pipelines.

GitLab CI

GitLab CI is a widely-used CI provider for GitLab-based orgs. It’s generally considered more feature-rich than alternatives. Interface-wise, it’s similar to GitHub Actions — you’ll be configuring and viewing your CI runs from your repository dashboard. GitLab CI is known for being highly-customizable for more complex CI tasks, but still has thorough documentation, so onboarding shouldn’t be a huge lift. You can create and maintain reusable modules for your pipelines by creating a “component project” on GitLab, or use existing components from the CI/CD Catalog.

Who it’s for: Orgs who are in the GitLab ecosystem who want a full-featured CI tool that they can use for the most complex of pipelines.

Hosting model: GitLab CI offers self-hosted and cloud-hosted runners, with usage-based billing.

Pipeline format: YAML workflows with public or custom components.

Harness

Harness is a complete CI/CD platform -- it goes above and beyond the standard CI toolkit, offering detailed deployment metrics, visual pipeline editors, governance tools, and test insights. It's designed for larger teams who want a plug-and-play, self-service solution, which in turn means it runs somewhat costlier than many alternatives. Harness is notably fast at building. It has a wide tooling ecosystem, with a recently-introduced SCM, and even an IDP solution.

Who it’s for: Larger, enterprise org who are ready to buy into a CI solution that benefits from the rest of the Harness ecosystem.

Hosting model: Harness offers managed, cloud-hosted runners, and some plans with self-hosted options. It is billed per seat.

Pipeline format: Workflows can be edited in the Harness Pipeline Editor visually or in YAML.

Conclusion

When you look at the CI tool landscape, you’ll notice one thing pretty quickly: most offerings are rather similar to each other. Pricing and performance are two metrics that tend to drive decision making, however it might be more important to consider a few other things, such as the size of your engineering org, the hosting model, user support offerings, and integrations with your SCM/other tools.

Need to unblock your CI with testing environments? Good news: Shipyard integrates with all CI providers. Talk to us today.