We all know that crypto mining is negatively impacting many things in the world. And now it's ruining something else in a way no one has seen coming. This is why mining crypto currencies is killing every free CI / CD platform.
Video
As usual, if you are a visual learner, or simply prefer to watch and listen instead of reading, here you have the video, which to be fair is much more complete than this post.
Link to the video: https://youtu.be/9TOJqJSHVvI
If you rather prefer reading, well... let's just continue :)
Intro
Today I have to talk about something I'd prefer not to, but unfortunately this is happening, and it's happening hard. So let's talk about crypto mining and its deleterious effect on free CI/CD platform.
A Work on Mining
We all know crypto mining, the process in which transactions for various forms of cryptocurrency are verified and added to the blockchain digital ledger, using the computing power of computers or graphics card, and for which miners are rewarded with crypto currencies directly.
We probably all know that this is affecting many aspects of our current time. For example, the current and past generation of graphics cards are so good and fast for mining that it's basically impossible to buy a graphics card right now, or if you find one the price is crazy. All the supply is basically taken up by miners, and few very lucky gamers.
How Does Mining Cryptocurrencies Affect CI Platforms
Ok, but how does this affect the CI/CD platforms? I'm glad you asked.
Due to the lack of availability of graphics cards, and the constantly increasing number of miners thanks to the rise in value of cryptocurrencies, miners have started trying to find alternative ways for mining.
They first started using Cloud services but quickly realized that the cost for always running large instances was higher than the gain they were able to get. And this is when they started looking at the free CI providers.
Hosted Build agents are fairly powerful, having to take care of compilation etc., and most platforms have a free tier, especially for public repositories. Powerful machines for free, a miner's dream come true.
And this is exactly the problem. They have started writing script, pushing them to public repositories, and take advantage of those free CI agents to run their mining software. And as the different providers started blocking those attempts, miners adapted and started writing fairly complex software and scripts to "mask" the real reasons why they were using the repos and CI agents.
An Example
There are countless examples, but here is one just to make you understand the gravity of the problem. There was a user on GitHub who created a simple repo, which seemed a legit one at a first look.
In the repo this user had the definition for 5 different CI providers, including GitHub Actions, CircleCI, TravisCI and others, and all were configured in automatic CI. The user had roughly 1 commit every hour, which in turn kicked off all 5 of those CI... and the script that was run was in fact a crypto miner. You can imagine how much resources that user alone has consumed.
The Effects
And in fact, if you have noticed your hosted CI agent being slower than usual or picking up jobs with a greater delay most likely it's because of this. And not only on free CI, but also on paid CI platforms... because the resources are the same. But if the problem was just some slowness, we wouldn't be here talking about this.
The problem is much bigger. So much so that basically all the CI providers have stopped offering free tiers or, in the best cases, they've implemented great limitations on the services.
Industry Reactions
Azure DevOps
Microsoft is not providing anymore free concurrent CI for their Azure Pipelines for new organizations. If the users want them, they need to request for them and provide additional information to verify they are eligible.
TravisCI
TravisCI is taking it a step further, completely removing the free tier, and giving to existing users a trial with an amount of free credits. When the credits are exhausted, if a user wants to keep using CI then they will have to buy a paid plan.
GitLab
GitLab, takes a different approach.
First, they require new users to verify their account adding a credit card to their account before they can start using the hosted CI agents. Existing users are not currently required to insert a credit card number, but they may be in future.
Second, they are removing the unlimited free minutes that were previously assigned to public projects, and setting a limit to 400 free minutes instead.
CircleCI
Circle CI has never had a completely free plan, but only a free grant of 2500 credits per month.
While they haven't change that, at least not yet, they 've published an article saying that they have a whole team, and I quote, "of security experts, operations engineers, data scientists, and developers whose ongoing work comprises spotting and eradicating abuse of our platform".
This of course is a huge cost for the company, and if things will continue like this they will need to find a way to get the money back... you make of this what you want.
GitHub Actions
Finally, GitHub Actions is the only provider that I'm aware of which has still a completely free unlimited use of their CI and has not changed that.
However, they did mention in a post on their public blog that the Actions teams have spent thousands of hours fighting against miners. As in the CircleCI case, this comes at a cost. Having engineering teams focusing on fighting miners most likely means they have less time to focus on improving and developing the service.
And they are also saying that they are rolling out features and improvements to help maintainer of Open Source projects having a better control of their CI when it comes to Pull Requests and Forks.
And I could continue for long, because similar things are happening from each and every CI provider.
The Solution?
Is there anything we can do to avoid this? Unfortunately, I'm afraid the answer is no.
Providers can do their best to enforce terms of service and take other measures, but as long as it's profitable and untraceable to make such attacks, miners will continue to become more sophisticated and circumvent measures.
The only hope is for crypto networks to fully disable the current computation-based mining as a way to earn new coins, switching entirely to a proof-of-stake (POS) validation model. It sounds impossible, but it is actually already happening. Ethereum in fact recently announced they will do exactly that.
Conclusions
Let me know in the comment section below what you think about this sensitive topic.
Like, share and follow me 🚀 for more content:
📽 YouTube
☕ Buy me a coffee
💖 Patreon
🌐 CoderDave.io Website
👕 Merch
👦🏻 Facebook page
🐱💻 GitHub
👲🏻 Twitter
👴🏻 LinkedIn
🔉 Podcast
Top comments (37)
This is insane.
I won't deny it was a genius discovery and clever tactic but horribly detrimental to those of us who rely on these free and excellent services daily for genuine use cases.
I like the free minutes approach rather than paywall or credits.
However I wouldn't mind GitHub requiring credit or debit card information to prevent abuse. Similar to how Firebase prevents abuse by requiring your card details. It works.
Not everyone has easy access to a credit card. Nothing Genius about it.
I agree with you. It’s a real problem, I hope as I’ve mention that they will find a solution that doesn’t require limiting others (aka us) for something abusers do, but rather changing the approach of the whole system
What about privacy issues?? I don't want to give my card info to Microsoft or any company for that matter.
I just pay for my CI. One of the few things that can be done.
This isn't always enough. If you have an open-source project that allows pull requests, and automatically runs tests against these, someone can weasel in there and inject a miner in that process.
They get the rewards, you get stuck with the tab.
The most insulting part of this is the amount gained by the attacker is usually a fraction of the cost to you. For each $1 you spend on CI services they might make a tenth of a cent, or in many cases even less. To make any amount of money at all they need to operate at a huge scale, which is why this problem is so bad.
There are other problems with letting the CI run on PRs without supervision. For example, someone could try to steal credentials being used in the build. I don't recommend doing this.
Letting strangers run arbitrary code in PRs with responsibility falling under the repository owner was always gunna turn out bad surely?
Seems like self-hosted might be the way to go in the really soon. 🤔
I don't like the idea of running and managing my own Jenkins server because a few bad actors are exploiting CI pipelines.
Its a good opportunity to learn more on devops for me. If you want something hands off you can use the AWS / DO / azure cloud plugin to create and destroy VMs on demand. If this isn't your thing you can always pay for the CI services themselves.
Personally, I'd recommend terraform or Ansible specifically for the creation of cloud resources. The Jenkins agent would be able to launch those using plug-ins.
Yes I create the master using terraform, but the agents are spun up on demand. Keeps the costs reasonably low without loosing out on performance.
Sure, but there are other scenarios in which users may not be able to do so… like for example for OSS projects
I pay for CI on my OSS projects.
You’re one of a kind 😉 jokes aside, there are many factors. It is always good have more choices and not being limited, especially if the limitations are in place because of abusers
Great article/video! I fully support the newer Proof-of-Stake mining method rather than the Proof-of-Work(PoW) mining like they do to current Bitcoin/Ethereum/Monero.
But as long as Bitcoin still living, the PoW will never ceased. At this rate only hope those CI/CD services can have a mechanism to ban all mining activities as soon as they detected it.
This would be just like dealing with hackers and cyber criminals. You create a game of cat and mouse between the exploiter and the security apparatus.
Thanks. The problem is that it’s almost impossible detecting it until it’s too late 😟
This is Bad.
I started doing open source projects before CI/CD and I know how painful it was to release software without CI/CD. It's just so painful. CI/CD is like a dream come true. These crypto monkeys are just shit. Why would they damage the very industry that brought these crypto currencies in the first place.
I think there is a way to prevent this by identifying which programs are invoked in CI and just block that user. Like is the program invoked is a mining software, that's pretty doable.
It’s not that easy. You can write your own mining software… how would a platform recognize that?
Well for most cases, miners only use popular software so it's easy to block most cases.
Thanks for the article. I learned something new and horrifying. Before, I was hoping that the Chinese govt and others were likely to wake up and charge these miners for their externalities. But now I see that they are just thieves stealing computing (and therefore also environmental) resources. I’m even more disgusted by the business.
Just now (June 14:th and 15:th), AWS CodeBuild had extended delays in execution for a number of accounts.
Also, a number of accounts in at least us-east-1 and eu-west-1 could (can not) not run CodeBuild due to AccountServiceLimitExceededException errors.
My guess is that AWS CodeBuild may have been hit by this kind of activity and that AWS blocked some account/organisation+region combinations, where they may have found suspicious execution patterns and perhaps somewhat new accounts.
Yes I’m pretty sure that’s the case… unfortunately 😟
To be fair with Travis CI, they were also bought, which is probably part of the reason to abandon the free tier.
But it is very inconvenient. Yesterday changed the CI from Travis to Github Actions, for a legitimate Open Source project, which on average maybe runs once a month. And before Travis became so slow the integration tests started to break.
Super interesting article! I didn't realise cryptomining was affecting so many services.
Thanks! Unfortunately it is 😕
Very interesting article!
Out of curiosity, do you know how cloud providers realized their platforms were being used for mining?
I’m not sure. If I have to guess, checking the network traffic to wallets/crypto accounts. But that would be “too late” because all the computation is already done 😩
THX YOU !!!!!
It’s so bad to devs that use this services in real projects and need to CI services. The future is pay or not use that.
Yes, it has a big impact, especially for small projects or devs that can't / won't afford to pay for CI.
What did we do to deserve this?
I know… it really sucks…
Hi Davide,
Excellent discussion.
I did not realize that the scope of this problem impacted CI/CD so strongly.
Keep up the good work!
Interesting, I have always wondered what's stopping them from abusing these free workflows, or some services that charges based on task executed rather than duration or compute power required.
As it turns out... not much :)
Some comments have been hidden by the post's author - find out more