I've built a Cypress test runner that completes all tests (no matter how many you've got) in less than a minute. In contrast, based on my experience, a medium-size Cypress test suite takes about 15 minutes to complete.
The only catch is that individual tests have to be under 30 seconds.
The idea came out of frustration running Cypress in my previous company, where it was taking 10 minutes+ to run all Cypress tests while parallelizing across 30 VMs and costing us in excess of USD 2k/month.
In order to achieve this, I have effectively built a new Cypress test runner from the ground up. It understands Cypress syntax, but otherwise have nothing in common with how Cypress works. The way it achieves this performance is by splitting each spec into individual tests and starting all of the tests at once, i.e. if you have 10 specs with 5 tests each, this program will start 50 VMs to run your tests.
I have two companies trialing this at the moment and the feedback has been incredibly positive, saying that it is saving hundreds of engineering hours.
I am trying to establish how to price this. The challenge is that this model is profitable at scale, but losing money if there is not high density of clients. This is because it costs me USD ~0.15 to run a VM for 1 hour and I need to spin up enough VMs to complete all tests, and I am charged in increments of an hour.
My thinking is to charge 2 cents per a test-minute, i.e. Using previous example of 10 specs with 5 tests each, it would cost USD 1 to run all tests once. If you run integration tests 100 times per day, that's USD 100/day.
This may sound much. However, if prior to using this you were waiting 15 minutes to run all tests, that is 14 minutes saved. If avg. engineer in your company is earning USD 60/hour, you are saving USD 14 by having engineers get immediate results rather than waiting for them. If positioned that way, it doesn't sound expensive.
I am currently targeting companies with 30-50 engineers (existing customers are series A and series B companies). At this size, they don't have crazy amount of tests, so I can deliver on the 1 minute promise, and they care a lot about moving fast.
What sounds reasonable?
Top comments (0)