P.S. Pardon my viral title, I'm trying to test out if viral titles on DEV have an effect on stats (-/+), it might be a separate post later 😄
Let's face it; you are not exceptional. In all likelihood, you will perform worse than long-term investors. You will trade more often than them, and you will be learning. The bottom line, doing algo-trading, you will lose most of the time. After a couple of years (in the best-case scenario), you might see positive yearly returns in areas where the markets are not efficient. Those inefficiencies will be temporary and will have limited capacity or high risks. If not, Big Corp would already be sucking all alpha out of it.
Now if you accept the things above and you still want to go into it, the name of your game will be preserving the capital. Cloud costs money and time. Money is pretty obvious. Time is something less so.
Money in cloud
The first step, you would need to have OHCL data pulled daily. A higher resolution is better since you can always resample to a lower resolution and not the other way around. At this point, you would probably get a data provider for 1min OHCL, which would cost you anything from 20$-90$ USD per month. For that much money, you would be able to pull daily or the last 30 days of OHCL with 1min resolution.
You would still have a rate limit. With a rate limit of 300 requests per minute will take you 23 min to pull 7000 stocks or 10 min to pull 3000. Now, how much would it cost you to run an EC2 machine for 23 min a day? (30*23) / 60 * 0.0255 = 0.29325 so it will be 30 cents per month to host this on your EC2 machine given that the EC2 machine runs daily for 23 minutes. This is the cheapest Vcpu: 1 and 2 GiB machine. Now, you would need to figure out how to provision EC2 machine with your pulling logic and figure out how to trigger it on schedule. So that's extra couple of cents per year. Now, AWS prices are not set in stone and could increase so will your budget. All of this data would need to be written to somewhere cheapest and easiest in EC2 is S3. You would need to read this data onto you machine for analysis which is an extra $+ for you. What if you make a mistake and you need to know that the job has failed? That is another expense +.
So what costs money for us so far: data vendor + EC2 + S3 + monitoring + management time. I added management time since it's the time that you are not spending working on your trading strategy. Now let's say you have found a strategy that requires data to be scraped from a browser or vendor has to be pulled 8h per day. Now your AWS costs will start growing more than your data vendor costs. There will be a point when the money you spend for your infra will outgrow the possible edge you can find.
Now let's talk about self-hosting:
This is also not rosy; it will require a higher starting cost but will be kept flat for far longer than your cloud solution. To get started, you need hardware if you have hardware lying around, this is all fine for you. You can set up a Linux machine with ssh access and deploy your fetcher/strategy. You could set up inexpensive monitoring with Cronitor or Discord bot so that you know when the bot fails. The downsides to this approach are that you need to operate your own server and keep it up and running. In my experience, if you keep your server simple, do not run k8s or docker and just deploy bare metal. It will only fail if there is an electricity outage. It should not be a problem initially since the costs you save will make up for it. If you know Rust or C++ you can compile your fetcher to your hardware. Keep it simple, no abstractions, just fetching and writing to storage. All of your data operations will be done on your main machine using Python during the exploration phase.
So what costs money for us so far: data vendor + initial hardware. Note, initial hardware and scaling will be the same cost and the more you will run it cheaper it will become.
When to move to the cloud?
When you start making more money than you would pay for cloud hosting. Moving to the cloud will not make you money automatically. Unless you rely on the speed of access and you want to position your server as close as possible to the exchange for some HFT stuff.
TLDR
In the early stages of algo-trading, you'll face losses; thus, maintaining a cost-effective setup for data handling and processing becomes vital to survival - you should consider transitioning to the cloud only when your trading earnings consistently exceed the costs associated with cloud services.
Top comments (0)