Setting up TWS & IBC on EC2 instance

Probably if you are a developer who was always interested in the stock market or any other type of investing, you have at least once thought about creating some sort of system that will help you get the market data and then use it for analysis

Especially these days when the stock market is crashing, the Japanese yen and British pound are hitting record lows against the dollar, etc.

Probably this is you when you open your portfolio

And because you definitely do not want to be in such a situation, you made a research on what kind of options you have to get the market data and found something called IB Trader Workstation (or TWS)

You start digging more and turns out that this TWS is actually exactly what you need - it gives you access not only to historical data but real-time too. Moreover, you get access to analysis tools, interactive, customizable charts, you can execute orders, etc.

Sounds quite interesting, right?
So, you go and install it on your local machine, play with it for a week as you absolutely loved it

Then you finally want to start coding and you stumble into TWS API which is a simple interface to TWS through which you can do a lot of stuff - from automating your trading strategies & requesting market data to monitoring your account balance and portfolio in real-time.

And the main requirement for this API to work is a running instance of Trader Workstation app (or IB Gateway, but we are not here to talk about it)

As TWS been already installed on your machine, you start writing code using TWS API - either its native version or any third-party libraries such as ib_insync.

You test your code and it works perfectly fine!

But then you start thinking that running TWS on your local machine is not a long-term solution and you decide to switch everything to the cloud

You think about which cloud platform you should use and decide to go with AWS, cause Jeff Bezos is the best

The first thing you have to do is launch the EC2 instance which we will use to run our Trader Workstation app

We are going to use Ubuntu as our AMI and for the instance type we will choose t2.medium (cause if you choose a less powerful instance type, TWS will work a little bit slow)

I suppose now you have a running EC2 Ubuntu instance with the instance type mentioned above and now let's start working on the installation of TWS on this EC2 machine

First, let's connect to your EC2 instance.
One of the easiest ways is through using SSH (you also need to have access to the file with the key pair that you used when creating the EC2 instance)

Choose your EC2 instance and click on the "Connect" button

From here, choose "SSH client" and just copy the example code

Then, use the terminal of your choice, and run the copied command (make sure that you run this command from the same folder where your key file is located)

You successfully connected to your EC2 and now let's run the following general commands

sudo apt update
sudo apt -y upgrade

After that, we have to install 2 additional things: XVFB and VNC server

XVFB - is an X server that can run on machines with no display hardware and no physical input devices. It emulates a dumb framebuffer using virtual memory.
In simple words, we will use it to add GUI to our EC2 instance.

VNC - is basically a screen-sharing system that is used to remotely control another computer.
It works on a client/server model which means that we install the VNC server on the remote computer - the one we want to control (in our case the EC2 instance) and on our local machine we install the VNC Client that we use to control another computer

To install XVFB run this command:

sudo apt install -y xvfb

As our VNC server, we will use x11vnc which allows us to view remotely and interact with real X displays (i.e. a display corresponding to a physical monitor, keyboard, and mouse) with any VNC viewer.

sudo apt install -y x11vnc

Then, run this command to start virtual frame buffer

/usr/bin/Xvfb :0 -ac -screen 0 1024x768x24 &

When you run this command, you should get the ID of the Xvfb background process

To start the VNC server, we will run this command where we specify 2 passwords - full access (keyboard + mouse) and second is view only.

/usr/bin/x11vnc -ncache 10 -ncache_cr -viewpasswd view_only_password -passwd full_access_password -display :0 -forever -shared -bg -noipv6

Also, we define what server display it should connect to (in our case, it is :0).
noipv6 option states that it should not try to use IPv6 for any listening or connecting sockets.
The shared option allows more than one viewer to connect at the same time and the forever option tells our VNC server to keep listening for more connections rather than exiting as soon as the first client disconnects

Now VNC server is listening on port 5900 and we can connect to it using any VNC client you want

I am going to use TightVNC but there are many other options such as TigerVNC, UltraVNC, etc.

Before connecting to our server, we have to make a few adjustments to the security group assigned to our EC2 instance.
Basically, we need to allow inbound traffic to port 5900 and we can easily do this by going to the AWS console, choosing our instance's security group and editing its inbound rules

Here we add a new rule of type Custom TCP and define our IP as a source

After that, we connect to the VNC Server using TightVNC Viewer.
In the hostname field, we enter the public DNS of our EC2 instance

And we use the full access password that you entered when starting the VNC Server

After connecting to our VNC server, let's make our life a little bit easier and install metacity - a lightweight window manager.

sudo apt install -y metacity

And run this command:

DISPLAY=:0 metacity &

Now we are ready to install the Trader Workstation app.
First, let's install offline TWS for Linux on our local machine

Offline TWS for Linux

After installation is complete, we will copy this file to our EC2 instance.

In order to do that, we will use the scp CLI tool that allows you to securely copy files and directories between two locations

scp -i /path_to_key_file tws-latest-standalone-linux-x64.sh ubuntu@public-ip-address-of-ec2-instance:

Do not forget to add ":" at the end of a public IP address

This command should copy the installation file from our local machine to the EC2 instance (to the home directory)

We can check this by running the ls command:

Now let's make this file executable

chmod a+x ~/tws-latest-standalone-linux-x64.sh

And as we have Xvfb installed, we can run the TWS installer on display ":0"

DISPLAY=:0 ~/tws-latest-standalone-linux-x64.sh

In TightVNC Viewer, you can see that the installation window is opened

Follow the steps to complete the TWS installation.
When you finish installation, you can run the ls command

As you can see, the Jts folder was created which contains all TWS-required files

Now you just have to use TWS credentials to log in and configure TWS as you want

And now you have the Trader Workstation app hosted on an EC2 instance!

So, let's say you've been using it for a few days and noticed that sometimes your TWS loses connection and asks for manual authentication
And it is definitely not the way you wanted it to work

Wouldn't it be perfect if there were a tool that can run TWS in 'hands-free' mode (without any manual intervention), right?

I agree that it would be nice to have such a tool and fortunately for us, there is one called IBC

You can follow the link above and read more about what IBC can do, but for our purpose, we will use it to start the TWS app and login automatically in case our TWS loses connection

First, we need to install it by running this command:

wget https://github.com/IbcAlpha/IBC/releases/download/3.14.0/IBCLinux-3.14.0.zip

Then, let's install unzip command (if you don't have it installed already)

sudo apt install -y unzip

Before extracting the contents of the downloaded ZIP file, I want you to create a new folder - /opt/ibc/

We do this because several IBC script files assume the default paths used for particular files (config file, TWS & IBC program files) and by using these default paths we will minimize customizing the configuration file and the shell scripts.

mkdir -p opt/ibc/
cd opt/ibc

Now let's unzip a file

unzip ../../IBCLinux-3.14.0.zip

Run these 2 commands to make sure all .sh files are executable:

chmod u+x *.sh
chmod u+x scripts/*.sh

Let's go back to the home directory and create another new folder - ibc where we will store the configuration file

cd ../..
mkdir ibc

Copy configuration file from /opt/ibc to the newly created folder

cp opt/ibc/config.ini ~/ibc/

Now we have to make a few changes to that config file (use the editor of your choice):

vi ibc/config.ini

Here we have to change the username & password fields

Save the changes and let's go back to the /opt/ibc/ folder
Here open the twsstart.sh file using an editor of your choice

cd opt/ibc/
vi twsstart.sh

For the trading mode, specify the one you need - either live or paper (I will use live)

And also, we need to change the TWS version.
We can find the TWS major version number by running the TWS app, clicking the Help button on the top navigation menu, then About Trader Workstation option

In the displayed information you'll see a TWS version

Going back to twsstart.sh file, modify the TWS_MAJOR_VRSN property
And for the IBC_PATH parameter add ~ before /opt/ibc

Save the changes and let's try to start IBC!
Before doing this, make sure that the previously started TWS is no longer running

DISPLAY=:0 ~/opt/ibc/twsstart.sh

After running this command, you should see TWS been started in VNC Viewer

And if you want to make IBC run every hour to check whether TWS lost connection and requires re-login, we can use crontab and schedule a cronjob that will run this command (DISPLAY=:0 ~/opt/ibc/twsstart.sh) every hour

Run this command:

crontab -e

And add the following line to the end of the file (you can adjust the time according to your needs)

0 * * * * DISPLAY=:0 /home/ubuntu/opt/ibc/twsstart.sh

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, I want to mention that I used this amazing article as a reference and my post is just a shorter and easier version of that article

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)