About
Ark invest ETF is a rising star on the stock market. If you subscribe to the ETF trading notifications, you could receive their trading information at the end of each day. But, I don't really want to get all their trading information. I just want to know their new holdings. So, I'd like to build a system to send me notifications only I care about.
Desgin
Build the app step by step
1. Create a Google Developer App
Follow this post to create a Gmail app on google cloud.
Note: choose Desktop Application when you create OAuth token
2. Create first Lambda to process email
First, let's create a script to get OAuth token for accessing the Gmail app. (client_secret.json is the file which we get from step 1)
import os
from googleapiclient import errors, discovery
from oauth2client import client, tools, file
SCOPES = ['https://mail.google.com/']
def get_credentials():
wd = os.getcwd()
# creates credentials with a refresh token
credential_path = os.path.join(wd,
'credentials_out.json')
store = file.Storage(credential_path)
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
return creds
if __name__ == '__main__':
get_credentials()
In the out file (credentials_out.json), we should be able to find client_id, client_secret, and refresh_token information.
Second, let's create build the lambda function to process email content. Find source code on github
import os
from googleapiclient import errors, discovery
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from email.mime.text import MIMEText
import base64
import re
import requests
from datetime import date, datetime
from pytz import timezone
import boto3
from oauth2client import client, tools, file
import httplib2
from bs4 import BeautifulSoup
import csv
# TODO
# 1. Validate the date of the email.
# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://mail.google.com/']
# ARK_INVEST_QUERY="from:ark@arkinvest.com subject:ARK Investment Management Trading Information"
ARK_INVEST_QUERY=os.environ['EMAIL_QUERY']
S3_BUCKET ="ark-fly"
OBJECT_KEY_PATTERN="dailytradingtrans/{today}-trading.csv"
SEND_NOTIFICATION_TO = "guojiayanc@gmail.com"
SENDER = "noreply@arkfly.com"
TEMP_TRADING_CSV_FILE="/tmp/trading.csv"
def main():
service = login()
try:
result = get_messages(service, 'me')
if 'messages' in result:
messageId = result['messages'][0]['id']
data = get_message('me', messageId, service)
print(data)
generate_csv(data)
today = get_date()
upload_to_s3(OBJECT_KEY_PATTERN.format(today=today))
delete_massage(service, 'me', messageId)
else:
print("No message found!")
except Exception as error:
# TODO Sender is not set correctly
print("ARK Fly processing failed " + str(error))
message = create_message(SENDER, SEND_NOTIFICATION_TO, "ARK Fly processing failed", str(error))
send_message(service, 'me', message)
def delete_massage(service, user_id, message_id):
try:
message = service.users().messages().delete(userId=user_id, id=message_id).execute()
except Exception as error:
raise error
def login():
try:
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
return discovery.build('gmail', 'v1', http=http, cache_discovery=False)
except Exception as error:
print("Log into gmail failed!" + str(error))
raise error
def get_credentials():
client_id = os.environ['GMAIL_CLIENT_ID']
client_secret = os.environ['GMAIL_CLIENT_SECRET']
refresh_token = os.environ['GMAIL_REFRESH_TOKEN']
credentials = client.GoogleCredentials(None,
client_id,
client_secret,
refresh_token,
None,
"https://accounts.google.com/o/oauth2/token",
'my-user-agent')
return credentials
def upload_to_s3(object_name):
client = boto3.client('s3')
try:
response = client.upload_file(TEMP_TRADING_CSV_FILE, S3_BUCKET, object_name)
except Exception as error:
raise Exception("faile to upload to s3! " + str(error))
def get_date():
tz = timezone('EST')
today = datetime.now(tz).strftime("%Y-%m-%d")
return today
def get_message(user_id, message_id, service):
try:
message = service.users().messages().get(userId=user_id, id=message_id).execute()
data = ''
print(message)
if 'parts' in message['payload']:
for part in message['payload']['parts']:
data += base64.urlsafe_b64decode(part['body']['data']).decode("utf-8") + "\n"
else:
data += base64.urlsafe_b64decode(message['payload']['body']['data']).decode("utf-8") + "\n"
return data
except Exception as error:
raise error
def generate_csv(data):
try:
bs=BeautifulSoup(data, 'html.parser')
table_body=bs.find('table')
rows = table_body.find_all('tr')
csv_rows = []
for row in rows:
cols=row.find_all('td')
cols=[x.text.strip().replace('\r\n', ' ') for x in cols]
csv_rows.append(cols)
with open(TEMP_TRADING_CSV_FILE, "w") as f:
wr = csv.writer(f)
wr.writerows(csv_rows)
except Exception as error:
raise Exception("Today's trading table not found!" + str(error))
def create_message(sender, to, subject, message_text):
message = MIMEText(message_text)
message['to'] = to
message['from'] = sender
message['subject'] = subject
raw_message = base64.urlsafe_b64encode(message.as_string().encode("utf-8"))
return {
'raw': raw_message.decode("utf-8")
}
def send_message(service, user_id, message):
try:
message = service.users().messages().send(userId=user_id, body=message).execute()
print('Message Id: %s' % message['id'])
return message
except Exception as error:
print('An error occurred: %s' % error)
raise error
def get_messages(service, user_id):
try:
return service.users().messages().list(userId=user_id, q=ARK_INVEST_QUERY).execute()
except Exception as error:
print('An error occurred: %s' % error)
raise error
# For local test
if __name__ == '__main__':
main()
def lambda_handler(event, context):
main()
return {
"status":200
}
Then, let's containerize the python code and push it to ecr on aws.
FROM amazon/aws-lambda-python:3
WORKDIR /var/task/
COPY . ./
RUN pip install -r requirements.txt
CMD [ "app.lambda_handler" ]
And build a simple script to push the docker image to ecr (change account id to your aws accountid)
#!/bin/sh
set -e
docker build . -t ark-go-email
docker tag ark-go-email:latest 120400168286.dkr.ecr.us-west-2.amazonaws.com/ark-go-email:latest
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 120400168286.dkr.ecr.us-west-2.amazonaws.com
docker push 120400168286.dkr.ecr.us-west-2.amazonaws.com/ark-go-email:latest
Next, create an AWS lambda from docker container image, check this post for how.
Note: we need to create 4 lambda environment variables.
EMAIL_QUERY=from:ARK Trading Desk <ark@ark-funds.com> subject:ARK Investment Management Trading Information is:unread newer_than:1d
GMAIL_CLIENT_ID=<client_id>
GMAIL_CLIENT_SECRET=<client_secret>
GMAIL_REFRESH_TOKEN=<refresh_token>
Last, create a cloud watch event to trigger the lambda on cron based.
3. Create Second Lambda to fetch ark holding information
Source code can be found here
All the steps should be the same as in section 2. But, the cron express is cron(0 4 ? * TUE-SAT *).
4. Create Third Lambda to process data generated by section 2 and 3
Source code can be found here
5. Create Fourth Lambda to send an email notification
Source code can be found here
6. Setup s3 event notifications
Choose the s3 bucket -> Properties -> Event notifications.
Note: In section 2, after it processes the email, it uploads a generated csv to s3. And it should trigger the third lambda in section 4. In section 4, if the lambda finds any new holds, it will upload the data s3, which will trigger the last lambda to send a notification. If no new holdings, do nothing. It will not trigger the last lambda.
Top comments (1)
This is a very generic document. Add a comment if you have have any questions. Happy to help. Build something almost free is really cool.