Extending Traditional Software with Serverless Microservices

So we have “traditional” software running on an AWS ec2 instance. It’s awesome, but now we want new features added. Here’s some traditional approaches:

• If we purchased the software (or open source), we have to request these features and hope they make the next release, then we have to wait for the next release.
• If we developed the software using a traditional approach, we have to add the features and go through a rigorous release process, regression testing, etc.

Luckily we now have another option: Microservices. Great, but now I have to deal with deploying those: using containers, something like Kubernetes for orchestration, auto scaling, cost of more ec2 instances, etc. Eh...it seems like a hassle.

Luckily we new have yet another option: Serverless Microservices. We can do it quickly and without any hassle, and only get charged when we use them. Let’s dig in to a real scenario.

Scenario: We’re running enterprise search software on ec2. We can index data from pretty much any datasource with this software, so all of our enterprise content is easily searchable, including our S3 buckets where we put much of our content. This is great, but when something is added to S3 we want it to show up in our search results right away. The software doesn’t support “watching” S3 for changes. What do we do?

Thankfully the software has lots of REST apis, including an api to perform a “scan” on a repository and pick up the changes. Perfect! This makes it super easy! In a matter of minutes, we can add a Lambda Function that calls our REST api on our ec2 instance. S3 allows us to trigger our new Lambda Function when anything is added to our Bucket. Voila! We have our new S3 “watch” feature...and it’s a self contained Microservice. No traditional full release process/regression testing needed!

Let’s do this! This will only take a few minutes!

Let’s assume we’re already using AWS, our “traditional” software is already running on ec2, and it’s connected to an S3 bucket we already created. We’re just going to add our new “watch” feature using an AWS Lambda Function.

Step 1: Add a Lambda Function

Select - Services > Compute > Lambda

Click Create Function

Create your Function. We’re going to use Node.js because I love the idea of one language for the full stack, no language context switching, increased productivity, etc (but that is another story).

Before Create see the next screenshot to choose your Role.

Choose your Role and click “Create function”

1. Add a Trigger to your Lambda Function

1. Add your Javascript code to call the REST api.

This is performing a simple HTTP PUT to our ec2 instance running our “traditional” software which will Scan our S3 Repository for changes and Index only the changes so the new content is searchable.

Here we’ve hardcoded the http options for simplicity sake, but we could also easily make these environment variables.

See full code here (it’s only 20ish lines of code without the AWS handler wrapper function): https://gist.github.com/scottlepp/0a12873095336c9cc1f0fa296c3581ef

1. Save it. That’s it!

Now let’s test it.

Step 1. Configure a new test event

Step 2. Choose event template and Create

Step 3. Run it! It works! Optional: Click the “logs” link to confirm/see what was logged.

Now let’s really test it by manually adding something to our S3 Bucket and check the results.

Choose Services > S3 and choose your bucket (assuming you know this part since you have a bucket).

Confirm

I’ve uploaded the files above and more. Let’s hit our ec2 instance and see if we get them in our search results. Note: You’ll need to install the enterprise search software on your ec2 instance to do this step. You can get an eval here: http://www.voyagersearch.com/download-trial

Yes! The documents are now searchable!

We’ve successfully added a new feature without touching the software source code! Serverless functions are a game changer allowing us to build software with a completely new architecture, or as we’ve seen here, a hybrid approach to add new features to software built using a traditional approach. If you can’t just scrap all the hard work put in to your software and start over (which most can’t), this is a nice alternative to start migrating toward a microservices architecture.