How to add Geo Location Search with AWS Amplify and ElasticSearch

ElasticSearch is a powerful tool that allows flexible searches based on all kinds of parameters, which also happens to include geospatial data.

It just so happens that AWS Amplify supports ElasticSearch with a few commands and with a little bit of tweaking, we can modify CloudFormation to integrate Geo Locations.

I'll walk you throw how to use Amplify and GraphQL Api via AppSync to set up ElasticSearch for geo searches.

This tutorial assumes you have some familiarity with Amplify and have already initiated a project

Step 1:

Add an Establishment Type w/ @searchable directive and type GPS

type Establishment @searchable {
  gps: GPS
}

type GPS {
  lon: Float
  lat:  Float
}

Then in the terminal

$: amplify api push

(This usually takes about 15-20 minutes as its waiting for the deployment of the ElasticSearch instance)

What happens here is Amplify creates resources for your ElasticSearch instance, including a lambda function (in python) that will process data from a DynamoDB stream and insert the data into ElasticSearch.

If you're unfamiliar with DynamoDB streams, they're notifications/callbacks that occur when records are created, updated or deleted. These streams can trigger multiple AWS services, including Lambda

If your curious about the python function Amplify creates in order to insert data in ElasticSearch, go unzip the file here : amplify/api/your-api-name/build/functions/ElasticSearchStreamingLambdaFunction.zip

Step 2

Update Access Policy for ElasticSearch / Kibana

Go to the ElasticSearch console in AWS and click Modify Access Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:<AWS-ACCOUNT-ID>:domain/amplify-elasti-fw023i2ikk3/*"
    }
  ]
}

🚨 WARNING 🚨

This will grant full access to ANYONE and this should only be used in the development process. For this project, I'll scope these permission down to my auth/unauth roles from Cognito and IAM afterwards.


After

 {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AWS-ACCOUNT-ID>:role/amplify-restaurantreviewapp-dev-000000-authRole"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:<AWS-ACCOUNT-ID>:domain/amplify-elasti-w023i2ikk3/*"
    }

Step 3:

Create ElasticSearch mappings for Geo Coordinates

In order to search by coordinates in ElasticSearch, we have to tell it specifically which fields from our DB are considered geo_point, so that it can index them to be queried by lat, lon.

There's a couple ways you can add this mapping to our ElasticSearch index, including directly from the Kibana Console. The problem with adding our mapping directly in Kibana is that it can be overwritten after deployments, forcing us to manually add our geo_point after each new build.

There's a 100% chance I would forget to do that some point 😳

Instead, we're going to add a post deployment lambda to our amplify/backend/api/your-api-name/stacks/CustomResources.json.

Add the following to the Resources block: gist here

Inside that gist, you'll see an inline python function under ConfigureES. The important line to notice here is:

action = {\"mappings\": {\"doc\": {\"properties\": {\"gps\": {\"type\": \"geo_point\"}}}}}"

This will create a mapping for a field from DynamoDB called gps that takes a geo_point type value, which is lat, lon.

Now it's time to deploy our new lambda function, so run

$: amplify api push

Before we go any further, I have to give a lot of credit to
Ramon Postulart and his commentor Ross Williams in this article on Geo Search with Amplify. Ross's comment in this article was the inspiration and base for the code I shared in the gist, so thanks to the both of you 🙏🏻

Step 4:

Create Query findEstablishments and attach the ElasticSearch data source in CustomResources along with VTL templates

When we added the @searchable directive in Step 1, Amplify created a Query called searchEstablishments, which is awesome for keyword searching values or doing calculations or other search combinations. It won't have the logic for searching by gps though, so we have to add it manually by creating a Query called findEstablishments

Add the to your schema.graphql:

type Query {
  findEstablishments(
    input: FindEstablishmentsInput!
  ): SearchableEstablishmentConnection @aws_iam @aws_cognito_user_pools
}

type GPS {
  lon: Float
  lat: Float
}

input GPSInput {
  lon: Float!
  lat: Float!
}

input GPSQueryInput {
  gps: GPSInput!
  radius: Float!
}

input FindEstablishmentsInput {
  byGPS: GPSQueryInput
  byPlaceID: String
  limit: Int
  nextToken: String
  from: Int
}

type SearchableEstablishmentConnection @aws_iam @aws_cognito_user_pools {
  items: [Establishment]
  nextToken: String
  total: Int
}

Next we need to go back to our CustomResources.json and add a resolver for findEstablishments that's attached to ElasticSearch

"FindEstablishmentsResolver": {
      "Type": "AWS::AppSync::Resolver",
      "Properties": {
        "ApiId": {
          "Ref": "AppSyncApiId"
        },
        "DataSourceName": "ElasticSearchDomain",
        "FieldName": "findEstablishments",
        "TypeName": "Query",
        "RequestMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              },
              "ResolverFileName": {
                "Fn::Join": [".", ["Query", "findEstablishments", "req", "vtl"]]
              }
            }
          ]
        },
        "ResponseMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              },
              "ResolverFileName": {
                "Fn::Join": [
                  ".",
                  ["Query", "searchEstablishments", "res", "vtl"]
                ]
              }
            }
          ]
        }
      }
    },

The ElasticSearch data source ("DataSourceName": "ElasticSearchDomain") was created for us by @searchable directive, so we're just referencing it by name.

If you looked carefully, you'll see I used ["Query", "searchEstablishments", "res", "vtl"] for the response mapping template, instead of ["Query", "findEstablishments", "res", "vtl"]. Amplify already generated an appropriate response template for ElasticSearch with searchEstablishments, so instead of writing it again, we're pointing to the searchEstablishments response.

Next up, we need to create our resolver template for findEstablishments. Create a file at amplify/api/<your-api-name>/resolvers/Query.findEstablishments.req.vtl and add the following:

#set( $indexPath = "/establishment/doc/_search" )

#set($query = {
  "bool": {

  }
})

#set($sort = [])

#if (!$util.isNull($context.args.input.byGPS ))
  $util.qr($query.bool.put("must", {"match_all" : {}}))
  $util.qr($query.bool.put("filter", {
    "geo_distance": {
      "distance": "${context.args.input.byGPS.radius}km",
      "gps": $context.args.input.byGPS.gps
    }
  }))

  $util.qr($sort.add({
    "_geo_distance":  {
      "gps":  $context.args.input.byGPS.gps,
      "order": "asc",
      "unit": "mi", 
      "distance_type": "plane" 
    }
  }))
#end


#if (!$util.isNull($context.args.input))
  #set($from = $util.defaultIfNull($context.args.input.nextToken, 0))
  #set($size = $util.defaultIfNull($context.args.input.limit, 20))
#else
  #set($from = 0)
  #set($size = 20)
#end

{
  "version":"2017-02-28",
  "operation":"GET",
  "path":"$indexPath",
  "params":{
    "body": {
      "from": $util.toJson($from),
      "size": $util.toJson($size),
      "query": $util.toJson($query),
      "sort": $util.toJson($sort)
    }
  }
}

Okay, let's give it a push

$: amplify api push

Step 5:

Searching Locations

Now that we have a Query called findEstablishments that's attached to ElasticSearch, let's test it out. First, we need to create an Establishment with gps data.

mutation {
  createEstablishment(input:{
    id:"test-restaurant",
    name: "Test Raustaurant",
    gps:{
      lat: 41.88337459649123,
      lon: -87.69204235097645
    }
  }) {
    id
    gps {
      lat
      lon
    }
  }
}

Next, we need to query byGPS (Radius is in kilometers):

query {
  findEstablishments(input:{
        byGPS:{
        gps: {lat: 41.86260812331178, lon: -87.79148237035405},
      radius:10
    }
  }) {
    items {
      id
      gps {
        lat
        lon
      }
    }
  }
}

And drumrollllllll please, the final results:

{
  "data": {
    "findEstablishments": {
      "items": [
        {
          "id": "cc7ea996-1568-4ef7-a640-bd4136eb88f9",
          "gps": {
            "lat": 41.88337459649123,
            "lon": -87.69204235097645
          }
        }
      ]
    }
  }
}

That's right, we got ourselves a Geo Location search with AWS Amplify and ElasticSearch working y'all, give yourself a pat on the back!

Hope you enjoyed this tutorial and for more Amplify and Serverless content, follow me on twitter @andthensumm