Matt's Blog

How to add Geo Location Search with AWS Amplify and ElasticSearch - Day 4-5

Matthew's photo
Matthew
·

9 min read

As part of the #AmplifyHashnode hackathon, I've decided to build a restaurant review app that leverages React-Native and Amplify. This is day 4-5 of my journey.

The next big piece of the app is the discovery part and ElasticSearch just happens to be a really awesome tool for the job. It allows for incredibly flexible searches based on all kinds of data including geospatial search based on latitude, longitude coordinates. I'll walkthrough how to use Amplify to quickly get your ElasticSearch instance up and running.

Step 1:

Add @searchable directive to Type Establishment

type Establishment @searchable {
  gps: GPS
}

type GPS {
  lon: Float
  lat:  Float
}

Then in the terminal

$: amplify api push

(This usually takes about 15-20 minutes as its waiting for the deployment of the ElasticSearch instance)

What happens here is Amplify creates resources for your ElasticSearch instance, including a python function that will process data and insert the data into ElasticSearch. The function receives data from our Establishment Table via a DynamoDB stream

Any time a record in our table is created, updated or deleted, that data is sent to the stream and can be consumed by other functions, like our python function.
If your curious about what that looks like, go to amplify/api/your-api-name/build/functions/ElasticSearchStreamingLambdaFunction.zip and unzip it.

Step 2

Update Access Policy for ElasticSearch / Kibana

Go to the ElasticSearch console in AWS and click Modify Access Policy elasticsearch-accessplicy-add.png elasticsearch-access-policy.png { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Resource": "arn:aws:es:us-east-1:<AWS-ACCOUNT-ID>:domain/amplify-elasti-fw023i2ikk3/*" } ] }

🚨 WARNING 🚨

This will grant full access to ANYONE and this should only be used in the development process. For this project, I'll scope these permission down to my auth/unauth roles from Cognito and IAM afterwards.

After { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<AWS-ACCOUNT-ID>:role/amplify-restaurantreviewapp-dev-000000-authRole" }, "Action": "es:*", "Resource": "arn:aws:es:us-east-1:<AWS-ACCOUNT-ID>:domain/amplify-elasti-w023i2ikk3/*" }

Step 3:

Create ElasticSearch mappings for Geo Coordinates

In order to search by coordinates in ElasticSearch, we have to tell it specifically which fields from our DB are considered geo_point, so that it can index them to be queried by lat, lon.

There's a couple ways you can add this mapping to our ElasticSearch index, including directly from the Kibana Console. The problem with adding our mapping directly in Kibana is that it can be overwritten after deployments, forcing us to manually add our geo_point after each new build.

There's a 100% chance I would forget to do that some point 😳
Instead, we're going to add a post deployment lambda to our amplify/backend/api/your-api-name/stacks/CustomResources.json.

Add the following to the Resources block: gist here

Inside that gist, you'll see an inline python function under ConfigureES. The important line to notice here is:

python action = {\"mappings\": {\"doc\": {\"properties\": {\"gps\": {\"type\": \"geo_point\"}}}}}"

This will create a mapping for a field from DynamoDB called gps that takes a geo_point type value, which is lat, lon. Now it's time to deploy our new lambda function, so run terminal $: amplify api push
Before we go any further, I have to give a lot of credit to Ramon Postulart and his commentor Ross Williams in this article on Geo Search with Amplify. Ross's comment in this article was the inspiration and base for the code I shared in the gist, so thanks to the both of you 🙏🏻

Step 4:

Create Query findEstablishments in AppSync and attach the ElasticSearch data source

When we added the @searchable directive in Step 1, Amplify created a Query called searchEstablishments, which is awesome for keyword searching values or doing greater than/ less than on numbers. It won't have the logic for searching by gps though, so we have to add it manually.

Add the following to your schema.graphql. I'm cheating a little and using some of the autogenerated resources from the build/schema.graphql including SearchableEstablishmentConnection.

Next we need to go back to our CustomResources.json and add a resolver for findEstablishments that's attached to ElasticSearch 

"FindEstablishmentsResolver": {
      "Type": "AWS::AppSync::Resolver",
      "Properties": {
        "ApiId": {
          "Ref": "AppSyncApiId"
        },
        "DataSourceName": "ElasticSearchDomain",
        "FieldName": "findEstablishments",
        "TypeName": "Query",
        "RequestMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              },
              "ResolverFileName": {
                "Fn::Join": [".", ["Query", "findEstablishments", "req", "vtl"]]
              }
            }
          ]
        },
        "ResponseMappingTemplateS3Location": {
          "Fn::Sub": [
            "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
            {
              "S3DeploymentBucket": {
                "Ref": "S3DeploymentBucket"
              },
              "S3DeploymentRootKey": {
                "Ref": "S3DeploymentRootKey"
              },
              "ResolverFileName": {
                "Fn::Join": [
                  ".",
                  ["Query", "searchEstablishments", "res", "vtl"]
                ]
              }
            }
          ]
        }
      }
    },

If you looked carefully, you'll see on the response mapping template, I used searchEstablishments, instead of findEstablishments. Thats because the VTL response template will be the same, so instead of writing it again, we're pointing to the searchEstablishments response.

We still need to create our resolver template for findEstablishments. Create a file at amplify/api/<your-api-name>/resolvers/Query.findEstablishments.req.vtl and add the following:

#set( $indexPath = "/establishment/doc/_search" )

#set($query = {
  "bool": {

  }
})

#set($sort = [])

#if (!$util.isNull($context.args.input.byGPS ))
  $util.qr($query.bool.put("must", {"match_all" : {}}))
  $util.qr($query.bool.put("filter", {
      "geo_distance": {
      "distance": "${context.args.input.byGPS.radius}km",
      "gps": $context.args.input.byGPS.gps
    }
  }))

  $util.qr($sort.add({
    "_geo_distance":  {
      "gps":  $context.args.input.byGPS.gps,
      "order": "asc",
      "unit": "mi", 
      "distance_type": "plane" 
      }
  }))
#end

#if(
  !$util.isNull($context.args.input.byPlaceID ) || 
  !$util.isNull($context.args.input.query) || 
  !$util.isNull($context.args.input.street ) || 
  !$util.isNull($context.args.input.city ) || 
  !$util.isNull($context.args.input.state ) || 
  !$util.isNull($context.args.input.zipcode )
  )


  #if (!$util.isNull($context.args.input.byPlaceID ))
    $util.qr($query.bool.must.add({ "match": { "placeID": "$context.args.input.byPlaceID" } }))
  #end


  #if (!$util.isNull($context.args.input.query))
    $util.qr($query.bool.must.add({ "match": { "name": "$context.args.input.query" } }))
  #end


  #if (!$util.isNull($context.args.input.street ))
    $util.qr($query.bool.must.add({ "match": { "address.street": "$context.args.input.street" } }))
  #end
  #if (!$util.isNull($context.args.input.city ))
    $util.qr($query.bool.must.add({ "match": { "address.city": "$context.args.input.city" } }))
  #end
  #if (!$util.isNull($context.args.input.state ))
    $util.qr($query.bool.must.add({ "match": { "address.state": "$context.args.input.state" } }))
  #end
  #if (!$util.isNull($context.args.input.zipcode ))
    $util.qr($query.bool.must.add({ "match": { "address.zipcode": "$context.args.input.zipcode" } }))
  #end
#end

#if (!$util.isNull($context.args.input))
  #set($from = $util.defaultIfNull($context.args.input.nextToken, 0))
  #set($size = $util.defaultIfNull($context.args.input.limit, 20))
#else
  #set($from = 0)
  #set($size = 20)
#end

{
  "version":"2017-02-28",
  "operation":"GET",
  "path":"$indexPath",
  "params":{
    "body": {
      "from": $util.toJson($from),
      "size": $util.toJson($size),
      "query": $util.toJson($query),
      "sort": $util.toJson($sort)
    }
  }
}

Okay, let's give it a push

$: amplify api push

Now we have a Query called findEstablishments, that's attached to ElasticSearch. Let's test it out. First, we need to create an Establishment with gps data.

mutation {
  createEstablishment(input:{
    id:"test-restaurant",
    name: "Test Raustaurant",
    gps:{
      lat: 41.88337459649123,
      lon: -87.69204235097645
    }
  }) {
    id
    gps {
      lat
      lon
    }
  }
}

Next, we need to query byGPS (Radius is in kilometers):

query {
  findEstablishments(input:{
        byGPS:{
        gps: {lat: 41.86260812331178, lon: -87.79148237035405},
      radius:10
    }
  }) {
    items {
      id
      gps {
        lat
        lon
      }
    }
  }
}

And drumrollllllll please, the final results:

{
  "data": {
    "findEstablishments": {
      "items": [
        {
          "id": "cc7ea996-1568-4ef7-a640-bd4136eb88f9",
          "gps": {
            "lat": 41.88337459649123,
            "lon": -87.69204235097645
          }
        }
      ]
    }
  }
}

Party time! That's right, we got ourselves a Geo Location search with AWS Amplify and ElasticSearch working y'all, give yourself a pat on the back!

I hope you enjoyed days 4-5, check back in for the next 2 weeks as I try and finish a restaurant review app for the Amplify Hashnode hackathon.

Follow me on twitter @andthensumm

AWSAmplify HashnodeelasticsearchGraphQL