DEV Community

Cover image for Journey from Elastic Cloud to AWS Elastic Search

Journey from Elastic Cloud to AWS Elastic Search

This article demonstrates the steps that we should follow when we migrate Elastic search data from Elastic cloud to AWS.

Let's get started.

The Elastic Cloud supports Major.Minor.Patch (eg: x.y.z) version of Elastic Search, where AWS supports only Major. Minor (eg: x.y).

When you build the target cluster on AWS, ensure you created the cluster using the next latest Minor version of the source cluster. (If the source cluster version is 7.7.1, the target cluster version must be 7.8).

Elastic Search doesn’t allow migrating from the upper version to the lower version.

{"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapshot was created with Elasticsearch version
 [7.7.1] which is higher than the version of this node [7.7.0]"}],"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapsh
ot was created with Elasticsearch version [7.7.1] which is higher than the version of this node [7.7.0]"},"status":500}
Enter fullscreen mode Exit fullscreen mode

Elasticsearch to AWS Opensearch

At first, create an S3 bucket in your AWS account as a custom repository to store ES snapshots/backups.

You could find the steps to set up a custom repository on the Elastic Cloud at their site. Therefore, I am not going to describe them here.

Create an EC2 instance which will be used to execute commands during the data restoration. (Even t2.nano is sufficient).

Create two roles such as Role-S3, Role-ES and two policies such as Policy-S3 and Policy-ES and attach them to the roles respectively.

Policy-S3
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [ 
 "arn:aws:s3:::<es-snapshot-bucket-name>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [ 
  "arn:aws:s3::: <es-snapshot-bucket-name>/*”
            ]
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode
Policy-ES
{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "iam:PassRole",
                "Resource": "arn:aws:iam::<account_id>:role/Role-S3"  
            },
            {
                "Effect": "Allow",
                "Action": "es:ESHttpPut",
                "Resource": [
                         "arn:aws:es:<region>:<account_id>:domain/<es_domain_name>/*"
                  ]
            }
        ]
    }
Enter fullscreen mode Exit fullscreen mode

Attach the role Role-ES to the EC2 instance and log in to the EC2.

Trigger a curl command to AWS ES endpoint to check if the cluster is reachable.

curl <es_endpoint>
Enter fullscreen mode Exit fullscreen mode

Install the following python libs in the EC2.

$ pip3 install boto3
$ pip3 install requests
$ pip3 install requests_aws4auth
$ pip3 install --upgrade request
Enter fullscreen mode Exit fullscreen mode

Copy and Paste the given python script, change the appropriate values and run the script.

This script will register the S3 bucket as a snapshot repository for the ES cluster that we created.

import boto3
import requests
from requests_aws4auth 
import AWS4Auth

host = <es_domain_endpoint/>    # Enter the ES domain endpoint and trailing ‘/’
region = <region>       
service = 'es'

credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Steps to Register snapshot repository
path = '_snapshot/<repository-name>'   # the ES API endpoint
url = host + path
payload = {
  "type": "s3",
  "settings": {
    "bucket": <es_snapshot_bucket_name>, 
    "region": <region>,  # Specify region for S3 bucket. If the S3 bucket is in the us-east-1 region use endpoint
    "endpoint": "s3.amazonaws.com", 
    "role_arn": "arn:aws:iam::<account_id>:role/Role-S3
  }
}
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
Enter fullscreen mode Exit fullscreen mode

The script would print output as shown below if it is executed successfully.

200
{"acknowledged":true}
Enter fullscreen mode Exit fullscreen mode

List down the snapshots.

curl <es_endpoint>/_snapshot/<repository_name>/_all?pretty
Enter fullscreen mode Exit fullscreen mode

Restore a snapshot

curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore
Enter fullscreen mode Exit fullscreen mode

Restore a specific index from a snapshot with settings

curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore -d '{
"indices": "<index_name>",
"index_settings": {
  "index.routing.allocation.require.data": null
  }
}' -H 'Content-Type: application/json'
Enter fullscreen mode Exit fullscreen mode

List down the indices.

curl -XGET <es_endpoint>/_cat/indices
Enter fullscreen mode Exit fullscreen mode

You'll be able to view the indices, if the restoration is successful.

** Key Points **

Don't restore the .kibana or system indices. It could throw errors.
Don't delete .kibana index in target cluster when restoring.

Discussion (0)