DEV Community

Abhimanyu Babbar
Abhimanyu Babbar

Posted on

8 3

SFTP to S3 Backup

In our company, we had a simple SFTP Server where third parties used to supply us with reports, files etc. In order to backup the information onto S3 where we could start processing the files, I wrote a simple utility in Go to achieve the same sftp-s3-backup.

The purpose of this article is to walk through a simple process of backing up the files present in SFTP onto S3.

S3 Login Credentials

In order to backup onto S3 our system needs to be able to login to the platform. Below environment variables needs to be set for the program to automatically pick them up. The variables are as follows:

  1. AWS_ACCESS_KEY_ID
  2. AWS_SECRET_ACCESS_KEY
  3. AWS_REGION
  4. AWS_SESSION_TOKEN // In case you have 2 factor auth enabled.

Simple Backup

package main

import (
  "log",
  backup "github.com/babbarshaer/sftp-s3-backup"
)

func main() {

  config := backup.Config{
    User:              "user",
    Address:           "address",
    Port:              22,
    PublicKeyLocation: "/home/user/.ssh/id_rsa.pub",
  }

  client := backup.Client(config)
  err = client.Init()
  if err != nil {
    log.Fatalf("Unable to initialize the client: %s", err.Error())
  }

  defer client.Close()

  err := client.Backup(
    "/data/user/uploads",
    "aws.bucket",
    backup.DefaultPathTransformer)

  // If we don't want the default path transformer
  // we can override it with our own implementation.

  if err != nil {
    log.Fatalf("Unable to backup, err: %s", err.Error())
  }

We will go through each part of the program step by step:

We start by creating a config object which mainly contains the credentials of the sftp server who's directory we need to backup onto S3.

config := backup.Config{
    User:              "user",
    Address:           "address",
    Port:              22,
    PublicKeyLocation: "/home/user/.ssh/id_rsa.pub",
  }

We then create a new backup client object by supplying in the config. We initialize the client by calling the Init() method.

client := backup.Client(config)
client.Init()

If everything goes smoothly, we should be able to establish a sftp connection after this point. Otherwise, it would return an error. Handling of error depends upon the user at this point.

We should always remember to free up the client connection.

defer client.Close()

Finally, we can now perform a directory backup. This method will backup all the files in the directory to the corresponding bucket in s3. We supply an additional transformer variable which dictates how the path needs to be transformed when storing the object from sftp onto s3.

directoryToBackup := "/data/user/uploads"
awsBucket := "aws.bucket"
transformer := backup.DefaultPathTransformer

err := client.Backup(
    directoryToBackup,
    awsBucket,
    transformer)

A user can supply there own transformer function as follows:

func MyTransformer(path string) string {
  newPath := path
  // computation ...
  return newPath
}


Sentry workshop image

Sick of your mobile apps crashing?

Let Simon Grimm show you how to fix them without the guesswork. Join the workshop and get to debugging.

Save your spot →

Top comments (1)

Collapse
 
couchdrop_ profile image
Couchdrop

You could also use couchdrop.io if you wanted to use something other than S3, or a more generic S3.

Billboard image

Deploy and scale your apps on AWS and GCP with a world class developer experience

Coherence makes it easy to set up and maintain cloud infrastructure. Harness the extensibility, compliance and cost efficiency of the cloud.

Learn more

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay