Use Rclone with the Koyeb Serverless Platform to Manage Data Across Cloud Storage Providers

Introduction

This guide demonstrates how to use Rclone on the Koyeb serverless platform and perform diverse operations such as data synchronization, copying, and moving data between two cloud storage providers. Rclone is a command-line program offering a large set of features to manage your data stored across different cloud storage providers. You can imagine Rclone as the Unix rsync command for the cloud.

As Koyeb provides a unified interface to manage your data and preview the result of each operation we will do in this guide from a single interface, we will create two Koyeb Stores connected to each of the providers used in this guide: AWS S3 and Scaleway Object Storage.
However, you can use Rclone with whichever cloud service providers you want! Find the full list of providers that Rclone supports here.

Below is a non-exhaustive list of operations you can perform using Rclone:

• Back up data stored with your cloud storage providers,
• Synchronize files and directories from different cloud storage providers,
• Copy files and directories from a cloud storage provider to another,
• Move files and directories from one cloud storage provider to the next,
• Compare the content of different repositories and buckets across different cloud storage providers,
• And much more all between any cloud storage providers.

Using Rclone with Koyeb lets you manage and move your data across different cloud storage providers through a reliable and easy-to-use serverless platform, Koyeb. Since Koyeb allows long-running functions, you can synchronize your data or copy data from a cloud storage provider to another without having to care about the execution time the operation took to perform. Combining Rclone with Koyeb allows you to run Rclone in batch mode or at a regular interval using Koyeb's Cron functions.
This tutorial will cover how to use Rclone to:

• Synchronize directories to modify the destination directory to match the source directory by either adding or removing files,
• Copy files from source to destination to copy the content from the source bucket to the destination bucket without duplicating files,
• Check directories differences to compare the two directories or buckets and to identify the difference if there is any.

Requirements

To successfully follow this guide, you need:

• A Koyeb account
• An AWS account and an AWS bucket
• A Scaleway account and a Scaleway bucket
• A Koyeb Store connected to your AWS S3 bucket
• A Koyeb Store connected to your Scaleway bucket

Steps

To use Rclone with Koyeb to manage and move your data across different cloud storage providers, follow these steps:

1. Create a Stack and deploy a new revision to sync your data
2. Copy files from source to destination
3. Check Directories for Missing or Extra Files

Create a Stack and deploy a new revision to sync your data

Stacks are processing environments you deploy functions and Docker containers. There are different ways to invoke your Stack functions using:

• Cron job: To trigger a function on a time-based period
• Store events: To trigger a function when an event occurs on your Stores
• Manual invocation: To invoke your function manually

All Stacks you create benefit from native versioning to track changes and rollback to previous versions easily.

To create a new Stack:

1. In the Koyeb Console, click the Create button in the top navigation bar and select Stack.
2. You land on the Stack creation page, Select Koyeb Simple Versioning, and give your Stack a name.

On your Stack page, you are asked to deploy your first Stack revision containing your Stack configuration. A Stack configuration is defined by a YAML file. Each time you deploy a new Stack revision, Koyeb reads your YAML instructions to build and deploy your functions.

In this guide, we use the Rclone Docker image to showcase how to sync, copy, or check data across two different cloud storage providers.

Below is the Stack configuration YAML used to perform a sync operation.

functions:
  - name: rclone
    image: rclone/rclone
    command: ["/bin/sh", "-c"]
    args: [
      RCLONE_CONFIG_AWS_TYPE=s3
      RCLONE_CONFIG_AWS_ACCESS_KEY_ID=$KOYEB_STORE_aws_ACCESS_KEY
      RCLONE_CONFIG_AWS_SECRET_ACCESS_KEY=$KOYEB_STORE_aws_SECRET_KEY
      RCLONE_CONFIG_AWS_ENDPOINT=$KOYEB_STORE_aws_ENDPOINT
      RCLONE_CONFIG_AWS_REGION=$KOYEB_STORE_aws_REGION
      RCLONE_CONFIG_SCALEWAY_TYPE=s3
      RCLONE_CONFIG_SCALEWAY_ACCESS_KEY_ID=$KOYEB_STORE_scaleway_ACCESS_KEY
      RCLONE_CONFIG_SCALEWAY_SECRET_ACCESS_KEY=$KOYEB_STORE_scaleway_SECRET_KEY
      RCLONE_CONFIG_SCALEWAY_ENDPOINT=$KOYEB_STORE_scaleway_ENDPOINT
      RCLONE_CONFIG_SCALEWAY_REGION=$KOYEB_STORE_scaleway_REGION
      rclone sync aws:store-connected-to-aws-s3 scaleway:store-connected-2-scaleway
    ]
    volumes:
      - name: scaleway
        store: store-connected-2-scaleway
      - name: aws
        store: store-connected-to-aws-s3

About the YAML

• name: The name instruction is used to define your function name. The name is then displayed in your Stack revision functions list and function executions history. Here rclone.
• image: The Docker image instruction being pulled from the Docker Hub and deployed on Koyeb. Here rclone/rclone.
• command: The command passed to the Rclone container. Here we use ["/bin/sh", "-c"] as we need to override the default command to pass additional environment variables containing our configuration to rclone
• args: The command passed to the command. Here we set environment variables to define the Rclone configuration. These environment variables are set using the value of the environment variables set by the volume instruction. The final part contains the rclone command, here sync.
• volumes: The volumes instruction is used to expose temporary Koyeb Store credentials as environment variables during function execution.

If you want to read more about Koyeb Stacks YAML Syntax, check out our documentation here.

This operation syncs the content of aws:store-connected-to-aws-s3 to scaleway:store-connected-2-scaleway, modifying only the destination directory. Since the destination directory will be modified to match the source directory, files will either be removed or added to your destination directory.

Click Deploy Revision and wait for your Stack to build.

Once your Stack has been built, click Invoke to manually invoke this function.

You can test if your Stack has successfully synchronized your data by:

• Manually comparing if the Koyeb Store connected to your Scaleway matches the content of your Koyeb Store connected to AWS S3 without leaving the Koyeb control panel
• Using Rclone's check operation to verify the bucket contents match

Copy files from source to destination

You can use Rclone to copy files from one directory to another, only adding files to the destination directory and skipping files that are already in both directories. Copy is different than the sync operation because copy will not remove files from the destination directory, it will only add non-duplicate files.

In your Stack, click Manage Stack Configuration and edit the Rclone command in the YAML:

Enter the following Stack configuration:

functions:
  - name: rclone
    image: rclone/rclone
    command: ["/bin/sh", "-c"]
    args: [
      RCLONE_CONFIG_AWS_TYPE=s3
      RCLONE_CONFIG_AWS_ACCESS_KEY_ID=$KOYEB_STORE_aws_ACCESS_KEY
      RCLONE_CONFIG_AWS_SECRET_ACCESS_KEY=$KOYEB_STORE_aws_SECRET_KEY
      RCLONE_CONFIG_AWS_ENDPOINT=$KOYEB_STORE_aws_ENDPOINT
      RCLONE_CONFIG_AWS_REGION=$KOYEB_STORE_aws_REGION
      RCLONE_CONFIG_SCALEWAY_TYPE=s3
      RCLONE_CONFIG_SCALEWAY_ACCESS_KEY_ID=$KOYEB_STORE_scaleway_ACCESS_KEY
      RCLONE_CONFIG_SCALEWAY_SECRET_ACCESS_KEY=$KOYEB_STORE_scaleway_SECRET_KEY
      RCLONE_CONFIG_SCALEWAY_ENDPOINT=$KOYEB_STORE_scaleway_ENDPOINT
      RCLONE_CONFIG_SCALEWAY_REGION=$KOYEB_STORE_scaleway_REGION
      rclone copy aws:store-connected-to-aws-s3 scaleway:store-connected-2-scaleway
    ]
    volumes:
      - name: scaleway
        store: store-connected-2-scaleway
      - name: aws
        store: store-connected-to-aws-s3

Click Deploy Revision and wait for your Stack to build.

Once your Stack has been built, click Invoke to manually invoke this function.

To see if your files were successfully copied from your source bucket to your destination bucket, you can:

• Manually comparing if the Koyeb Store connected to your Scaleway match the content of your Koyeb Store connected to AWS S3 without leaving the Koyeb control panel
• Using Rclone check operation to verify the bucket contents match

Check Directories for Missing or Extra Files

You can use Rclone to compare one directory to another, checking if there are missing or extra files between the source and destination directories.

In your Stack, click Manage Stack Configuration and edit the Rclone command in the yaml:

Enter the following Stack configuration:

functions:
  - name: rclone
    image: rclone/rclone
    command: ["/bin/sh", "-c"]
    args: [
      RCLONE_CONFIG_AWS_TYPE=s3
      RCLONE_CONFIG_AWS_ACCESS_KEY_ID=$KOYEB_STORE_aws_ACCESS_KEY
      RCLONE_CONFIG_AWS_SECRET_ACCESS_KEY=$KOYEB_STORE_aws_SECRET_KEY
      RCLONE_CONFIG_AWS_ENDPOINT=$KOYEB_STORE_aws_ENDPOINT
      RCLONE_CONFIG_AWS_REGION=$KOYEB_STORE_aws_REGION
      RCLONE_CONFIG_SCALEWAY_TYPE=s3
      RCLONE_CONFIG_SCALEWAY_ACCESS_KEY_ID=$KOYEB_STORE_scaleway_ACCESS_KEY
      RCLONE_CONFIG_SCALEWAY_SECRET_ACCESS_KEY=$KOYEB_STORE_scaleway_SECRET_KEY
      RCLONE_CONFIG_SCALEWAY_ENDPOINT=$KOYEB_STORE_scaleway_ENDPOINT
      RCLONE_CONFIG_SCALEWAY_REGION=$KOYEB_STORE_scaleway_REGION
      rclone check aws:store-connected-to-aws-s3 scaleway:store-connected-2-scaleway
    ]
    volumes:
      - name: scaleway
        store: store-connected-2-scaleway
      - name: aws
        store: store-connected-to-aws-s3

Click Deploy Revision and wait for your Stack to build.

Once your Stack has been built, click Invoke to manually invoke this function.

Click Show Logs to see the results.

The example above shows 2 files in the destination directory are missing compared to the source directory.

Conclusion

In this guide, you learned how to manage and move your data across different cloud storage providers with Rclone using Koyeb's serverless platform. Running Rclone on Koyeb's serverless platform ensures you only use the computing time your Rclone command requires, nothing more.

This tutorial covered how to use Rclone to synchronize, copy, move, and check your data across your different cloud service providers. You can check out all other Rclone supported operations here.

Want to follow this tutorial and use different third-party cloud service providers than AWS and Scaleway? Learn how to connect your cloud storage provider to Koyeb.

If you need any assistance or have additional questions, do not hesitate to reach out to us via Koyeb's in-app support or over Slack. We're happy to help!