Directly upload large files to the cloud with Anvil

You might need to build a web app that allows users to upload very large files. If you were building an Anvil app, you would normally use a FileLoader component to capture the user's file then store the file in a Data Table. But if your file is many gigabytes in size, you might see timeouts while uploading or exceed the storage limits of your Data Tables. Instead, it makes sense to store giant files directly in a cloud storage service, such as Amazon S3.

Fortunately, Anvil makes it easy to use third-party cloud services on the back end, and Javascript libraries on the front end. In this example, we'll walk through how to use the use the uppy.io JavaScript widget and Amazon boto3 library to upload files from Anvil directly to an S3 bucket.

New to Anvil? Anvil is a platform for building full-stack web applications with nothing but Python. This is a fairly advanced how-to guide -- you might want to start with a quick tutorial.

Check out the app we're going to build by cloning the source code.

We'll build the app in five steps:

1. First, we'll create a new Anvil app.
2. We'll then set up an Amazon S3 bucket with the proper permissions.
3. Next, we'll add the Uppy widget to our app.
4. We then need to get a presigned URL using Amazon's boto3 library.
5. Finally, we'll provide the presigned URL to Uppy which will upload the file to S3.

Step 1: Create an Anvil app

First, log into Anvil and create a new blank app. We'll need to make use of our app's App Secrets to store the AWS access keys that we'll acquire in the next step, so ahead and add the App Secrets service to your app.

Step 2: Set up an S3 bucket

If you don't already have an S3 bucket, you'll need to set one up. You can find a get started guide here: Getting Started with Amazon S3.

Your S3 bucket will need to have cross-origin access enabled. The following cross-origin policy is sufficient to allow uploads from browsers – you can paste it into the CORS policy text box in the S3 console:

[{"AllowedHeaders": ["content-type"], "AllowedMethods": ["POST", "PUT"], "AllowedOrigins": ["*"], "ExposeHeaders": []}]

You'll also need an AWS Access Key and a Secret Access Key that has permission to write to your S3 bucket.

Save your access keys in your app's App Secrets. Store the AWS access key ID in a secret named aws_key_id and the secret key in a secret named aws_secret_key.

Step 3: Add the Uppy widget to the app

Uppy is a third-party JavaScript that displays a form where users can drag and drop files to upload to S3. We can add the widget to an Anvil app using Native Libraries and the anvil.js module.

First, add the following line to your app's Native Libraries:

<link href="https://releases.transloadit.com/uppy/v3.3.1/uppy.min.css" rel="stylesheet">

Next, go to the Design View of your Form, add a ColumnPanel, and name it uppy_target. This is where the Uppy widget will go.

Switch to Code View and add the following import statements to your Form's code:

from anvil.js import window, import_from, get_dom_node
import anvil.js

We'll need to set up two functions for the Uppy widget to use:

• get_upload_parameters will return AWS credentials from a Server Module that will give Uppy temporary permission to access the S3 bucket.

• on_complete will be called by Uppy when the upload completes, whether or not it's successful.

For now, the functions won't do anything, but we'll fix that in Steps 4 and 5.

  def get_upload_parameters(self, file_descr):
    # This function is called to get the pre-signed URL
    # for the S3 upload.
    pass


  def on_complete(self, result, *args):
    # This function is called when the upload is finished,
    # successful or not.
    pass

Next, in the Form's __init__ method, add the following code:

    mod = anvil.js.import_from("https://releases.transloadit.com/uppy/v3.3.1/uppy.min.mjs")
    self.uppy = mod.Uppy().use(mod.Dashboard, {
      'inline': True,
      'target': get_dom_node(self.uppy_target)
    })
    self.uppy.use(mod.AwsS3, {
      'getUploadParameters': self.get_upload_parameters
    })
    self.uppy.on('complete', self.on_complete)

If you run the app now, you'll see an empty Uppy uploader. Nothing will happen if you try to upload a file because we haven't set up the get_upload_parameters function yet. In the next step, we'll use boto3 to return the AWS credentials we need.

Step 4: Get a presigned URL from S3 using boto3

The AWS access key we set up in Step 2 will provide access to S3, but it's a powerful credential for this very reason. We don't want to expose it to the client and give anyone unlimited access to our S3.

Instead, we can generate a presigned URL, which grants temporary access to upload one file to the target S3 bucket. We can use the boto3 library from a Server Module to generate the presigned URL.

We can trust the server code with powerful AWS credentials, because we have control over the server. Because the client is controlled by anyone who visits our app, we can't trust it with anything except a time-limited, single-purpose presigned URL. Read more about client and server security here.

In a Server Module, add the required import statements and set up variables for your S3 bucket and AWS region:

import boto3, string
from datetime import datetime
from random import SystemRandom

BUCKET_NAME="anvil-upload-demo" # Replace with your S3 bucket name
REGION_NAME="eu-west-2" # Replace with your AWS region

Next, we'll need to create a function to generate the presigned URL and make it callable from the client. This function will be called in the get_upload_parameters function we created in the previous step:

@anvil.server.callable
def get_presigned_url(filename, content_type):
  s3_client = boto3.client(
      's3',
      aws_access_key_id=anvil.secrets.get_secret('aws_key_id'),
      aws_secret_access_key=anvil.secrets.get_secret('aws_secret_key'),
      region_name=REGION_NAME
  )

  # Because the filename is user-supplied, we use the current date and a random string to ensure
  # a unique filename for each upload
  random_string = ''.join(SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(10))
  filename_to_upload = f"{datetime.now().isoformat()}-{random_string}-{filename}"

  r = s3_client.generate_presigned_post(BUCKET_NAME, filename_to_upload)

  # Return the data in the format Uppy needs
  return {
    'url': r['url'],
    'fields': r['fields'],
    'method': 'POST'
  }

Step 5: Provide the presigned URL to Uppy

When Uppy is ready to upload a file, it will call the get_upload_parameters function. We can now change the function so that it calls the get_presigned_url server function and returns the results (the pre-signed URL and associated data) to Uppy.

Because we're using this function as a callback from Javascript, we need to add the @anvil.js.report_exceptions decorator. This makes sure that if the code throws an exception, it appears correctly in the Anvil console.

Go back to your Form's client code and update get_upload_parameters:

  @anvil.js.report_exceptions
  def get_upload_parameters(self, file_descr):
    return anvil.server.call('get_presigned_url', file_descr['name'], file_descr['type'])

Provided you've configured your bucket and access keys correctly, you should now be able to upload files to S3.

Optionally, we can configure the on_complete function to report feedback from Uppy. We can add this to an alert to display to the user:

  @anvil.js.report_exceptions
  def on_complete(self, result, *args):
    successful = result['successful']
    failed = result['failed']
    alert(f"{len(successful)} succeeded, {len(failed)} failed")

That's it!

That's all we need to upload files directly from our user's browser into S3.

You can copy the sample app here. Of course, you'll need your own S3 bucket and access keys for it work.

For our next steps, consult the Uppy documentation for how to customise the Uppy upload widget, or the boto3 documentation for how to handle your file now it's safely uploaded into S3.

Using other cloud storage services

Amazon S3 isn't the only cloud storage service out there. But because it was the first, a lot of other services use the same API. You can adapt the sample code from this example to use other services, such as Cloudflare R2, DigitalOcean Spaces or Backblaze B2.

Just check your preferred service's documentation for how to use it with boto3. (For example, here's CloudFlare's page on configuring boto3 to work with their R2 service.)

Build your own app with Anvil

If you're new here, welcome! Anvil is a platform for building full-stack web apps with nothing but Python. No need to wrestle with JS, HTML, CSS, Python, SQL and all their frameworks – just build it all in Python.

Sign up for Anvil!