MinIO - Mock S3 in local development

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services that provides object storage through a web service interface. Amazon S3 is undoubtedly one of the popular storage services around. Most of the developers have worked with it either in their professional projects or pet projects to store files. But to work with it, we must need an AWS account, set up the buckets, etc. If you work on a company project, you would have to wait for the credentials from the infrastructure before developing and testing the functionalities. That's where MinIO comes to help.

Minio is an open source distributed object storage server written in Go, designed for Private Cloud infrastructure providing S3 storage functionality. As it implements the same API as amazon S3, any application which can communicate with S3, also can communicate with MinIO. In simple terms, it’s like Amazon S3 but hosted locally. MinIO is a high performance object storage solution that provides an Amazon Web Services S3-compatible API and supports all core S3 features.

Why we are talking about MinIO because you can create S3 storage functionality locally, develop and test your API without needing any actual S3 bucket or account. Later at production, you can just change the credential from MinIO to S3 without changing any internal code.

S3 can serve many purposes including replacing S3 in production but we will talk about from a developer perspective where we will only use it for development, testing and mocking S3 functionalities.

Architecture

MinIO has three major components.

1. MinIO Server: MinIO Server serves as the root component.

2. MinIO Client: Referred to as mc provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, find etc.

3. MinIO Client SDK: MinIO Client SDK provides an API to access any Amazon S3 compatible object storage server. These are the available SDK:

   • Java
   • Go
   • Python
   • JavaScript
   • .NET
   • Haskell

In this article, we will learn how to setup a MinIO server in our localhost (which ideally can be replaced with S3 server in production). But we will not use the MinIo client. Rather we will use the existing amazon S3 client SDK.

Installation & Setup

We will now start the installation of the MinIO Server. The best and fastest way to run a instance of MinIO server locally is to use Docker (read more about Docker). You can use either Docker Run or Docker Composer.

Docker Run

mkdir -p ~/minio/data

docker run \
    -p 9000:9000 \
    -p 9090:9090 \
    --name minio \
    -v ~/minio/data:/data \
    -e "MINIO_ROOT_USER=root" \
    -e "MINIO_ROOT_PASSWORD=password" \
    quay.io/minio/minio server /data --console-address ":9090"

• mkdir creates a new local directory at ~/minio/data in your home directory.
• docker run starts the MinIO container.
• -p binds a local port to a container port.
• -name creates a name for the container.
• -v sets a file path as a persistent volume location for the container to use. When MinIO writes data to /data, that data mirrors to the local path ~/minio/data, allowing it to persist between container restarts. You can replace ~/minio/data with another local file location to which the user has read, write, and delete access.
• -e sets the environment variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, respectively. These set the root user credentials. Change the example values to use for your container.

Finally, access the MinIO Console by going to a browser and visiting http://127.0.0.1:9000.

Docker Compose

In the docker-compose.yml:

version: '3'
services:
  minio:
    image: 'minio/minio:latest'
    ports:
      - '${FORWARD_MINIO_PORT:-9000}:9000'
      - '${FORWARD_MINIO_CONSOLE_PORT:-9090}:9090'
    environment:
      MINIO_ROOT_USER: 'root'
      MINIO_ROOT_PASSWORD: 'password'
    volumes:
      - 'minio:/data/minio'
    command: minio server /data/minio --console-address ":9090"
volumes:
  minio:
    driver: local

You can access the MinIO Console at http://127.0.0.1:9000.

MinIO Play

You can get started exploring MinIO features using their play server at https://play.min.io. Any file uploaded to play should be considered public and non-protected.

Credentials:

Username: Q3AM3UQ867SPQQA43P2F

Password: zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

Native Install

If you do not want use Docker, you can install MinIO server natively in your machine.

• For Linux
• For MacOS
• For Windows

Usage

Now that you have installed MinIO server, let's use it on your project. I will be using the official S3 JavaScript SDK as example. But should be applicable for all other S3 SDKs.

We will instruct the SDK to use the endpoint of our locally hosted MinIO server. Find the the initialization point of the SDK in your code. Modify the config as follows.

For JavaScript SDK:

const config = new S3({
  endpoint: 'http://localhost:9000', // for docker, http://minio:9000
  credentials: {
    accessKeyId: 'root', // MINIO_ROOT_USER
    secretAccessKey: 'password', // MINIO_ROOT_PASSWORD
  },
  s3ForcePathStyle: true, // important
});

If you are using MinIO Play:

const config = new S3({
  endpoint: 'https://play.min.io:9000',
  credentials: {
    accessKeyId: 'Q3AM3UQ867SPQQA43P2F',
    secretAccessKey: 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG',
  },
  s3ForcePathStyle: true,
});

For Laravel:

AWS_ACCESS_KEY_ID=root
AWS_SECRET_ACCESS_KEY=password
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=http://localhost:9000 # for docker, http://minio:9000
AWS_USE_PATH_STYLE_ENDPOINT=true

As simple as that. You do not need to change anything else. With these modifications, you now have a working connection with your client and the local server that you can use to mimic S3 functionalities without needing actual S3 configuration and access.