Bucket Storage on Railway with Go

A little about Railway and what we are trying to solve

Railway has changed how I will build any side projects (and future start up!). One of the best things about building software is actually writing the code and solving the problems you intended with it. But a lot of times I feel like I am wrestling configurations for AWS or GCP to get things to work in an automated way. Thats time lost into building your actual application.

Railway has changed that for me. Deploying and adding resources like a database is seamless and serverless. I haven't been this excited about a new product in a long time. Thats why I decided to write this blog post. Currently Railway doesn't have native object storage, so I wanted to share one way to use some sort of object storage for your project. It's the least I can do for their generous free tier.

Pre-requisites & Setup.

1. You have a google cloud account you can sign up here with a gmail account.Google Cloud
   Why Google cloud? Well I thought there is a lot s3 and AWS guides out there to try something different. (If you are interested in AWS guide let me know!).

2. You have a Railway.app account. Their free tier is really generous and should be sufficient for this guide.

3. The way I like to use Railway is to connect to an existing project from Github to it. In this case it is the object storage railway project https://github.com/ralaruri/object-storage-railway. Every-time you push a PR you can run a development deployment and when you merge to your main branch you can push to production Add Project

4. If you want to skip the guide check out the repo here you can deploy to Railway on your personal account. https://github.com/ralaruri/object-storage-railway

Google Cloud setup

We will create a project & service account in order to give our code access in our project to create, write, delete and update buckets along with data.

1. Create a new Project or use a current project you have in the UI. I created a Temporary project for this guide called object-storage-railway
   

2. Go to IAM & Admin Panel and create a service account. Click + CREATE SERVICE ACCOUNT
   

3. Name the Service account whatever you like mine is named: bucket-creator-and-writer
   

4. Add Permissions to your service account:(Remember to give it the least amount of access needed i.e. reading, writing and creating buckets) In this guide I will keep it simple and give it storage admin access. This allows it to create, update, read, write and delete storage. Go to Permissions
   

Then add storage admin access.

1. Create a private JSON key and download it locally. Do not commit this publicly to your github or share this anywhere what I am sharing in this guide is just a test account I have deleted these and are no longer valid!

Setting up the Service Account Locally or on Railway.

1. Convert your JSON keyfile into a base64 key. using this function in your terminal cat object-storage-railway-f5a8a25dd590.json|base64 it will return the base64 key Again don't expose this publicly this is a demo account for the purpose of this blog post and has already been deleted. Save the output

This allows us to keep the code in our .env locally or inside of Railway's variable storage for the project and decode it as needed at runtime.

base64 decoding in Go part of the bucket_creater.go file.

func CovertStringToJSON(env_details string) []byte {
    decoded_json, err := base64.StdEncoding.DecodeString(env_details)
    if err != nil {
        panic(err)
    }

    return decoded_json

}

1. For your project you can access variables here we will add the encoded base64 key as a variable in our Railway Project.

Adding Key

Key added

Additionally you will want to create a variable called ENV=PROD this allows us to use Railway as our production env and locally we can use our .env if we choose to.

Understanding the Code:

1. Establishing a connection to GCP We are using the Google Cloud library in Go to create a bucket depending if you are running locally or on Railway we use either the .env you have local variables or the variables on Railway. This allows us to test locally if we need to.

This client is the underlying method to use any GCP service in our case we are using to create, read and write to a bucket.

in create_bucket.go:

func clientConnection() *storage.Client {
    ctx := context.Background()

    // Set up credentials to authenticate with Google Cloud
    var bucketVar string

    if env := os.Getenv("ENV"); env != "PROD" {
        bucketVar = utils.GoDotEnvVariable("./.env", "BUCKET_CREATOR")
    } else {
        bucketVar = os.Getenv("BUCKET_CREATOR")
    }

    new_creds := CovertStringToJSON(bucketVar)
    creds := option.WithCredentialsJSON(new_creds)

    // Create a client to interact with the Google Cloud Storage API
    client, err := storage.NewClient(ctx, creds)
    if err != nil {
        log.Fatalf("Failed to create client: %v", err)
    }

    return client

}

1. Creating a Bucket

We can now use the client connection to create a bucket just providing the bucket name and the project we are using.

in create_bucket.go:

func CreateBucket(projectID string, bucketName string) {

    ctx := context.Background()

    client := clientConnection()

    // Create a new bucket in the given project with the given name if the bucket
    // already exists then it will just print (bucket already exists)
    if err := client.Bucket(bucketName).Create(ctx, projectID, nil); err != nil {
        fmt.Printf("Failed to create bucket %q: %v", bucketName, err)
    }

    fmt.Printf("Bucket %q created.\n", bucketName)
}

in main.go:

buckets.CreateBucket("object-storage-railway", "food-bucket-dev")

1. Reading and Writing Data

We have two operators we care about writing and reading. So the two functions take the name of the bucket and the file path of the data you are writing to the bucket. You can change the path of the file you want to write in the bucket it does not need to match the file path you have when reading the file.

in bucket_operator.go:

func WriteToBucket(bucket_name string, file_path string) {

    ctx := context.Background()

    // Set up credentials to authenticate with Google Cloud

    client := clientConnection()

    bucketName := bucket_name
    filePath := file_path

    file, err := os.Open(filePath)

    if err != nil {
        log.Fatalf("Failed to open file: %v", err)

    }

    defer file.Close()

    // timestamp the file
    today := time.Now().Format("2006_01_02")

    // write the date part of the file path name
    filePath = fmt.Sprintf("%s_%s", today, filePath)

    // Create a new writer for the file in the bucket
    writer := client.Bucket(bucketName).Object(filePath).NewWriter(ctx)

    // Copy the content
    if _, err := io.Copy(writer, file); err != nil {
        log.Fatalf("Failed to write file to bucket: %v", err)
    }

    // Close the writer to flush the contents to the bucket
    if err := writer.Close(); err != nil {
        log.Fatalf("Failed to close writer %v", err)

    }

    log.Printf("File %q uploaded to bucket %q. \n", filePath, bucketName)

}

Note the date is currently hardcoded and will need to be changed.

in main.go:

buckets.WriteToBucket("food-bucket-dev", "cheese.json")
buckets.ReadFromBucket("food-bucket-dev","2023_03_12_cheese.json")

1. Creating Mock Data:

The purpose of this is jsut to create some mock data to use for the purpose of the project. we create a json of different cheeses.

// write the cheeseMap to a json file
func WriteCheeseMapToJSONFile() {
    cheeseMap := map[int]string{100: "chedder", 200: "swiss", 300: "gouda", 400: "mozzarella"}
    // create a file
    file, err := os.Create("cheese.json")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    // create a json encoder
    encoder := json.NewEncoder(file)
    encoder.SetIndent("", "  ")

    // write the cheeseMap to the file
    err = encoder.Encode(cheeseMap)
    if err != nil {
        log.Fatal(err)
    }
}

func DeleteJsonFile() {
    err := os.Remove("cheese.json")
    if err != nil {
        log.Fatal(err)
    }
}

Deploying and Checking out Deployment

As mentioned before deploying in Railway is as simple as PR push or manually clicking the deploy button on a project.

From there your deployment will run and you can check the logs that everything ran successfully.

Logs should show the JSON and the bucket creation:

And we can check in GCP in the cloud storage panel which looks like it was a success!

Conclusion

Now we have established a way to setup Object storage that you can use with you Railway project. I created this for a personal project I am working on where I needed to store some files. As I mentioned before I am a huge fan of Railway and looking forward to creating more content and sharing my next personal project with everyone on Railway!

If you enjoyed this post let me know! I am happy to create more guides using different languages (Python, Typescript, even Rust!)