Your files go to Cloud : Uploading Images with Express to Cloudinary

As soon as your website or app reaches a point when you realize you cannot always display static or dummy images among all the dynamic content fetched from your backend, you start looking for ways to integrate image uploads in your project. After looking around on the internet for a while, you begin to hear some names and if you aren't willing to let AWS have your card details, you will fall back to Cloudinary. Now, Cloudinary is quite convenient for small project and specially when you are still studying. But last time I tried getting it set up in my project, it wasn't the most "copy-from-docs and go" set up. Along with Cloudinary's own configuration, an important aspect is how you actually bring your files to Cloudinary's API for the upload.

In this one, we will be implementing image uploads to cloudinary wherein we will demonstrate it using a form to set profile pictures for the user. Our focus here would be on how we will handle the file so that it can be uploaded. I will use a node.js backend with authentication already implemented. In the client, I will use a form that handle user's profile information but we mostly care about the profile picture.

Preface

I will be going in depth in this one. If you need a to-the-point solution to how you can get image upload set-up with cloudinary you can skip to In short

Let's get started

So, I am trying to add a profile picture to the users in my database by uploading them to cloudinary. I would have a form in the front end, I would use a file input to select a file and then it would send this file to the backend where it would be uploaded to cloudinary. Now, these things come with hidden complexities at each steps. The file input gives us a Javascript File Object which the database or cloudinary doesn't really like or understand. We need raw image data or maybe a url. Also, JSON wouldn't send images as it is basically just fancy text. Even if we do manage to send these File Objects to our endpoint, this file object would not be enough to upload the image to cloudinary. Let's test each of these using an express server and a mongodb database.

Situation

I have a user collection in my local mongodb and I have added one user to it manually which looks like the following:

blogdb> db.users.find()
[
  {
    _id: ObjectId("629f08b134f91b645470ca97"),
    username: 'suyious',
    name: 'Suyash Kumar',
    email: 'solo@gmail.com',
    password: '$2b$10$eT7nsqWx3jVjlH9xbHOU2.EcowoHlXf8VWMldb2klFKL2CzxSeC5a',
    role: 'user',
    createdAt: ISODate("2022-06-07T08:13:37.436Z"),
    __v: 0,
    avatar: {
      public_id: 'xetra/products/vunlvjcijttmyu6aah1o',
      url: 'https://res.cloudinary.com/djv5txrzp/image/upload/v1667916852/xetra/products/vunlvjcijttmyu6aah1o.jpg'
    }
  }
]

This should give an idea of how the user collection looks like. We have an avatar field which is an object with a key url. Now I would like to change this avatar field and by changing it, I imply sending an image from my frontend to the server which would then get uploaded to cloudinary and the avatar field would have it's new url. Now we can always manually upload our images to cloudinary and just send it's url through a JSON, but we need that to happen on it's own and we should just need to use the html file input.

Sending the image from the frontend

I guess even now, you would have started getting a hint that JSON would probably not send our file object to our server. But, let's test that anyway. All we need is an <input type="file"/> and an axios.put('/upload', body). When we are sending the image from the frontend, the request body looks like the following in Javascript:

Looks fine for now but as soon as this request reaches the backend, thing begin to look different.

Clearly, JSON doesn't seem to know what a file exactly is. Guess, it's time to run to stack overflow.

Turns out, apart from application/json, there are other content-type for your request body and one of them is mutipart/form-data. In short, this is what we use when we have files along with text to send to server. Looks like that should solve our problem and the File would show up in our backend.

multipart/form-data

In order to send multipart/form-data through an HTMl form, you need <form enctype='multipart/form-data'> tag and when using axios, you need axios.put('/put', body, { "Content-Type": "multipart/form-data" }). You can also create a FormData Object in Javascript and set key/value pairs to construct your request body.

So, now we should just try sending form-data from our frontend similar to before, only this time our request body would be constructed using:

const body = new FormData(formElement);

The formElement is a reference to am html form which can be the name of a form ( or ref.current when using React.useRef)

In the frontend, the request body looks like this:

But now, we turn to our server only to see an even more obscure output.

We should also try to send the same form-data using postman just in case we are making some mistake in our frontend code.

Clearly, even postman gives us a similar output. It seems like the backend isn't able to understand what we are throwing at it. Hmm. That doesn't happen often. Or does it? One might begin to remember that we couldn't earlier even read the JSON output if we just threw it at the backend. If that didn't ring a bell, I am talking about the express.json() or earlier bodyparser() middlewares that we needed to parse our JSON requests. It is fairly obvious that we would also need to parse form-data before our backend can understand it.

Now, there are multiple similar libraries for accomplishing this and the most famous one that I can find is multer. Naturally, we would start by installing it and including it in our server. We could add the middleware for every route similar to express.json() with express.use(), but we would want to only add it for the routes that actually need form data.

multer

So, if we go forward, install multer and add a middleware multer().single('avatar') (see docs) to our upload route, we find this in our console:

Looks like our form-data did pass through. But where is the file. Turns out the files are not a part of req.body. Instead they are in req.file.

We can finally see our file reach the backend. This looks good. Now all we have to do is throw this file at Cloudinary and it should just work out of the box. Right? Well, we all know our history with throwing things around here.

Cloudinary

Now, let's install cloudinary and set it up with the configurations. You need following environment variables for the config:

cloudinary.config({
  cloud_name: process.env.CLOUDINARY_NAME,
  api_key: process.env.CLOUDINARY_API_KEY,
  api_secret: process.env.CLOUDINARY_API_SECRET
})

Above is also the code that you need to add to your server for initial configuration of cloudinary. Once this is done, you should be able to talk to your cloudinary account. ( If it wasn't obvious, you need to create an account. It won't charge you or ask for card details. You will get your api_key and api_secret from the dashboard

Now, as from the docs, we need the following to upload our image:

import { v2 as cloudinary } from 'cloudinary'
cloudinary.uploader
  .upload(file, options)
  .then(callback);

Once we are set up, we should try to use the file we received from the frontend on the upload function provided by the api. I have the following code added to my upload route:

cloudinary.uploader.upload(req.file)
  .then((result => {
    console.log(result);
  }))

This time, the server gives us a new error. Depending on how your server is set up, the error might show up at different places. As for me, the error comes as a 500 Error Response.

The error message tells us that cloudinary's API does not accept a Javascript File object and instead is asking for some string. If we look into the docs again, we find the following:

Given above, the upload function takes only the allowed types as the file parameter. Out of these, we do not have a local file path in the server as the file comes from the frontend. Neither do we have a file URL or S3 bucket URL. We are left with using a byte-array buffer or a Data URI (Base64 encoded).

As we can see above, we do seem to be getting a buffer key in the file object received. Let's inspect it's value and see if it can be of our use. If we just plug the buffer value to the upload function, the cloudinary api still cannot process it and gives a similar error.

We could try to process this byte array and make it work with the API but if we look at multer's documentation, we notice using the buffer option requires storing the images into memory which could be bad for the server in terms of performance.

Now, we are left with either trying to convert the file into a byte-array in the frontend or using the data URI option. Also, we should try to do the computations in the frontend itself as much as possible, now that we are reminded.

If we dig around a little to find some information about byte-array or Data URI, we come across the FileReader object provided by Javacript's File API. This object provides functions for dealing with both byte-arrays and Data URIs.

FileReader

This FileReader object delivers the data using events, as reading from disk may take time. The Syntax for using it is generally of the form:

const reader = new FileReader();

reader.onload = () => {
  <your-processing-function>(reader.result);
}

reader.<instance function>(blob);

We should try the readAsDataURL(file) function to send the image file as a Data URI. Note that the function is readAsDataURL and NOT readAsDataURI. The result of this function call returned through reader.result can also be used with <img src=""/> to preview the image. The following code demonstrate how we would process the file before sending it.

const reader = new FileReader();
reader.onload = () => {
  console.log(reader.result);
}
reader.readAsDataURL(input.files[0])

Here, input may be e.target if you are listening to events on a file input or inputref.current if inputref is a ref to an input element when using React.useRef. The resulting Data URI is returned as a string through reader.result. Now, that this is a string, it should work as an argument to cloudinary API's upload function.
The result looks like the following:

It is basically a bunch of gibberish to us but will represent our image file to the server. When we send this new Data URI string to the server, it gets sent as is, that is as a string. But now that it is no longer a file. It is part of req.body itself and not of the req.file

Now, we could plug this string into cloudinary API's upload function and see if it works:

cloudinary.uploader.upload(req.body.avatar)
  .then((result => {
    console.log(result);
  }))

This, after all this work, actually works and we get the following result in our server console:

We can go to the provided secure url and it is actually uploaded to cloudinary. We can also see this image in our cloudinary media library. Here, I have used the folder option that can be provided in the second argument of the upload function as follows:

const result = await cloudinary.uploader
  .upload(req.body.avatar, {
    folder: "/fraise/avatars"
  })

The API returns an object with a url for the uploaded image as secure_url, which can now be stored in the database to update our user profile.

const user = await User.findByIdAndUpdate(req.user.id, req.body, {
  new: true,
  runValidators: true,
  useFindAndModify: false
})

Here, my server is setting req.user while validating authentication, though it is only the providing the id of the entry to update in our database. This updates the current user's profile picture.

In short

In short, all I did all this while was:

• send files as form-data and not JSON.

here you might need:

<form enctype='multipart/form-data'>

axios.put('/put', body, { "Content-Type": "multipart/form-data" })

• receive form-data using multer.

you might need to install multer and add the middleware:

multer().single('name')

for other usages, here's the docs

• setup cloudinary and upload

cloudinary.config({
  cloud_name: process.env.CLOUDINARY_NAME,
  api_key: process.env.CLOUDINARY_API_KEY,
  api_secret: process.env.CLOUDINARY_API_SECRET
})

import { v2 as cloudinary } from 'cloudinary'
cloudinary.uploader
  .upload(path, options)
  .then(callback);

• send files as dataURI for the path in upload() above:

const reader = new FileReader();
reader.onload = () => {
  console.log(reader.result);
}
reader.readAsDataURL(input.files[0])

But this makes me think

Even though we used multer to be able to read multipart/form-data in the server, we don't quite seem to have used it for actual file transfer between the frontend and the backend. It makes me wonder if using form-data was for nothing. I did try to use JSON for the same but it didn't quite seem to work. Now, I didn't want to pull apart all that code again. So, I urge you to try it.

Additionally, the data URI seems to be a very long string which might be too long to send back and forth. Tell me if there are other simpler ways to upload the images. I do know of another method, creating a temporary file on the server and using it's file path for the upload. This would require creating and removing files on the server.

I would be really happy if I am pointed to a better direction here, but if my solution was helpful to you, that would make me really happy too.