Don't keep me waiting

The Problem

We were working on this project where we had to upload n products to an ecommerce application. Sure it isn't such a big deal. The catch was we had to scrape the data from another place and upload to another place. And for some reason the upload APIs were damnnnnnn slow.

Our UI was the simplest. It had a white screen with a button that said "Scrape & Up". Little did the client know so many things happened behind it. Yes am an obsessed backend dev. Anyways the problem was the user experience. We couldn't just show a loader to the user while the products were uploaded. At any moment we might have from 10 to 10,000 products. If we decided to show a loader for 10k product upload, there was high chance the user might get hypnotized by looking at the loader.

The somewhat solution ...

So here is what we decided.

1. We implemented a queue on the server. A queue as we all know is just a FIFO structure. You add some task at the back and pick ones from the front.
2. We decided whenever the user clicks the button, we will create a job and add it to the queue. We returned the job details to the front-end in response.
3. On the front-end we started showing a nice little animation of a progress bar.
4. We made another endpoint which basically gave the status of a particular job. This endpoint told the browser how to update the progress bar component.

And voila we were done. The user didn't have to look at the loader anymore. Sure we didn't speed up the upload process that is for sure. But we somehow managed to allow the user to do something else while the products were being uploaded.

At times that is a win situation too :'(

The showoff !!!

Oh and in case you guys are wondering how we did all of that, here is some sample code.

1. We used the bull queue of nestjs. Please go through this link. Its pretty simple if you have already worked with Nest.
2. The only thing we added to the above link was our endpoint

@Get("products/progress")
async fetchProductUploadProgress(@Query() query: Progress)
{
    return await this.NetoService.fetchNetoUploadProgress(query.jobId);
}

1. The service for the above controller

async fetchNetoUploadProgress(jobId)
{
    const jobDetails = await this.NetoQueue.getJob(jobId);

    const response =
    {
        id: jobDetails.id, 
        isActive: await jobDetails.isActive(), 
        state: await jobDetails.getState(),
        progress: await jobDetails.progress(),
        end: jobDetails.data["end"]
    };

    return response;
}

1. The response was what the front-end kept checking at intervals. The progress is basically an integer value which ran till end which is again an integer. The progress-bar component just kept incrementing itself based on this "progress number" and once done, it showed the user "Hoorraayyy"!!!

2. The best part of this entire exercise is bull queues use redis by default and hence even if your project crashes for some reason, the queue persists.i.e. next time you start your project the queue starts from where you left it.

Do leave your comments.

Happy programming !!!