Full Stack Search with Algolia, Nextjs and MongoDB | Part 1

Today, you will learn how to build a search engine using Algolia in Next.js. This is part 1 where we will handle the backend part. In part 2, we will handle the frontend part.

Table of Contents

• What is Algolia?
• How Algolia Works?
• Syncing Data with Algolia
• Setup Database
• Setup Algolia
• Listen for Changes
• Add a record to Algolia
• Update a record in Algolia
• Delete a record in Algolia
• Add existing data to Algolia

What is Algolia?

Algolia is a search engine API that allows you to build a fast and relevant search experience for your users. It provides a powerful search engine that can be easily integrated into your application. You might have seen it on documentation websites like React, Tailwind, Redux, etc.

I recommend checking the video tutorial on my youtube channel to better understand.

How Algolia Works?

A regular query to a database can be slow and inefficient, especially when dealing with large datasets.
Algolia works by indexing your data. Indexing means storing some additional data about your data in the Database. This additional data is optimized for search queries, making it much faster and more efficient than a regular query.

To do indexing, Algolia needs your data from the Database. So, you need to copy the relevant data from your Database to Algolia. Never copy secret data like passwords, credit card numbers, etc.

Records: A record is a single piece of data that you want to search for, such as a product, a user, etc. Think of it as a document in MongoDB or a row in an SQL database.

Index: It is a collection of records. Think of it like a table in an SQL database or a collection in MongoDB. An Algolia project can have multiple indexes, such as users_index, products_index, etc.

Syncing Data with Algolia

Since you are copying data from your Database to Algolia, you need to keep the data in sync. This means that whenever you add, update, or delete data in your Database, you need to update the corresponding data in Algolia. Otherwise you will have outdated/wrong search results. You can do this by:

Cron Job: You can run a cron job that periodically syncs the data between your Database and Algolia.
I have a detail video on Cron Jobs in case you are interested.

Manual Sync: You can manually sync the data whenever you make changes to your Database. Let's say you have a function updateData the modifies the data in your Database. YOu can call another function that will apply the same changes to Algolia. You must do the same thing for every function that modifies the data.

Event-Driven Sync: You can listen for changes in your Database. Every time you create, update, or delete, a function will be triggered. There, you apply the same changes to Algolia. This will depend on the DB. You can use ChangeStream in MongoDB. We are going to use this approach. But I haven't found a way to do this on Nextjs, so I will use a separate node server for that.

Setup Database

You can check the following guide to setup mongodb.

Setup Algolia

• Create a new account on Algolia
  • A new application will be created asking you to add data. You can skip that for now.
• Get keys:
  • Go to API keys page
  • Copy application_id and admin_key. Admin key is secret so only use it on server and never expose.

Listen for Changes

Let's create a simple node server with express:

const { searchClient } = require('@algolia/client-search')
const mongoose = require('mongoose')
const express = require('express')
require('dotenv').config()

const app = express()

const client = searchClient(
  process.env.ALGOLIA_APP_ID,
  process.env.ALGOLIA_ADMIN_KEY,
)

mongoose
  .connect(process.env.MONGO_URI, {})
  .catch(console.error)
  .then(() => console.log('Connected to MongoDB'))

// A dummy model
const User = mongoose.model('User', new mongoose.Schema({}))

const changeStream = User.watch()

const USERS_INDEX = 'users_index_yt'

changeStream.on('change', async data => {
  console.log(data)

  const {
    operationType,
    fullDocument,
    documentKey: { _id },
  } = data

  const documentId = _id.toString()

  switch (operationType) {
    case 'insert':
      break

    case 'update':
      break

    case 'delete':
      break

    default:
      break
  }
})

app.listen(8000, () => {
  console.log('Server running on port 8000')
})

Explanation:

• Create a searchClient with Algolia keys.
• Connect to MongoDB and create a dummy User model.
• Create a changeStream on the User model. This will listen for changes in the User collection.
• Listen for changes using the on method and pass the change string. Then, pass a callback function that will be called whenever there is a change in the User collection.
• data will contain the change information. We are interested in operationType, fullDocument, and documentKey.

Add a record to Algolia

const addToIndex = async data => {
  try {
    const { _id, name, email } = data

    const userObject = {
      objectID: _id.toString(),
      name,
      email,
    }

    await client.saveObject({
      indexName: USERS_INDEX,
      body: userObject,
    })
  } catch (error) {
    console.log(error)
  }
}

switch (operationType) {
  case 'insert':
    await addToIndex(fullDocument)
    break
}

Explanation:

• We are extracting the _id, name, and email from the fullDocument.
• We are creating a userObject with the objectID, name, and email.
• Then, we call the saveObject method on the client object. We are passing the indexName and the body object. saveObject method will add a new record to the index if there is no record with the same objectID. If there is a record with the same objectID, it will replace the record. Also if the indexName doesn't exist, it will create a new index.

Update a record in Algolia

const updateIndex = async (id, data) => {
  try {
    const { password: _, ...updatedFields } = data

    await client.partialUpdateObject({
      indexName: USERS_INDEX,
      objectID: id,
      attributesToUpdate: updatedFields,
    })
  } catch (error) {
    console.log(error)
  }
}

switch (operationType) {
  case 'update':
    const {
      updateDescription: { updatedFields },
    } = data
    await updateIndex(documentId, updatedFields)
    break
}

Explanation:

• We are extracting the updatedFields from the data. Only the updated fields will be present.
• Then call the partialUpdateObject method on the client object. Pass the indexName, objectID, and attributesToUpdate.

Delete a record in Algolia

const deleteFromIndex = async id => {
  try {
    await client.deleteObject({
      indexName: USERS_INDEX,
      objectID: id,
    })
  } catch (error) {
    console.log(error)
  }
}

switch (operationType) {
  case 'delete':
    await deleteFromIndex(documentId)
    break
}

Explanation:

• Call the deleteObject method and use the document id as objectID.

Add existing data to Algolia

You can add a simple script to add existing data to Algolia:

const processRecords = async () => {
  await mongoose.connect(MONGO_URI)

  console.log('Connected to MongoDB')

  let users = await User.find({}, 'name email')

  const usersObject = users.map(user => {
    const { _id, ...restUser } = user.toObject()

    return {
      objectID: _id.toString(),
      ...restUser,
    }
  })

  return await client.saveObjects({
    indexName: 'users_index_yt',
    objects: usersObject,
  })
}

Explanation:

• Get all the data from DB
• This time, use the saveObjects method to add multiple records at once. It is the same as saveObject, but you can pass an array of objects.
• If adding a record fails, no other records will be added. So, you won't have partial data in Algolia.

That's it. In part 2, we will learn how to add a search bar in Next.js and use Algolia to search the data.

Thank you for reading. If you have any questions, please leave a comment below.
Make sure to check out my YouTube channel and subscribe for more tutorials! 🎥✨