DX_1: How we created an Optimized Search for our Faculty's Career Fair Website

Last Update - 19th Feb, 2024 | 1347

DX_ Series

This is an article of my DX_ series where I post my developer experience one article at a time.

Introduction

The annual career fair organized by our Faculty of IT, University of Moratuwa stands as a vital bridge connecting students with top industry professionals. 2024 FIT Future Careers Career fair was held on 8th of February,2024.

Recognizing the need to enhance this connection, the student body INTECS responsible for the event has launched a dedicated website. However, the site's search feature initially fell short of expectations.

In this article, I'll share how I tackled these challenges to refine the search function, ensuring the website effectively matches the best talents with the greatest opportunities.

What was the problem?

Upon joining the career fair website team, my main focus was on deployment. However, I quickly identified a major flaw in the website's search functionality.

Originally designed to support 300 students, the site inefficiently fetched all student records in one API call for every load of the search page, performing the filtering on the client side.

This not only strained the website's performance but also risked overloading the API server, given our limited server resources. For instance, if even if the user views only the first page of the results, all 300 student details are already loaded, which is a huge overhead.

We faced a critical need for a more efficient search solution that balanced functionality with our budget constraints.

My thought pattern

Facing the search functionality issues on our career fair website, I recalled using Elasticsearch in a past project for efficient search and filtering. This experience inspired me to apply a similar solution, but cost was a concern due to our tight budget.

I explored open-source options, finding Typesense, a cost-effective, typo-tolerant search engine offering fast, relevant search capabilities and a managed cloud service with a free tier.

Despite initially considering self-hosting, their managed cloud service - Typesense Cloud proved to be the most practical choice, allowing us to enhance search efficiency without financial burden or extensive setup.

A little bit of problematic implementation

Earlier implementation looked something like this.

You can see when the Student.jsx component of the student search page is loaded, it loads all the users from the database via the API, which is not good.

useEffect(() => {
  axios.get(`user/get-all-users`).then((res) => {
    setAllUsers(res.data.data);
  });
}, []);

The data returned from the API endpoint are then used to update the state variable allUsers.

const [allUsers, setAllUsers] = useState([]);

The users were filtered using a simple JS filter() method. This frontend filter doesn't do much other than just simply filtering the already-retrieved database data.

This doesn't provide the performance improvements a dedicated search system would provide, such as typo-tolerance etc. multiple search criteria etc.

const filterUsers = (users, query, selectedAreas) => {
  return users.filter(user => {
    const fullName = `${user.f_name || ''} ${user.l_name || ''}`.trim().toLowerCase();
    const isNameMatch = fullName.includes(query.toLowerCase());
    let preferredAreas = [];
    try {
      preferredAreas = JSON.parse(user.preferred_area || '[]');
    } catch (e) {
      console.error('Error parsing preferred_area:', e);
    }
    const isAreaMatch = selectedAreas.length === 0 || preferredAreas.some(area => selectedAreas.includes(area));
    return isNameMatch && isAreaMatch;
  });
};

The solution

To improve the search functionality without changing the UI, I chose Typesense for its high performance and React compatibility. Typesense offers fast, typo-tolerant searches, ideal for instant feedback on user queries.

I integrated it using React InstantSearch components & hooks, adapting them for Typesense to enhance search without altering the UI. The setup involved initializing a Typesense server for the search index, facilitated by Typesense’s clear documentation.

This allowed for a smooth transition to Typesense’s optimized search components, ensuring easy customization and adaptability to our UI needs. Also automatic pagination and filter components are provided with this library.

A Node.js program was developed to create the indexes of the fields using which we needed to filter the students contained the schema definitions as below.

// Define schema for the 'students' collection
const schema = {
    name: "students",
    num_documents: 0,
    fields: [
        { name: "uni_index", type: "string", facet: false, optional: true },
        { name: "f_name", type: "string", facet: false, optional: true },
        { name: "l_name", type: "string", facet: false, optional: true },
        { name: "profile", type: "string", facet: false, optional: true },
        { name: "preferred_area", type: "string[]", facet: true, optional: true },
        { name: "skills", type: "string[]", facet: true, optional: true },
        { name: "cv_link", type: "string", facet: false, optional: true },
        { name: "front_picture_link", type: "string", facet: false, optional: true },
    ],
};

The following is how the indexes look in the managed cloud server.

Then in the react frontend application, we consumed the search and configured which fields and the weights we need to assign to each field during the searching.

export const typesenseAdapter = new TypesenseInstantsearchAdapter({
  server: TYPESENSE_SERVER_CONFIG,
  additionalSearchParameters: {
    queryBy: "f_name,l_name,skills,preferred_area,uni_index",
    query_by_weights: "3,3,2,2,1",
    numTypos: 3,
    typoTokensThreshold: 1,
  },
});

I tackled the challenge of integrating Typesense’s search components without altering the existing design by customizing React InstantSearch components to match the application's aesthetic.

By applying custom styles and utilizing modified hooks like a debounced useSearchBox and an adapted useHits, I ensured the search functionality blended seamlessly and efficiently with the UI.

This resulted in an enhanced user search experience—faster and more accurate—without disrupting the design continuity. The successful integration underscored my ability to improve functionality while maintaining the visual integrity of the application.

Who keeps the search index updated? - Cron!

The above mentioned Node.js application was deployed as a Cronb job to automate updating the Typesense search index, ensuring our database records were consistently synchronized for reliable search functionality.

This application, designed to run as a cron job, leverages Node.js's strengths in asynchronous operations and database compatibility, making it an efficient solution for data indexing.

It periodically (every 15 minutes) checks for new or modified database entries, formats them for Typesense, and uploads them to the search server on managed Typesense instance.

Deployed as a serverless function on Digital Ocean using do-functions and do-functions-server packages, this setup streamlines the indexing process, maintaining the search feature's efficiency and reliability. Otherwise even if the students update their details and save to database, those updated details would not be returned in search results without being indexed.

Conclusion

In this instalment of DX_ series, I shared how we upgraded our faculty career fair website's search function by shifting from basic front-end filtering to an advanced, type-ahead search with Typesense.

This cost-effective improvement was achieved by integrating Typesense's search engine and automating data indexing through a custom Node.js cron job.

The result was a significant enhancement in connecting students with employers, without altering the user interface, demonstrating the impact of open-source solutions and innovation in addressing project challenges and user requirements.

References

• https://github.com/typesense/typesense-instantsearch-demo
• https://cloud.typesense.org/
• https://lucene.apache.org/core/
• https://www.digitalocean.com/products/app-platform

BONUS: Tech stack for the whole project

• Frontend - React.js
• Backend - Node.js
• Database - MySQL
• Search - Typesense
• Image storage - Firebase storage
• User tour - driver.js

Disclaimer

• Although I contributed to the search feature, I was just one member of the whole project. The whole project was a successful effort of many undergraduates of the Faculty from different batches.
• Code snippets and screenshots in this article may not be production-ready. They are edited as needed for educational purposes.