lonka pardhu

Posted on Oct 19

I Fumbled on a Next.js MongoDB Error and Learned the Key Differences Between Mongoose and MongoClient

#webdev #mongoose #mongoclient #mongodb

Recently, while building a Next.js application with MongoDB, I stumbled upon an error⚠️ that stumped me for a while

MongooseError: Operation `spaces.insertOne()` buffering timed out after 10000ms

It was confusing because my connection seemed fine, and yet, I couldn't insert a document using Mongoose. This error taught me some valuable lessons about how Mongoose and MongoClient (MongoDB's native driver) handle database connections in different ways. Let’s dive into what happened, how I fixed it, and the key differences I learned between these two.

The Setup

Using MongoDB with Next.js

I had a Next.js app where users could create "spaces" (think of them as categories or containers for user data). I wanted to store these spaces in MongoDB, and I had two ways to interact with the database:

MongoClient (MongoDB's native driver): Offers low-level database interaction.
Mongoose: An Object Data Modeling (ODM) library that provides a schema-based solution for MongoDB. It’s great for validating, organizing, and manipulating data models.

My Initial Approach

In my Next.js project, I was using NextAuth.js (now Auth.js) for authentication and MongoDB for database storage.

//MongoDB connection file (db.ts):

import { MongoClient, ServerApiVersion } from 'mongodb';

if (!process.env.MONGODB_URI) {
  throw new Error('Invalid/Missing environment variable: "MONGODB_URI"');
}

const uri = process.env.MONGODB_URI;
const options = {
  serverApi: {
    version: ServerApiVersion.v1,
    strict: true,
    deprecationErrors: true,
  },
};

let client: MongoClient;

if (process.env.NODE_ENV === 'development') {
  let globalWithMongo = global as typeof globalThis & { _mongoClient?: MongoClient };
  if (!globalWithMongo._mongoClient) {
    globalWithMongo._mongoClient = new MongoClient(uri, options);
  }
  client = globalWithMongo._mongoClient;
} else {
  client = new MongoClient(uri, options);
}

export default client;

thats just as it is from the official docs of authjs Read here

In the route handler, I tried using Mongoose to create a new document, while also managing the connection with MongoClient:

import Space from '@/models/space.model';
import client from '@/lib/db'; // MongoClient connection

export async function POST(req: Request) {
  const session = await auth(); // Authentication middleware

  if (!session || !session.user) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
  }

  try {
    const { spaceName, title, message, questions } = await req.json();

    if (!spaceName || !title || !message || !questions || !questions.length) {
      return NextResponse.json({ message: 'All fields are required' }, { status: 400 });
    }

    await client.connect(); // MongoClient connection

    const space = await Space.create({
      spaceName,
      spaceOwner: session.user.id,
      title,
      message,
      questions,
    });

    return NextResponse.json({ message: 'Space created successfully', spaceId: space._id }, { status: 201 });
  } catch (error) {
    return NextResponse.json({ message: 'Failed to create space' }, { status: 500 });
  }
}

This is when I hit the following error⚠️:

MongooseError: Operation `spaces.insertOne()` buffering timed out after 10000ms

The Root Cause

After some digging, I realized that I was mixing up two different database management systems:

MongoClient: This is the native MongoDB driver that works directly with MongoDB by establishing a raw connection to the database.
Mongoose: An ODM that sits on top of MongoClient and provides schema-based models to interact with MongoDB.

The Problem

MongoClient and Mongoose are not interchangeable. In my code, I was using client.connect() (MongoClient) to establish the connection but trying to interact with Mongoose models (Space.create()).
Mongoose manages its own connection pooling and lifecycle. So, when I used client.connect() from MongoClient, Mongoose didn’t know about this connection, causing it to wait (or "buffer") until its own connection was established. This led to the buffering timeout error: Operation 'spaces.insertOne()' buffering timed out after 10000ms.

Why It Worked with MongoClient but Not Mongoose

When I switched to the following code, it worked without issue:

const db = (await client.connect()).db();

const space = await db.collection("spaces").insertOne({
  spaceName,
  spaceOwner: session.user.id,
  title,
  message,
  questions,
});

Here, I was using MongoClient's native API directly, which correctly handled the connection because I used client.connect() to establish the connection and then interacted with the MongoDB collections.
MongoClient doesn't depend on Mongoose's connection pool, so it worked as expected.

Why Mongoose Failed

await client.connect(); // MongoClient connection

const space = await Space.create({
  spaceName,
  spaceOwner: session.user.id,
  title,
  message,
  questions,
});

Mongoose was still in a buffering state because I hadn't initialized a Mongoose connection. Mongoose waits for its own connection before performing any operations on the model, which led to the timeout error.

Fixing the Issue: Properly Managing Mongoose Connections

To fix the issue, I had to properly establish a Mongoose connection and let Mongoose handle the database connection itself instead of using MongoClient. Here’s what I did:

Correct Mongoose Connection Setup (db.ts):

import mongoose from 'mongoose';

const MONGODB_URI = process.env.MONGODB_URI;

if (!MONGODB_URI) {
  throw new Error('Please define the MONGODB_URI environment variable');
}

let cached = global.mongoose;

if (!cached) {
  cached = global.mongoose = { conn: null, promise: null };
}

async function connectToDatabase() {
  if (cached.conn) {
    return cached.conn;
  }

  if (!cached.promise) {
    cached.promise = mongoose.connect(MONGODB_URI, {
      useNewUrlParser: true,
      useUnifiedTopology: true,
    }).then((mongoose) => {
      return mongoose;
    });
  }
  cached.conn = await cached.promise;
  return cached.conn;
}

export default connectToDatabase;

Updated Route Handler Using Mongoose:

import Space from '@/models/space.model';
import connectToDatabase from '@/lib/db'; // Mongoose connection

export async function POST(req: Request) {
  const session = await auth(); // Authentication middleware

  if (!session || !session.user) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
  }

  try {
    const { spaceName, title, message, questions } = await req.json();

    if (!spaceName || !title || !message || !questions || !questions.length) {
      return NextResponse.json({ message: 'All fields are required' }, { status: 400 });
    }

    await connectToDatabase(); // Use Mongoose connection

    const space = await Space.create({
      spaceName,
      spaceOwner: session.user.id,
      title,
      message,
      questions,
    });

    return NextResponse.json({ message: 'Space created successfully', spaceId: space._id }, { status: 201 });
  } catch (error) {
    return NextResponse.json({ message: 'Failed to create space' }, { status: 500 });
  }
}

Key Takeaways

Mongoose and MongoClient are not interchangeable:

MongoClient: Best for low-level, direct interaction with MongoDB.
Mongoose: Provides schema-based modeling and is more suitable for structured data interactions, but you need to establish its own connection using mongoose.connect().

Use Mongoose with Mongoose models:

If you're using Mongoose models like Space.create(), make sure you initialize the connection with Mongoose, not MongoClient.

Connection pooling:

Mongoose manages its own connection pooling. If you manually manage connections with MongoClient, Mongoose will not automatically know about this connection, leading to issues like buffering or timeouts.

Global connection caching:

In Next.js, using global variables to cache your MongoDB connection (whether through MongoClient or Mongoose) is important in development mode to prevent creating multiple connections during hot module reloads.

Conclusion

Through this experience, I learned a lot about the internal workings of Mongoose and MongoClient. Understanding their differences and when to use each is crucial to avoiding connection-related errors. If you're using Mongoose, make sure to properly initialize the connection with Mongoose itself, rather than trying to use MongoClient alongside it.

I hope this blog helps anyone who might be facing similar issues when dealing with Mongoose and MongoClient in a Next.js application. Understanding the differences between these tools can save a lot of time.

Let me know if you have any questions or thoughts on this.🙌

Btw I wrote this blog as part of sharing the learnings and challenges I'm facing while building my project and learning Next.js. If you're interested in what I'm working on or want to follow my journey, check out my Twitter for more updates and to see where it takes me .

Before you can reach anything you have to believe it

DEV Community