DEV Community 👩‍💻👨‍💻

DEV Community 👩‍💻👨‍💻 is a community of 963,864 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Cover image for Fast and Efficient Pagination in Golang and MongoDB
Neeraj Kumar
Neeraj Kumar

Posted on

Fast and Efficient Pagination in Golang and MongoDB

MongoDB is a document based data store and hence pagination is one of the most common use case of it. So when do you paginate the response? The answer is pretty neat; you paginate whenever you want to process result in chunks. Some common scenarios are

  • Batch processing

  • Showing huge set of results on user interfac

Paginating on client and server side are both really very expensive and should not be considered. Hence pagination is generally handled at database level and databases are optimized for such needs to

2 approaches through which you can easily paginate your MongoDB responses. Sample Document

 {
        "_id" : ObjectId("6936d17263623919cd5145db"),
        "name" : "Neeraj Kumar",
        "age" : 25
    }
Enter fullscreen mode Exit fullscreen mode

Approach 1: Using cursor.skip and cursor.limit
MongoDB cursor has two methods that makes paging easy; they are

  • cursor.skip()
  • cursor.limit()

skip(n) will skip n documents from the cursor while limit(n) will cap the number of documents to be returned from the cursor. Thus combination of two naturally paginates the response.
In Mongo Shell your pagination code looks something like

 // Page 1
    db.students.find().limit(10)

    // Page 2
    db.students.find().skip(10).limit(10)

    // Page 3
    db.students.find().skip(10).limit(10)
Enter fullscreen mode Exit fullscreen mode

implement pagination:

func GetPagination(limit, page int) error {
  ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()
   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)

   l := int64(limit)
     skip := int64(page * limit - limit)
     fOpt := options.FindOptions{Limit: &l, Skip: &skip}

     curr, err := coll.Find(ctx, bson.D{{}}, &fOpt)
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}


Enter fullscreen mode Exit fullscreen mode

Approach 2: Using _id and limit

This approach will make effective use of default index on _id and nature of ObjectId. I bet you didn’t know that a Mongodb ObjectId is a 12 byte structure containing

Using this property of ObjectId and also taking into consideration the fact that _id is always indexed, we can devise following approach for pagination:

  • Fetch a page of documents from database
  • Get the document id of the last document of the page
  • Retrieve documents greater than that id
    // Page 1
    db.students.find().limit(10)

    // Page 2
    last_id = ...  # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)

    // Page 3
    last_id = ... # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)
Enter fullscreen mode Exit fullscreen mode
func GetPagination(limit, page int)  error {
    ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()

   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)
   ctx, _ := context.WithTimeout(context.Background(), contextTimeout)

   curr, err := coll.Find(ctx, bson.D{{}}, newMongoPaginate(limit,page).getPaginatedOpts())
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}

Enter fullscreen mode Exit fullscreen mode

Top comments (0)

🌚 Life is too short to browse without dark mode