Zach

Posted on Apr 10 • Originally published at circumeo.io

Handling Big Data with Django and AgGrid

#python #django #aggrid

We're going to build an app that can handle filtering and sorting large amounts of data with AgGrid.

Sending all of your data to the browser is a quick way to get started with AgGrid, but what if you have a million rows? That's when server-side filtering and sorting come to the rescue.

This guide includes:

using AgGrid in infinite scroll mode so that the server is responsible for paginating, filtering, and sorting
implementing filtering, sorting, and pagination for AgGrid on top of a Django ListView class
creating a Django management command to populate test data

But first, here's a quick peek at what we're building today:

Previewing the project

Want to see a live version of the app? You can create a copy of this project now and try it out.

Create a fork of the AgGridDemo project on Circumeo.

Setting up the Django app

Install packages and create the Django application.

pip install --upgrade django faker
django-admin startproject bigdata .
python3 manage.py startapp core

Add core to the INSTALLED_APPS list.

# settings.py
INSTALLED_APPS = [
  "core",
    ...
]

Adding the templates

Create a directory named templates within the core app.
Create a file named index.html within the templates directory.

<!DOCTYPE html>
<html>
  <head>
    <script src="https://unpkg.com/ag-grid/dist/ag-grid.min.noStyle.js"></script>
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-grid.css">
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-theme-balham.css">
    <style>
      #container {
        max-width: 1250px;
        width: 100%;
        height: 100%;
        margin: 0px auto;
        margin-top: 30px;
      }
    </style>
  </head>
  <body>
    <div id="container">
      <div id="data-grid" style="height: 600px; width: 100%" class="ag-theme-balham"></div>
    </div>
    <script type="text/javascript" charset="utf-8">
      const numberFilterParams = {
        filterParams: {
          filterOptions: ["equals", "notEqual", "lessThan", "greaterThan"],
          suppressAndOrCondition: true,
        },
      };

      var columnDefs = [
        { headerName: "Name", field: "name", filter: "agTextColumnFilter" },
        {
          headerName: "Description",
          field: "description",
          filter: "agTextColumnFilter",
        },
        { headerName: "Category", field: "category", filter: "agTextColumnFilter" },
        {
          headerName: "Price",
          field: "price",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Stock Quantity",
          field: "stock_quantity",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Rating",
          field: "rating",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
      ];

      var gridOptions = {
        columnDefs: columnDefs,
        defaultColDef: {
          filterParams: {
            suppressAndOrCondition: true,
          },
        },
        enableServerSideSorting: true,
        enableServerSideFilter: true,
        rowModelType: "infinite",
        cacheBlockSize: 100,
        maxBlocksInCache: 10,
      };

      var dataSource = {
        getRows: function (params) {
          var filtering = encodeURIComponent(JSON.stringify(params.filterModel));
          var sorting = encodeURIComponent(JSON.stringify(params.sortModel));

          var startRow = params.startRow;
          var endRow = params.endRow;

          var url = `/products?startRow=${startRow}&endRow=${endRow}&filter=${filtering}&sort=${sorting}`;

          fetch(url)
            .then((response) => response.json())
            .then((data) => {
            params.successCallback(data.rows, data.totalRows);
          })
            .catch((err) => {
            params.failCallback();
          });
        },
      };

      var gridDiv = document.querySelector("#data-grid");

      new agGrid.Grid(gridDiv, gridOptions);
      gridOptions.api.setDatasource(dataSource);
      gridOptions.api.sizeColumnsToFit();
    </script>
  </body>
</html>

Adding the views

Remove the existing views.py file.
Create a new directory named views within the core folder.
Create a new file named index.py within the views directory.
Copy and paste the following into index.py within the views directory.

from django.shortcuts import render

def index_view(request):
    return render(request, "core/index.html")

Create another file named products.py within the same directory.
Copy and paste the following into the products.py file.

import json

from django.views.generic.list import ListView
from django.http import JsonResponse
from django.db.models import Q

from core.models import Product


class ProductListView(ListView):
    model = Product

    def get_queryset(self):
        """
        Convert AgGrid filter and sort objects into a Django query.
        An example filter:
            {"category": {"type": "contains", "filter": "electronics"}}
        """
        queryset = super().get_queryset()

        filter_params = self.request.GET.get("filter", None)
        if filter_params:
            filters = json.loads(filter_params)
            q_objects = Q()

            for key, filter_info in filters.items():
                filter_type = filter_info.get("type")
                filter_value = filter_info.get("filter")

                if filter_type == "contains":
                    lookup = f"{key}__icontains"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "equals":
                    lookup = f"{key}__exact"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "notEqual":
                    lookup = f"{key}__exact"
                    q_objects &= ~Q(**{lookup: filter_value})
                elif filter_type == "greaterThan":
                    lookup = f"{key}__gt"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "lessThan":
                    lookup = f"{key}__lt"
                    q_objects &= Q(**{lookup: filter_value})

            queryset = queryset.filter(q_objects)

        sort_params = self.request.GET.get("sort", None)
        if sort_params:
            sort_objects = json.loads(sort_params)
            sort_fields = []
            for sort_object in sort_objects:
                col_id = sort_object["colId"]
                sort_order = sort_object["sort"]
                if sort_order == "asc":
                    sort_fields.append(col_id)
                elif sort_order == "desc":
                    sort_fields.append(f"-{col_id}")

            if sort_fields:
                queryset = queryset.order_by(*sort_fields)
        return queryset

    def get(self, request, *args, **kwargs):
        start_row = int(request.GET.get("startRow", 0))
        end_row = int(request.GET.get("endRow", 100))

        queryset = self.get_queryset()

        total_rows = queryset.count()
        queryset = queryset[start_row:end_row]

        products = list(
            queryset.values(
                "name", "description", "category", "price", "stock_quantity", "rating"
            )
        )

        return JsonResponse({"rows": products, "totalRows": total_rows})

Updating URLs

Create urls.py in the core directory.

from django.urls import path

from core.views.index import index_view
from core.views.products import ProductListView

urlpatterns = [
    path("", index_view, name="index"),
    path("products", ProductListView.as_view(), name="products"),
]

Update the existing urls.py within the project bigdata directory.

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]

Adding the database models

Overwrite the existing models.py with the following:

from django.db import models

class Product(models.Model):
    name = models.CharField(max_length=255)
    description = models.TextField(blank=True)
    category = models.CharField(max_length=255)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    stock_quantity = models.IntegerField()
    rating = models.FloatField(blank=True, null=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

Adding a Django Management Command for Test Data

Create the directory structure management/commands within the core folder.
Open a file named populate_products.py within the new directory and enter the following.

from faker import Faker
import random

from django.core.management.base import BaseCommand
from core.models import Product

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        faker = Faker()

        categories = [
            'Electronics',
            'Books',
            'Clothing',
            'Home & Garden',
            'Toys & Games',
            'Sports & Outdoors',
            'Health & Beauty',
            'Automotive',
            'Groceries',
            'Pet Supplies'
        ]

        for _ in range(1000):
            Product.objects.create(
                name=faker.text(max_nb_chars=20).capitalize(),
                description=faker.text(),
                category=random.choice(categories),
                price=round(random.uniform(5.0, 500.0), 2),
                stock_quantity=random.randint(0, 100),
                rating=round(random.uniform(1.0, 5.0), 1)
            )

        self.stdout.write(self.style.SUCCESS('Successfully populated the database with products.'))

Open a shell session in order to run the Django management command.

python3 manage.py populate_products

Up and Running with Django and AgGrid

You're now ready to handle millions of rows without crashing browser tabs! Add database indexes where needed, and you can scale this approach pretty far.

DEV Community

Handling Big Data with Django and AgGrid

Previewing the project

Setting up the Django app

Adding the templates

Adding the views

Updating URLs

Adding the database models

Adding a Django Management Command for Test Data

Up and Running with Django and AgGrid

Top comments (0)

Read next

Build Your Own AI Language Model with Python and NumPy

Top AI Tools to Use in 2024 for Developers, Creators, and Innovators

A Guide to dbt Macros - Purpose, Benefits, and Usage

Building a Document QA with Streamlit & OpenAI