We're going to build an app that can handle filtering and sorting large amounts of data with AgGrid.
Sending all of your data to the browser is a quick way to get started with AgGrid, but what if you have a million rows? That's when server-side filtering and sorting come to the rescue.
This guide includes:
- using AgGrid in infinite scroll mode so that the server is responsible for paginating, filtering, and sorting
- implementing filtering, sorting, and pagination for AgGrid on top of a Django ListView class
- creating a Django management command to populate test data
But first, here's a quick peek at what we're building today:
Previewing the project
Want to see a live version of the app? You can create a copy of this project now and try it out.
Create a fork of the AgGridDemo project on Circumeo.
Setting up the Django app
Install packages and create the Django application.
pip install --upgrade django faker
django-admin startproject bigdata .
python3 manage.py startapp core
Add core to the INSTALLED_APPS list.
# settings.py
INSTALLED_APPS = [
"core",
...
]
Adding the templates
- Create a directory named
templateswithin thecoreapp. - Create a file named
index.htmlwithin thetemplatesdirectory.
<!DOCTYPE html>
<html>
<head>
<script src="https://unpkg.com/ag-grid/dist/ag-grid.min.noStyle.js"></script>
<link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-grid.css">
<link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-theme-balham.css">
<style>
#container {
max-width: 1250px;
width: 100%;
height: 100%;
margin: 0px auto;
margin-top: 30px;
}
</style>
</head>
<body>
<div id="container">
<div id="data-grid" style="height: 600px; width: 100%" class="ag-theme-balham"></div>
</div>
<script type="text/javascript" charset="utf-8">
const numberFilterParams = {
filterParams: {
filterOptions: ["equals", "notEqual", "lessThan", "greaterThan"],
suppressAndOrCondition: true,
},
};
var columnDefs = [
{ headerName: "Name", field: "name", filter: "agTextColumnFilter" },
{
headerName: "Description",
field: "description",
filter: "agTextColumnFilter",
},
{ headerName: "Category", field: "category", filter: "agTextColumnFilter" },
{
headerName: "Price",
field: "price",
filter: "agNumberColumnFilter",
...numberFilterParams,
},
{
headerName: "Stock Quantity",
field: "stock_quantity",
filter: "agNumberColumnFilter",
...numberFilterParams,
},
{
headerName: "Rating",
field: "rating",
filter: "agNumberColumnFilter",
...numberFilterParams,
},
];
var gridOptions = {
columnDefs: columnDefs,
defaultColDef: {
filterParams: {
suppressAndOrCondition: true,
},
},
enableServerSideSorting: true,
enableServerSideFilter: true,
rowModelType: "infinite",
cacheBlockSize: 100,
maxBlocksInCache: 10,
};
var dataSource = {
getRows: function (params) {
var filtering = encodeURIComponent(JSON.stringify(params.filterModel));
var sorting = encodeURIComponent(JSON.stringify(params.sortModel));
var startRow = params.startRow;
var endRow = params.endRow;
var url = `/products?startRow=${startRow}&endRow=${endRow}&filter=${filtering}&sort=${sorting}`;
fetch(url)
.then((response) => response.json())
.then((data) => {
params.successCallback(data.rows, data.totalRows);
})
.catch((err) => {
params.failCallback();
});
},
};
var gridDiv = document.querySelector("#data-grid");
new agGrid.Grid(gridDiv, gridOptions);
gridOptions.api.setDatasource(dataSource);
gridOptions.api.sizeColumnsToFit();
</script>
</body>
</html>
Adding the views
- Remove the existing
views.pyfile. - Create a new directory named
viewswithin thecorefolder. - Create a new file named
index.pywithin theviewsdirectory. - Copy and paste the following into
index.pywithin theviewsdirectory.
from django.shortcuts import render
def index_view(request):
return render(request, "core/index.html")
- Create another file named
products.pywithin the same directory. - Copy and paste the following into the
products.pyfile.
import json
from django.views.generic.list import ListView
from django.http import JsonResponse
from django.db.models import Q
from core.models import Product
class ProductListView(ListView):
model = Product
def get_queryset(self):
"""
Convert AgGrid filter and sort objects into a Django query.
An example filter:
{"category": {"type": "contains", "filter": "electronics"}}
"""
queryset = super().get_queryset()
filter_params = self.request.GET.get("filter", None)
if filter_params:
filters = json.loads(filter_params)
q_objects = Q()
for key, filter_info in filters.items():
filter_type = filter_info.get("type")
filter_value = filter_info.get("filter")
if filter_type == "contains":
lookup = f"{key}__icontains"
q_objects &= Q(**{lookup: filter_value})
elif filter_type == "equals":
lookup = f"{key}__exact"
q_objects &= Q(**{lookup: filter_value})
elif filter_type == "notEqual":
lookup = f"{key}__exact"
q_objects &= ~Q(**{lookup: filter_value})
elif filter_type == "greaterThan":
lookup = f"{key}__gt"
q_objects &= Q(**{lookup: filter_value})
elif filter_type == "lessThan":
lookup = f"{key}__lt"
q_objects &= Q(**{lookup: filter_value})
queryset = queryset.filter(q_objects)
sort_params = self.request.GET.get("sort", None)
if sort_params:
sort_objects = json.loads(sort_params)
sort_fields = []
for sort_object in sort_objects:
col_id = sort_object["colId"]
sort_order = sort_object["sort"]
if sort_order == "asc":
sort_fields.append(col_id)
elif sort_order == "desc":
sort_fields.append(f"-{col_id}")
if sort_fields:
queryset = queryset.order_by(*sort_fields)
return queryset
def get(self, request, *args, **kwargs):
start_row = int(request.GET.get("startRow", 0))
end_row = int(request.GET.get("endRow", 100))
queryset = self.get_queryset()
total_rows = queryset.count()
queryset = queryset[start_row:end_row]
products = list(
queryset.values(
"name", "description", "category", "price", "stock_quantity", "rating"
)
)
return JsonResponse({"rows": products, "totalRows": total_rows})
Updating URLs
- Create urls.py in the core directory.
from django.urls import path
from core.views.index import index_view
from core.views.products import ProductListView
urlpatterns = [
path("", index_view, name="index"),
path("products", ProductListView.as_view(), name="products"),
]
- Update the existing urls.py within the project bigdata directory.
from django.contrib import admin
from django.urls import include, path
urlpatterns = [
path("admin/", admin.site.urls),
path("", include("core.urls")),
]
Adding the database models
Overwrite the existing models.py with the following:
from django.db import models
class Product(models.Model):
name = models.CharField(max_length=255)
description = models.TextField(blank=True)
category = models.CharField(max_length=255)
price = models.DecimalField(max_digits=10, decimal_places=2)
stock_quantity = models.IntegerField()
rating = models.FloatField(blank=True, null=True)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
Adding a Django Management Command for Test Data
- Create the directory structure
management/commandswithin thecorefolder. - Open a file named
populate_products.pywithin the new directory and enter the following.
from faker import Faker
import random
from django.core.management.base import BaseCommand
from core.models import Product
class Command(BaseCommand):
def handle(self, *args, **kwargs):
faker = Faker()
categories = [
'Electronics',
'Books',
'Clothing',
'Home & Garden',
'Toys & Games',
'Sports & Outdoors',
'Health & Beauty',
'Automotive',
'Groceries',
'Pet Supplies'
]
for _ in range(1000):
Product.objects.create(
name=faker.text(max_nb_chars=20).capitalize(),
description=faker.text(),
category=random.choice(categories),
price=round(random.uniform(5.0, 500.0), 2),
stock_quantity=random.randint(0, 100),
rating=round(random.uniform(1.0, 5.0), 1)
)
self.stdout.write(self.style.SUCCESS('Successfully populated the database with products.'))
Open a shell session in order to run the Django management command.
python3 manage.py populate_products
Up and Running with Django and AgGrid
You're now ready to handle millions of rows without crashing browser tabs! Add database indexes where needed, and you can scale this approach pretty far.
Top comments (0)