Simple infinite scroll in Django

If you've used Twitter, Instagram, or Facebook, you would have used something called "infinite scroll", sometimes also called "infinite loading" or "endless pagination".

Basically, it means that once you scroll down near the bottom of a list of items, the page fetches new items automatically and adds them to the page. That makes it a smoother experience compared to traditional pagination in some cases.

You may have wondered how to do that in Django. I'm going to show you a very simple way to do this using no JavaScript libraries.

Note: The solution here is inefficient for lists with a large number of items (e.g. thousands of items). Efficient infinite scroll uses windowing, and removes items not in view from the DOM. However, if you are only dealing with a few hundred items, this is a very simple way to achieve this, with no dependencies.

Don't worry about the specific details of the code snippets here, just make sure you understand the basic concepts. At the end of the article, I will link to some example code.

Detecting when we've scrolled to the bottom

In the frontend, we will need a way to detect when you've scrolled to the bottom.

This used to be pretty difficult to do, but there is a new browser API or feature called Intersection Observer.

We can think of it as having two components, the scrollable element, and a "sentinel":

Basic concept

The idea is that when the user scrolls down to where the sentinel element is visible, we fetch new items from the Django backend.

First, let's look at how we can detect when this happens using the Intersection Observer API.

First, the Django template, as you can see it's just a list of items and then a sentinel element:

<div id="scrollable-element">
    {% for post in posts %}
        {% include "_post.html" with post=post %}
    {% endfor %}
</div>

<div id="sentinel"></div>

Now the JavaScript

document.addEventListener("DOMContentLoaded", () => {
  let sentinel = document.getElementById("sentinel");

  let observer = new IntersectionObserver((entries) => {
    entry = entries[0];
    if (entry.intersectionRatio > 0) {
        alert("This happened");
    }
  })
  observer.observe(sentinel);
})

When you scroll to the bottom where the sentinel is visible, you should get a popup alert.

That's great! Now we can replace that popup alert with an AJAX request to our backend. But let's add that functionality to our backend first.

We can choose to have our backend return JSON, and render it client-side (e.g. using a templating library), but in this tutorial I have opted to have the backend return HTML, and appending this to the scrollable element through innerHTML. This is a pretty old-fashioned AJAX technique that you sometimes still see in websites like GitHub. If you do this, you need to be very careful with XSS, but we'll discuss this later.

Django pagination

You might be familiar with Django's pagination, if not check out the documentation on that topic here.

Here's how it works. Let's say you have a simple list view like this:

from django.shortcuts import render
from django.views.decorators.http import require_GET, require_POST
from .models import Post

@require_GET
def post_list(request):
    posts = Post.objects.order_by('-created_at').all()
    context = {'posts': posts}
    return render(request, 'post_list.html', context)

You can paginate this by changing it like this:

from django.shortcuts import render
from django.core.paginator import Paginator
from django.http import Http404
from django.views.decorators.http import require_GET, require_POST
from .models import Post

@require_GET
def post_list(request):
    all_posts = Post.objects.order_by('-created_at').all()
    paginator = Paginator(all_posts, per_page=10)
    page_num = int(request.GET.get("page", 1))
    if page_num > paginator.num_pages:
        raise Http404
    posts = paginator.page(page_num)
    context = {'posts': posts}
    return render(request, 'post_list.html', context)

However, we want to have our backend return a short HTML snippet, rather than the full HTML page. So what we want to do is a bit of conditional processing.

First we add another partial template like this, let's call it _posts.html:

    {% for post in posts %}
        {% include "_post.html" with post=post %}
    {% endfor %}

And include that in our list view:

<div id="scrollable-element">
    {% include "_posts.html" with posts=posts %}
</div>

<div id="sentinel"></div>

Then we need to conditionally change the response when the request is an AJAX request. We used to be able to do this using request.is_ajax(), but starting from version 3.1 this is deprecated.

Luckily, it's easy to replicate that functionality:

def is_ajax(request):
    """
    This utility function is used, as `request.is_ajax()` is deprecated.

    This implements the previous functionality. Note that you need to
    attach this header manually if using fetch.
    """
    return request.META.get("HTTP_X_REQUESTED_WITH") == "XMLHttpRequest"

Then, we simply change the above paginated view to do this:

@require_GET
def post_list(request):
    """
    List view for posts.
    """
    all_posts = Post.objects.order_by('-created_at').all()
    paginator = Paginator(all_posts, per_page=10)
    page_num = int(request.GET.get("page", 1))
    if page_num > paginator.num_pages:
        raise Http404
    posts = paginator.page(page_num)
    if is_ajax(request):
        return render(request, '_posts.html', {'posts': posts})
    return render(request, 'post_list.html', {'posts': posts})

That's all we need for the backend.

Making AJAX requests

Now, we can update our JavaScript code to fetch the data from the backend when we scroll down to the sentinel.

Note that we are using fetch in this example, and it doesn't add the X-Requested-With header by default (which is actually why it request.is_ajax() was deprecated). So we need to manually add that header.

const fetchPage = async (url) => {
  let headers = new Headers()
  headers.append("X-Requested-With", "XMLHttpRequest")
  return fetch(url, { headers })
}

document.addEventListener("DOMContentLoaded", () => {
  let sentinel = document.getElementById("sentinel");
  let scrollElement = document.getElementById("scroll-element");
  let counter = 2;
  let end = false;

  let observer = new IntersectionObserver(async (entries) => {
    entry = entries[0];
    if (entry.intersectionRatio > 0) {
        let url = `/posts/?page=${counter}`;
        let req = await fetchPage(url);
        if (req.ok) {
            let body = await req.text();
            // Be careful of XSS if you do this. Make sure
            // you remove all possible sources of XSS.
            scrollElement.innerHTML += body;
        } else {
            // If it returns a 404, stop requesting new items
            end = true;
        }
    }
  })
  observer.observe(sentinel);
})

A note about innerHTML and XSS

In this example, we appended HTML from the server's response to innerHTML. This is a technique that used to be pretty common, and we still see in websites like GitHub. Try opening your "Network" tab in dev tools the next time you're on GitHub, and see the responses when you interact with the website!

As mentioned, if you do this, you need to be very careful to remove sources of XSS in your HTTP response from the backend, otherwise an attacker can inject malicious JavaScript that runs on a user's browser.

If you use Django templates, your template is escaped by default. However, if you use the safe template filter (e.g. you're allowing user to input some HTML) this method is unsafe.

First, you should re-evaluate whether you should be allowing users to enter untrusted HTML that you will display. In many cases you shouldn't need to.

If you can't get around this, you will need to sanitise the HTML. You can do it in the backend using a library like bleach, or a frontend library like DOMPurify.

Alternatively, you can return a JSON response from the backend, and render the HTML client-side. This is a more common way to do this today, as there are frontend frameworks and libraries to do exactly this. This is beyond the scope of this tutorial.

Example code

If you want to see a full working example, I have pushed some sample code to my repository here:

https://github.com/spikelantern/simple-infinite-scroll.git

Summary

In this tutorial, we covered a very simple infinite scroll implementation that doesn't use any special libraries. This uses the new Intersection Observer API.

As mentioned, this technique isn't ideal for very large lists. For an efficient solution, it's necessary to delete DOM elements to prevent the DOM from growing too much. However, for a few hundred items, this should work just fine.

We also discussed the use of innerHTML and its security implications, and recommended some possible mitigations and alternatives.

Hopefully, this post helped you. If you liked this article, make sure to subscribe. I plan to post more articles about using JavaScript with Django, so make sure you subscribe to get notified!