The goal of this tutorial is to build an api to create and list subscribers.
In this tutorial we will inspect the data, create models, serializers, and views, time the difference between create and bulk create, and learn how to filter a queryset.
Well-known libraries we will use include Django, and Django REST framework.
- Installing libraries
- Creating a Django project and app
- Project structure
- Understanding the data
- Creating models
- Creating serializers
- Creating views
- Creating routes (urls)
- Adding a command for test data
- Creating a bulk serializer
- Queryset filtering
Installing Django and Django REST framework
pip3 install django djangorestframework
Creating a Django Project and Django App
-
Create a working directory for the django project
mkdir django_api_tutorial && cd django_api_tutorial -
Create the django project
django-admin startproject api . -
Create the django app for subscribers
python3 manage.py startapp subscribers -
Connect subscribers and rest_framework app to the API settings in
api/settings.py
INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', # new (djangorestframework) 'subscribers' # new ]Add the following to the bottom of the settings for paged list results
REST_FRAMEWORK = { 'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination', 'PAGE_SIZE': 32 }
Project structure
./django_api_tutorial
│ manage.py
│ README.md
│
├───api
│ asgi.py
│ settings.py
│ urls.py
│ wsgi.py
│ __init__.py
│
├───data
│ fake_users.csv
│
└───subscribers
│ admin.py
│ apps.py
│ models.py
│ serializers.py
│ tests.py
│ urls.py
│ views.py
│ __init__.py
│
├───management
│ └───commands
│ bulktestdata.py
│ testdata.py
│
└───migrations
__init__.py
Learn more about the django project structure here.
Understanding the data
Here are the first 10 rows of the fake data for the tutorial. This data was created using Mockaroo.
| first_name | last_name | gender | city | state | |
|---|---|---|---|---|---|
| Mohammed | Poad | mpoad0@cisco.com | Male | Watertown | Massachusetts |
| Briana | Liddall | bliddall1@odnoklassniki.ru | Female | Indianapolis | Indiana |
| Jodie | Pattington | jpattington2@telegraph.co.uk | Male | Brockton | Massachusetts |
| Cari | Worcs | cworcs3@youku.com | Female | Richmond | Virginia |
| Shane | Pickford | spickford4@arstechnica.com | Male | Newark | New Jersey |
| Bethany | McColm | bmccolm5@comsenz.com | Female | Evansville | Indiana |
| Elsi | Wyrill | ewyrill6@opera.com | Female | Lexington | Kentucky |
| Kylynn | Hartill | khartill7@webeden.co.uk | Female | Columbia | South Carolina |
| Lonnie | Elliot | lelliot8@msn.com | Male | Joliet | Illinois |
| Kellyann | Kelso | kkelso9@sbwire.com | Female | San Bernardino | California |
Now we will break down each one of these columns to their corresponding data type.
first_name VARCHAR(64)
last_name VARCHAR(64)
email TEXT
gender VARCHAR(8)
city VARCHAR(256)
state VARCHAR(24)
Knowing how we want to handle each of these variables is going to enable us to easily create models and serializers.
Creating models
The model definitions can be found in subscribers/models.py.
from django.db import models
class Location(models.Model):
city = models.CharField(null=False, max_length=256)
state = models.CharField(null=False, max_length=64)
class Subscriber(models.Model):
first_name = models.CharField(null=False, max_length=64)
last_name = models.CharField(null=False, max_length=64)
email = models.TextField()
gender = models.CharField(null=False, max_length=8)
location = models.ForeignKey(Location, related_name='subscriber_location', on_delete=models.DO_NOTHING)
Above, we defined the models for both a Subscriber and Location.
The Subscriber model is a Many to One relationship with the Location model since we can have many subscribers in one location.
Creating serializers
We will now define serializers for the models. This allows us to easily convert the model objects into json objects for API responses.
When we create serializers, we can add a lot of functionality to the API easily with Django REST framework.
The serializers are defined in subscribers/serializers.py. Notice that in the SubscriberSerializer we have to overwrite the create method.
This is normal when you are using relational models.
from rest_framework import serializers
class LocationSerializer(serializers.Serializer):
city = serializers.CharField(required=True, max_length=256)
state = serializers.CharField(required=True, max_length=64)
class Meta:
fields = ['city', 'state']
class SubscriberSerializer(serializers.Serializer):
id = serializers.IntegerField(required=False)
created = serializers.DateTimeField(required=False)
first_name = serializers.CharField(required=True, max_length=64)
last_name = serializers.CharField(required=True, max_length=64)
email = serializers.CharField(required=False)
gender = serializers.CharField(required=True, max_length=8)
location = LocationSerializer(required=True)
class Meta:
fields = ['first_name', 'last_name', 'email', 'gender', 'location']
read_only_fields = ['id', 'created']
def create(self, validated_data):
# remove location from serialized data and add model object
location = validated_data.pop('location')
city = location.get('city', None)
state = location.get('state', None)
if not city and not state:
raise serializers.ValidationError('No location input found')
# call get or create to reuse location objects
location_obj = Location.objects.get_or_create(city=city, state=state)[0]
# add location back to validated data
validated_data.update({'location': location_obj})
# unpack validated_data to create a new Subscriber object
return Subscriber.objects.create(**validated_data)
We have now defined the serializers so that we can easily do things like create, read, update, delete and list objects from the models.
We will see this in action when we implement the Views.
Creating views
The views are defined in subscribers/views.py and contain the functionality that will be available to users of the API.
In this tutorial we will focus on being able to create and list subscribers, but along with adding data we will also likely need a way to easily remove data.
I will give an example how to easily add a delete operation to the API using Django REST framework mixins below.
from rest_framework import viewsets, mixins
from .models import Subscriber
from .serializers import SubscriberSerializer
class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin):
queryset = Subscriber.objects.all()
serializer_class = SubscriberSerializer
This simple view above provides a generic API interface with list and create functionality for Subscribers.
To easily add the delete method and functionality it would look like the following:
from rest_framework import viewsets, mixins
from .models import Subscriber
from .serializers import SubscriberSerializer
class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin,
mixins.DestroyModelMixin):
queryset = Subscriber.objects.all()
serializer_class = SubscriberSerializer
Notice the only new change was the addition of mixins.DestroyModelMixin in the class definition. More info about mixins can be found on the Django REST framework docs.
Creating routes
Now to be able to navigate to the API we will need to add the urls. The urls are defined in subscribers/urls.py.
We will also want to tell the base API where to find the routes from subscribers and this is defined in api/urls.py.
# api/urls.py
from django.contrib import admin
from django.urls import path, include
urlpatterns = [
path('admin/', admin.site.urls),
path('tutorial/', include('subscribers.urls'))
]
# subscribers/urls.py
from rest_framework import routers
from .views import SubscriberView
router = routers.DefaultRouter(trailing_slash=False)
router.register(r'subscribers', SubscriberView, basename='subscribers')
urlpatterns = router.urls
In the subscribers urls we use a router from Django REST framework to easily add all of the functionality that we defined in the views.
The router with the viewset enabled the following routes for the API.
GET /tutorial/subscribers List view of subscribers
POST /tutorial/subscribers Create a new subscriber
Adding a command for test data
Now we will add a command to django to easily allow us to add test data to the API. The commands are defined in subscribers/management/commands and the first command is named testdata.py.
import csv
from time import time
from django.core.management.base import BaseCommand, CommandError
from subscribers.serializers import SubscriberSerializer
class Command(BaseCommand):
help = 'Adds the fake test data to the API'
def handle(self, *args, **options):
try:
with open('data/fake_users.csv', 'r') as fin:
csvreader = csv.reader(fin)
headers = next(csvreader)
data = [{'first_name': row[0],
'last_name': row[1],
'email': row[2],
'gender': row[3],
'location': {'city': row[4], 'state': row[5]}
} for row in csvreader
]
# time how fast it takes to add all records 1 by 1
start = time()
for item in data:
serializer = SubscriberSerializer(data=item)
if serializer.is_valid():
serializer.create(item)
stop = time()
print(f'{len(data)} items added in {stop-start} seconds')
except FileExistsError:
raise CommandError('No testdata found')
This command will add test records to the API. It will also track how many records and how quickly they were added.
We will run the command and see what the output is.
python3 manage.py testdata
Output: 6000 items added in 31.6553955078125
Next we will implement a bulk serializer and add a new bulk command to see if we can speed up creating records.
Creating a bulk serializer
Instead of having to create objects one by one we will create a bulk serializer to create many at a time.
class BulkSubscriberSerializer(serializers.Serializer):
subscribers = SubscriberSerializer(many=True)
class Meta:
fields = ['subscribers']
def create(self, validated_data):
# store the Subscriber objects to be created in bulk
create_objects_list = []
# iterate over the validated_data and add Subscriber objects to a list to be created
for data in validated_data:
# notice the same functionality from the regular serializer
location = data.pop('location')
city = location.get('city', None)
state = location.get('state', None)
location_obj = Location.objects.get_or_create(city=city, state=state)[0]
# combine data and {'location': location_obj} and unpack to the Subscriber model
create_objects_list.append(Subscriber(**{**data, **{'location': location_obj}}))
return Subscriber.objects.bulk_create(create_objects_list)
We will also create a new command called bulktestdata that is defined in subscribers/management/commands/bulktestdata.py.
This will use the bulk serializer to add the records and track how long it takes.
import csv
from time import time
from django.core.management.base import BaseCommand, CommandError
from subscribers.serializers import BulkSubscriberSerializer
class Command(BaseCommand):
help = 'Adds the fake test data to the API'
def handle(self, *args, **options):
try:
with open('data/fake_users.csv', 'r') as fin:
csvreader = csv.reader(fin)
headers = next(csvreader)
data = [{'first_name': row[0],
'last_name': row[1],
'email': row[2],
'gender': row[3],
'location': {'city': row[4], 'state': row[5]}
} for row in csvreader
]
# time how fast it takes to add records in bulk
start = time()
bulk_serializer = BulkSubscriberSerializer(data={'subscribers': data})
if bulk_serializer.is_valid():
bulk_serializer.create(data)
stop = time()
print(f'{len(data)} items added in {stop-start} seconds')
except FileExistsError:
raise CommandError('No testdata found')
Now when we run the new command lets see how fast all of the records get added.
python3 manage.py bulktestdata
Output: 6000 items added in 5.3229029178619385 seconds
Lastly, we will update the views to use the regular or bulk serializer based on the data sent to the route.
from rest_framework import viewsets, mixins
from rest_framework.response import Response
from .models import Subscriber
from .serializers import SubscriberSerializer, BulkSubscriberSerializer
class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin,
mixins.DestroyModelMixin):
queryset = Subscriber.objects.all()
serializer_class = SubscriberSerializer
def create(self, request, *args, **kwargs):
# if the data is a dictionary, use parent create that relies on serializer_class
if isinstance(request.data, dict):
return super(SubscriberView, self).create(request, *args, **kwargs)
# if the data is a list, send to the bulk serializer to handle creation
elif isinstance(request.data, list):
serializer = BulkSubscriberSerializer(data={'subscribers': request.data})
if serializer.is_valid():
serializer.create(request.data)
return Response(serializer.data, status=201)
else:
return Response(serializer.errors, status=400)
else:
return Response('Invalid data received', status=400)
At this point, the API can now create one to many records at a time and allow users to browse the current subscribers.
Here is a snippet of a response from the API for a GET request to http://127.0.0.1:8000/tutorial/subscribers.
{
"count": 6000,
"next": "http://127.0.0.1:8000/tutorial/subscribers?page=2",
"previous": null,
"results": [
{
"id": 1,
"created": "2020-10-13T15:51:50.850563Z",
"first_name": "Mohammed",
"last_name": "Poad",
"email": "mpoad0@cisco.com",
"gender": "Male",
"location": {
"city": "Watertown",
"state": "Massachusetts"
}
},
{
"id": 2,
"created": "2020-10-13T15:51:50.862560Z",
"first_name": "Briana",
"last_name": "Liddall",
"email": "bliddall1@odnoklassniki.ru",
"gender": "Female",
"location": {
"city": "Indianapolis",
"state": "Indiana"
}
},
...
]
}
Queryset filtering
We can easily list all of the subscribers but what if we only want to see subscribers from a specific state?
Currently, as a user, we would have to pull all of the subscribers from the API and filter our own results.
This is where queryset filtering from Django can help give the users more control.
The user can send a query parameter in the request and we can use it to filter the results. The new view will look like the following.
from rest_framework import viewsets, mixins
from rest_framework.response import Response
from .models import Subscriber
from .serializers import SubscriberSerializer, BulkSubscriberSerializer
class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin,
mixins.DestroyModelMixin):
serializer_class = SubscriberSerializer
def get_queryset(self):
queryset = Subscriber.objects
if 'state' in self.request.query_params:
queryset = queryset.filter(location__state__icontains=self.request.query_params['state'])
return queryset.order_by('created')
def create(self, request, *args, **kwargs):
# if the data is a dictionary, use parent create that relies on serializer class
if isinstance(request.data, dict):
return super(SubscriberView, self).create(request, *args, **kwargs)
# if the data is a list, send to the bulk serializer to handle creation
elif isinstance(request.data, list):
serializer = BulkSubscriberSerializer(data={'subscribers': request.data})
if serializer.is_valid():
serializer.create(request.data)
return Response(serializer.data, status=201)
else:
return Response(serializer.errors, status=400)
else:
return Response('Invalid data received', status=400)
We have added the get_queryset method and can now send state as a query parameter on a GET request.
For example if we send a GET request to http://127.0.0.1:8000/tutorial/subscribers?state=Texas we can see that we have less total results.
{
"count": 629,
"next": "http://127.0.0.1:8000/tutorial/subscribers?page=2&state=Texas",
"previous": null,
"results": [
{
"id": 13,
"created": "2020-10-13T19:51:29.461522Z",
"first_name": "Laure",
"last_name": "Chitter",
"email": "lchitterc@t-online.de",
"gender": "Female",
"location": {
"city": "Corpus Christi",
"state": "Texas"
}
},
...
]
}
We now have an API that can create one to many subscribers based on the payload, list all subscribers, and list subscribers from a certain state.
Hope you enjoyed the tutorial, all the code can be found here.
Top comments (0)