Dumping JSON data to Django Models: Using Django Setup and Commands

The need of data become most important when you complete your first version of website using Django or REST API using Django Rest Framework (DRF). For, similar problem I have written my previous article , in which, i have discussed a direct approach of dumping JSON data into Django model by inserting directly to SQLite database and tables. However, i have also mentioned that Django have loaddata, dumpdata and fixtures for similar problems but that need little more understanding of framework.

Today, I am going to travel that learning curve and discuss how to convert your normal JSON data into a fixture and then use loaddata to dump JSON data into a specific model. Similarly, if you have some data for models (may be added via admin page etc.) then how to dump that as fixtures to use with other model or use in other project.

The previous article has two parts: First part is normal setup required for data dumping and second part, is about python script to dump JSON data into the model. So, keep the article short, In this article, i will start from second part and you can follow Part -I from previous article to follow this article. So, assuming you have a book app and in which there is Book model inside models.py, similarly the API endpoints are ready, such as (Github repo of previous post) :
http://127.0.0.1:8000/api/books/1

or you simply want to load data to Book model for a Django-based website. You can follow code from following git repository.
JSON Data Dumping to Model

Given two repositories, you can use anyone for follow article from here, you just need following things:

1. A working Django project having a Book model with or without API features
2. A json file having book information as per the Book model.

Example 1 : The Book Model in book/models.py

from django.db import models

class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.CharField(max_length=100)    
    isbn = models.CharField(max_length=13, unique=True)
    pages = models.PositiveIntegerField()   
    language = models.CharField(max_length=30)

    def __str__(self):
        return self.title

books.json

[
  {
    "title": "Example Book 1",
    "author": "John Doe",
    "isbn": "1234567890123",
    "pages": 350,
    "language": "English"
  },
  {
    "title": "Example Book 2",
    "author": "Kumar Ajit",
    "isbn": "9876543210987",
    "pages": 280,
    "language": "Hindi"
  }
]

Example 2: The Book model from previous article

from django.db import models

class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.CharField(max_length=100)
    price = models.DecimalField(max_digits=6, decimal_places=2)

    def __str__(self):
        return self.title

You can find data.json file here.

Data Dumping to model

We have a working Django project with a model Book and a extra JSON file having books information. So, now what is the problem?
We want to dump the JSON data into the model so that can be either use with Django website (for example passing data to template for display to user) or serving data via API to frontend.

Django have loaddata, fixtures and dumpdata, along side of admin page (you can register your model and insert data) and shell (to directly manipulating database using SQL or ORM). However, loaddata via fixtures is seems better if want to merge large data or want to do it many times (often a usecase during development and testing).

Let starts by explaining these tools and then will dive into the python script to achieve the data dumping task.

Loaddata

$django-admin loaddata <fixture label>

Fixtures

A fixture is a collection of files that contain the serialized contents of the database. Each fixture has a unique name, and the files that comprise the fixture can be distributed over multiple directories, in multiple applications. [source]

Let jump to future part of article and have a look on the books fixtures.

[
  {
    "model": "book.book",
    "pk": 40,
    "fields": {
      "title": "Example Book 1",
      "author": "John Doe",
      "isbn": "123456734890123",
      "pages": 350,
      "language": "English"
    }
  },
  {
    "model": "book.book",
    "pk": 41,
    "fields": {
      "title": "Example Book 2",
      "author": "Kumar Ajit",
      "isbn": "9876543455210987",
      "pages": 280,
      "language": "Hindi"
    }
  }
]

If you had a look on fixture file, you must have found that it is just another JSON file with more key-value pairs and those are metadata for Django models /tables. So, yes, you are right and our aim to to convert our books.json format to fixture format so the Django admin tool will recognize it and dump data in suitable table/s.

Now, for creating a fixture you can have two situations: 1) you have data in models and you want to create fixture for future usage or testing etc.
2) You don't have data in models/tables but you have data in external sources like json, csv or any other formats and you want to load that data in Django via previously discussed command loaddata .( This article is about this situation but towards the end, we will use dumpdata to compare manual and dumpdata output.)

Before going to python script to convert normal json file to fixture format, let understand how Django look for fixture file and how use it.

There are three point of interest and order [source]:

1. In the fixtures directory of every installed application : This is similar to the recommendation for other directory like template or media or static folder organized inside a Django project. However, you can also give direct path like mentioned below.

2. In any directory listed in the FIXTURE_DIRS setting

3. In the literal path named by the fixture: Works with smaller project or app but it is better to have a proper recommended location for fixtures so it will easy to use with testing or debugging etc.

Note:One major benefit of using fixture is to apply and use compression on fixtures.

Dumpdata

This is django-admin command to output all data of table to standard output.

Note 1: Outputs to standard output all data in the database associated with the named application(s).
Note 2 : The output of dumpdata can be used as input for loaddata.(as fixture )

django-admin dumpdata [app_label[.ModelName] 

you can read more from this link.

Python script from simple json to Django Fixtures.

from django.apps import apps
import django
import os
import json


# Set the Django settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'dataloading.settings')
django.setup()

# get all models
models = apps.get_models()

# Get the model names
for model in models:
    print(model)
    print(model.__name__)

# Choose Model to create fixtures
from book.models import Book 

# Get all field names
field_names = [field.name for field in Book._meta.get_fields()]
fixture_name = "books.json"

#read the data file json, csv etc.

with open('data.json','r') as fp:
    books= json.load(fp)


books_fixture =[]
# get the pk value from model , if there is data
pk=1 
# iterate all the books in json file
for book in books:
    book_fixture={}
    # Add model name (most important for fixture)
    book_fixture['model']= 'book.book'
    pk+=1
    book_fixture['pk']=pk
    # assuming the data has same fields name as model else mapping is required
    book_fixture['fields']=book
    books_fixture.append(book_fixture)
# Writing converted Json as file for fixture

with open(fixture_name,'w') as fp:
    json.dump(books_fixture,fp)

# print(field_names)
# if __name__ =='__main__' :
    #create_book()  

Once your fixture is created as books.json. It time to move it /copy to suitable location (book/fixtures/book/) so that Django manager can find it easily and then run loaddata command to dump all data to Book model.

$mv books.json book/fixtures/book/
$python manage.py loaddata book/fixtures/book/books.json

You can also use similar script to directly write data to model sing .save() method. Below is create_book() method that can be use to insert one or more (looping) data records into Book model.

# def create_book():
#     book = Book(
#         title="This is from script",
#         author ="ajit",
#         isbn = "2024-0809",
#         pages=300,
#         language ='English'
#     )
#     book.save()
#     print(f'Book {book.title} by {book.author} created.')

You can also look into Dumpscript from Django-extensions for similar task.

I hope it will be helpful.

Note: The article in not properly proofread and formatted because of time constraint. So, please use it carefully and leave comments below for any errors, dobuts or questions. I will reply and edit the article overtime.