Data Migration from Monolith to Microservice in Django

Introduction

Monolith Architecture is a design model for a software program. Under this design model, we follow a traditional approach of a unified program i.e. whole code is composed in one single piece.

Due to a unified approach monolith architecture does not provide flexibility in terms of size and changes. If the application is too large then changes or updates are difficult to make. Continuous development is difficult due to a unified architecture and also fault detection is difficult. Due to a unified architecture, it provides a restriction both in size and complexity of maintenance.

To overcome the shortcomings of monolith architecture nowadays microservices are used. The main idea of microservices is to break the unified architecture into smaller segments and interconnect them, using such a design we would reduce complete failure of the system due to breaking at one point and also help in the maintenance, testing, and continuous integration.

To break a monolith into microservices we need to make microservices in such a way that it is interconnected to others but also be able to stand alone in times when an interconnected microservices breaks off or if one microservice face downtime or some bug, then it should not lead to whole system failure.

In this article, we would learn about how to separate the data of a microservice already present from a monolith and populate it into the database linked with the microservice using Django management commands. We would learn about Django Management commands and how data migration can be done using these commands.

What is Django?

Django is a python high level web framework that helps in rapid and clean development. It is free and open source. Django provides features like fast, secure and scalable thus making it one of the most popular choices for web development.

What are Django Management commands?

Django management commands are a part of Django apps that provide us the ability to fulfill repetitive or complex tasks through a one keyword command line instruction. Every management command has a proper script that defines how the command has to be executed in order to perform tasks

How to migrate data from monolith to microservice?

In order to perform the migration we need to follow the below steps to get and populate the data of a microservice present in a monolith to the microservice database:

1. A management command at monolith to filter out the data related to the microservice present in monolith in a specific format like JSON or CSV(here I will be using JSON)
2. Use this JSON or CSV file and extract the data and populate them in Microservice Database.

Step 1: Extracting data from monolith

1. This will be a management command in python so we need to write this under the following directory {django-app-name}/management/commands/fetch_data(file_name).py
2. Code:

import json
import sys
from django.core.management.base import BaseCommand
from django.core.serializers.json import DjangoJSONEncoder
class Command(BaseCommand):
    help = "Extracting user data to JSON format"

    def handle(self, *args, **options):
        # Get User Data from User Model in monolith        
        user_microservice_data = User.objects.all()
        for user_data in user_microservice_data:
            data = {
                "model": "User",
                "id": user_data.id,
                "username": user_data.flip_collection.id,
                "mobile_number": user_data.flip_image.id,
                "created_at": user_data.created_at,
            }
            # Dumping Data into JSON Format
            json.dump(data, sys.stdout, cls=DjangoJSONEncoder)
            sys.stdout.write("\n")

After saving this command extract using the following shell command:

python manage.py fetch_data > {data_file_name}.json

Step 2: Populating the Microservice database using this JSON File

1. For this we will again need to write a management command on microservice project and upload the json file in microservice project.
   Directory :{django-app-name}/management/commands/populate_data(file_name).py

2. Code:
   

import json
import sys
import logging
from dateutil import parser
from django.core.management.base import BaseCommand

logger = logging.getLogger(__name__)

class Command(BaseCommand):
    help = "Populating User data obtained in JSON from Monolith."

    def handle(self, *args, **options):
        for line in sys.stdin:
            data = json.loads(line)

            # Populating User Model
            if data["model"] == "User":
                user= User(
                    user_id=data["id"],
                    username=data["username"],
                    mobile_number=data["mobile_number"],
                    created_at=parser.parse(data["created_at"]),
                )
                user.save()
                logger.debug("User populated:{}".format(user.user_id))

After saving this command populate using the following shell command:

cat {data_file_name}.json | python manage.py populate_data

Conclusion:

The above process is a sample code about how you can extract data from monolith and populate in microservice. The above code can be used to get a JSON containing data of multiple tables of monolith in a same file. We can use other file types too like CSV etc. Bulk create could also be used in Django to save all entries in one go.