Introduction & Motivation

In today's digital age, software development has become an increasingly important field of study, with universities and colleges offering a range of courses to prepare students for the industry. One such course is the Database System course on the Faculty of Informatics and Information Technologies STU in Bratislava, which teaches students how to design, build and manage databases. As part of this course, students have to create an HTTP server which interacts with a database, to demonstrate their understanding of the concepts covered in the course.

However, running and validating these HTTP servers can be a time-consuming and challenging task for instructors, especially as the number of students increases. To address this problem, an web application was developed which can accept Docker images created by students, run the image, provide environment variables for database connections and perform a set of HTTP requests to validate the implementation. This web-based application is created using the Django web framework, uses Redis for queue management and the Docker Python SDK to communicate with the Docker Daemon.

By automating the testing process, this application not only saves time for instructors but also ensures a consistent and fair evaluation of student work. Moreover, it provides students with an opportunity to learn about the use of Docker in a real-world scenario, thereby preparing them for the industry.

This post is structured as follows: The first chapter Requirements and Design, describes the requirements for such an application, defines its processes, breaks it down into logical components, and proposes a data model. The second chapter Implementation, provides an introduction to key implementation issues, such as implementing asynchronous tasks and LDAP authentication. It also showcases the usage of Docker with Python SDK in the project, including network configuration, and describes the deployment configuration using supervisord. The final chapter summarizes the efforts and provides links to the code repositories.

Requirements and design

Our university courses rely on GitHub for Education. Our idea was to build a web application that enables students to log in with their academic credentials (OpenLDAP) and submit a link to their Docker image hosted by GitHub. The simplified workflow is shown in the sequence diagram below.

When students submit a URL to their Docker image, details are saved in the database and a task is added to the Redis queue. The student is then redirected to the detail page of the Task, which refreshes periodically and shows the current status of the task (including its results if finished).

Tasks are processed by the worker, as visualized in the simplified diagram below. We use the django-rq package to implement an async queue using Redis.

The worker creates a Docker container for the student from the provided image and performs a series of pre-defined URL requests. The results are saved to the database.

The database model of the application is shown in the image below.

The following entities are present in the database model:

• Assignment: Assignments for students that they have to complete during the term.
• Scenario: The definition of the test request for a specific Assignment (including expected results). Some scenarios are private and are only used by the course supervisors for the final evaluation.
• Task: The test request created by the student. It contains information about the Docker image and waits in the queue until processing. The entity has the following states: ok, fail and pending.
• TaskRecord: An instance of a Scenario for the Task. Entity has following states: success, fail and mismatch.
• AuthSource: LDAP configurations.
• AuthUser: User definitions.

The entire project is shown in the component diagram below. The Tester package represents our application. We use PostgreSQL for data storage, Redis for the queue, and Docker for containerization.

The diagrams presented are simplified to improve readability.

Implementation

The implementation of the application was done using the Django web framework. Django was chosen for its user-friendly ORM, customizable authentication, auto-generated administration, and rich ecosystem of libraries and resources. In this chapter, we will take a closer look at some of the key implementation issues and how they were addressed.

The chapter is structured into five main sections, each covering a specific aspect of the implementation. The first section covers the implementation of asynchronous tasks, which were necessary to handle the running of Docker images in a non-blocking and scalable way. The second section discusses the implementation of LDAP authentication, which was used to authenticate users against the university's OpenLDAP server.

The third section delves into the usage of the Python Docker SDK in the project. This SDK was used to communicate with the Docker daemon and perform various operations, such as creating and managing Docker containers. The fourth section is dedicated to configuration of performing the HTTP requests using the long-pooling method (in case if the container applications is not yet fully loaded).
Finally, the fifth section describes the deployment using the supervisord and systemd.

Asynchronous tasks

Performing tests on student images can be a time-consuming process, taking several minutes. Application has to download the image from the registry, create a container, wait for the application to load, and perform the test scenarios. To efficiently utilize the available hardware and handle increased server demand during peak times, a task queue was implemented using django-rq.

Student creates a Task, which contains all the necessary information about the test scenarios that will be executed (by providing a path to the Docker image and choosing a Assignment). The task is then added to the Redis queue, which is consumed by workers that process the tasks. Redis provides atomic access to its structures, ensuring that the same task is not processed multiple times.

The implementation of the task queue can be divided into three main steps: configuring the queue, implementing the workers, and adding tasks to the queue. The first step involves configuring the Redis queue using Django settings. The second step requires implementing the workers to process the tasks in the queue, which is done by creating a Python function that performs the necessary test scenarios. The final step involves adding tasks to the queue, which is done by calling the function with the necessary parameters and pushing it onto the queue.

Configuring the queue

Configuring the task queue involves defining the Redis queue and its connection settings in the Django settings file. The example below defines a task queue called default and connects to the Redis server using REDIS_HOST, REDIS_PORT, RQ_REDIS_DB, and REDIS_PASSWORD environment variables (with fallback to localhost:6379 on database 0 without a password).

# dbs_tester/settings/base.py
RQ_QUEUES = {
    'default': {
        'HOST': os.getenv('REDIS_HOST', 'localhost'),
        'PORT': int(os.getenv('REDIS_PORT', 6379)),
        'DB': int(os.getenv('RQ_REDIS_DB', 0)),
        'PASSWORD': os.getenv('REDIS_PASSWORD', None),
        'DEFAULT_TIMEOUT': 360,
    }
}

RQ_EXCEPTION_HANDLERS = [
    'apps.core.jobs.exception_handler'
]

RQ_SHOW_ADMIN_LINK = True

The RQ_QUEUES setting defines the task queue and its connection settings. The DEFAULT_TIMEOUT setting defines the default timeout for tasks in the queue.

The RQ_EXCEPTION_HANDLERS setting is used to define custom exception handlers for handling task failures. To handle failed tasks, a custom exception handler is implemented in the Django application. When a task fails with an exception, the custom exception handler can change the state of the task. This ensures that failed tests are properly recorded and can be reviewed later.

# apps/core/jobs.py
def exception_handler(job, exc_type, exc_value, traceback):
    try:
        task = Task.objects.get(pk=job.args[0])
    except Task.DoesNotExist:
        return

    task.status = Task.Status.FAILED
    task.message = str(exc_value)
    task.save()

When a task fails with an exception, the exception handler is triggered, and it changes the state of the task to fail. Additionally, the handler saves information about the exception, which can be used to help diagnose and fix any issues with the student's container.

By providing this information to the students, they can quickly identify and fix any issues with their containers, leading to a more efficient testing process. This approach helps to ensure that students are able to learn from their mistakes and improve their skills, while also ensuring that the application is able to provide accurate and reliable feedback.

The RQ_SHOW_ADMIN_LINK setting is used to provide access to a simple web-based interface for monitoring the status of the task queue. This setting displays a link to the Django administration panel on the RQ dashboard, providing a convenient way to monitor and manage the task queue.

With this approach, multiple tasks can be processed in parallel, making the process more scalable and efficient.

Implementing the workers

In this implementation, the worker is defined as a class called BasicJob, which contains four methods: prepare, run, cleanup, and a static method called execute. The workflow for the worker is defined in the Requirements and Design chapter of the blog post.

# apps/core/jobs.py
class BasicJob:
    def __init__(self, task: Task, public_only: bool):
        self._task = task
        self._public_only = public_only
        self._database_name = ''.join(random.choices(string.ascii_letters, k=10)).lower()
        self._database_password = ''.join(random.choices(string.ascii_letters, k=10)).lower()

    def prepare(self):
        with connection.cursor() as cursor:
            cursor.execute(
                f"CREATE DATABASE {self._database_name} TEMPLATE {self._task.assigment.database or 'template0'};"
            )
            cursor.execute(
                f"CREATE USER {self._database_name} WITH ENCRYPTED PASSWORD '{self._database_password}';"
            )
            cursor.execute(f"GRANT ALL PRIVILEGES ON DATABASE {self._database_name} TO {self._database_name};")

    def run(self):
        pass

    def cleanup(self):
        with connection.cursor() as cursor:
            cursor.execute(f"DROP DATABASE {self._database_name};")
            cursor.execute(f"DROP USER {self._database_name};")

    @staticmethod
    def execute(task_id: UUID, public_only: bool) -> Optional[Task]:
        try:
            task = Task.objects.get(pk=task_id)
        except Task.DoesNotExist:
            logging.error("Task %s does not exist!", task_id)
            return None

        if task.status != Task.Status.PENDING:
            logging.warning("Task %s is already done! Skipping.", task.pk)
            return None

        job = BasicJob(task, public_only)
        job.prepare()
        try:
            job.run()
        except Exception as e:
            task.status = Task.Status.FAILED
            task.message = str(e)
            task.save()

        job.cleanup()

        return task

The prepare method creates a temporary database user with a database according to the Assignment specification, while the run method executes the container and performs the tests. The cleanup method removes the temporary user and database. We use the super sophisticated design pattern called Gotta catch them all to catch any exceptions that may be raised from the run method (I was not able to find the author of the image so I can't credit him).

However, workers cannot be specified as a class or static method. Instead, only a simple Python function can be used. To work around this limitation, a simple wrapper was implemented that calls a static method to create and execute BasicJob. The attributes for our job are the task_id, which is the UUID of the student task, and the public_only flag, which is used to determine if only public tests will be performed. Private scenarios were also defined for the final evaluation, so students have to think about edge-cases.

# apps/core/jobs.py
def basic_job(task_id: UUID, public_only: bool) -> Optional[Task]:
    return BasicJob.execute(task_id, public_only)

Adding tasks to the queue

Once we have our task definition prepared, we can create a view that will add it to the queue.

# apps/web/views/tasks.py
class CrateTaskView(LoginRequiredMixin, CreateView):
    model = Task
    form_class = TaskForm
    template_name = 'web/task.html'

    def __init__(self):
        self.object = None

    def form_valid(self, form):
        self.object = form.save(commit=False)
        self.object.user = self.request.user
        self.object.save()
        django_rq.enqueue(basic_job, self.object.pk, not self.request.user.is_staff)
        return HttpResponseRedirect(self.get_success_url())

In the provided code block, the CrateTaskView uses LoginRequiredMixin to ensure that only logged-in users can add tasks to the queue. CreateView is used to simplify the implementation of the form validation. If the TaskForm is valid, the form_valid method is executed.

The form_valid method first creates a new Task object from the form data and saves it to the database. The django_rq.enqueue method is then used to add the apps.core.jobs.basic_job function to the queue, along with the task.id and a flag to determine if only public tests will be performed (only staff users can perform private scenarios). The basic_job function will then be executed by the worker.

LDAP

To ensure that only users from the university can access the application, LDAP authentication was implemented. To make the application as customizable as possible, we created a database entity called auth_source that stores the LDAP configurations. A custom Django authentication backend called LdapBackend was developed, which uses the auth_source to create user accounts on login.

# apps/core/auth/LdapBackend.py
class LdapBackend(ModelBackend):
    class Config(TypedDict):
        URI: str
        ROOT_DN: str
        BIND: str
        USER_ATTR_MAP: Dict[str, str]
        GROUP_MAP: Dict[str, str]
        FILTER: str

    def _ldap(self, username: str, password: str, auth_source: AuthSource) -> Optional[User]:
        config: LdapBackend.Config = auth_source.content
        connection = ldap.initialize(uri=config['URI'])
        connection.set_option(ldap.OPT_REFERRALS, 0)

        try:
            connection.simple_bind_s(config['BIND'].format(username=username), password)
        except ldap.LDAPError as e:
            logging.warning(
                f"Unable to bind with external service (id={auth_source.pk}, name={auth_source.name}): {e}"
            )
            return None

        try:
            user = User.objects.get(username=username)
        except User.DoesNotExist:
            user = User(
                username=username,
            )
            user.set_unusable_password()

        result = connection.search(
            f"{config['ROOT_DN']}", ldap.SCOPE_SUBTREE, config['FILTER'].format(username=username), ['*']
        )

        user_type, profiles = connection.result(result, 60)

        if profiles:
            name, attrs = profiles[0]

            # LDAP properties
            for model_property, ldap_property in config['USER_ATTR_MAP'].items():
                setattr(user, model_property, attrs[ldap_property][0].decode())

            user.last_login = timezone.now()
            user.save()

            # LDAP groups
            user.groups.clear()
            for ldap_group in attrs.get('memberOf', []):
                if ldap_group.decode() in config['GROUP_MAP']:
                    try:
                        group = Group.objects.get(name=config['GROUP_MAP'][ldap_group.decode()])
                    except Group.DoesNotExist:
                        continue
                    user.groups.add(group)
        else:
            logging.warning(
                f"Could not find user profile for {username} in auth source {auth_source.name}"
                f" (id={auth_source.pk}, name={auth_source.name})"
            )
            return None

        connection.unbind()

        user.save()

        return user

    def authenticate(self, request, username=None, password=None, **kwargs):
        user = None

        for auth_source in AuthSource.objects.filter(is_active=True):
            logging.debug(f'Checking {auth_source.name}')
            if auth_source.driver == AuthSource.Driver.LDAP:
                user = self._ldap(username, password, auth_source)

            if user:
                break

        return user

The _ldap method establish a connection with the LDAP server. If the user credentials are valid, it retrieves the user profile and creates a new user account in th database, if one does not already exist. It also updates the user's information, including their groups.

To use the LdapBackend authentication backend, it must be added to the AUTHENTICATION_BACKENDS list in the Django settings.

# dbs_tester/settings/base.py
AUTHENTICATION_BACKENDS = [
    'django.contrib.auth.backends.ModelBackend',
    'apps.core.auth.LdapBackend'
]

With this implementation, only users from the university who have valid LDAP credentials can access the application.

Docker

As previously mentioned, the students' assignments are submitted as Docker images hosted on the GitHub Container Registry, and to run these images, the machine running the application needs to have access to this registry. This is a one-time configuration that needs to be done on the production server using the docker login command.

We use the official Docker Python SDK to manipulate images and containers. All students' containers run in a separate NAT Docker network.

To illustrate the creation of a Docker container, we can refer to the example code below:

# apps/web/views/tasks.py
client = docker.from_env()

params = {
    'image': self._task.image,
    'detach': True,
    'environment': {
        'DATABASE_HOST': settings.DATABASES['default']['HOST'],
        'DATABASE_PORT': settings.DATABASES['default']['PORT'],
        'DATABASE_NAME': self._database_name,
        'DATABASE_USER': self._database_name,
        'DATABASE_PASSWORD': self._database_password,
    },
    'name': self._task.id,
    'privileged': False,
    'network': settings.DBS_DOCKER_NETWORK,
    'extra_hosts': {
        'host.docker.internal': 'host-gateway',
        'docker.for.mac.localhost': 'host-gateway'
    },
    'ports': {
        '8000/tcp': '9050'
    }
}

container: Container = client.containers.run(**params)
sleep(5)
container.reload()

The docker.from_env() method creates a Docker client connected to the Docker daemon running on the local machine, which allows us to manage the Docker containers and images. Then, we define the parameters for the container and run the container using the client.containers.run() method.

When creating a new container, we pass a dictionary object to the run() method which contains the required parameters, such as:

• image: the name of the image to create a container from,
• detach: to run the container in the background and return immediately,
• environment: a dictionary of environment variables to set inside the container,
• network: the name of the NAT Docker network that the container is connected to.

We use the environment variables to pass information about the database connection with the temporary credentials (which will be deleted after the test).

We also pass extra hosts which allow us to access the host's Docker daemon from within the container. This is required when using Docker for Mac or Docker for Windows. Finally, we specify port mapping using the ports to make the container's application accessible from the host machine.

In this example, we also wait five seconds to let the container initialize before reloading the container object. This is necessary to get the container's IP address so we can make HTTP requests to it from the test runner.

To retrieve the container IP address, the container.reload() method must first be called. The network adapters can then be accessed via container.attrs['NetworkSettings']['Networks']. Since the network is specified using the network parameter, the IP address can be accessed using the following code:

container: Container = client.containers.run(**params)
sleep(5)
container.reload()

ip_address = container.attrs['NetworkSettings']['Networks'][settings.DBS_DOCKER_NETWORK]['IPAddress']
print(ip_address)

As we want to minimize the execution time of the tests, we run them asynchronously in the background. Since the tests are run on separate Docker containers, we need to make sure that the containers are stopped after the tests are completed. This can be achieved by using the container.stop() method, which stops the container, and then using the container.remove() method to remove the container from the Docker host.

Performing requests

In order to handle cases where the student's application might take longer to load, we use the requests package with a Retry object to perform HTTP requests. The Retry object is configured to attempt 6 requests with a 2 second backoff if an attempt fails. The Session object is used to persist parameters across requests and to take advantage of connection pooling.

from requests import Session, Request
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

s = Session()
retry = Retry(connect=6, backoff_factor=2)
adapter = HTTPAdapter(max_retries=retry)
s.mount('http://', adapter)
s.mount('https://', adapter)
req = Request(
    method=scenario.method,
    url=record.url,
)

Deployment

The application is currently deployed as a single Docker container, which utilizes supervisord to run all the necessary modules including the Django application and the workers. The Docker image is mounted to the path to the Docker socket from the host machine, which allows it to operate in a docker-in-docker manner.

The supervisord configuration file contains two programs - gunicorn and worker. The gunicorn program runs the Django application, while the worker program executes the workers using RQ. The number of worker processes can be specified using the numprocs parameter.

[supervisord]
nodaemon=true

[program:gunicorn]
directory=/usr/src/app
command=/opt/venv/bin/gunicorn -w 4 --log-level=debug --log-file=/var/log/gunicorn.log --bind unix:/var/run/gunicorn/gunicorn.socket --pid /var/run/gunicorn/gunicorn.pid dbs_tester.wsgi
autostart=true
autorestart=true
priority=900
stdout_logfile=/var/log/gunicorn.stdout.log
stderr_logfile=/var/log/gunicorn.stderr.log

[program:worker]
directory=/usr/src/app
command=/opt/venv/bin/python manage.py rqworker default
autostart=true
autorestart=true
priority=900
stdout_logfile=/var/log/rqworker.stdout.log
stderr_logfile=/var/log/rqworker.stderr.log
numprocs=4
process_name=%(program_name)s_%(process_num)02d

The application is executed using Systemd by running a Docker image. A unit file is used to configure the service. The Docker image is executed with parameters specifying the environment variables used in the Django settings file. The add-host parameter is used to make the host.docker.internal accessible from within the container. The unit file specifies that the docker.service is required and the network-online.target is wanted before the service is started. The ExecStartPre section is used to stop and remove the existing container before starting a new one. The ExecStartPost section is used to remove the container after stopping it. The dbs network is used to allow communication between the application container and other containers running in the same network.

[Unit]
Description=DBS Tester
Wants=network-online.target
After=network-online.target docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker stop -t 5 dbs-tester
ExecStartPre=-/usr/bin/docker rm dbs-tester
ExecStartPre=-/usr/bin/docker pull ghcr.io/fiit-databases/tester:master
ExecStart=/usr/bin/docker run -p 9200:9000 -v /var/run/docker.sock:/var/run/docker.sock -v ./logs:/var/log/ --env BASE_URL= --env ALLOWED_HOSTS= --env DATABASE_HOST= --env DATABASE_NAME= --env DATABASE_PASSWORD= --env DATABASE_PORT= --env DATABASE_USER= --env DJANGO_SETTINGS_MODULE=dbs_tester.settings.production --env REDIS_HOST= --env SECRET_KEY= --env GITHUB_TOKEN= --env GITHUB_USER= --name dbs-tester --network dbs --add-host=host.docker.internal:host-gateway ghcr.io/fiit-databases/tester:master
ExecStop=/usr/bin/docker stop -t 5 dbs-tester
ExecStopPost=-/usr/bin/docker rm dbs-tester

[Install]
WantedBy=multi-user.target

Conclusion

In conclusion, we have developed a web application for testing Docker images submitted by students using a series of predefined URL requests. We have used various technologies, such as Django, Redis, PostgreSQL, and Docker, to create a customisable and efficient system for automating the evaluation of student assignments.

The application includes several features to improve security, such as using LDAP authentication to restrict access to authenticated users from our university, and running student containers in a separate NAT Docker network.

The system is designed to handle a large number of simultaneous requests, and we have used the django-rq library to implement an asynchronous queue using Redis. This approach ensures that the system is scalable and can process a large number of requests efficiently.

Overall, the system provides an automated and efficient way to evaluate student assignments, reducing the workload of instructors and providing a more objective evaluation process.

The source code of the project is available on the GitHub repository FIIT-Databases/tester. Additionally, an example assignment for the project is also available on GitHub as FIIT-Databases/dbs-python-example. The project is being used in the education process at the Faculty of Informatics and Information Technologies STU in Bratislava. Anyone can access and use the project to create their own testing platform.

In addition to the features and problems that were mentioned, there are many more that were encountered during the development of this project. To get a more detailed overview, feel free to check the repositories provided or ask any questions in the comments. Your feedback and questions are appreciated, and I look forward to hearing from you!