What is Soft Deletion?
Sometimes, you want to delete something from your database without actually deleting it. You want to preserve it, in case you ever decide to restore it in the future, or to use it for some other purpose, such as analytics … but it should still appear as if it were actually deleted, not show up in unexpected places in your application. This is called the 'soft delete' pattern.
Instead of actually deleting a record from your database, which is irreversible, you simply 'mark' it as deleted, usually in another column in your table. Now the challenge remains, how to prevent soft-deleted records from 'leaking' into your application.
In this article, we will learn how to implement soft deletion in Django by taking the example of a simple note-taking app backend.
Soft Delete Model
We will start by creating a base SoftDeleteModel
that can be inherited by the rest of our models.
class SoftDeleteModel(models.Model):
is_deleted = models.BooleanField(default=False)
def soft_delete(self):
self.is_deleted = True
self.save()
def restore(self):
self.is_deleted = False
self.save()
class Meta:
abstract = True
Note that we have marked this model as abstract = True
. This means that Django will not create a database table for it.
Now, we can create our models as subclasses of SoftDeleteModel
to grant them the ability to be soft-deleted. Let's take the example of a Note
model.
class Note(SoftDeleteModel):
user = models.ForeignKey(User, on_delete=models.CASCADE, related_name="notes")
title = models.CharField(max_length=32)
content = models.CharField(max_length=255)
We can query our database filtering by is_deleted
in order to exclude records that have been soft-deleted.
Note.objects.filter(is_deleted=False)
Trying It Out
Let's try playing around a bit with the code we've written so far. First, open up the Django shell by typing python manage.py shell
in your terminal.
Import the models required:
from django.contrib.auth.models import User
from tutorialapp.models import Note
Since each note is foreign-keyed to a user, our first step is to create a User
object:
john = User.objects.create_user('john', 'lennon@thebeatles.com', 'johnpassword')
Now we can create a couple of notes:
my_note = Note.objects.create(user=john, title="Strawberry Fields", content="Strawberry Fields Forever")
another_note = Note.objects.create(user=john, title="Here Comes The Sun", content="It's All Right")
You are now ready to soft delete and restore notes:
my_note.soft_delete()
my_note.restore()
You can query for all notes, whether they have been soft deleted or not:
Note.objects.all()
You can also filter only for notes that have not been soft deleted:
Note.objects.filter(is_deleted=False)
Soft Delete Manager
While our code is functionally correct, the disadvantage is that we will have to remember to filter by is_deleted=False
each time we write a query.
We can improve upon this behaviour by creating a custom model manager to apply the filter automatically, behind the scenes. If you've used Django in the past, you might be familiar with statements that look like this: MyModel.objects.all()
. The .objects
part in the statement is the manager. Managers act as the 'bridge' between your Django code and the database. They control the database operations performed on the tables that they 'manage'.
Our new custom manager can be defined as:
class SoftDeleteManager(models.Manager):
def get_queryset(self):
return super().get_queryset().filter(is_deleted=False)
We will need to add the new manager to our SoftDeleteModel
base class:
class SoftDeleteModel(models.Model):
is_deleted = models.BooleanField(default=False)
objects = models.Manager()
undeleted_objects = SoftDeleteManager()
def soft_delete(self):
self.is_deleted = True
self.save()
def restore(self):
self.is_deleted = False
self.save()
class Meta:
abstract = True
Note that, since we have added a custom manager to our class, we are required to explicitly add the default objects
manager as well.
Then, we can simply rewrite our query as,
Note.undeleted_objects.all()
to get a QuerySet
of undeleted notes.
We can still use
Note.objects.all()
to get the full list of notes, including those that have been soft-deleted.
Handling Foreign Key Relationships
Now, what if you have multiple users, and you want to fetch all the notes belonging to a specific user? The naive approach is to simply write a query filtering against the user:
Note.objects.filter(user=john, is_deleted=False)
However, a more elegant and readable solution is to make use of the reverse relationships Django provides for this purpose.
john.notes.all()
Try soft deleting some of your notes and running this query. Do you notice something unusual about the results?
We find that the resultant QuerySet
contains records that we had soft deleted. This is because Django is using the default objects
manager to perform the reverse lookup, which, as you may recall, does not filter out soft deleted records.
How can we force Django to use our custom SoftDeleteManager
to perform reverse lookups? We can simply replace the default objects
manager in our SoftDeleteModel
:
class SoftDeleteModel(models.Model):
is_deleted = models.BooleanField(default=False)
objects = SoftDeleteManager()
all_objects = models.Manager()
def soft_delete(self):
self.is_deleted = True
self.save()
def restore(self):
self.is_deleted = False
self.save()
class Meta:
abstract = True
Now, the objects
manager will automatically filter out soft-deleted objects when querying our database, ensuring they never leak into our application under any circumstances! If we want to, we can still include soft deleted objects in our queries by making use of the all_objects
manager.
Note.all_objects.all()
Storing More Information
We've already got a pretty solid soft deletion framework in our Django app, but we can make one final improvement. Knowing whether a record is soft deleted or not is useful, but another piece of information that would be nice to know is when the record was soft deleted. For this, we can add a new a attribute deleted_at
to our SoftDeleteModel
:
deleted_at = models.DateTimeField(null=True, default=None)
We can also update our soft_delete
and restore
methods as follows:
def soft_delete(self):
self.deleted_at = timezone.now()
self.save()
def restore(self):
self.deleted_at = None
self.save()
For undeleted records, the value of deleted_at
will be null, while for soft deleted records, it will contain the date and time at which it was deleted.
The addition of the new deleted_at
attribute makes our previously created is_deleted
attribute redundant, because we can simply perform a null-check on deleted_at
to find out whether the record is soft deleted or not.
Our rewritten SoftDeleteModel
now looks like this:
class SoftDeleteModel(models.Model):
deleted_at = models.DateTimeField(null=True, default=None)
objects = SoftDeleteManager()
all_objects = models.Manager()
def soft_delete(self):
self.deleted_at = timezone.now()
self.save()
def restore(self):
self.deleted_at = None
self.save()
class Meta:
abstract = True
And our rewritten SoftDeleteManager
looks like this:
class SoftDeleteManager(models.Manager):
def get_queryset(self):
return super().get_queryset().filter(deleted_at__isnull=True)
In addition to our previous capabilities, we can now also see the exact date and time at which our record was soft deleted:
my_note.deleted_at
In Conclusion
Soft deletion is a powerful software pattern that comes with several advantages, including better data preservation and restoration, history tracking, and faster recovery from failures.
At the same time, it should be used with care. Sensitive and personal data including payment-related information should always be hard-deleted. Users should always have the option to have their data permanently deleted, if they wish. Several jurisdictions around the world have information privacy and data protection laws that include the 'right to be forgotten', such as the European Union's GDPR. It might also make sense to periodically delete or archive data that is very old, to avoid eating up excess database storage space.
If you would like to view the complete source code for the example used in this tutorial, it is available on GitHub.
References and Further Reading
How to use managers in Django
What Are Soft Deletes, and How Are They Implemented?
Soft Deletion in Django
Top comments (7)
Nice article and an important subject. A more sophisticated pattern is to implement a valid timestamp-range to allow a certain object to be valid for some timestamps and not valid for others. Implementation is similar to the above, with different operators and a lot more functionality.
Thanks Joost! The valid timestamp range pattern is an interesting extension of the soft deletion functionality. I suppose it will give us the capability to view the complete history if the object has been deleted and restored multiple times, am I right?
Yes, if you use an array of timestamp ranges, anything is possible.
the cascade is not handled though
Could you clarify on that?
I think he means, what happens if you delete a User? Is there a way to delete the User and Keep the notes? Thanks for sharing, great article!
This is a really nice approach! Is there a way to still show all_objects within the admin panel?