DEV Community

Cover image for Data-Oriented Programming is dope
Xavier Barbosa
Xavier Barbosa

Posted on

Data-Oriented Programming is dope

Photo by Ravi Roshan on Unsplash.
This post was published first in https://techblog.deepki.com/data-oriented-programming/. This is an English translation by the same author.

Data-oriented programming (DOP) is not a new concept. It's a paradigm that can be used by developers in any programming language, it's purpose is to reduce complexity of information system that they are designing.

Yehonathan Sharvit explains that in his book Data-oriented programming.
The book explores tenets of this paradigm, as a dialog between two people.

The narrator is a junior javascript developer, he develops a Library Management System of a client. Initial features are easy, the software is coded in object oriented. But when the client asks for some new features at the very last moment, everything becomes complicated. He seeks support from a veteran developper, Joe.

Throughout these pages Joe demonstrates difficulties he encounters, shows him how to confront them. In fine, he teaches him a new way to organize his source code that is more easy to decipher and to evolve.


Examples in the book are in Javascript, I wanted to introduce my interpretation in Python on a small part of these rules: code and data separation.

In the book, the heroes talk about user management. The narrator had to design two types of users, the librarian and the member:

Image description

Once this logic implemented, the client requested him to add super members, then VIP members.
It achieves to be this UML class diagram:

Image description

It's really hard to manage for the young narrator. Although everything is perfectly logic, the classes hierarchy is hard to work with, mixing inheritance with dependencies.

Joe understands affirms that is "feelings" are due to « Data encapsulation has its merits and drawbacks: Think about the way you designed the Library Management System. According to DOP, the main cause of the complexity of systems and their lack of flexibility is because code and data are mixed together in objects »

That's what Yehonathan Sharvit fights all along the book: he depicts the difficulty to just understand something and be able to upgrade it without difficulties.

Complexity is a thing that have been accumulated insidiously. When it's not kept under control, implementing new features can take weeks instead of days. But DO comes with a radical approch to fight this complexity. To achieve this, data and code must be separated:

Image description

In order to explain this separation, here is my implementation in python.

I've followed technics that are described in the book. I started from the client specifications, I've made a list of names that seem to represent Entities of the system, and another list of everything that look like a feature. Then I've organized what I've found:

  • Two kind of users: members and librarians
  • Users log in to the system via email and password
  • Members can borrow books
  • Members and librarians can search books by title or by author
  • Librarians can block and unblock members
  • Librarians can list the books currently lent by a member
  • There could be several copies of a book

Entities classified by groups:

Image description

Features put in several code modules:

Image description

On this basis, I will implement book lending.

The catalog's data part:

$schema: "https://json-schema.org/draft/2020-12/schema"
properties:
  lendings:
    additionalProperties:
      type: object
      properties:
        id: { type: string }
        user_id: { type: string, format: uuid }
        book_item_id: { type: string }
      required: [id, user_email, book_item_id]
  propertyNames: { type: string, format: uuid }
required: [lendings]
Enter fullscreen mode Exit fullscreen mode

The user_management's data part:

$schema: "https://json-schema.org/draft/2020-12/schema"
properties:
  members_by_id:
    type: object
    additionalProperties:
      type: object
      properties:
        is_blocked: { type: boolean }
      required: [is_blocked]
    propertyNames: { type: string, format: uuid }
required: [members_by_id]
Enter fullscreen mode Exit fullscreen mode

Here I've used JSON Schema, because data does not have to be contained in rigid structures. Only the keys are relevant and need to be specified. In DO, data requires to obey three other rules:

  • all types are generic
  • all types are immutable
  • shape of data and data schema are separated

Here is a mock that validates these two schemas:

library_data = {
    "catalog": {
        "books_by_isbn": {
            "9781234567897": {
                "title": "Data Oriented Programming",
                "author": "Yehonathan Sharvit",
            }
        },
        "book_items_by_id": {
            "book-item-1": {
                "isbn": "9781617298578",
            },
            "book-item-2": {
                "isbn": "9781617298578",
            }
        },
        "lendings": [
            {
                "id": "...",
                "user_id": "member-1",
                "book_item_id": "book-item-1",
            }
        ],
    },
    "user_management": {
        "members_by_id": {
            "member-1": {
                "id": "member-1",
                "name": "Xavier B.",
                "email": "xavier@deepki.com",
                "password": "aG93IGRhcmUgeW91IQ==",
                "is_blocked": False,
            }
        }
    },
}
Enter fullscreen mode Exit fullscreen mode

By convention, dict are used like some Mapping, and I forbid myself to update them.

Please note that examples will use the classes+static method form in order to make this article readable. In a production code, the modules+functions form is the way to go.

And now the code part:

from __future__ import annotations

from typing import Tuple, TypeVar
from uuid import uuid4

T = TypeVar("T")


class Library:
    @staticmethod
    def checkout(library_data: T, user_id, book_item_id) -> tuple[T, dict]:
        user_management_data = library_data["user_management"]
        if not UserManagement.is_member(user_management_data, user_id):
            raise Exception("Only members can borrow books")
        if UserManagement.is_blocked(user_management_data, user_id):
            raise Exception("Member cannot borrow book because he is bloqued")
        catalog_data = library_data["catalog"]
        if not Catalog.is_available(catalog_data, book_item_id):
            raise Exception("Book is already borrowed")
        catalog_data, lending = Catalog.checkout(catalog_data, book_item_id, user_id)
        return (
            library_data | {
                "catalog": catalog_data,
            },
            lending,
        )


class UserManagement:
    @staticmethod
    def is_member(user_management_data: T, user_id) -> bool:
        return user_id in user_management_data["members_by_id"]

    @staticmethod
    def is_blocked(user_management_data: T, user_id) -> bool:
        return user_management_data["members_by_id"][user_id]["is_blocked"] is True


class Catalog:
    @staticmethod
    def is_available(catalog_data: T, book_item_id) -> bool:
        lendings = catalog_data["lendings"]
        return all(lending["book_item_id"] != book_item_id for lending in lendings)

    @staticmethod
    def checkout(catalog_data: T, book_item_id, user_id) -> Tuple[T, dict]:
        lending_id = uuid4().__str__()
        lending = {"id": lending_id, "user_id": user_id, "book_item_id": book_item_id}
        lendings = catalog_data["lendings"]
        return (
            catalog_data | {
                "lendings": lendings + [lending]
            },
            lending
        )
Enter fullscreen mode Exit fullscreen mode

As we can see, code is a series of pure functions.
Functions that modify a state return a new state object rather than upgrading the previous state.

In each module, functions are made simple and easy to test. They can be reused in any context, like the main module. Globally, they are composed with other existing function. It becomes very easy to adapt them for the client's needs.

And now, which path will data lead if my alter-ego borrows another copy of the book?

library_data, lending = Library.checkout(
    library_data,
    user_id="member-1",
    book_item_id="book-item-2",
)
Enter fullscreen mode Exit fullscreen mode

Two things occur:

  1. Data is systematically transmitted to every function calls. This object is quite opaque, each level use only a fragment that he knows without worrying about the remaining:

    # 1. injects data into Library.checkout module
    library_data, lending = Library.checkout(library_data, ...)
    
    # 2. extracts data from user_management
    user_management_data = library_data["user_management"]
    
    # 3. uses this data fragment into UserManagement module
    if not UserManagement.is_member(user_management_data, ...):
        ...
    if UserManagement.is_blocked(user_management_data, ...):
        ...
    
    # 4. picks catalog data
    catalog_data = library_data["catalog"]
    
    # 5. uses this data fragment into Catalog module
    if not Catalog.is_available(catalog_data, ...):
        ...
    ... = Catalog.checkout(catalog_data, ...)
    
  2. When a function is about to change a state, it returns a new version of data. Every level of the call stack must returns a new version of data:

    # 1. handles the request in Catalog.checkout
    lending = ...
    lendings = catalog_data["lendings"]
    # 2. creates a new version of catalog_data
    catalog_data = catalog_data | {
        "lendings": lendings + [lending]
    }
    # 3. interception of the new catalog_data by Library.checkout
    catalog_data, ... = Catalog.checkout(...)
    # 4. creation of a new version of library_data
    library_data = library_data | {
        "catalog": catalog_data,
    }
    

    Then, this new version of data can be exposed to whole system.

In my example, I don't talk about data consolidation. I suggest you to read the book which gives informations concerning this subject.

Is it pythonic?

Broadly speaking, this paradigm fits well in Python if we shelve the object oriented capabilities of the language.
The notion of modules in data-oriented are naturally superimposed on modules in Python, which facilitates adherence.

Borrowings of functional languages such as map(), filter() as well as functions from standard operator module also contribute to make this paradigm quite natural in Python.

In our example, standard typing will not work. However, it is quite easy to do custom typing, such as:

from __future__ import annotations

from typing import Any, Mapping


class M(Mapping[str, Any]):
    def __or__(self, other: dict) -> M:
        return {**self, **other}  # type: ignore

    def __hash__(self) -> int:
        ...

# which can be used as well in the source code

def my_func(members_data: M[str, M], member_id: str):
    member = members_data[member_id]
Enter fullscreen mode Exit fullscreen mode

Code purity

In my implementation my functions are not really pure because I have used exceptions. However, this disgression is acceptable if it is applied under a certain condition. Indeed, exceptions are only used to express illegal operations, in a throw early, catch late way. Using them this way contributes to the readability of the code. The higher layers of the system will know how to deal with them.

For example in a Flask application:

@app.post("/checkout")
def checkout_view():
    ...
    try:
        ..., lending = Library.checkout(library_data, user_id, book_item_id)
        # la fonction 'checkout' peut lever des exceptions
        ...
        return jsonify(lending), 201
    except Exception as error:
        result = {"error": str(error)}
        return jsonify(result), 400
Enter fullscreen mode Exit fullscreen mode

The author is also clear that embracing DO comes at a cost. For example, the fact that DO is relatively agnostic of any programming language undermines the guarantees offered by object modeling (or other tools such as code analysis that some IDEs allow). However, he sometimes offers alternatives for this, such as JSON Schema used here.

What I presented to you was just a preview of DOP using pure Python. The author gives a lot of details about unit tests, data structures, state management, structural sharing, atomicity, transformation pipeline, etc.

I highly recommend you to read Data-oriented programming by Yehonathan Sharvit, and please follow his approach in his blog.

Finally

The author is a multilingual developer and without citing it, many concepts come from the Clojure language. According to its defenders, Clojure is the easiest programming language in the world because it has almost no syntax or grammar and it was designed by Rich Hickey in such a way as to facilitate code changes.

This language can be inspiring for other languages. To convince yourself, you can consult this other talk Design, Composition, and Performance Short by Rich Hickey.

It makes me happy to see some of these principles reused in other languages. Indeed, languages must nurture on each other.

Bref, making functional code in Python is dope.

Discussion (3)

Collapse
vrolok profile image
Vladislav Bereznyuk

Thanks for a brief and interesting intro to DOP. Now, I'm off to a book.

Collapse
ivanpierre profile image
Ivan Pierre

Nice showcase... ;)

Collapse
xba profile image
Xavier Barbosa Author

thank you :)