psLoaders devlog 01: A PySide2 threading helper

I'm a developer in the Visual Effects industry creating the pipeline that ties together different departments and create helpful tools for artists' day-to-day tasks. Most of the GUI apps I write load some stuff in a background thread and once that's done load some more stuff depending on the first stuff and so on until eventually displaying all of the stuff in a UI. When that UI is changed by the user some of the stuff may have to re-load again. Business as usual for me and lots of other developers.

This post series is a devlog for myself and anyone else interested in developing this kind of applications.

Example

For a simple example let's say we've got an app that searches our digital assets library of VFX stock footage and loads it into our Visual Effects program (such as Nuke by Foundry).

Its capabilities are:

1. Search the asset library (query a database server)
2. Show a grid of found assets (display the query results)
3. Show an asset's details and potentially additional info
4. Load the selected asset

Each section in the GUI is dependent on another one having finished loading or having changed. The interactive sections (search, grid, buttons) can trigger a change in the state of the app. Some of these changes can cause duplicate requests and race conditions such as a sending 2 search queries to the database and potentially receiving the second result before the first, causing the search terms to not match the actual result. To avoid this we'd like to lock the chain until each section has completed loading:

I'll explain the flowchart on the right in the Solution section below

The flow of the app is (see yellow arrows above):

• User enters a new search term
  • We query the database server
  • We receive a result from the database server
• The Grid changes a. It finished populating b. The user changes the asset selection
• The Details pane updates
  • If the asset selection has changed: Populate details at the top of the panel and clear the detailed info text
• The "Inspect" button is pressed
  • Query the database for detailed info based on the asset selection
  • Populate the detailed info text
• The Load button is pressed
  • Load the actual file into the VFX app

Whenever any of the sections trigger a change, the whole chain is locked until all sections have completed loading.

The Challenge

A common challenge I often face when writing apps like this is linking all the different loading procedures together in a meaningful manner while keeping the UI responsive or displaying a loading icon, progress bar or similar so the users knows what's happening. I write in Python with PySide2 (Qt for Python) which makes it pretty easy to link loading chains like this up via Slots and Signals. However when the app starts growing it can get messy and hard to manage all the different connections, remembering which loading procedure depends on another and when to lock which part of the UI for input etc.

The Solution

So I've started to write a Python package that aims to make scaling these kinds of apps easier. Its philosophy is to encourage separating each "loading procedure" into its own, separate Loader class. That class then provides a standardized set of Signals, that are usually sent at certain times in the loading procedure. These separate, flexible "blocks" of loading logic are then connected into meaningful "networks" in the main section of the program where we control the flow (meaning: which procedures should wait on each other, which can run simultaneously etc.) of the procedures.

I am still working out how exactly these links should be established but the goal is to create meaningful "networks" or "families" of Loaders that make it clear which can load simultaneously and which sequentially. For example:

In this model the loading chains are quite logical and easy to build. However one limitation is that Loaders from separate "families" can't easily be connected to each other in this way without creating a bit of a mess. Will look into if it would make sense to have "distant relatives" (yes, I like the family analogy 😄)

Next Up

I expect to post updates about this whenever I have made some major progress, either in research or actual development of the module itself. I am of course hoping to make this module useful for a wide range of uses cases but as this can be approached in countless ways it will likely be biased toward a certain software architecture. I hope You'll like to follow along and will find it useful for your project(s). I welcome any comments, thoughts, ideas or experiences with similar challenges You've faced.

Resources

Here are articles and posts that I've found useful in my research so far.

• Concurrent processes in PySide2/PyQt5 applications
• Stateless vs stateful
• Software Architecture