Pycraft progress report!

Hello there!
It’s Pycraft progress update time, and this week we really started to focus in on testing the installer. Last week we briefly started work on it, but this week we devised an entirely new testing strategy to test the installer.

Something that may potentially cause confusion is that the installer is both the sister project to Pycraft (in that it can run with or without Pycraft installed) and also a system/component of this sister project – as the installer is named after its primary purpose, installing Pycraft. Where possible we will likely be referring to the system, not the sister project as a whole for this progress summary.

With that out the way, lets talk briefly about what we changed in our testing method for the installer this week and how that compares with the way that we tested other areas of Pycraft. So firstly in the old testing process we would simply keep identifying and fixing bugs as we discovered them in a single ‘testing session’, where a ‘testing session’ is a block of time that we will spend working on testing on that system, and this amount of time is flexible to allow for unintentional delays.
This works great for low stakes applications, however when we got to the installer (in this case the entire sister project to Pycraft) we realised that we wanted to make absolutely sure we had identified all the bugs we could find. To do this, instead of splitting our time into ‘testing sessions’ we would instead run passes on each system of the installer (the sister project to Pycraft). This is essentially a 2-step process:

1. Run the system and create a log of all the problems we identify.
2. Fix each problem we identify This process is only run once a day for every system we are testing, which is another big advantage of this in that we can test multiple systems in a single day.

We are making absolutely sure here that we have identified and fixed all the bugs we possibly can, as unlike Pycraft, the installer (the sister project to Pycraft) has the ability to – with the users consent – create, download or remove files. So naturally we want to make the user as confident as we can that this process of installing Pycraft works and we want to be absolutely certain that nothing will go wrong as to avoid damage to personal data. We are also checking in this process to see if something does go wrong, what would happen and will the installer be smart enough to detect the problem and either correct it or otherwise cancel the process.

With the user’s security in mind, we elected to make the following rules about our new pass style system for the large-scale testing of the installer (the sister project to Pycraft):

• We can run only one pass on each system a day.
• We can test multiple systems in series – but not at the same time – to a maximum of 2.
• We will not stop testing a system until we are able to run 2 sequential passes it and identify no bugs.

This additional caution here meant that the entire installer is taking longer to test than the rest of Pycraft, but where the user’s files and personal data are involved, we want to be absolutely certain that nothing goes wrong. However, this does not mean therefore that the rest of Pycraft is still potentially ‘dangerous.’ We stuck with using the same approach to testing the whole of Pycraft as its quick, effective and we have shown it to work for our source code, using the passes style of approach we are using for the installer would work fine for the rest of Pycraft, but at the cost of a considerably longer testing process, as where a single ‘block’ of testing in the old style may take perhaps half a day, but the more stringent pass style approach would take at a minimum 4 times as long.

So, with all the theory out of the way, how did this work in practise? And how far through testing are we?

This pass style approach to testing worked perfectly, and although admittingly has lengthened the time taken to test the installer, this process has meant that we were able to more rigorously test each component/system of the installer. At the time of writing, we are currently finished with the installer and uninstaller and are currently working on the update system, although believe that we will be able to finish that off in a few days’ time.

So, what happens next?

Well, we are very close to a release, this will be the release of Pycraft v9.5.5, although with the extensive testing and reworking of so many features in this update it almost feels like the full release of Pycraft v9.6! Our aim is to release the article for Pycraft in the next few weeks, then in the meantime work on improving docstrings and non-code based areas of Pycraft (for example settings descriptions, as some could do with some considerable improvement), before releasing Pycraft v9.5.5 in mid-December of 2022!

Attached below are some visual highlights from the last week of development!