ClickyController

For my final year project I wanted to do something related to computer task automation. I got into programming from a book by Al Sweigart called 'Automate the Boring Stuff with Python' and have used what I learned in that book a lot in my everyday life.

Originally, I wanted to make a script or program that would automate a task, but couldn't really think of anything that would have enough substance to work on for the duration of the year. Instead, I took inspiration from the Python library PyAutoGUI and decided to make my own GUI automation library in C# for use by programmers. I would also attempt to make a WPF application that would sit on top of this library - allowing people without programming experience to automate GUI interaction tasks.

Why C#? It's one of the programming languages that I use the most, meaning that it would be easier for me to start working on the project and it can also be used as a scripting language (although it's not traditionally used in this fashion).

Link to Code

How I built it

I kept a physical developer diary during development to refer back to in the future for potential future projects and to use in my dissertation. There are a lot of notes and my handwriting isn't the greatest, so I'll try to write out some key developments below.

Most of the project relies on the Windows API to move the mouse or send keyboard input, so a lot of the initial research went into learning how to use it and what was available. The API takes various structs containing the required data for mouse and keyboard input, but thankfully the layout and data types used in the structs were available on the Windows Docs site. Because the Windows API is built in C++, the C# equivalent data types had to be researched in order for the API to recognise the data that was being passed in. This caused a number of issues originally, as there was some conflicting information on C# to C++ data types and I ended up using the wrong ones without realising, causing the Windows API to throw an error.

When all of this was working however, creating the rest of Clicky Controller was rather straight forward and simply involved creating various functions that performed specific tasks, such as a Left Click, Scroll Down, Enter Text, Key Down etc. I couldn't find any easy way to write tests for most of the code other than just observing what happened on screen and seeing it was what I expected.

The next step was to create a WPF application that would use this library to allow non-programmers to automate their own tasks with ease (well, hopefully).

I've developed programs with WPF before, so I was able to dive straight in. I decided the use the MVVM pattern for development to make the code cleaner and easier to debug and test. The design of the application went through a few variations, with the earlier versions being prone to errors from user input not being easy to check. These issues have been ironed out (mostly) in the final product thanks to a redesign in the way the program stores the commands.

The user's scripts can be saved and opened like you would expect from any program in 2020 and are saved in JSON format to make it super easy to serialize and deserialize using a slightly tweaked version of Newtonsoft's JSON.NET Library.

Additional Thoughts / Feelings

Overall, I'm mostly pleased with the outcome of the project. Is it the most amazing thing in the world? No, obviously not, considering it was put together on and off over the course of about 3 months - but it works. I'm more happy with the outcome of the Clicky Controller library than the GUI application, simply because it's easier to use and implement. The GUI program suffers a lot from 'programmer design' meaning it makes sense to use from my point of view (obviously), but if I was to give it to some random office worker they would probably be scratching their head using it. Also, the GUI application doesn't have any way of aborting the script whilst it's running, which could cause some serious issues if a script is programmed incorrectly.

Other programs like this allow a user to simply record their input and that program will then simply play it back, reducing the potential for mistakes significantly. Research was done into implementing this, but I decided that it would have taken up too much time and effort and would have put other parts of my work on the sideline. Now that my work has been submitted and I'm done with uni, it might be something that I go back to.