Common programming mistakes: just one more thing...

This is a story about how a simple side project can escalate easily, how to make irrational decisions, and how to have fun doing it!

The idea

I've had a Raspberry Pi with a 5" screen for years now, and it's just been gathering dust on my shelf. But last weekend, I had an awesome idea on how to make use of it. The hypothesis was:

At the company I work with, we are running a platform with a dozen micro-services. Those programs are in charge of processing everything. It would be useful for me to always see the state of each of those programs so that when one crashes, I can react immediately.

That shouldn't be too hard, right? The platform is using Kubernetes so it's pretty easy to access the data (using kubectl), and I'll have it shown nicely on the screen. It will be nice!

The prototype

My mission was simple enough - to have data for the most important micro-services displayed on the screen. This data consists of: how long a service is running, how many times it was restarted, and is it running at the moment.

The setup process was simple:

1. Download Raspbian OS (bare-bone version, because who needs a Desktop)
2. Install it on the MicroSD card
3. Set everything up on the Raspberry (WiFi, ssh, screen drivers, kubectl)
4. Set up a simple one-liner bash command that will refresh the data every second
5. Make it run on startup

Basically, kubectl is a command with which I can pull the latest data, WiFi lets the Raspberry talk to the outside world, ssh lets me talk to the Raspberry, and the screen drivers let me actually see stuff on the screen.

And the command? Pretty simple actually:

watch -n 1 kubectl get pods | grep "sandbox\|memcached\|mongo\|postgres"

This is what I got as a result:

Time spent: 1.5 hours

But wait, there is a problem... I felt that this was way to easy. I just didn't feel the gratification of a hard day's work. I know, let me add just one more thing!

Down the rabbit hole

Let me just code in some color. It will mean the world to me in the future. Also, the future me will definitely want the coloring logic to be easily programmable, so it could support all the complex logic it's surely gonna need.

For this, I decided to move away from Bash towards Python. This is where I rock and it will be done in no time. I decided on just one small restriction - the program could use only standard Python libraries since otherwise it would make things too easy, and I wanted to learn something new.

And so it was. With 2 classes, 1 recursive and 5 regular functions, totaling 127 lines of code, I had my custom color-coded monitoring solution. While looking at it on my 15" laptop with an i7 processor, 16GB of RAM and an Nvidia graphics card, I thought to myself: This was something I could be proud of.

...until I ran it on the Raspberry. There, I could see glitches every time the screen would refresh. It couldn't clear the whole screen and reprint everything fast enough. It was so slow that the refresh would happen every 3 seconds, instead of the 1 second I programmed it to be. Also, some colors made the data harder to read. Pretty disappointing.

Time spent: 3 hours

Ok, ok, no problem. I just need to optimize the code a little. It doesn't have to refresh every second, but I need to get rid of the glitch.

I had a couple of optimizations in mind - every time I get new data I'll compare it to the old data. If it's the same, I will just ignore it because that means nothing changed. Also, I will store everything that needs to be printed out so that when I print, I print everything at once.

After I coded the changes, I was happy! I've finally had my day's work that made everything so satisfying. Too bad that the glitches wouldn't go away.

Do or do not, there is no try

This wasn't going to defeat me. I knew the solution was to find a way to reprint just the changed data. After a lot of (StackOverflow) research, I found that my only solution was this curses library which allowed me to edit every row and column displayed separately.

This meant that I had to rewrite the whole printing process. I ended up mapping the position of every piece of information on the screen, looking for changes to that information and then replacing it appropriately.

This brought a lot of pain (and pleasure?). If I wanted to print a new piece of information over the old one, I had to compare the strings and in case the new one was shorter, I had to add spaces to overwrite the difference. When a service terminated, it needed to disappear from the screen which meant that I had to clear that line, and then shift all other lines up to cover the blank space. I also had to watch out when the color of a line changes based on data changes, because the whole line needed to be reprinted.

A lot of these small things would add up, ending with 236 lines of code and me spending almost the whole day programming. And even now I'm still not done, due to the small screen size, there are some modifications that I'll have to do for everything to render properly.

To think that I had almost the same functionality with that 1 line of Bash code. Was I insane?

Was it worth it?

Yes. And no.

I had a lot of fun programming and solving those problems, mixed with some frustrations that usually comes with this line of business. And this is something that I think often gets lost when you start programming for a living because we get caught up in specifications and deadlines. I learned a lot as well - how to manipulate the screen on a character level, that it often pays off to think about hardware limitations, and how to make a Raspberry look cool.

On the other hand, the end result is not much better than what I started with in Bash, and even now I still have some modifications to do. So maybe I could have used that time to do something else and break my standard routine.

What was interesting for me were the (not so) subtle irrational decisions that I made in this project. So learn from my experience. Not necessarily how to be less irrational, but how to recognize that behavior and take it into account in your decision-making process. Also, experiment and have fun - that is the crucial part :)

Time spent: 8 hours

What do you think?

I'd really like te know what do you think. Do you know of a better way to solve the problems I encountered? Did you find this useful and would you like me to publish the code on GitHub?