Punishment Driven Development?

In a previous job I developed 'Industrial Monitoring Systems', otherwise known as a PC, a Camera and Image Processing Software to analyse products moving down a production line.

If everything was fine the keg of cider, pizza box or carrot would be sent out to a shop, otherwise it would be diverted to the reject pile! The systems looked something like this:

But why Punishment Driven Development, well because when my software stopped working the entire production line would grind to a halt and some poor 'volunteer' would have to stand there and inspect the parts manually, clicking a red or a green button for each one.

(Fun fact: I was once told that allegedly some factory workers would mess with the PC until it broke, hitting the same key 100 times in a row, endlessly restarting it, etc, etc. They did this because whilst the production line was down, they got to go on an extended tea-break 😊)

Needless to say, if things went wrong the 'Head of Production' would soon be on the phone and because they had a support contract, it was my job to answer it and fix the problem.

During the 5 years that I worked this job, I discovered the following:

1. Most industrial systems are located far away from the office where there is a phone and internet access
2. People don't like being repeatedly asked 'can you just try this?' if each time it involves a 10 minute walk
3. All sorts of things can go wrong with software running on a PC mounted to the side of a production line, inside a noisy, dirty factory
4. If the problem can't be solved over the phone, someone (me), has to drive to the factory and stay till it's fixed

Not surprisingly it is the last lesson that caused me the most problems!! Turns out, most factories are in the middle-of-nowhere. In addition, you have limited access to the machine you're fixing and there's no chance you'll get a nice comfy desk and a chair to sit on.

However, it also forced me to implement reliable and useful diagnostics in my software and make it easy to extract them from the PC. Also I made it possible for the software to update itself from the contents of a USB stick, so no more evenings in a factory for me!!

All in all I learnt:

1. Having useful diagnostics doesn't happen by accident, you have to design it into your product
2. Spending time thinking about what information to log pays off (I always assumed that one day I'd be relying on it)
3. Be nice to the person using your software, so that when you visit their factory, they'll want to help you out
4. Having a motivation to make your software better really helps, hence Punishment Driven Development

Otherwise you end up being sent to Coventry (literally in my case!!)

What about you, do you have a similar story?

Have you ever learnt a valuable lesson about software development, due to an external motivation?