Hunting for spirits with the Internet of Things

There are moments in life when you might find yourself building something you just can't believe they are asking you to do. A couple of years ago, IBM held a small conference on the Queen Mary in Long Beach. I had the opportunity to support a Watson Internet of Things workshop where, and I'll quote exactly, "we want to hunt ghosts with Watson." My face probably turned as white as a ghost, but, I was up for the engineering challenge.

I've meant to write about the experience, alas, other things have taken my attention. Yet, this story keeps resurfacing so I thought I would detail some of the technical aspects as they still apply in solutions today. Happy Halloween!

Legends of America states the "Queen Mary is one of the most haunted places in the world with as many as 150 known spirits lurking upon the ship. Over the past 60 years, the Queen Mary has been the site of at least 49 reported deaths, not to mention having gone through the terrors of war, so it comes as no surprise that spectral spirits of her vivid past continue to walk within her rooms and hallways."

The goal of the workshop was to capture sensor data and visual imagery that could then be analyzed by participants on the main deck. Four custom designed rigs included a cellphone camera, a Raspberry Pi with a SenseHAT, a speaker to play audio, and microphone to capture sound, were mounted in key places that are said to be haunted: the pool area, cargo hold, the Green room, and the Safe Room.

Participants used a dashboard to subscribe to IoT events and visualize sensor data, which for much of the two days remained consistent. The Raspberry Pi SenseHAT captured temperature, humidity, and pressure.

Using an IP Camera app on each mobile device made the camera images available over the private network specifically run throughout the ship for this project (remember ships are metal and Wi-Fi wasn't available).

Using Node-RED, we captured the four camera images every five seconds and updated a simple webpage. The images were stored into the Object Storage service in IBM Cloud and analyzed using Watson Visual Recognition.

Which leads into a fun side-story. Prior to the event, with my limited knowledge of paranormal activity, I searched Google for pictures of ghosts to train the model with. Here's what you might find:

Which is exactly what ghosts do not look like, without going into too much depth (they can be orbs of lights). If I did this project again, I would use image comparison and when the image changed significantly (which for much of the two days remained constant as you would expect), classify the image using Watson's custom image classifiers (could identify objects like people) to determine what object(s) now have. Again, this wasn't a scientific project by any means.

Participants could also record their voices and send the audio down to these rooms to attempt to summon a response from a spirit. This was built using Node-RED's dashboard nodes. There was also an option to have Watson's Text to Speech synthesize a message.

While this workshop ranks high on my list of memorable workshops, it also had a number of lessons that I learned from.

Hardware is difficult

During the setup day, there were a number of technical issues ranging from the camera app going blank because the phone turned off to confirming the devices were still functioning after overheating with the displays being set to full brightness for hours. While these devices were still accessible (though a long walk from the control room), I learned why having as much control remotely is important with IoT. Can you restart devices remotely? If they lose connectivity, what was the last state a device was in? Capturing and saving this data can help debug failures. At one point we even used the camera to watch whether the LEDs on the SenseHAT changed colors to see if the network was intermittent.

Validate your assumptions

As I mentioned earlier, searching for ghost pictures on the internet leads to a lot of false positives. AI is only as good as the data it is trained on. Garbage in, garbage out is absolutely accurate.

Be selective in what data you capture/save

Eager not to miss that one split second a spirit was present, we tinkered with the interval at which images were saved: 1, 5, 10 seconds. If you do the math, that calculates to 240 images a minute or 691,200 images over two days. Over half a million images of the same four shots, yikes! At five second intervals, that is still 138,240 images over two days. A possible alternative was to add an additional system to compare images, and if they look the same, avoid saving or analyzing images when you can recreate the data.

Fatigue messes with your mind

If you've ever worked late into the night debugging problems, you know it can be both mentally and physically exhausting. And if you're already a little spooked, sometimes this can also cause you to overthink things and incorrectly imagine things happen when they actually don't. Fear can mess with your mind! It was surreal to be down in the bowel of the ship and wonder if I really saw something.

Disappointment is real

So, the question everyone asks, did you find any ghosts?!? Given the nature of spirits, you can't just summon a spirit on demand (like you might see on an edited television show). While we didn't see any abnormalities in the data, it doesn't conclusively say we did or did not have a spirit among us. This is a good reminder that while we can all have expectations, we can't completely control the environment just because we want something to be.