Intro

I am a semi-beginner in all things tech and coming up on actually finally completing my coding bootcamp course with Flatiron on full stack web development! Woo 🎉 The course covers standard software engineering, and not AI/machine learning/data science, but I've always been interested in it, and wanted to learn general programming before one day moving into something more directly AI related. But I could definitely be like this meme about wannabe AI enthusiasts - so this is a post for - cannot emphasize enough - beginners.

COVID-19

Yes, this is a post about COVID-19. AI and coronavirus. Who needs another one of those? Not me! Send tweet.

I promise I'll try to be original here. I've always been very interested in healthcare tech and AI - maybe because I love to watch Grey's Anatomy and I've always wanted the glory of medicine without having to go to medical school! But for real - and at the risk of sounding like Silicon Valley:

I really do want to use technology to make... the world... at least slightly less terrible than it is now. Whether or not that's possible is a question for another day!

Most coding bootcamps have a 'final project' that combines everything we've learned in the past 15 weeks, and mine is due at the end of this week, so I've been spending a lot of time looking at trending projects/hackathon submissions to gather inspiration and try to think of something that hadn't been totally done a million times before (this might also be impossible). As I said, I've been generally interested in something healthcare-related for this final project since before I even started, and now with COVID, I couldn't help it.

For inspiration, this project in particular caught my attention, because of the use of audio recognition and coughs. There doesn't seem to be a lot of specific research on the use of cough recognition and coronavirus, but the research on uses in other medical conditions/cases seems very promising.

There are many other very cool uses for AI/COVID - using imaging, fever detection glasses, cleaning robots, and - potentially more controversial - 'smart' doors that monitor people's activity and movement in quarantine. If all of these seem a little too sci-fi and impractical, there is also the more familiar example of AI chatbot technology, such as IBM Watson, in helping to automate COVID diagnosis, questioning, and spread of information. While this is a beginner's post, it's no secret that AI in general has a ton of varying uses, some controversial and some, frankly, mundane, so I won't go into that too much more here.

TensorFlow

All of this got me very interested in a basic implementation of TensorFlow, Google's open-source machine-learning platform. My final bootcamp project is a COVID-19 self-report tracker and symptom checker, with a Rails backend and React.js frontend, so I looked into using TensorFlow.js, specifically for audio recognition. TensorFlow and AI in general seem more common in Python, so I found this tutorial on implementing a simple audio recognizer in JavaScript especially exciting!!

As I said, I was inspired by examples of using AI to diagnose COVID, but I am nowhere near being able to do that in any meaningful way, so I really just wanted to implement the most basic possible use of AI in my app to show that I could. I was able to "train" the TensorFlow model by coughing with my microphone on for a few seconds, and there you go, a sliding bar would move to the right if I was trying to record a cough, and to the left if I was trying to record anything other than a cough (i.e. regular talking).

I really did nothing impressive here - all the cool stuff is handled by TensorFlow and this excellent tutorial! The simple audio recognizer uses Transfer learning, which "is a technique that shortcuts much of this by taking a piece of a model that has already been trained on a related task and reusing it in a new model" (straight from the old copy and paste here) but it's nice to know that so many of these seemingly complex AI technologies are actually just building on a ton of past work. So something relatively simple, like an image or audio recognizer, doesn't require as much re-training. This is why all I had to do to make my simple audio function the way I wanted was to "teach" it the sound of a cough. It already "knew" basic human speech pretty well, so then the distinction between that library and a new sound wasn't far off.

const INPUT_SHAPE = [NUM_FRAMES, 232, 1];
let model;

async function train() {
 toggleButtons(false);
 const ys = tf.oneHot(examples.map(e => e.label), 3);
 const xsShape = [examples.length, ...INPUT_SHAPE];
 const xs = tf.tensor(flatten(examples.map(e => e.vals)), xsShape);

 await model.fit(xs, ys, {
   batchSize: 16,
   epochs: 10,
   callbacks: {
     onEpochEnd: (epoch, logs) => {
       document.querySelector('#console').textContent =
           `Accuracy: ${(logs.acc * 100).toFixed(1)}% Epoch: ${epoch + 1}`;
     }
   }
 });
 tf.dispose([xs, ys]);
 toggleButtons(true);
}

But I thought, this could even be used just for a simple validation on an online symptom submission so the user knows if they are coughing enough into the microphone, and/or prevents the user from submitting useless audio. While small, these are all things that can make the process of doctors remotely diagnosing and treating patients even just a little bit easier, which is very exciting to me!

Conclusion

This isn't a tutorial, and I don't have any unique insights on using TensorFlow (yet!) but there seem to be a ton of amazing tutorials and it really was easier than I expected to use TensorFlow in this simple way. If you, like me, are interested in AI/ML but haven't had time to do that Coursera course in full yet and are feeling overwhelmed, my advice is to just go for it and see what you can do!

References

*https://devpost.com/software/faco-fight-against-corona-jfcza9
*https://devpost.com/software/corona-mask-detector
*https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7187882/
*https://venturebeat.com/2020/03/03/how-people-are-using-ai-to-detect-and-fight-the-coronavirus/
*https://www.scmp.com/tech/article/3077964/ring-smart-doorbell-monitors-people-covid-19-isolation
*https://www.bbc.com/news/technology-52340651
*https://techcrunch.com/2020/04/16/chinese-startup-rokid-pitches-covid-19-detection-glasses-in-u-s/