Azure Trial Hackathon - EmotionalSound - perfect spotify song for your emotions

Project Links
Demo
Frontend
BackEnd

Overview of our project
This project goal is to suggest real Spotify songs by understanding face emotions.
This project help you to find perfect sound for your actual mood!
You only have to enter in the demo, take a photo of your face and see which new Spotify song EmotionalSound suggest for you.
If you click the button you can real time listen the song on Spotify!.
Every 5 minutes the back-end search for new real time released and add it to song list.

Preview

Mobile
This project have full mobile support (also Safari), and if you click the Spotify Button you can open the Song in your mobile application

Submission Category:
The category for this project is: AI Aces.
I am using Azure Face API for detecting the face emotions and a custom classification neural network made with Tensorflow and compiled to be used by TensorFlowJs.

In addition, to publish the frontend i am using an Azure Static Webapp (great Angular support) and for the backend an Azure App Service with custom Docker images.

For docker images i use Azure Container Registry.

For frontend i have implemented CI/CD with GitHub Actions.
Instead for back-end i prefer to use Bitbucket with Custom Pipelines for build And Deploy the Docker images.

Neural Network Model:
the neural network model is a quite simple classification model that use the normalized data from Spotify API to understand the mood of a song Based on Song Background Frequency and other values.
To better Understand this part i suggest you to visit and try the Google Collab project here!: Collab Url

Tech Stack
The frontend is written in Angular 11.
From the homepage of the application,you can take picture from your webcam and the image is send as a base64 string directly to the Backend, through an API.

YOUR DATA ARE NOT STORED IN ANY WAY

The backend is written in Node js 16 and Express.
For the database to store the song retrived by Spotify Api i use MongoDb 4.4.
an example of Database doc is:

{ 
    "_id" : ObjectId("6224cff6eff1e8fefdf67e45"), 
    "name" : "Don’t Forget My Love", 
    "__v" : NumberInt(0), 
    "image" : "https://i.scdn.co/image/ab67616d0000b2732cfa4b40e2d7b56fb171dd53", 
    "mood" : "2", 
    "spotifyUrl" : "https://open.spotify.com/track/5L7ddKqhj6xZFeb8ZOoI4r"
}

For interact with Mongodb i use mongoose, an npm library for easy wrap all.
The API receives the base64 string image from the Frontend and sends it to the Azure Face API directly using a simple axios call.
The Back-end get the response from azure and understand the max mood (es happiness) and send it to the client with the saved songs generated from The Neural Network and Spotify Api.
Every 5 minutes the back-end get new spotify releases from the API and Neural Network analise the data for understand the mood of the new song, and save it to Mongo.
On the results page, we display a “dynamic” mood label with the founded songs and a button to real-time listen it.

Azure resources
This is the list of our services deployed on Azure:

CI/CD
I have automated the Frontend builds through CI/CD thanks to two GitHub Actions generated from Azure.

For the back-end i need a more complicated one that make a docker image and release it on the Azure Container Registry

image: atlassian/default-image:2
clone:
  depth: full
pipelines:
  branches:

    master:
      - step:
          name: Deploy to Azure
          caches:
            - docker
          services:
            - docker
          script:
            - docker login emotionalsound.azurecr.io -u $AZURE_USER -p $AZURE_PASS
            - docker build -t emotionalsound.azurecr.io/prod:latest .
            - docker push emotionalsound.azurecr.io/prod:latest
definitions:
  services:
    docker:
      memory: 3072  # increase docker memory for unit testing into the pipe

RoadMap
the project is official on Beta stage, my roadMap is to improve before the end of Hackathon the import system to get more song to suggest, and random shuffle it for get more result variety.

If anyone encounters any kind of bug, please report it in the comments

About Me
My name is Marco Bertelli, i am a 21 years old back-end developer that work in Runelab.
This is my first Hackathon, if you have suggestions or comments are Welcome.
If you like the project Comment and Share this article on Linkedin or with your friend!

If anyone has any questions or suggestions, don't hesitate to leave a comment!