Image classification in web applications with ML.NET

Hey, there! It's a pleasure to greet you. In this tutorial article we will learn step by step to generate a personalized Machine Learning model from ML.NET for the classification of images, particularly to classify images of people with and without masks, and then learn to consume that model from a web application on ASP.NET with the MVVM pattern (Model, View, ViewModel).

What is ML.NET and how does it work?

ML.NET is an open-source Machine Learning platform that allows us to transform input data into an ML model, in this case images, and then be able to integrate it into any type of application on the .NET ecosystem, whether web, mobile, desktop, or any other type of application.

In our case study we will focus on the categorization of images.

Image Classification

For our example, the intention is to be able to classify images of people into two groups: people with masks and people without masks.

To achieve this goal, we will carry out three activities in this article:

• 1. Meet the necessary requirements and prepare our solution in Visual Studio.
• 2. Create our Machine Learning model with ML.NET.
• 3. Consume the model generated from ASP.NET.

With this mentioned, let's get started!

Part 1: Prerequisites and Solution Preparation in Visual Studio

In order to generate our Machine Learning model, we will use the Model Builder interpreter of ML.NET. For this, we need to have the following workload in Visual Studio (.NET Cross-Platform Development):

With this workload, we can now create our solution. In this case, the intention is to consume the ML model that we are going to generate from a web application, for this purpose we will use the DotVVM framework that allows us to work with HTML pages and C# classes on ASP.NET, and with the MVVM pattern (Model, View, ViewModel).

Part 2: Building the Machine Learning Model

It's time to create the model with ML.NET. As in any other process in the field of Machine Learning to generate a model, the steps to build the sentiment analysis model are as follows:

• Set the ML scenario.
• Prepare the environment for data training.
• Load the dataset.
• Train the model.
• Evaluate the model.
• Consume the model.

To begin with, in the main project of our solution we will add an element of type Machine Learning:

With this option, the ML.NET Model Builder interpreter will be initialized and we can start customizing the model.

Scenario:
The first step is to select the scenario with which we want to work, in this case it will be Image classification:

Environment:

Now, we must select the computing resources for the training of our model. In this scenario we have three options available, either use the CPU of our computer, a graphics card to take advantage of the GPU (since we will work with images), or perform training in the cloud with Azure Machine Learning. In this example we will use the local CPU:

Data source:

This is one of the most important parts, since with the data we can customize our model. For this purpose, we must specify the directory where the files of the images that we want to classify are located. In this sense, it is important that there is a folder for each category that we are going to classify, in this case we can find the subfolder "With mask", and another called "Without mask". By selecting the directory from the model builder ML.NET we can visualize something like this:

Training:

With everything we have already established so far, we can already perform the training process, and finally generate the desired model. In this section we can only start with the training, and wait for it to end. The time will depend on the number of images in the dataset, the characteristics of our CPU (previously selected method), and the size of the corresponding files.

Evaluation:

Up to this point we already have the model established, anyway, from the Model Builder of ML.NET we can evaluate the model with a particular example.

Consume:

Finally, the interpreter will give us options to consume the created model, either by creating other projects (by console or with a web api), or indicating the calls we must make from our own application.

In this case, we can already consume the model from our web application with DotVVM on ASP.NET with the new generated package: MLModel.mbconfig.

Part 2: Consume the generated model from ASP.NET

To consume the model we must refer to the MLModel class, which was generated by ML.NET in the group MLModel.mbconfig:

In this sense, it is necessary to establish an instance of ModelInput with the reference of the image that we want to classify, and then call the Predict method, and obtain an object of type ModelOutput with the corresponding classification (With mask, Without mask), and its probability.

ViewModel

For this purpose, and to be able to establish a web page with DotVVM, in a ViewModel, in this case the DefaultViewModel.cs, we will define four attributes or global variables:

private IUploadedFileStorage storage;
public UploadedFilesCollection Files { get; set; }

public string Result { get; set; } = null;
public decimal? Score { get; set; } = null;

These variables will allow us to refer to the image file in a storage, and represent the results of the prediction in the attributes Result and Score.

Then we can set a method in this class, in such a way that it represents the action of a button to make the prediction:

public void Predict()
{
    var uploadPath = GetUploadPath();
    var targetPath = Path.Combine(uploadPath, Files.Files[0].FileId + ".bin");
    storage.SaveAsAsync(Files.Files[0].FileId, targetPath);

    var sampleData = new MLModel.ModelInput()
    {
        ImageSource = targetPath,
    };

    //Load model and predict output
    var output = MLModel.Predict(sampleData);
    Result = output.Prediction;

    if (Result.Equals("With mask"))
    {
        Score = decimal.Round((decimal)(output.Score[0]) * 100, 2);
    }
    else
    {
        Score = decimal.Round((decimal)(output.Score[1]) * 100, 2);
    }

    //storage.DeleteFileAsync(Files.Files[0].FileId);
    Files.Clear();
}

In this code we have three important parts:

• Store the image file in a local storage, so that we can have the path of this file for the model.
• Initialize an MLModel.ModelInput with the image path.
• Call the MLModel.Predict with the ModelInput as a parameter, thus obtaining the 'ModelOutput' with the results.
• Prepare the Result to visualize the result of the classification later, and its accuracy in the prediction.

View

Although, we can now construct the view by referencing the attributes and method created in the ViewModel. This view will be encoded in the file Default.dothtml as follows:

<table style="border: hidden">
    <tbody style="border: hidden">
        <tr style="border: hidden">
            <td style="border: hidden" align="center">
                <dot:FileUpload UploadedFiles="{value: Files}"
                                AllowMultipleFiles="false"
                                SuccessMessageText="Predicting..."
                                NumberOfFilesIndicatorText=""
                                UploadButtonText="Search Image"
                                AllowedFileTypes="jpg,png"
                                UploadCompleted="{command: Predict()}" />
            </td>
            <td style="border: hidden" align="center">
                <b>{{value: Result}}</b> {{value: Score}}
            </td>
        </tr>
    </tbody>
</table>

Here the most important thing is the FileUpload control so that the user can upload the file of an image from his computer, to then make the call to the Predict() method with a Button, and set the display of the results by showing the values of the variables Result, and Score.

Examples running

With this View, and your ViewModel, our website is ready. Here we can see two examples at runtime:

The source code for the sample can be found in the following repository: github.com/esdanielgomez/MLNET_ImageClassification.

Thank you!

Thank you very much for reading, I hope this demo can be useful to you. If you have any questions or ideas that you need to discuss, it will be a pleasure to be able to collaborate and together exchange knowledge with each other.

See you on Twitter! Or if you like you can also write to me by email: daniel@esdanielgomez.com. :)