Few months back, I posted a small video clip on my LinkedIn to show how I used ML.NET (w/ Azure), WinML & UWP to build a custom real-time object detection application. I received a few messages of folks on how I actually did that therefore, I decided to submit my talk & spoke to NDC Sydney, 2021 to actually showcase the working of it.

After its implementation, one of former colleagues asked if I could achieve this using Power Platform i.e. AI Builder (because of its object detection capability) and Canvas Apps. I responded in negative because I remember this limitation when I wrote about using AI Builder with Camera Stream in one of my previous posts.

The limitation was that we could not pass the image (or encoded base 64) to the AI Model in Canvas Apps because you had just 4-5 built-in controls which were actually not super helpful. Even if you had a requirement to pass an image (via camera stream) to your object detection model, you had to send it to Power Automate and process its response. So whenever I got a chance, I spoke about this limitation with the awesome team at Microsoft and to my fortune (with many others), we didn't have to wait long enough. 🙂

On November 8, at Microsoft Ignite 2021, Lobe & AI Builder team announced fantastic features to add intelligence to your Power Apps. Now, not only you can easily add your model as a data source to your Canvas Apps but also you can use them with PowerFx. This became the reason for me to re-visit my problem which I couldn't solve because of the discussed limitation.

The purpose of this post is to show how you can achieve near real-time object detection using AI Builder models with Power Apps. In case you're interested in building the AI Builder Object Detection model, you may want to check out this post where I have discussed its process step by step.

Object detection using Camera Stream

The capability of using the AI into your apps with Power Fx allows you to actually access your AI models by any control, be it a Gallery, DataTable and so on. Therefore, we will try to build (or mimic) the same experience as I did for my .NET app. Not because I couldn't think of any other idea but because I want to show the limitless capabilities of Power Platform. Yes I know one works offline while other does not but I think you got my point, didn't you?

So I already have got a template app ready which I will be using to add necessary controls to achieve above. 👆

Camera

Let's start by adding a Camera, set its StreamRate and OnStream properties with a variable that is going to capture the image from the stream. The property we call is captureTrigger

Video Walkthrough

Timer

Add the Timer control to the screen and hide it. Set its Duration to 2000 and OnTimerEnd should have Set(captureTrigger,100);which would set the trigger's value to 100 after every 2 seconds to take a picture from the camera.

Video Walkthrough

Note: Do not forget to add toggle Repeat and Auto start. I missed this in the video.

Gallery

Now, at the same position (X and Y axes), add the Vertical Gallery and align its height and width similar to the Camera. The reason for this is that Gallery will be used to create Bounding Boxes on top of the Camera control. Change the TemplateSize to 0 otherwise it is going stack the items vertically one by one (if there are more than one object identified in an image). Delete all the items inside your Gallery as we're going to add some new controls here.

Lastly, add go to Add Data and add your AI Model, set the Items property of a Gallery to 'Your Model Name'.Predict(YourCameraControl.Stream).results - that's the beauty of using your AI Model along with PowerFx in any control.

Video Walkthrough

Add a Container with Text Label and Rectangle inside your gallery control.

Container

• Change its X value to ThisItem.boundingBox.left * YourCameraControl.Width
• Change its Y value to ThisItem.boundingBox.top * YourCameraControl.Height
• Change its Width value to ThisItem.boundingBox.width * YourCameraControl.Width
• Change its Height value to ThisItem.boundingBox.height * YourCameraControl.Height

Text Label

• Choose your choice of Color with the Border and set its value to 1
• Change its PaddingTop to 1
• Change its Height to something of your own choice, I kept it as 28
• Keep the Width as Parent.Width
• Also, please ensure you set X and Y both to 0
• Set its Text property to ThisItem.tagName

Rectangle

• Choose your choice of BorderColor with the Border and set its value to 8
• Keep the Height as Parent.Height
• Keep the Width as Parent.Width

Note: Please do not set the Color property, just set the color of the Border instead.

You may notice a bit of a lag it is because unlike .NET (or any other framework which runs offline), this is being run online and it takes a second or two to get the response back.

Bounding Boxes (Bonus Control)

There's a bonus custom control for you to use instead of the above discussed Gallery and that is Bounding Boxes.

Motivation

Few days back I had a chat with Jake Cohen and Bill Barnes from AI Builder & Lobe Team at Microsoft on the above topic and we discussed a couple of options to achieve that. During our discussion, Bill went ahead and instead of Gallery approach, he came up with a smart idea of using a HtmlText instead. He then converted HtmlText into a PowerApps Component called Bounding Boxes. As the name suggests, this custom component exposes a property called Boxes which actually takes a Bounding Box of an AI Model as an argument and shows the bounding box on the screen. The best part of this control is its weight which is not heavy as the Gallery and simple for anyone to use.

Video Walkthrough

The good news is, Bill has just shared his repo with me so you can have this (and several other controls) added to your Canvas Apps easily. Here's how you add the Bounding Boxes to your application. 👆

Add the Component to your app and then set the Boxes property of the component to 'Your Model Name'.Predict(StreamCamera.Stream).results so after every 2 seconds, it will start showing the bounding box as they are processed by the model.

As I discussed above that the purpose of this post is to show you the limitless capabilities of AI Builder. It is now up to your business needs where you really want to use this, for example, as a business owner of a store, you'd like to keep a periodic (~10 minutes) check on people wearing masks within a location, counting the number of allowed people in a café and so on. In case you get stuck with whatever I have shown you here - please feel free to comment down and I'll be happy to assist you.

Until next time.