Author
Dave Glover, Microsoft Cloud Developer Advocate
Solution
Creating an image recognition solution with Azure IoT Edge and Azure Cognit...
For further actions, you may consider blocking this person and/or reporting abuse
This is really a useful and interesting article. Thank you very much.
That being said, I'm trying to run the project on a PC running Ubuntu 18.04.
I'm using a local registry for testing.
It seems that docker images were pushed to the local registry correctly:
However, deployment can't be completed. I found this in the edgeAgent logs:
So it seems that there is a syntax error or something like that in the deployment file, but I can't find it. This file looks like this:
Any help would be greatly appreciated.
The problem is related to edgeAgent 1.0.7, as explained here.
After updating to 1.0.8, I can deploy the modules:
Ah, fantastic. Thanks and great you got working. I'll update the deployment template so it starts with 1.0.8. Let me know how you get on. Cheers Dave
Dave, I'm diving into your project to understand how it works in more detail.
First of all, I'm exploring the camera capture process. I'm running the project on a Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz.
I noticed that if the scene shot by the camera is still, no frames are processed. If the scene changes or if I move the camera, on average about 4 frames per second are processed:
Did you achieve similar performances on your development host?
What about the RPi?
I've just found this in
CameraCapture.py
...It answers my question ;)
Hey yes, I did some optimisations, 1) if pixel change was greater that 70000 pixels RGB then send frame to ml model. 2) slowed down frame rate down, logic was the model would be more available to process a frame if something changed...
A raspberry pi 4 take approx 0.8 seconds per inference. Raspberry pi 3b plus approx 1.2 seconds per inference
Thank you, Dave. These numbers are very useful.
How did you get them?
My goal is
That's why I would like to measure the inference time the same way you did.
hey from Bash I just did 'time curl ....' and just used the curl example in the readme from the downloaded custom vision docker container
Hi Dave
I tested the custom docker container on my PC first and it worked fine:
Then, I built your project for
arm32v7
architecture and pulled the resulting image from my embedded device (I had to use a registry on the docker Hub because I couldn't pull from the local registry running on my PC).I tried to run the same test on my embedded device running armbian distribution, but it didn't work although the container seems up and running:
Any advice on how I could analyze this issue?
I just noticed that, after running this test on the embedded device, the container stops and the following warning message appears in its log:
Hey there, I've not tried armbian. Those Tensorflow messages are just warnings. You can try the arm image I built from glovebox/image-classifier-service:1.1.111-arm32v7 ie docker run -it --rm -p 80:80 glovebox/image-classifier-service:1.1.111-arm32v7. And test with 'curl -X POST xxx.xxx.xxx.xxx/image -F imageData=@image.jpg' My Pi is running Docker version 19.03.3, build a872fc2. Cheers Dave
Hi Dave,
I verified the docker version running on my board:
Unfortunately, the outcome is the same even with your image:
I'm afraid I have to debug at a lower level to understand what's going on (that is quite common for embedded devices ...).
I'm not an expert of Azure-based development approach, so I don't know what is the best thing to do in such a situation.
If there are no better ideas, I'm thinking of:
Hey there, I'm pretty sure that the contents of the container are fine, and they are isolated too. Do you have a Raspberry Pi you can test against? There is nothing to stop you running the contents of the docker project that is exported by Custom Vision directly on the device (ie outside of a container). dg
also try curl to localhost curl -X POST localhost/image -F imageData=@red-apple.jpg or by hostname curl -X POST mydevice.local/image -F imageData=@red-apple.jpg. I've seen issues where name resolution doesnt always work as you'd expect...
Hi Dave,
unfortunately, neither localhost nor mydevice.local worked :(
So I tried the other approach that doesn't make use of any container.
For convenience, I first tried to make it work on my development PC. I followed this tutorial, but it didn't work either :(
Apart from several warning messages, the simple Python program I wrote crashes because of this error:
I'll try to figure out what's going on, but I don't think I'll be able to solve it quickly, as I'm not an Tensorflow expert ...
That being said, as far as I know, I can't exclude that the docker version of the classifier doesn't work on my embedded device for the same problem ...
I had a stupid bug in my code.
I fixed it and now everything works fine. I'm gonna run it on my embedded device.
Yah awesome!
Hi Dave,
installing tensorflow and all its dependencies wasn't easy on armbian at all!
I tried several TF/Python combinations, but none of them worked :(
This table lists the combinations I tried and the reason why they fail.
I think that the Illegal instruction problem might explain why your container doesn't work either on this device.
By the way, does your container make use of Python 2.x o 3.x?
In the meantime, I think I'm gonna try a different distro.
Hi Dave
I also tried Armbian Stretch (Debian 9), but nothing changed. I got an Illegal Instruction error as well.
Then I managed to get an RPi 3. I set it up by following this tutorial. On this platform, my simple test program runs correctly:
I used mounted the same raspbian root file system used with RPi from my embedded platform and I got an Illegal Instruction error again.
So it seems there is a structural incompatibility between one of the software layers (maybe TensorFlow) and my platform, which is based on NXP i.MX6Q.
Hey, I had a brief look at armbian and I spotted that it was on a fairly old kernel release - 3.x from memory. I think Stretch on RPi was on 4.3 or something similar. I did wonder if that was where the issue is. There is nothing to stop you from retargeting the Custom Vision model Docker image to different base a image... I think you said you got the CV/Tensorflow running directly on Armbian so that might be a good starting point...
Actually, I used only the armbian root file system.
Regarding the Linux kernel, I used the one that belongs to the latest official BSP of our platform. It is based on release 4.9.11.
Anyway, I agree with you, in the sense that I can't exclude that the root cause is somehow related to the kernel.
Hi Dave,
finally, I managed to solve the problem.
The root cause is related to how the Tensor Flow packages I used were built. Because of the compiler's flags, these packages make use of instructions that are not supported by the i.MX6Q SoC.
So I rebuilt TF with the proper flags ... et voilà:
Woohoo, well done!
Hi, just wanted to let you know that I got your demo working on a Linux desktop x86_64 architecture, running Ubuntu 18.04.3, using a Logitech USB C922 webcam.
There were several challenges in finding all the things that needed tweaking, so I thought I'd share for others who may run into some of these issues.
1) deployment.template.json: Edit the azureSpeechServicesKey to match your Azure Cognitive Service's Speech service key (not BingKey, as stated in the tutorial)
2) module.json: In each module's folder, edit the "repository" line to point to your localhost:5000 instead of glovebox
3) azure_text_speech.py: Edit the TOKEN_URL to point to the one that Azure provides for you when you set up your speech service. Also edit the BASE_URL to point to the text-to-speech base URL for your region. For example, I had to edit mine to point to my region:
4) text2speech.py: For whatever reason, wf.getframerate() would not return the correct frame rate of my audio, causing an error.
Expression 'paInvalidSampleRate' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2048
So I ran 'pacmd list-sinks' to find my actual audio sample rate (48000) and hardcoded it in place of wf.getframerate.
5) predict.py: Lastly, my camera-capture module kept getting connectivity issues, which was actually because it kept returning a response error stating:
Error: Could not preprocess image for prediction. module 'tensorflow' has no attribute 'Session'
This method was deprecated, so to fix this, edit the predict.py's line from 'tf.Session()' to 'tf.compat.v1.Session()'
After all was said and done, I was able to get it working:
Hi ,
Thanks for sharing.I am receiving the same error of your 5 point. In my predict.py file "tf.Session" is missing. Any help will be great.
if possible could u share your this predict.py file.
Sure I could share my predict.py edit. It's simply a one line edit on line 123 from tf.Session() to tf.compat.v1.Session().
Hi ,
This article is helpful.
How can i make the same image classification module without raspberry pi and on my Ubuntu platform.
Any help would be great.
Yes absolutely. I mostly built the project on Ubuntu 18.04 on my laptop and then ported to Raspberry Pi. You will see there are Dockerfiles for x86 in the project. Cheers Dave
Thanks .
Also do u have any document for connecting a physical device like camera to this project.
On the bottom bar of Visual Studio Code there is the option to switch the project from armv32 to amd64. That is how you build the containers for arm64. The project will work with most USB cameras and the camera module is using OpenCV to capture frames from the USB camera.
Thanks a lot sir for your efforts.
Also , do i have to delete the arm 32v7 files and from platform also if i am not using raspberry pi.
Thanks a lot again.
I tried it but it's showing a 500 error in azure portal. How to know whether my camera is connected or not.
Thank you very much, you've saved me countless hours over the last couple days while I've been learning Azure IoT.
Your Dockerfile for OpenCV are also a life saver (i found the ones in the Azure-Samples github.com/Azure-Samples/Custom-vi... to fail to build properly)
p.s. nice surname.
Very helpful tutorial and relevant to my current work.
Hey, that is awesome and great that you find helpful.
Would you mind telling me how you found the posting - did you come to dev.to and what did you search for as I was not sure how to tag this post.
Cheers and thanks Dave
I follow you on twitter and came across this post in twitter updates. Also I came to know about dev.to only through your post.
ah cool - thanks for the follow. Feel free to ask any questions. Cheers Dave
HI,need a little help.I am receiving this in my camera capture log. Camera is not opening.I am using Ubuntu and not raspberry pi. Following is my code:
Camera Capture Azure IoT Edge Module. Press Ctrl-C to exit.
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:1286:(snd_func_refer) Unable to find definition 'cards.ICH.pcm.surround71.0:CARD=0'
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM surround71
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
@heldss
All of those "ALSA lib" lines have nothing to do with your camera, those are a result of your audio device, so you can ignore them as they are just warnings.
Is there any LED light on your camera that light up when the camera-capture module is running? For example, I have a C922 Logitech webcam, and on the device you can see the white LED lights turn on when camera-capture module starts running.
If you see no lights, or suspect camera-capture module isn't connecting to your camera, then most likely it means your camera device index doesn't match what is in your deployment.template.json file. For example, my webcam shows up as "/dev/video0" and "/dev/video1" whenever I plug it in. Find yours by opening up a terminal window, unplug your camera, type "ls /dev/video*" and see what shows up, then plug in your camera and type "ls /dev/video*" again to determine the number of your camera index. Then in your deployment.template.json, edit the "PathOnHost" and "PathInContainer" parameters to match your device.
Restart your camera-capture from Terminal with "iotedge restart camera-capture", and see if it works.
Hi,
I am facing this issue.
Camera Capture Azure IoT Edge Module. Press Ctrl-C to exit.
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm_route.c:867:(find_matching_chmap) Found no matching channel map
Hey there - as Tom pointed out about the ALSA messages are to do with audio and are just warnings. I think there must be an issue with your USB camera and OpenCV. What camera are you using? Cheers Dave