I have included following topics in this article,
- How to test run Mediapipe mobile app using their github code
- Details about codebase structure
- In brief about some key components of code
Build docker image using command
$ docker build . -t mediapipe
Run docker container over image
$ docker run -i -t IMAGE_ID /bin/bash
Inside the container set up Android using command
$ bash ./setup_android_sdk_and_ndk.sh ~/Android/Sdk ~/Android/Ndk r21
In the next step, build android binary using bazel.
$ bazel build -c opt mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu:facedetectioncpu
this command is for building facedetectioncpu project. Replace project_name (facedetectioncpu here) with the name of the project you want to build.
Name of the folders under path
mediapipe/mediapipe/examples/android/src/java/com/google/mediapipe/apps/is name of the in built projects.
This command would take long for the first build. For any subsequent builds, it would use the cached data and shouldn't take long.
apk will be stored at
/mediapipe/bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps. Copy the apk file using
sudo docker cp CONTAINER_ID:mediapipe/mediapipe/bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps
copy this apk file to your android device and install.
or directly install from terminal using commands
$ adb install project.apk
keep the phone connected and logs could be viewed using
$ adb logcat
You can greb specific logs for example errors using
$ adb logcat | greb 'error'
Image at the top has general structure of the mediapipe project.
we call bazel build on BUILD file present at
This build file refers to graph present at
This graph at
/mediapipe/graphs/facemeshgpu/facemeshgpu.pbtxtrefers to one or more calculators present at 'mediapipe/calculators/folder_name/calculator.cc`
mediapipe/graphs/project_name/.has BUILD files that contains information about dependencies.
graphs could be visualised here https://viz.mediapipe.dev/
copy paste content from any of your .pbtxt file e.g
https://github.com/google/mediapipe/blob/master/mediapipe/graphs/face_detection/face_detection_full_range_mobile_gpu.pbtxtto editor on visualiser. It would show the graph on the left side.
Each box is calculator and has input and output.
This input output relations are specified in .pbtxt graph files.
each box in graph visualiser is calculator. (or subgraph made of calculators)
calculators has process function defined that performes calculations. e.g render text on frame or render assets like spectacles at specific position on frame. e.g https://github.com/google/mediapipe/blob/6cdc6443b6a7ed662744e2a2ce2d58d9c83e6d6f/mediapipe/calculators/util/landmark_visibility_calculator.cc#L66
other than process, calculator have getcontract and open function. Open function is called once at the start and process function is called for each input again and again. see below for example.
consider we need to display frame count on the camera preview. We start with 0 (for the first frame as the camera starts) and increment count for every subsequent frame. For this, we would define variable count in open function of calculator and assign value 0 to it.
We will add increment logic
count = count + 1 in process function and display it on frame.
count will be incremented for every frame because process function will be called for each input, i.e frame. Open is called once at the beginning and value 0 will be assigned to count.
- add new calculator in project_folder inside calculator folder
- add entry in BUILD file of the project calculator folder
- create project folder inside graph folder and create .pbtxt graph for new project.
- add entry in BUILD file of project graph folder
- modify BUILD
- compile using command
bazel build PATH TO BUILD