Mediapipe | Implementing custom feature on Android

I have included following topics in this article,

• How to test run Mediapipe mobile app using their github code
• Details about codebase structure
• In brief about some key components of code

Compile and try out already implemented example projects on android

1. Build docker image using command
   
   $ docker build . -t mediapipe
   

2. Run docker container over image
   
   $ docker run -i -t IMAGE_ID /bin/bash
   

3. Inside the container set up Android using command
   
   $ bash ./setup_android_sdk_and_ndk.sh ~/Android/Sdk ~/Android/Ndk r21
   

4. In the next step, build android binary using bazel.
   
   $ bazel build -c opt mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu:facedetectioncpu
   
   this command is for building facedetectioncpu project. Replace project_name (facedetectioncpu here) with the name of the project you want to build.
   Name of the folders under path mediapipe/mediapipe/examples/android/src/java/com/google/mediapipe/apps/ is name of the in built projects.
   This command would take long for the first build. For any subsequent builds, it would use the cached data and shouldn't take long.

5. apk will be stored at /mediapipe/bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps
/project_name. Copy the apk file using
   
   sudo docker cp CONTAINER_ID:mediapipe/mediapipe/bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps
/project_name /home/username/downloads/.
   

6. copy this apk file to your android device and install.

7. or directly install from terminal using commands
   
   $ adb install project.apk
   

8. keep the phone connected and logs could be viewed using
   
   $ adb logcat
   

9. You can greb specific logs for example errors using
   
   $ adb logcat | greb 'error'
   

Structure of Mediapipe project repo

1. Image at the top has general structure of the mediapipe project.

2. we call bazel build on BUILD file present at mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu

3. This build file refers to graph present at /mediapipe/graphs/facemeshgpu/facemeshgpu.pbtxt

4. This graph at /mediapipe/graphs/facemeshgpu/facemeshgpu.pbtxt refers to one or more calculators present at 'mediapipe/calculators/folder_name/calculator.cc`

5. both folders mediapipe/calculators/folder_name/. and mediapipe/graphs/project_name/. has BUILD files that contains information about dependencies.

Graphs

1. graphs could be visualised here https://viz.mediapipe.dev/

2. copy paste content from any of your .pbtxt file e.g https://github.com/google/mediapipe/blob/master/mediapipe/graphs/face_detection/face_detection_full_range_mobile_gpu.pbtxt to editor on visualiser. It would show the graph on the left side.

3. Each box is calculator and has input and output.

4. This input output relations are specified in .pbtxt graph files.

Calculators

1. each box in graph visualiser is calculator. (or subgraph made of calculators)

2. calculators has process function defined that performes calculations. e.g render text on frame or render assets like spectacles at specific position on frame. e.g https://github.com/google/mediapipe/blob/6cdc6443b6a7ed662744e2a2ce2d58d9c83e6d6f/mediapipe/calculators/util/landmark_visibility_calculator.cc#L66

3. other than process, calculator have getcontract and open function. Open function is called once at the start and process function is called for each input again and again. see below for example.

About open vs process function in calculators

consider we need to display frame count on the camera preview. We start with 0 (for the first frame as the camera starts) and increment count for every subsequent frame. For this, we would define variable count in open function of calculator and assign value 0 to it.
We will add increment logic count = count + 1 in process function and display it on frame.
count will be incremented for every frame because process function will be called for each input, i.e frame. Open is called once at the beginning and value 0 will be assigned to count.

Modify current project

1. add new calculator in project_folder inside calculator folder
2. add entry in BUILD file of the project calculator folder
3. create project folder inside graph folder and create .pbtxt graph for new project.
4. add entry in BUILD file of project graph folder
5. modify BUILD mediapipe/examples/android/src/java/com/google/mediapipe/apps/BUILD
6. compile using command bazel build PATH TO BUILD