Automated Image Annotation using Auto-Annotate Tool

As simple as saying — “Annotate all the street sign (label) in the autonomous car dataset (directory)” and BAM! DONE.

On average, it takes approximately 30+ seconds to create a single polygon mask and annotate it using some tool. Using Auto-Annotate, this wearisome manual overlapped part of annotating images can be automated by the hit of a single command. The open-source Auto-Annotate tool provides auto annotation of segmentation masks for the objects in the images inside some directory based on the labels.

In this article, I will walk you through the basics of image annotation, functionality, code, and usage of the Auto-Annotate tool. GitHub Repository for Auto-Annotate can be found here.

Image Annotation

Image annotation forms the basis of building datasets for computer vision models. Image annotation is the process of assigning metadata in the form of labels to various entities in an image. It is a data labeling technique that marks the features in an image we want the machine learning model to get trained on.

The most common types of Image Annotations are:

• 2D Bounding Boxes
• Polygonal Segmentation
• Lines and Splines
• Semantic Segmentation
• 3D Cuboids

In this tool, we perform Polygonal Segmentation and provide annotation for those as they cut the noise and let the model focus on the desired set of intended pixels.

Auto-Annotate Tool

The Auto-Annotate tool is built on top of Mask R-CNN to support auto annotations for each instance of an object segment in the image. Auto-Annotate is able to provide automated annotations for the labels defined in the COCO Dataset and also supports custom labels.

The Auto-Annotate tool works in two modes:

• COCO Label Annotation — NO TRAINING REQUIRED. Just use the weights of the Coco dataset, point to the image dataset directory and the annotations are ready.
• Custom Label Annotation — Train the model for the custom label. Use the newly trained weights on the image dataset directory and the annotations are ready in a matter of time.

NOTE: Please refer to knownIssues.md file in the repo for known issues and their resolution. Please feel free to contribute in case of any errors/issues pops up during the installation and usage of the tool.

Sample JSON Annotation:

(This example below illustrates the Custom Label Annotation, wherein the model trained was trained to identify and annotate “Bird Houses”)

NOTE: The JSON response structure remains the same for both the flavors of the Auto-Annotate tool and is highly customizable to suffice your need. Refer to the “JSON Format” section in Code Walkthrough section below for any customization.

JSON Format Description:

• filename: The image file name.
• id: Unique id.
• label: label to annotate. (Coco based label or custom label)
• bbox: [ x, y, w, h ] —x, y coordinate of the top-left point of the bounding box. w, h specifies the width and height of the bounding box. “Format correspond to Coco JSON response format for the bounding box.”
• segmentation: [ x1, y1, x2, y2,…] — For Xi, Yi belonging to the pixel location of the mask segment.
• “Format correspond to Coco JSON response format for segmentation.”

Auto-Annotate Tool Installation:

• Clone the Auto-Annotate GitHub Repository.
• Install the dependencies using:

pip3 install -r requirements.txt

• If annotating labels which are supported by COCO Dataset, download the pre-trained COCO weights (mask_rcnn_coco.h5) from the Mask R-CNN releases page. If annotating Custom Objects, train the model using Mask R-CNN and use the newly trained model’s weight. (Refer to Custom Training as defined below for more details)
• Run the commands given below based on the mode.

COCO Label Annotation:

python3 annotate.py annotateCoco --image_directory=/path/to/the/image/directory/ --label=object_label_to_annotate --weights=/path/to/weights.h5 --displayMaskedImages=False

Note: — label=object_label_to_annotate should be in accordance with the COCO dataset labels. Refer to the COCO Dataset for more details.

Custom Label Annotation:

python3 annotate.py annotateCustom --image_directory=/path/to/the/image/directory/ --label=object_label_to_annotate --weights=/path/to/weights.h5 --displayMaskedImages=False

Note: — label=object_label_to_annotate should be a label for which the training is performed and the weights are provided.

Refer to the Code Walkthrough section below for more details on arg params.

• Find the annotations in the directory — /path/to/the/image/directory/ specified above.

Training on Your Own Dataset

Read the original post by Waleed Abdulla in his blog post about training using the Splash of Color article where he very beautifully explained the process of custom training, starting from annotating images to training using Mask R-CNN.

Then use the customTrain.py which is a modified version of the original balloon.py written by Waleed, which now focuses only on the training part.

# Train a new model starting from pre-trained COCO weights
    python3 customTrain.py train --dataset=/path/to/custom/dataset --weights=coco
# Resume training a model that you had trained earlier
    python3 customTrain.py train --dataset=/path/to/custom/dataset --weights=last

Code Walkthrough

This section briefs about the building blocks of the annotate.py file in the Auto-Annotate GitHub repository.

Argument Parser

This describes the usage of various command-line arguments that can be passed to annotate.py.

• command: Can take either of the two values based on the desired mode. Values — (annotateCoco or annotateCustom)
• image_directory: Points to the dataset directory. Currently supported image formats are “.jpg”, “.jpeg”, “.png” and they can be extended by modifying code in the annotateImagesInDirectory() method.
• weights: If command = annotateCoco, the path to the pre-trained COCO weights (mask_rcnn_coco.h5). If command = annotateCustom, the path to the newly trained model’s weight. (can be found in the /log directory, after training custom model is complete).
• label: If command = annotateCoco, the labels that can be identified by the COCO dataset. For the complete list of labels and sequences of the Coco dataset, refer here. If command = annotateCustom, the label for which the training is performed and the weights are provided above.
• displayMaskedImages: By default, this boolean value is set to False, when toggled to True, it displays the masked image wherein it identifies the object it has annotated.

JSON Format

This method can be altered to suffice to the JSON response annotation format to support any type of response customization curated to your needs.

The bounding box structure “bbox” could be altered to support top-left and bottom-right fashioned bounding box.

Segmentation can be altered in order to support [x1, x2, …, xn, y1, y2, …,yn] format.

Could add an area in the annotation as well.

Thanks for reading this article! If you found this article or the tool insightful, then do show some love. ❤️ 👏

Connect with me on Linkedin here.

Cheers!