DEV Community

loading...

Image Segmentation With 5 Lines of Code.

ayoolaolafenwa profile image Olafenwa Ayoola Updated on ・6 min read

Computer vision is evolving on a daily basis. Popular computer vision techniques such as image classification and object detection have been used extensively to solve a lot of computer vision problems. In image classification, an entire image is classified. Object detection extends image classification by detecting the location of individual objects present in an image.

Image Segmentation:

Some computer vision problems require deeper understanding of the contents in the images. Classification and object detection may not be suitable to solve these problems. The need for an effective technique to solve some critical computer vision problems gives rise to the invention of Image Segmentation.
Every image is made up of a group of pixel values. Image Segmentation is the task of classifying an image at the pixel level. A machine is able to analyse an image more effectively by dividing it into different segments according to the classes assigned to each of the pixel values present in the image.
The distinct technique employed in Image Segmentation makes it applicable in solving critical computer vision problems. These are problems that require detailed information about the objects present in an image, details that cannot be provided by classifying the entire image or providing bounding boxes for the objects present in the image. Some of the major contributions of Image Segmentation include:

-Efficient vision system for driverless cars for an effective road scene’s understanding.
-Medical Image Segmentation provides segmentation of body parts for performing diagnostic tests.
-Satellite images' analysis.

There are two major types of Image Segmentation:
Semantic Segmentation: Objects classified with the same pixel values are segmented with the same colormaps.

Instance Segmentation: It differs from semantic segmentation because different instances of the same object are segmented with different color maps.

semantic
Semantic Segmentation

instance
Instance Segmentation

PixelLib: is a library built for an easy implementation of Image Segmentation in real life problems. PixelLib is a flexible library that can be integrated into software solutions that require the application of Image Segmentation.
Semantic and instance segmentation is possible with five lines of code.

Install PixelLib and its dependencies:

Install the latest version tensorflow(tensorflow 2.0) with:
pip3 install tensorflow

Install opencv-python with:
pip3 install opencv-python

Install scikit-image with:
pip3 install scikit-image

Install Pillow with:
pip3 install pillow

Install Pixellib:
pip3 install pixellib

Implementation of Semantic Segmentation with PixelLib:

The code to implement semantic segmentation with deeplabv3+ model is trained on pascal voc dataset.

import pixellib
from pixellib.semantic import semantic_segmentation

segment_image = semantic_segmentation()
segment_image.load_pascalvoc_model("deeplabv3_xception_tf_dim_ordering_tf_kernels.h5") 
segment_image.segmentAsPascalvoc("path_to_image", output_image_name = "path_to_output_image")
Enter fullscreen mode Exit fullscreen mode

We shall observe each line of code:

import pixellib
from pixellib.semantic import semantic_segmentation 
segment_image = semantic_segmentation()
Enter fullscreen mode Exit fullscreen mode

The class for performing semantic segmentation is imported from PixelLib and we created an instance of the class.

segment_image.load_pascalvoc_model(deeplabv3_xception_tf_dim_ordering_tf_kernels.h5)
Enter fullscreen mode Exit fullscreen mode

In the code above we loaded the xception model trained on pascal voc for segmenting objects. The model can be downloaded from here.

segment_image.segmentAsPascalvoc(path_to_image, output_image_name = path_to_output_image)
Enter fullscreen mode Exit fullscreen mode

We loaded the function to perform segmentation on an image. The function takes two parameters…
path_to_image:- this is the path to the image to be segmented.
output_image_name:- this is the path to save the segmented image. It will be saved in your current working directory.

sample1.jpg

Smaple1

import pixellib
from pixellib.semantic import semantic_segmentation

segment_image = semantic_segmentation()
segment_image.load_pascalvoc_model("deeplabv3_xception_tf_dim_ordering_tf_kernels.h5") 
segment_image.segmentAsPascalvoc("sample1.jpg", output_image_name = "image_new.jpg")
Enter fullscreen mode Exit fullscreen mode

Alt Text

The saved image after segmentation, the objects in the image are segmented. You can apply segmentation overlay on the image if you want to.

segment_image.segmentAsPascalvoc("sample1.jpg", output_image_name = "image_new.jpg", overlay = True)
Enter fullscreen mode Exit fullscreen mode

overlay

We added the extra parameter overlay and set it to true and we obtained an image with a segmentation overlay on the objects.

You can check the inference time required for performing segmentation by modifying the code below.

import pixellib
from pixellib.semantic import semantic_segmentation
import time

segment_image = semantic_segmentation()
segment_image.load_pascalvoc_model("pascal.h5")

start = time.time()
segment_image.segmentAsPascalvoc("sample1.jpg", output_image_name= "image_new.jpg")

end = time.time()
print(f"Inference Time: {end-start:.2f}seconds")
Enter fullscreen mode Exit fullscreen mode
Inference Time: 7.38seconds
Enter fullscreen mode Exit fullscreen mode

It took 7.38 seconds to run semantic segmentation on the image.
This xception model is trained on pascal voc dataset, a dataset with 20 object categories.
Objects and their corresponding color maps

Alt Text

Specialised uses of PixelLib may require you to return the array of the segmentation’s output:

Obtain the array of the segmentation’s output by using this code,

output, segmap = segment_image.segmentAsPascalvoc()
Enter fullscreen mode Exit fullscreen mode

You can test the code for obtaining arrays and print out the shape of the output by modifying the semantic segmentation code below.

import pixellib
from pixellib.semantic import semantic_segmentation
import cv2

segment_image = semantic_segmentation()
segment_image.load_pascalvoc_model("pascal.h5")
output, segmap = segment_image.segmentAsPascalvoc("sample1.jpg")
cv2.imwrite("img.jpg", output)
print(output.shape)
Enter fullscreen mode Exit fullscreen mode

INSTANCE SEGMENTATION WITH PIXELLIB:

Instance segmentation with PixelLib is based on Mask R-CNN framework.

Code to implement instance segmentation:

import pixellib
from pixellib.instance import instance_segmentation

segment_image = instance_segmentation()
segment_image.load_model("mask_rcnn_coco.h5") 
segment_image.segmentImage("path_to_image", output_image_name = "output_image_path")
Enter fullscreen mode Exit fullscreen mode

Observe each line of code

import pixellib
from pixellib.instance import instance_segmentation
segment_image = instance_segmentation()
Enter fullscreen mode Exit fullscreen mode

The class for performing instance segmentation is imported and we created an instance of the class.

segment_image.load_model("mask_rcnn_coco.h5")
Enter fullscreen mode Exit fullscreen mode

This is the code to load the mask r-cnn model to perform instance segmentation. Download the mask r-cnn model from here.

segment_image.segmentImage("path_to_image", output_image_name = "output_image_path")
Enter fullscreen mode Exit fullscreen mode

This is the code to perform instance segmentation on an image and it takes two parameters:
path_to_image: The path to the image to be predicted by the model.
output_image_name: The path to save the segmentation result. It will be saved in your current working directory.

sample2.jpg
sample2

import pixellib
from pixellib.instance import instance_segmentation

segment_image = instance_segmentation()
segment_image.load_model("mask_rcnn_coco.h5") 
segment_image.segmentImage("sample2.jpg", output_image_name = "image_new.jpg")
Enter fullscreen mode Exit fullscreen mode

mask

This is the saved image in your current working directory.

You can implement segmentation with bounding boxes. This can be achieved by modifying the code.

segment_image.segmentImage("path_to_image", output_image_name = "output_image_path", show_bboxes = True)
Enter fullscreen mode Exit fullscreen mode

Alt Text

We added an extra parameter show_bboxes and set it to true, the segmentation masks are produced with bounding boxes.

You can check the inference time required for performing segmentation by modifying the code below.

import pixellib
from pixellib.instance import instance_segmentation
import time

segment_image = instance_segmentation()
segment_image.load_model("mask_rcnn_coco.h5")

start = time.time()
segment_image.segmentImage("former.jpg", output_image_name= "image_new.jpg")

end = time.time()
print(f"Inference Time: {end-start:.2f}seconds")
Enter fullscreen mode Exit fullscreen mode
Inference Time: 12.87seconds
Enter fullscreen mode Exit fullscreen mode

It took 12.87 seconds to run instance segmentation on the image.

The Mask R_CNN model is trained on Microsoft Coco dataset, a dataset with 80 common object categories. The model can perform instance segmentation on these object categories.

A list of the object categories present in Coco dataset:
[‘BG’, ‘person’, ‘bicycle’, ‘car’, ‘motorcycle’, ‘airplane’, ‘bus’, ‘train’, ‘truck’, ‘boat’, ‘traffic light’, ‘fire hydrant’, ‘stop sign’, ‘parking meter’, ‘bench’, ‘bird’, ‘cat’, ‘dog’, ‘horse’, ‘sheep’, ‘cow’, ‘elephant’, ‘bear’, ‘zebra’, ‘giraffe’, ‘backpack’, ‘umbrella’, ‘handbag’, ‘tie’, ‘suitcase’, ‘frisbee’, ‘skis’, ‘snowboard’, ‘sports ball’, ‘kite’, ‘baseball bat’, ‘baseball glove’, ‘skateboard’, ‘surfboard’, ‘tennis racket’, ‘bottle’, ‘wine glass’, ‘cup’, ‘fork’, ‘knife’, ‘spoon’, ‘bowl’, ‘banana’, ‘apple’, ‘sandwich’, ‘orange’, ‘broccoli’, ‘carrot’, ‘hot dog’, ‘pizza’, ‘donut’, ‘cake’, ‘chair’, ‘couch’, ‘potted plant’, ‘bed’, ‘dining table’, ‘toilet’, ‘tv’, ‘laptop’, ‘mouse’, ‘remote’, ‘keyboard’, ‘cell phone’, ‘microwave’, ‘oven’, ‘toaster’, ‘sink’, ‘refrigerator’, ‘book’, ‘clock’, ‘vase’, ‘scissors’, ‘teddy bear’, ‘hair drier’, ‘toothbrush’]

Specialised uses of PixelLib for Instance Segmentation.

Obtain the following arrays:
-Detected Objects’ arrays
-Objects’ corresponding class_ids’ arrays
-Segmentation masks’ arrays
-Output’s array

By using this code

segmask, output = segment_image.segmentImage()
Enter fullscreen mode Exit fullscreen mode

You can test the code for obtaining arrays and print out the shape of the output by modifying the instance segmentation code below.

import pixellib
from pixellib.instance import instance_segmentation
import cv2

instance_seg = instance_segmentation()
instance_seg.load_model("mask_rcnn_coco.h5")
segmask, output = instance_seg.segmentImage("sample2.jpg")
cv2.imwrite("img.jpg", output)
print(output.shape)
Enter fullscreen mode Exit fullscreen mode

Obtain arrays of segmentation with bounding boxes by including the parameter show_bboxes.

segmask, output = segment_image.segmentImage(show_bboxes = True)
Enter fullscreen mode Exit fullscreen mode
import pixellib
from pixellib.instance import instance_segmentation
import cv2

instance_seg = instance_segmentation()
instance_seg.load_model("mask_rcnn_coco.h5")
segmask, output = instance_seg.segmentImage("sample2.jpg", show_bboxes= True)
cv2.imwrite("img.jpg", output)
print(output.shape)
Enter fullscreen mode Exit fullscreen mode

Install PixelLib and test it with as many images you desire.
Visit the official github repository of PixelLib.

Visit the official documentation of PixelLib

Reach to me via:
Email: olafenwaayoola@gmail.com
Twitter: @AyoolaOlafenwa
Facebook: Ayoola Olafenwa
Linkedin: Ayoola Olafenwa

Discussion

pic
Editor guide