Labeling Multiple Barcodes with Augmented Reality and Dynamsoft Barcode Reader

The term "augmented reality (AR)" generally refers to the integration of digital information with the user's environment in real time. AR can make user interfaces more immersive and intuitive, overlaying real-time information onto the real world to make tasks more efficient and less error-prone. For example, in warehouse management, AR can be used to track the inventory of items on shelves. In hospitals, AR can help identify tubes and bottles, and in retail, it can display product information. In this article, we will demonstrate how to use ARCore and Dynamsoft Barcode Reader to label multiple barcodes and avoid misreading.

Prerequisites

• An ARCore compatible device running Google Play Services for AR (ARCore) 1.24 or later
• Android Studio 4.1 or later

2D Barcode Overlay on Camera Preview

Usually, we display barcode results on the camera preview as follows:

The overlay is rendered on a 2D canvas that sits above the camera preview, displaying both the barcode information and its bounding box in real time.

Drawing an overlay on a camera view can be considered a basic form of augmented reality. A more advanced AR system often involves understanding the geometry of the environment and placing digital objects in a way that makes them appear to exist within the real world, responding appropriately to changes in the camera's view or the environment itself.

In the following sections, we will combine ARCore and the Dynamsoft Barcode Reader SDK to enhance the barcode scanning experience.

About ARCore and Dynamsoft Barcode Reader

• ARCore is Google's platform for building augmented reality (AR) experiences. By utilizing three key technologies — motion tracking, environmental understanding, and light estimation — ARCore enables your phone to sense its environment, understand the world, and interact with information in a more comprehensive and contextually aware manner.
• Dynamsoft Barcode Reader SDK is a commercial barcode scanning library for Windows, Linux, macOS, Android, iOS, and web. It supports various barcode types, including QR Code, DataMatrix, PDF417, Code 128, Code 39, EAN 13, UPC-A, etc. It also provides a free trial license for 30 days.

Combining Google's ARCore with Dynamsoft Android Barcode SDK

Let's get started with the sample project of ARCore: https://github.com/googlesamples/arcore-ml-sample.

The Android sample written in Kotlin demonstrates how to utilize ARCore and computer vision algorithms for object detection and tracking. For us, the primary goal is the detection and tracking of barcodes. Integrating the Dynamsoft Barcode Reader into the sample project involves a series of steps.

Add Dynamsoft Barcode Reader SDK to the Project

1. Add the Dynamsoft repository source https://download2.dynamsoft.com/maven/aar to the project's build.gradle file:
   

   allprojects {
       repositories {
           google()
           mavenLocal()
           mavenCentral()
           maven {
               url "https://download2.dynamsoft.com/maven/aar"
           }
       }
   }
   

2. Configure the dependency in the app's build.gradle file:
   

   implementation 'com.dynamsoft:dynamsoftbarcodereader:9.6.20'
   

Detect Barcode, QR Code and DataMatrix in Kotlin

1. In MainActivity.kt, set the license key. You can get a free trial license from here.
   

   import com.dynamsoft.dbr.BarcodeReader
   
   BarcodeReader.initLicense("LICENSE-KEY") { isSuccessful, e ->
     runOnUiThread {
       if (!isSuccessful) {
         e.printStackTrace()
         Log.e(TAG, "Failed to verify the license: $e")
       }
     }
   }
   

2. Create a barcode reader object in AppRenderer.kt:
   

   import com.dynamsoft.dbr.BarcodeReader
   import com.dynamsoft.dbr.BarcodeReaderException
   import com.dynamsoft.dbr.EnumImagePixelFormat
   import com.dynamsoft.dbr.Point
   
   var reader: BarcodeReader? = null
   
   fun bindView(view: MainActivityView) {
       try {
         reader = BarcodeReader()
         val settings = reader!!.runtimeSettings
         settings.expectedBarcodesCount = 999
         reader!!.updateRuntimeSettings(settings)
       } catch (e: BarcodeReaderException) {
         e.printStackTrace()
       }
   }
   

3. Find the code block that starts with launch(Dispatchers.IO). You can utilize the acquired camera image to detect barcodes:
   

   if (reader != null) {
       var bytes = ByteArray(cameraImage.planes[0].buffer.remaining())
       cameraImage.planes[0].buffer.get(bytes)
   
       var results = reader!!.decodeBuffer(bytes, cameraImage.width, cameraImage.height, cameraImage.planes[0].rowStride, EnumImagePixelFormat.IPF_NV21)
       objectResults = emptyList()
       if (results != null && results.isNotEmpty()) {
       val tmp: MutableList<DetectedObjectResult> = mutableListOf()
       for (result in results) {
           var points = result.localizationResult.resultPoints
           var confidence = 100
   
           val (x1, y1) = points[0].x to points[0].y
           val (x2, y2) = points[1].x to points[1].y
           val (x3, y3) = points[2].x to points[2].y
           val (x4, y4) = points[3].x to points[3].y
           val centerX = (x1 + x2 + x3 + x4) / 4
           val centerY = (y1 + y2 + y3 + y4) / 4
           val content = result.barcodeText
           val label = "✓"
   
           val detectedObjectResult = DetectedObjectResult(confidence.toFloat(), label, centerX.toInt() to centerY.toInt(), content)
           tmp.add(detectedObjectResult)
       }
       objectResults = tmp
       }
   }
   

A barcode detection result contains four points, which are the four corners of the barcode. We can calculate the center point of the barcode and use it as the anchor point for the label.

val (x1, y1) = points[0].x to points[0].y
val (x2, y2) = points[1].x to points[1].y
val (x3, y3) = points[2].x to points[2].y
val (x4, y4) = points[3].x to points[3].y
val centerX = (x1 + x2 + x3 + x4) / 4
val centerY = (y1 + y2 + y3 + y4) / 4

We convert the barcode results to a DetectedObjectResult list, which will be rendered on the screen later.

If we set the label to result.barcodeText, the content of the barcode will be displayed on the screen.

The effect works well for isolated barcodes. However, if multiple barcodes appear in the camera view, the display can become overcrowded. To address this issue, we can set the label to ✓ and present the barcode content in a list view.

Enable Camera Auto Focus

By default, the ARCore camera is set to fixed focus, which can make the barcode appear blurry. To enhance barcode detection accuracy, enabling auto-focus is necessary.

A switch button is added to the UI to toggle the focus mode.

1. Create a switch button in the activity_main.xml layout file :
   

   <androidx.appcompat.widget.SwitchCompat
         android:id="@+id/switch_focus_mode"
         android:layout_width="wrap_content"
         android:layout_height="wrap_content"
         android:layout_alignParentEnd="true"
         android:checked="false"
         android:text="@string/switch_focus_mode"
         android:textColor="#ffffff" />
   

2. In MainActivityView.kt, obtain the reference to the switch button:
   

   var focusModeSwitch = root.findViewById<SwitchCompat>(R.id.switch_focus_mode)
   

3. Register the OnCheckedChangeListener of the switch button in AppRenderer.kt:
   

   view.focusModeSwitch.setOnCheckedChangeListener { _, isChecked ->
     val session = activity.arCoreSessionHelper.sessionCache ?: return@setOnCheckedChangeListener
     val config = session.config
     config.focusMode = if (isChecked) Config.FocusMode.AUTO else Config.FocusMode.FIXED
     session.configure(config)
   }
   

Source Code

https://github.com/yushulx/android-camera-barcode-mrz-document-scanner/tree/main/examples/9.x/ARCore