TL;DR This is notes of my first trial of synthetic dataset generation for machine learnings. Just generating simple torus rendered images and trained machine learning model to detect them.

Training method

I referred this article about training a model for Tensorflow.js to make a model. I tested with the kangaroo dataset and tomato dataset on Kaggle. Both worked fine and the trained model gave me good predictions. I applied the procedure to my dataset generated by Blender.

Steps

Set up scene

Let's put a torus at the origin of the scene. I left the default point light and the camera as it was.

Next, go to the Shading tab and create a material. By default, Principled BSDF is used as the surface shader. I left as it is. As I talk in the next section, I changed the Base Color randomly in the Principled BSDF to set the color of the torus.

Scripting

On Blender, we can write python scripts to control objects' properties, set up rendering parameters etc. I automated rendering images, moving / rotating the torus and changing colors randomly. After setting those parameters, I calculated the bounding boxes (x min, y min, x max, y max) of torus in each rendered images to generate annotation dataset for training and rendered images as inputs. I generated 600 images for training and 200 images for testring. Let me share a part of my script. (The full script is on Github. Thanks for a Blender forum article. I use the code on the article to calculate the bonding box.)

base_dir = "d:/rendering_result"
geometry_file_path = os.path.join(base_dir, "geometry.txt")
donut = bpy.data.objects["Torus"]
with open(geometry_file_path, "w") as file:
    for i in range(800):
        # Setting rotation
        donut.rotation_euler[0] = random() * math.pi
        donut.rotation_euler[1] = random() * math.pi
        donut.rotation_euler[2] = random() * math.pi
        # Setting rotation
        donut.location.x = random()
        donut.location.y = random()
        donut.location.z = random()
        # Setting material color
        color = (random(), random(), random(), 1)
        donut.active_material.node_tree.nodes['Principled BSDF'].inputs["Base Color"].default_value = color

        # Naming a file and render an image
        f = "image" + str(i) + ".png"
        path = os.path.join(base_dir, f)
        bpy.context.scene.render.filepath = path
        bpy.ops.render.render(write_still = True)

        # Calculating the bounding box
        b = camera_view_bounds_2d(bpy.context.scene, bpy.context.scene.camera, bpy.data.objects['Torus'])

        # Generating annotation CSV row and writing it in a file
        row = "%s,%i,%i,donut,%i,%i,%i,%i" % (f, 512, 512, b.x, b.y, b.x + b.width, b.y + b.height)
        file.write(row + "\n")

The followings are the example images and the annotation data

filename,width,height,class,xmin,ymin,xmax,ymax
image0.png,512,512,donut,207,143,333,285
image1.png,512,512,donut,262,157,324,311
image2.png,512,512,donut,253,122,411,267
image3.png,512,512,donut,268,146,420,301
image4.png,512,512,donut,243,185,401,286
...

Training with the dataset

I do not explain the details of my training because the steps are written in the article I referred. However, I trained my models on my PC although the article is using Colab. My notes may help if you want to make your models in your local PC.

I used 1000 as the training step. First I tried 7500 as the same as the kangaroo model training in the article, but it was too much for this training.

Result

With the 200 testing dataset, I checked detection results. Almost all torus are detected with confidence over 70% confidence.

However, the model did not recognized the torus in the images taken by my webcam (I took pictures of my PC screen with the webcam). I guess the training torus in the dataset is too clean comparing to the real world. I should include real photo image as the background and some noise, motion blur and so on in the dataset.

Through this process, I learned synthetic dataset generation somewhat work but I still have many improvements I should introduce to use my model in the real world (like AR). Please share your experiences about synthetic dataset generation as comments!