TL;DR This is notes of my first trial of synthetic dataset generation for machine learnings. Just generating simple torus rendered images and trained machine learning model to detect them.
I referred this article about training a model for Tensorflow.js to make a model. I tested with the kangaroo dataset and tomato dataset on Kaggle. Both worked fine and the trained model gave me good predictions. I applied the procedure to my dataset generated by Blender.
Let's put a torus at the origin of the scene. I left the default point light and the camera as it was.
Next, go to the
Shading tab and create a material. By default,
Principled BSDF is used as the surface shader. I left as it is. As I talk in the next section, I changed the
Base Color randomly in the
Principled BSDF to set the color of the torus.
On Blender, we can write python scripts to control objects' properties, set up rendering parameters etc. I automated rendering images, moving / rotating the torus and changing colors randomly. After setting those parameters, I calculated the bounding boxes (x min, y min, x max, y max) of torus in each rendered images to generate annotation dataset for training and rendered images as inputs. I generated 600 images for training and 200 images for testring. Let me share a part of my script. (The full script is on Github. Thanks for a Blender forum article. I use the code on the article to calculate the bonding box.)
base_dir = "d:/rendering_result" geometry_file_path = os.path.join(base_dir, "geometry.txt") donut = bpy.data.objects["Torus"] with open(geometry_file_path, "w") as file: for i in range(800): # Setting rotation donut.rotation_euler = random() * math.pi donut.rotation_euler = random() * math.pi donut.rotation_euler = random() * math.pi # Setting rotation donut.location.x = random() donut.location.y = random() donut.location.z = random() # Setting material color color = (random(), random(), random(), 1) donut.active_material.node_tree.nodes['Principled BSDF'].inputs["Base Color"].default_value = color # Naming a file and render an image f = "image" + str(i) + ".png" path = os.path.join(base_dir, f) bpy.context.scene.render.filepath = path bpy.ops.render.render(write_still = True) # Calculating the bounding box b = camera_view_bounds_2d(bpy.context.scene, bpy.context.scene.camera, bpy.data.objects['Torus']) # Generating annotation CSV row and writing it in a file row = "%s,%i,%i,donut,%i,%i,%i,%i" % (f, 512, 512, b.x, b.y, b.x + b.width, b.y + b.height) file.write(row + "\n")
filename,width,height,class,xmin,ymin,xmax,ymax image0.png,512,512,donut,207,143,333,285 image1.png,512,512,donut,262,157,324,311 image2.png,512,512,donut,253,122,411,267 image3.png,512,512,donut,268,146,420,301 image4.png,512,512,donut,243,185,401,286 ...
I do not explain the details of my training because the steps are written in the article I referred. However, I trained my models on my PC although the article is using Colab. My notes may help if you want to make your models in your local PC.
1000 as the training step. First I tried 7500 as the same as the kangaroo model training in the article, but it was too much for this training.
With the 200 testing dataset, I checked detection results. Almost all torus are detected with confidence over 70% confidence.
However, the model did not recognized the torus in the images taken by my webcam (I took pictures of my PC screen with the webcam). I guess the training torus in the dataset is too clean comparing to the real world. I should include real photo image as the background and some noise, motion blur and so on in the dataset.
Through this process, I learned synthetic dataset generation somewhat work but I still have many improvements I should introduce to use my model in the real world (like AR). Please share your experiences about synthetic dataset generation as comments!