Solution to 3D Modeling on Mobile Apps

"3D modeling" is commonly referred to the use of 3D production software to build a model with 3D data. 3D modeling is useful in a wide range of scenarios like product modeling, 3D printing, teaching demos, game development, and animation creation.
However, It takes at least several hours to build a low polygon model for a simple object, and even longer for a high polygon one. The average cost of building a simple model can be more than one hundred dollars, and even higher for building a complex one.
3D Modeling Kit is the best tool to improve 3D content production efficiency, which is designed to facilitate fast, simple, and cost-effective 3D content creation, to help create 3D models and animations.

3D Modeling Kit currently offers three capabilities: 3D object reconstruction, material generation, and motion capture. In the near future, more capabilities, such as human body modeling and facial motion capture, will be available.

3D Object Reconstruction

• Introduction

This capability is used for modeling, but differs from the other two modeling methods, of which one uses a 3D modeling program and the other uses a scanner. 3D object reconstruction helps develop the user-generated content at lower costs for mobile apps. Instead of images with depth information, the capability requires images collected from a standard RGB camera and captured from multiple angles of an object. Using the images uploaded to the cloud, the capability can automatically generate a 3D model with textures.

When it comes to hardware requirements, 3D object reconstruction now supports Android and iOS devices. It could offer web APIs in the future, bringing powerful functions to all kinds of apps.

As this capability implements image-based modeling, it has some special requirements for objects, such as requiring them to have rich textures with rigid bodies that are non-reflective and medium-sized. Such objects include goods (plush toys, bags, and shoes), furniture (sofas and pillows), and cultural relics (bronzes, stone artifacts, and wooden artifacts).

The optimal dimensions for an object are between 10 x 10 x 10 cm and 200 x 200 x 200 cm. The capability also supports objects with bigger dimensions, but will require more time to model such objects. It typically takes less than five minutes for 3D object reconstruction to use 1080p images to generate a model, which will be saved as a standard model file format, OBJ or glTF.

Last but not least, the size of the SDK for this capability is only 83 KB, which will not greatly increase the app size.

• Application Scenarios

3D object reconstruction is ideal for the e-commerce industry. The image below demonstrates an example of this function's usefulness by showing how to model a shoe and cultural relic. Put it on a turntable in a light box, and then collect images by following the image collection suggestions mentioned above.

3D models can be a key factor for boosting user conversion, as it provides a more immersive shopping experience for users and allows them to grasp a realistic impression of the products.

Regarding the cultural relic industry, 3D object reconstruction can be used to generate 3D models for cultural relics, ensuring that they can be digitally preserved and displayed in multimedia.

Material Generation

• Definition of the Material

The second capability from 3D Modeling Kit is material generation. Appearance in computer graphics, also known as material, is a property that describes how light interacts with the surface and inside of the object. A material model and a set of control parameters define the surface appearance.

Materials play a vital role in making a virtual scene look more realistic. Take the following two images as an example. The image on the right appears more realistic than the one on the left, because the former is texturized to define the appearance of the walls, floor, oil drums, and table cloth.

Developers and users are faced with several obstacles when trying to create materials.

First, material creation is time-consuming, strenuous, and hard to standardize. Before being used to create life-like materials, images of textures in the real world have to be processed by an art designer using professional software. For example, obtrusive lighting must be removed, and roughness and dimensions must be adjusted.

Second, the cost of material creation is high. The classic approach of texture mapping requires skills and specific renderers. A further barrier to texture mapping is that images used in a project and renderer can rarely be reused in others. Besides, it's hard for art designers to accumulate their experience.

To overcome these obstacles, we propose two solutions:
First, improve the efficiency and quality of material creation, by using the deep learning network to easily generate physically based rendering (PBR) texture maps.

Second, make texture maps usable in different projects and renderers and spread artistic experience, by standardizing the experience and material creation specifications of technical artists into data that complies with the PBR standards.

• Introduction to Material Generation

This capability supports one-tap generation of four types of texture maps via one or more RGB images.

All you need for this is an Android device with a standard RGB camera, which does not have to be a time-of-flight (ToF) camera or light detection and ranging (LiDAR) sensor. The supported material types include concrete, marble, rock, gravel, brick, gypsum, clay, metal, wood, bark, leather, fabric, paint, plastic, and composite material.

The generated texture maps come in four types, which are the diffuse map, normal map, specular map, and roughness map. Regarding input image quality, this capability requires that the image's resolution be between 1K to 4K and that the image have no seam, bright spot, shadow, or reflection. The SSIM for the rendered material is greater than 0.9, which indicates that the material generation capability performs rather well.

The image below illustrates how to use material generation to quickly create a house. To begin with, we need to generate texture maps. Once this is done, drag the maps to the spheres, and then copy the texture spheres to the blank models as needed. Finally, the house is created via offline or online rendering.

Motion Capture

This capability becomes available recently. It quickly and accurately calculates the 3D data of 24 key skeleton points of the human body, through the RGB image or continuous video frames shot from a common monocular camera.

It supports common poses including standing, walking, and running.

The optimal resolution for input images and video is within the range from 320p to 1080p. When the resolution exceeds this, the capability will take a longer time to perform calculation, and this will not significantly boost accuracy. Therefore, to ensure that the capability can offer the ideal results for the input video frames or RGB image with a resolution higher than 1080p, you are advised to scale them down first.

Let's see the output. This capability simultaneously outputs quaternions and 3D coordinates of 24 key skeleton points, which can be used directly in some engines for skeleton animation.

When the capability runs on a phone with the CPU, the detection frame rate is up to 30 fps; when it runs on a phone with the NPU, the detection frame rate is up to 80 fps.

Motion capture can be integrated in either full SDK mode or base SDK mode. If your app runs on Huawei phones, you need to integrate the base SDK which has a file size of 200 KB and download the algorithm package when the capability is used.