The Bubble Figure: Integrating Tensorflow.js With Three.js and Cannon-es to Create Interactive Art

Recently I had the pleasure to work on one of the funnest projects ever: The Bubble Figure.

The Bubble Figure was created by a team of volunteers at Wunderdog for Grand One 2023, annual design awards that take place in Helsinki. Our designers created the visuals for the event and wanted to do something extra.

After exploring various ideas, tech, and taking the time and resource constraints into mind, we decided to go for The Bubble Figure.

What in the world is The Bubble Figure?

The Bubble Figure is an interactive experience that runs in the browser. When a person is recognized via the webcam it makes them appear as a colorful bubble figure. They can then interact with various shapes that fall from above, for example, by hitting them. Simple and fun, and all in the visual language of Grand One 2023.

How does it work?

The Bubble Figure uses TensorFlow.js pose detection to recognize a person. Pose keypoints (e.g. left shoulder, right wrist) are then mapped onto a Three.js scene.

The figure is randomized every time a new person is detected. The shapes falling from above are also randomized, while using certain geometries and textures.

To simulate physics, cannon-es is used, a maintained fork of cannon.js (we found out the hard way cannon.js wasn't maintained anymore). Shape coordinates are mapped from the physical world to the Three.js scene, while the figure is mapped from the scene to the physical world.

How did we do it?

Funnily, none of us had any experience with these libraries. We learned everything the best way: on the job, with lots of trial and error.

ChatGPT was very useful to understand some concepts from the tech stack and generate basic code. But of course, this required some fixing and modifying to get it to work.

We did quite a lot of tinkering from the design perspective, to make sure The Bubble Figure looks alright. This required adjusting the number of bubbles, their sizes, offsetting their positions, and adding randomness to create various interesting results.

Can I try it?

Absolutely! You can check out the latest demo on GitHub pages.

Please use Google Chrome, other browsers are not supported. Allow the browser to use the camera. Then take a few steps back to make sure that your pose is detected, and knock some shapes around.

You can also hit Command + K on Mac or Ctrl + K on Windows to adjust various parameters. Clicking Apply will save the parameters in the browser's local storage.

Are there any limitations?

Of course, it wouldn't be a software project without limitations.

For instance, cylinders and cones don't always collide with the figure in a smooth manner. We improved it a bit by randomizing the figure's bubble z coordinates. Still, some weird collisions may happen. Isn't it part of the fun though?

We also found it rather difficult to render the bubble figure always in the same size, regardless of the person's distance from the camera. Maybe given more time we would have succeeded, but we had to release the project.

The frame rate also changes once the pose is detected. Nothing really bad, even though ideally it should be consistent.

Also, for some reason using TensorFlow pose detection as NPM packages caused various JavaScript errors. That's why we had to resort to loading it via script tags.

Where is the code?

The Bubble Figure has been released as open source in the Wunderdog GitHub account.

Since we were short on time, we did rapid prototyping and the code got very messy. So, following the event, we spent about a week cleaning it up and also adding TypeScript.

One thing I might have done differently, is to create a the figure with a skeleton. Then it should be easier to move the skeleton's joints and possibly to keep the figure the same size regardless of camera distance. Oh well, next time.

I hope that you like this little project. May you find it useful if you would also like to create an interactive browser experience with pose detection, 3D graphics, and physics.