In this demo, we will try to reproduce one of the main interface of the Death Stranding game.

Demo Link
Demo repository

About the game

Death Stranding is a game produced by Hideo Kojima (especially known for its Metal Gear series games). The game takes place in a post-apocalyptic future where an unknown phenomenon has ravaged most of the world. You play a character, Sam, responsible for making merchandise deliveries to the scattered remains of the population in a world that became quite dangerous. If Sam looks familiar to you it’s because its model is based on the actor who played Daryl in Walking Dead.

About this interface

On this interface, the player must arrange the merchandise he will carry from point A to point B.
The arrangement done by the player will have a significant consequence on the success of the delivery.

This interface is really interesting for a number of reasons:

• The player is expected to spend some time in this interface so it is really important that it doesn’t break the flow of the game.
• It should also keep the player fully immersed in the universe of the game
• How it uses both a 2D overlay on top of a 3D Scene
• Its aesthetic choices

For the sake of this article, I reduced the scope of the interface but I tried to keep the essence of what makes it interesting. Our goal will be to reproduce:

• The 3D scene to display the merchandises
• The 2D overlay to manage the merchandises
• Keeping some interactions between the 2D overlay and the 3D scene

For the 3D scene, there will be 3 different positions to display the merchandises:

• Private locker (the main storage)
• Shared locker (alternative storage)
• Sam cargo (represents merchandises carried by Sam)

Target audience

This article requires some knowledge about threejs and react-three-fiber.
If you have no experience in threejs the best resource on the web to get started is the course made by Bruno Simon: ThreejsJourney
If you’re looking for resources on react-three-fiber you can take a look at this repository

Format

There are 2 possibilities to consume this article. You can simply read it to get a global understanding of how the demo works or you can try to reproduce the demo to have a deeper understanding.
If you choose the latter, I created a starter project on codesanbox with all the assets to get started more easily. You can also download it if you prefer to work locally.
Feel free to choose what suits you best.

Starter

Complete demo

The stack

The base project is a classic create-react-app. Here’s the list of the additional libraries used in it:

• react-three-fiber (for the 3D scene)
• react-spring (for 2D and 3D animations)
• valtio (state management)
• twind (styling solution based on Tailwind)
• drei (react-three-fiber helpers collection)

A note on Twind:
This library is a CSS-in-JS version of TailwindJS. If you're more comfortable with another styling solution don't hesitate to replace it. If you prefer vanilla Tailwind, Twind can be used just like that by using the following shim (already included in the starter).

Interface components

We're going to start building our interface with the 3D part. First, we will create the 3D grid of the private locker. The grid cell delimitations will be done using particles.
Then we will create two smaller grids (for shared locker and sam cargo) without particles. Finally, we need to be able to move the camera between these 3 positions.

3D

Components List

Briefcase

This component will be responsible for loading and displaying the model. We will go through the whole process but some parts are already done in the starter.

• download our gltf model from sketchfab (credit goes to luac for the model)
• convert it to a react component using gtltfjsx locally or the new online version
• convert PNG to JPEG and optimize them
• using draco to convert our gltf file to GLB and compress it at the same time.
• put the GLB file in our /public folder

At this point, we should be able to see the model. Now we have to position/rotate/scale the model correctly so it fits the original UI.

We will also handle a secondary display for the model. It will be useful later on to separate the selected item from the other. For this secondary display, we will try to display it with a translucent blue color and a wireframe on top of it.

• First, we need to duplicate the main material (the first one) of the briefcase into two meshes
• For the translucent blue color we can use a simple shader by using component-material on the first material

const SelectedMaterial = ({ blue = 0.2, ...props }) => {
 return (
   <>
     <Material
       {...props}
       uniforms={{
         r: { value: 0.0, type: 'float' },
         g: { value: 0.0, type: 'float' },
         b: { value: blue, type: 'float' },
       }}
       transparent
     >
       <Material.Frag.Body children={`gl_FragColor = vec4(r, g, b, blue);`} />
     </Material>
   </>
 )
}

• For the wireframe it’s already built-in threejs, we just have to use the wireframe attribute on the second material

To simulate the selected state you can try to use react-three-a11y. By wrapping our model with the <A11y> component we will have access to hover, focus, and pressed state through useA11y() hook. We can try to display a SelectedMaterial based on the hover state for example.

Since we will have a 2D overlay on top of the 3D scene we won’t need react-three-a11y afterward but it’s good to know that you can bring accessibility to your 3D scene quite easily with it.

Particles grid

This will be the most complex part of the demo.
To recreate this grid we will need 2 components:

• A Grid component to display the particles
• A GridContainer to compute the positions of the particles and the briefcases

There are 2 different kinds of particles which are called smallCross and bigCross. In the end, we will have to compute these 2 position arrays plus the one for the briefcases.

Grid

First, we will start with the Grid component.

const Grid = ({ texture, positions = [], ...props }) => (
 <points {...props}>
   <pointsMaterial
     size={0.6}
     opacity={0.5}
     color="#316B74"
     alphaMap={texture}
     transparent
     depthWrite={false}
     blending={THREE.AdditiveBlending}
   />
   <bufferGeometry attach="geometry">
     <bufferAttribute attachObject={['attributes', 'position']} count={positions.length / 3} array={positions} itemSize={3} />
   </bufferGeometry>
 </points>
)

Here we’re using an alpha map texture to recreate the “cross” particle effect. We’re also tweaking a few parameters for the colors and the transparency. The particle’s positions and count are given to the bufferAttribute tag. The positions array needs to have the following format [x1, y1, z1, x2, y2, z2, ...].

GridsContainer

Let’s continue with the GridsContainer.
We said that we have 3 position arrays to compute but we can do the 3 of them at the same time.

First question, how many particles do we need for the small cross particles array?

Let’s say we want

• 20 particles per line
• 6 lines
• 2 layers

Also for one particle weed 3 values (x, y, z).
In the end, we will need an array 720 values (20 * 6 * 2 * 3) to display a grid of 20 columns, 6 lines, and 2 layers.

This is only for the small cross particles position array, the big cross array has 2 times less coordinate and the briefcases one 4 times less.

This is because for each cell we want to display:

• 4 small cross particles
• 2 big cross particles
• 1 briefcase

There are probably several ways of doing this. Here’s one method:

• loop over the array with 720 placeholder values
• for each loop, we need to know if we’re computing an x, y, or z coordinate
• for each case, we compute 3 differents coordinates (small cross, big cross, briefcase)
• we push these 3 coordinates in their respective arrays

At the end of the loop, we can filter the coordinates we don’t need for the big cross and briefcases arrays (remember that we have 2 times and 4 times fewer coordinates for these too).

Don’t hesitate to put every configuration variable (columns, lines, layers, spacing …) for this grid in a tool like leva to make it look like what you want.

In the actual render, we need to:

• map over an arbitrary number (we will change that later)
• render our Briefcase components with positionsBriefcases values
• render a Grid components with positionsSmallCross values
• render a Grid components with positionsBigCross values

External grid

This one is simpler than the grid we just build since it doesn’t use any particles.
Here we just want to display briefcases on the same Z value, 3 columns, and any number of lines. In our new ExternalGrid component we will map just the briefcases list and call a util function to get the position.

Our util function to get the position could look like this:

const X_SPACING = 2
const Y_SPACING = -1

export const getPositionExternalGrid = (index, columnWidth = 3) => {
 const x = (index % columnWidth) * X_SPACING
 const y = Math.floor(index / columnWidth) * Y_SPACING
 return [x, y, 0]
}

Floor and fog

To make the scene look right color-wise on the background we have to add a floor and a fog.

Floor:

   <Plane rotation={[-Math.PI * 0.5, 0, 0]} position={[0, -6, 0]}>
     <planeBufferGeometry attach="geometry" args={[100, 100]} />
     <meshStandardMaterial attach="material" color="#1D2832" />
   </Plane>

Fog:

<fog attach="fog" args={['#2A3C47', 10, 20]} />

Add these 2 elements to the main canvas.

2D

State and data

Before going into building the HTML UI we need to create our state with the data.
For this demo, I wanted to give a try to valtio as the state manager.

We will need to create a state with proxyWithComputed, because we will have to computed values based on the state.

In the actual state we have only two values:

• allItems (list of all the briefcases)
• selectedItem (index of the selected briefcase inside allItems)

To populate it we need a function to generate data. This function already exists in the starter.

So our state looks like this for now:

proxyWithComputed(
 {
   selectedItem: 0,
   allItems: [...generateItems(9, 'private'), ...generateItems(3, 'share'), ...generateItems(3, 'sam')],
 },

The second parameter takes an object and is used to define the computed values.
Here’s the list of computed values we will need:

• isPrivateLocker (based on the selectedItem)
• isShareLocker (based on the selectedItem)
• isSamCargo (based on the selectedItem)
• itemsPrivateLocker (filter allItems)
• itemsShareLocker (filter allItems)
• itemsSam (filter allItems)
• allItemsSorted (use filter computed values to sort the array)
• selectedId (ID of the selected item)
• selectedCategory (category of the selected item)
• totalWeight (sum of briefcase weight inside Sam cargo)

Components List

Inventory

This is the component that will display our list of briefcases. As we saw on the schema it uses the following child components:

• MenuTab (pure UI component)
• MenuItems (display a portion of the list, ie: briefcases in PrivateLocker)
• ActionModal (will be discussed just after)

The component should also handle the following events:

• keyboard navigation
• mouse events
• update the selected briefcase in the store
• open ActionModal

Action modal

In this modal, we add actions to move the selected briefcase from one category to another.
To do that we just need to update the category of the selected item in the store. Since we’re using computed values to display the lists, everything should update automatically.

We will also need to handle keyboard navigation in this modal.

Item description

This is the right side part of the UI. We just need to display all the data of the selected item here.

The only interaction is about the like button. Whenever the user clicks on it we should update the likes count of the selected briefcase. This is simple to do thanks to Valtio, we just update allItems[selectedItem].likes in the state directly and the likes counts should update in the Inventory.

Combining 2D and 3D

We now have a 2D UI and a 3D scene, it would be nice to make them interact with each other.

Selected briefcase

Currently, we just highlight the selected item in the UI part. We need to reflect this to the 3D briefcase as well. We already made the selected material, we just need to use it inside the Briefcase component.

Scene transition

From now on, our camera was only looking at the main grid, the private locker. We will create 3 components to move the camera and display them based on the properties isPrivateLocker, isShareLocker, and isSamCargo that we created earlier in the state.

Here for example the code that look at the main grid:

function ZoomPrivateLocker() {
 const vec = new THREE.Vector3(0, 1.5, 4)
 return useFrame((state) => {
   state.camera.position.lerp(vec, 0.075)
   state.camera.lookAt(0, 0, 0)
   state.camera.updateProjectionMatrix()
 })
}

Adding perspective

To give our UI a more realistic look we must make it look like it is slightly rotated from the camera. We can do that with the following CSS:

body{
  perspective 800px;
}

.htmlOverlay {
  transform: rotate3d(0, 1, 0, 357deg);
}

Animations

We’re now going to add some animations to both the UI and the 3D scene.
All animations has been done using react-spring.

2D

MenuEffect

This is the animation that happens inside Inventory whenever the selected item changes.

There are actually 3 parts to this animation:

• a sliding background going from left to right
• the item background going from 0 to 100% height
• a slight blinking loop for the background-color

We will go through each of them and combine them together with the useChain hook.

Sliding animation

To reproduce this animation we will need custom SVGs (they are already available in the starter). I used the tool https://yqnn.github.io/svg-path-editor/ to make 3 SVGs.

I think we could have an even better effect with more SVGs, feel free to try adding more frames to animation.
To animate these 3 SVGs, we will declare a x property inside a useSpring going from 0 to to 2 and in the render we will have this:

         <a.path
           d={
             x &&
             x.to({
               range: [0, 1, 2],
               output: [
                 'M 0 0 l 16 0 l 0 3 l -16 0 l 0 -3',
                 'M 0 0 l 25 0 l -10 3 l -15 0 l 0 -3',
                 'M 0 0 l 16 0 L 16 3 l -5 0 l -11 -3 m 11 3',
               ],
             })
           }
         />
       </a.svg>

Now we just need to animate the opacity and the width and we should have a good sliding animation effect.

background height

Here we’re just expending the item’s background with a default spring:

const [{ height }] = useSpring(() => ({
   from: { height: 0 },
   to: { height: 24 },
   ref: heightRef,
 }))

glowing color animation
To reproduce this part we will make a spring between 2 colors and play with the opacity at the same time:

 const [{ bgOpacity, color }] = useSpring(() => ({
   from: { bgOpacity: 1, color: '#456798' },
   to: { bgOpacity: 0.5, color: '#3E5E8D' },
   ref: bgOpacityRef,
   loop: true,
   easing: (t) => t * t,
   config: config.slow,
 }))

All together
Finally, we just have to use these 3 animations with the useChain hook

 useChain([opacityRef, heightRef, bgOpacityRef], [0, 0.2, 0])

SideMenuEffect

The SideMenu animation will use the same technique we just saw. It will be a spring that goes through 3 SVGs. Again I was a bit lazy on the number of SVG frames, feel free to try with more.
Here are the 3 SVGs I used for the demo:

             output: [
               'M 0 0 l 16 0 l 0 3 l -16 0 l 0 -3',
               'M 0 0 l 25 0 l -10 3 l -15 0 l 0 -3',
               'M 0 0 l 16 0 L 16 3 l -5 0 l -11 -3 m 11 3',
             ],

AnimatedOuterBox

Here our OuterBox component:

const OuterBox = () => (
  <div>
    <div className="h-1 w-2 bg-gray-200 absolute top-0 left-0" />
    <div className="h-1 w-2 bg-gray-200 absolute top-0 right-0" />
    <div className="h-1 w-2 bg-gray-200 absolute bottom-0 left-0" />
    <div className="h-1 w-2 bg-gray-200 absolute bottom-0 right-0" />
  </div>
)

This component is displayed inside ItemDescription one. It shows four little white stripes at the edges of ItemDescription.

On the animation side, we will have to animate the height property of the component from 0 to 100%.

AnimatedBar

For the bar that shows an item's durability, we will make an animated bar (like a loader).
We need to animate the width property based on the damage attribute of the item.

3D

For the 3D scene, we will add just one animation that will be triggered whenever a briefcase is changed from one category to another. We will make it seem like the briefcases, those that have changed, are falling from above.

We can handle this animation in the Briefcase component. Whenever the position of a briefcase will change, we will animate the new value on the Y-axis from the new value plus a delta to the new value.

Until now the spring animations were triggered whenever a component was mounted. Here we need to animate briefcases that are already mounted.
To trigger a spring that has already been played once we need the second parameter received from the useSpring hook.

  const [{ position: animatedPosition }, set] = useSpring(() => ({
    from: { position: [position[0], position[1] + 5, position[2]] },
    to: { position },
  }))

Be careful to use @react-spring/three instead of @react-spring/web here.

Sounds

For the sounds part we’re going to create a sound manager component using useSound hook from Joshua Comeau. After that, we’re going to put our sound functions newly-created into our state so that we can everywhere in the app.

Here’s the list of sounds we need to handle:

• like button
• menu change (played whenever the item selected change)
• menu action (played whenever the action modal is opened)
• menu validate (played whenever the action modal is closed)

Conclusion

We’re done with the tutorial, I hope you liked it. If you’re trying to make your own version of the Death Stranding UI, don’t hesitate to share it with me on twitter. If you're interested in more GameUI on web demos I share updates on the upcoming demos on this newsletter.