Last three months at MLH Fellowship, I worked on Facebook WebXR samples project, developing immersive VR experiences on web using WebXR.

I worked with my teammate Schezeen Fazulbhoy on this project under the mentorship of our mentor and the project maintainer Rik Cabanier from Facebook.

MLH Fellowship has been an amazing experience, where I got to learn a lot, meet new people, collaborate and work on some awesome projects. In this blog, I will share my journey about MLH Fellowship.

MLH Fellowship

“The MLH Fellowship is a remote internship alternative for aspiring technologists. Spend 12 weeks building your skills by collaborating on real-world projects.”

In MLH Fellowship, students get to work on different latest technologies. They are matched with the projects as per the skillsets and interest. Its a global platform where students from across the globe collaborate and contribute.

In MLH Fellowship, fellows are put into small groups ‘pods’. The fellows collaborate to contribute to assigned projects under mentorship of professional software engineers.

And apart from the coding and project aside, there are a ton of skills building, team building and fun activities and workshops.

Learn more about MLH Fellowship here.

The Inception

So, it was in late 2020, I came to know about MLH Fellowship. I found about it on Facebook and it sounded like a great program. And, on learning more about it from a YouTube video by Kunal Kushwaha, I really wanted to be a part of this program.

As soon as my exams finished in December, I started looking for the fellowship. The application form required basic information, a code project sample and three question essays.

About the project, I submitted a group project specifying my role in the project. The project was in JavaScript. And as for the questions, I answered those honestly about my interest, learnings and more on the project.

The Process

After submitting my application, I was eagerly waiting for the result.

My initial application had passed, and I received the email regarding that. However, I had submitted my application very close to deadline. And that had its effect.

In short, my application had passed, and I also gave my Initial Interview, but the class for Spring Batch was already full. So, my application was deferred for the next batch.

Everything happens for a reason

And that was the very very good thing that happened to me.

It was because in March 2021, I got to be in the World Finals of Microsoft Imagine Cup 2021.
Imagine Cup was also a life changing experience.

More on My Journey to the World Finals – Imagine Cup 2021

Had I been in the Fellowship Spring Batch, which would have started in February, it would have been quite difficult to manage time for both.

Next Steps

My Initial Interview had been with Sashrika Kaur who was very friendly and the interview had gone smooth.

So, after my Initial Interview, I had my Technical Interview. My technical interview was with Kunal Kushwaha. It would be around the project code sample I had submitted, and technologies and tools I had mentioned in the application.

Both the interview went very good and smooth. The people form MLH are very nice and friendly, and that makes the interview process very smooth and easy.

The Results

I had a feeling that my interviews had gone good and was expecting to get accepted.

Around late May, I received the email, “Congratulations, you’re going to be an MLH Fellow! 🎉”.

I was very happy that finally, I was going to be an MLH Fellow.

MLH Fellowship – The Journey Starts

The journey starts, sign ups in different platforms and getting to know the fellowship system.

I was an Open Source Fellow and I was in Pod 3.0.2. Karan Sheth and Alex Wilkes were our pod leaders. And in my pod were Abishek, Akashdeep, Ben, Daniela, Howard, Jake, John, Khairo, Schezeen, Stuti and me.

We were from different parts of the world. And this diversity is the beauty of programs like MLH Fellowship, where we get to meet people from different culture and background. This helps to get newer perspectives and learn new things.

We named our pod Bashful Boolions, and here is our pod logo designed by Ben.

The first week was about getting to know the MLH Fellowship better and getting to know the team members.

Each fellow had to do an 1-on-1 meeting with other fellows in the pod. This was a great way get to know each other better.

Then, we also had the Orientation Hackathon. There were groups formed of 3-4 members. There were many amazing projects presented in the hackathon.

Our team created Koding Buddy, which is a platform providing a collaborative working space in the form of a code-editor and compiler along side a video chat window to help people connect in a more fruitful manner.

The second week started with assignment of projects and this summer I was going to be contributing to the WebXR Samples project.

The Project – WebXR Samples

In MLH Fellowship, I worked on Facebook / WebXR Samples projects along with my pod mate Schezeen Fazulbhoy. We collaboratively worked on the project under the guidance of our project maintainer Rik Cabanier.

Hello WebXR

We were introduced to the project by our pod leaders Alex and Karan, and we got to learn more about it from our maintainer Rik.

Before we started on the project, we had meetings with Rik who explained about the project and the issues that we could be working on.

We would be working on the existing code base by previous fellows Soham Parekh and Zhixiang Teoh. We had a call with both of them and got insights about the project, reference implementations and resources to learn about WebXR.

Previous fellows were mainly focused on WebXR Layers. As per the instructions from Rik, we mainly focused on creating and implementing Hand Interactions and Gestures with WebXR Hand Inputs API.

Getting Started

AR/VR side of development is a something which I find interesting but had never tried before the fellowship. It was amazing to know that using my existing knowledge of JavaScript, I could build immersive experiences for the web.

I had some experiences with 3D Graphics, but I was new to XR, WebXR and three.js. So, initially I learnt about three.js, creating 3D environments and objects with it, and leveraging it to access the WebXR API.

Then I learnt about WebXR and creating the immersive VR experiences for web with it. After that, I started working on the project, first with basic VR scenes and then moving on to the hand gestures and interactions.

We used the WebXR browser Emulator until the time our Oculus Quest 2 had arrived.

The Implementation – Under the Hood

The WebXR Device API gives access to hand-tracking XR systems primarily exposing the poses of hand skeleton joints. It can be used to render hand models in VR scenes as well as performing gesture detection with hands.

Accessing the API, we performed calculations on the positions of the hand skeleton joints in the VR world. Then, we created some mathematical functions to detect the hand gestures and used three.js 3D rendering library to render the hands and other objects in the VR scene.

By calculating distances amongst the hand joints and the objects, we detected simple hand gestures as pinching, touching and selecting.

Finally, the Project

By the implementation of above mentioned hand gestures, we built hand interaction samples.

Some of the samples we built are:

• Drawing in space with fingers
• Creating and erasing three.js objects
• Grabbing, transferring and changing properties the three.js objects
• Using two hands interaction to resize the objects

Apart from that, we worked on some interesting features like Spiderman pose and Thanos snap pose.

In Spiderman pose, one would throw the spider web with hands when we give the Spiderman go-web-go pose.

In Thanos example, half of the objects in scene would be deleted with a snap.

We faced some challenges setting up the project and development environment with the Oculus.

So, keeping that in regards, we created detailed documentation on project and oculus development environment setup. So that, future contributors can get started with the project quickly and easily.

Takeaways

WebXR is very exciting but it might be intimidating at first.

Once we start to build stuffs, it’s really fascinating to see how the project just comes to life.

As a child I used to do a lot of paper crafts. And coming to this day, creativity and a sense of building something is one of the best things I love about software engineering.

And the fact that, we could give our creative inputs in the project, and how we could actually be inside of the project once we put on the Oculus and press that ‘Enter VR’ button, is something really exciting.

In the past three months, I have taken a new journey to the world of AR/VR and learnt and build some awesome stuffs that I would not have imagined before the fellowship.

Apart from technical skills, I got to know and work with Schezeen, who have been an awesome teammate. I am grateful to have had Rik as our maintainer whose insights were highly valuable and encouraging for us.

This project has definitely given me the confidence to tackle challenges in my future ventures and contributing to other open source projects.

Knowing that the samples we have built will serve as reference for future implementations is one of the best takeaways.

Apart from the Project

Apart from the project, there were many other activities going on in the fellowship. As we have already talked about the hackathon. Then there were daily events or workshops by MLH for different different skills development.

The pod meetings and weekly retrospections where we would share about progresses, shoutouts or any blockers were really helpful to keep oneself on track.

There were game nights, retrospections, capturing the flag, and many others which were fun and were a great opportunity to learn.

Show and Tells in MLH Fellowship

Then there were the show and tells. I did my show and tell on paper craft and some basic robotics stuff.

I created a paper toy car model, and connected it with ESP8266 and some motors. The system (car) would be controlled from a webpage remote that I deployed on Azure. Finally, I let my podmates remotely control the car.

And all this connections, paper crafts, and coding and deploying stuffs I did live in show and tell. It was a challenge for myself as well, and it went great.

Around the End of the MLH Fellowship

I had a great time with my pod this summer. Here is one from our last pod meeting.

And around at the end of the fellowship, we had demo days, where every fellows would be giving the demo of their projects or works.

Me and Schezeen prepared for it and we shared about WebXR, Our Project, the Community, How it works, Challenges, Learnings and Takaways.

And at last, we shared our project demo live by sharing the Oculus Screen casting. The demo went very good.

Finally at the end of August, we had our graduation. And, I graduated from the MLH Fellowship Batch 3.

And,

MLH Fellowship has been a great experience. Working on amazing projects while collaborating with amazing teammates, learning from our awesome project maintainer, its been an amazing experience.

I got to work on WebXR Samples project. The project itself was awesome and the fact that we were able to give our creative inputs to it was a great thing. I got to learn about XR, 3D Graphics, three.js and WebXR Library. Also, I learnt about the power of pair programming sessions. Moreover, I also got to work with Facebook Oculus Quest 2.

I got an amazing pod with awesome pod leaders and pod mates. Everyone was so much helpful and we got to learn so much from each other, via show and tells and one on ones. Not only about the technical stuffs I also got to learn more about the cultures, geographies of my podmates.

From the initial hackathon, where we got to learn about good GitHub practices and many more, to our demo day, where I showed my Thanos Snap hand interactions with WebXR project saying, ‘This one is INEVITABLE’, every single experience in MLH has been amazing.

MLH Fellowship is one of the best thing that has happened to me.

The MLH Chapter has added strong skillsets, amazing talented friends, and tons of memories to cherish for the future. Its truly been a lifetime experience.