VR Hand Customization

Subject

I’ve held a fascination with Virtual Reality for quite a while, not only for the possibilities of a far higher degree of immersion, compared to classic Videogames, but also for the possibility to incorporate different bodies. Further, character customization can be quite impactful in games and I’ve held curiosity for a long time about the effects customization could have in VR. That’s where the idea for my thesis was born.

I originally would’ve liked to do the study on a full bodied customizable character, but that would’ve gone far beyond the scope of a bachelor’s thesis, so I settled for hands, which, interestingly is as well an aspect not usually focused on too much in character customization. Usually you get the hands you get for your selected gender and skin color and can change the nails if you’re lucky. But since hands are such an integral part of the VR experience, exploring both aspects together held a lot of interest for me.

Study

To research the effect of hand customization on different people I divised an in person study, fully within VR since I didn’t want to break immersion halfway through with an analog questionnaire.

The process would consist of:

• an out of VR demographic questionnaire
• getting into VR and getting accustomed with it
• a simple task done with standard hands
• the first part of the questionnaire
• hand customization
• the same task done with customized hands
• the second part of the questionnaire

The following video shows a run down of the VR elements of this process:

Development

As a base Engine for my Application I chose Unreal Engine 5, since I had some experience with it already, but because there weren’t any customization tools out there yet I had to build the hands and their functionality myself, as well as most things they interacted with for the task and the UI for taking the survey and the customization menu, and the environment for a consistent style. 

I sculpted the hands in Mudbox and then retopologized, rigged and animated them in Maya, did some light textures and masks in Photoshop and Substance Painter and built the rest of the functionality in Unreal. The modelling of the interactible devices and the environment was also done in Maya.

The following video shows a quick overview of the development process:

More on the details of the development and the study can be read in my thesis itself, which I’ve included below:

Behind the Scenes

What isn’t included in my thesis though are more visual details of the development and a look behind the scenes, which I’d like to give a bit more of.

Rigging and Animation

First I want to provide a more thorough look in my rigging and animation process. In this picture you can see the skeletal joints as well as the control rig of the right hand:

This was all done in Maya since I did the animations in Maya already and then just imported the skeletal rig as well as the animations into Unreal Engine 5. You can see more of this process, and especially my struggle with the lower thumb in the video below. I originally had plans to add writing on a whiteboard as a task as well, therefore I made animations for a pen, but sadly adding the physical and functional mechanics of using the pen would’ve taken up too much time, so I had to focus on easier implementable tasks.

Texturing and shading

Due to scope limitations I wasn’t able to include actual skin textures, but I sculpted veins and wrinkles in the sculpting process and managed to extract them via Substance Painter into Normal Maps so I could have that detail within Unreal Engine. Here is the curvature map since it’s the most intuitive, as well as the opacity mask of the hand.

Since there were issues with a basic white to grey to black fade, I used dotting to enable a fade at the wrist. which I projected cylindrically within Substance Painter.

For the texture of the natural nails I photographed my own thumb and then edited the picture to fit the UVs of the nails. Pictures as well as a video of the editing steps can be seen below.

To enable the user to customize the skin tone by adjusting the amount of Melanin and the undertone, I devised a 2D map with the Melanin on the x Axis and the Undertone on the Y Axis, as well as a second map for the insides of the hands since they don’t get as dark as the rest with higher Melanin levels. 

Here is a small overview of how this worked within the Shading Nodes:

Above is the Skin Shader, below the Nail Shader (admittetly a bit more messy) and a more detailed view of the shading of the natural nails.

Environment

For the environment I wanted something futuristic, because the interactive elements had a futuristic design as well. To enable the user to see the hands in direct and indirect lighting I wanted to incorporate a mix of them in the scene and so I oriented my concept on architectural elements from the 2019 game Control, which incorporates a brutalist style with light shafts and massive concrete structures.

Here are some pictures of the early prototyping phase of the environment and how it would work together with lighting to give a mix between direct and indirect lighting the user would interact with.

The following pictures is the final environment from the IntroLevel and TaskLevel respectively. In comparison to before, I extended the ceiling cutout to provide a bit more direct light and added a light shaft behind the user to brighten the environment up a bit.

Failure

And finally a little fun at the end: a compilation of some funny failure of hands that didn’t behave as I expected. I originally planned to make the hands fully physically interactable, i.e. enable fingers to be bent by colliding with other objects or also to have them conform to the object they were holding. That proved to be a gigantic difficulty and I sadly couldn’t make it work in time and had to scale my ambitions back a bit. My efforts yielded some very funny moments though: