Background
As a designer who loves type design and typography, I have been falling in love with holographic types since I met HoloLens back in 2015. Microsoft HoloLens allows you to place and see holographic objects in your physical environment. Just like real-world physical objects, you can place them on a table or wall, you can move around and see them from different angles.
Typography Insight for HoloLens(2016) was my first experiment with holographic types in space. I created an app that allows you to lay out and experience beautiful holographic types in a real-world environment. The composition component has evolved into my new project Type In Space(2018) for HoloLens.
Being able to see and interact with a holographic type in a real-world environment is one of the most exciting experiences. Now, with HoloLens 2’s new hand-tracking and eye-tracking input, I can directly touch, grab, and manipulate the holographic type with my hand. This instinctual interaction opens up totally new creative possibilities.
In this story, I would like to share my journey on bringing the Type In Space app from HoloLens 1st gen to HoloLens 2.
HoloLens 2’s Instinctual Interactions
HoloLens 2 is packed with a lot of new technologies such as a completely new MEMS(Micro-electromechanical systems) laser display with larger FoV, and HPU(Holographic Processing Unit) 2.0 with AI coprocessor. However, one of the most exciting new capabilities in HoloLens 2 is the fully articulated hand-tracking input. With hand-tracking input, you can directly interact with holograms by touching, grabbing, and pressing them. This new input interaction dramatically changes how we interact with holographic objects and makes it very natural and ‘instinctual’.
With hand-tracking input, finally, I was able to touch and grab a gorgeous holographic type.
MRTK: Building blocks for spatial interactions and UI
MRTK(Mixed Reality Toolkit) is an open-source project by Microsoft. MRTK-Unity provides a set of foundational components and features to accelerate mixed reality app design and development in Unity. The latest release of MRTK v2 supports HoloLens/HoloLens 2, Windows Mixed Reality, and OpenVR platforms.
Since MRTK v2 is completely redesigned from the ground up, foundational components such as the input system are not compatible with MRTK v1 (HoloToolkit). Since I used HoloToolkit for the original version of my app Type In Space, I started from scratch to create a new version with MRTK v2. It was much faster to start from scratch because most of the core interactions can be achieved with the building blocks in MRTK v2.
UX Elements: Text Object
Text is the most important component in the Type In Space app. The text object is composed of the following elements.
Text Mesh Pro
Since HoloLens has a high-resolution display with 47 PPD(Pixels Per Degree), it can display sharp and beautiful text. To properly leverage this high-resolution, it is important to use a properly optimized text component. Unity’s TextMesh Pro uses SDF(Signed Distance Field) technique to display sharp and clear text regardless of the distance. For more details, see Typography guideline and Text in Unity on Mixed Reality Dev Center.
Near & Far Manipulation
Being able to directly grab and manipulate the holographic type is one of the most important core interactions. MRTK’s ManipulationHandler script allows you to achieve one or two-handed direct manipulation.
ManipulationHandler also allows far-field interactions with hand rays in HoloLens 2. You can use two hand rays to grab/move/rotate/scale objects.
Bounding Box
The bounding box is a standard interface for the precise scale and rotation of an object in HoloLens. For the Type In Space app, I used it to indicate the currently selected text objects by displaying the corner handles. MRTK’s Bounding Box provides various configurable options for the visual representation of the handles as well as the behaviors.
UX Elements: Menu UI for Text Properties
Button
The button is one of the most foundational UI components. In HoloLens 2, you can directly press buttons with hand-tracking input. However, since you are essentially pressing through the air without any physical tactile feedback, it is important to amplify visual and audio feedback.
MRTK’s HoloLens 2 style button provides rich visual/audio cues and handles complex logic for the speed/trajectory/direction of the finger movements. Visual feedback includes proximity-based lighting, highlight box, compressing front cage, hover light on the surface, pulse effect on press event trigger, and the fingertip cursor.
Since it prevents back-pressing, it could be used for overlay menus too. (examples below in the hand menu videos) I used the HoloLens 2 style button prefab for all menu UIs including fonts and colors list.
Hand Menu
In the original version, I had a floating menu with tag-along behavior. It followed the user so that it can be accessed anytime. In HoloLens 2, there is an emerging pattern called the ‘hand menu’. It uses hand tracking to display quick menus around the hands. This is very useful to display a contextual menu when it is needed then hide and continue to interact with the target content. To learn more about the hand menu, see Hand Menu design guidelines on Mixed Reality Dev Center.
I started using this UX pattern for the text property menus and tested them. Below are the Mixed Reality Capture videos from HoloLens 2 device.
As you can see, it works quite well. You can quickly raise your palm to display the menu and change the properties. Since it could cause muscle fatigue if you want to change multiple properties and adjust details, I have added an additional option to make the menu world-locked. Tried a pin button and drag-to-pull-out behavior.
The hand menu worked well with the target text object in the near-field interaction range. However, I found that it becomes problematic for far-field interaction. Since I had to look at the target text at a distance and the hand menu in the near field, my eyes had to continuously switch focus/depth. It quickly caused eye strain.
A solution could be attaching the text-related menus to the target text object. However, for the text far away, to make the menu usable with a pointer (hand ray or gaze cursor in HoloLens 1), the size of the menu should be very large. It will visually overwhelm the target text which should be the hero object of the experience. (Content is King, Content before Chrome — sounds familiar? :)) I also wanted to keep the menu directly interactable with my fingers when I interact with the text objects in the near-field.
In-Between Menu
My solution was to place the text property menus between the target object and my eyes(headset). Of course, closer to the target text object, since the goal is to minimize the focus/depth switching. After playing with the values, I made the menu placed 30% far from the target object. (70% far from my eyes) This allowed me to directly interact with the menus easily with the text objects in the near-field. This menu positioning/sizing also works well with HoloLens 1st gen’s smaller FOV size.
Fortunately, one of the MRTK’s awesome Solvers called ‘InBetween’ provides the positioning mechanism. Using InBetween Solver, you can easily position something between two objects and specify the distance % ratio between those two.
To maintain the target size of the menu regardless of the distance, I used MRTK’s ‘ConstantViewSize’ Solver. As you can see in the video, the menu automatically scales up when it moves far and scales down when it is moving into the near-interaction range. This makes the menu easily interactable with both direct finger press and hand ray pointer + air-tap/pinch.
UX Elements: Main Menu for the global features
For the buttons that provide global(non-object specific) features, I left them in the hand menu. As described before, you can easily grab and pull out to make the menu world-locked.
The main menu includes:
- New Text | Clear Scene | Save & Load Scene
- Spatial Mapping | Mesh Visualization | Snap to Surface
- Physics: Rigid Body | Kinematic | Slider UI for gravity(force)
- Grab & Duplicate Toggle
- Random Composition
- About
Below is the latest design iteration result, a compact version with automatic world-lock on hand drop. The world-locked menu makes it easy to interact with multiple UI controls.
UX Elements: Annotation
Being able to place text objects in the physical space means that you can annotate physical objects. To help visual connection to the target object I have added an optional line and sphere component. By simply grabbing and moving the text object(anchor) and the sphere(pivot), you can easily create annotations with a connected line between the text and the sphere. For this feature, I used MRTK’s Tooltip component.
UX Elements: Spatial Mapping
Spatial Mapping is one of the most exciting features of HoloLens. With Spatial Mapping, you can make holographic objects interact with the real-life environment. In the original version of the Type In Space app, I used the Gaze cursor to move the text on the physical surface. In the new version of HoloLens 2, I was able to use the hand ray’s endpoint to attach the text and follow the surface. The placing and moving states are toggled by air tap.
To use Spatial Mapping in MRTK, you can simply turn on the Spatial Awareness feature. This gives you the spatial mesh of the environment. Surface Magnetism solver is a convenient utility that can make any objects snap to the surface.
UX Elements: Physics
With Spatial Mapping, I have physical surfaces that I can use. How about applying gravity force to make the type fall or fly and let them collide with the physical environment? I already had a simple ‘gravity’ option in the original version. In the new version, I have added a slider so that the user can control the amount of force as well as the direction.
UX Elements: Text Input with keyboard and speech
HoloLens 2’s direct interaction with holograms dramatically improved the text input experience too. Just like using the physical keyboard, you can use your fingers to type in the text with the holographic keyboard. Of course, you can still use the speech input for dictation. The system keyboard has a built-in speech dictation input button. MRTK provides great examples of using system keyboard and speech input.
Below is an example of using Dictation input. The dictation service is provided by Windows.
UX Elements: Grab & Duplicate
The original version had a simple duplicate feature that allows you to quickly create a new text with the same text properties. To make the duplicated text visible, I made the duplicated text placed with some offset position. This created an interesting visual effect with an array of instances.
I have modified this feature to make it duplicatable by simply grabbing and moving with the hands which is much easier than keep pressing the duplicate button. World-locked trails of holographic text look gorgeous.
Supporting HoloLens 1st gen and Windows Mixed Reality VR devices
One of the benefits of using MRTK is cross-platform support. MRTK’s interaction building blocks and UI components support various types of input such as HoloLens 1’s GGV (Gaze, Gesture, Voice) and Windows Mixed Reality immersive headset’s motion controller.
The text properties menu (In-Between menu) works well with GGV input without any modification. Since it is always displayed on the right side of the currently selected text object, even with HoloLens 1’s smaller FOV, it works well. Interacting with the motion controller’s pointer in VR also works well.
Since I cannot use hand-tracking for the hand menu, I made the Main menu a floating tag-along menu with a toggle pin/unpin option. Other than this small modification, I was able to publish the app for both HoloLens and HoloLens 2 with a single Unity project.
English 
Korean