Q&A in Webinar

QUESTIONS ABOUT CHROMA KEY

Q:Can early access users use Chroma Key features through the APIs?

A: Chroma Key is part of Varjo’s Mixed Reality C++ API that was shared with all the Varjo users as part of the software release v2.2. The API provides functionality for managing chroma key configurations and to enable/disable the feature for specific client layers. ChromaKeyTool is an experimental helper tool for setting up chroma key configurations. It can be provided for selected early access partners. Unreal and Unity plugins and Varjo UI support will be added in the following releases. API documentation is available here.

Q: Do the camera images pass through the computer before they get to the display, or is all chroma-keying and warping taking place on the headset itself?

A: Chroma keying happens in our video-see-through post-processing and compositor pipeline on PC. The path is the same regardless of whether the chroma key feature is enabled or not.

Q:Can you explain the “green flakes” seen in the Chroma Key demo?

A:As a technique, chroma keying requires a controlled environment and lighting. “Green flakes” seen in the live demo are due to the natural lighting in the room and XR-1 cameras set to the automatic mode trying to adapt to the varying conditions. This leads to some false metering when the camera sees only the green screen surface. Optimal set up should have dedicated lights to create an even light-scape, fine tuned parameters for the environment and camera settings locked to prevent white balance changing over time. We are working on improvements to our chroma key algorithm for automatically adjusting it to the camera settings.

Q: Can I use my own code here for creative control of the image blending?

A: This is something we are currently working on, stay tuned for future updates.

Q:Will Varjo’s headsets offer a 360 video player that takes advantage of Chroma Key by compositing the viewer’s body into the live-action experience? If so, will it support stereo or only mono?

A:While Varjo doesn’t provide a 360 video player, it’s possible to create one using the C++ API or any of the supported engines, such as Unity or Unreal. Alternatively, one can use a video player from the Steam store.

Q:Is there a method for adjusting chroma settings (instead of random nudging)?

A:Varjo will provide a user interface for adjusting the settings in the upcoming software updates. That will be based on Varjo’s experimental ChromaKeyTool utility that allows the user to pick base parameters from the camera image by pointing the headset towards the green screen. Parameters can then be fine-tuned manually.

Q:Can you explain the Chroma Key occlusion? Is there also occlusion for real hands and virtual objects?

A: Chroma key masking is based on color matching – it does not recognize real hands in any way. The occlusion is based on the fact that human skin color is a different color than the chroma backdrop, which is usually green or blue as it is a color that is furthest away from the human skin tone.

Q:Are the real-world included objects projected in mono or stereo in the chroma keyed environment? I understand the cameras are stereo but what about the image?

A:Everything is projected in stereo. The high-resolution focus area has a higher resolution chroma key mask. If eye tracking is enabled and calibrated, that high-resolution area will also follow your gaze.

Q:What is the difference between Depth masking vs. Stencil masking vs. Chroma?

A: All of the above mentioned can be used to solve the same problem. Depth masking requires that VR is positioned in the physical free space (within room walls), and hands occlude VR content when they are closer to the HMD than the VR content. For hand occlusion, chroma keying usually provides higher resolution and more stable silhouettes. Stencil masking (using a model of the real environment in VR rendering where alpha is rendered as 0) allows the occlusion of static objects, e.g., a plane cockpit, but does not solve hand occlusion when hands are brought over VR content (such as scenery seen through cockpit windows). For more details, please have a look at our blog post discussing the masking issue.

Q:What’s the max distance between HMD and Chroma?

A:There are no limitations to the distance between HMD and Chroma. The only thing that is relevant in a set-up that is that the camera needs to see the color.

QUESTIONS ABOUT VISUAL MARKERS

Q:What’s the longest distance to track markers?

A: Below is the list of different sized markers and their recommended maximum distance from the headset when the marker is being tracked:

• 25mm (~1”) Best in distances up to 0.5 meter (1.5 feet)
• 50mm (~2”) Best in distances up to 1 meter (3 feet)
• 150mm (~6”) Best in distances up to 3 meters (9 feet)

Q: Can I use custom markers? For example, a giant 4-meter high marker. Can you use different sized markers at the same time?:

A:The custom markers are not supported currently.

Q: Can you use the Vive pucks instead of print markers?

A: Vive pucks can be used through OpenVR API.

Q:Will there be a possibility of using the hand to interact with a marker that has a virtual screen put on top of it or the hand will interfere with the marker?

A:The marker keeps its last known location when obstructed. The timeout of an inactive marker can be changed to a desired length of time using the native API.

QUESTIONS ABOUT BOTH FEATURE？

Q:Can this be integrated with OpenVR?

A:Chroma keying is supported only in Varjo native C++ API at the moment. However, using our experimental ChromaKeyTool utility, we can force global override mode that will work in every application, no matter if it is OpenVR, Unity, Unreal, or Varjo native app. It will work even if it is a pure VR application without any mixed reality support. Visual Markers are accessible through the native C++ API only at the moment.

Q:Was there hand occlusion with your marker tracker combined with the green screen?

A: At the beginning of the Visual Marker demo (see timestamp 35:20 on webinar recording), Varjo’s co-founder Urho Konttori brought a PacMan Arcade machine against the green canvas. Then hand occlusion was disabled for showing other marker tracked objects in a non-green environment.

Q: Can you describe the interaction with the built-in eye-tracking?

A: Chroma key mask is evaluated on a higher resolution in the focus area. Our foveated video-see-through follows the user’s gaze and if the eye-tracking is enabled, the user will always see the sharper mask where the gaze is pointed. This enhances the visual experience significantly.

Q: What is the cause of the slight “swimming” visible between virtual and real objects? Are the latencies of the virtual images and camera images equal?

A: Typically VST latency is lower than VR latency; however, we correct this by doing positional and rotational timewarp to align both views. The tracking system (e.g., SteamVR) may introduce drifting that is more apparent in MR scenes than full VR scenes; however, one solution to this problem is to use Visual Markers, which by definition, align VR positioning with the visual VST image.

QUESTIONS ABOUT XR-1

Q: If I have two XR-1 headsets, I would love to feed the camera from one headset to the display of the other, and vice-versa. Does your system have this kind of flexibility?

A: While we do not have such a feature built-in, a client application can read the VST video stream and send it over the network to another client, who then renders this as VR content. It should be noted that in this case, the latency would be much higher.

Q: How is the latency from front-facing cameras to rendering? Is it dizzying if you move your head around? Is there some sort of re-projection?

A: We are achieving 15ms photon-to-photon latency when measured from the end-of-exposure-time of VST cameras to photons coming out from the displays. From camera to rendering, the latency is lower at roughly 9ms. This latency is low enough for most people not to be able to notice at all; however, we further reduce the apparent latency by performing render-time rotational timewarp to eliminate lag originating from turning your head.

Q: Are your cameras capturing the real world in stereo or mono?

A: The cameras are capturing the real world in stereo.

Q: Is there a hand-tracking option in XR-1 for seamless interaction with virtual content and actual reality?

A: Varjo’s VR-2 Pro features built-in hand-tracking technology from Ultraleap and with XR-1 you can add Leap Motion module separately yourself in front of the XR-1.

QUESTIONS ABOUT FUTURE OF THE PRODUCTS

Q: Do you have a wireless version coming in the future?

A:As all product companies, we are always developing new things in our labs, but we do not comment on future products ahead of time.

Q:Are Varjo’s libraries stand-alone for ensuring future applications to be compatible not only with the next Varjo generation but also with other headsets?

A:Varjo APIs have been made to be optimal for Varjo products. We do not comment on future developments, be them hardware or software-related.

Presentation：

Urho Konttori

Co-founder & Chief Product Officer at Varjo

Jussi Mäkinen

Chief Marketing Officer at Varjo

Ville Timonen

Mixed Reality Lead at Varjo

Saku Tiainen

Mixed Reality Developer at Varjo

Source:Vajor