Let’s dive deeper into the key insights from the webinar–Vajor VR

Implementing chroma key enables mixed reality scenarios where you can place real-life physical objects within a fully immersive virtual reality environment. You can also effortlessly switch between virtual, mixed, and real scenarios, enabling new levels of collaboration between you and your colleagues.

Chroma keying – seamless blending of physical objects within virtual scenery

Chroma keying is a technology that the broadcasting and film industry have been using for decades to blend virtual content with the surrounding reality. While this technology itself isn’t new, applying it in a real-time mixed reality setting is novel – especially with the photorealistic quality that is only achieved with Varjo’s mixed reality headset, XR-1.

What chroma keying technology allows the users to do is assign any one color of choice as the canvas on which the virtual reality scenario will appear. Users can then choose different colors for different outputs. For example, you can select the color blue for all the dynamic movements, while using green to showcase the more static objects in a mixed reality setting. Varjo currently supports up to four different colors at the same time, which are also adjustable. In practice, this means users can set chroma and thresholds for each of the colors separately via the API or the command line tool. This way, you can pick e.g. normal floor and ceiling color as one and then have a proper green screen as another.

Chroma key and visual marker tracking give users a true mixed reality experience where real-world elements are blended within the virtual environment, enabling collaboration in both worlds as naturally as sitting together in your brick-and-mortar office.

Visual markers for tracking objects

Visual markers are a few decades-old technology, and are typically used in augmented reality applications. At Varjo, we’re using existing libraries that let you track markers, enabling you to place and anchor virtual reality interactions on top of them or next to them.  By using visual markers with the XR-1, professional users can place a virtual object in a pixel-perfect, fixed location. This accuracy makes it easy to replace real-life physical objects – such as replicas or prototype parts – with interactive virtual content in the mixed reality setting.

Varjo’s easy-access and convenient API gives you access to the identity of the marker. Our markers support individual tracking of up to 1,000 objects, without the need to use active controllers or tracking pucks. This support makes building interactions on top of them as easy as possible. Varjo currently provides this implementation for Unreal, Unity, and native SDK for XR-1 Developer Edition, which saves users from utilizing the lower-level libraries.

Varjo’s XR-1 currently supports three different sizes of markers

• 25mm marker, with an active area of up to 1 meter (3 feet). Best usability within 0.5 meters. These small markers are excellent for tracking something in your hand.
• 50mm marker, with an active area of up to 2 meters (6.5 feet). Best usability within 1 meters. These are the most stable options for interactions within arms reach.
• 150mm markers, with an active area of up to 5 meters (16 feet). Best usability within 2 meters for dynamic movement. Beyond 3 meters, we recommend setting a static position or using multiple markers to stabilise.

A game-changer for design work

For designers and engineers, combining chroma key with visual markers is a game-changer. With the latest update, XR-1 users are now able to create digital twins of the real-life objects they want to include in a mixed reality scenery.

Users can 3D print any object with the colors that Varjo’s chroma markers support. The combination of chroma keying technology and visual markers enables users to hold the 3D printed objects, like phones or other gadgets, in their hand with perfect occlusion and “dress” them instantly to look like a material-finished product.

To make designing even more intuitive and life-like, Varjo’s light model enables accurate reflections of the real world around the virtual objects.  The object follows the chroma visual markers perfectly through Varjo’s XR-1, making it possible to visualize and re-design prototypes in seconds, without the need to actually build a physical prototype.

Track objects accurately using visual markers

Our latest software release introduces visual marker-based object tracking. Using visual markers and the XR-1, professionals can anchor virtual objects exactly where they want them in their surroundings. This allows the exact positioning of virtual displays, controls or other objects to be fixed in the reality around you. Varjo markers support individual tracking of up to 1,000 objects, without the need to use active controllers or tracking pucks.

Full body presence by a perfect occlusion of the virtual

Chroma can place individual physical things in the virtual experience – we are talking about tools, equipment, control sticks, etc. To understand how this works, let’s think about pilots in training – flying a plane is much more than just knowing the ins and outs of the cockpit. To fly a real aircraft, pilots must develop non-visual muscle memory of the location and operation of certain controls, such as a throttle, in relation to their position in the cockpit. This is critical know-how for any pilot and flight mission.

With chroma key markers, you can still rely on the cost-efficient virtual scenery currently used in training simulations, but can now mix physical objects in the scenery as well to bring the experience as close as possible to a real-life simulation experience. For example, you can use chroma key markers to virtualize a physical, real-life control stick in the mixed reality setting of the simulation. This means that the pilot-in-training can experience the tactile sensation of using the control stick, developing the much-needed muscle memory, while operating a fully simulated aircraft.

Chroma keying enhanced virtual scenery provides the training pilot with a sensation of immersion like never before, even including scenarios where one needs side-by-side visibility of other pilots or aircraft within the simulation.

Perfect connection between people through virtual interaction

With marker tracking, you can share objects with other users inside the collaborative virtual reality spaces. You can also share the coordinate system with other users while working on tasks – the coordinate system stays the same for everyone in the scenery as markers are “glued” to certain points.

To support the collaborative setup, users can set multiple markers in place to make the coordinate system more reliable. Marker tracking also enables professional users to scale up the coordinate space significantly with accuracy and precision of millimeters.

With chroma key and marker tracking, you can share a space together with other users and see each other physically at the same time, in real time. It’s not only a new way of working together more efficiently, but it also enables you to express yourself with body language and movement, just as you would in real life.

When collaborating over design settings with others through chroma key in the XR-1, you are able to experience the real deal. Compared to the isolated nature of virtual reality, mixed reality with chroma key and visual markers offers a ground-breaking shared experience where you can interact naturally with your colleagues and clients on any projects, reviewing and discussing in real-time.

Presentation：

Urho Konttori

Co-founder & Chief Product Officer at Varjo

Jussi Mäkinen

Chief Marketing Officer at Varjo

Ville Timonen

Mixed Reality Lead at Varjo

Saku Tiainen

Mixed Reality Developer at Varjo

Source:Vajor