Cgangs V3D Produces AR Performance Effect

Billboard is an American music magazine founded in 1894.  It contains various music contents and billboards. Within which the most important charts are Billboard Hot 100 (Singles Ranking) and Billboard 200 (Album Ranking).

At the 2018 Billboard Music Awards in Las Vegas, Madonna collaborated with rising star Maluma, performed her comeback single “Medellin.” During the show, Madonna danced with a series of AR-characters of her own image, including an agent, a dance teacher, a bride, and a musician. It’s all done with the Unreal engine and 3D capture, which come on and off at the right time, complementing Madonna’s performance and making the show more artistic.

The AR experience was created by Sequin, a new augmented reality company founded by members of The Future Group. It’s worth noting that this immersive effect production team has produced a series of cool mixed-reality weather effects for the Weather Channel, as well as for the virtual women’s group K/DA for the 2018 Hero League Global Finals. Co-founder Lawrence Jones said: “What’s new about this AR effect is that Madonna and Maluma are able to work with AR Madonna in perfect choreography. All productions, animations, and most of the lighting are done ahead of time. ”

A key aspect of making this kind of show is real-time camera tracking. The team used the Cgangs V3D Infinity Virtual System to layer objects above the Unreal engine, including animation, character generation, and real-time data. Cgangs V3D Infinity provides data from physical cameras to the Unreal engine so that everything in the real settings is combined with the digital copy model to ensure AR rendering in the right place at the right time.

Madonna herself was captured in three dimensions at a studio in London. Sequin then used the tools within Cgangs V3D Infinity to ensure that all data were transferred to the Unreal engine so that the live video matched exactly with the stage performance.

Madonna’s model was built using Intel studios’ 3D Presenter technology. Instead of shooting from a few different angles, it used more than 70 cameras to capture the entire scene at 360-degree angles. Real 3d figures were generated automatically through the calculation of camera signals, and a real-time 3D volume was continuously established, which was then repositioned and mapped according to the camera data.

3D Presenter supported the seamless import of figures into Cgangs V3D Infinity’s 3D virtual studio environment and their interaction with real and virtual objects in the scene. For example, shadows on a real desktop were also reflected on the virtual ground.

The Sequin team came to the show and adjusted the settings so they were fully integrated with the actual venue, including lighting, shadow, reflection, timing and position. Achieving such a goal was crucial to the Billboard music awards venue.  And it was a “super challenge” for Sequin and Madonna. Adequate stage time was critical to ensure that live and virtual events were properly arranged, which was no easy task for a show with a dozen performers and also needed time to rehearse.

This complex performance was driven by time codes. Everything from lighting, music and flame effects to graphics has its own time code trigger. Of course, the same goes for rendering. The show ended up using three AR enabled cameras: rocker arm, front-facing camera and wireless steadicam. Attendees at the ceremony were able to see Madonna’s AR stage effects on screens at the MGM grand garden arena.