Volkswagen Group is collaborating with Microsoft to enable the HoloLens 2 mixed reality headset to be used in moving vehicles for the first time. The new ‘moving platform mode for HoloLens 2 creates potential for the AR technology to be used in new ways, such as training drivers to handle challenging road conditions, for example, or creating new user experiences for autonomous vehicles.
HoloLens uses two main types of sensors that measure its motion — visible light cameras and an inertial measurement unit, or IMU, that gauges acceleration and rotation speed. Together, the sensors mimic how humans see and move through the world.
The Volkswagen researchers established a bidirectional data connection between the vehicle and the HoloLens in order to display and control real-time information from the car.
The team implemented several demo use cases investigating how virtual interfaces could enhance the interior of future vehicles.
Volkswagen introduced an augmented reality head-up display in its ID. family of electric cars in 2020 that projects navigation arrows, lane markings and other information onto the environment in front of the cars.
Dr. Andro Kleen, head of the data science team at Volkswagen Group Innovation, says: “We think mixed reality information is the most intuitive information we could provide to enhance our customers’ user experiences. Because what you see there, and what you need to process, is very close to what humans normally see and process. It’s not so abstract.”
Marc Pollefeys, director of Microsoft’s Mixed Reality and AI Lab in Zurich, Switzerland, adds: “We had extensive discussions. They presented their use cases and what they were hoping to enable. They were eager to work with us to find a solution and be able to use HoloLens in those situations.”
Michael Wittkämper, augmented reality expert at Volkswagen, comments: “We connected a positioning system that tracks the location of the vehicle. This way we were able to also place 3D elements such as information on point of interests outside of the car. This opens up completely new possibilities to not only display holograms within the driver’s forward-facing field of view, but also wherever the user wearing the glasses is looking.”
HoloLens’ moving platform feature is currently supported for use on large ships, and Microsoft plans to further refine it for use in elevators, trains, cars and other moving environments.
