For many drivers, trying to follow digital route guidance in an unfamiliar place amid intense rush-hour traffic can cause sensory overload, high anxiety—and compromise safety. New technology recently tested in advanced prototype form by Automotive Engineering is aimed at mitigating and even preventing such scenarios.

We were among the first media to drive a vehicle equipped with a new Augmented Reality Head-Up Display (AR-HUD) system developed by Continental. Slated for production in 2017, the prototype AR-HUD system we sampled on the roads and autobahn between Frankfurt and Darmstadt, Germany, was fitted to a Kia K9 sedan, specially configured by Continental’s Interior Division engineers for demonstrating the system to OEM customers. While still in development, the system we evaluated was significantly more useful, capable, and engaging than Conti’s current-production HUDs also on hand in BMW and Mercedes cars for comparison.

As its name indicates, the AR system augments the conventional windshield-type HUD’s display data—i.e., vehicle road speed, local speed limit, no-passing zones, and basic navigation aids—with real-time and highly accurate graphic portrayals of the vehicle’s turn-by-turn route guidance, lane-departure warning (LDW) and adaptive cruise control (ACC) functions. These were selected from 150 use cases provided by Continental researchers as most relevant for the demo vehicle, from among the company’s advanced driver assistance systems (ADAS) suite.

AR enables the vehicle to “show” the driver what the ADAS sensors, GPS, etc., “see” and what they are doing. For example, while approaching your next turn, a sequence of arrowheads (Ʌ Ʌ Ʌ or > > >) appears in your field of vision, seemingly laid on top of the road surface and leading you to and through the turn, then disappearing until the next prescribed route change. We found this feature to be intuitive and well executed in the test vehicle, keeping our eyes on the direction of travel and replacing the need to look back and forth from a navi screen to the road ahead.

To augment the LDW’s buzzing seat-cushion alert, if you drift across the road’s center line or onto the shoulder, a row of red “cat eyes” appears to hover precisely over the violated line, then disappear (along with the haptic feedback) as you steer back into your lane. Likewise when the ACC is activated, the driver sees a virtual arc that appears to cradle the rear of the vehicle directly ahead of you, indicating that the ADAS camera and radar are "painting" it at a defined distance and pacing its speed. As the gap between the vehicles changes, a sequence of blue bars illustrates your relative safe following distance. A red triangle signals imminent danger.

To see a video of the ACC function: http://youtu.be/Xn_VF0chwsM. Lane Departure Warning function: http://youtu.be/TRNGkZs3xuQ. Turn-by-turn route guidance: http://youtu.be/X7T67Gdpp1c.

Automotive Engineering found the LDW and ACC graphics to be instructional—an accurate visual portrayal of the car’s dynamic safety electronics doing their job in real time. However, during complex traffic situations where the vehicle was receiving multiple sensor inputs and driver instructions—such as swerving to avoid an object on the roadside (triggering the red cat’s eyes) while approaching an intersection turn (arrowheads sequencing) with the ACC "painting" the car ahead with the virtual arc)—the HUD info area of the windshield becomes pretty busy. Conti engineers said they’re keen to avoid putting too much information in front of the driver and thus potentially distracting him. But clearly this system has the potential to be minimalist, displaying only the essential information for a given traffic or road situation.

AR in various forms is a key technique used in cinema-quality digital video projection, but the technology wasn’t easy to adopt for the far more extreme automotive use, noted Pablo Richter, Ph.D., the physicist directing Continental’s advanced HUD programs. He respectfully cited the early (1993) research done by AR pioneer Paul Milgram of the University of Toronto’s Industrial Engineering Dept. (see http://etclab.mie.utoronto.ca/publication/1994/Milgram_Takemura_SPIE1994.pdf).

“AR is the biggest trend for the future of head-up displays,” said Eelco Spoelder, head of Continental’s Instrumentation & Driver HMI business unit. The technology, he explained, is the next major step in creating a “holistic HMI (human-machine interface) approach” for improving driver awareness, confidence, and safety. It will also be an enabler for Continental’s ongoing work in sensor-fusion technologies, as the company integrates its expanding ADAS into automated and eventually autonomous vehicles.

Spoelder told Automotive Engineering that HUDs will be a “priority information display”—effectively evolving into an operating system in combination with instrument clusters, which are "not going away,” he said. Spoelder expects the cluster to transition to a “secondary” priority over time.

Continental is forecasting 1.5 million HUD unit sales in 2014, increasing to 5 million units sold by 2018. The supplier entered HUD production in 2003, and has been developing the AR-HUD since 2012.

Two "layers" and a DMD

Richter and his colleague Dr. Thorsten Kern, Manager of the HUD Competence Center at Conti’s Babenhausen interior-electronics development complex, explained that the various data and graphics are arranged in two picture levels. These “layers” (imagine two vertical planes, one closer than the other) are created by two different image generation units. One unit consists of a thin-film transistor, a smaller optical mirror, and LED array; it produces an image “near” to the viewer (in the prototype this was the base data such as road speed). Its field of vision of 5°x 1° (corresponding to 210 x 42 mm/8.2 x 1.65 in) has a projection distance of 2.4 m (7.8 ft)—roughly the equivalent of a conventional HUD.

The second image generator in the AR-HUD produces the “remote” (augmentation) layer, which in the prototype supports the ADAS-related graphics that “appear” to be on the road surface ahead of the vehicle. Unveiled at the 2013 IAA show, Conti's remote-image generator is a source of pride among Richter’s team. It’s based on a digital micro-mirror device (DMD)—an optical semiconductor supplied by Texas Instruments.

The DMD contains a matrix of several hundred thousand tiny mirrors, each of which can be electrostatically tilted. The micro-mirror matrix is alternately lit by three colored (red, blue, green) LEDs in rapid, timed sequence. The collimation, or parallel direction, of the tri-color light takes place through a tilted mirror with a color-filter (dichroic mirror) function. Depending on the color, these mirrors either allow the light to pass through or reflect it, Richter noted.

All micro-mirrors of a color are tilted synchronously with the color currently lit, so that they reflect the incoming light through a lens and thus depict the color on a focusing screen as individual pixels. This happens at the same time for all three colors. The human eye "averages" all three color images on the focusing screen and gives the impression of a full-color picture.

According to Richter, the AR generator has a longer projection distance of 7.5 m (22.5 ft) in front of the driver, compared with conventional HUDs. The emitting area of the optical system for the augmentation is nearly A4-paper size. This produces a considerable field of vision of 10° x 4.8° in the augmentation level—the equivalent of a viewing area 130 x 63 cm (51 x 25 in) in the driver’s direct field of vision.

The AR-HUD offers OEMs a 2° deviation angle—the angle the driver must lower his eyes from horizontal in order to view the HUD data. By comparison, a conventional HUD’s deviation angle is typically 6°. The prototype AR display has a luminance adapted to the ambient lighting of over 10,000 cd/m², which Dr. Kern said makes it easy legible in nearly all ambient-light conditions. (Due to two days of overcast weather in Germany, we were not able to test the AR-HUD in bright sunlight or at night.)

Continental uses precision-manufacturing and quality control processes, including proprietary coating technology, to produce its HUD mirrors at the Babenhausen factory, where it also assembles the complete HUD units for a variety of OEMs including Audi, BMW, and Mercedes. Key optical components of the AR-HUD are expected to enter series production in 2016 as the "DMD-HUD." It will offer a larger field of vision in a windshield HUD, before the AR arrives.

Meet the Creator

Using simulations and subject tests, the Conti AR-HUD team found that most drivers are comfortable when the augmentation begins approximately 18 to 20 m (59 to 66 ft) in front of the vehicle and continues up to a distance of around 100 m (328 ft), depending on route. Their research was vital for developing another key innovation, the AR-Creator—a control unit/signal processor that receives a stream of sensor data then arranges the graphic information for the driver’s view. At its heart is an automotive-spec, 1.2 GHz quad-core processor.

Harald Dittmann, Systems Architecture and Integration engineer, praised his team’s work in managing time delays in the data acquisition and fusion from three sources—the front-mounted radar, a CMOS mono camera integrated into the interior rear-view mirror mount (range-optimized for lane detection at 4 to 60 m), and Continental's eHorizon, which provides the map data translation from the navi system. The vehicle's position is shown on a digital map using GPS technology. If the GPS signal is lost, the last known position is calculated using vehicle dynamics data. In the demo vehicle, the data for this is provided by a chassis-mounted gyroscopic sensor module.

“Ten to twenty milliseconds is a big time gap,” Dittman asserted, noting the mathematical challenge the time delays represented in developing the unit. “To correct, we make a prediction based on calculations using a Kalman filter”—an algorithm that uses a series of measurements observed over time that produces estimates of unknown variables. Dittmann noted that his team has investigated using a stereo camera, which would provide better side-to-side views of the vehicle on the road.

The AR-Creator (see accompanying Systems Overview and Signal Processing diagrams) also uses the fused data to calculate the road geometry as it looks from the driver's position. Dittmann explained that the driver's eye position is established according to the "eye box"—a 160 x 60 mm (6.3x 2.3 in) rectangular space around the driver’s head that determines the correct viewing baseline, and we adjusted for this in the demo vehicle before our test drives. Conti engineers said when the system enters production, an interior camera may be used to detect the driver’s eye position and track location of the eye box.

For OEM vehicle integrators, two main challenges in incorporating the AR-HUD are establishing the proper look-down angle, and packaging the unit, which in current prototype form consumes 13 L, in the IP/dash assembly, said engineers. Eelco Spoelder said that while he doesn’t expect an AR-HUD to have a volume of less than 4 L (similar to a conventional HUD), an 11-L unit may be feasible in series production.