Research into potential automotive applications for biometrics ─ facial and retina scans or fingerprints ─ is increasing, despite rising concerns about privacy and data security, and outcries from civil liberties advocates. A growing number of OEMs, Tier 1s and startups are striving to add facial recognition capability to driver monitoring systems (DMS). Interior cameras are becoming commonplace as DMS are needed for advanced safety systems and autonomous driving. Biometrics are being added to provide more functionality, though suppliers must deal with the privacy issues that surround these technologies.

The retina and iris of the human eye offer various opportunities for personal identification, as their internal patterns are essentially unique from person to person. Facial recognition also can provide the basis for personalized offerings for passengers as well as drivers. Byton, a Chinese startup that’s planning to enter the U.S. luxury electric vehicle market next year, is making facial recognition an important element in a user interface (top) that includes a 48-inch screen. Its vehicles will customize various parameters to every person in the vehicle.

Byton is betting that buyers will decide the “wow” factor outweighs any privacy concerns. “All our user experiences flow from personalization,” said Elizabeth Edwards, the company’s director of imaging technology programs. “We could do it with a simple personal identification number, but that’s not as cool as using facial recognition so customization is seamless when people enter the vehicle. Facial recognition is an important element in our design,” she told Autonomous Vehicle Engineering.

Hardware and processing requirements
When the industry transitions to SAE Level 3 and 4 autonomous driving, DMS systems will watch to see whether drivers are ready to take over when automated systems can’t handle the driving conditions. Tier 1s are already heavily involved in research to see how biometrics can be added efficiently. “For autonomy, driver monitoring and biometrics like facial recognition are very key aspects,” said Doua Vang, senior manager of Denso’s Elevate R&D Lab. “They’re how vehicles determine whether the user can capably take over the vehicle.”

Startups are also a part of this emerging field. Israel-based Adam Cogtec aims to augment DMS and facial recognition by gaining insight into the driver’s mental state. That is far more helpful than simply determining that eyes are open and hands are on the wheel. “Facial recognition does more than a conventional driver monitoring system; it’s better at detecting drowsiness or impairment,” said Carl Pickering, CEO at Adam Cogtec. “We add a software layer that sits on top of any driver monitoring system, looking at the driver’s cognitive state. We can determine whether the driver is cognitively impaired by using a visual stimulus that lets us see change in pupil dilation and eye movement.”

Adam Cogtec’s system requires a small LED that sits in the driver’s peripheral vision, flashing sporadically to see how the eyes respond to a change in light input. When the eye’s response strays below a baseline measurement, the driver isn’t mentally active even though they may be sitting in a good driving position.

Many technologists think handling social issues and developing use-case strategies will be more challenging than augmenting DMS with biometric technologies. Though facial recognition is still an emerging technology, implementing it doesn’t require the latest hardware. For example, conventional CMOS cameras meet sensor requirements. “We’re using fairly low-resolution systems compared to surveillance cameras,” Edwards noted. “We can use readily available automotive-grade cameras for facial recognition. We have five cameras in the cabin, one per occupant, with a redundant camera for the driver.”

Processing requirements in autos will be far less than those used in applications like airport surveillance systems, since automotive systems will only be comparing faces for known drivers whose information is on file. Software will be a critical aspect of any system design.

Occupant health monitoring
Major players are looking for intellectual property (IP) from major vendors and startups alike. The growing usage of open source software, coupled with increasing numbers of alliances, makes it important to look at the sources of all relevant technology. “This is an area that’s growing rapidly,” observed Vang. “We’re looking at a variety of companies from large to small. We need to look at the company’s IP portfolio, which is important in any fast-moving field. We want to make sure IP elements are not held by another company.”

Though facial recognition can be handled by most DMS cameras, high-end users may want more accuracy. In low lighting conditions or when the driver’s face is partially covered, conventional cameras may have trouble comparing new images with faces in the system. Infrared cameras can be added to improve accuracy.

“Standard RGB cameras have some limitations in low light, and large sunglasses can reduce the effectiveness of facial recognition,” Edwards explained. “For many of the corner cases, people can be identified with an optional monochromatic camera that can see through sunglasses. For base systems, we leverage the RGB camera as much as possible.”

To help justify the cost of DMS and biometric systems, developers are looking at additional tasks that can be handled by the systems. In fully autonomous (L4) vehicles such as robotaxis, cameras make video conferencing possible. In the nearer term, many suppliers are looking at ways to monitor the driver’s health conditions. Cameras can also prevent drivers from leaving kids or pets in cars.

“We are seeing the need or desire to monitor occupants of vehicles, not only for access or starting the vehicle, but also as a supplement to health monitoring,” said Zachary Bolton, head of systems and technology r&d for Continental’s Interior division. He noted that a vehicle could potentially notify a driver of an increased heart rate or spike in blood pressure that is concerning. Further integration of monitoring in automated driving systems will enable greater capability “to do more while we are in these vehicles,” he said.

Facial recognition could also be used to let drivers unlock doors. However, this application may not leverage installed cameras. Interior cameras will focus on drivers, while exterior sensors generally take a wide view for 360-degree coverage. Facial recognition is only possible “when you have a clear visual of a face from a camera system,” Bolton said. “Unlocking doors is also possible if the vehicle is equipped with a surround view or rear back-up camera system, but some of these systems do not have the optimum orientation or field of view for facial recognition.”

Big Brother is watching
Experts maintain that privacy issues and security will be critical elements in any rollout. Security will be critical, since biometric data serves as the “keys to the car” as well as access to sensitive apps loaded onto the vehicle. Automotive systems can’t be fooled into giving away such valuable access. “We want to make sure systems can’t be spoofed,” asserted Denso’s Vang. “You don’t want someone to be able to hold up a picture or get a copy of a fingerprint and gain access to the vehicle. The biggest piece for our customers is the system’s robustness in a vehicle setting.”

Many privacy advocates are particularly leery of facial recognition, which has been outlawed in San Francisco over privacy concerns and is driving regulatory oversight in related areas. The Illinois Biometric Information Privacy Act requires that companies that collect their employees’ data must first notify them about how the data will be used and stored and obtain their consent. Texas and Washington have similar laws but only in Illinois do those who gave up this information have the private right to sue ─ whether or not they feel their data was misused or if they suffered damages. Florida and New York are considering similar laws.

While automakers promise that data collected in their vehicles will typically stay within them, there’s growing concern that data collected will be sent to the cloud, where anything can happen. The auto industry will probably follow the lead of early adopters. “We’re watching industries outside of automotive,” Vang said. “With facial recognition, there’s concern about what information users want to give up. We have to see where that goes; there’s a lot of concern about Big Brother watching.”

Edwards at Byton said that that although her company’s data will be encrypted and protected, privacy remains a big factor. However, she noted that it’s difficult to use stolen biometric data in programs that weren’t developed using the same techniques. “If two different facial recognition algorithms were developed by different people, you can’t use the same coefficients with the other software,” she said. “The data is only useful to the algorithm written to use it.”