The auto industry is under increasing pressure to prove that autonomous vehicles (AV) can be safe. But before AV manufacturers can begin to establish the necessary benchmarks for safety, the industry needs a common framework for describing the infinite scenarios and roadway variabilities that a self-driving vehicle might encounter. Foretellix, a Tel Aviv-based software company, is fulfilling that need with its Measurable Scenario Description Language (M-SDL), a machine- and human-readable programming language for AV validation.
The open-source language, described in a 120-page technical document, has been downloaded by hundreds of engineers. Companies using M-SDL in pilot projects include Volkswagen. Foretify, Foretellix’s software product, compiles high-level scenarios written in M-SDL into specific scenarios that are run via an application programming interface (API) in an AV simulation tool or other test environment. Foretify is either licensed per developer seat or on a simulation runtime basis.
Because the scenario is delineated in a standardized fashion, engineers can confirm that disparate simulators are testing the same well-defined scenarios. The parameters could include vehicle speed, the distance between vehicles, weather and thousands of other variables. In one scenario (see gallery) called “cut-in,” Car1 begins by passing the DUT (device under test) car. The DUT car drives at a speed between 30 and 70 kph. Meanwhile, Car1 starts in a position that’s 5 to 100 meters behind the DUT car, but cuts in front to a location that’s 5 to 15 meters ahead of the DUT. That’s just part of one scenario written in M-SDL.
“Autonomous vehicles need to do the right thing not 99 percent of the time, but 99.9999 percent of the time,” said Ziv Binyamini, chief executive and co-founder of Foretillix. Discovering the rare but potentially catastrophic edge cases is the job of AV safety engineers and regulators, a process that requires a robust validation process. Historically, testing has relied on racking up millions of miles of driving and hoping that your validation vehicle would eventually encounter the most critical situations.
“You didn’t know where you were in the process or when you were done,” Binyamini said. And there was no common language for all the stakeholders to confirm they were working on the same scenario. M-SDL represents a shift in AV-safety validation from millions of miles driven (on actual roads, track testing and simulation) to the quality of coverage. M-SDL is an aspect-oriented programming language, an extension of an object-oriented approach.
Binyamini said it bears some resemblance to SystemVerilog, Python and C++. However, the closest match to M-SDL is the e-language developed in 1992 by Foretellix co-founder Yoav Hollander to automate functional verification of hardware, specifically semiconductors. It turns out that searching for the infinite possible flaws in a silicon chip and the limitless shortcomings of an AV algorithm are similarly daunting challenges, both of which require automation to conquer.
Software-generated, five-star safety ratings
Safety engineers, commonly statisticians, start with a verification plan that’s shared with the entire team working on scenarios. M-SDL is human-readable, so it can be understood by non-programmers on the team. “AV engineers tell us that the formalism and structure of the language by itself, even without automation, is very useful,” Binyamini said.
Foretify generates the tests as well as the so-called monitors that record the performance of the vehicles and scenario coverage, which are measured against the conditions outlined in the verification plan. The software-based monitors identify whether the scenario being evaluated occurred during the drive. If the scenario did occur, the monitor also documents which parameter values were encountered, such as an on-ramp or a dirt road. It’s like an infinite, automated, qualitative checklist.
Scenarios also can be coded using official data from past AV accidents. Then, the scenario can be methodically tested against AV algorithms and sensor configurations to prove that any shortcomings caused by the accident (under the same or a wide variety of similar conditions) are no longer a threat. Binyamini said he hopes that safety agencies like NHTSA and the Euro NCAP will one day use M-SDL to delineate the level of coverage required for autonomous vehicles to be deemed safe.
A prescribed set of scenarios would be used to test an AV company’s algorithms the same way vehicles today are crashed into a wall to determine how sheetmetal and vehicle substructures handle impact to produce a five-star rating. The code can find the locations within maps, such as a roundabout or three-lane highway, to run simulation tests that are best suited to a given scenario.
“By defining the common language of a scenario, we allow different organizations to share and exchange data,” Binyamini said. “So, for the same cut-in scenario, you could evaluate with three different simulators and three different maps. One at an urban junction, one semi-urban and another one on the highway. Somebody wrote just 30 lines of code and the same scenario is automatically landing on all of these different simulators and all of these different maps.”
Safety freak puppet master
The M-SDL code also can place the ego vehicle at a precise location, turn sensors on and off, modify the behavior of other vehicles, change the weather or lighting conditions and allow random acts, such as a rock thrown into the roadway. “We are the puppeteers that control all the actors,” Binyamini said. The code might specify, for example, a set of simulations in which another vehicle cuts in from the right, but only after the camera facing in the same direction experiences a failure – a good test for adequate sensor redundancy.
Perhaps more importantly, as edge cases within various constrained scenarios are discovered, the coder adds them to the library to produce an increasingly refined scenario. Each scenario, enshrined in code, also is a building block that can be combined with other scenarios. For example, a cut-in scenario might be combined with another scenario describing a merge event. An AV needs to handle both at once.
Moreover, the industry can use M-SDL to systematically raise the bar for safety over time, even if there’s no way to capture every possible threat. “Our system has mathematics under the hood. We generate randomness but within the constraints,” Binyamini said, referring to the concept as “infinity minus.”Continue reading »