When in recent years it became apparent to developers of autonomous vehicles (AVs) that they could never cumulatively acquire the equivalent of billions of miles of on-road validation necessary to create robust (read: markedly safer than human drivers) automated-driving systems, the emphasis shifted to simulation. By some estimates, developers now are counting on simulation for more than 95% of their entire validation and verification effort.

But could scores of AV simulation-software suppliers – not to mention deep-pockets AV players such as Waymo and any number of automakers and top-tier suppliers – be getting it all wrong? Michael DeKort, founder and CTO of AV simulation developer Dactle and holder of a broad and deep resume in the simulation discipline, thinks 95% simulation isn’t nearly sufficient. And the fundamental strategies and technologies on which nearly all simulation developers currently rely is flawed. 

DeKort insists the solution – what he generally refers to as adoption of U.S. Department of Defense (DoD) and aviation-industry simulation techniques and technology – not only is attainable and affordable, it’s more efficient. So efficient, he claimed, “that if Waymo used the right simulation technology, it could get to [SAE Level 4 autonomy] within five years for most locations in the U.S.”

With many major automakers and developers now conceding high-level automated driving for everyday vehicles may be a decade or more from reality, DeKort’s statement is provocative. But there are others who agree that AV simulation practices need a shakeup. Some say that upheaval already is underway.

What DeKort professes – and what he and a handful of other simulation developers are in the early stages of executing – is “the new dawn of simulation,” according to Celite Milbrandt, CEO of simulation developer monoDrive. In a similar timeline for how enhanced processing power has advanced the film-making industry to almost unimaginable new vistas of computer-generated imagery (CGI), so too, Milbrandt said, has sophistication advanced for AV simulation.  

Determinism and federation
Dactle’s DeKort maintains his proposed DoD-inspired simulation method doesn’t rely so much on new levels of processing horsepower as on the structure to use processing vastly more efficiently. He contends the industry’s reliance on gaming processors and the simulation practices based around those architectures have led developers down the wrong path. Instead, he said, simulation needs to be based on the concepts of federation and determinism.

Determinism, DeKort explained, is the notion of using a synchronous core or host architecture versus an asynchronous arrangement to determine what runs, when it runs, how often it runs and the order in which runs. This includes every process or model.

Federation is making each system, model or parts of models their own processes or executables – and is what enables determinism to work properly. “The combination of determinism and federation creates extreme math efficiency,” DeKort asserted.

Those who rely on gaming rendering engines for more than visual processing and associated practices may fall behind when it comes to simulating highly complex AV scenarios, he said. “Their approach is so inefficient that they’re throttling themselves all the time. Others run, or try to run, massive amounts of code or models simultaneously. You have to break up and federate the pieces [of a simulation task],” DeKort said. “That’s the way aircraft simulation works,” he added. And to those who counter that aviation scenarios effectively are much simpler than those required for AVs, he said the DoD takes aviation simulation to an immensely complex level with war-game simulations that are run with the same techniques and hardware.

Doug Farrell, principal offering manager, Autonomous Driving Validation at NI (formerly National Instruments), has seen firsthand that thoughts about gaming-based simulation may be changing and calls DeKort’s position “pretty spot-on. I've heard customers who were actually building their own in-house simulation development teams because they don't want to build on game engines or use off-the-shelf tools that are built on game engines,” Farrell said. “Rather than doing that, they're going to spend a whole bunch of money and develop something themselves in-house to avoid that.”

But monoDrive’s Milbrandt doesn’t necessarily agree gaming-based simulation architectures are inherently overmatched. “I think that when you sort of talk about a gaming engine, about what it does – ignore the capability of the animations and the physics and stuff like that, because a lot of times that's generally re-written,” he contended. “But you're really talking about the rendering engine, of whether the rendering engine is correct for testing this level of granularity. If you can model the camera correctly with a rendering engine, then I think it's good enough. If you can make the frame rate deterministic – and if you can model all of the parameters that you need to model – then you're just waiting on the next spin of GPU so you can fit more into it.”

Sébastien Lozé, Simulations Industry Manager at Epic Games, has experience in both the defense and gaming worlds – and sees an optimistic middle ground. “Globally speaking, I agree with Michael; in the grand scheme of things, the defense and the civil-aviation simulation domains are slightly more mature than the other ones. Return on experience, development methodology and focus on accuracy are essential.

“I think the questions we should ask ourselves are not in terms of technology types but really about how to better combine together technologies,” Lozé continued. “No simulators can decently be created in a monolithic fashion as we did 20 years ago – to deliver a well-balanced simulation solution which would leverage the benefits of all worlds.”

Yeah, but what’s it gonna cost?
Typically, increased sophistication comes with a cost. But DeKort told Autonomous Vehicle Engineering that the cost of Dactle’s system “should be in line with most of simulation products. The reason for this is the creation of our simulation and models does not require significantly more effort than the gaming-based simulation companies require to make their products.”

He said cost is contained because his company uses methods and tools that require similar in labor hours but produce a higher fidelity, as well as real-time capable output. And he does concede, “For most scenarios required to get to L4/5, a generic high-end 8-core gaming PC is all that is required, the exception being when an extremely non-federated vehicle model like Adams is used. For that we would have to add cores or put it in a federated machine.

“The important part is the fidelity,” DeKort concludes. “Now I could do the most-demanding scenarios on Day 1.” This system, he said, is all that’s required to create a true digital twin of any environment, “everything you need to stop relying so heavily on the real world. The DoD determined this – out of necessity – 25 years ago.”

DeKort said the industry retains the belief that it is not possible to adequately simulate or model enough facets of real-world development and design to replace the real world to a productive degree – to create a complete “digital twin.” That belief has led to skepticism high-fidelity simulation can be run in the volume necessary for the robust validation AV development requires.

But he said his proposed system’s architectural and engineering upgrades, in combination, yield extremely detailed models and simulation – as well as the massive increase in computational or math processing efficiency necessary for widespread use. Dactle’s simulation “facilitates significant real-time performance differences, especially when the scenarios are complex and loaded, he said. “Without these capabilities a true digital twin cannot be created nor run at or faster than real-time to cover the most complex and loaded scenarios. Not only for the last 10% of the development of a Level 4-5 driverless vehicle but, we believe, possibly the last 50%.”

Typical simulation models not only aren’t accurate enough, DeKort contends. They also become a massive math burden for the system to run at or faster than real-time. His company’s position, he said, is to help “make the industry aware of and utilize DoD/aerospace simulation and modeling technology to build effective and complete digital twins, especially as they relate to physics.”

Then there’s the increasing realization that accurate sensor models are a foundational necessity. DeKort said his techniques include modeling of production sensors and how they interact with exact objects in the real-world. “We model how results are created. Other simulation providers leverage ray tracing, which largely models an anticipated outcome versus creating the outcome.”

NI’s Farrell said current model information about perception systems is “incredibly brittle. They are very fragile to the specifics that you have trained them with and tested them with. So things like leaves rendering differently on a tree could ‘break’ a perception system.”

The former simulation lead for an OEM’s driverless vehicle division, who asked not to be named, supports Dactle’s approach and believes that OEM should have adopted the strategy, but ultimately did not – perhaps because it strayed too far from the conventional, iterative approach that has ruled the auto industry for a century.

“By the frantic pace that the AV folks have imposed on themselves – due to the nature of entrepreneurship, hype curves and need for capital – they are in a rush-rush mode. This kind of mode may be engendering a short-sightedness in certain aspects, including the strategy around AV simulation,” the source said.

“One bias I see is being very ready to spend a lot of resources – money and hours – on self-invented methods with not fully understood value towards development and verification and validation (which includes AV simulation) instead of adapting methods like federated simulation and full determinism towards the new needs at hand,” the source concluded.

Epic Games’ Lozé envisions a productive evolution and merging of simulation disciplines. “I believe that there are several technologies and approaches which can coexist and carry us collectively towards excellence in the very large and still emerging field of AV simulation,” he said. “We could name some examples of very smart and serious companies that are taking a hybrid approach – using both super-solid and rigorous simulation modules for dynamics and sensor data collection in collaboration with Game Engines to ensure both accuracy of the simulation process as well as a fast development process. I don’t believe in ‘Or’ – I believe in ‘And.’”

“I think where you sort of you differentiate yourself, at least in the simulation business, is you really need to verify, validate that your sensor models are accurate to real-world models, said MonoDrive’s Milbrandt. “Whether you're talking about lidar, radar, ultrasonic or camera, you need to model all of the physics associated with those sensors and you need to integrate them into your simulation. If you're not, then you're really not simulating anything.”

Nobody in the simulation industry is going to agree completely with how this “new dawn” of technology and techniques can or will be applied, but Milbrandt believes there is one certainty: “The company with the best simulation technology wins.”