SAE recommended practice on behavior reference models for evaluating automated driving systems in traffic conflict scenarios
J3330
This Recommended Practice provides general definitions, dimensions, and taxonomies for the application of driver behavior reference models in the context of automated driving system (ADS) evaluation in traffic conflict scenarios.
Questions addressed by this recommended practice include:
● What is a behavior reference model for ADS (and what is it not)?
● What types of behavior reference models exist and what dimensions can be used to characterize them (e.g., quantitative vs. qualitative, subjective vs. objective, heuristic vs. automated)?
● What are the key requirements for behavior reference models?
- Scenario coverage: What types of scenarios should be covered by the models (e.g., front-to-rear, straight crossing path etc.)?
- Behavior coverage: What types of behaviors should the models represent (e.g., vehicle following, hazard response, evasive maneuvering) ?
- Model complexity: What aspects of driving should the behavior reference models minimally capture and at what level of detail?
- Validation: How can behavior reference models be validated?
- How can/should behavior reference models be used to validate ADS and how should the results be interpreted?
Applicable driving behaviors
While behavior reference models may, in principle, be established for any aspect of driving behavior, this Recommended Practice focuses on driving behavior directly associated with traffic conflicts (i.e., collisions or situations where an imminent response is required to avoid collision), including behaviors prior to, during, and after the conflict.
Applicable automated driving systems, vehicles and road types
This document applies to the evaluation of ADS, that is, hardware and software that are collectively capable of performing the entire dynamic driving task (DDT) on a sustained basis, regardless of whether it is limited to a specific operational design domain (SAE J3016). ADS thus includes driving automation at SAE levels 3, 4 and 5 in both dual-mode and dedicated vehicles. The document primarily applies to OEM or aftermarket ADS installed onboard in light vehicles, heavy vehicles, including articulated vehicles and single vehicles towing single and double trailers, corresponding to vehicle Class 2-13 in the vehicle classification scheme defined by the US Federal Highway Administration. The document does not apply to tracked or partially tracked vehicles, vehicles with skis, vehicle platoons, off-road vehicles or vehicles driven remotely. Construction vehicles other than those designed for use on public roads (e.g., forestry and mining equipment) are also excluded.
Applicable evaluation contexts
This document is applicable to any study or test of an ADS where the ADS behavior is compared to a benchmark represented by a behavior reference model. This includes, but is not limited to, simulated testing on synthetic scenarios, structured testing on test tracks, and re-simulation of driving logs. The document focuses entirely on offline evaluation whereas methods for onboard, real-time, application of reference models is not in scope.
Rationale: Driving behavior reference models here broadly to models, measures, and statistics used as benchmarks in ADS behavior evaluation. They represent benchmarks of behavior for how an ADS should perform in a particular driving scenario or event, including scenarios in which crashes occur. Today, a variety of driving behavior reference models have been proposed or are under development. Examples include, for example, models intended to represent aspects of an attentive and/or careful and competent human driver, safety envelope models operationalizing defensive driving in terms of kinematically defined margins or models representing the extent to which an ADS’s behavior aligns well with the expectations of other road users. Furthermore, models range from relatively basic rule-based (heuristic) models to full-blown computational simulated agent models, often covering very different aspects of driving behavior (e.g., vehicle following, hazard response) and types of traffic scenarios (e.g., front-to-rear, straight crossing path etc.).
Given this heterogeneity, there is a need to establish stronger consensus around the concept of a driving behavior reference models and their use in the context of ADS evaluation. The objective of this report is to (1) define what constitutes a behavior reference model, (2) categorize the different types of behavior reference models, (3) define general requirements for the models, and (4) provide guidance for their use and documentation. This recommended practice describes the general characteristics of these models and their application to offline scenario evaluation but does not specify or recommend a particular model.
