Efficient On-Premises Data Storage Solution for Autonomous Driving 2021-26-0042

Total autonomy will be 100% accident free by testing more than 8 Billion miles of sensor data*. However, testing the software in real driving situations on the road would be extremely time consuming, expensive, cumbersome. Numerous improvements of the algorithms are necessary to perfect the control software for autonomous driving. To train and test the algorithms we need sensor data. Here the challenge is to store and manage petabytes of sensor data across the global locations with data integrity. In autonomous vehicle to maintain the road safety, multiple sensors are mounted with varying levels of maturity. Multisensory data is the process of combining observations from different sources to provide a robust and complete description of an environment and to overcome the limitation in term of availability.

This paper describes an end to end architecture and design of global data ingest, data processing accessing sensor data effectively, data service and data protection method. It gives a scalable and reliable solution that is capable to handle hundreds of petabytes using hundreds of computer cores. Proposed design helps location-independent data Ingest, faster data availability, On-Demand usage of data effectively and recovery from accidental deletion of data. It helps to store exabytes of data in a tiered storage model in different categorizations. This architecture will reduce it to within 3 to 5 days for further processing of data with data integrity & failover management, Having right strategy will be beneficial in mitigating the risks in sensor data management.