Autonomous vehicle testing: in pursuit of perfection

In March 2019, Boeing’s entire fleet of 737 MAX airliners was grounded indefinitely following two crashes in Indonesia and Ethiopia. “For two brand-new aircraft to crash within five months of each other is unprecedented in modern times,” concluded the US House Committee investigation published this year.

Although clearly an aerospace topic, the Boeing storm speaks to many of the pitfalls around cutting-edge vehicle tech: the importance of training on new equipment; the risks inherent in the human-machine interface; and the limitations of computer-aided safety systems. Yet the most striking detail was the scale of the response. The story dominated the global news cycle for months and the reason for such a profound reaction is that airliner crashes are rare.

In contrast, road deaths are so commonplace in 2020 that they rarely enter the news cycle at all. Yet now, as vehicle development speeds towards an autonomous future, this could be about to change. Autonomous technology is presenting ways to mitigate human error and drive down accident numbers like never before.

Perfection is unlikely and the goal of zero crashes probably unattainable. Yet there’s no denying: vehicle automation, advanced safety and driver monitoring systems could create a world in which motoring fatalities become, if not as rare as aircraft crashes, certainly markedly rarer.

Future will be virtually assured

Ensuring test procedures fairly and accurately replicate real-world driving scenarios is a borderline obsession.

Much of our testing is carried out for the consumer, and physical evidence of safety tests is still crucial in maintaining their trust. The ‘Dieselgate’ emissions scandal has eroded public trust in manufacturers’ ability to mark their own homework. This credibility gap is something testers and regulators can help bridge. Whatever opportunities data modelling offers in terms of virtual testing (VT)‚ and they are many, this means physical validation will continue to play a vital supporting role long into the future.

Physical, or biomechanical, testing has been used since the 1960s. It was the only tool we had at the time. Yet since at least the 2000s, VT has really come into its own. Improved crash-data models and better computing power have made this possible. VT gives us and others the chance to make test procedures increasingly realistic, and to assess how safety tech performs in hundreds, even thousands, of different scenarios. As our understanding of big data and modelling grows, the opportunities are mind-blowing and VT will help organisations embrace the challenge of testing automated and autonomous vehicles, and ensure our procedures become ever more robust.

Autonomous vehicles are not going to be perfect. There will still be crashes, still be system malfunctions, things will still go wrong

VT allows us to increase complexity, and variability, to better replicate real-world driving situations. Today, for example, we might have a single pedestrian passing in front of a parked car on our track. With VT we can dive into a vehicle’s algorithm and ‘trick’ it into responding to a greater range of scenarios.

We might introduce a city environment to a vehicle’s sensor set. Or employ virtual animation to fool the car into seeing a city street with dozens of people on it.

VT is also a more effective way to test how a vehicle’s systems work together. Rather than isolating one element and testing its ability, VT allows us to better assess how the vehicle’s driver monitoring, guidance and crash-avoidance systems work together.

What we’re now beginning to do, and this is illustrative of where testing is going in the future, is produce a library of test data relating to real-world crashes that models a range of scenarios. Thatcham today has a library of some 330 incidents and counting. We could in the future make this matrix available to automakers so they have full visibility. We could then ask them to provide physical evidence of the matrix of tests in a randomised but transparent way.

Collaboration with carmakers

The challenge for the testing community, one of our greatest, in fact, is treading that careful line between supporting the sector we love and the automakers for whom we have infinite respect, while at the same time protecting the lives of motorists. Therefore, Thatcham has always prioritised collaborative relationships with carmakers.

Ultimately, we need the support of manufacturers. Keeping automakers appraised of what we’re doing and how we’re doing it is the best way to achieve the co-operation and response times that are so crucial to our work. That’s why Euro NCAP publishes a roadmap document, which anticipates the challenges ahead in terms of automated and autonomous vehicle testing, and makes clear where we all need to be in the coming years. Automakers commit millions to new tech products and it’s crucial they have a clear view ahead.

We consult closely with carmakers on each roadmap and together we are currently co-authoring the 2025-30 edition, for which we’re discussing the role of CAE (Computer Aided Engineering) modelling data to increase the range of variables in our virtual testing.

Instead of conducting physical tests, automakers are increasingly providing virtual test data as evidence. This means forging a new way of working together. Automakers need to be confident they are not revealing proprietary data; we need to be confident we can translate this IP-free data into physical-world tests that tell the consumer the whole truth.

Changing regulatory environment

Thatcham is heavily involved with the ‘Geneva Process’, the vehicle regulatory forums that form part of the United Nations Economic Commission for Europe (UNECE). And we work closely with UK regulators through the DfT’s Centre for Connected and Autonomous Vehicles (CCAV). European type approval regulations are generally based on Euro NCAP test procedures, much of which is grounded in Thatcham’s active safety research.

Regulators recognise the need to evolve in response to vehicle automation. To look at all the constituent parts of safe systems for automation, and the resulting testing that needs to occur. This challenge will intensify as we accelerate towards L4 and eventually L5-enabled vehicles, and new technology is already outpacing legislation and guidance.

The potential of autonomous vehicles could create a world in which motoring fatalities become, if not as rare as aircraft crashes, certainly markedly rare

The work Thatcham and Euro NCAP have recently completed on Level 2 Assisted testing could pave the way. It is really the first time any organisation has developed modular procedures to test and validate assisted or automated vehicles. The framework ensures vehicles must have good balance between vehicle assistance, driver engagement, and safety backup to achieve top marks.

The L2 procedure is an excellent starting point for automakers to work with. As we have many times before, Thatcham and Euro NCAP have set the standard and given regulators a basis for where we might go in the future.

Near nirvana

It is unlikely that motoring will come to resemble air travel any time soon and that vehicle crashes will become so rare that we can investigate every one in forensic detail. And it’s even more unlikely that crashes and prangs will ever reach zero.

Autonomous vehicles are not going to be perfect. There will still be crashes, system malfunctions, things will still go wrong—and motorists will still need insurance cover.

Yet there is no denying that autonomous vehicles, with support from the testing and regulatory communities, could reduce risk to levels previously thought impossible. Near nirvana, if you like. And virtual testing will underpin that.

Matthew Avery is Director of Research at Thatcham Research, the UK’s only not-for-profit insurer-funded research centre.