rFpro believes that one element that is critical to the successful adoption of AVs is a collaborative driving environment specifically created for their training and development. The concept is similar to that of online gaming, imagine up to 50 AV manufacturers or technology providers sharing a virtual world. It would be a critical step to enable AVs to react to other vehicles which are being independently controlled, without resorting to physical testing in the real world. It would offer a faster, safer route to validation without the need to introduce unproven technology into a physical public space.
In this interview, Matt Daley explains the concept of collaborative driving and why it is critical for the road to autonomy. He highlights the challenges of such a concept and calls for the industry to work together to accelerate development.
When it comes to autonomous vehicles, what does ‘collaborative driving’ really mean?
In essence, it is enabling open communication between all the vehicles on the road and the infrastructure around it to create a much more efficient and safer road network.
Correctly perceiving the world ahead is the single biggest challenge for autonomous vehicles. If the vehicle could communicate with other cars on the road as well as the road infrastructure this would become significantly easier.
By sharing information, such as what one vehicle has already perceived to another, the intended direction and actions of the vehicle or even when a traffic light is about to change, navigating the world becomes much easier. The more accurate information that is added to the collective pot of knowledge, the better the system works for everybody.
This would be a big shift for the industry, how do we reach this point?
Absolutely. This would require a monumental collaborative approach from all areas and players of the industry. One of the biggest challenges would be determining the chosen ‘language’ or communication system that would need to work universally with both vehicles and infrastructure. There are a number of industry bodies around the world that are trying to work towards this utopian goal.
Simulation is a fantastic way to test and evaluate the various communication systems before spending any money on physical infrastructure. Now is the time to be doing this as the same simulation environments are also being used to conduct the majority of autonomous vehicle development. There are huge volumes of simulation experiments being run each day by many individual companies for their own vehicles. But now is the time to start rapidly expanding the collaborative driving simulations too, showing that every AV can interact and communicate correctly with vehicles and infrastructure nodes from all of the different manufacturers.
Why are AVs being developed predominantly using simulation?
Virtual testing is the only safe means to validate AI without introducing immature, unproven technology into a physical public space. Any AV must be capable of co-existing with other road users without having any control over their behaviour. Currently, the industry relies on validating this capability through real world driving because it’s the only situation where AVs can interact with genuinely uncontrolled vehicles or road users. It would be much safer if AVs could be thoroughly evaluated by discovering their limitations in a multi-user simulation within a shared virtual environment. This shared simulation world also then allows the development and validation of the communications between those multiple actors – to improve the safety levels for all.
What would a simulation of collaborative driving in a shared virtual environment look like?
The concept has been around in both the military and also online gaming for many years already. Now, instead of having multiple military personnel and units trying to achieve a common mission goal or instead of 20 gamers racing each other around Silverstone or Daytona, imagine 50 or 100 AV vehicle manufacturers, technology providers and infrastructure management companies sharing a virtual Paris city centre navigating the Arc De Triomphe.
Each vehicle under the control of its manufacturer is termed an ‘ego’ vehicle, with the capacity to take independent action. Collaborative driving in a virtual environment requires multi-ego simulation in which all the vehicles can interact and respond to actions from other road users, which may be random or uncontrolled, in order to validate their systems. The virtual environment would also simulate the various ‘nodes’ or sensors used for infrastructure, so the communication system can be evaluated. For example, a roadworks zone can broadcast its position, speed restrictions and any lane closures out to the vehicles before they arrive to avoid dangerous last-minute actions.
The simulation environment would form a large virtual proving ground for all the players in the industry, including vehicle manufacturers, tier suppliers, sensor developers, communication suppliers, type approval bodies and even local councils.
How far along the road are we today?
We have already worked with partners, such as Zenzic in the UK, who have shown via their interoperable simulation project that different parts of the simulations like ego vehicles, traffic vehicle inputs and traffic signals can all exist in the same simulation, even though they are physically in different locations. The technology could benefit not only AV development but could be used by the authorities to standardize and greatly accelerate the vehicle approval process.
Multi-ego collaborative driving within rFpro is already being done by a number of our automotive manufacturers and Tier 1 customers, within their own private networks. We don’t yet know of any customers who have taken the step of running a joint simulation between different remote companies. They are already using our Dedicated Server node to join their multiple ego vehicles and suites of sensors onboard each one. Each vehicle only has to share its own vehicle speed and position, with no requirement to share its critical driving IP. It is up to the participants involved to contribute to or listen to any additional shared collaborative driving communications that are being transmitted.
Our next step is to help open up these connections between businesses. In our environment, traffic and the world are defined by the central hub and participants would ‘hire’ a vehicle space for a set period. They would not even need to know the identities of the companies operating the other vehicles.
The main computing power still resides at the various users’ sites, with all of their own sensor simulations and autonomous driving decision-making code being run within their own protected locations. The central server only exists to link everybody together.
There is also an ongoing project called VOICES (Virtual Open Innovation Collaborative Environment for Safety), run by the US Department of Transport, which aims to offer a centrally managed data exchange system whilst leaving the end customers to run their own vehicle and sensor simulations. Users will be able to connect their full vehicle systems simulations into VOICES network and drive around in virtual worlds with other vehicle manufacturers and infrastructure management operators.
What have been the main challenges in creating this environment?
Decision making by an AV is a two-step process; the first is to correctly identify all the features in its surroundings, the second is to take appropriate action. Of these, the first step, perception, is the most challenging to accomplish accurately and repeatably under all conditions, so high-grade sensor modelling is one of the foundations of effective simulation.
Collaborative driving involves passing information from multiple actors about what they perceive in the world to each other, so the highest-fidelity simulations must replicate all the different sensors on the vehicles and roadside infrastructure. rFpro’s hugely scalable simulation size allows these to all be run at the same time to create the world’s first engineering-grade, multi-user, full system simulation. This can be run in real-time or, where engineers want to go to even higher levels of fidelity the simulation can run in a fully synchronous system mode. Essentially, when the test scenario gets very complex we can slow the simulation down to enable the computers to keep up.
This overcomes challenges such as the blurring of images due to fast vehicle motions, or road vibrations, and can produce accurate rolling shutter effects and multiple exposure periods for specific camera manufacturers’ models.