The Next Step for Digital Twin – Virtual World

Digital Twin is a widely spread concept of creating a virtual representation of object state. The object may be small, like a raindrop, or huge as a factory. The goal is to simplify the operations on the object by creating a set of plain interfaces and limiting the amount of stored information. With a simple interface, the object can be easily manipulated and observed, while the state of its physical reflection is adjusted accordingly.

In the automotive and aerospace industries, this is a common approach to use virtual objects representation to design, develop, test, manufacture, and operate both parts of a vehicle, like an engine, drivetrain, chassis/fuselage, or a full vehicle – a whole car, motorcycle, truck or aircraft. Virtual representations are easier to experiment with, especially on a bigger scale, and to operate – especially in situations when connectivity between a vehicle and the cloud is not stable ability to query the state anyway is vital to provide a smooth user experience.

It’s not always critical to replicate the object with all details. For some use cases, like airflow modeling for calculating drag force, mainly exterior parts are important. For computer vision AI simulation, on the other hand, user checking if the doors and windows are locked only requires a boolean true/false state. And to simulate the combustion process in the engine, even the vehicle type is not important. 

Today, artificial intelligence takes a significant role in a lot of car systems, to name a few: driver assistance, fatigue check, predictive maintenance, emergency braking, and collision avoidance, speed limit recognition, and prediction. Most of those systems do not live in a void – to operate correctly they require information about the surrounding world gathered through V2X connections, cameras, radars, lidars, GPS position, thermometers, or ABS/ESP sensors.

Let’s take Adaptive Cruise Control (ACC). The vehicle is kept in lane using computer vision and a front-facing camera. The distance to surrounding vehicles and obstacles is calculated using both a camera and a radar/lidar. Position on the map is gathered using GPS, and the speed limit is jointly calculated using the navigation system, road sign recognition, and distance to the vehicle ahead. This is an example of a complex system, which is hard to test – all parts of it have to be simulated separately, for example, by injecting a fake GPS path. Visualizing this kind of test system is complicated, and it’s hard to use data gathered from the car to reproduce the failure scenarios.

Here the Virtual World comes to help. The virtual world is an extension of the vehicle shadow concept where the multiple types of digital twins coexist in the same environment knowing their presence and interfaces. The system is composed of digital representation of physical assets whenever possible – including elements recognized via computer vision. Vehicles, road infrastructure, positioning systems, or even pedestrians are part of the virtual world. All vehicles are part of the same environment meaning they can share the data regarding the position of other traffic participants.

• Such a system provides multiple benefits: Improved accuracy of assistance systems, as the recognized infrastructure and traffic participants can come from other vehicles, and their position can be estimated even when they are still outside the range of sensors.
• Easier, more robust communication between infrastructure, vehicles, pedestrians, and cloud APIs as everything remains in the same digital system. 
• Possibility to fully reproduce conditions of system failure as the state history of not just vehicle, but all of its surrounding remains in cloud and can be used to recreate and visualize the area.
• Ability to enhance existing systems leveraging data from the greater area – for example, immediately notifying about an obstacle on the road in 500 meters and suggestion to reduce speed.
• The extensive information set can be used to build new AI/ML applications, like real-time weather information (rain sensor) can be built to close sunroofs of vehicles parked in the area.
• The same system can be used to better simulate its behavior, even using data from real vehicles.
• Common interfaces allow for quicker implementation.

Obviously, there are also challenges – the amount of data to be stored is huge, so it should be heavily optimized, and storage has to be highly scalable. There is also an impact of the connection between the car and the cloud. Overall, the advantages overweight the disadvantages, and the Virtual World will be a common pattern in the next years with the growing implementation of software-defined vehicles and machine learning applications requiring more and more data to improve its operations.