8 in-vehicle E-Commerce applications that pave the way for a seamless drive

 It cost a staggering $200 for that time. The antenna took up almost the entire roof of the car, the batteries barely fit under the front seat, and the huge speakers had to be fixed to the back of the seat backrest. The year was 1922, just over 20 years after the launch of the first mass-produced Oldsmobile Curved Dash car. Entertainment had just made its entrance into the car industry - Chevrolet introduced the first car radio. From then on it only got more exciting.

Nowadays, 100 years on from that event, we can no longer envisage a car without radio, music, or news. In fact, we can no longer imagine a car without entertainment in the broadest sense of the word. Because the radio - at least in its traditional form - is slowly becoming obsolete. It's being replaced by a "personal radio station" created by the driver - streaming music, favorite podcasts, audiobooks, and even video content.

Although we are still a far cry from the catchy phrase  "a smartphone on wheels" , first uttered in 2011 by Akio Toyoda, the automotive industry is indeed heading in this direction. Cars are ceasing to be vehicles designed to take us from A to B. Like any other device connected to the Internet, they are becoming a gate to new worlds of entertainment, shopping, learning, or gaming.

When finishing shopping or listening to an audiobook on one device, we want to seamlessly continue the activity on a laptop or desktop computer. Whether we like it or not,  the car is becoming another medium that will allow us to stay virtually connected all the time.

Akio Toyoda was wrong. A car is much more than a "smartphone" on wheels!

A potentially larger screen than a smartphone (not only the touchscreen in-vehicle infotainment system panel, but the windscreen too, which can also be used to display content), at least 4 seats that can be independently paired with the in-car entertainment system, and, ironically, much more mobility than mobile devices.

As we look at the development of V2X (vehicle-to-everything) technology, which will turn vehicles into the Internet of Things devices, the opportunities that lie ahead for the automotive industry in the entertainment field are hard to estimate.

One thing is certain. This process cannot be stopped. Every company in  the automotive industry must be aware of the upcoming changes.

According to IHS Markit data, in 2014 only 53% of cars in the USA had a dashboard touch screen, while today this percentage has already reached 82%. These types of solutions can bring automotive companies entirely new revenue streams, and most importantly they will be less dependent on vehicle production cycles and with much higher margins.

The in-vehicle infotainment system market is estimated to be worth $78.9 billion by 2025. [Allied Market Research].

Quo Vadis in-vehicle infotainment systems?

In-vehicle voice assistants for infotainment control

Siri, Alexa, or Google Now are names that have become part of the consumer market and make life easier for most of us, allowing us to make phone calls, send messages or manage our own calendars. While sending voice commands to our phone or the speaker in our home or office is nothing new, communicating with our own car is still some kind of novelty.

And it is here while driving when we need to focus on the road and have our hands free, that voice technology can be of the most benefit and make driving more efficient and smooth. And of course, more fun.

Navigant Research (Guidehouse) predicts that by 2028, 90% of vehicles will be equipped with a voice assistant. Already today - looking at Voicebot.ai data - a large proportion of commands given by drivers are entertainment-related. Playing music, listening to podcasts, finding out about movies, ordering food, or making purchases directly from behind the wheel is becoming increasingly popular among drivers with enhanced IVI systems.

The main players in this section are certainly the manufacturers already known for their other platforms, namely Google and Apple, which are integrating their Android Auto and Carplay technologies in partnership with major OEMs. Hot on its heels is Amazon, which has not only begun collaborating to bring Alexa into Toyota, Ford, and BMW vehicles but also released an Amazon Echo device that any driver can install in their car themselves (as long as it meets the manufacturer's technical requirements).

Vehicle manufacturers, however, are no longer just waiting for the offers of the largest players in this market, but are developing their systems or working with smaller business partners to help them develop such solutions.

Korea's Hyundai has entered into an operation with Saltlux, a company specializing in semantic networks. Honda, Kia Motors, and Daimler are working with the SoundHound start-up. And Volkswagen has invested $180 million in the Chinese start-up Mobvoi.

Gesture-recognition

Voice command in the car is a trend that will continue to grow every year. Yet, there are situations in which gestures are much better than voice commands - for example when you are on a call or have a cold and don't want to strain your throat. Gestures are universal for every driver, while voice assistant applications are often still hampered by technological limitations, for example, due to the variety of accents or the system's adaptation to the driver's language.

As the system recognizes a gesture made with the palm of your hand, fingers, or even your head, you can stay focused on your driving and at the same time activate a specific function when you cannot use your voice command. Scrolling through songs on the radio, raising or lowering the temperature in the car, launching a text message application - all these actions can be configured using gestures. Instead of clicking and scrolling through a touchpad, which always entails taking your eyes off the road, gestures will allow you to boost safety and easily manage the entire system.

Virtual reality & Augmented reality

While currently the introduction of virtual reality in vehicles only makes sense for passengers who do not need to focus on driving, augmented reality technologies are already being successfully implemented in vehicles. Unlike VR, augmented reality does not distract drivers from reality and allows them to concentrate on driving. And they can even increase safety.

Although today this type of technology can only be found in the most innovative and prestigious IVI systems (one of the first cars in which this technology was used was Mercedes-Benz GLE 2020), we should expect this type of solution to develop in the near future, as it brings a whole new quality to in-car entertainment.

Their direct equivalent to the automotive field is the heads-up display system, which is an additional head-up display integrated into the vehicle's windscreen in addition to the IVI control panel. This screen can be used to display destination-related information, traffic warnings, or information about other vehicles on the road (so-called intelligent terrain mapping).

In the near future, these technologies may also be applied in entertainment itself - for instance in the form of augmented marketing. The windscreen will then display interesting offers and discounts from the restaurants, shops or shopping malls we have just passed. The displayed images will of course adapt to our driving speed, and we can decide for ourselves what kind of messages we wish to see.

On-demand in-car services

In-vehicle infotainment systems are the point of contact between different parties: customers, internet providers, companies producing vehicles, making entertainment, or electronic equipment (e.g. smartphones).

In most cases, drivers already have their favorite apps (Google and Apple being in the lead, of course) and use their favorite streaming services. Competing with platforms like Spotify, Netflix, Pandora or Slacker may not necessarily be the best strategy for automotive companies. It is much better to make use of the recognisability of brands that provide entertainment content and, based on this, extend it with a unique offer for their own clients. Opening up to partnerships with third-party platforms is the best way to address  customer needs and create a stream of data that can be monetized .

One of the interesting market examples of this type are the efforts of the GM concern, which has created its own car application in the form of a marketplace, from which the driver can make purchases at Starbucks or Dunkin' Donuts, pay for the fuel at selected petrol stations, and book a hotel or a table at a restaurant.

We should expect that the trend of shopping straight from the car and making the most of the time we have on our commute to/from work while being stuck in traffic jams will not be limited to listening to music and podcasts only. With the development of the Internet of Things, drivers will also be able to control other devices within their "smart" network from their vehicles.

Samsung is already creating solutions that allow the driver to look into their own fridge and decide whether they need to go shopping, turn up the thermostat to prepare the perfect temperature for the return home, activate the alarm when going on holiday, or open the gate automatically.

Rear seat entertainment

Most modern IVI systems are not just an integrated head-unit, i.e. a touch panel on the vehicle dashboard for the driver, but more and more often, interactive panels dedicated to the passengers. These offer practically endless opportunities for entertainment. And we don't just mean the extensive range of streaming video services that can be subscribed to in the vehicle.

After all, the interactivity of the screens makes it possible to implement various applications and gamification elements in the car. These can take the form of quizzes, common picture drawing, shopping via third-party applications, or even karaoke singing, which can also engage the driver.

But what if the sound or type of music doesn't suit the driver, who wants to concentrate on driving? There are already solutions that direct the sound from different areas of the vehicle so that each passenger can listen to different music without wearing headphones.

This is how, for example, the Separated Sound Zone (SSZ) works in KIA cars. Based on multiple loudspeakers and the physical wave acoustics principles, the sounds do not overlap but instead reach their intended audience. Even if powerful beats dominate in the back seat, you can still relax while listening to calmer music in the driver's seat.

In-vehicle infotainment enters a new era

In-car entertainment has a long history. Ever since mobile devices became part of our lives, it is nothing new to connect a smartphone to a Bluetooth radio or for passengers to watch videos on their own smartphones/tablets. The only difference was that, until recently, in-vehicle infotainment was just an accessory, an element that makes a difference and highlights a brand. Today it is a factor on which customers often rely when buying a new vehicle.

In-vehicle infotainment is increasingly rarely limited to a touch screen panel on the dashboard. Right before our eyes, it is growing to be omnipresent and taking precedence over other vehicle functions. Brands that miss this moment and, like Blockbuster in the video content market or Nokia in the mobile market, may find themselves in a completely new reality. A reality in which totally different companies will be on top of the bunch.

Generative AI is becoming a major player in automotive innovation. The market is already valued at  USD 480.22 million in 2024 , and it’s expected to grow to USD 3,900.03 million by 2034, with a steady annual growth rate of 23.3%. Moreover, by 2025, the global automobile sector will invest $11.1 billion in cognitive and AI technologies. These numbers show how quickly the industry is picking up on this technology’s potential.

GenAI is making its mark across various areas. From manufacturing optimization to autonomous driving, its impact is undeniable. Predictive maintenance systems identify issues early, AI-powered tools optimize vehicle development, and talking to  in-car assistants is starting to feel like a scene out of a sci-fi movie.

Speaking of sci-fi, pop culture has always loved the idea of talking cars. There is K.I.T.T. (Knight Industries Two Thousand), of course, but also all Transformers and tons of cartoons, starting with Lightning McQueen. Is it just pure fiction? Not at all (except McQueen, for many reasons 😊)! Early attempts at smarter cars started with examples like a 2004 Honda offering voice-controlled navigation and Ford’s 2007 infotainment system. Fast forward to now, and we have a VW Golf with a GPT-based assistant that’s more conversational than ever.

But honestly, the most resourceful one is K.I.T.T. – it activates all onboard systems, diagnoses itself, and uses company resources (there is an episode when K.I.T.T. withdraws money from the company bank account using an ATM). In 1982, when the show first aired, it was just pure science fiction. But what about now? Is it more science or fiction? With  Generative AI growing rapidly in automotive, we have to revisit that question.

Let’s break it down!

Prerequisites

Let’s assume we would like to create  a solution-oriented chatbot connected with a car. By “solution-oriented,” I mean one that is really useful, able not only to change the attractive interior lighting but also to truly solve owners’ issues.

The idea is to use Generative AI, a large language model with its abilities in reasoning, problem-solving, and language processing.

 Therefore, the first question is – where should the model be planted – in the cloud or a car?

For the first option, you need a constant Internet connection (which is usually not guaranteed in cars). In contrast, the second option typically involves a smaller and less versatile model, and you still need a lot of resources (hardware, power) to run it. The truth lies, as usual, in between (cloud model if available, local one otherwise), but today we’ll focus on the cloud model only.

 The next step is to consider the user-facing layer. The perfect one is integrated into the car, isn’t it? Well, in most cases, yes, but there are some drawbacks.

The first issue is user-oriented – if you want to interact with your car when being outside of it, your mobile phone is probably the most convenient option (or a smartwatch, like Michael from Knight Rider). Also, infotainment systems are comprehensively tested and usually heavily sealed into cars, so introducing such a bot is very time-consuming. Therefore, the mobile phone is our choice.

We don’t want to focus on this application today, however. Depending on the target operating system, it probably should use speech-to-text recognition and text-to-speech generation and stream data both ways for a better user experience.

The core part is the chatbot backend – a regular application connecting the frontend and the LLM. It should be able to call external APIs and use two sources of knowledge – live car data and company-owned data sources.

Basics

Let’s gather the components. There is a customer-facing layer – the mobile application; then there is our main backend application, the LLM, of course, and some services to provide data and functionalities.

The diagram above is conceptual, of course. The backend is probably cloud-hosted, too, and cloud services linked to car services form the essence of the “connected cars” pattern.

 The main concept for the application is “tool calling” – the LLM ability to call predefined functions with structuralized arguments. That’s why the backend is surrounded by different services. In a perfect world, those should be separated microservices designed for different use cases. However, this architecture is not scenario-based. There is no “if-else-if” ladder or so. The LLM determines how to utilize the tools based on its own decision-making process.

The sample conversation schema might look like the one presented below.

As you can see, the chatbot service calls the LLM, and the LLM returns command “call function A.” Then, the service calls the function and returns the response to the LLM (not the user!).

This approach is very flexible as functions (a.k.a. tools) might execute actions and return useful data. Also, the LLM may decide to use a function based on another function result. In the case above, it can, for example, use one function to check the climate control system status and discover that it’s running in the “eco mode”. Then, it might decide to call the “set mode” function with the argument “max AC” to change the mode. After that, the LLM can return an answer to the user with a message like “It should be fixed now”.

To build such an application, all you need to call the LLM like that (OpenAI GPT-4o example):

{
 "model": "gpt-4o",
 "messages": [
   {
     "role": "user",
     "content": "My AC is ineffective! Fix it!"
   }
 ],
 "tools": [
   {
     "type": "function",
     "function": {
       "name": "get AC status",
       "description": "Return current status of the climate control system"
     }
   },
   {
     "type": "function",
     "function": {
       "name": "set AC mode",
       "description": "Sets up the specified mode for the climate control system",
       "parameters": {
         "type": "object",
         "properties": {
           "mode": {
             "type": "string",
             "description": "Desired mode",
             "enum": ["ECO", “NORMAL”, "MAX AC"]
           }
         },
         "required": ["mode"]
       }
     }
   }
 ],
 "tool_choice": "auto"
}

As you can see, the response schema does not bother us here – the assumption is that the LLM is able to understand any reasonable response.

Dive

The subtitle should be a “deep dive”, but honestly, we’re just scratching the surface today. Nevertheless, let’s focus a little bit more.

So far, we have the user-facing application and the backend service. Now, let’s make it useful.

The AC example mentioned above is perfectly valid, but how can it be achieved? Let’s say there is an API for interaction with the AC in the car. It’s typical for all PHEVs and EVs and available for some HEVs, too, when you can turn on your AC remotely via the mobile app. However, the real value lies in the connected car

There is no IP address of the car hardcoded in the application. Usually, there is a digital twin in the cloud (a cloud service that represents the car). The application calls the twin, and the twin notifies the vehicle. There should also be some pub/sub queue in between to handle connectivity tier disruptions. Also, the security layer is extremely important. We don’t want anybody even to play the radio at max volume during a quiet night ride, not to mention turning off the lights or engaging breaks.

 Which brings us to the list of possible actions.

Let’s assume all systems in the car are somehow connected, maybe using a common bus or a more modern ethernet-like network. Still, some executors, such as brakes, should be isolated from the system.

So, there is no “brake API” to stop a car. However, it may be beneficial for mechanics to execute some "dangerous" actions programmatically, e.g., to increase the pressure in the braking system without actually pressing the pedal. If this is the case, such functionalities should be accessible exclusively through a local connection without the need for digital twin integration. Therefore, we can assume there are two systems in the car – local and cloud-integrated, no matter the nature of the isolation (physical, network, or software). Let’s focus on the  connected car aspect.

I believe the system should be able to change the vehicle settings, even if there is a risk that the driver could be surprised by an unauthorized change in the steering feel while taking a turn. This way, the chatbot might be useful and reduce support load by adjusting car settings based on the user's preferences. To avoid misusage, we can instruct the chatbot by prompt engineering to confirm each change with the user before execution and, of course, implement best-in-class security for all components. We can also allow certain operations only if the car is parked.

 Which brings us back to the list of possible actions.

For the sake of this article, let’s assume the chatbot can change various car settings. Examples include:

•  Climate control settings
•  Driver assistant sensitivity and specific functions toggles
•  Navigation System settings, like route type or other functions toggles
•  360 camera system settings, like brightness adjustment
•  Sound system settings like equalizer
•  Wiper settings
•  Notifications settings
•  Active steering system settings

This list is not complete, and the best thing is – it doesn’t need to be, as adding new functions (tool definition + API availability) might be a part of the future system OVA update.

What about reading real-time data? Should we connect to the car directly and read the status? Let’s leave this option for another article 😉 and focus on communication via the cloud.

 There are two possibilities.

We can provide more tools to get data per source/component (a reminder – LLM decides to call for data, which then triggers an API call, and the LLM processes the received response). Alternatively, we could implement a single tool, “get vehicle data,” that collects and merges all data available from all data sources.

For the latter approach, two ways are available – do we really need a tool? Maybe we should inject the current state into each conversation, as it’s probably beneficial to have the current state anyway to solve all cases?

Let me give the standard consultant reply to those questions.

 It depends.

More data in the context means extended response time and a higher bill. Also, some cases don’t need the entire context or don’t need it at all. On the other hand, if you let the LLM decide which part of the context is necessary (which function to call), it will also affect the time and cost.

The next part is the “cost” of collecting data. Some data sources might be slow or consume valuable in-car resources to calculate (maybe not appropriate) data.

My advice would be to start collecting all data in the background during session creation and attach it to the session progressively when ready. Additionally, provide the LLM with the capability to wait for data if it has not yet been supplied. This should be implemented as a function to handle instances where the user requests data-related action, but the necessary data is not yet available

The second part is splitting the data. You can use a much smaller LLM to decide which context data is necessary to answer the current question and remove unnecessary information to keep it concise.

The last part is company data. All LLM models are trained based on thousands of manuals available over the Internet, but this is usually not sufficient. Your chatbot should have the car manual at its fingertips, so the general knowledge is not enough. Therefore, you need to provide it. You can execute the  RAG procedure and add some content to the context for each prompt, but my advice would be to hide it behind a function, too. Then, the LLM may decide when to call it to get some extra information if needed.

Use cases of Generative AI for connected cars

OK, we have all the components in place – there is the frontend, the backend, the model, and the toolbox. Can it be useful? As you would expect!

In the examples below, I will skip the part where the LLM commands the backend to perform an action, and the backend calls an API. I will also omit the process of the API response traveling back through the backend to the LLM. Additionally, I will assume that no new data is added to the context, so the LLM must decide which data it needs. Besides that, make sure that your context contains some system prompts to better explain all possibilities to the assistant.

Here are some examples.

                   User:         I think my Lane Keeping Assist activates too frequently. Can you check it?    
                Assistant:               [get driver assistance settings]    
                Assistant:               [diagnose sensors]    
                Assistant:               Everything looks good, but I can see that the Lane Keeping Assist mode is set to “fast”. Do you want me to lower the setting to “medium”?    
         User:         Yes, please.    
                Assistant:               [change LKA settings]    
                Assistant:               It should be better now.          

In the example above, the user has a problem with one of the driver assistance components. The system proposes and, after confirmation, executes settings changes, so the driver doesn’t need to find a proper setting in the on-board computer.

Check another case.

                   User:         Whenever I’m parking the car in my garage, I can barely see anything in my rear camera. There is not much light there, but maybe you help me in some other way?    
                Assistant:               [get camera settings]    
                Assistant:               I can increase the brightness of the camera, but then it might be too bright for other places. What do you think?    
         User:         Yes, please do that.    
                Assistant:               [change settings]    
                Assistant:               Here you are; let me know if you want me to roll back the change.          

Usually, there is no brightness setting available for the user. To adjust it, the user would need to schedule a service department meeting, spend some time, and pay for the facility. The assistant may do it in a few seconds for free.

                   User:         There are so many notifications in the car, so I can’t hear the important ones!    
                Assistant:               [check last notifications]    
                Assistant:               If you find some of my notifications unnecessary, I can mute some of them. What do you think?    
         User:         Yes, please turn off the speeding avoidance warning.    
                Assistant:               I can’t turn it off due to legal requirements, but I can replace the sound with vibration. feedback on the acceleration pedal. Do you want me to do that?    
         User:         Sounds great!    
                Assistant:               [adjust settings]    
                Assistant:               It’s done. I hope you’ll be less distracted.          

In this case, the assistant solves the issue of annoying notifications. Note the alternative solution suggestion and knowledge about the legal aspect. We can achieve it with proper function descriptions.

                   User:         It seems my car can’t connect to the Internet.    
                Assistant:               [check the connection] -> no connection to the car from the cloud service    
                Assistant:               It seems your car is disconnected. Can you please check the cellular network signal icon on the infotainment main screen?    
         User:         I can see the signal is good.    
                Assistant:               [check last logs]    
                Assistant:               I can see there were some issues with your connection before. Please try to open the settings in the car infotainment and restart the communication module. I’ll guide you through the process.    
         User:         OK, what should I do?    
                Assistant:               [check user manual]    
                Assistant:               Go to “Settings”, then “Connectivity”, then “Cellular” and then click “Restart module”. Confirm when asked.    
         User:         OK    
                Assistant:               [check the connection]    
                Assistant:               I can see the connection is restored. Can you confirm that? If not, I can schedule a service meeting for you.    
         User:         No need; it works now!          

In the last example, the Assistant guides the user using logs, the manual, and after checking real-time conditions. Without the LLM, the user would need to visit the dealer or at least use online support assistance.

Final words

With modern technology, we can bring some science fiction concepts to real life. Maybe WARP speed is not possible yet, but at least a car itself can give us solutions to problems recently solved only in much more expensive ways.

The beauty of it is – it’s really simple. Of course, there is a user-facing application to be made, the security to be ensured, and a bunch of functions to be implemented, but with modern connected cars, those elements are in place anyway. The tricky, novel part is a very simple application that uses the LLM as the brain power of the system.

As you can see the “  Attention is all you need ” paper that started the LLM revolution has allowed humanity to bring to life concepts present in our culture for decades. On the other hand, would this article have been ever written if its authors hadn’t watched the K.I.T.T. in their childhood? We will never know.