Self-driving car development has historically relied on a modular approach involving Perception, Localization, Planning, and Control. A decade later, companies have shifted toward End-To-End learning, replacing these modules with single neural networks to predict steering and acceleration.
The article questions whether Large Language Models (LLMs) can replicate these four pillars of autonomous driving. It notes that while current approaches are known, they do not yet solve the self-driving problem.
Before discussing applications, the text explains LLM fundamentals. These models convert input text into tokens, which can represent words, subwords, or characters depending on the model architecture.
Most LLMs utilize a Transformer encoder-decoder structure. The decoder performs next-word prediction by generating a series of tokens based on learned features and context from the encoder.
Adapting this to autonomous driving involves changing the input format. Instead of text, the system would process images, sensor data like LiDAR or RADAR point clouds, or algorithmic data such as lane lines and object detections.
Source: https://thegradient.pub/car-gpt/
