DRIFT: Detachable Reasoning for Runtime-Flexible and Efficient End-to-End Driving

Emad Aghajanzadeh1,2,*, Bethany Anne Wong1,2,*†, Leonhard Kunze1,2,†, Moritz Thoma2
Maximilian Forstenhäusler2,3, Nael Fasfous2,†, Manoj Rohit Vemparala2, Pierpaolo Mori2
Christos Anagnostopoulos3, Daniel Mueller-Gritschneder4, Ulf Schlichtmann1

1Technical University of Munich   2BMW Group   3University of Glasgow   4TU Wien

* Equal contribution. † Employed at BMW at time of submission.

Abstract

The integration of vision-language models (VLMs) into end-to-end (E2E) autonomous driving has demonstrated strong potential by providing scene description and reasoning capabilities. However, VLMs are computationally expensive, and switching language reasoning on and off at runtime without hurting closed-loop performance has not been demonstrated, which hinders practical adoption. To address these issues, we propose DRIFT, an E2E autonomous driving framework that integrates a lightweight language model (LM) for visual question answering and action commentary. By efficiently projecting intermediate driving representations into the language embedding space, we demonstrate that a compact LM (<150M parameters) can achieve strong scene description and action commentary. Furthermore, we introduce a runtime-flexible planning module that is trained on surrogate reasoning features distilled from the LM and can be conditioned on either the LM or the surrogate reasoning features at inference. This design enables bypassing the LM entirely when required by deployment constraints, without sacrificing driving performance. With only 255M parameters, DRIFT reaches up to a Driving Score (DS) of 97.03 and a Success Rate of 89.39% (mean over three runs) on Bench2Drive by interleaving its LM and surrogate reasoning at a chosen reasoning rate. This is on par with the strongest non-language models while being far more compact than existing VLM-based models. DRIFT further achieves competitive results on the real-world NAVSIM and AlpaSim benchmarks at significantly lower computational cost than existing VLM-based approaches.

Architecture

DRIFT architecture: a backbone feeds a planner, with a language model that can be attached or detached at runtime and a surrogate reasoning projection that replaces it.
Overview of the DRIFT architecture. The language model can be attached or detached at runtime; the planner is conditioned on either the LM reasoning features or the surrogate reasoning features.

Results

Bench2Drive: closed-loop driving performance

MethodParametersFramesDS ↑SR (%) ↑
Language-based models
DriveMoE [39]3B234K74.2248.64
ETA [10]7B234K74.33±0.9648.33±3.56
VLR-Driver [17]8.1B234K75.0150.00
ORION [8]7B234K77.7454.62
SpaceDrive+ [19]7.4B–78.0255.11
AutoVLA [45]3B600K78.8457.73
MindDrive-L [9]3B234K80.5958.26
SimLingo [31]1B3.1M85.07±0.9567.27±2.11
VLADriver-RAG [42]–234K89.1270.42
LinkVLA [35]1B–91.0174.55
Non-language models
SimLingo-BASE [31]355M3.1M85.9466.82
TFv6 [27]126M1M95.2±0.386.8±0.7
BridgeDrive [23]–1M96.34±0.5589.25±0.50
DRIFT w/o LM120M1M95.97±0.6586.52±1.60
DRIFT w/ LM255M1M97.03±0.0989.39±0.95

DRIFT results are mean ± standard deviation over three closed-loop runs. DRIFT w/ LM uses a reasoning rate of 2. The paper notes that training data sizes differ across methods, so the numbers are not strictly comparable.

Language evaluation

DatasetModelBLEU-4 ↑SPICE ↑CIDEr ↑ROUGE-L ↑METEOR ↑
DriveLM-CARLADriveLM-CARLA (LLaMA-7B)–72.711.1571.20–
DriveLM-CARLASimLingo (Qwen2-0.5B)72.0787.247.0884.7750.57
DriveLM-CARLAOurs (SmolLM2-135M)72.6385.566.9684.5950.08
CommentaryOurs (SmolLM2-135M)83.9082.206.9389.1057.01

Real-world benchmarks

MethodParametersPDMS ↑EPDMS ↑AlpaSim score ↑
FSDrive [40]2B85.1––
AdaThinkDrive [25]8B86.2––
FLARE [37]4B86.9––
SGDrive [18]2B87.427.1–
Qwen-Drive-1.0, SFT [44]4B87.930.591.7
LTFv6 (reported) [27]69M85.428.3–
LTFv6 RegNetY-032‡117M87.030.288.6
DRIFT w/o LM‡125M87.432.589.5
DRIFT w/ LM‡260M87.332.889.5

PDMS is reported on NAVSIM navtest; EPDMS on NAVSIM navhard; AlpaSim score is on the nuPlan track. ‡ Test-time heading post-processing is applied. Table 5 reports 260M parameters for DRIFT w/ LM.

Examples

Example visual question answering and action commentary on NAVSIM and CARLA scenes.
Example questions and answers on NAVSIM (left) and CARLA (right).