Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction

Understanding drivers’ decision-making is crucial for road safety. Although pre dicting the ego-vehicle’s path is valuable for driver-assistance systems, existing
methods mainly focus on external factors like other vehicles’ motions, often ne glecting the driver’s attention and intent. To address this gap, we infer the ego trajectory by integrating the driver’s gaze and the surrounding scene. We intro duce RouteFormer, a novel multimodal ego-trajectory prediction network com bining GPS data, environmental context, and driver field-of-view—comprising
first-person video and gaze fixations. We also present the Path Complexity Index
(PCI), a new metric for trajectory complexity that enables a more nuanced evalu ation of challenging scenarios. To tackle data scarcity and enhance diversity, we
introduce GEM, a comprehensive dataset of urban driving scenarios enriched with
synchronized driver field-of-view and gaze data. Extensive evaluations on GEM
and DR(eye)VE demonstrate that RouteFormer significantly outperforms state-of the-art methods, achieving notable improvements in prediction accuracy across
diverse conditions. Ablation studies reveal that incorporating driver field-of-view
data yields significantly better average displacement error, especially in challeng ing scenarios with high PCI scores, underscoring the importance of modeling
driver attention. All data and code is available at meakbiyik.github.io/routeformer.