Motion Forecasting for Autonomous Driving

Anticipating the future movements of dynamic traffic agents is a critical link between environment perception and safe path planning in autonomous driving. This research project, developed as part of the Automated and Connected Driving Challenges initiative (09/2026), focuses on trajectory prediction using the large-scale Argoverse 2 (AV2) Motion Forecasting dataset.

Our team tackled this complex problem by building a suite of models, transitioning from basic physics-based kinematic baselines to advanced interaction-aware and map-aware deep learning architectures.

My Contribution: Pedestrian Motion Forecasting

Traffic participants operate under fundamentally different dynamics; unlike vehicles that follow structured lane geometry, pedestrians exhibit short-range, highly irregular movements and require a tighter, localized spatial context. To address this, I spearheaded the design and development of a pedestrian-specific Map-Aware Social-LSTM model, building foundational architectural inspiration from hierarchical trajectory modeling SS-LSTM and social pooling frameworks Social-LSTM.

• Architectural Design: The model utilizes a dual-branch design that explicitly decomposes social and scene information into dedicated encoders, drawing conceptual inspiration from Social-Scene LSTM hierarchical architectures SS-LSTM and Social-LSTM pooling strategies Social-LSTM.
• Context Integration: It combines grid-based social pooling; aggregating the hidden states of neighboring agents to capture human-human interactions, with hand-crafted map features over a 2-second history and forecasting horizon.
• Performance Metrics: Evaluated against 19,603 pedestrian validation scenes, the model achieved highly accurate forecasts, logging an Average Displacement Error (ADE) of 0.2187 m and a Final Displacement Error (FDE) of 0.5107 m.
• Reliability: The model successfully secured a 23.67% miss rate at a strict 0.8 m threshold.

Technical Stack & Compute Environment

• Frameworks & Libraries: Built primarily in PyTorch (v2.8.0), leveraging the Argoverse 2 API (av2), pandas, pyarrow, and NumPy for data manipulation.
• Hardware: Developed, trained, and validated on an institute GPU server equipped with an NVIDIA A40 (47.7 GB VRAM) running CUDA 12.8.
• Data Pipeline: Engineered multi-worker DataLoaders and parallel map preprocessing scripts to convert 11-second multi-agent raw scenarios into focal-centric spatial matrices for high-speed GPU transfer.

Results

While the architecture successfully fused social and map context, evaluating its true real-world viability revealed both strong predictive capabilities in structured environments and critical limitations in complex edge cases.

Successful Trajectory Forecasting

In scenarios with clear spatial constraints and observable intent, the model accurately captured distinct pedestrian maneuvers:

Architectural and Data-Driven Limitations

Despite these successes, edge-case analysis exposed scenarios where the prediction collapsed, driven primarily by either architectural rigidity or underlying data quality issues.

1. Architectural Constraints: The Purple Pedestrian Failure

   In the complex interaction shown below, the model fails to predict the true path of the purple pedestrian.

   This failure highlights two specific architectural flaws:

   • Overly Rigid Social Repulsion: The grid-based social pooling forces the purple pedestrian to maintain a parallel path, effectively preventing it from cutting across the adjacent blue pedestrian to avoid a perceived, mathematically heavily-penalized collision.
   • Map Context Dilution: Using mean pooling for the map context geographically blurs the distinct boundary of the crosswalk. Consequently, the attractive signal of the crosswalk becomes too weak to override the aggressive social repulsion. Implementing a weighted average or max pooling mechanism could have preserved this critical map feature better.

2. Data Limitations: Missing History

   A significant portion of prediction errors stemmed directly from imperfect sensor recordings.

   During preprocessing, tracks with missing frames were padded with the latest available positional data to maintain tensor dimensions. If a pedestrian’s history was entirely truncated before the prediction threshold, the padding forced the model to perceive the agent as stationary. Consequently, the model predicts a stationary future, completely failing to capture the sudden, unrecorded onset of movement.

3. Sensor Noise and the ADE/FDE Metric Fallacy

   Finally, irregular trajectories, whether caused by erratic human movement or sensor tracking noise-frequently caused the prediction to collapse, as there were no discernible kinematic patterns for the LSTM to latch onto.

   Crucially, the example above exposes a fundamental flaw in how motion forecasting is evaluated. Despite the prediction collapsing visually and behaviorally, the Average Displacement Error (ADE) and Final Displacement Error (FDE) values for this plot remain misleadingly low. Because human motion operates within a much smaller spatial radius than vehicle motion, absolute error values in meters inherently stay small. This confirms that while ADE and FDE are standard benchmarks, they are heavily flawed metrics for evaluating the true behavioral accuracy and safety viability of pedestrian trajectory predictions.

The Broader Team Effort

While I led the pedestrian architecture, this project was a highly collaborative effort alongside my teammates, Ishita Jain and Manish Rajasekar. They focused on the agent-agnostic models designed for vehicles and other dynamic agents.

Their work progressed from Constant Velocity (CV) and Constant Acceleration (CA) baselines to a socially-aware LSTM utilizing multi-head cross-attention. The team’s final multimodal architecture combined this cross-attention social pooling with a 1D-CNN encoder processing the nearest 16 map lane centerlines to output six diverse future trajectories per agent. Together, our combined research supported the hypothesis that integrating explicit map context resolves significant ambiguity in agent intent.

Future Work & Links

Future iterations of this research will extend the analysis beyond LSTM backbones to explore state-of-the-art transformer-based vectorized models, such as HiVT and QCNet, which currently define the benchmark standards on the Argoverse dataset.