PCICF: a Pedestrian Crossing Identification and Classification Framework

We have recently observed the commercial roll-out of robotaxis in various countries, including the USA and Germany. They are deployed within an operational design domain (ODD) on specific routes and environmental conditions, and are subject to continuous monitoring to regain control in safety-critical traffic situations. Since ODDs typically cover urban areas, robotaxis must reliably detect and interact with vulnerable road users (VRUs) such as pedestrians, bicyclists, and e-scooter riders. To better handle such varied traffic situations, end-to-end AI, which directly computes vehicle control actions from multi-modal sensor data instead of only for perception, is on the rise. High quality data is needed to systematically train and evaluate such systems within their ODD. In this work, we propose PCICF, a framework to systematically identify and classify VRU situations to support ODD’s incident analysis. We base our work on the existing synthetic dataset SMIRK, and enhance it by extending its single-pedestrian-only design into the MoreSMIRK dataset, a structured dictionary of multi-pedestrian crossing situations constructed systematically. We then use spacefilling curves (SFCs) to transform multi-dimensional features of scenarios into characteristic patterns, which we match with corresponding entries in MoreSMIRK. We evaluate PCICF with the large real-world dataset PIE, which contains more than 150 manually annotated pedestrian crossing videos. We show that PCICF can successfully identify and classify complex pedestrian crossing situations, even when groups of pedestrians merge or split during their crossing. By leveraging computationally efficient components like SFCs, PCICF has also potential to be used onboard of robotaxis for out-of-distribution (OOD) detection, for example. We share an opensource replication package for PCICF, including its algorithms, the complete MoreSMIRK dataset and dictionary, and our experiment results, available at https://github.com/Claud1234/PCICF.