Home

The general objective of this project is to design a system for continuously estimating the potential of dangerous situations during driving, focusing not only on the state of the external traffic, and not only on the state of the driver, but on the synergy of these factors. The external factors will be analyzed by new and improved perception algorithms, which will extract high level features from video streams and other generally available sensors (GPS, inertial sensors).
The state of the driver will be monitored by machine learning algorithms that are fed data provided by wearables, or by sensors that equip the car and show the driver’s actions (steering wheel angle, pedals, etc).
The project’s contributions will be focused on improved sensing of the driver’s and the traffic state, with emphasis on the critical details that define dangerous situations, and also on automatic learning and inference on the combination of these factors to actually detect and prevent dangerous situations.


Team

From the Technical University of Cluj-Napoca the following key persons:


Workplan

The workplan of the project is organized in three phases, corresponding to the calendar years covered.

Phase 1 (January 2021 – Dec 2021):

  • data acquisition system (software), driving data (data), initial prototype of environment perception system (software), components of the driver and vehicle monitoring system (software), data acquisition and annotation protocol (report), phase report (report)

Phase 2 (Jan 2022 – Dec 2022):

  • final prototype of environment perception system (software), initial prototype of driver and vehicle monitoring system (software), driving data (data), initial prototype of driver-vehicle-environment model (software), initial prototype of response model (software), articles and manuscripts (documents), phase report (report).

Phase 3 (Jan 2023 – Dec 2023):

  • final prototype of driver-vehicle-environment model(software), final prototype of response model (software), demonstrator application (software), articles (document), final report (report).


Publications

Scientific papers:
Phase 1
M. P. Muresan, S. Nedevschi, R. Danescu, “Robust Data Association using Fusion of Data-Driven and Engineered Features for Real Time Pedestrian Tracking in Thermal Images”, Sensors, Volum 21, Numar 23, Numar articol 8005, 2021. [ISI journal - Q1]
R. Itu and R. Danescu, "Object detection using part based semantic segmentation," 2021 IEEE 17th International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, 2021, pp. 227-231.
R. D. Brehar, C. C. Vancea, M. P. Mureşan, S. Nedevschi and R. Dănescu, "Pose Based Pedestrian Street Cross Action Recognition in Infrared Images," 2021 IEEE 17th International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, 2021, pp. 41-46.
M. P. Muresan, M. Raul, S. Nedevschi and R. Danescu, "Stereo and Mono Depth Estimation Fusion for an Improved and Fault Tolerant 3D Reconstruction," 2021 IEEE 17th International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, 2021, pp. 233-24.

Phase 2
R. Itu and R. Danescu, “Part-Based Obstacle Detection Using a Multiple Output Neural Network,” Sensors, vol. 22, no. 12, număr articol 4312, 2022. [ISI journal – Q1]
R. D. Brehar, R. O. Băbuţ, A. Fuzes and R. Dănescu, "Outdoor Traffic Scene Risk Estimation in the Context of Autonomous Driving," 2022 IEEE 18th International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, 2022, pp. 129-134.
M. P. Muresan, R. Schlanger, R. Danescu, S. Nedevschi, “Real-Time Obstacle Detection using a Pillar-based Representation and a Parallel Architecture on the GPU from LiDAR Measurements”, 2023 International Conference on Computer Vision Theory and Applications (VISAPP), 2023, pp. 779-787.

Phase 3
R. Brehar, G. Coblişan, A. Füzes, R. Dănescu, “Driver Attention Estimation Based on Temporal Sequence Classification of Distracting Contexts”, In Proceedings of the 20th International Conference on Informatics in Control, Automation and Robotics - Volume 1 (ICINCO), 2023, SciTePress, pp. 578-585.
R. Itu, R. Danescu, “On-Board Estimation of Vehicle Speed and the Need of Braking Using Convolutional Neural Networks”, In Proceedings of the 20th International Conference on Informatics in Control, Automation and Robotics - Volume 1 (ICINCO), 2023, SciTePress, pp. 600-607.
R. Itu, R. Danescu, “Predicting Emergency Braking in Vehicles Using a CNN with Sequential Image and Velocity Data”, 2023 IEEE 19th International Conference on Intelligent Computer Communication and Processing (ICCP), 2023.
R. Itu, R. Danescu, “Fully Convolutional Neural Network for Vehicle Speed and Emergency Brake Prediction”, Sensors, in review. [ISI journal]

PhD thesis:
Mircea Paul Mureșan, “Multimodal Measurement Approaches for Autonomous Systems”, supervisor Prof. Sergiu Nedevschi, 2023.

Reports

Phase 1 report here.

Phase 2 report here.

Final report here.


Results

Project results visualized


1. Detection of objects in images using convolutional neural networks

The images are processed by a U-Net convolutional neural network for semantic segmentation, and the network's output consists of four images encoding the component parts of each object: the four quarters, top left, bottom left, top right, bottom right. A geometric algorithm uses the relationship between these parts to identify the object both as a cuboid and as a set of pixels (pixel mask), thereby achieving instance segmentation.

Prediction multi-output.


2. Identification and tracking of objects in thermal images

Thermal images (IR - Infrared) are particularly useful for pedestrian detection, as they have a characteristic temperature that can be distinguished from the background both in cold and warm seasons. The images are processed using a neural network for instance segmentation (individual identification of each object, as opposed to semantic segmentation that identifies the type of object to which the pixel belongs). Subsequently, they are tracked in successive frames using the association between frames based on various similarity features.




3. Monitoring driver attention

Using neural networks, the orientation of the driver's head and the position of their eyes are continuously monitored. Based on these positions, a level of attention is inferred.

4. Determining the level of danger caused by pedestrians

Pedestrians are an unpredictable category of traffic participants who can cross our vehicle's trajectory at any time. Using a neural network for object detection of various categories, combined with a tracking algorithm, the likelihood of a pedestrian appearing in front of the car is estimated, and a level of danger is inferred from this information.

5. Prediction of the need for emergency braking

Using a sequence of 20 consecutive images along with the vehicle's speed associated with each of these images, a neural network will generate a signal indicating the need for emergency braking. The neural network is not a recurrent network. The 20 images will be combined into a single one, with the pixel depth being 20. This way, the braking signal can be predicted even if the objects causing the danger are not explicitly identified.

Prediction on test trip #1.


Prediction on test trip #2.



Contact

Radu Danescu

Address:
Technical University of Cluj-Napoca
Computer Science Department,
Str. Memorandumului, Nr. 28, 400 114, Cluj Napoca
Romania

Office: Baritiu str. 26, room 37
Phone: +40 264 401457

E-mail:
Radu.Danescu@cs.utcluj.ro



Full project name:
MEDALS - Modeling, Estimation and Management of Dangerous Situations through Continuous Analysis of the Driver-Vehicle-Environment System

Contract:
PN-III-P4-ID-PCE2020-1700

Funding:
Ministery of Research and Innovation, CNCS – UEFISCDI