Author: Karlo Crnek
Mentor: Matej Rojc
Comentor: Izidor Mlakar
Degree: Level 2 study program Electrical Engineering, course Electronics
Date: September, 2019
DKUM: KARLO CRNEK
The problem we are dealing with in this master’s thesis is lane detection on RGB images, i.e. images of the road in front of the vehicle during driving. To solve this problem, we have used “deep learning” techniques, specifically convolutional neural networks for the semantic segmentation task. We designed three different network architectures, which were trained on BDD100k dataset. Those network models were then evaluated and compared based on IoU metric used for semantic segmentation. We performed several experiments with the goal of improving IoU results and the generalization of the models. Finally, we tested the models on the Nvidia Jetson TX2 platform and proposed the possibilities for incorporating such models into the autonomous vehicle system.
In the master’s thesis we presented comprehensively how we can use convolutional CNN neural networks for semantic segmentation on the problem of lane detection for autonomous vehicles. The task presented is also a good starting point for further work on the design of dedicated lane detection architectures. The task also presents the integration options of the proposed solution on the integrated platforms of autonomous vehicles. Segmentation results have shown very good performance even under very challenging environmental conditions such as night-time footage and in different weather conditions. Of course, segmentation does not give satisfactory results in all cases, but it still provides a good basis for building a complex module of perception and integration with other sensors on an autonomous vehicle. Future work in this context could involve additional processing steps, such as: eliminating segmented pixels incorrectly in an isolated area, applying time consistency between sequential video frames to produce more robust results, etc.
The research in the thesis was also focused on improving the result of the selected metric – IoU, since we noticed that it is very difficult to objectively compare the results of the models and based on them, which one would be best used in autonomous driving. In the future, either a new benchmarking metric should be developed that would be more representative of the use of autonomous driving models, or we would first test the models on an autonomous driving simulator and then use them on a realistic platform.
We also tested different algorithms for optimization, regularization, and different levels of learning regarding the model learning approach. Of course, the ideal configuration of the model also depends on many other parameters. Problems arose with the use of the TensorRT tool, which is used to optimize models and, consequently, faster inference time. Nevertheless, we were able to build models that could meet the real-time specification depending on the speed of the camera. In our case, the U-Net and ENet models performed lane segmentation on the Nvidia Jetson TX2 platform between 14 fps and 18 fps. In the future, we intend to optimize such models to achieve faster inference times and more ease of use. This is also a very active area of research, since the goal is to apply deep learning models to less powerful devices such as cell phones or even microcontrollers. It is also worth noting that the use of models for real-time operation depends not only on the model’s closing time, but also on the processing before and after the model is completed. E.g. in our case it was necessary to reduce the dimensions of the input RGB image to a dimension of 256×256. If it is necessary to convert the segmentation results back to the original dimension after inference, the corresponding processing operations take time. Therefore, it is often necessary to find a compromise between the output image from the camera and the input image into the model itself.
We used the BDD100k dataset in the task, which is a very diverse data set and can be used for various perception tasks in autonomous driving. One of the options we did not consider in the assignment is transfer learning. Since we found it difficult to teach models with more data on the GPU platform used (in our case, 70,000 images), it might be useful to use a model already learned for some similar task, and then just refine the model on our data. In the end, we used 21,000 images to learn the U-Net model and also presented the results.
BQ portal
