Driving Scene Understanding using Spiking Neural Networks

Abstract

One of the applications of AI lies in developing intelligent systems for safe on-road driving, other than building and perfecting self-driving vehicles, and many others. Driving Scene Understanding (DSU) is one such area where AI algorithms can be used to infer the current actions of driver, pedestrians, nearby vehicles, etc. to improve the on-road decision making capability of the in-vehicle driver. Another related front of technological advancement in transportation is the production and development of electric vehicles. A future with battery electric vehicle and safe driving necessitates the creation of AI algorithms which not only assist in increasing the on-road safety but are also energy efficient.

This thesis is an attempt towards developing such an energy efficient AI model for DSU using Spiking Neural Networks (SNNs). Low power neuromorphic hardware (e.g. Intel’s Loihi) can be leveraged for the deployment of such SNNs which offer low inference latency and energy efficiency. Out of a number of ways to build SNNs, an established method is to first train an Artificial Neural Network (ANN) with traditional neurons (e.g. ReLU) and then replace those neurons with spiking neurons (e.g. Integrate & Fire neurons) along with some other network modifications. Therefore, Chapter 4 first presents a 3D-CNN based ANN model, and identifies the appropriate spatial resolution and temporal depth of the incoming video frames for DSU. Through extensive experiments, it was found that MaxPooling performs better than AveragePooling in such a 3D-CNNs based model; however there exists no method to convert a network with MaxPooling layers into an SNN which can be entirely deployed on a specialized neuromorphic hardware.

Chapter 6 presents two novel approaches to implement MaxPooling on a neuromorphic hardware; thus facilitating the conversion of networks with MaxPooling layers to fully neuromorphic-hardware compatible SNNs. These approaches have been tested with 2D-CNNs based SNNs for image recognition, and can be extended to the 3D-CNNs based SNNs as well; thus, theoretically realizing an energy efficient SNN for DSU.