Confidence estimation in deep neural networks
Author: Daniel Hari
Mentor: izr. prof. dr. Matej Rojc univ. dipl. inž. el.
Co-mentor: Danilo Zimšek, univ. dipl. inž. tel.
Degree: 2.
Date: september, 2020
DKUM: DANIEL HARI
The master’s thesis presents approaches to evaluate trust in deep neural networks in the case of digit recognition
In the thesis, we focus mainly on two approaches, namely Bayesian training and Dropout layer. Bayesian learning is a more mathematically demanding procedure because it works by treating each input data into a neural network as a probability distribution and not as a deterministically determined value. In the Dropout layer technique, a stochastic Dropout layer is added behind each hidden layer of the network, so that the output from the model can be viewed as a random sample created from the a posteriori distribution. Such a procedure is less computationally demanding, but gives a similar result.
Confidence estimation in neural networks is an approach that provides information from a deep neural network about the certainty of recognition or perception.
Machine learning and neural networks have experienced a great boom in recent years. Confidence estimation in neural networks is an approach by which we obtain from a deep neural network information about the certainty of recognition or perception. With this additional information, we can improve the performance of the model by excluding from the final set of outputs those with a low level of certainty.
Intelligence is the ability to effectively solve problems in a creative way that is not pre-programmed, but is the ability to acquire knowledge and to adapt in way of solving problems. Intelligence is determined by the ability to adapt to changed circumstances, the ability to remember knowledge and its use, the ability to reason and abstract thinking, the ability to learn and to understand, thinking, the ability to evaluate and judge/decide, the ability to create original and creative thoughts. Intelligent behavior should include the following characteristics: perceiving the environment, working in demanding environments, learning and understanding from examples, reasoning to solve problems, acquiring hidden knowledge by reasoning, using knowledge in new circumstances, communicating with others and understanding languages, creativity, originality and curiosity.
Methods for determining neural network certainty
There are two methods for determining the certainty of the NN result, namely the Bayesian training procedure, and the Dropout layer procedure, which otherwise has several derivatives. As already mentioned, the sampling process with Dropout layers is computationally less demanding, but yields a similar result as the Bayesian training process. Bayes’ view on statistics uses an approach in which we treat unknown parameters as random variables, and we want the model to learn the parameters of the variables distribution. The distribution depends on what can be observed in the data on which one learns. During the Bayesian neural network training process, unknown model weights are adjusted based on what we know or can observe. This is a problem of inverse probability, and we solve it using Bayes’ theorem. NN building blocks consist of one input layer, one or more hidden layers, and an output layer. When we train a neural network or a model by updating weights, the network can also become too dependent on the data set we use in training. When the model then has to predict or classify unseen data, it will not give satisfactory results. This is known as overfitting. To solve this problem, we use a technique introduced in 2012 by Geoffrey Hinton. This technique is known as the Dropout layer process. In these processes, we use Dropout layers for sampling, but this technique is often also used to eliminate overfitting.
Experimental environment
Jupyter Notebook is an open source web application that can be used to create and share various documents. In the Jupyter Notebook we can enter text, program code, image, etc. The core is IPython, which allows us to write programs in the Python programming language. Keras is a high-level interface for programming applications for TensorFlow 2.0, and a highly productive interface for solving machine learning problems with an emphasis on modern deep learning. It includes important abstractions and building blocks for developing and sending machine learning solutions with high iteration speed. The scalability and performance of the TensorFlow 2.0 framework can be transferred to several platforms, namely: Keras can be run on a TPU, or on large clusters of GPUs, Keras models can also be exported to run in a browser or on a mobile phone. Keras contains many realizations of the most commonly used neural network block building blocks, such as layers, targets, activation functions, an optimization tool, and many tools to facilitate working with image and text data to simplify the coding needed to write deep neural network code. In addition to standard neural networks, Keras also has support for convolutional and feedback neural networks. It also supports other general layer uses, such as discharge layer techniques, package normalization, and aggregation.
The Bayesian training process
In the model, we first used the convolution network function (Conv2D), which performed the convolution over the input data. We repeated this twice. We then used the MaxPooling2D aggregation function, which is used to reduce the number of neurons. After this function, we also use the function of discharge layers. In this case, we used it to reduce the problem of overfitting. The flattening function aligned our inputs. Then we used the Dense function and again the Dropout layer function. The Dense function is related to the inner layer or deep learning, and is the most commonly used function, in addition to Dropout layers. The presented model was used in all procedures in the experimental part of the thesis.
Sampling procedures with Dropout layers
The model is similar to that of the Bayesian training process. After the convolution layer and the aggregation layer were repeated twice, the Dropout layers were added. We used it for sampling, and it is also included in the training phase. Flattening function aligns inputs. Then we use the Dense function, and again the Dropout layer function. The Dense function is related to the inner layer or deep learning and is the most commonly used function, in addition to Dropout layers.
Results
We focused on the approaches for estimating the probability of digit recognition and, with the help of the results of the procedures, improved the accuracy of the recognition. In which we have presented several possible ways. We found that the Bayesian procedure is very computationally demanding, which was also observed in the experimental work during processing, or recognition. We observed similar results in the Dropout layer process, but the training process was faster. We also found that the Monte Carlo procedure can be interpreted as a Bayesian approach, as it is based on probability distribution. When we compared the approaches with each other, we found that the procedures differed quite a bit in terms of accuracy. In general they all achieve an accuracy above 90%, which is very good. We also had a process of recognizing digits with convolutional neural networks, which even exceeded the ability of human recognition, which is usually between 97.5% and 98%.
Diskusija in rezultati
