An outlier detection method to improve gathered datasets for network behavior analysis in IoT

Abstract

Outlier detection is a subfield of data mining to determine data points that notably deviate from the rest of a dataset. Their deviation can indicate that these data points are generated by errors and should therefore be removed or repaired. There are many reasons for outliers in a network dataset such as human or instrument errors, noise or system behavior changes. On the other side, Network Behavior Analysis (NBA) is a way to monitor traffic and recognize unusual actions in a network. Analyzing data trends in NBA methods is a common way to interpret network situation. Outliers can deviate and produce erroneous trends that influence the results of the NBA methods. This paper presents an approach that based on a method for trend detection divides the data set into subsets where contextual outliers are discovered. The outliers can then be removed to have a clear dataset that better shows the network behavior when using NBA methods. Increasing the accuracy and reliability are the goals of our method. We compare the proposed method with the Hampel method on simulated IoT network data.