NIM: Scalable Distributed Stream Processing System on Mobile Network Data

Abstract

The amount of 3G MBB data has grown from 15 to 20 times in the past two years. Thus, real-time processing of these data is becoming increasingly necessary. The overhead of storage and file transfer to HDFS, delay in processing, and etc make off-line analysis inefficient. Analysis of these datasets are non-trivial, examples include personal recommendation, anomaly detection, and fault diagnosis. We describe NIM - Network Intelligence Miner, which is a scalable and elastic streaming solution that analyzes MBB statistics and traffic patterns in real-time, and provides information for real-time decision making. The design and the unique features (e.g., balanced data grouping, aging strategy) of NIM help not only the network data analysis tasks but also other applications like Intelligent Transportation System (ITS), etc.