Data Channel Integration in Hadoop Environments

Abstract

In the field of distributed computing is growing and speedily becoming a natural part of large as well as smaller enterprises IT processes. Driving the progress is the cost efficiency of distributed systems compared to centralized options, the physical limitations of single machinery and reliability concerns. There are frameworks within the field which aims to create a standardized platform to facilitate the progress and implementation of distributed services and applications. Apache Hadoop is one of those papers. Hadoop is a framework for distributed processing and data storage. It contains support for many different modules for different purposes such as Distributed database management, safety, data streaming and processing. In calculation to offering storage much cheaper than traditional centralized relation database, Hadoop chains powerful methods of handling very large amounts of data as it streams through and is stored on the system. These methods are widely used for all kinds of big data dealing out in large IT companies with a need for low-latency, high-throughput processing of the data. More and more companies are looking towards implementing Hadoop in their IT process; one of them is Unomaly, a company which offers agnostic, proactive anomaly detection. The anomaly detection system analyses system logs to detect discrepancies. The anomaly finding system is reliant on large amounts of data to build an exact image of the target system. Integration with Hadoop would result in the possibility to consume incredibly large amounts of data as it is streamed to the Hadoop storage or other parts of the system.

In this degree paper an integration layer application has been developed to allow Hadoop integration with Unomalys system. Research has been conducted throughout the paper in order to determine the best way of implement the integration. The first part of the result of the paper is a PoC application for real time data channel between Hadoop clusters and the Unomaly system. The second part is a recommendation of how the integration should be designed, based on the studies conducted in the paper work.