Framework to Monitor Big Data Processing in the Cloud

Abstract

Big data is processed on the cloud as a series of Map and Reduce jobs. Often, huge chunks of data is pushed onto the cloud for processing, thereby overloading the processor. As a result, the jobs working on such huge chunks of data run for many days, making it quite difficult for the Technical Analysts to keep track of such jobs and to find the root cause of the delay in completing them.  This paper proposes an idea to build a framework and explains the implemented details to overcome the problems currently faced in cloud data processing, thus enhancing the data processing activities in the cloud. The purpose of this implementation is to make Hadoop Big data processing management simpler by developing web based application for provisioning, managing, and monitoring data processing activities on Cloudera apache Hadoop clusters.  This paper provides an intuitive, easy-to-use Hadoop Data processing monitoring management web dashboard backed by reporting framework to provide the collective view of the data as obtained from Job Trackers.