A Hybrid Scheduling Approach for Scalable Heterogeneous Hadoop Systems

The scalability of Cloud infrastructures has significantly increased their applicability. Hadoop, which works based on a Map Reduce model, provides for efficient processing of Big Data. This solution is being used widely by most Cloud providers. Hadoop schedulers are critical elements for providing desired performance levels. A scheduler assigns Map Reduce tasks to Hadoop resources. There is a considerable challenge to schedule the growing number of tasks and resources in a scalable manner. Moreover, the potential heterogeneous nature of deployed Hadoop systems tends to increase this challenge. This paper analyzes the performance of widely used Hadoop schedulers including FIFO and Fair sharing and compares them with the COSHH (Classification and Optimization based Scheduler for Heterogeneous Hadoop) scheduler, which has been developed by the authors. Based on our insights, a hybrid solution is introduced, which selects appropriate scheduling algorithms for scalable and heterogeneous Hadoop systems with respect to the number of incoming jobs and available resources.