Skip to content

NextGen MapReduce

by Vaibhav Aggarwal on March 5th, 2012

Hadoop 0.23 delivers the new version of MapReduce called YARN. It has some significant differences from old MapReduce.

1. Introduction of Resource Manager responsible for managing and assigning global cluster resources.

2. Introduction of per application Application Master. The application master interacts with resource manager to request compute resources.

3. Introduction of Node Manager responsible for managing user processes per node.

In earlier version of Hadoop the JobTracker was closely tied to MapReduce framework. It was responsible for both resource management and application management. JobTracker would allow running Hadoop MapReduce jobs only. The new resource manager allows running other services like MPI within the same cluster via Application Master.

In old Hadoop Map and reduce slots could not be interchangeably used. This would mean that the cluster would be largely underutilized during a pure map or reduce phase. In the newer avatar of Hadoop slots can be reused there is much better resource utilization.

 

From → Hadoop

No comments yet

Leave a Reply

Note: XHTML is allowed. Your email address will never be published.

Subscribe to this comment feed via RSS