Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise, Part 2

Challenge #2: Current Hadoop MapReduce implementations lack flexibility and reliable resource management

As outlined in Part 1 of this series, here at Platform, we’ve identified five significant challenges that we believe are currently hindering Hadoop MapReduce adoption in enterprise environments.  The second challenge, addressed here, is a lack of flexibility and resource management provided by the open source solutions currently on the market.

Current Hadoop MapReduce implementations derived from open source are not equipped to address the dynamic resource allocation required by various applications.  They are also susceptible to single points of failure on HDFS NameNode, as well as on JobTracker. As mentioned in Part 1,  these shortcomings are due to the fundamental architectural design in the open source implementation in which the job tracker is not separated from the resource manager.  As IT continues its transformation from a cost center to a service-oriented organization, the need for an enterprise–class platform capable of providing services for multiple lines of business will rise.  In order to support MapReduce applications running in a robust production environment, a runtime engine offering dynamic resource management (such as borrowing and lending capabilities) is critical for helping IT deliver its services to multiple business units while meeting their service level agreements.  Dynamic resource allocation capabilities promise to not only yield extremely high resource utilization but also eliminate IT silos, therefore bringing tangible ROI to enterprise IT data centers.

Equally important is high reliability. An enterprise-class MapReduce implementation must be highly reliable so there are no single points of failure. Some may argue that the existing solution in Hadoop MapReduce has shown very low rates of failure and therefore reliability is not of high importance.  However, our experience and long history of working with enterprise-class customers has proved that in mission critical environments, the cost of one failure is measured in millions of dollars and is in no way justifiable for the organization. Eliminating single points of failure could significantly minimize the downtime risk for IT. For many organizations, that translates to faster time to results and higher profits.

Is Platform HPC 3 Just a GUI around Linux?

Platform HPC 3 has a new skin, a brand new web interface that is modern and future looking. While we are emphasizing the usability of Platform HPC with this new product release, some people think Platform HPC is just a GUI around Linux. They couldn’t be more mistaken.

GNOME and KDE are graphical desktop interfaces for Linux, which are included in most Linux distributions. They make Linux easier to use. However, making Linux easier to use does not turn a bunch of servers into a cluster. A cluster that acts as a single system for users and administrators requires more capabilities from management software.

First, it requires a cluster management system. This allows administrators to easily manage a large number of nodes within a cluster without perming tasks node by node. Management tasks include installing the OS and software, patching the OS and upgrading applications, etc. on nodes. Cluster monitoring is also an important component of the cluster management. Without cluster monitoring capabilities, administrators would have to login to individual nodes to get cluster health and performance information, which is impractical for a cluster with more than 10 nodes. Using consolidated alerts can release administrators from having to closely watch the cluster all day long. Cluster management is the foundation that allows all nodes within a cluster to act like a single system from an administration perspective.

Second, a cluster requires workload management. This helps make a cluster into a single system for multiple users. Without workload management, a cluster is many discrete servers from user’s perspective. Instead, workload management helps run each user’s application instance as a “job”. It schedules jobs according user specified requirements and available resources. Workload management software automates fault tolerance and load balancing. It also turns multiple discrete commodity servers into a single reliable system.

Finally, a reporting system is essential for management so administrators can understand how cluster capabilities have been used and whether or not additional capabilities are required. It also shows how well users are being served using performance indicators like average job wait time, job run time, etc. This is to ensure a good return on investment for the valuable computer cluster.

There exist open source software solutions for cluster management, workload management, and reporting on the market today. However, nothing replaces a solution that integrates them together with an easy-to-use web interface while the command line interface is still there. This dramatically reduces system set-up time and the learning curve for users. It also helps users focus on their work rather dealing with complex cluster issues. For power users, the flexible command lines are still there for customization and extension. This is what Platform HPC 3 delivers. It is far more than just a nice GUI around Linux.