Linux Interacting with Windows HPC Server

There are many interesting technology showcases at SC’11 this week. One of the Birds of a Feather sessions this week will discuss a solution implemented at Virginia Tech Transportation Institute (VTTI) that mixes Linux with Windows HPC Server in a cluster for processing large amount of data.

Without a proper tool or a lot of practice, getting Linux and Windows to work together seamlessly to provide a unified interface for end users is a very challenging task. Having both systems coexist in an HPC cluster environment adds an order of magnitude of additional complexity compared to an already complex enough HPC Linux cluster.

This is because Windows and Linux “speak very different languages” in many areas such as user account management, file path and directory structure, cluster management practice, application integrations etc.

The good news is the Platform Computing engineering team did some heavy lifting in product development for this project. Platform HPC integrates with the full software stack required to run an HPC Linux cluster. Its major differentiator compared to alternative solutions is that Platform HPC is application aware. When adding Windows HPC Server into the HPC cluster, the solution delivered by Platform HPC ensures it provides a unified user experiences across Linux and Windows, and hides the difference and complexity between the two OSs.

Platform HPC team has developed a step by step guide for implementing an end-to-end solution with provisioning both Windows and Linux, unified user authentication, unified job scheduling, automated workload driven OS switch, application integrations, and unified end-user interfaces.

This solution significantly reduces the complexity of a mixed Windows and Linux cluster, so users can focus on their applications and their productive work, as opposed to managing the complexity of the mixed Windows and Linux cluster.

Wading throughthe Pre-SC’11 HPC News

The pre-SC11 dawn has already started to heat up with announcements being made by new vendors who see the promise of cloud computing aiding the needs of HPC users everywhere.

Just recently, a competitor of Platform Computing announced their entry into the “cloud bursting” space . They claim that their technology functions with all of the common workload management systems. Without much detail, the only conclusion that can be drawn from the announcement is that they have built a system to “poll – act – sleep” in a loop. While possibly illustrative of the promise of cloud computing, this simplistic view ignores what we believe is the most fundamental challenge to cloud bursting – data locality.

By “data locality” we refer to the fact that compute resources must have some access to data for them to be useful. When datasets get large (input or output), getting access to globally distributed compute resources may have dubious value until workload schedulers understand what data exists where and can affect transfers of data between localities to take better advantage of those remote resources.

Separately, we are very pleased to see others in the HPC industry focusing on the ease of use / ease of build idea. In fact just recently, Rightscale made an announcement to this effect for building clusters in the cloud Platform has been talking about the importance of this for some time, namely with our Platform HPC product. Stay tuned for more announcements as we take this same story into the cloud.

Finally, we were also very happy to see that HPC as a service continues to gain momentum. Just recently there was a story from the Netherlands where a small cluster of very “thick” servers running KVM have been used to create a serviced based HPC infrastructure to be rented to various researchers in the academic and government communities.

No doubt these announcements will be just the beginning of a coming onslaught from many vendors waiting to announce during the week. We expect that cloud and HPC will take one step closer together than they were last year in New Orleans. Stay tuned to the Platform blogs, we’ll provide a summary of the show and any key items we hear about.

Big Data’s Big Show at SC’11

It’s less than a week away, everyone in the HPC community are drumming up for SC’11. As someone who has been to SC for the past seven years, I was pleased to see that Big Data appears to be the  new kid on the block this year. Roughly 20 sessions will be dedicated to Big Data related topics at this year’s Supercomputing show. From basic training on Hadoop and MapReduce to the challenges and opportunities for exascale data analytics, we will hear wide range of discussions on  Big Data.  Platform Computing will also be hosting a breakfast briefing (free!) on how to overcome your MapReduce barriers in the morning of Wed., Nov 16 at the show. Registration details can be found here.

Traditionally, hot topics in HPC are often around performance, scaling, latency and bandwidth. It’s only been in the last couple of years that data intensive computing has become an area of interest in HPC, and it is heating up quickly! The reality is, now that hardware is getting faster and cheaper - thanks mainly to the advancement in processor technologies -- users can run more problems faster. As a result, more data is being generated and a lot of that data contains tremendous insights we could utilize to make better products and decisions. Sure, Big Data exists in Web 2.0, retail, telco companies as well as many other verticals in the enterprise space, yet there is no shortage of use cases in HPC. Areas such as cybersecurity, fraud detection, next-gen sequencing analysis are just a few such examples that fall into HPC arena – often applications in these space are both computationally and data intensive. 

HPC has long been considered the incubator for many bleeding edge innovations that will later trickle down and benefit mainstream applications. We believe Big Data is no exception. The annual Supercomputing show is always about showcasing leading edge science and technologies, Big Data certainly fits the bill.  I am looking forward to learning about new solutions for Big Data problems developed in HPC, as well as getting a better understanding of the specific requirements for this particular market. It will be an exciting week ahead, and we expect to hear some big buzz around Big Data at SC’11!

 

Stop by and visit Platform Computing at SC11 in booth #1117!

Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise, Part 5

Challenge #5: Lack of Multiple Data Source Support

With this blog entry, we have reached the final chapter of the Hadoop challenge series. In this blog, I am going to discuss the fifth challenge for current Hadoop implementations – the lack of multiple data source support.

The current Hadoop implementation does not support multiple data sources; instead, it supports only one distributed file system at a time, the default being the Hadoop Distributed File System (HDFS). This restriction creates barriers for organizations whose data is stored in different file systems other than HDFS while implementing MapReduce applications in Hadoop.  For non-HDFS users, enabling MapReduce applications running in Hadoop environment means they have to first move their data from the current file system into HDFS, which can turn into a timing consuming and very expensive operation.  This limited capability for heterogeneous data support also leads to inferior performance and poor resource utilization due to an inflexible infrastructure.

The reality is, users want a platform that 1) supports various types of input data at the same time and outputs to their desired data sources (which could be different from the input type); 2) completes the   data conversion at runtime so no extra ETL steps are needed after the run.  (See the chart below for a high-level architecture layout for heterogeneous data support.) For instance, a user can send his input data stored in HDFS and output to a relational database such as Oracle and MySQL upon the completion of the run.  Such capabilities eliminate the data movement at both the beginning and the final stages of a MapReduce run, therefore, dramatically reducing the cost while improving the operation efficiency and driving   faster time to results. 

A high level diagram on heterogeneous data support

This capability of  heterogeneous data support in runtime can be considered as an alternative approach to traditional ETL function. The advantage of the former, while compared with the existing ETL tools, is that it provides a faster, cheaper and integrated new platform for users running Big Data applications.   

Having identified the 5 major challenges in the current Hadoop MapReduce implementation, we at Platform Computing has developed a solution – Platform MapReduce, an enterprise-class distributed runtime engine to not only address those barriers mentioned in this blog series, but also bring additional capabilities requested by users wanting to move their Big Data applications into production.  For detailed features and benefits delivered by Platform MapReduce, please visit: http://www.platform.com/mapreduce

Please join us at SC11 for a free breakfast briefing: “Overcoming Your MapReduce Barriers”.  Register today to secure your spot!

Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise, Part 4

Challenge #4: Lack of Quality of Service

We are back after a short break.  The challenge with the current implementation of Hadoop MapReduce continues.  In this blog let’s take a look at the fourth challenge in the existing Hadoop stack – the lack of quality of service.

By high quality of service, we are referring to the capability of dynamically allocating available IT infrastructure based on workloads requirements, maximizing resource utilization and preventing silos. Those capabilities lead to better application performance and faster time to results, and therefore, provide high return on investment for the IT organization.  The current architecture design of the existing open source Hadoop stack puts limitations on the above capabilities. As mentioned in part 2 of this blog series, the single job tracker in the current Hadoop implementation is not separated from the resource manager, so as a result, the job tracker does not provide sufficient resource management functionalities to allow dynamic lending and borrowing of available IT resources. This creates a static IT environment in which each Hadoop application can only run on a pre-assigned set of resources at a given time and no exceptions are allowed. As the requirements of the application changes, resources will need to be re-configured manually to meet new demand. Such a static IT infrastructure brings the following issues for an IT organization:

·         Unable to provide the necessary and guaranteed services to multiple lines of businesses

·         Unable to manage real-time workload requirements

·         Slower performance and time to discovery

·         Increased management complexity

The result?  Underutilized resources and a higher total cost of ownership for IT.

In contrast to a static IT infrastructure, a sophisticated runtime built on a service oriented architecture (SOA) evolution.  brings quality of service to IT organizations committed to providing high quality services to their internal and/or external clients. Such a runtime solution will help transform IT into a true service provider and help meet demanding requirements (-high availability, dynamic resource allocations, ease of management, etc.) in the production environment. As new technologies such as Hadoop and MapReduce continue their penetration into the mainstream market, more applications will be developed and moved into production. Quality of service will undoubtedly become a critical consideration for IT in the next wave of the Big Data

Please join us at SC11 for a free breakfast briefing: “Overcoming Your MapReduce Barriers”.  Register today to secure your spot!