HPC from A-Z (part 19) - S

S is for Space exploration!

Are we alone in the cosmos? What is dark matter? What is the universe expanding into? The nature of the cosmos has fascinated us earthlings since we first looked up at the sky and began to wonder. We’ve come a long way since then. We've ditched the loin cloths, created the telescope and even set foot on the moon – but there are many questions which remain unanswered and HPC can be used to help to answer them.

In 1999, a project called SETI@home was set up to search for signs of intelligent life in the universe using volunteers’ idle computers. Back then, SETI borrowed cycles from these computers across the Internet, using their compute resources to analyse intergalactic data. Now while SETI@home didn’t quite manage to find E.T, it’s just one great example of how HPC can benefit space exploration. Put a little more thought into it though, and HPC could be used for a whole lot more; everything from crunching telescope data, analysing rocket and shuttle stability, to scrutinising far away galaxies which could host potential alien life forms.

Whatever the next great step for mankind is, you can bet that HPC will be somehow involved.

Is Platform Cluster Manager just commercial support of Kusu?

Many people think Platform Cluster Manager is just a commercial version of the open source software, Kusu. When Kusu was born a few years ago, that actually was the case. Since then, Platform Cluster Manager has evolved significantly, and by comparing the latest release of Platform Cluster Manager with Kusu today, one will find there are many differences now.

Kusu is open source cluster provisioning and management software developed by Platform Computing. The commercially supported open source version was initially called Open Cluster Stack (OCS), and that name was later changed to Platform Cluster Manager. In version 2.0, which was released in early 2010, we started to package proprietary code into Platform Cluster Manager for a better interface. In subsequent releases since then, more proprietary code has been added to Platform Cluster Manager. In the latest version 3.0, the original Kusu code is now just small part of the Platform Cluster Manager product. Installation, a graphical web interface, and the monitoring system are all proprietary code that have been added into the product. Although the Kusu code has gained enhancements for functionality and reliability release by release, Platform Cluster Manager 3 just uses Kusu to power its provisioning engine. The rest of the product’s functionality is not open source any more.

Today, Platform Cluster Manager as a licensed product is sold by many Platform Computing channel partners. It contains the following functional modules:

1. High quality and flexible open source provisioning engine developed by Platform Computing.
2. Web interface framework shared by most Platform Computing products
3. Web interface for cluster management
4. Monitoring framework based on reliable and scalable agent technology used in Platform LSF
5. Installer that supports un-attended or factory install

Some key new features added to Platform Cluster Manager 3 include:

1. An intuitive web interface
2. Management node high availability
3. Support of NIC bonding
4. Monitoring and alerting
5. More flexible network interface types
6. Enhanced kit building process

Platform Cluster Manager is also included in Platform HPC as the cluster management tool. It is used for deploying and managing the additional software modules in Platform HPC. With one package and one web interface, customers can easily expand their management functionalities from Platform Cluster Manager to Platform HPC by just adding Platform HPC licenses.

With the addition of these new features over the last two years, Platform Cluster Manager has truly evolved from a community supported open source package to a commercial grade product.

Platform MapReduce: Tackling Big Data, One Enterprise at a Time

“Big Data” seems to be on the tip of everyone’s tongue in recent months, and here at the Platform this has been no exception. Applying MapReduce applications to the data deluge has so much potential, but in Derrick Harris’ sage words “Hadoop may be hot, but it needs to be useful” (source: GigaOM).  With this in mind, Platform has set itself to applying its 18-year experience in policy-driven workload scheduling to the development of a MapReduce solution ready for the enterprise and to tackle its unique challenges. Back in March, Platform strongly hinted at a forthcoming product, but now it’s official.

Platform announced the launch of Platform MapReduce, the industry’s first enterprise-class, distributed runtime engine for MapReduce applications, with general availability to come at the end of July. The new solution will be able to manage MapReduce applications in a cluster (even multiple applications on a shared cluster) across an entire distributed file system. With more than 10,000 policy levels and support for up to 300,000 concurrent tasks, Platform MapReduce provides unparalleled manageability and scale, while ensuring high resource utilization to maximize ROI. Applicable to industries across multiple sectors, these key features can enable such diverse functions as compliance and regulatory reporting for financial services and government agencies; customer churn prevention for telecommunications; and genome sequencing analysis for life sciences.

Platform MapReduce also supports open distributed file system architecture, including immediate support for Hadoop Distributed File System (HDFS) and Appistry Cloud IQ – with more to come! To ensure that open source solutions, in this case those used with the Platform MapReduce distributed runtime engine, receive the world-class support enterprise customers demand, Platform has also signed the Apache Corporate Contributor License Agreement to contribute to the development of Apache-based, open-source Hadoop Distributed File System (HDFS).

With Platform MapReduce and world-class support, the enterprise is now ready to tackle the data deluge!

The Experience of Building a Scalable Supercomputer

This week, the TOP500 ranked the system at Taiwan’s National Center for High Performance Computing as #42 in their June 2011 bi-annual supercomputer list. This system was provided by Acer together with its technology partners, AMD, DataDirect Networks, QLogic, and Platform Computing. Platform Computing provided the management software and MPI libraries for the system, as well as services for deploying these software components.

During the period of system installation and configuration, a number of areas demonstrated the advantages of partnering with Platform Computing:

(1) Management software: Platform HPC was chosen to manage the system. The scalability and maturity of the software components simplified the installation and the configuration of the management software layer. Both the workload scheduler (based on Platform LSF) and MPI library (Platform MPI) on the system scale effortlessly.

(2) MPI expertise: To achieve maximum Linpack performance results, it is critical to ensure MPI performance is optimized. During the installation and configuration stage, the Platform MPI development team provided numerous best practices to help maximize the benchmarking results, from checking cluster healthiness to MPI performance tuning. They collaborated closely with developers from QLogic, who provided Infiniband interconnects.

(3) Dynamic zoning: The system will be used by multiple research user groups. There is a separate workload management instance for each user group. Based on the workload of each user group, the size of the workload management zone will change from time to time. Each zone has its own user account management system and scheduling policies. Platform HPC was set up to easily manages such dynamic configuration changes.

The maturity of Platform HPC, as well as the expertise from Platform Computing’s development and services teams played a key role in ensuring the success of this Acer project. The maximized performance and stability of the benchmarking runs enabled the results to be submitted in time for the June TOP500 list. But mostly importantly, when the system is in hands of hundreds of users in production, the robustness of the workload management, the performance of MPI, as well as the support from experts who built the software will make a difference in delivering the quality of services from this top Taiwanese supercomputer.

HPC from A-Z (part 18) - R

R is for Reservoir modelling

It might just look like thin brown treacle to you and I, but crude oil is a big money business.

Millions of years worth of pressure under the earth’s surface has turned the tiny plants and animals of prehistoric Earth into the modern world’s most valuable resource – powering vehicles, industries and economies across the globe. As such, the financial rewards for finding and trading in oil are substantial. However, when you’re using millions of pounds worth of equipment including a 30ft drill to bore holes into the planet’s crust then equally, so are the risks. Choosing the wrong spot to drill can be an expensive mistake.

StatoilHydro ASA, a Norway-based oil and gas company, is one of the world’s largest crude oil traders. It relies on sophisticated 3D simulation programmes to search for natural oil-wells in the Earth’s crust -- if you want to strike it rich you need to be drilling in the right place. I don’t need to tell you that this process involves vast amounts of data, large numbers of complex calculations and requires thousands of iterations to produce accurate results. To put it simply, it’s a very, very big job.

To ensure StatoilHydro had the required resources to power such colossal calculations it installed an HPC environment, which is now invaluable to its reservoir engineers worldwide. It allows its users to run significantly more simulations which in turn means for much greater accuracy when drilling.

And accuracy is important when only a small error in location can cost many millions of dollars. This isn’t ‘pin the tail on the donkey’ – it’s an exact science.