Hadoop MapReduce low latency matters! BGI Shenzen wins IDC Innovation Award


Last week IDC released the winners for the HPC Innovation Excellence Awards.

As per IDC’s announcement, BGI Shenzen saved millions of dollars while enabling faster processing of their large genome data sets. The IBM Platform Symphony product was used for the Hadoop MapReduce applications. Platform Symphony provides a low latency Hadoop MapReduce implementation. In conjunction with its unique shared memory management and the data aware low latency scheduler, it accelerates many life sciences applications by as much as 10 times over open source Apache Hadoop

Below is an excerpt from the release:

BGI Shenzhen (China). BGI has developed a set of distributed computing applications using a MapReduce framework to process large genome data sets on clusters. By applying advanced software technologies including HDFS, Lustre, GlusterFS, and the Platform Symphony MapReduce framework, the institute has saved more than $20 million to date. For some application workloads, BGI achieved a significant improvement in processing capabilities while enabling the reuse of storage, resulting in reduced infrastructure costs while delivering results in less than 10 minutes, versus the prior time of 2.5 hours. Some of the applications enabled through the MapReduce framework included: sequencing of 1% of the Human Genome for the International Human Genome Project; contributing 10% to the International Human HapMap Project; conducting research in combating SARS, and a German variant of the E. coli virus; and completely sequencing the rice genome, the silkworm potato genome, and the human gut metagenome. Project leader: Lin Fang

Congratulations to the BGI team for their well deserved recognition.

Rohit Valia
Director, Marketing
HPC Cloud and Analytics

0 comments:

Post a Comment