What do we have?

General Overview

We have three High Performance Computing (HPC) systems on campus. Mio, Mc2 (Energy), and AuN (Golden). Mc2 and AuN are collectively know as BlueM.

BLUEM OVERVIEW

BlueM is a unique high performance computing system from IBM. The overall specifications are:

   
Feature Value
Teraflop rating 154 teraflops
Memory 17.4 terabytes
Nodes 656
Cores 10,496
Disk 480 terabytes

One of the unique characteristics of this machine is its small footprint both in physical size and energy usage. It requires only 85 kW. The new machine occupies a total of five racks, requiring only three compute racks, a management rack and a file system rack.

The IBM supercomputer is also unique in configuration. It contains two independent compute partitions that share a common file system. The combined compute partitions and their file system are known collectively as BlueM.

The two partitions are built using different architectures. The first partition, known as Mc2 (Energy), runs on an IBM BlueGene Q (BGQ). The second partition, known as Aun (Golden), uses the iDataplex architecture. Each of the architectures is optimized for a particular type of parallel application.

MC2 (ENERGY)

Mc2, the IBM BlueGene Q is designed to handle programs that can take advantage of large numbers of compute cores. Also, the BGQ is designed to run applications that use multiple levels of parallelism, such as combining threading and message passing. Multilevel parallelism is expected to be the dominant paradigm in the future of HPC. Our BGQ contains 512 nodes with each node having 16 cores. It has 8.192 terabytes of memory and a peak computational rate of 104.9 teraflops. The BGQ rack is currently half populated. That is, there is room for an additional 512 nodes within the same cabinet.

MC2 (Energy) Specifications
Specification
Blue Gene Q
PowerPC A2 17 Core
512 Nodes
8,192 Cores
8,192 Gbytes
104 Tflops
Features
New Architecture
Designed for large core count jobs
Highly scaleable
Multilevel parallelism - Direction of HPC
Room to Grow
Future looking machine

The processors on the Blue Gene are in a different family from those on RA and Mio and the network is significantly different. Code will need to be recompiled to run on Mc2. We have below two lists of programs and libraries that have been built on other Blue Gene machines. Some of the listed items are from a earlier model of the Blue Gene, the "P". Some listed have been ported but there has not been a high level of optimization performed

Aun (Golden)

Aun, based on the IBM iDataplex platform, is designed to handle applications that may require more memory per core. The nodes employ the x86 Sandy Bridge generation architecture. Each of the 144 nodes has 64 gigabytes of memory and 16 processors for a total of 2,304 cores and 9.216 terabytes of memory.

Aun uses the same compiler suite as Mines' Mio supercomputer. Many applications that are being run on these machines today could run on the new machine without a recompile. However, because of the updated processor instruction set available on the new machine we would expect improved performance with a recompile.

Aun (Golden)
Specification
iDataPlex
Intel 8x2 core SandyBridge
144 Nodes
2,304 Cores
9,216 Gbytes
50Tflops
Feature
Latest Generation Intel Processors
Large Memory / Node
Common architecture
Similar user environment to RA and Mio
Quickly get researchers up and running

MIO OVERVIEW

The machine Mio.mines.edu represents a new concept in computing at Mines. Mio is a shared resource funded in part by the Mines Administration and in part by money from individual researchers. Mio came on line March 2010. Initially was a relatively small cluster dedicated to a single group of research projects. We expect that Mio will quickly grow in to a supercomputing class machine.

Concept

Supercomputing has become an important part of engineering and scientific research. Most current generation supercomputers are actually comprised of a collection of compute nodes with each node containing several compute cores. Such machines are often called clusters. A typical cluster supercomputer might have hundreds to thousands of compute cores. The individual compute cores work on the same computation simultaneously. The compute nodes or cores communicate with each other via a high speed network. The nodes are normally housed in a rack with infrastructure such as the communications hardware, management nodes, network connections, and power supplies.

The Mio concept is simple. Mines funds provide the infrastructure discussed above and individual professors purchase compute nodes that are added to the cluster. The professors own their nodes, that is, they have exclusive access when they need them. When they are not in use by the owners the nodes are available for use by others.

Mio will be managed by the High Performance Computing group at Mines. The advantage of Mio for the professors is that they:

What’s in a name?

The name "Mio" is a play on words. It is a Spanish translation of the word “mine” as in belongs to me, not the hole in the ground. The phrase “The computer is mine.” can be translated as "El ordenador es mío."

Financial Considerations

The Mines Administration has purchased the initial infrastructure for Mio at a cost of roughly $19,000. Professors can purchase nodes at a cost of $5,500-$6,600. These nodes contain high end processors, 16 cores and are populated with 4 Gbytes of memory per core or 64 Gbytes per node.

Initial Configuration

Initially, Mio consisted of a Relion 2701 Head Node, 2 Relion 1702 Twin Compute Nodes (each Relion 1702 contains 2 nodes in a 1u enclosure), Infiniband and Ethernet connectivity, power supplies and a single enclosure rack. Each of the compute nodes contained two Intel 5570 Nehalem processors running at 2.93 GHz. Each Intel 5570 Nehalem processor contains 4 cores. There were be a total of 4 nodes x 2 processors per node x 4 cores per processor = 32 cores. For a complete machine description click on the Configuration link.

Current Compute Node Configuration (Updated 06/14/15):

That is, each node contains 2 x Intel 14 core processor for a total of 28 cores, along with 256 GB of memory, 2 TB of internal disk, and an FDR Infinband connector. These come grouped in a 2 node box and the cost per node is about $6,600. With 64 GB of memory the cost is about $5,300.

Mio also contains a number of "special" nodes. It has two Intel Phi nodes, three x86 nodes with GPUs and two IBM Power8 nodes with K80 GPUs. The IBM nodes have 20 cores, supporting 160 threads across 256 GB of memory. They have two K80 cards each with two GPUs. We have a number of specific examples for these nodes, included at the pages:

  1. Show me some Power8 and GPU examples!
  2. Show me some Machine Learning examples!

With a purchase of a Mio node you are gaining several advantages. You will not need to manage the node. You have the infrastructure provided by the school. This includes the Infiniband network which will greatly increase the scalability of your multinode applications. You will gain the option of using other peoples nodes when they are not in use. To purchase a node or get pricing information email Dr. Timothy Kaiser tkaiser@mines.edu.