Joining GECO
Opens On Campus Only

• 2144 processing cores in 268 nodes
  • 256 nodes with 512 Clovertown E5355 (2.67 GHz)
    
    • (quad core dual socket)
    • 184 with 16 Gbytes & 72 with 32 Gbytes
  • 12 nodes with 48 Xeon 7140M (3.4 GHz)
    
    • (quad socket dual core)
    • 32 Gbytes each
• Memory
  • 5,632 Gbytes ram (5.6 terabytes)
  • 300 terabyte disk
  • 16/32 gigabytes RAM per node
  • Library of Congress has 20 terabytes
  • All the films made in the world in 1989 would take up 166 terabytes
• Performance
  • 17 teraflop sustained performance
  • like every human on the planet doing 2500 calculations per second

Admin Nodes: 2 total
PE2950 servers
E53352 quad-core Xeons

Management Server: 1 total (page 37 or original quote)
PE1950 server
5130 dual-core Xeons

Nodes/Footprint: Dual/quad-socket racks, 12 cabinets

Interconnect: IB, DDR, Cisco
Cisco SFS 7024 IB Server Switch - 48 Port
Std Configuration with additional 12 Port Cards to support up to 288 ports of 4X DDR IB Host Channel
Adaptors. Each compute node will host one DDR 1-Port HCA (4X IB, PCIex8, 0MB, Tall Bkt) [LINUX].
InfiniBand to Fiber Channel Gateway will be provided by Data Direct Networks.
Administrative network will be provisioned by Gigabit Ethernet resident on each motherboard and Dell Power Connect Managed 6248 GigE Switches.

Disk Storage: Data Direct Networks
CSM: Scratch~200 TB raw, Primary ~100 TB raw, Tape ~300 TB raw
2 S2A9550s with 10 48 Bay JBODs and 400 750GB SATA drives
300TB Raw, 240TB Usable
Bandwidth will be between 3 and 4GB/sec depending on I/O
Lustre File System will be implemented and supported by DDN

Tape Archive:
300TB
Power Vault ML6030

Data Center

The GECO cluster will housed in the CSM CTLM Data Center which was completed in July 2005. It has a robust utility infrastructure and was designed to accommodate high performance servers. The data center features redundant systems for both power and air conditioning including backup room air conditioning units, a backup chiller, an emergency generator and Uninterruptible Power Supplies (UPS). The room includes surveillance cameras and remote temperature sensing and alarms. A description of each system is provided below.

The CTLM Machine room is cooled by 4 chilled water, computer room A/C (CRAC) units with 25% load redundancy. The CRAC fans are powered by an emergency generator in the event of a loss of normal building power. Chilled water serving the space is provided by a chilled water plant adjacent to the building which has multiple pumps and chillers. A backup chiller, with 62.5 tons of capacity, is available on the CTLM roof with emergency generator backup power. This is sufficient to cool an anticipated room buildout of 271 KVA. Auto-dialers are used to notify Academic Computing personnel if the room temperature rises above set point.

There are 3 Power Distribution Units (PDUs) in the data center. Each PDU is capable of distributing 100 KVA of power to the space. The current average load on this system is 45 KVA while the cluster load estimate is approximately 100 KVA. Beyond being adequate to supply power to the cluster, this facility has 171 KVA of surplus which can be used for future expansions/upgrades.

Power to the PDUs is conditioned and backed up by 3 UPSs. The UPS batteries are capable of providing continuous power for up to 15 minutes. An emergency generator starts within 10 seconds of a power outage to provide backup power to all of the UPS/PDU circuits, the CRAC units, the temperature controls and the backup chiller and chilled water pumps.

A pre-action fire protection system protects the data center while minimizing the risk of accidental flooding. Sensors are installed below the raised floor to detect the presence of water.
Access to the CTLM Data Center is restricted. Electronic card readers are used to secure doors and record access to the machine room. Video surveillance cameras are located throughout the entire building and within the Data Center itself to document personnel movement throughout the building.

Networking Plan

Over the last ten years, CSM has invested heavily in a campus-wide high-speed data network. Sufficient infrastructure and backbone bandwidth is in place to provide 100Mb/s links to all edge devices on campus, with the option of using 1Gb/s as needed. The campus is fully wired (category 5e). Construction done in the last 5 years used category 6 wiring. A wireless (802.11g) network is available throughout the campus. All major academic buildings as well as all student residential units are interconnected via fiber optic cable. Major building uplinks run at 1Gb/s; expansion to multi-GigiBit connections is possible with minimal cost. Each building is connected directly back to the core switch: a Cisco 6509 with a 480Gb/s backplane.

CSM is a member of the Front Range GigaPop (FRGP)—a consortium of sixteen government, educational and research institutions. The FRGP provides high-speed connectivity between member institutions as well as the Commodity Internet, Internet2 and National Lambda Rail. CSM subscribes to 40Mb/s of Commodity Internet bandwidth and has access to significantly more bandwidth to Commodity sources via the TransitRail service provided by NLR. CSM currently purchases 45Mb of bandwidth to Internet2 and 1Gb to NLR.

An aggressive strategy to improve and extend CSM data networking has been approved by the CSM administration. Funding has been secured for a significant upgrade to the university network core allowing the deployment of a fully redundant core physically located in two different buildings. Legal agreements are now being finalized which will provide long-term access (40 years) to metro-area “dark” fiber. Significant for the CSM-NREL collaboration is that one branch of the fiber ring connecting CSM to the Denver FRGP POPs runs past the main gate of NREL and includes a potential splice point at that intersection. The agreements will allow for NREL to join the FRGP with connectivity on the CSM ring and/or for a dedicated fiber connection between the NREL and the CSM campuses.