|
Processors:
2144 total cores
Standard nodes are dual-socket/quad-core
PE 1950 servers
512 Clovertown E5355 (2.67 GHz), 2048 cores
2 GB RAM per core (16 GB/node)
Fat nodes are quad-socket/dual-core
48 Xeon 7140M (3.4 GHz), 96 cores
4 GB RAM per core (32 GB/node)
Admin Nodes: 2 total
PE2950 servers
E53352 quad-core Xeons
Management Server: 1 total
(page 37 or original quote)
PE1950 server
5130 dual-core Xeons
Nodes/Footprint: Dual/quad-socket
racks, 12 cabinets
Interconnect: IB, DDR, Cisco
Cisco SFS 7024 IB Server Switch - 48 Port
Std Configuration with additional 12 Port Cards to support up to
288 ports of 4X DDR IB Host Channel
Adaptors. Each compute node will host one DDR 1-Port HCA (4X IB,
PCIex8, 0MB, Tall Bkt) [LINUX].
InfiniBand to Fiber Channel Gateway will be provided by Data Direct
Networks.
Administrative network will be provisioned by Gigabit Ethernet resident
on each motherboard and Dell Power Connect Managed 6248 GigE Switches.
Disk Storage: Data Direct
Networks
CSM: Scratch~200 TB raw, Primary ~100 TB raw, Tape ~300 TB raw
2 S2A9550s with 10 48 Bay JBODs and 400 750GB SATA drives
300TB Raw, 240TB Usable
Bandwidth will be between 3 and 4GB/sec depending on I/O
Lustre File System will be implemented and supported by DDN
Tape Archive:
300TB
Power Vault ML6030
Data Center
The GECO cluster will housed in the CSM CTLM Data Center which was
completed in July 2005. It has a robust utility infrastructure and
was designed to accommodate high performance servers. The data center
features redundant systems for both power and air conditioning including
backup room air conditioning units, a backup chiller, an emergency
generator and Uninterruptible Power Supplies (UPS). The room includes
surveillance cameras and remote temperature sensing and alarms.
A description of each system is provided below.
The CTLM Machine room is cooled
by 4 chilled water, computer room A/C (CRAC) units with 25% load
redundancy. The CRAC fans are powered by an emergency generator
in the event of a loss of normal building power. Chilled water serving
the space is provided by a chilled water plant adjacent to the building
which has multiple pumps and chillers. A backup chiller, with 62.5
tons of capacity, is available on the CTLM roof with emergency generator
backup power. This is sufficient to cool an anticipated room buildout
of 271 KVA. Auto-dialers are used to notify Academic Computing personnel
if the room temperature rises above set point.
There are 3 Power Distribution
Units (PDUs) in the data center. Each PDU is capable of distributing
100 KVA of power to the space. The current average load on this
system is 45 KVA while the cluster load estimate is approximately
100 KVA. Beyond being adequate to supply power to the cluster, this
facility has 171 KVA of surplus which can be used for future expansions/upgrades.
Power to the PDUs is conditioned
and backed up by 3 UPSs. The UPS batteries are capable of providing
continuous power for up to 15 minutes. An emergency generator starts
within 10 seconds of a power outage to provide backup power to all
of the UPS/PDU circuits, the CRAC units, the temperature controls
and the backup chiller and chilled water pumps.
A pre-action fire protection system
protects the data center while minimizing the risk of accidental
flooding. Sensors are installed below the raised floor to detect
the presence of water.
Access to the CTLM Data Center is restricted. Electronic card readers
are used to secure doors and record access to the machine room.
Video surveillance cameras are located throughout the entire building
and within the Data Center itself to document personnel movement
throughout the building.
Networking Plan
Over the last ten years, CSM has invested heavily in a campus-wide
high-speed data network. Sufficient infrastructure and backbone
bandwidth is in place to provide 100Mb/s links to all edge devices
on campus, with the option of using 1Gb/s as needed. The campus
is fully wired (category 5e). Construction done in the last 5 years
used category 6 wiring. A wireless (802.11g) network is available
throughout the campus. All major academic buildings as well as all
student residential units are interconnected via fiber optic cable.
Major building uplinks run at 1Gb/s; expansion to multi-GigiBit
connections is possible with minimal cost. Each building is connected
directly back to the core switch: a Cisco 6509 with a 480Gb/s backplane.
CSM is a member of the Front Range
GigaPop (FRGP)—a consortium of sixteen government, educational
and research institutions. The FRGP provides high-speed connectivity
between member institutions as well as the Commodity Internet, Internet2
and National Lambda Rail. CSM subscribes to 40Mb/s of Commodity
Internet bandwidth and has access to significantly more bandwidth
to Commodity sources via the TransitRail service provided by NLR.
CSM currently purchases 45Mb of bandwidth to Internet2 and 1Gb to
NLR.
An aggressive strategy to improve
and extend CSM data networking has been approved by the CSM administration.
Funding has been secured for a significant upgrade to the university
network core allowing the deployment of a fully redundant core physically
located in two different buildings. Legal agreements are now being
finalized which will provide long-term access (40 years) to metro-area
“dark” fiber. Significant for the CSM-NREL collaboration
is that one branch of the fiber ring connecting CSM to the Denver
FRGP POPs runs past the main gate of NREL and includes a potential
splice point at that intersection. The agreements will allow for
NREL to join the FRGP with connectivity on the CSM ring and/or for
a dedicated fiber connection between the NREL and the CSM campuses. |
