The Aggregate Cluster Design Rules Form

Costs & Internal Parameters

Obviously, one of the most important cluster design constraints is cost... but cost changes rapidly, and is not necessarily the same for everybody depending on where you live, what deals your institution has with vendors, etc. To get truly accurate prices, the best approach is probably to first discuss the cluster with someone who knows your institution's purchasing procedure and any vendor relationships, then to get prices yourself (via the WWW, phone calls, etc.), and finally to write a document explaining precisely what you need so that people in purchasing will not make the common mistake of treating your request like purchase of generic PCs. For the purpose of computing approximate price/performance for the cluster configurations that are otherwise viable, there are a few cost parameters used by our code. The default prices used are very approximate; you may want to alter the values to more accurately reflect the prices you find.

The full set of cost and internal parameters used by the CGI is stored in a database. You can select any of these parameter sets, some of which have been made available by other users, or create one with your own values to better reflect your circumstances. Each email address is allowed one parameter set.

NOTE: all prices for components with an Amazon ID are updated daily. These price updates may cause the CDR to produce different designs given the same input depending on what the best prices are on any given day.

Default Parameter Set

To edit your own parameter set you must first login. You need not login to evaluate a design using any of the parameter sets that appear above. If you are unsure, you probably do not need to login.

Email address:
Password:

If you would like to login, you must first create a new user account if you have not done so before.

Warning: The password entered in this form is sent across the network in the clear. Though unlikely, it is possible that someone could intercept it. Do not use the same password here that you use for more secure accounts.

This form contains a series of questions about your supercomputing application and constraints on the cluster design. A CGI program will use your answers to provide a set of cluster designs that might be appropriate for your application. The search space is generally very large, so the CGI limits the search time to 10 seconds. Of course, the designs generated by the CGI should be treated only as a starting point for your own detailed design analysis.

Memory

Although data size often increases somewhat when a program is parallelized (due to copies buffered to avoid communication), the memory space that would be taken by your serial program's data for the largest problem you hope to solve can be used as an estimate of the data space needed in parallel execution. In integer MB, about how much data does your program need to hold in main memory?
1MB or less 1MB to 2MB 2MB to 4MB 4MB to 8MB 8MB to 16MB 16MB to 32MB 32MB to 64MB 64MB to 128MB 128MB to 256MB 256MB to 512MB 512MB to 1GB 1GB to 2GB 2GB to 4GB 4GB to 8GB 8GB to 16GB 16GB to 32GB 32GB to 64GB 64GB to 128GB 128GB to 256GB 256GB to 512GB 512GB to 1TB 1TB to 2TB 2TB to 4TB per cluster

Your application program will need to be running on each node, thus, each node memory will need a copy of the executable code. Not counting data, how big is your compiled program?
1MB or less 1MB to 2MB 2MB to 4MB 4MB to 8MB 8MB to 16MB 16MB to 32MB 32MB to 64MB 64MB to 128MB 128MB to 256MB 256MB to 512MB 512MB to 1GB 1GB to 2GB per node

Operating systems differ, but any OS you use on each node will have to fit in node memory. This OS memory use includes not just the OS kernel, but also any background OS tasks and a reasonable set of I/O buffers. How much main memory will your OS use on each node?
16MB (e.g., minimal Linux) 32MB (e.g., Linux + X Windows) 64MB (e.g., minimal MS Windows) 128MB 256MB 512MB 1GB 2GB per node

Ever since cache moved on-chip, cache bandwidth has been very good for most processors. Unfortunately, codes that have cache-unfriendly memory access patterns are very sensitive to main memory bandwidth. If you specify a particular memory bandwidth (GBytes/second per GFLOPS) here, the GFLOPS rating quoted for each design will be limited to the GFLOPS supported by the memory bandwidth of the system.
my code is very cache-friendly 0.125GB/s per GFLOPS 0.25GB/s per GFLOPS 0.5GB/s per GFLOPS 0.75GB/s per GFLOPS 1.0GB/s per GFLOPS 1.25GB/s per GFLOPS 1.5GB/s per GFLOPS 2.0GB/s per GFLOPS 3.0GB/s per GFLOPS 4.0GB/s per GFLOPS 8.0GB/s per GFLOPS 16.0GB/s per GFLOPS 32.0GB/s per GFLOPS

Hard Disk Drives

Although use of disk-based virtual memory should be avoided to achieve maximum performance, many existing software packages have "memory leaks" that make their memory image slowly grow despite the fact that the memory space they are truly using does not grow. If your code leaks, you may want to use local disks for virtual memory swap space. How much swap space should be allocated?
no virtual memory swap space swap same size as main memory swap 2X main memory size swap 3X main memory size swap 4X main memory size

In total, how many GB of hard disk storage do you need for live data in your cluster? (Keep in mind that your cluster can borrow disk space from other machines, so no disk is needed within the cluster nodes.)
none within the cluster nodes 2GB or less 2GB to 4GB 4GB to 8GB 8GB to 16GB 16GB to 32GB 32GB to 64GB 64GB to 128GB 128GB to 256GB 256GB to 512GB 512GB to 1TB 1TB to 2TB 2TB to 4TB 4TB to 8TB 8TB to 16TB 16TB to 32TB 32TB to 64TB 64TB to 128TB 128TB to 256TB 256TB to 512TB 512TB to 1PB

One of the advantages of clusters is that you can use additional disk drives in each node to increase reliability (using simple disk mirroring or fancier RAID techniques) or to hold backups. Which of these techniques do you intend to use?
no RAID add RAID for recovery from one disk failure
no extra disk space add space for a backup copy

Network

The minimum message latency of the network plays a major role in determining the finest grain parallelism you can use across the nodes of a cluster. What is the largest minimum latency that you expect your code can tolerate?
50 to 100 microseconds 10 to 50 microseconds 10 microseconds or less latency is not a concern 1600 to 3200 microseconds 800 to 1600 microseconds 400 to 800 microseconds 200 to 400 microseconds 100 to 200 microseconds

Latency in performing various types of aggregate functions (aka, collective communications) is different from latency on point-to-point messages. What is the largest minimum latency that you expect your code can tolerate on aggregate operations like barrier synchronization?
latency is not a concern 1600 to 3200 microseconds 800 to 1600 microseconds 400 to 800 microseconds 200 to 400 microseconds 100 to 200 microseconds 50 to 100 microseconds 10 to 50 microseconds 10 microseconds or less

Bisection bandwidth is a very important in achieving good performance from some applications, but is relatively unimportant for others. If all the processor cores in the cluster are communicating simultaneously, what is the minimum amount of network bandwidth that must be available to each processor core? (I.e., what is bisection bandwidth / number of processor core?)
0 (only one node talks at a time) 10Mb/s or less 10Mb/s to 20Mb/s 20Mb/s to 30Mb/s 30Mb/s to 40Mb/s 40Mb/s to 50Mb/s 50Mb/s to 60Mb/s 60Mb/s to 70Mb/s 70Mb/s to 80Mb/s 80Mb/s to 90Mb/s 90Mb/s to 100Mb/s 100Mb/s to 125Mb/s 125Mb/s to 150Mb/s 150Mb/s to 175Mb/s 175Mb/s to 200Mb/s 200Mb/s to 250Mb/s 250Mb/s to 300Mb/s 300Mb/s to 350Mb/s 350Mb/s to 400Mb/s 400Mb/s to 500Mb/s 500Mb/s to 600Mb/s 600Mb/s to 700Mb/s 700Mb/s to 800Mb/s 800Mb/s to 900Mb/s 900Mb/s to 1000Mb/s 1Gb/s to 1.2GMb/s 1.25Gb/s to 1.5Gb/s 1.5Gb/s to 1.75Gb/s 1.75Mb/s to 2Gb/s more than 2Gb/s

When a node needs to communicate, on average, how many other nodes does it need to talk with? We call this number the coordinality of communication. For example, to update boundary regions in a simple 2D grid code, each node typically communicates with every node that is handling an adjacent portion of the 2D space; thus, communication would involve 4 other nodes.
more than 5 other nodes all other nodes 0 (nodes never communicate) 1 other node 2 other nodes 3 other nodes 4 other nodes 5 other nodes

Physical Parameters

No matter how reliable your vendors are, even if you have a maintenance contract, you need spare nodes. Without spares, hardware failures not only will take time to be repaired, but also probably will result in your hardware becoming heterogeneous, because identical parts are no longer available. "Hot" spares give you higher availability than "cold" spares, but increase your network costs. How many spare nodes do you want to include?
cold spares, 1 for every hot spares, 1 for every cold spares, exactly hot spares, exactly

Not counting spares, does your cluster need to have a specific number of nodes, processors, or processor cores? What are the constraints?
number of nodes number of processors number of cores is unconstrained must be a power of 2 must be a square must be a cube must be exactly

The cluster nodes and switches need to be mounted in some kind of rack or shelving unit. A typical rack needs about 2x2 feet of floor space; shelving units come in various sizes, but most require about 2x4 feet of floor space (i.e., roughly the space of two 2x2 racks). Allowing at least a 2 foot wide path to walk around the racks, what is the maximum number of 2x2 rack units that you can fit in the space you have available?
rack space is not an issue 1 rack 2 racks 3 racks 4 racks 5 racks 6 racks 7 racks 8 racks 9 racks 10 racks 11 racks 12 racks 13 racks 14 racks 15 racks 16 racks 17 racks 18 racks 19 racks 20 racks 21 racks 22 racks 23 racks 24 racks 25 racks 26 racks 27 racks 28 racks 29 racks 30 racks 31 racks 32 racks 33 racks 34 racks 35 racks 36 racks 37 racks 38 racks 39 racks 40 racks 41 racks 42 racks 43 racks 44 racks 45 racks 46 racks 47 racks 48 racks 49 racks 50 racks

Power requirements for nodes vary depending on the precise contents of the nodes, but a large cluster needs a lot of power. In the USA, that power is typically 110VAC, usually with a 15A or 20A limit on each circuit coming from the main power box. Approximately how much 110VAC power can be dedicated to the cluster?
power is not an issue 5A to 10A 10A to 15A 15A to 20A 20A to 25A 25A to 30A 30A to 40A 40A to 50A 50A to 60A 60A to 70A 70A to 80A 80A to 90A 90A to 100A 100A to 120A 120A to 140A 140A to 160A 160A to 180A 180A to 200A 200A to 220A 220A to 240A 240A to 260A 260A to 280A 280A to 300A 300A to 350A 350A to 400A 400A to 450A 450A to 500A 500A to 600A 600A to 700A 700A to 800A 800A to 900A 900A to 1000A 1000A to 1200A 1200A to 1400A 1400A to 1600A 1600A to 1800A 1800A to 2000A 2000A to 2500A 2500A to 3000A 3000A to 3500A 3500A to 4000A 4000A to 4500A 4500A to 5000A

Essentially all the power consumed by a cluster gets turned into heat, which has to be removed from the room housing the cluster. How much air conditioning is available in the cluster room?
Air Conditioning is not a factor 0.25tons to 0.5tons 0.5tons to 0.75tons 0.75tons to 1.0tons 1.0tons to 1.5tons 1.5tons to 2.0tons 2.0tons to 2.5tons 2.5tons to 3.0tons 3.0tons to 3.5tons 3.5tons to 4.0tons 4.0tons to 5.0tons 5.0tons to 7.5tons 7.5tons to 10.0tons 10tons to 15tons 15tons to 20tons 20tons to 25tons 25tons to 30tons 30tons to 50tons 50tons to 75tons 75tons to 100tons 100tons to 125tons 125tons to 150tons 150tons to 175tons

Costs

The cost of the cluster does not end after it is purchased. Operating a cluster incurs costs as well. Enter local cost factors below to help the CDR find designs that fit within your budget.

Operating Costs

In addition to money you spend to buy your cluster, it will cost money to operate. Often some or all of these costs are born by other parts of the support organization, and thus are "free" to the cluster purchaser. However, operating costs can be a significant percentage of the cluster purchase price, especially when considered over the life of the cluster. These operating costs tend to vary widely based on location.

How much does electricity cost at the location at which the cluster will be housed?
$/KWh + $/month
What kind of cooling system will be used to cool the cluster? High efficiency chilled water (0.35 KW/ton) Regular chilled water (0.5 KW/ton) High efficiency heat pump (SEER 14) Heat pump meeting minimal efficiency standard (SEER 10) Old, inefficient heat pump (SEER 7)

How much does floor space cost per square foot per month?
$per ft² per month

What is your annual operating budget? (Enter 0 to disregard operating costs)
$ per year

Software

A cluster needs an operating system, system management tools, run-time libraries, and application software to do useful work. Many cluster users are happy with free/open source software. However, some users want specific commercial software packages that require licensing fees.

How much does your software cost?
per cluster
per node processor core

Acquisition Cost

What is your budget for the cluster? Be somewhat conservative, because there are always a few unexpected costs.

Metrics

What is the minimum number of GFLOPS your cluster must be able to achieve?

The HPL model estimates cluster performance based on a model of the HPL benchmark. What is the minimum performance (in GFLOPS) your cluster must be able to achieve on the High Performance Linpack (HPL) benchmark (experimental)? A value of 0 means no constraint on High Performance Linpack (HPL) performance.

The SWEEP3D model estimates cluster performance based on a model of the SWEEP3D benchmark. What is the minimum performance (in 1/sec) your cluster must be able to achieve on the SWEEP3D benchmark (experimental)? A value of 0 means no constraint on SWEEP3D performance.

The Total Cost of Ownership model measures total operating cost over five years plus acquisition cost. The operating cost is computed based on power consumption for the cluster itself, power for air conditioning, and space rental costs. Acquisition cost include all costs to buy the necessary hardware and software. What is the minimum performance (in 1/$) your cluster must be able to achieve on the Total Cost of Ownership benchmark (experimental)? A value of 0 means no constraint on Total Cost of Ownership performance.

What is the maximum number of designs that should be displayed?
the best 256 designs the best 512 designs the best 1024 designs the best 2048 designs the best 4096 designs the best 8192 designs only the best design the best 2 designs the best 4 designs the best 8 designs the best 16 designs the best 32 designs the best 64 designs the best 128 designs

There are a variety of metrics computed for each feasible configuration.
Use weightings below SWEEP3D (experimental) High Performance Linpack (HPL) (experimental) Total Cost of Ownership (experimental)

The following parameters allow you to set weightings for how these metrics will be combined to determine the sorted order of configurations, and hence which design is best. Weightings are multiplicative factors; higher factors imply that extra performance is more important in that metric.
0 * Memory space 1 * Memory space 10 * Memory space 100 * Memory space 1000 * Memory space 10000 * Memory space 100000 * Memory space 1000000 * Memory space + 0 * Memory bandwidth 1 * Memory bandwidth 10 * Memory bandwidth 100 * Memory bandwidth 1000 * Memory bandwidth 10000 * Memory bandwidth 100000 * Memory bandwidth 1000000 * Memory bandwidth + 0 * Disk space 1 * Disk space 10 * Disk space 100 * Disk space 1000 * Disk space 10000 * Disk space 100000 * Disk space 1000000 * Disk space + 0 * 1/Net latency 1 * 1/Net latency 10 * 1/Net latency 100 * 1/Net latency 1000 * 1/Net latency 10000 * 1/Net latency 100000 * 1/Net latency 1000000 * 1/Net latency + 0 * Net bandwidth 1 * Net bandwidth 10 * Net bandwidth 100 * Net bandwidth 1000 * Net bandwidth 10000 * Net bandwidth 100000 * Net bandwidth 1000000 * Net bandwidth + 0 * 1/Cost 1 * 1/Cost 10 * 1/Cost 100 * 1/Cost 1000 * 1/Cost 10000 * 1/Cost 100000 * 1/Cost 1000000 * 1/Cost + 1 * GFLOPS 10 * GFLOPS 100 * GFLOPS 1000 * GFLOPS 10000 * GFLOPS 100000 * GFLOPS 1000000 * GFLOPS 0 * GFLOPS + 0 * Amps 1 * 1/Amps 10 * 1/Amps 100 * 1/Amps 1000 * 1/Amps 10000 * 1/Amps 100000 * 1/Amps 1000000 * 1/Amps + 0 * Tons 1 * 1/Tons 10 * 1/Tons 100 * 1/Tons 1000 * 1/Tons 10000 * 1/Tons 100000 * 1/Tons 1000000 * 1/Tons + 0 * 1/Cost per year 1 * 1/Cost per year 10 * 1/Cost per year 100 * 1/Cost per year 1000 * 1/Cost per year 10000 * 1/Cost per year 100000 * 1/Cost per year 1000000 * 1/Cost per year

Component Limitations

For non-technical reasons, you may be restricted to including certain components with or excluding certain components from your cluster. You may limit the search to either include or exclude systems that have a particular string in the name or type of at least one component.

Include only designs with any all of the space separated strings below in the name or type of at least one component. Surround strings that have spaces with quotes (e.g., "Fast Ethernet").

Exclude all designs with any all of the space separated strings below in the name or type of at least one component. Surround strings that have spaces with quotes (e.g., "Fast Ethernet").