The following topics about our personal HPC cluster solutions are discussed as a series of questions and answers.
General/Audience/Market
Is this a real cluster?
Yes, it is a real cluster. The basic system has four motherboards which are cluster nodes in a standard off the shelf case with a single power supply. The main motherboard is always powered and functions just like a workstation. The three compute nodes can be powered-on when needed. The software is identical to that running on large clusters.
What can I run on such a system?
Any software that is designed to run on a Linux cluster including both High Performance
Computing (HPC) and Apache Hadoop applications.
Can I do real HPC and/or Hadoop work on such a system?
Yes. Considering that surveys have shown that around 40% of HPC users use less than 16 cores (over 50% use less than 32 cores), it should be a very usable system. Similarly, Hadoop systems can support turn-key feasibility environments with up to 24 TB of SSD storage, 96 threads, and 512 GB of memory.
How fast is it?
The first Limulus version achieved 385.5 double precision CPU GFLOPS (52% of peak running HPL). Note, benchmarking results in HPC are very application specific. While a GPU might match the performance of a small cluster for some applications, it is not a general purpose computing device, and therefore is not as flexible as a CPU cluster.
What Does Limulus Mean?
Limulus is an acronym for LInux MULti-core Unified Supercomputer.
What is the difference between a Limulus and a 8/16-core workstation?
In terms of core count, there is no difference. In terms of price and performance there can be a big difference. A multi-core SMP system (such as a dual socket workstation or server motherboard) can provide many cores, but depending on workload, you may not be able to get effective use from all of the cores due to memory contention (See Benchmarking A Multi-Core Processor For HPC and Counting Effective Cores). In a cluster design, such as Limulus, each node has one processor socket with exclusive access to the local memory (memory bandwidth remains constant as more cluster nodes are added).
Also, as more cores are added to a processor the amount heat generated increases. To account for this excess heat either the clock frequency must be lowered for the cores (resulting in less performance) or the heat dissipation requirements are increased resulting in very hot processors that can exceed 200W (often requiring loud and specialized cooling). Limulus systems provide a balanced approach where high frequency (fast) computing cores (typically 65-80 Watts) are distributed with a consistent and scalable memory bandwidth. Finally, unused nodes can be powered off when not in use saving more power.
In terms of price, comparable core-heavy workstation solutions usually cost more than a comparable Limulus personal workstation cluster.
Why don't I just use the cloud?
The cloud is basically "renting cycles on some else's computer." If that use case fits, then the cloud is good solution. However, if you need a 24x7 resource (as in most HPC and Hadoop installations) the cloud can be quite expensive. The one year AWS cost for hardware equivalent to a Limulus system can easily cost 1.5 to 2 times the Limulus purchase price and the subsequent years require more cloud funding. An effective use case for the cloud must include cost, data transfer, security, and flexibility. While cloud has its place in HPC and Hadoop processing, there is nothing like "owning the reset switch."
What is Limulus' market?
There are several areas where a personal cluster can be useful (i.e where you own the reset switch). In addition, Edge Computing whereby compute resources are needed locally outside of a data center or cloud, has become and important factor in many deployments. Some examples are as follows:
- Edge systems - high-performance in non-data center environments (lab, office, home, factory where low noise, limited power, and low heat are important)
- System administrators - a cluster sandbox to try new things, test software packages
- Software developers - a private software development environment
- Academic projects - instructional hardware, student projects, learn to run real HPC/Hadoop codes
- Cloud staging - stage and develop cloud HPC software before launching it to the cloud
- Small medium scale production work - test ideas, run applications under your control with up to 96 physical cores
- Small and medium-sized business HPC - explore how HPC can help manufacturing without a huge investment
- Big Data/Hadoop/Analytics - try and test big data and Apache Hadoop/Spark projects using 192 cores/threads and up to 32 TB of SSD storage without the administration overhead and cost of a data center cluster
Hardware
How many cores can you fit on one case?
Currently the base Limulus model 212 can provide up to 64 physical cores (128 threads). Model 1212 can provide up to 128 physical cores (256 threads) cores (128 threads). The rack mount option can support up to four single Limulus systems for a total of 512cores (256 threads).
What kind of processors do you use?
Our current designs include Ryzen Zen3 processors from AMD (Intel available upon request). For the nodes we use low power (65 Watt) TDP six/eight/twelve/sixteen-core x86_64 processors. Some the the higher TDP processors are tuned to run at 65 Watt TDP.
What kind of motherboards do you use?
We can use almost any standard Micro-ATX (µATX) motherboard. However, we prefer to test them before we recommend any specific motherboard. Geometry and component issues (i.e. Gigabit Ethernet chipset) make some boards more desirable than others.
How do you fit those extra motherboards in a standard case?
Our second generation uses µATX blades that can hold almost any standard µATX motherboard. We designed the blades with plug-able power (i.e. inserting the blade automatically connects to the power bus). We tried to keep the cost of the custom parts as low as possible by 3D printing all needed brackets, blades and spacers. We also took the time to create a clean design and keep the cabling neat.
Why do you use 3D printing?
By using 3D printing we can keep costs lower (i.e. we don't have to buy sheet metal or injection molded parts in high quantity), easily manage design changes, and allow customized solutions. In addition 3D printing can be used to create sophisticated shapes that might otherwise would be expensive, difficult, or impossible to create with other manufacturing methodologies.
Will 3D printed plastic parts melt in a high performance computer?
In a word, no. First, our systems are designed to run cool and have plenty of moving air and case ventilation to remove heat. Second, during normal
operation, the hottest part of the system are the CPUs and active coolers. These devices are not near or in contact with any printed parts. With modern processors, temperatures
can reach 70-85° C before they internally throttle the clock (thus reducing the amount of heat generated). Limulus nodes are also monitored
and have a default throttling temperature (72C) that will initiate kernel level throttling, if needed. Finally, all parts are printed (and annealed) using
high performance PLA. The resulting parts start to become soft between 75-85°C. If would take some work to seal off the case so that the internal
temperature is in this range (plus a bunch of other things would not work).
Can I attach a keyboard and monitor to the node motherboards?
Yes, in some cases. There is a front panel that provides video, USB, and a power switch for each motherboard, however, AMD Ryzen nodes are "headless" and therefore will have not video output. Unless you want to debug node startup issues (something we do to make sure systems work for you) there is no need for node video during normal operation.
Why don't you pack a bunch of 20-core processors into the case?
Limulus is designed with a desk-side heat/power/performance/noise envelope. These type of systems are designed to operate at "Edge" or the cloud or data center. An HPC server can pack in many cores because in data centers there is dedicated power, cooling, and a tolerance for fan noise. Have you ever run an HPC server (or two) next to your desk?
Why don't you pack a bunch of GPU processors into the case?
Using GPUs is a great solution if it fits your problem, but GPUs require more power and more heat must be removed from the case. The Limulus DL (Deep Learning) systems employ GPUs.
Do you use dual socket motherboards?
No. Limulus is designed to use single socket Micro ATX motherboards. These offer a balance of expansion (RAM and PCIe slots), memory bandwidth, power and size.
Can I connect multiple Limulus cases and create a bigger cluster?
Yes, a whole "classroom cluster" can be built with Limulus systems. We also offer a four node rack-mount configuration that is low power and quiet.
How are the nodes connected?
Gigabit Ethernet (GbE). There is also a low-cost 10-GbE option (all internal switching).
Does each motherboard have a hard drive?
In the standard HPC design only the main motherboard has persistent storage. The nodes run using a RAM disk and are stateless (We use the Warewulf Provisioning software). It is possible to connect drives to the worker nodes, however, all drives will remain powered up while nodes are off. The Limulus software will place these drives in standby mode and they will become locally available to the node when restarted.
Our Hadoop models have dedicated SSD data drives (for HDFS) connected to each motherboard and we install the base operating system and all Hadoop software in a resident M.2 drive.
Can I add extra hard drives?
Yes, each Limulus HPC system comes with an M.2 SSD for system software. There is an additional two 2.5 inch bays and six open 3.5 inch drive bays that can be used to increase the total number of drives to eight (two SSD, six HDD). RAID storage is now an option on all systems.
The Limulus Hadoop version has eight 2.5 bays (for SSD disks, used for HDFS) and six 3.5 inch bays for large spinning RAID sets.
Can I add expansion cards to the node motherboards?
Yes, there is an optional bracket for adding one low profile PCIe card (usually 10GbE). And, it depends on the model and motherboard.
Can I add expansion cards to the main motherboard?
Yes, depending on options, there are usually some slots are available, but long cards may not fit (e.g. huge video cards).
Can I add video cards to the node motherboards?
We include a low power video card and do not recommend using higher powered cards (in addition there may not be adequate PCI slots available.
How big is the power supply?
Depending on the specific hardware installed 850 -1200 Watts.
Does it have ECC memory?
When using AMD Ryzen processors the base Limulus models now support ECC (Error Correcting Memory). We ship ECC memory with all AMD systems.
Depending on the Intel processor, ECC an memory option may be available (Contact us if you need to use Intel processors). Keep in mind, however, the requirement for ECC depends on your needs. In our experience, and in our testing, we have found excellent results with "quality" memory (not the bargain priced brands). We have never had a problem with non-ECC memory in our personal cluster systems. We have run (self checking) codes for days without any issues.
Can it use 10 GbE, 25 GbE, or IB?
We support 10 GbE and 25 GbE using a low-cost switchless design. Single byte latencies range from 8-10 µsec (using Netpipe). IB is not currently supported.
Power/Heat/Noise
How much power does a Limulus use?
When running HPL (16 cores) we measure 320 Watts for the model 100. Of course it also depends on what you put in the system (i.e. disk, video card, etc.)
As new generations of processors become available (with faster clocks and more cores), we continue to use 65 Watt versions to keep Limulus systems within the same power envelope.
How many standard wall plugs (receptacles) do Limulus systems use?
One. Unless you have the redundant power supply option, then two receptacles are needed.
Can I manually turn nodes off and on?
Yes.
Can I automatically turn nodes off and on?
Yes, and you can even integrate this into the HPC resource scheduler.
Does it create a lot of heat?
Like all electronic devices, it generates heat. Unlike a high-end servers, it would make a poor space heater. We use 65-80 Watt processors. Limulus will not create anymore heat than most older PCs.
How loud is it?
It is very quiet. The use of large fans helps reduce noise considerably. It can sit next to a desk in an office without any significant impact on the ambient noise environment (i.e., it will not disturb conversations, listening to music, or phone calls).
Software
What software does it run?
Linux of course. It comes installed with HPC cluster stack based on OpenHPC (you can use all OpenHPC packages).. All base packages are open source and built on top of CentOS 7.x. Other RPMS and SRPMS are freely available. All Limulus Hadoop systems use an open source Hadoop distribution.
Will updated software be available?
Yes.
Is software support available?
Yes, each system comes with 90 days of support. Extended
support options are available.
Is administrative support available?
Yes, if we can remotely access the system(s) vis ssh and/or VPN. See our
administrative support options are available.
Is it the same software that runs on big clusters?
Yes. Applications can be moved from/to other clusters with little or no effort.
Will there be other open source application software available?
Yes.
Can I install my own software?
Yes, this is an open source platform, you control your destiny!
Can I run commercial software?
If your software can run on a large Linux cluster, it can probably run on a Limulus system. Currently, we use CentOS 7.X, which is a community rebuild of Red Hat 7.X. Of course it all depends on the software as well as vendor policies.
Can I run Windows on it?
Probably, but we have not tried.
Support
Does it come with hardware support?
Yes. One year return to depot support is included.
You must pay shipping to the depot.
Does it come with software support?
Yes. Ninety days of software support is included.
Extended support options are available.