For the past couple of months, I have been experimenting with different setups of lab equipment because I wanted to take my
I wanted to approach the homelab design like I would approached any other customer design, so I started out with some requirements (although these requirements might not be usable in a customer design 😉 )
- The homelab must be low-powered, preferably limited to 75 watts per host.
- The homelab must be powerful enough to run multiple virtual desktops similarly for benchmark testing.
- The homelab must support half-height PCIe NVIDIA Tesla GPUs.
- It should be easy enough to scale up/out (more memory, disk capacity/extra hosts).
- vSAN All-Flash must be supported, preferably with multiple disk groups.
- Because vSAN All-Flash needed to be supported, 10GbE had to be supported as well.
- The form factor of the individual hosts need to be as small as possible (since I have
half-height19-inch rack in my man cave.
- The hosts should make as less noise as possible.
- The hosts and vSAN components need to be on the VMware/vSAN HCL.
- IPMI is a must-have since I want to be able to remotely power-on/off the hosts.
The road to the build
My employer ITQ stimulates people to be creative (hence the F1 simulator on VDI). Another example of this is the fact that my colleague Laurens (
The E300-8D has a 1U small form-factor and after replacing the standard fans with
Although a Tesla P4 fits perfectly in a Supermicro E300-8D, the PCIe x8 to x16 riser and some extra cooling didn’t show the result I was hoping for. The PCIe x8 slot is unable to handle the bandwidth a Tesla P4 requires to perform well. And so, I had a perfect first management node for my new lab. Management because it’s unable to run virtual desktops and other VMs containing a (v)GPU.
As the power consumption did fit my needs (it only used about 45 Watts while running a vCenter, vROps, Horizon Connection Servers, an NSX Manager/Edges, and my UAGs). And since I only populated half the RAM slots, I could also expand if needed.
So, the search continued. The E300-9B was already announced, but the expected price for a single unit was double the
The challenge was the housing though. Because an NVIDIA Tesla P4 requires a lot of cooling to avoid it overheating and that cooling isn’t simply possible with an E300-8D housing. So, I decided to go for a slightly bigger form-factor, the Supermicro CSE-721TQ-250B Micro tower case.
So I got the board and the case, I ordered an Intel 760p NVMe 512GB drive as vSAN Cache disk and a Crucial MX500 2TB as a capacity disk, added some memory and as shown on the picture below, Noctua fans 🙂
The challenge with the board was that it is passively cooled and relies on an active airflow. Since the case doesn’t provide that, I had to come up with a solution. Luckily for me, someone else already did that. In this
The second challenge was the GPU. The Tesla P4 (like more Tesla GPUs) are passively cooled and not designed to run in such a tiny case without
The fan is attached to the front of the mount. The whole thing is attached to the back of the Tesla P4 and pushes the air in (instead of sucking the air out, which doesn’t offer sufficient cooling).
The file can be downloaded for free here:
With it you can print your own. After attaching everything, the host looks like this:
After mounting everything in my 19-inch rack, it looks like this:
You can see a Ubiquiti 24 port 1GbE switch, two Synology appliances and an Apple Timecapsule (which I will consolidate in a single Synology appliance in the future) and underneath the Supermicro E300-8D and two hosts ready for VDI/vGPU/vSAN All-Flash.
The following list is an overview of the components I used:
|Total per |
|2x||Supermicro X10SDV-4C-TLN4F||€ 527,57||€ 1.055,14|
|2x||Supermicro CSE-721TQ-250B||€ 172,66||€ 345,32|
|1x||Ubiquiti UniFi Switch 24-poorts||€ 190,-||€ 190,-|
|1x||Supermicro SuperServer SYS-E300-8D||€ 739,13||€ 739,13|
|6x||Noctua NF-A6x25 PWM, 60mm||€ 14,14||€ 84,84|
|4x||Crucial CT32G4RFD4266||€ 310,50||€ 1.242,-|
|3x||Crucial MX500 2,5″ 2TB||€ 279,-||€ 837,-|
|3x||Intel 760p 512GB||€ 122,95||€ 368,85|
|2x||NVIDIA Tesla P4||€ 2492,20||€ 4984,40|
|incl. GPUs||€ 9.846,68|
I added the prices of all components through a dutch site called tweakers.net. Everything is clickable to their
The installation of all the components was pretty straight forward. In
- Announcing vSphere Bitfusion as GPU acceleration platform - June 2, 2020
- Mastering voodoo: Building Linux VDI for Deep Learning workloads, Part 2 - April 2, 2020
- Lightboard Series: VMware Horizon DaaS - March 5, 2020