That depends, when you say you are building out a new server, are we talking a proper 1 or 2u dell, HPE, etc type server? If so you’ll have to contend with the GPU footprint, for example my 1u servers can only take up to 2, half height, half length GPU’s, and they can only be powered by PCIE so I’m limited to 75w.

In my 2u servers I can get the “GPU enablement kit” which is essentially smaller form factor heatsinks for the CPU’s and some long 8pin power connectors to go from the the mobo to the PCIE riser, allowing many more options, but still there are problems to address with heat, power draw (CPUs are limited to 130TDP I believe) and the server firmware complaining about the GPU/forcing the system fans to run at an obnoxious level, etc…

If you are homebrewing a 3u, a tower or using consumer parts than things change quite a bit.

Can you please share a bit more about your setup and experiences?

I’ve been looking to use some of my idle enterprise gear for LLM’s but everyone tells me not to bother. I’ve got a few dual xeon boxes with quad channel DDR4 in 256 & 384GB capacities, NVMe or RAID10 SSDs, 10GBe, etc and I guess (having not yet experienced it) I have a hard time imagining that the equivalent of 120Ghz, 1/2 - 1tb of RAM and 7GB/s disk reads “not being fast enough.” I don’t need instant responses from a sex chatbot, rather I would like to run a model that can help my wife (in the medical field) with work queries, to help my school age kid with math and grammar questions, etc.

Thank you much!