I’ve read a lot of hardware discussions here, but never a mention of cards like AMD 102D1631200 Radeon Instinct MI60 32GB HBM2 . Their VRAM and bandwidth are certainly fine, the price more than fine (used only), is there a devil in the details?
You must log in or register to comment.
I’d just be worried they will drop support for them in ROCm 6.0. They dropped the MI-50’s already. Technically you can still run them and the other MI25 but ROCm is kernel specific so before long you might have to maintain a system with an old kernel to have it working. I have a pair of MI100s and while they do work fine, they are slower than NVidia 3090s when used with llama.cpp, exLLama, and Koboldcpp for some reason. It looks like with the new release of flashattention-2 the MI210 is the oldest card they support which I find very frustrating. I also have a couple W6800’s and they are actually as fast or faster than the MI100s with the same software and about the same price and have built in cooling.
They work fine but have a couple pitfalls:
- rocm support across software is spotty, but does work when available (mostly)
- rocm is less well supported os wise (have to use specific, blessed distros / kernels)
- most llm software is optimized for nvidia hardware, perf is worse than it could be w/ bespoke cuda kernels