minus-squaretvetus@alien.topBtoLocalLLaMA•AMD EPYC CPU or 1x RTX 4090?linkfedilinkEnglisharrow-up1·3 years ago at full precision Full precision is not as useful as you think. Even at 4bit, the losses are not that large. 70B What is your motivation for such large models? You’re sacrificing a lot of speed for the larger model. linkfedilink
minus-squaretvetus@alien.topBtoLocalLLaMA•I am going to buy H100s. There are too many options.linkfedilinkEnglisharrow-up1·3 years agoCan be hard to rent if all the capacity is bought out. But if it’s just 1 DGX then they might be better off renting. linkfedilink
Full precision is not as useful as you think. Even at 4bit, the losses are not that large.
What is your motivation for such large models? You’re sacrificing a lot of speed for the larger model.