I dont have budget for hosting models on dedicated GPU, what are the alternative options or platforms that let me use Opensource models like mistral, Llamas, etc in a pay per API call basis ?
I dont have budget for hosting models on dedicated GPU, what are the alternative options or platforms that let me use Opensource models like mistral, Llamas, etc in a pay per API call basis ?
It may be out of your range, but you can pick up the dell precision 7720 with a 16gb P5000 GPU for about $500 on eBay. The Quadro P5000 is also in a few other workstation laptop models around that era. Note: They had other graphics options so only go for P5000 models.