llama.cpp mostly, just on console with main.exe. Wrote a simple python file to talk to the llama.cpp server which also works great. LM Studio is good and i have it installed but i dont use it, i have a 8gb vram laptop gpu at office and 6gb vram laptop gpu at home so i make myself keep used to using the console to save memory where ever i can. My experience with text gen web ui has not been great, its takes far far too long to update, and sometimes it gets the torch installation right and sometimes torch is not installed with cuda. I really dont want to waste my time on that. I like to install everything manually and want some really light weight web ui to just use the server hosted with llama.cpp.
llama.cpp mostly, just on console with main.exe. Wrote a simple python file to talk to the llama.cpp server which also works great. LM Studio is good and i have it installed but i dont use it, i have a 8gb vram laptop gpu at office and 6gb vram laptop gpu at home so i make myself keep used to using the console to save memory where ever i can. My experience with text gen web ui has not been great, its takes far far too long to update, and sometimes it gets the torch installation right and sometimes torch is not installed with cuda. I really dont want to waste my time on that. I like to install everything manually and want some really light weight web ui to just use the server hosted with llama.cpp.