WizardLM is really old by now. Have you tried any of the Mistral finetunes? Don’t discount it just because of the low parameter count. I was also running WizardLM-33b-4bit for the longest time, but Mistral-Hermes-2.5-7b-8bit is just so much more capable for what I need.
I haven’t been able to use that with my 3090Ti yet. I tried TheBloke’s GPTQ and GGUF (4bit) versions. The first runs into memory issues, the second, loaded with llama.cpp (which it seems to be configured on) loads, but is excruciatingly slow (like 0.07t/sec).
I must admit that I am a complete noob regarding all the different variants and model loaders.
Koboldcpp is the easiest way.
Get nous-capybara-34b.Q4_K_M.gguf (it just fits into 24GB VRAM with 8K context).
Here are my Koboldcpp settings (not sure if they are optimal but they work)
A 34B model is the best fit for a 24GB GPU right now. Good speed and huge context window.
nous-capybara-34b is a good start
I’ve been going with WizardLM-33B-V1.0-Uncensored-GPTQ for a while and it’s okay. Is Nous-Capybara-34b better?
Well yes, WizarldLM-33b is 5 months old, a lot of things happened since then.
WizardLM is really old by now. Have you tried any of the Mistral finetunes? Don’t discount it just because of the low parameter count. I was also running WizardLM-33b-4bit for the longest time, but Mistral-Hermes-2.5-7b-8bit is just so much more capable for what I need.
I’ve tried that one. It is… strange.
I haven’t been able to use that with my 3090Ti yet. I tried TheBloke’s GPTQ and GGUF (4bit) versions. The first runs into memory issues, the second, loaded with llama.cpp (which it seems to be configured on) loads, but is excruciatingly slow (like 0.07t/sec).
I must admit that I am a complete noob regarding all the different variants and model loaders.
Koboldcpp is the easiest way.
Get nous-capybara-34b.Q4_K_M.gguf (it just fits into 24GB VRAM with 8K context).
Here are my Koboldcpp settings (not sure if they are optimal but they work)
https://preview.redd.it/dco0bokvic1c1.jpeg?width=540&format=pjpg&auto=webp&s=bf188ea61481a9464593db79d690b26eb7989883