minus-squarelongtimegoneMTGO@alien.topBtoLocalLLaMA•Question about GGUF, gpu offload and performancelinkfedilinkEnglisharrow-up1·1 year agoI have a 3080 12Gb, and can run a 20B-Q4_K_M with about 50 layers offloaded and 8k context. It starts off at just under 4 t/s, and once the context is filled it gets as slow as just over 2 t/s It might be worth setting up a linux partition to boot into for this, I was getting much slower speeds under windows. linkfedilink
I have a 3080 12Gb, and can run a 20B-Q4_K_M with about 50 layers offloaded and 8k context.
It starts off at just under 4 t/s, and once the context is filled it gets as slow as just over 2 t/s
It might be worth setting up a linux partition to boot into for this, I was getting much slower speeds under windows.