alchemist1e9@alien.topB to LocalLLaMAEnglish · 2 years agoExLlamaV2: The Fastest Library to Run LLMstowardsdatascience.comexternal-linkmessage-square22linkfedilinkarrow-up11arrow-down10file-text
arrow-up11arrow-down1external-linkExLlamaV2: The Fastest Library to Run LLMstowardsdatascience.comalchemist1e9@alien.topB to LocalLLaMAEnglish · 2 years agomessage-square22linkfedilinkfile-text
minus-squareCardAnarchist@alien.topBlinkfedilinkEnglisharrow-up1·2 years agoCan you offload layers with this like GGUF? I don’t have much VRAM / RAM so even when running a 7B I have to partially offload layers.
Can you offload layers with this like GGUF?
I don’t have much VRAM / RAM so even when running a 7B I have to partially offload layers.