I’m currently using 1650 4GB, AMD 5600, 32GB RAM.

I got some spare cash to throw to learn more about local llm.

Should I get: A. 64 GB RAM (2 X 32GB) B. 3060 12GB C. Intel A770 16GB.

I’m using openhermes 2.5 Mistral 7b q5k_m gguf, ok-ish Performace for Silly tavern with koboldcpp. But when context goes above 3k, it crawled.

Please let advise which option you think I should take first. Thanks bunch.

  • tech92yc@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 years ago

    get a used Nvidia GPU , the Cuda acceleration changes everything (x20-x50 performance)

    Don’t waste your time on CPU inference, also Intel A770 doesn’t have the software support