I’m currently using 1650 4GB, AMD 5600, 32GB RAM.

I got some spare cash to throw to learn more about local llm.

Should I get: A. 64 GB RAM (2 X 32GB) B. 3060 12GB C. Intel A770 16GB.

I’m using openhermes 2.5 Mistral 7b q5k_m gguf, ok-ish Performace for Silly tavern with koboldcpp. But when context goes above 3k, it crawled.

Please let advise which option you think I should take first. Thanks bunch.

  • bebopkim1372@alien.topBEnglish
    1·
    1 year ago

    My main computer is M1 Max Mac Studio. It has 64GB of memory and I can use up to 48GB of video memory. However, it is difficult to use because the modules, libraries, and software support are not very good to use. If you’re a software developer, you will have tough time to make everything working well.

    I bought 4060 Ti 16GB 2 months ago, and I felt that it was very easy making everything runs well on CUDA Development Kit. With Metal from Apple Silicon, I had quite tough time. With Metal environment, I almost always had quite minor problems, and sometimes there was no solution at all. But with NVIDIA’s GPU, things like this never happened. Only small VRAM is the problem.

    I have no experience with A770, but I guess it is similar with Metal.

  • uti24@alien.topBEnglish
    1·
    1 year ago

    Well, it depends.

    You can not run 70B models with RTX 3060, but you can with 64Gb of memory.

  • mcmoose1900@alien.topBEnglish
    1·
    1 year ago

    So, definitely a new GPU. You can’t go wrong with either, both will easily hold 7B or 13B at long context.

    The 3060 will work better right now. It has support for much better backends at the moment and will be way faster.

    I’m partial to the A770 because its stronger on paper, and I believe its going to get faster imminently with more support from various backends. It should be faster in the longer term. Also, I’m very salty about Nvidia’s price gouging and anti competitiveness.

    So… I guess it depends on when your next upgrade will be. I myself am thinking I will replace my 3090 with Intel’s next gen GPUs if they’re any good (and 24GB+).

    • yahdahduhe@alien.topOPBEnglish
      1·
      1 year ago

      This is the dilemma for me. The a770 is the first gen GPU of Intel, while the news has been lately very promising, but battlemage seems to be just around the corner, based on rumours.

    • twisted7ogic@alien.topBEnglish
      1·
      1 year ago

      I’m in the same mind. Nvidia is the best choice right now, but it’s also a super overpriced brand that really skimps on vram and is acting pretty toxic to it’s end users.

      Long term, Intel (and AMD) might be better options, possibly probably, eventually.

  • iamkucuk@alien.topBEnglish
    1·
    1 year ago

    3060 would cause fewer headaches. A770 would be better once you get things working. Btw, a770 has somewhat mature support for ai workload, too.

  • tech92yc@alien.topBEnglish
    1·
    1 year ago

    get a used Nvidia GPU , the Cuda acceleration changes everything (x20-x50 performance)

    Don’t waste your time on CPU inference, also Intel A770 doesn’t have the software support