Hi there, Im looking to buy an apple laptop and I saw a macbook pro m1 max with 64gb ram and 2TB ssd for 2400 usd Will this computer be able to run the big models at reasonable speed?

I was going to buy the simple macbook air m1 8gb ram for 700usd but I saw this and I always wanted to play with LLMs but never could.

Any advice is appreciated, thanks

  • irregardless@alien.topBEnglish
    1·
    1 year ago

    Totally feasible to run LLMs at useful speeds. I’m running a 64gb 10/32 M1 Max. With LM Studio, I typically get

    • 3-4 T/s using q5_k_m quants of ~70B models
    • 6-9 T/s from q5_* and q6_k quants of ~30G models
    • 25-30 T/s from q6_k and q8 quants of 7B models
    • around 20 T/s from unquantized fp16 7B models

    And this is my daily work and play machine, so I usually have all sorts of browser tabs and applications open simultaneously while running the models. From a fresh boot, it’s cool to be able to load an entire model into memory and still be able to do “normal” work without having to use any swap space at all.