I recently got a 32GB M1 Mac Studio. I was excited to see how big of a model it could run. It turns out that’s 70B. It is a Q3_K_S model so the 2nd smallest for 70B in GGUF format, but still it’s a 70B model.
As many people know, the Mac shouldn’t be able to dedicate that much RAM to the GPU. Apple limits it to 67%, which is about 21GB. This model is 28GB. So it shouldn’t fit. But there’s a solution to that thanks to these smart people here.
https://github.com/ggerganov/llama.cpp/discussions/2182
They wrote a program to patch that limit in the kernel. You can set it to anything you want. So I cranked mine up to 92%. I also do these couple of things to save RAM.
-
I don’t use the GUI. Just simply logging in and doing nothing uses a fair amount of RAM. I run my Mac headless. I ssh in.
-
I stopped the mds_stores process from running. I saw that it was using up between 500MB and 1GB of RAM. Its the processes that indexes the drives for faster search. Considering my drive is 97% empty, I don’t know what it was doing to use up 1GB of RAM. I normally turn off indexing on all my machines always.
With all that set, the highest I’ve seen in use memory is 31.02GB while running a 70B Q3_K_S model. So there’s headroom. There maybe a lot more. Since my goal is to not swap. I noticed that when I log into the GUI while it’s running a model, the compressed RAM goes up to around 750MB but it still doesn’t swap. So I wonder how far memory compression would let me stretch it. I do notice that it’s not as snappy. With no GUI login, the model runs right away after the model is cached after the first run. With a GUI login, it pauses for a few seconds.
As for performance, it’s 14 t/s prompt and 4 t/s generation using the GPU. It’s 2 and 2 using the CPU. Power consumption is remarkably low. Using the GPU, powermetrics reports 39 watts for the entire machine but my wall monitor says it’s taking 79 watts from the wall. Using the CPU powermetrics reports 36 watts and the wall monitor says 63 watts. I don’t know why it’s so much more efficient at the wall between GPU and CPU. It’s only a 3 watt difference in the machine but 16 watts at the wall.
All in all, I’m super impressed. The M1 32GB Studio may be the runt of the Mac Studio lineup but considering that I paid about what a used 3090 costs on ebay for a new one, I think it’s the best value for performance I have to run LLMs. Since I plan on running this all out 24/7/365, the power savings alone compared to anything else with a GPU will be several hundreds of dollars a year.
You must log in or register to comment.
Last comment on GitHub seems a way safer option. On every reboot just do sudo sysctl iogpu.wired_limit_mb= and you’re done. No weird patching and booting and what nots.
But amazing knowledge again. Especially for the 192 gb version, this could open up some extra doors. I assume the m3 next year will be 256gb so that would bring some as we one models to the table!
Definitely. It’s a much better way to do it for a variety of reasons. Not least of which is that the kernel patch is kernel dependent so will need to be kept up to date. Setting this system variable isn’t. Unless Apple removes it. It should keep working in future releases of Mac OS.
Yes, this is great news! Just the other day I was trying to get the yi-34 model running at q3 on my 24GB MacBook Air and was literally like 50 MB away from getting it to load. I had a search for a way to bump up the max allocable RAM but didn’t see anything. This method works great, just tested it.
Pretty cool hack. Beats CPU inference at those speeds for sure.
The bandwidth utilization is not the best yet on gpu, its only 1/3rd of the potential 400GB/s.
The cpu RAM bandwidth utilization in llama.cpp on the otherhand, is nearly 100%, For my 32gb of DDR4, I get 1.5t/s with the 70B Q3_K_S model.
There will hopefully be more optimizations to speed this up.
I can’t wait for ultrafastbert. If that delivers on the promise then it’s a game changer that will propel CPU inference to the front of the pack. For 7B models up to a 78x speedup. The speedup decreases as the number of layers increase, but I’m hoping at 70B it’ll still be pretty significant.
There will hopefully be more optimizations to speed this up.
Speculative, Jacobi, or lookahead decoding could speed things up quite a bit.
How do you run headless to save on the excess ram of the GUI? I’ve looked for on google but there only tutorials about screen sharing.
I just don’t login using the GUI. There indeed doesn’t seem to be a way to turn it off like in Linux. So it still uses up 10s of MB waiting for you to login. But that’s a far cry from the 100’s of MB if you do login. I have thought about killing those login in processes but the Mac is so GUI centric that if something really goes wrong and I can’t ssh in, I want that as backup. I think a few 10’s of MB is worth it for that instead of trying to fix things in the terminal in recovery mode.
So it still uses up 10s of MB waiting for you to login. But that’s a far cry from the 100’s of MB if you do login
Thanks that clears things up a lot. I’m going to try it. And then I’ll look into maybe killing the login screen once I’ve ssh’ed in.
I think it means no display in.
There are no new 3090 so comparing the cost to a new 3090 is pointless as its basically just scalped overprized new 3090s left.
There are no new 3090 so comparing the cost to a new 3090 is pointless as its basically just scalped overprized new 3090s left.
I’m not comparing it to the cost of a new 3090. I clearly said I was comparing it to the price of a used 3090.
“The M1 32GB Studio may be the runt of the Mac Studio lineup but considering that I paid about what a used 3090 costs on ebay for a new one”
Awesome! I think I remember us talking about this at some point, but I didn’t have the courage to try it on my own machine. You’re the first person I’ve seen actually do the deed, and now I want to as well =D The 192GB Mac Studio stops at 147GB… I also run headless, so I can’t fathom that stupid bricks really needs 45GB of RAM to do normal stuff lol.
I am inspired. I’ll give it a go this weekend! Great work =D
please followup for actual memory limits, if you hit them
It’s impressive, but from what I have heard on this sub the big weakness of mac computers is the time it takes to evaluate the prompt. For example, if you are roleplaying with 8k context in Sillytavern and your prompt is continuously changing as world entries are triggered and character sheets are swapped during a group chat, then the evaluation speed of the prompt matters greatly.
That’s where context shifting comes into play. You don’t re-evaluate the entire context. You just process the additions.
“Previously, we had to re-evaluate the context when it becomes full and this could take a lot of time, especially on the CPU. Now, this is avoided by correctly updating the KV cache on-the-fly:”
Interesting, so even if the middle of the context changes between replies as world entries appear and disappear in the context, it will still not force a re-evaluation of the greater whole?
can the ram be upgraded, guessing no since it’s apple
Yea, I’m pissed I can’t upgrade the ram on my 3090 either. /s
what a nonsensical comparison.
have you ever posted in this sub before? also what country are you posting from?
It’s a perfectly sensical comparison. If anything, it’s far easier to upgrade the RAM on a 3090 than a M Mac.
you can’t but its not just apple being assholes like what they did with SSD’s, the “M” chips are completely integrated SOC’s that use unified memory that is shared between the GPU and CPU
I can run Q4 Falcon-180B on my M3 Max (40 GPU) with 128GB RAM. I get 2.5 t/s, it’s crazy for a mobile chip.
That is pretty good! What quant do you use?
Have you tried Goliath-120B, how fast it runs? It might be even better that a Falcon-180B, so might be worth trying.
There’s hardly any case for using the 70B chat model, most LLM tasks are happening just fine with Mistral-7b-instruct at 30tok/s
7b models are absolute garbage for RP with large context and world info entries.
Mine are mostly summarisation & extraction work so Mistral-instruct is way better than llama13b
Mine are mostly summarisation & extraction work so Mistral-instruct is way better than llama13b
I guess you’ve only used a 7B model. IMO, the magic doesn’t really start happening until 30B.
I am new to Mac Studio, primarily a laptop user. How do you run it headless?
You may try and run one of Q4 models without problems: because llama.cpp uses mmap to map files into memory, you can go above available RAM and because many models are sparse it will not use all mapped pages and even if it needs it, it will swap it out with other pages on demand… I was able to run falcon-180b-chat.Q6_K which uses about 141GB on a 128GB Windows PC with less than 1% SSD reads during inference… I could even run falcon-180b-chat.Q8 which uses about 182GB but in this case SSD was working heavily during inference and it was unbearably slow (0.01 tokens/second)…
Yes. I’ve done that before on my other machines. Llama.cpp in fact defaults to that. The hope for me was that since the models are sparse that the OS would cache the relevant parts of the models in RAM. So the first run through would be slow but subsequent runs would be fast since those pages are cached in RAM. How well that works or not really depends on how much RAM the OS is willing to use to cache mmap and how smartly it does it. My hope was that if it did it smarty with sparse data that it would be pretty fast. So far, my hopes haven’t been realized.
how about for those mere mortals with an entry level Macbook Pro M1 Pro 2022? Share a link to a friend?
Thanks for this, I’m about 3GB short of running Goliath-120b on my 64gb mbp.
I should probably give that a go on my 96GB M2 Max. I haven’t done much reading on that model,
Hey mate, no need to do this now. There’s a terminal command you can run instead. I did this on my M1 and it works fine.
> sudo sysctl iogpu.wired_limit_mb=57344
I did that for my 64GB, you’d want to change the 57344 to whatever you want for your 96GB
Does anyone know how many stream of LLAMA 2 70b a apple studio can run in parrallel ? Does it need the same amount of ram for each completion, or does llama.cpp manage to share it between different stream ?
As so often happens, the real LPT is in the comments. Using sysctl to change vram allocation is amazing. Thanks for this post.
Absolutely. That is a much better way to do it. But that was a recent development. More recent than this thread. That post at github about doing it that way happened 3 hours after I posted this thread.