1·
1 year agoSwitch to using YARN is the best I’m aware of at the moment.
YARN is basically dynamic alpha scaling with extra steps, functions better without fine tuning, and also benefits from fine tuning.
- 0 Posts
- 8 Comments
Joined 1 year ago
Cake day: October 30th, 2023
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
Gonna be honest, you can totally just skip LlamaSharp and call the Llama.dll methods using interop in C#
Its really not difficult to do and it cuts an entire layer of dependency out of your project.
I actually don’t know how much overhead that’s going to be. I’d start by just kicking it off on the command line first as a proof of concept, its super easy,
5_K_M is just the quantization I use. There’s almost no loss of perplexity with 5_K_M, but its also larger than 4 which is what most people use.
Name Quant method Bits Size Max RAM required Use case goat-70b-storytelling.Q2_K.gguf Q2_K 2 29.28 GB 31.78 GB smallest, significant quality loss - not recommended for most purposes goat-70b-storytelling.Q3_K_S.gguf Q3_K_S 3 29.92 GB 32.42 GB very small, high quality loss goat-70b-storytelling.Q3_K_M.gguf Q3_K_M 3 33.19 GB 35.69 GB very small, high quality loss goat-70b-storytelling.Q3_K_L.gguf Q3_K_L 3 36.15 GB 38.65 GB small, substantial quality loss goat-70b-storytelling.Q4_0.gguf Q4_0 4 38.87 GB 41.37 GB legacy; small, very high quality loss - prefer using Q3_K_M goat-70b-storytelling.Q4_K_S.gguf Q4_K_S 4 39.07 GB 41.57 GB small, greater quality loss goat-70b-storytelling.Q4_K_M.gguf Q4_K_M 4 41.42 GB 43.92 GB medium, balanced quality - recommended goat-70b-storytelling.Q5_0.gguf Q5_0 5 47.46 GB 49.96 GB legacy; medium, balanced quality - prefer using Q4_K_M goat-70b-storytelling.Q5_K_S.gguf Q5_K_S 5 47.46 GB 49.96 GB large, low quality loss - recommended goat-70b-storytelling.Q5_K_M.gguf Q5_K_M 5 48.75 GB 51.25 GB large, very low quality loss - recommended goat-70b-storytelling.Q6_K.gguf Q6_K 6 56.59 GB 59.09 GB very large, extremely low quality loss goat-70b-storytelling.Q8_0.gguf Q8_0 8 73.29 GB 75.79 GB very large, extremely low quality loss - not recommended
If you’re only getting 0.1 then you’ve probably overshot your layer offloading.
I can get up to 1.5 t/s with a 3090, at 5_K_M
Try running Llama.cpp from the command line with 30 layers offloaded to the gpu, and make sure your thread count is set to match your (physical) CPU core count
The other problem you’re likely running into is that 64gb of RAM is cutting it pretty close. Make sure your base OS usage is below 8GB if possible and try memory locking the model on load. The problem is that with that amount of system ram, its possible you have other applications running causing the OS to page the model data out to disk, which kills performance
/u/Noxusequal
I just went through this exact issue building Llama.cpp inside an Ubuntu docker container. If you don’t have nvcc installed it will compile without error, but wont include CUDA support regardless of what you set for options. Check to make sure nvcc is installed in the machine
If you’re trying to do something novel as part of a learning experiment, just pick a good UI framework and then either wrap LlamaSharp, or interop directly with Llama.cpp using PInvoke.
Personally I just use PInvoke to cut out the middle man
As for a UI framework, I’ve been having a lot of fun with Avalonia lately.
You’re not going to get 100% interop with all models using Llama.cpp as a core but if this is a learning exercise then I’m sure that’s not an issue.
That being said, if you really want to you can fuck around with Python.net but you may find yourself spending way more time trying to manage model interop and execution than you want to.
I tried coaxing an answer out of it, and as far as I got was
- One random redditors comment saying “next year” was announced at Meta Connect
- “It makes sense” due to the spacing between 1 and 2
So I don’t know much about architecture but I’m assuming if we want to run something like this in Llama, we’re going to need to submit a request? If its ground up, then pretty much everything is going to need to be implemented, right?