I use C#. Initially I’d gone all out trying to wrap Llama.cpp myself, but I was getting outdated in a matter of weeks and it was going to take a ton of effort to keep up.
So instead I run a local ooba server and use the api. I get to do all my business logic in nice, structured C#, while all the python stuff says in ooba and I don’t have to dig into it really at all.
Let me start off by saying I haven’t gotten this right yet. But still…
When autogpt went viral, the thing everyone started talking about was vector db’s and how they can magically extend the context window. This was not a very informed idea and implementations have been lacking.
It turns out that merely finding similar messages in the history and dumping them into the context is not enough. While this may sometimes give you a valuable nugget, most of the time it will just fill the context with repetitive garbage.
What you really need for this to work imo, is a structured narrative around finding the data, reading it, and reporting the data. LLMs respond extremely poorly to random, disconnected dialogue. They don’t know what to do with it. So for one thing, you’ll need a reasonable amount of pre-context for each data point so that the bot can even understand what’s being talked about. But now this is prohibitively long, 4 or 5 matches on your search and your context is probably full. So you’ll need to do some summarizing before squeezing it into the live conversation, which means your request takes 2x longer, at a minimum, and then you need to weave that into your chat context in as natural a way as possible.
Honestly RAG as a task is so weird that I no longer expect any general models to be capable of it. Especially not 7B/13B. Even gpt4 can just barely do it. I think with a very clever dataset somebody could make an effective RAG Lora, but I’ve yet to see it.
I have the 16gb version of 4060ti, so the cards have nearly identical capabilities.
i can’t speak for the desktop 3080ti, but i have that laptop card and it’s roughly equivalent in performance to my 4060ti desktop card.
i did this 1.1B TinyLlama quant last week. it’s not smart by any means but it can hold a conversation somewhat. and, crucially, it’s 600mb
my personal lora does this just because it was trained on actual human conversations. it’s super unnatural for people to try answering just any off the wall question, most people will just go like “lmao” or “idk, wtf” and if you methodically strip that from the data (like most instruct datasets do) then it makes the bots act weird as hell
If you’re not using a chat/instruct tuned model you should be using the notebook, the input that the chat tab creates will be chat/instruct formatted
I can’t speak for 1B models but you’re going to have a really hard time training with no gpu. It’s just going to take an insanely long time.
For $500 though you can get a 4060ti with 16gb of ram which is good enough to train a 13B lora
without knowing what you’ve tried it’s impossible to really know what to recommend. tiefighter or openhermes2.5 is probably your best bet
Mistral and Llama2 (and Llama) are foundation models, meaning they actually trained all the weights given. Almost anything worth using is a derivative of these 3 foundation models. They are really expensive to train.
Just about everything else is a Lora fine tune on top of one of them. Fine tunes only change a small fraction of the weights, like 1%. Functionally speaking, the important part of these is the additional data they were trained on, and that training can be done on any underlying model.
So Open hermes is a Lora tuning on top of mistral, and is some opensource offshoot of nous hermes, which is an instruction dataset for giving good smart answers (or something) in a given instruction format.
It doesn’t, it just has a function to go look at your files
i’ve been having some success just asking gpt4 to write a quiz over each chapter,and then write a scenario in which my character answers those questions. this was pretty easy to automate and costs a few dollars per book basically. I had to use gpt-3.5-16k for a few very long chapters, and the quality is very much worse.
if you’re making a lora, training on wikipedia directly will pretty much make it output text that looks like wikipedia. which is to say it will (probably) be worse at chatting.
a strategy i’ve been using lately is to get gpt4 to make a conversation in my chosen format *about* each chapter of my “textbook”, i can automate this with pretty good results and it’s done in about 10 minutes. It does kind of work, it’ll at least get the bot to talk about the topics I chose, but as far as actually comprehending the information it’s referencing… it’s bad. It gets better as I increase rank, but it takes a lot of VRAM. I can only get to around 256 before it’ll die
you could do a rag that uses vector embeddings, but you could also just ask the llm for a search query and use that to search a database and that would still be a rag
ggml is totally deprecated, so much so that the make-ggml.py script in llama.cpp now makes ggufs
This isn’t an original concept but the data transfer rates make distributed training like this pretty impractical. Unless you’re doing some galaxy brained techniques you will need to distribute like 120gb every step (for a 70B model) and training on an old step is a waste of time. So parallelizing it on internet connected machines is a horrible option.
You don’t really need this setup anyways as you can train a 4bit 13B Lora on a gaming pc. This would only be useful for big foundation models, maybe, in which case you’re going to get way faster results by just renting some cloud gpus
I’m not doing that but my guess is it’s fun, easy, and cheap to do (only $8/mo!) and potentially lucrative if you can cheese a following somehow.
Using gpt is really lazy though when it’s so easy to do a custom 13B Lora that will actually interact like a human