So Mistral-7b is a pretty impressive 7B param model … but why is it so capable? Do we have any insights into its dataset? Was it trained very far beyond the scaling limit? Any attempts at open reproductions or merges to scale up # of params?
You must log in or register to comment.
It’s simply the time bonus - coming after all the big models.
- better filtering - kill outright junk
- you use already big models (OpenAI and LLama) that you can use for data tuning and filtering
- use available synthetic data
They matched parameters and tokens when training.
Podcast on Spotify “No Priors” has the CEO of Mistral on who discusses this.
I don’t know what this means but will listen to the podcast to find out
Having used it a lot, I can say for sure that without much prompting it readily produces junk web text, urls etc, so it is not a fully filtered or fully synthetic dataset.
My guess would be that it’s just ‘a bit better filtered than llama-2’, and maybe slightly more trained on that set. Slightly better quality set, slightly more trained on that set.
My intuition based on this, is that per parameter size EVERYTHING open source could be optimized considerably more.
Is there any version of mistral or llama2 with RHLF applied to make tasks of text summarisation without having the censorship. Sometimes the output is totally different from what one could expect with the input sentences. Even if I state in the prompt to avoid applying censorship and focus on the input.
We’re only in the first inning too. Buckle up
My current hunch is that they use a lot of non easily accessible online ressources (including a specific archive owned by someone named Anna).
oh, anna !
Do people find that it holds up in use? Or are we mostly going on benchmarks? I’m skeptical of benchmarks, and a highly performant 7B model would be of great use.
French qualité. Yes, this is a thing now. Get used to it. HuggingFace is french too.
I second this. Mistral-7B gave me good results. After fine-tuning it’s result is even better.
Mistral-7B gave me good results
Can you expand upon that? Do you mean in terms of its ability to write at a college level without major grammatical errors?
Are there notable finetunes to your knowledge? I’ve started using LLMs today, starting with openorca mistral 7B and it seems pretty good.
On HuggingFace you can find many fine-tuned/quantized models. Look for models from TheBloke on HuggingFace.
It doesn’t seem too capable. Has anyone else tried running this locally or on runpod?
Trained on a larger # of tokens. All the llama models are under trained it appears, especially the 70b
They didn’t lobotomize it for safety.
Lack of censorship is a key factor as it maximises the predictive abilities of the model.
I assume the progress is based on well structured, high quality training data, combined with an incremental “learning schedule”. At least that’s where some reports of massive progress seem to be coming from and it’s also very intuitive that this would help a lot.
The results are okay, but I’m hard-pressed to call it “very capable”. My perspective on it is that other bigger models are making mistakes they shouldn’t be making because they were “trained wrong”.