We’ve seen pretty amazing performance of mistral 7b when comparing with Llama 34B & Llama2 13B. I’m curious, theoretically, will it be possible to build an SLM, with 7-8B parameters, able to outperform GPT4 in all tasks? If so, what are potential difficulties / problems to solve? And when do you expect such SLM to come?

ps: sorry for the typo. This is my real question.

Is it possible for SLM to outperform GPT4 in all tasks?

  • steph_pop@alien.topBEnglish
    1·
    1 year ago

    It’s unfair to compare standalone LLMs with GPT4 which is whole engineering system we know nothing about.

    People are working for improving quality of LLM and reduce their sizes for sure and you can always train a 7B to be very good at some tasks and beat a bigger model but only on this small task.
    However the lower the number of parameters, the less the model can handle complex tasks, and the less it can be good at several different tasks at the same time.

    Take a look to the tests made by https://www.reddit.com/r/LocalLLaMA/comments/17vcr9d/llm_comparisontest_2x_34b_yi_dolphin_nous/

    • bortlip@alien.topBEnglish
      1·
      1 year ago

      It’s not really about fairness though, it’s about knowing where things stand.

      I’ve used GPT 4 a lot so I have a rough idea of what it can do in general, but I’ve almost no experience with local LLMs. That’s something I’ve only played a little with recently after seeing the advances in the past year.

      So, I don’t really see it as a question that disparages local LLMs, so I don’t see fairness as an issue - it’s not a competition to me.