What is there differentiating factor, or are they planning on being another one of maybe hundred or so companies copy-pasting the same basic architecture, and the same basic training data?
I think the proliferation of smaller LLMs is wonderful but none have really placed a dent on the capabilities of the best closed source models (mostly OpenAI), which is largely due to model size. Beyond model size even, there seems to be no real innovation happening in architecture, design, or UX between Falcon, Mistral, Llama, Yi, etc. etc.
LLMs seem like a black hole in VC space, gambling at the level of billionaires. It reminds me of the talk given by Warren Buffet years back on how hard difficult it is to predict winners even when you know a technology is inevitable:
"There were two thousand auto companies: the most important invention, probably, of the first half of the twentieth century. It had an enormous impact on people’s lives. If you had seen at the time of the first cars how this country would develop in connection with autos, you would have said, ‘This is the place I must be.’ But of the two thousand companies, as of a few years ago, only three car companies survived.21 And, at one time or another, all three were selling for less than book value, which is the amount of money that had been put into the companies and left there. So autos had an enormous impact on America, but in the opposite direction on investors.”
And also with respect to airline companies:
“Now the other great invention of the first half of the century was the airplane. In this period from 1919 to 1939, there were about two hundred companies. Imagine if you could have seen the future of the airline industry back there at Kitty Hawk. You would have seen a world undreamed of. But assume you had the insight, and you saw all of these people wishing to fly and to visit their relatives or run away from their relatives or whatever you do in an airplane, and you decided this was the place to be.
As of a couple of years ago, there had been zero money made from the aggregate of all stock investments in the airline industry in history."
This was an insightful comment. The winnowing effect of market conditions should not be underestimated.
I love the Wild West that is the local LLM scene right now, but I wonder how long the party will last. I predict that the groups with the capacity to produce novel, state-of-the-art LLMs will be seduced by profit to keep those models closed, and as those models that could run on consumer hardware become increasingly capable, the safety concerns (legitimate or not) will eventually smother their open nature. We may continue to get weights for toy versions of those new flagship models, but I suspect their creators will reserve the top-shelf stuff for their subscription customers, and they can easily cite safety as a reason for it. I can’t really blame them, either. Why give it away for free when you can become rich off your invention?
What is there differentiating factor, or are they planning on being another one of maybe hundred or so companies copy-pasting the same basic architecture, and the same basic training data?
I think the proliferation of smaller LLMs is wonderful but none have really placed a dent on the capabilities of the best closed source models (mostly OpenAI), which is largely due to model size. Beyond model size even, there seems to be no real innovation happening in architecture, design, or UX between Falcon, Mistral, Llama, Yi, etc. etc.
LLMs seem like a black hole in VC space, gambling at the level of billionaires. It reminds me of the talk given by Warren Buffet years back on how hard difficult it is to predict winners even when you know a technology is inevitable:
And also with respect to airline companies:
Taken from The Snowball by Alice Schroeder
This was an insightful comment. The winnowing effect of market conditions should not be underestimated.
I love the Wild West that is the local LLM scene right now, but I wonder how long the party will last. I predict that the groups with the capacity to produce novel, state-of-the-art LLMs will be seduced by profit to keep those models closed, and as those models that could run on consumer hardware become increasingly capable, the safety concerns (legitimate or not) will eventually smother their open nature. We may continue to get weights for toy versions of those new flagship models, but I suspect their creators will reserve the top-shelf stuff for their subscription customers, and they can easily cite safety as a reason for it. I can’t really blame them, either. Why give it away for free when you can become rich off your invention?
Hopefully I’ll be proven wrong. 🤞 We’ll see…
I disagree. I would argue GPT-4 is useful because of the sea of augmentations built up around it.
If it was just raw responses from a single base model (like most local LLMs), with no preprocessing, I believe GPT-4 would be much less impressive.