Local LLMs are a gold mine for IT consultants right now.

Not only is there a market, I’d say a substantial portion of data departments across the economy will dedicate significant resource to this. Once generic models become common place, specialized ones will be where you edge out the competition

Aren’t there people selling such services to companies here? Implementing RAG, etc.

Yes. We are currently planning AI project and implementations in my company. We are handling sensitive data which requires us to do it local. And we want to setup a team and establish compentence and more experience in ML/LLMs. For our current use cases we don’t need a “super intelligence” or “the best” LLM on the market ! RAG with smaller models is totally fine and sufficient for us.

Yes there is. Just to mention one of many scenarios:

Make a chatbot based on a LLM. Learn how to train an LLM from data from the customer. Voila! Your customer just aquired a new employee to handle customer support or internal support.

There is a market. Might be a small market, but there definitely is. Just like Linux!

The entire market will eventually use local LLMs, it’s simply as that.

Online services are never an ideal solution for any business. It’s not just about privacy.

- The owners can do whatever they want, so if they change settings or even simply shut down, you’re screwed.

- Online services are a public traffick, so in case of high-density, it bottlenecks. Just like highroads, expect major slowdowns of your job if you use an online service that happens to be saturated. And slowdown means financiary loss.

- In case of internet issues, you’re screwed.

- you have to pay for the service, which can get freaking expensive depending on how much you use it.

​

Local LLMs have none of these issues. And more than that:

- While general intelligence like ChatGPT or Claude is incredible, it will never be enough for every use case. There will always be cases where you need a more specialized alternative, even if less intelligent.

- The gap between the big ones and local LLMs is frankly not that high. I’m not going to say “they’re as intelligent as ChatGPT!”, but as a matter of fact, everything I was able to make with ChatGPT, I succeeded with a local LLM as well or even better. Analysing code and rewriting it/completing it? Managed with a 7B. Writing creative short stories? Easy even with a 7B.

- An online service has its own abilities, and the devs can update it but you have no guarantee they will. In the case of LLMs, context length matters so much! OpenAI did raise GPT’s context length regularly, but what if they don’t?

- Intelligence isn’t the only point about an AI! A local LLM has its own default language style, and even the big ones are hard to steer away from it. ChatGPT’s answers, for example, are very lengthy, constantly. With a Local LLM, it’s easier to steer. You can even force it to adopt a certain format.

OpenAI is also so overwhelmed with demand that their API “limits” are a complete joke. I have a tier 4 account and have always hit server error issues when running any kind of single threaded application with their API well before hitting any of their published limits. I get more TPS from my local systems than I was ever able to get from the OpenAI API.

Yes, there are a few reasons. Companies might want them offline, fine-tuned to their industry/company and lower operating cost are the ones that come to mind.

Absolutely (I’m a datascientist too), I own a fresh company in ML/DL/AI and it works pretty well.

There is so much room for creation even with current models (RAG pipeline on private data, prototyping locally and train or finetune on instances, real world application mixing others branch like CV, cinematography with SD, implementing liquid NN with LLM and so on)