Try HuggingFace Endpoints with any of the cheap T4 based serverless instances these go to sleep as well in 15mins.

Mine are mostly summarisation & extraction work so Mistral-instruct is way better than llama13b

Mine are mostly summarisation & extraction work so Mistral-instruct is way better than llama13b

HuggingFace has inference endpoint which is private & public as needed with sleep built in

Yeah 7B is no problem on phones even at 4tok/s

There’s hardly any case for using the 70B chat model, most LLM tasks are happening just fine with Mistral-7b-instruct at 30tok/s

It should be model page on HuggingFace, they also have a explicit template module which you can import automatically when interacting using model-id.

Llama ones are forgiving for not using structure but the mistral-instruct is very bad if structure is not maintained

The fastest way would be to ingest the ggerganov server.cpp module & make HTTP calls to it. Way easier to package into other apps & supports parallel decoding with 30tok/s on Apple Silicon(M1 Pro)

Try to use the instruct models like Mistral. Ensure your template is the correct one a well.

Have you tried a combination of mistral-instruct & Langchain? If not can you share some sample inputs you’re having problems with

7B models will work just fine with FP4 or INT4. Similarly 13B as well with little offloading if needed.

I reverted to mistral-instruct

Have you tried with FP4 & RAM offloading combined?

Most likely plugs multiple retrievers to figure out best candidates or linked content chunks to derive an answer

If you have a GPU then I’d suggest setting up a TGI container with the correct model. If no GPU is available then use the server.cpp example in llama.cpp repository & simply invoke it from your GUI.

I’m looking to build one called OtherBrain, should work with local files as well & will live in the menu bar on macOS. Since Apple Silicon can do ~20tok/s all companies with employees who have newer Mac’s should just leverage this amazing local inference device for daily workflows

I feel like LLMs should just do basic calculations themselves & everything else they should invoke a tool for verification

Use langchain with tools

The attention layers get replaced with flash attention 2, there’s kv caching as well so you get way better batch1 & batchN results with continuous batching for every request

Run this with TGI or vLLM