Some 70b llama in 8bit GGUF would be cool, you can play with Goliath 120b in <8 bpw.

over Skype, right?;)

I am not able to select and copy any text while generating. Seems like a UX bug where the selection disappears with each token streamed in.

the values here seem off, not normalized, but I like the idea.

https://preview.redd.it/rmmx9kclh33c1.png?width=183&amp;format=png&amp;auto=webp&amp;s=03a94c6b8d21dee12ab9822f4206752296e08172

not sure if usable, but “rounds” or “amount” seem good alternatives.

Maybe wrong suggestion, but I got used to have /docs endpoint with description of the endpoints available, would you consider adding it too u/Evening_Ad6637?
It could point to/render https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md#api-endpoints at first, anyway seems helpful to have it served.

Can it serve on a CPU-only machine?

I’ve got mixed experiences with Bavarder, native UI, fair choice of models to grab, but offen not working reliably. They seem to improve it slowly but steadily.

What is needed to get it done? Can anyone help or only a few days of your focused time are expected to lead to it?

I assume auth_token is for storing the merged model in HF? Seems worth noting/clarifying.

I’ll get back with more feedback when I get to test it.+)

keep my friends in https://alignmentjam.com/jams cool,
they are amazing and fun!

Most alignment folks do not care about the polite correctness sht at all, but want humanity not killed nor enslaved.

tried twice the NurtureAI model and failed in various ways both times:

https://preview.redd.it/ntxebw31rq0c1.png?width=1526&amp;format=png&amp;auto=webp&amp;s=510045cfdb55d353f8f5215e8c4c441754fb1a02

Fairly great. There is no Save button for the generation Configuration and I see max_new_tokens reseting quickly( like reload page or selecting other model).

Plans to support markdown display?+)