Doesn’t the LlamaCpp server host a GUI for multimodal? You could potentially visit it, open the developer panel in your browser, and observe the HTTP requests being sent.

Thanks for the input.

What inference engine did you use? It’s possibly a bug as these things tend to happen with the new models.
I for one can’t wait for the lookahead decoding in llamacpp and others, combine that with some smaller models and we’ll have blazing fast speeds on pennies worth of hardware from what i recon.

There is new rocket 3b that might be worth a try. It’s suspiciously high in benchmarks so i suspect contamination of the dataset, but I saw people have good experience with it.

Future is going to be interesting. With this kind of CPU speedup we can run blazing fast LLMs on a toaster if it has enough RAM.

Oh wow, this seems almost too good to be true

Oh wow, I know the results are probably cherry picked, but this still seems like such a step-up.

Yes! I’ve been waiting for progress in video for a while! Imagine dyi automated classification for the sake of compilations and edits. This is going to be sick! Can’t wait and see an implementation on llamacpp

I can’t wait to see some finetunes of openchat-3.5. this thing is way too smart for a 7b. Frankly I am amazed at how fast we went from 7b can’t keep it togheter to “this 7b is pretty much on par with chatgpt-3.5” (for a lot of use cases at least)

I hope we can get quantized gguf soon from the legendary TheBloke

Also i’d give the new openchat 3.5, if the benchmarks are indeed correct then it’s the best 7B model so far (altough there are so many of them that i might be wrong, but it’s better than base mistral 7B)

I know these benchmarks are a tough topic, but this on paper looks really impressive. It states to be better than mistral and I loved the progress mistral brought. If someone tries this model out can you give feedback under this post? Much appreciated