That’s a sharp comment.
Potentially beyond my technical ability but I can vaguely see where you’re going with it.
Next step was embeddings anyway (hence attempt to clean the data - get it ready for that).
I’ve not heard of pagerank applied to this before though. Thanks!
What is the intended use case? At 10s/token I’d imagine not chat
Swapping out layers on the fly is an interesting approach though
There is also the issue of pcie slots. Currently running a second card in a x4 slot and it’s noticeably slower. Getting four full speed x16 slots is going to be some pretty specialised equipment. All the crypto rigs are slow slots to my knowledge since it doesn’t matter there
It is good to see more competitive cards in this space though. Dual 770 could be very accessible
Liking this one - seems particularly good at long form story telling.
NB you may need to update your software…seems to rely on something pretty recent at least for text gen / llama.cpp. Crashed till I updated (and existing copy was max 48hr old)
Also, something odd on the template. Suggested template from the gguf seems to be alpaca while bloke model card says chatml. Under both it seems to spit out <|im_end|> occasionally but chatml seems better overall
multi-GPU
That’s the question I guess. If you can get say 5x of these for the price of a 4090 then that may look interesting. Though that’s a hell of a lot of overhead & hassle on power and pcie slots etc.
Athena V4
Think it’s aimed at ERP but remarkably pleasant at general upbeat female AI persona.
dolphin.2.1 with possibly a more serious tone.
Tried the yi dolphin one a bit…seems to provide much shorter & curt responses. Def doesn’t feel story telling like to me. Maybe the mistral version is better
More of an adjacent observation than answer but I was stunned by how many of the flagship models at decent size/quant get this wrong.
Grammar constained to Yes/No:
Is the earth flat? Answer with yes or no only. Do not provide any explanation or additional narrative.
Especially with non zero temp the answer seem near coin toss. idk maybe the training data is polluted by flat earthers lol
This doesn’t smell right to me.
All references around Q* and the drama around proto-AGI…e.g. Altman talking about veil of ignorance being pulled back seem to point to something that happened in the last couple of weeks. Not 2020.
Langchain has serpai plugins…but that’s more one shot type questions than a convo
Also the free limit on serpai is pretty low
tbh I think it’ll be a bad trade off. What you lose in steerability is huge, while I’m not convinced you’ll get any gains on less boring/overly positive. An instruction model you can at least tell it to write something dystopian.
more interesting outputs.
Try jacking up the temperature
Trial and error
It’s fine…you take a bit of a hit on the VRAM by virtue of well windows running though. Only really an issue if you’re on 8gb though
It’s vaguely how the Mac’s work.
The current APUs are still quite slow but maybe it’ll change. Also in most cases you need to designate memory as gpu specific. So not quite shared