I know that vLLM and TensorRT can be used to speed up LLM inference. I tried to find other tools can be do such things similar and will compare them. Do you guys have any suggestions?
vLLM: speed up inference
TensorRT: speed up inference
DeepSpeed:speed up for training phrase
Do you have any idea why MLC isn’t a more used format? It seems so much faster than GGUF or ExLlama architectures, yet everyone defaults to those
Thats an excellent question.