Correct, even for training the models, all the Python code you see is really just a friendly interface over highly optimized C/cuda code.
There are no “loops” or matrix multiplication being done in Python. All the heavy lifting is done in lower level highly optimized code.
Python is just the glue.
Phi-2
A great LLM is a great compression algorithm.
Yi is not trustable on standard benchmarks because they are easy to game by including them in training data and the LKF gang who built this has a high pressure to justify their 1 billion dollar valuation and continue to milk investors.
The only way to really evaluate this is on some hidden benchmark never seen before and / or rigorous qualitative experiments.
Until then, I’m not holding my breath.
Just do what we all do: .NET wrapping a Docker container service running your LLM with some Python framework exposing a micro service with FastAPI.
V7 is propriety.
This is the current business model / strategy:
Yes. Even the authors of the AI frameworks like PyTorch aren’t usually writing the low level cuda code for NNs. They are wrapping the cuDNN library from NVIDIA which has highly optimized cuda code for NN operations.