While I’m not an expert, I understand that, in theory, large language models can process an unlimited amount of context. However, there are practical limitations. If we start by training a base model, for example, one with 70 billion parameters, to excel in reasoning and insight extraction, we could then progress to using bigger models to fine-tune . These bigger models could teach our 70b how to handle context windows ranging from 2 to 10 million tokens, essentially allowing us to store up-to-date information in a document. RAG can come in handy here as well.
While I’m not an expert, I understand that, in theory, large language models can process an unlimited amount of context. However, there are practical limitations. If we start by training a base model, for example, one with 70 billion parameters, to excel in reasoning and insight extraction, we could then progress to using bigger models to fine-tune . These bigger models could teach our 70b how to handle context windows ranging from 2 to 10 million tokens, essentially allowing us to store up-to-date information in a document. RAG can come in handy here as well.