I am a newbie to running a llm locally, but I would like to make one and use it to convert long books to short notes. I am a med student with a slightly above average knowledge of this tech (compared to general public) will it be possible for me to achieve this with a non tech background? If it is then it would mean a lot if someone can point me in the right direction.
You must log in or register to comment.
I am not really an expert myself but I will give it my shot.
The problem:
If you load a downloaded LLM and just try to feed it a book and some instructions it will breakdown within seconds. LLMs have limits on their context window that can barely handle a roleplay after 10 minutes of chatting. If my roleplay character can’t even remember their best friend’s name after 10 minutes of chatting then it will not be able to process a whole book.
The Solution:
What you need to do is actually include the book in the training dataset. Instead of feed a LLM the book via a prompt you would need it to already know the book from it’s original training. The strong AI nerds here can probably do this but most people here are downloading models from the HuggingFace and testing/trying them out. Training a model with your own dataset could be fun but it will require you do some research/self teaching on how to do it and then you will still need the GPU processing power to build it.
Nural chat 7b is pretty good tho this seems a bit stupid imho , your better off using the model stated before and use somthing like privateGPT to ingest the book into a vector db. Then you can effectively “talk to your books”.
Others can help you with the LLM part of this, but I am just mainly curious whether your plan is worth it. You do know that converting an entire book into a summary is pretty much worthless if you don’t make the summary yourself?