Hi,
I’ve issues when Q&A my PDF:

  • - I don’t have consistency
  • - Tried many models (embedding/llm)
  • - Tried many methods
  • - I have 20% hallucination, especialy with “president” and “mickael jackson”

ex1 (this one is correct):

> question: How much net income for Amazon in 2020, 2021 and 2022 ?

> answer: In 2020, Amazon made a net income of $21.331 billion, in 2021 $33.364 billion and in 2022 the company made a loss of $2.722 billion.

ex2: (this one is incorrect):

> question: How much operating expenses for AWS in 2020, 2021 and 2022 ?

> answer: The operating expenses of AWS in 2020, 2021, and 2022 were $444.943 billion, $501.735 billion, and $567.608 billion respectively.

It always calculate the entire expenses of the company, i try GPT-4 and it is capable.

- PDF: Amazon 2022 annual report 10K (88 pages)
- Embedding: all-MiniLM-L12-v2
- Text splitter: Chunk_size = 1000, overlap = 20
- VectorDB: Chroma
- LLM: SciPhi-Self-RAG-Mistral-7B-32k-8.0bpw-h6-exl2 via Oobabooga (OpenAI extension) with 0.2temp, alpaca instruction template.
- Langchain: RetrievalQA, chain_type = stuff, retriever = vectDB.as_retriever()
- RTX 3090

If anyone resolve this issue, please can you help me :)