[LLM][RAG][AutoRAG]AutoRAG Part2: QA set(RAGAS)

AutoRAG Part2: QA set-RAGAS

using RAGAS framework

---

1. default

# 1. default method

import pandas as pd

from autorag.data.qacreation.ragas import generate_qa_ragas

corpus_df = pd.read_parquet(corpus_path)
qa_df_ragas = generate_qa_ragas(corpus_df, test_size=50)

---

2. distribution

# 2. distributions method

from ragas.testset.evolutions import simple, reasoning, multi_context, conditional
from autorag.data.qacreation.ragas import generate_qa_ragas

corpus_df = pd.read_parquet(corpus_path)
distributions = {  # uniform distribution
    simple: 0.25,
    reasoning: 0.25,
    multi_context: 0.25,
    conditional: 0.25
}
qa_df_ragas_dis = generate_qa_ragas(corpus_df, test_size=50, distributions=distributions)

---

3. critic (self-correct)

# 3. critic method

from autorag.data.qacreation.ragas import generate_qa_ragas
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

corpus_df = pd.read_parquet(corpus_path)

generator_llm = ChatOpenAI(model="gpt-4o", temperature=0.5)
critic_llm = ChatOpenAI(model="gpt-4", temperature=0.0)
embedding_model = OpenAIEmbeddings()

qa_df_critic = generate_qa_ragas(corpus_df, test_size=50, distributions=distributions,
                          generator_llm=generator_llm, critic_llm=critic_llm, embedding_model=embedding_model)

 

Best: 3. critic (self-correct) method

---

Evaluation

# evaluation
# qa_ragas_fin.parquet == outcome of 3. critic method

from autorag.evaluator import Evaluator

evaluator = Evaluator(qa_data_path="../data/qa_ragas_fin.parquet", corpus_data_path=vector_pq, project_dir='../benchmark')
evaluator.start_trial('../config.yaml')

---

Dashboard

- interpretation Results of AutoRAG

!autorag dashboard --trial_dir ./benchmark/4

---

Prototype (Streamlit)

- Deployment best pipeline

!autorag run_web --trial_path ./benchmark/4