国家官方网站百度指数功能

本文主要内容：基于Langchian实现RAG

Langchian中 RAG 中的 Retrieve（检索） 流程如下：

各种文档->各种loader->文本切片->嵌入向量化->向量存储->各种检索链

1. Loader 加载器

loader 就是从指定源进行加载数据，比如：

• Text文件：TextLoader
• 文件夹：DirectoryLoader
• CSV文件：CSVLoader
• Google网盘：GoogleDriveLoader
• 任意的网页：UnstructuredHTMLLoader
• PDF：PyPDFLoader
• Youtube：YoutubeLoader

上面只是简单的进行列举了几个，官方提供了超级的多的加载器供你使用。
这里，我们给出最为常见的几种数据loader的使用方式：

(1) 加载Text文件：TextLoader

from langchain_community.document_loaders import TextLoaderloader = TextLoader("./index.md")
data = loader.load()

(2) 加载CSV文件：CSVLoader

from langchain_community.document_loaders import CSVLoaderloader = CSVLoader("./data/test.csv")
data = loader.load()

(3) 加载JSON文件

# pip install jq
from langchain_community.document_loaders import JSONLoaderloader = JSONLoader(file_path="./data/student.json", jq_schema=".template", text_content=False
)
data = loader.load()

(4) 加载PDF文件：PyPDFLoader

# pip install pypdf
from langchain_community.document_loaders import PyPDFLoaderloader = PyPDFLoader("./data/paper.pdf")
data = loader.load()

2. Text Splitter 文本分割

文本分割就是用来分割文本的。为什么需要分割文本？因为我们每次不管是做把文本当作 prompt 发给 openai api ，还是还是使用 openai api embedding 功能都是有字符限制的。

比如我们将一份300页的 pdf 发给 openai api，让他进行总结，他肯定会报超过最大 Token 错。所以这里就需要使用文本分割器去分割我们 loader 进来的 Document。

示例如下：

from langchain.text_splitter import RecursiveCharacterTextSplitter# 加载需要切分的文档with open("./data/file.txt") as f:all_text = f.read()# 使用递归字符切分器
text_splitter = RecursiveCharacterTextSplitter(chunk_size=200,  # 切分的文本块大小chunk_overlap=20,  # 切分的文本块重叠大小length_function=len,  # 长度函数add_start_index=True  # 是否添加开始索引
)split_text = text_splitter.create_documents([all_text])
print(split_text)

输出如下：

[Document(metadata={'start_index': 0}, page_content='LangChain is a framework for developing applications powered by large language models (LLMs).\n\nLangChain simplifies every stage of the LLM application lifecycle:'), Document(metadata={'start_index': 163}, page_content="Development: Build your applications using LangChain's open-source components and third-party integrations. Use LangGraph to build stateful agents with first-class streaming and human-in-the-loop"), Document(metadata={'start_index': 341}, page_content='human-in-the-loop support.'), Document(metadata={'start_index': 368}, page_content='Productionization: Use LangSmith to inspect, monitor and evaluate your applications, so that you can continuously optimize and deploy with confidence.'), Document(metadata={'start_index': 519}, page_content='Deployment: Turn your LangGraph applications into production-ready APIs and Assistants with LangGraph Platform.')]

3. 文档的总结、精炼、翻译

首先安装 doctran 依赖包：

pip install doctran

具体实现如下：

from dotenv import load_dotenv
from doctran import Doctran
import os# 加载文档
with open("./data/file.txt") as f:content = f.read()load_dotenv("openai.env")
OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")
OPENAI_API_BASE = os.environ.get("OPENAI_API_BASE")
OPENAI_MODEL = "gpt-3.5-turbo-16k"
OPENAI_TOKEN_LIMIT = 8000doctrans = Doctran(openai_api_key=OPENAI_API_KEY,openai_model=OPENAI_MODEL,openai_token_limit=OPENAI_TOKEN_LIMIT,
)
documents = doctran.create_documents(content=content)# (1) 总结文档
summary = documents.summarize(token_limit=100).execute()
print(summary.transformed_content)# (2) 翻译文档
translation = documents.translate(language="chinese"
).execute()
print(summary.transformed_content)# (3) 精炼文档，仅保留与主题相关的内容
refined = documents.refine(topic=['langchain']
).execute()
print(refined.transformed_content)

4. 文本向量化

Embedding嵌入可以让一组文本或者一段话以向量来表示，从而可以让我们在向量空间中进行语义搜索之类的操作，从而大幅提升学习能力。

# from langchain.embeddings import OpenAIEmbeddings
from langchain_ollama import OllamaEmbeddingse_model = OllamaEmbeddings(model="nomic-embed-text")text = "今天天气真好"
single_vector = e_model.embed_query(text)
print("single_vector:", single_vector)multi_vector = e_model.embed_documents(["你好","今天天气真好","你叫什么名字","我叫小红",]
)print("multi_vector:", multi_vector)

输出如下：

single_vector: [0.023803584, 0.019944623, -0.18307257, 0.016884161, 0.013910737, 0.041673694, 0.0119952755, -0.0038184985, -0.0638501, -0.031662196, -0.04351526, 0.025420515, ......]]
multi_vector: [[-0.005196261, 0.01294648, -0.16610104, 0.0030547038, 0.06508742, 0.010399415, -0.03311033, -0.01913366, -0.017523672, -0.054374795, ......]]

参考博客：基于Ollama和LangChain使用embeddings模型进行文档检索

5. 嵌入向量缓存

加载文档，切分文档，将切分文档向量化并存储在缓存中：

首先安装 FAISS 向量数据库：

pip install faiss-cpu

完整代码如下：

from langchain.embeddings import OpenAIEmbeddings, CacheBackedEmbeddings
from langchain.storage import LocalFileStore
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import FAISS# 定义embedding模型
u_embeddings = OpenAIEmbeddings()
fs = LocalFileStore("./cache/")
cached_embeddings = CacheBackedEmbeddings.from_bytes_store(u_embeddings,fs,namespace=u_embeddings.model
)
list(fs.yield_keys())# 加载文档，切分文档
raw_documents = TextLoader("letter.txt").load()
text_splitter = CharacterTextSplitter(chunk_size=600, chunk_overlap=0)
documents = text_splitter.split_documents(raw_documents)# 将缓存写入向量数据库中
db = FAISS.from_documents(documents, cached_embeddings)  # 向量化到FAISS向量数据库中# 查看缓存中的键
list(fs.yield_keys())

6. 向量数据库

下面简单介绍几种常见的向量数据库：

7. 实战：ChatDoc 文档检索小助手

ChatDoc 文档检索小助手的功能：

• 可以加载pdf或者xls格式文档
• 可以对文档进行适当切分
• 文档进行向量化
• 使用Chroma db实现本地向量存储
• 使用智能检索实现和文档的对话

from langchain_community.document_loaders import Docx2txtLoader, PyPDFLoader, UnstructuredExcelLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain_ollama import OllamaEmbeddingsclass ChatDocAuto(object):def __init__(self, file_path):self.file_path = file_pathself.doc = Noneself.split_text = []  # 分割后的文本def get_file_content(self):loaders = {"docx": Docx2txtLoader,"pdf": PyPDFLoader,"xlsx": UnstructuredExcelLoader}file_extension = self.file_path.split(".")[-1]loader_class = loaders.get(file_extension)if loader_class:try:loader = loader_class(self.file_path)text = loader.load()return textexcept Exception as e:print("Error loading {}".format(file_extension))else:print("Unsupported file extension {}".format(file_extension))# 文档切割函数def split_sentence(self):full_text = self.get_file_content()if full_text is not None:text_splitter = CharacterTextSplitter(chunk_size=150, chunk_overlap=20)texts = text_splitter.split_documents(full_text)self.split_text = texts# return texts# 向量化与存储索引def embedding_and_vector(self):embeddings = OllamaEmbeddings(model="nomic-embed-text")db = Chroma.from_documents(documents=self.split_text,embedding=embeddings)return dbdef ask_and_find_files(self, question):db = self.embedding_and_vector()retriever = db.as_retriever()results = retriever.invoke(question)return resultsif __name__ == "__main__":file_path = "./data/paper.pdf"chatDocAuto = ChatDocAuto(file_path)# 查看文档内容content = chatDocAuto.get_file_content()# print(content)# 对文档进行分割chatDocAuto.split_sentence()# print(chatDocAuto.split_text)# 文档嵌入向量化chatDocAuto.embedding_and_vector()# 索引并使用自然语言找出相关的文本块question = "What is ITRANSFORMER?"result = chatDocAuto.ask_and_find_files(question)print(result)

[Document(metadata={'author': '', 'creationdate': '2023-12-04T01:26:30+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-06-03T09:01:40+08:00', 'page': 8, 'page_label': '9', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': './data/paper.pdf', 'subject': '', 'title': '', 'total_pages': 24, 'trapped': '/False'}, page_content='Analysis of multivariate correlations By assigning the duty of multivariate correlation to the\nattention mechanism, the learned map enjoys enhanced interpretability. We present the case visual-\nization on series from Solar-Energy in Figure 7, which has distinct correlations in the lookback and\nfuture windows. It can be observed that in the shallow attention layer...

可以看到虽然能检索到相关的结果，但是准确性不佳。这时候如果想优化检索结果，可以引入大模型进行多重查询，关键代码如下：

from langchain.retrievers import MultiQueryRetrieverclass ChatDocAuto(object):......def ask_and_find_files_by_llm(self, question):db = self.embedding_and_vector()# 把问题交给llm进行多角度扩展llm = OllamaLLM(model="llama3.1:8b")retriever_from_llm = MultiQueryRetriever(retriever=db.as_retriever(),llm=llm)results = retriever_from_llm.get_relevant_documents(question)return results

参考资料

AI Agent智能体开发，一步步教你搭建agent开发环境（需求分析、技术选型、技术分解）