基于ragflow中deepdoc对pdf文档的rag系统
基于ragflow中deepdoc对pdf文档的rag系统
一、安装
conda环境安装到指定的路径
conda create 包名/环境的名字
rag就是包的名字，ragflow就是环境名；
- 怎样激活环境？–我是在百度飞桨上面跑的
conda activate /home/aistudio/rag/ragflow
安装python库
datrie==0.8.2
akshare>=1.15.78,<2.0.0
azure-storage-blob==12.22.0
azure-identity==1.17.1
azure-storage-file-datalake==12.16.0
anthropic==0.34.1
arxiv==2.1.3
aspose-slides>=24.9.0,<25.0.0; platform_machine == 'x86_64' or (sys_platform == 'darwin' and platform_machine == 'arm64')
beartype>=0.18.5,<0.19.0
bio==1.7.1
blinker==1.7.0
boto3==1.34.140
botocore==1.34.140
cachetools==5.3.3
chardet==5.2.0
cn2an==0.5.22
cohere==5.6.2
Crawl4AI==0.3.8
dashscope==1.20.11
deepl==1.18.0
demjson3==3.0.6
discord-py==2.3.2
duckduckgo-search>=7.2.0,<8.0.0
editdistance==0.8.1
elastic-transport==8.12.0
elasticsearch==8.12.1
elasticsearch-dsl==8.12.0
fasttext==0.9.3
filelock==3.15.4
flask==3.0.3
flask-cors==5.0.0
flask-login==0.6.3
flask-session==0.8.0
google-search-results==2.4.2
groq==0.9.0
hanziconv==0.3.2
html-text==0.6.2
httpx==0.27.0
huggingface-hub>=0.25.0,<0.26.0
infinity-sdk==0.6.0-dev3
infinity-emb>=0.0.66,<0.0.67
itsdangerous==2.1.2
json-repair==0.35.0
markdown==3.6
markdown-to-json==2.1.1
minio==7.2.4
mistralai==0.4.2
nltk==3.9.1
numpy>=1.26.0,<2.0.0
ollama==0.2.1
onnxruntime==1.19.2; sys_platform == 'darwin' or platform_machine != 'x86_64'
onnxruntime-gpu==1.19.2; sys_platform != 'darwin' and platform_machine == 'x86_64'
openai==1.45.0
opencv-python==4.10.0.84
opencv-python-headless==4.10.0.84
openpyxl>=3.1.0,<4.0.0
ormsgpack==1.5.0
pandas>=2.2.0,<3.0.0
pdfplumber==0.10.4
peewee==3.17.1
pillow==10.4.0
protobuf==5.27.2
psycopg2-binary==2.9.9
pyclipper==1.3.0.post5
pycryptodomex==3.20.0
pypdf>=5.0.0,<6.0.0
pytest>=8.3.0,<9.0.0
python-dotenv==1.0.1
python-dateutil==2.8.2
python-pptx>=1.0.2,<2.0.0
pywencai==0.12.2
qianfan==0.4.6
ranx==0.3.20
readability-lxml==0.8.1
valkey==6.0.2
requests==2.32.2
replicate==0.31.0
roman-numbers==1.0.2
ruamel-base==1.0.0
scholarly==1.7.11
scikit-learn==1.5.0
selenium==4.22.0
selenium-wire==5.1.0
setuptools>=75.2.0,<76.0.0
shapely==2.0.5
six==1.16.0
strenum==0.4.15
tabulate==0.9.0
tencentcloud-sdk-python==3.0.1215
tika==2.6.0
tiktoken==0.7.0
umap_learn==0.5.6
vertexai==1.64.0
volcengine==1.0.146
voyageai==0.2.3
webdriver-manager==4.0.1
werkzeug==3.0.6
wikipedia==1.4.0
word2number==1.1
xgboost==1.5.0
xpinyin==0.7.6
yfinance==0.1.96
zhipuai==2.0.1
ruamel-yaml>=0.18.6,<0.19.0
google-generativeai>=0.8.1,<0.9.0
python-docx>=1.1.2,<2.0.0
pypdf2>=3.0.1,<4.0.0
graspologic>=3.4.1,<4.0.0
pymysql>=1.1.1,<2.0.0
mini-racer>=0.12.4,<0.13.0
pyodbc>=5.2.0,<6.0.0
flasgger>=0.9.7.1,<0.10.0
xxhash>=3.5.0,<4.0.0
bcembedding==0.1.5
fastembed>=0.3.6,<0.4.0; sys_platform == 'darwin' or platform_machine != 'x86_64'
fastembed-gpu>=0.3.6,<0.4.0; sys_platform != 'darwin' and platform_machine == 'x86_64'
flagembedding==1.2.10
torch>=2.5.0,<3.0.0
transformers>=4.35.0,<5.0.0
langchain
langchain-community
faiss-gpu
- 怎样从
pyproject.toml中使用conda进行安装；- 直接调用大模型上传
pyproject.toml加上提示词就能够生成requirements.txt
- 直接调用大模型上传
注意：版本有问题，使用最新版本的进行安装的pyproject.toml进行改装；
二、解析pdf文件
根据ragflow的demo中可以查看解析的格式
https://demo.ragflow.io
到达路径下面-cd ragflow/rag/app/接可以看到可以解析的各种切片的方法：
怎样进行切分呢？（基本上只有这些）
+ 法律文档：使用laws.py来运行
+ ppt：presentation.py
+ 论文：paper.py
+ 简历：resume.py
+ 问答：qa.py
都可以在https://demo.ragflow.io中查看简介
- 以paper.py为例
- 在import前面添加：
import sys
sys.path.append("../..")
2. 在主函数后面直接替换
if __name__ == "__main__":
import sys
def dummy(prog=None, msg=""):
print(prog,msg)
#chunk(sys.argv[1], callback=dummy)
import os
path = "/home/aistudio/majority"
files=os.listdir(path)
file_paths=[path+"/"+file for file in files]
print("file_paths=",file_paths)
for i in file_paths:
print(i)
all_chunks = []
# 遍历处理每个文件
for i,path in enumerate(file_paths):
print(f"第{i}path=",path)
chunks = chunk(path, callback=dummy)
with open("/home/aistudio/paper.txt","a") as f:
for chunk1 in chunks:
f.write(chunk1["content_with_weight"]+"\n\n")
all_chunks.extend(chunks) # 合并所有chunks
print(f"path={path}成功")
print("Total chunks=", len(all_chunks))
# 保存合并后的txt文件
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
embeddings = HuggingFaceEmbeddings(
model_name="/home/aistudio/bge_large",
model_kwargs = {'device': 'cuda'}
)
chunks_content = [chunk["content_with_weight"] for chunk in all_chunks]
FAISS.from_texts(chunks_content, embeddings).save_local(r"/home/aistudio/faiss_ppt")
print("多文件保存完毕")
- 运行
在ragflow/rag/app的路径下运行：
python paper.py
- 注意
path = "/home/aistudio/majority"是pdf的文件夹的路径；
三、rag检索向量数据库
def retriever(query):
#读取faiss_ppt的数据库
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
embeddings = HuggingFaceEmbeddings(model_name=r"D:\program\phone_check\bge_large")
db = FAISS.load_local("faiss_ppt", embeddings,allow_dangerous_deserialization = True)
# 从数据库中获取与问题最相关的10个的文档
docs = db.similarity_search(query, k=10)
#读取docs的内容
docs_content=[]
for doc in docs:
docs_content.append(doc.page_content)
return docs_content
- query:是字符串
- model_name:是本地的模型路径（保存faiss的embedding与加载faiss的我embedding相同）
四、错误
4.1 python库的版本不匹配
- 对requirements.txt根据最新版的ragflow进行更新
4.2 wordnet.zip和punk_tab.zip缺失
- 下载地址
https://aistudio.baidu.com/datasetdetail/319730
五、deepdoc的优势
可以提取表格；可以快速的应用pdf的切片。