当前位置： 首页 > news >正文

DeepSearch：WebThinker开启AI搜索研究新纪元！

news 来源：原创 2025/5/17 7:22:47

1，项目简介

WebThinker 是一个深度研究智能体，使 LRMs 能够在推理过程中自主搜索网络、导航网页，并撰写研究报告。这种技术的目标是革命性的：让用户通过简单的查询就能在互联网的海量信息中进行深度搜索、挖掘和整合，从而为知识密集型领域（如金融、科学、工程）的研究人员大幅降低信息收集的时间和成本。

项目地址：https://github.com/RUC-NLPIR/WebThinker

评价：很垃圾，需要接入微软宙斯的东西，Bing搜索。这些都需要替换成国产的，而且质量一般，不兼容图片等等等，远远不如字节的 DeerFlow 强大。

现有的开源深度搜索智能体通常采用检索增强生成（Retrieval-Augmented Generation, RAG）技术，依循预定义的工作流程，这限制了 LRM 探索更深层次网页信息的能力，也阻碍了 LRM 与搜索引擎之间的紧密交互。

  • 传统 RAG：仅进行浅层搜索，缺乏思考深度和连贯性
  • 进阶 RAG：使用预定义工作流，包括查询拆解、多轮 RAG 等，但仍缺乏灵活性
  • WebThinker：在连续深思考过程中自主调用工具，实现端到端任务执行

WebThinker 使 LRM 能够在单次生成中自主执行操作，无需遵循预设的工作流程，从而实现真正的端到端任务执行。

2，替换原LLM

【替换Bing的检索】run_web_thinker.py & run_web_thinker_report.py

async def generate_response(client: AsyncOpenAI,prompt: str,semaphore: asyncio.Semaphore,generate_mode: str = "chat",temperature: float = 0.0,top_p: float = 1.0,max_tokens: int = 32768,repetition_penalty: float = 1.0,top_k: int = 1,min_p: float = 0.0,model_name: str = "QwQ-32B",stop: List[str] = [END_SEARCH_QUERY],retry_limit: int = 3,bad_words: List[str] = [f"{END_SEARCH_RESULT}\n\n{tokenizer.eos_token}"],
) -> Tuple[str, str]:"""Generate a single response with retry logic"""for attempt in range(retry_limit):try:async with semaphore:if generate_mode == "chat":messages = [{"role": "user", "content": prompt}]if 'qwq' in model_name.lower() or 'deepseek' in model_name.lower() or 'r1' in model_name.lower():formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)else:formatted_prompt = aux_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)if ('deepseek' in model_name.lower() or 'r1' in model_name.lower()) and "<think>\n" not in formatted_prompt:formatted_prompt = formatted_prompt + "<think>\n"else:formatted_prompt = promptresponse = await client.completions.create(model=model_name,prompt=formatted_prompt,temperature=temperature,top_p=top_p,max_tokens=max_tokens,stop=stop,extra_body={'top_k': top_k,'include_stop_str_in_output': True,'repetition_penalty': repetition_penalty,# 'bad_words': bad_words,# 'min_p': min_p},timeout=3600,)return formatted_prompt, response.choices[0].textexcept Exception as e:print(f"Generate Response Error occurred: {e}, Starting retry attempt {attempt + 1}")# print(prompt)if "maximum context length" in str(e).lower():# If length exceeds limit, reduce max_tokens by halfmax_tokens = max_tokens // 2print(f"Reducing max_tokens to {max_tokens}")if attempt == retry_limit - 1:print(f"Failed after {retry_limit} attempts: {e}")return "", ""await asyncio.sleep(1 * (attempt + 1))return "", ""

替换为：

async def generate_response(client: AsyncOpenAI,prompt: str,semaphore: asyncio.Semaphore,generate_mode: str = "chat",temperature: float = 0.0,top_p: float = 1.0,max_tokens: int = 32768,repetition_penalty: float = 1.0,top_k: int = 1,min_p: float = 0.0,model_name: str = "QwQ-32B",stop: List[str] = [END_SEARCH_QUERY],retry_limit: int = 3,bad_words: List[str] = [f"{END_SEARCH_RESULT}\n\n{tokenizer.eos_token}"],
) -> Tuple[str, str]:"""Generate a single response with retry logic"""for attempt in range(retry_limit):try:async with semaphore:if generate_mode == "chat":messages = [{"role": "user", "content": prompt}]if 'qwq' in model_name.lower() or 'deepseek' in model_name.lower() or 'r1' in model_name.lower():formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)else:formatted_prompt = aux_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)if ('deepseek' in model_name.lower() or 'r1' in model_name.lower()) and "<think>\n" not in formatted_prompt:formatted_prompt = formatted_prompt + "<think>\n"else:formatted_prompt = promptclient = OpenAI(api_key="sk-*****",base_url="https://api.deepseek.com/beta")response = client.chat.completions.create(model="deepseek-chat",temperature=temperature,messages=[{"role": "system", "content": formatted_prompt},],stream=False)return formatted_prompt, response.choices[0].message.contentexcept Exception as e:print(f"Generate Response Error occurred: {e}, Starting retry attempt {attempt + 1}")# print(prompt)if "maximum context length" in str(e).lower():# If length exceeds limit, reduce max_tokens by halfmax_tokens = max_tokens // 2print(f"Reducing max_tokens to {max_tokens}")if attempt == retry_limit - 1:print(f"Failed after {retry_limit} attempts: {e}")return "", ""await asyncio.sleep(1 * (attempt + 1))return "", ""
def extract_relevant_info(search_results):"""Extract relevant information from Bing search results.Args:search_results (dict): JSON response from the Bing Web Search API.Returns:list: A list of dictionaries containing the extracted information."""useful_info = []if 'webPages' in search_results and 'value' in search_results['webPages']:for id, result in enumerate(search_results['webPages']['value']):info = {'id': id + 1,  # Increment id for easier subsequent operations'title': result.get('name', ''),'url': result.get('url', ''),'site_name': result.get('siteName', ''),'date': result.get('datePublished', '').split('T')[0],'snippet': result.get('snippet', ''),  # Remove HTML tags# Add context content to the information'context': ''  # Reserved field to be filled later}useful_info.append(info)return useful_info

替换为：

def extract_relevant_info(search_results):"""Extract relevant information from search results.Compatible with custom search result JSON (not Bing).Args:search_results (dict): JSON response containing search results.Returns:list: A list of dictionaries containing the extracted information."""useful_info = []if 'results' in search_results and isinstance(search_results['results'], list):for id, result in enumerate(search_results['results']):info = {'id': id + 1,'title': result.get('title', ''),'url': result.get('url', ''),'site_name': '',  # 该结构中无 site_name 字段'date': '',       # 该结构中无 datePublished 字段'snippet': result.get('content', ''),  # 使用 content 字段'context': ''}useful_info.append(info)return useful_info

3，替换Bing搜索

目前Bing关闭的API，原方式无法继续使用。修改 bing_search.py：

async def bing_web_search_async(query, subscription_key, endpoint, market='en-US', language='en', timeout=20):"""Perform an asynchronous search using the Bing Web Search API.Args:query (str): Search query.subscription_key (str): Subscription key for the Bing Search API.endpoint (str): Endpoint for the Bing Search API.market (str): Market, e.g., "en-US" or "zh-CN".language (str): Language of the results, e.g., "en".timeout (int): Request timeout in seconds.Returns:dict: JSON response of the search results. Returns empty dict if all retries fail."""headers = {"Ocp-Apim-Subscription-Key": subscription_key}params = {"q": query,"mkt": market,"setLang": language,"textDecorations": True,"textFormat": "HTML"}max_retries = 5retry_count = 0while retry_count < max_retries:try:response = session.get(endpoint, headers=headers, params=params, timeout=timeout)response.raise_for_status()search_results = response.json()return search_resultsexcept Exception as e:retry_count += 1if retry_count == max_retries:print(f"Bing Web Search Request Error occurred: {e} after {max_retries} retries")return {}print(f"Bing Web Search Request Error occurred, retrying ({retry_count}/{max_retries})...")time.sleep(1)  # Wait 1 second between retries

替换为：

async def bing_web_search_async(query, subscription_key, endpoint, market='en-US', language='en', timeout=20):client = TavilyClient("tvly-dev-*****")response = client.search(query=query)response.raise_for_status()search_results = response.json()print("=========",search_results)return search_results

4，命令行启动

 【问题解决模式】

python scripts/run_web_thinker.py \--single_question "What is OpenAI Deep Research?" \--bing_subscription_key "YOUR_BING_SUBSCRIPTION_KEY" \  # 用于调用 Bing 搜索API实现搜索增强（如 RAG）--api_base_url "YOUR_API_BASE_URL" \                    # 主模型的 API 接口基础地址，如 http://localhost:8000/v1--model_name "QwQ-32B" \                                # 主模型名称，可为本地模型或远程托管模型--tokenizer_path "PATH_TO_YOUR_TOKENIZER" \             # 主模型使用的分词器路径，如 ./tokenizers/qwq32b/--aux_api_base_url "YOUR_AUX_API_BASE_URL" \            # 辅助模型 API 接口地址--aux_model_name "Qwen2.5-32B-Instruct" \               # 辅助模型名称，用于多模型结果对比、增强或检验--aux_tokenizer_path "PATH_TO_YOUR_AUX_TOKENIZER"       # 辅助模型的分词器路径

【报告生成模式】

python scripts/run_web_thinker_report.py \--single_question "What are the models of OpenAI and what are the differences?" \--bing_subscription_key "YOUR_BING_SUBSCRIPTION_KEY" \--api_base_url "YOUR_API_BASE_URL" \--model_name "QwQ-32B" \--aux_api_base_url "YOUR_AUX_API_BASE_URL" \--aux_model_name "Qwen2.5-32B-Instruct" \--tokenizer_path "PATH_TO_YOUR_TOKENIZER" \--aux_tokenizer_path "PATH_TO_YOUR_AUX_TOKENIZER"

5，web 启动

cd demo
streamlit run run_demo.py

【报错】There is no current event loop in thread 'ScriptRunner.scriptThread'.

【解决】修改 bin_search.py 457 行：关闭加锁。

self.lock = asyncio.Lock()
👇
self.lock = None

【报错】Generate Response Error occurred: Connection error

【解决】修改 settings.py _load_client()

   def _load_client(self, api_base_url, aux_api_base_url):self.client = AsyncOpenAI(api_key="empty",base_url=api_base_url,)self.aux_client = AsyncOpenAI(api_key="empty",base_url=aux_api_base_url,)   👇 def _load_client(self, api_base_url, aux_api_base_url):client = OpenAI(api_key="sk-***",base_url="https://api.deepseek.com/beta")self.client = clientself.aux_client = client
# bing_search.py
use_model_name='QwQ-32B',
aux_model_name='Qwen2.5-72B-Instruct',
👇
use_model_name='deepseek-chat',
aux_model_name='deepseek-chat',

【报错】Invalid max_tokens value, the valid range of max_tokens is [1, 8192]

【解决】修改 bing_search.py

response = await client.completions.create(model=env.aux_model_name,max_tokens=4096,prompt=prompt,timeout=3600,)
👇
response = client.chat.completions.create(model=env.aux_model_name,messages=[{"role": "system", "content": prompt},],)

【报错】Generate Response Error occurred: Error code: 400 - {'error': {'message': 'Invalid max_tokens value, the valid range of max_tokens is [1, 8192]', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_request_error'}}, Starting retry attempt 1

【解决】修改 run_login.py generate_response()

async def generate_response(client: AsyncOpenAI,prompt: str,temperature: float = 0.0,top_p: float = 1.0,max_tokens: int = 4096,repetition_penalty: float = 1.0,top_k: int = 1,min_p: float = 0.0,model_name: str = "QwQ-32B",stop: List[str] = ["<|end_search_query|>"],retry_limit: int = 3,
):"""Generate a streaming response with retry logic"""for attempt in range(retry_limit):try:client = OpenAI(api_key="sk-****",base_url="https://api.deepseek.com/beta")response = client.chat.completions.create(model="deepseek-chat",temperature=temperature,messages=[{"role": "system", "content": prompt},],stream=True)async for chunk in response:if chunk.choices[0].message.content:yield chunk.choices[0].message.contentreturn  except Exception as e:print(f"Generate Response Error occurred: {e}, Starting retry attempt {attempt + 1}")if attempt == retry_limit - 1:print(f"Failed after {retry_limit} attempts: {e}")await asyncio.sleep(0.5 * (attempt + 1))yield ""

【报错】Generate Response Error occurred: 'async for' requires an object with __aiter__ method, got Stream, Starting

【解决】DeepSeek的输出不是标准的迭代格式，修改 run_logit.py generate_response()

async for chunk in response:if chunk.choices[0].message.content:yield chunk.choices[0].message.contentreturn  
👇
yield response.choices[0].message.content

相关文章：

  • 【LLM】大模型落地应用的技术 ——— 推理训练 MOE，AI搜索 RAG，AI Agent MCP
  • win10-django项目与mysql的基本增删改查
  • 泰迪杯特等奖案例深度解析：基于MSER-CNN的商品图片字符检测与识别系统设计
  • Yocto Project 快速构建
  • ISBI 2012 EM 神经元结构分割数据集复现UNet
  • RAG-MCP：突破大模型工具调用瓶颈，告别Prompt膨胀
  • 【实证分析】上市公司持续创新数据集+dofile（2008-2023年）
  • Apollo Client 1.6.0 + @RefreshScope + @Value 刷新问题解析
  • 【AT32】 AT32 移植 Freemodbus 主站
  • ssti刷刷刷
  • threejs小案例——贴图翻转
  • 华为云Flexus+DeepSeek征文｜SpringBoot开发实战：基于ModelArts Studio高效集成DeepSeek大模型服务
  • 归并排序：分治思想的优雅实现
  • 后端框架(1)：Mybatis
  • 在 C# 中将 DataGridView 数据导出为 CSV
  • ROS资料推荐学习
  • xss-labs靶场第11-14关基础详解
  • Microsoft Azure 服务4月更新告示
  • handsome主题美化及优化：10.1.0最新版 - 1
  • 网站运维基础 | 2. cms介绍及wordpress的搭建
  • 用贝多芬八首钢琴三重奏纪念风雨并肩20年
  • 3月中国减持189亿美元美债、持仓规模降至第三，英国升至第二
  • 特朗普政府涉税改法案遭众议院预算委员会否决
  • 外交部部长助理兼礼宾司司长洪磊接受美国新任驻华大使递交国书副本
  • 俄乌谈判开始
  • 侵害孩子者，必严惩不贷！3名性侵害未成年人罪犯被执行死刑