当前位置：首页 > news >正文

芝麻酱工作创新点分享1——SpringBoot下使用mongo+Redis做向量搜索

news 来源：原创 2025/6/4 14:39:43

一、背景介绍

有时，项目的数据来源于文档的解析。其特点如下：

总量并不大，并且也没有很明确的表关系。
相同业务模块不同的文档可能结构也不相同，都会有一些自己特有的字段。
文档中的一些词，可能填的比较模糊，常常数据关联不上。

基于这样的场景，我们首先在持久层选型上，会考虑使用mongodb。其好处是不需要事先设计表结构，把分几个集合想好即可。不同字段相同业务的数据，也可以放到一个集合中。等完成数据导入后，再根据查询需求建索引即可。

一般这种项目也不大，对于模糊查询的需求，为此引入el-search或者专业的向量数据库milvus，总觉得杀鸡用牛刀。而一般搭建Java应用，都会用一个redis。这时，我们就考虑建一个nick_name的集合，存放常见术语的昵称，并在redis中搭建一个向量结构。当查询时，先在redis中做向量匹配。

这个方案也可以做些扩展，应用于其他小型的带有一定数据交互的需求。毕竟向量计算可能需要调用open-api，可能是个耗时的操作。我们可以先把数据存mongodb，而后异步处理向量相关数据。
如果匹配不到，再去mongodb查。其实一般都会匹配到，但我们可以把术语的导入做成异步的，毕竟计算向量是个耗时操作。

二、redis安装

2.1 注意点

redis8.0，自带了redis-search模块，但需要在配置文件中开启。

2.2 ubuntu下安装

sudo apt-get install lsb-release curl gpg
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
sudo chmod 644 /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt-get update
sudo apt-get install redis

2.3 配置文件

#开放外部链接访问
bind 0.0.0.0  
#连接密码
requirepass ???@1314  
#新版本不推荐启用守护进程
daemonize no
supervised systemd
#log文件存放位置				 
logfile log/redis.log
#指定dump.rdb路径
dir /WORK/MIDDLEWARE/redis/current
#pid位置
#pidfile /WORK/MIDDLEWARE/redis/current/run/redis_6379.pid
#持久化设置
dbfilename dump.rdb
appendonly yes
appendfilename "appendonly.aof"
save 900 1
save 300 10
save 60 10000
appendonly yes
appendfsync everysec
# 加载各模块
loadmodule /usr/lib/redis/modules/redisearch.so
loadmodule /usr/lib/redis/modules/redisbloom.so
loadmodule /usr/lib/redis/modules/redistimeseries.so
loadmodule /usr/lib/redis/modules/rejson.so

2.4 自启动配置

Description=Redis Server
After=network.target[Service]
Type=simple
ExecStart=/WORK/SOFTWARE/redis/redis-8.0.0/src/redis-server /WORK/MIDDLEWARE/redis/conf/redis.conf
ExecStartPost=/bin/sleep 1
ExecStop=/WORK/SOFTWARE/redis/redis-8.0.0/src/redis-cli -a ???@1314 shutdown
Restart=always
User=redis
Group=redis
UMask=0077
#PIDFile=/WORK/MIDDLEWARE/redis/run/redis_6379.pid
RuntimeDirectory=/WORK/MIDDLEWARE/redis
RuntimeDirectoryMode=2755
TimeoutStartSec=infinity
TimeoutStopSec=infinity
NoNewPrivileges=yes[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable redis

三、springboot实现代码

3.1 技术选型

查阅官方文档发现，Java中的Jedis支持redis-search的API

3.2 pom引用

<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-mongodb</artifactId><version>${springData.mongo.version}</version>
</dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-autoconfigure</artifactId></dependency><dependency><groupId>redis.clients</groupId><artifactId>jedis</artifactId><version>5.2.0</version></dependency><dependency><groupId>com.alibaba</groupId><artifactId>easyexcel</artifactId><version>${easy-excel.version}</version><exclusions><exclusion><groupId>commons-io</groupId><artifactId>commons-io</artifactId></exclusion></exclusions></dependency><dependency><groupId>commons-io</groupId><artifactId>commons-io</artifactId><version>2.19.0</version></dependency>

3.3 向量查询Util

3.3.1 接口设计

由于昵称可能用于很多不同的场景，所以我们对昵称分分模块，ModuleName是大模块，SubModuleName是小模块。我们查询nickName时，在自己的小模块中查询。在建索引时，每个小模块建一个索引。
这样既减轻了用向量查询时带来的幻觉问题，也提升了索引的效率（相当于分库分表）

public interface INickRedisUtil {void addIndex(String pModuleName,  String pSubModuleName);void addNickName(String pModuleName, String pSubModuleName, String pName, List<String> pNickNames);String queryNickName(String pModuleName, String pSubModuleName,  String pNickName);void deleteNickName(String pModuleName, String pSubModuleName, String pName, List<String> pNickNames);
}

3.3.2 代码实现

下面我将提供该接口的代码实现，讲几个关键的点。

1. 向量维度
  向量维度必须和所使用的模型完全一致，这里要十分谨慎。
  我测试用的是GPT的text-embedding-3-small，向量的长度是1536；公司的模型长度是1024
1. 建索引代码解释
  TextField.of 描述其中的普通字段
  .addPrefix(nickNamePrefix) 是对每个redis的普通key-value数据，监听其键的前缀，只要符合规则，则进入索引。
1. 查找代码解释
  KNN $K @embedding $BLOB AS distance
  这句是核心代码，意思是把向量距离rename成distance，类似于mysql中的AS
  returnFields 类似于mysql中的select
1. redis的连接配置
  我这里的写法，借用的SpringData的配置，其实起作用的引用是spring-boot-autoconfigure
  VectorRedisProperties：

@Configuration
@ConfigurationProperties(prefix = "zhifa.redis.vector")
@Data
public class VectorRedisProperties extends RedisProperties {Pool pool;public VectorRedisProperties(){super();super.setTimeout(Duration.ofSeconds(30));super.setConnectTimeout(Duration.ofSeconds(30));super.setDatabase(0);}
}

1. 向量的请求
  向量计算，不要用Java做，python中有大量AI相关的库，可谓事半功倍。Java中做远程调用即可

@RequiredArgsConstructor
@Component
public class OpenAiEmbedUtil {private final OpenAiProperties mOpenAiProperties;private final RestTemplate mRestTemplate;public List<List<Float>> getEmbedding(List<String> pText) {String url = mOpenAiProperties.getApiUrl();EmbeddingRequest embeddingRequest = new EmbeddingRequest();embeddingRequest.initWithText(pText);HttpHeaders headers = new HttpHeaders();ParameterizedTypeReference<RestResponse<List<List<Float>>>> responseType = new ParameterizedTypeReference<RestResponse<List<List<Float>>>>() {};HttpEntity<EmbeddingRequest> requestEntity = new HttpEntity<>(embeddingRequest, headers);ResponseEntity<RestResponse<List<List<Float>>>> response = mRestTemplate.exchange(url, HttpMethod.POST, requestEntity,  responseType);if(response.getStatusCode().is2xxSuccessful()){return response.getBody().getData();}return null;/*WebClient webClient = WebClient.builder().baseUrl(url).defaultHeader(HttpHeaders.AUTHORIZATION, "Bearer "+mOpenAiProperties.getSecKey()).defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE).build();EmbeddingResponse result = webClient.post().bodyValue(embeddingRequest).retrieve().onStatus(status -> status.is4xxClientError() || status.is5xxServerError(),resp -> resp.bodyToMono(String.class).flatMap(body -> Mono.error(new ServiceException("OpenAI 返回错误: ", body)))).bodyToMono(EmbeddingResponse.class).block(mOpenAiProperties.getTimeOut());return result.getVector();*/}
}

下面是真正的代码：

@Slf4j
@RequiredArgsConstructor
@Component
public class NickNameRedisUtil implements INickRedisUtil {final OpenAiEmbedUtil mOpenAiEmbedUtil;final VectorRedisProperties mVectorRedisProperties;UnifiedJedis mJedis;static final String NICK_NAME_INDEX_NAME = "idx:nick_name";static final String NICK_NAME_KEY_PREFIX = "nick";static final int EMBEDDING_DIM = 1536;static final float DISTANCE_THRESHOLD = 0.6f;@PostConstructpublic void init(){DefaultJedisClientConfig defaultJedisClientConfig = DefaultJedisClientConfig.builder().clientName("jedis-vec").database(mVectorRedisProperties.getDatabase()).password(mVectorRedisProperties.getPassword()).timeoutMillis((int)mVectorRedisProperties.getTimeout().toMillis()).connectionTimeoutMillis((int)mVectorRedisProperties.getTimeout().toMillis()).build();HostAndPort hostAndPort = new HostAndPort(mVectorRedisProperties.getHost(), mVectorRedisProperties.getPort());if(null != mVectorRedisProperties.getPool() && mVectorRedisProperties.getPool().getEnabled()){GenericObjectPoolConfig<Connection> poolConfig = new GenericObjectPoolConfig<Connection>();poolConfig.setMaxIdle(mVectorRedisProperties.getPool().getMaxIdle());poolConfig.setMinIdle(mVectorRedisProperties.getPool().getMinIdle());poolConfig.setMaxTotal(mVectorRedisProperties.getPool().getMaxActive());mJedis = new JedisPooled(poolConfig,hostAndPort,defaultJedisClientConfig);}else{mJedis = new UnifiedJedis(hostAndPort,defaultJedisClientConfig);}}@Overridepublic void addIndex(String pModuleName, String pSubModuleName) {Set<String> indexSet = mJedis.ftList();String indexName = NICK_NAME_INDEX_NAME +":"+ pModuleName + ":" + pSubModuleName;String nickNamePrefix = NICK_NAME_KEY_PREFIX +":"+ pModuleName + ":" + pSubModuleName + ":";if(indexSet.contains(indexName)){return;}SchemaField[] schema = {TextField.of("nick_name"),TagField.of("name"),VectorField.builder().fieldName("embedding").algorithm(VectorField.VectorAlgorithm.HNSW).attributes(Map.of("TYPE", "FLOAT32","DIM", EMBEDDING_DIM,"DISTANCE_METRIC", "L2")).build()};mJedis.ftCreate(indexName,FTCreateParams.createParams().addPrefix(nickNamePrefix).on(IndexDataType.HASH),schema);}protected byte[] vec2ByteString(List<Float> vec) {byte[] bytes = new byte[vec.size() * Float.BYTES];Float[] boxed = vec.toArray(new Float[0]);float[] array = ArrayUtils.toPrimitive(boxed, 0f);ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().put(array);return bytes;}@Overridepublic void addNickName(String pModuleName, String pSubModuleName, String pName, List<String> pNickNames) {String nickNamePrefix = NICK_NAME_KEY_PREFIX +":"+ pModuleName + ":" + pSubModuleName + ":";List<List<Float>> embeddings = mOpenAiEmbedUtil.getEmbedding(pNickNames);for (int i=0;i<pNickNames.size();i++) {String nickName = pNickNames.get(i);int fingerPrint = MurmurHash3.hash32(nickName);String key = nickNamePrefix + fingerPrint;if(mJedis.exists(key)){return;}mJedis.hset(key,Map.of("nick_name", nickName,"name", pName));mJedis.hset(key.getBytes(),"embedding".getBytes(),vec2ByteString(embeddings.get(i)));log.info("已添加缓存:"+key+"|  "+nickName+" -> " + pName);}}@Overridepublic String queryNickName(String pModuleName, String pSubModuleName, String pNickName) {String indexName = NICK_NAME_INDEX_NAME +":"+ pModuleName + ":" + pSubModuleName;List<List<Float>> embeddings = mOpenAiEmbedUtil.getEmbedding(Arrays.asList(pNickName));if(embeddings.size() == 0){throw new ServiceException("获取"+pNickName+"向量失败");}byte[] bVec = vec2ByteString(embeddings.get(0));int K = 3;Query q = new Query("*=>[KNN $K @embedding $BLOB AS distance]").returnFields("nick_name", "name", "distance").addParam("K", K).addParam("BLOB",bVec).setSortBy("distance", true).dialect(2);List<Document> docs = mJedis.ftSearch(indexName, q).getDocuments();for (Document doc: docs) {System.out.println(String.format("ID: %s, Distance: %s, nick_name: %s, name: %s",doc.getId(),doc.get("distance"),doc.get("nick_name"),doc.get("name")));}if(docs.size() == 0){return null;}String distance = docs.get(0).getString("distance");Float fDistance = Float.parseFloat(distance);if(fDistance > DISTANCE_THRESHOLD){return null;}return docs.get(0).getString("name");}@Overridepublic void deleteNickName(String pModuleName, String pSubModuleName, String pName,List<String> pNickNames) {String nickNamePrefix = NICK_NAME_KEY_PREFIX +":"+ pModuleName + ":" + pSubModuleName + ":";List<String> delKeys = new ArrayList<>();for (int i=0;i<pNickNames.size();i++) {String nickName = pNickNames.get(i);int fingerPrint = MurmurHash3.hash32(nickName);String key = nickNamePrefix + fingerPrint;delKeys.add(key);}mJedis.del(delKeys.toArray(String[]::new));}
}

四、python向量服务器

其实python这边，用不到复杂的技术，所以选用tornado服务器，尽可能的轻量。
下面直接上代码：

from typing import List, Any
from dataclasses import dataclass, asdict,is_dataclass
import jsonimport tornado.ioloop
import tornado.web
from openai import OpenAIclient = OpenAI(api_key="我的OpenAi的key")@dataclass
class StringVector:questions: List[str]model: strkeyword: strdef ok_response(data: Any) -> str:# 1) 如果是 dataclass，就用 asdictif is_dataclass(data):payload = asdict(data)# 2) 如果是列表，且列表里每项都是 dataclass，也一并展开elif isinstance(data, list) and all(is_dataclass(item) for item in data):payload = [asdict(item) for item in data]# 3) 如果有 to_dict 方法（如 OpenAIObject），就调用它elif hasattr(data, "to_dict") and callable(data.to_dict):payload = data.to_dict()# 4) 其它一律原样返回（假定它已经是 dict、list、基本类型等可序列化的）else:payload = data# 使用 json.dumps 将对象转为 JSON 字符串return json.dumps({"data": payload,"code": 200,"message": None},ensure_ascii=False)# 定义一个处理器类，继承自 tornado.web.RequestHandler
class test2vec_handler(tornado.web.RequestHandler):def post(self):body = self.request.bodydata = json.loads(body)# 将字典转换为 StringVector 数据类实例string_vector = StringVector(**data)response = client.embeddings.create(input= string_vector.questions,model="text-embedding-3-small")embeddings = [item.embedding for item in response.data]print("已计算字符串"+",".join(string_vector.questions))self.set_header("Content-Type", "application/json")self.write(ok_response(embeddings))# 定义应用程序类，指定路由
def make_app():return tornado.web.Application([(r"/text2vec", test2vec_handler),  # 将 "/" 路径映射到 MainHandler])# 启动服务器
if __name__ == "__main__":app = make_app()app.listen(8082)  # 监听 8888 端口tornado.ioloop.IOLoop.current().start()  # 启动事件循环