当前位置：首页 > news >正文

讯飞AI相关sdk集成springboot

news 来源：原创 2025/5/25 5:40:00

星火认知大模型对话：（以spark 4.0 ultra 为例）

demo上的功能比较简陋，网络上搜到的比较残缺，很多功能缺失，我这里自己收集资料和运用编程知识做了整理，得到了自己想要的一些功能，比如持久化处理、对话历史记录持久化，自定义提示词传入等等。

希望能帮助到你们。

提前准备：

maven依赖：

        <dependency><groupId>org.springframework</groupId><artifactId>spring-context</artifactId></dependency><dependency><groupId>com.google.code.gson</groupId><artifactId>gson</artifactId><version>2.8.2</version></dependency><dependency><groupId>com.squareup.okhttp3</groupId><artifactId>okhttp</artifactId><version>4.10.0</version></dependency><dependency><groupId>com.squareup.okio</groupId><artifactId>okio</artifactId><version>2.10.0</version></dependency>

配置类：

@EnableConfigurationProperties
@Data
@Component
@ConfigurationProperties("xf.config")
public class XFConfig {private String appId;private String apiSecret;private String apiKey;private String hostUrl;private Integer maxResponseTime;}

配置信息放在yaml中：

xf:config:hostUrl: https://spark-api.xf-yun.com/v4.0/chatappId: # 填写你的控制台的apiSecret: #填写你的控制台的apiKey: #填写你的控制台的maxResponseTime: 40

listener目录：

webClient：用于连接星火服务，无需修改，直接复制粘贴到此目录下即可

@Slf4j
@Component
public class XFWebClient {@Autowiredprivate XFConfig xfConfig;//线程池private final ExecutorService makePool = Executors.newCachedThreadPool();public XFWebClient(XFConfig xfConfig) {this.xfConfig = xfConfig;}//通过webflux实现websocket长连接,异步流式响应,过滤掉除content的其他信息(可以控制响应速度)public Flux<String> send(List<RoleContent> messages) throws Exception {String authUrl = getAuthUrl(xfConfig.getHostUrl(), xfConfig.getApiKey(), xfConfig.getApiSecret());String url = authUrl.replace("http://", "ws://").replace("https://", "wss://");//获取连接器ReactorNettyWebSocketClient client = new ReactorNettyWebSocketClient();String json = JSON.toJSONString(createRequestParams(xfConfig.getAppId(), messages,null));System.out.println("Request JSON: " + json); //检验参数return Flux.create(fluxSink ->makePool.execute(()-> client.execute(URI.create(url), session -> session //开启异步线程.send(Flux.just(session.textMessage( //发送消息Objects.requireNonNull(JSON.toJSONString(createRequestParams(xfConfig.getAppId(), messages,null)))))).thenMany(session.receive().map(WebSocketMessage ->{XfResponse response = JSON.parseObject(WebSocketMessage.getPayloadAsText(), XfResponse.class);if(response.getHeader().getStatus() == 2){ //返回的是最后一个片段关闭fluxSink.complete();}//处理返回的数据String content = response.getPayload().getChoices().getText().get(0).getContent().replace("*","");return content;}).doOnNext(fluxSink::next)).then()).block(Duration.ofSeconds(120))));//阻塞线程120s}/*** @description: 发送请求至大模型方法* @author: ChengLiang* @date: 2023/10/19 16:27* @param: [用户id, 请求内容, 返回结果监听器listener]* @return: okhttp3.WebSocket**/public WebSocket sendMsg(String uid, List<RoleContent> questions, WebSocketListener listener) {// 获取鉴权urlString authUrl = null;try {authUrl = getAuthUrl(xfConfig.getHostUrl(), xfConfig.getApiKey(), xfConfig.getApiSecret());} catch (Exception e) {log.error("鉴权失败：{}", e);return null;}// 鉴权方法生成失败，直接返回 nullOkHttpClient okHttpClient = new OkHttpClient.Builder().build();// 将 https/http 连接替换为 ws/wss 连接String url = authUrl.replace("http://", "ws://").replace("https://", "wss://");Request request = new Request.Builder().url(url).build();// 建立 wss 连接WebSocket webSocket = okHttpClient.newWebSocket(request, listener);// 组装请求参数JSONObject requestDTO = createRequestParams(uid, questions);// 发送请求webSocket.send(JSONObject.toJSONString(requestDTO));return webSocket;}/*** @description: 鉴权方法* @author: ChengLiang* @date: 2023/10/19 16:25* @param: [讯飞大模型请求地址, apiKey, apiSecret]* @return: java.lang.String**/public static String getAuthUrl(String hostUrl, String apiKey, String apiSecret) throws Exception {URL url = new URL(hostUrl);// 时间SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss z", Locale.US);format.setTimeZone(TimeZone.getTimeZone("GMT"));String date = format.format(new Date());// 拼接String preStr = "host: " + url.getHost() + "\n" +"date: " + date + "\n" +"GET " + url.getPath() + " HTTP/1.1";// SHA256加密Mac mac = Mac.getInstance("hmacsha256");SecretKeySpec spec = new SecretKeySpec(apiSecret.getBytes(StandardCharsets.UTF_8), "hmacsha256");mac.init(spec);byte[] hexDigits = mac.doFinal(preStr.getBytes(StandardCharsets.UTF_8));// Base64加密String sha = Base64.getEncoder().encodeToString(hexDigits);// 拼接String authorization = String.format("api_key=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"", apiKey, "hmac-sha256", "host date request-line", sha);// 拼接地址HttpUrl httpUrl = Objects.requireNonNull(HttpUrl.parse("https://" + url.getHost() + url.getPath())).newBuilder().//addQueryParameter("authorization", Base64.getEncoder().encodeToString(authorization.getBytes(StandardCharsets.UTF_8))).//addQueryParameter("date", date).//addQueryParameter("host", url.getHost()).//build();return httpUrl.toString();}/*** @description: 请求参数组装方法* @author: ChengLiang* @date: 2023/10/19 16:26* @param: [用户id, 请求内容]* @return: com.alibaba.fastjson.JSONObject**/public JSONObject createRequestParams(String uid, List<RoleContent> questions,String functions) {JSONObject requestJson = new JSONObject();// header参数JSONObject header = new JSONObject();header.put("app_id", xfConfig.getAppId());header.put("uid", uid);// parameter参数JSONObject parameter = new JSONObject();JSONObject chat = new JSONObject();chat.put("domain", "4.0Ultra");chat.put("temperature", 0.5);chat.put("max_tokens", 4096);parameter.put("chat", chat);// payload参数JSONObject payload = new JSONObject();JSONObject message = new JSONObject();JSONArray jsonArray = new JSONArray();jsonArray.addAll(questions);message.put("text", jsonArray);payload.put("message", message);payload.put("functions",functions);requestJson.put("header", header);requestJson.put("parameter", parameter);requestJson.put("payload", payload);return requestJson;}public JSONObject createRequestParams(String uid, List<RoleContent> questions) {JSONObject requestJson = new JSONObject();// header参数JSONObject header = new JSONObject();header.put("app_id", xfConfig.getAppId());header.put("uid", uid);// parameter参数JSONObject parameter = new JSONObject();JSONObject chat = new JSONObject();chat.put("domain", "4.0Ultra");chat.put("temperature", 0.5);chat.put("max_tokens", 4096);parameter.put("chat", chat);// payload参数JSONObject payload = new JSONObject();JSONObject message = new JSONObject();JSONArray jsonArray = new JSONArray();jsonArray.addAll(questions);message.put("text", jsonArray);payload.put("message", message);requestJson.put("header", header);requestJson.put("parameter", parameter);requestJson.put("payload", payload);return requestJson;}
}

socketListener：用于监听和星火认知大模型的socket连接，也是直接复制粘贴到listener目录下：

可以在todo处进行持久化处理，我这里不在这里处理，下面有其他代码我会专门做处理。

@Slf4j
public class XFWebSocketListener extends WebSocketListener {//断开websocket标志位private boolean wsCloseFlag = false;//语句组装buffer，将大模型返回结果全部接收，在组装成一句话返回private StringBuilder answer = new StringBuilder();public String getAnswer() {return answer.toString();}public boolean isWsCloseFlag() {return wsCloseFlag;}@Overridepublic void onOpen(WebSocket webSocket, Response response) {super.onOpen(webSocket, response);log.info("大模型服务器连接成功！");}@Overridepublic void onMessage(WebSocket webSocket, String text) {super.onMessage(webSocket, text);JsonParse myJsonParse = JSON.parseObject(text, JsonParse.class);log.info("myJsonParse:{}", JSON.toJSONString(myJsonParse));if (myJsonParse.getHeader().getCode() != 0) {log.error("发生错误，错误信息为:{}", JSON.toJSONString(myJsonParse.getHeader()));this.answer.append("大模型响应异常，请联系管理员");// 关闭连接标识wsCloseFlag = true;return;}List<Text> textList = myJsonParse.getPayload().getChoices().getText();for (Text temp : textList) {log.info("返回结果信息为：【{}】", JSON.toJSONString(temp));this.answer.append(temp.getContent());}log.info("result:{}", this.answer.toString());if (myJsonParse.getHeader().getStatus() == 2) {wsCloseFlag = true;//todo 将问答信息入库进行记录，可自行实现}}@Overridepublic void onFailure(WebSocket webSocket, Throwable t, Response response) {super.onFailure(webSocket, t, response);try {if (null != response) {int code = response.code();log.error("onFailure body:{}", response.body().string());if (101 != code) {log.error("讯飞星火大模型连接异常");}}} catch (IOException e) {log.error("IO异常：{}", e);}}
}

实体类：

有点多，要慢慢复制粘贴。

@Data
public class Choices {private List<Text> text;}

@lombok.Data
public class Data {int status;String audio;
}

这个FileDetail实体类用于讯飞多模态分析的表情识别：

@Data
public class FileDetail {private int code;private String fileName;private int label;private List<Object> labels;private String name;private double rate; // 如果rate是字符串，请改为Stringprivate List<Double> rates;private boolean review;private List<Object> subLabels;private String tag;
}

@Data
public class Header {private int code;private int status;private String sid;}

@Data
public class JsonParse {private Header header;private Payload payload;}

public class NettyGroup {private static ChannelGroup channelGroup = new DefaultChannelGroup(GlobalEventExecutor.INSTANCE);/*** 存放用户与Chanel的对应信息，用于给指定用户发送消息*/private static ConcurrentHashMap<String, Channel> channelMap = new ConcurrentHashMap<>();private NettyGroup() {}/*** 获取channel组*/public static ChannelGroup getChannelGroup() {return channelGroup;}/*** 获取连接channel map*/public static ConcurrentHashMap<String, Channel> getUserChannelMap() {return channelMap;}
}

@Data
public class Payload {private Choices choices;}

@Getter
@Setter
@ToString(callSuper = true)
@AllArgsConstructor
@NoArgsConstructor
public class ResultBean<T> {private String errorCode;private String message;private T data;public ResultBean(T data) {this.errorCode = ErrorMessage.SUCCESS.getErrorCode();this.message = ErrorMessage.SUCCESS.getMessage();this.data = data;}public ResultBean(ErrorMessage errorMessage, T data) {this.errorCode = errorMessage.getErrorCode();this.message = errorMessage.getMessage();this.data = data;}public static <T> ResultBean success(T data) {ResultBean resultBean = new ResultBean(data);return resultBean;}public static <T> ResultBean fail(T data) {ResultBean resultBean = new ResultBean(ErrorMessage.FAIL.getErrorCode(), ErrorMessage.FAIL.getMessage(), data);return resultBean;}public enum ErrorMessage {SUCCESS("0", "success"),FAIL("001", "fail"),NOAUTH("1001", "非法访问");private String errorCode;private String message;ErrorMessage(String errorCode, String message) {this.errorCode = errorCode;this.message = message;}public String getErrorCode() {return errorCode;}public void setErrorCode(String errorCode) {this.errorCode = errorCode;}public String getMessage() {return message;}public void setMessage(String message) {this.message = message;}}
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class RoleContent {public static final String ROLE_USER = "user";public static final String ROLE_ASSISTANT = "assistant";private String role;private String content;public static RoleContent createUserRoleContent(String content) {return new RoleContent(ROLE_USER, content);}public static RoleContent createAssistantRoleContent(String content) {return new RoleContent(ROLE_ASSISTANT, content);}
}

这个类很重要，记住它：

@Data
@NoArgsConstructor
@TableName("t_role_content")
public class RoleContentPo implements Serializable {private static final long serialVersionUID = 1666L;@TableId(type = IdType.AUTO)private Long id;private String uid;private String role; //user,private String content;public RoleContentPo(String uid, String role, String content){this.uid = uid;this.role = role;this.content = content;}private Date sendTime;private Long scid;private Long rcid;private String type;private Long voiceLength;@TableLogic//是否删除private Integer isDeleted;
}

@Data
public class Text {private String role;private String content;}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class XFInputQuestion {private String uid;private String context;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class XFInputQuestionFunctionCall {private String uid;private String context;private Object functionsParams;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Choices {private int status;private int seq;private List<Message> text;}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class ResponseHeader {private Integer code;private String message;private String sid;private Integer status;
}

@Data
public class ResponsePayload implements Serializable {private Choices choices;private Usage usage;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class TextUsage implements Serializable {@JsonProperty("prompt_tokens")private Integer promptTokens;@JsonProperty("completion_tokens")private Integer completionTokens;@JsonProperty("total_tokens")private Integer totalTokens;@JsonProperty("question_tokens")private Integer questionTokens;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Usage implements Serializable {private TextUsage text;}

@AllArgsConstructor
@NoArgsConstructor
@Data
public class XfResponse implements Serializable {private ResponseHeader header;private ResponsePayload payload;
}

还可以再建一个history子目录，用于存放有关历史对话的实体类：

@Data
public class AiHistoryPage {//发送者的idprivate String scid;//当前页码private Integer pageNum;//每一页的数据量大小private Integer pageSize;//最大页数private Integer pageMaxCount;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class AiHistoryRecords {private Integer recordsNum;private ArrayList<HistoryInfo> history;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class HistoryDataVo {private String uid;private Integer userId;@JsonFormat(pattern = "yyyy-MM-dd", timezone = "GMT+8")private Date uTime;private String lastQuestion;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
public class HistoryInfo {@TableIdprivate Integer id;private String uid;private String scid;private String LastMsgTime;private String LastQuestionMsg;
}

mysql表：只有两个：

CREATE TABLE `t_role_content` (`id` int unsigned NOT NULL AUTO_INCREMENT,`uid` varchar(50) NOT NULL DEFAULT 'ergou' COMMENT '对话id',`role` varchar(50) NOT NULL DEFAULT 'user' COMMENT '对话角色',`content` text NOT NULL COMMENT '对话内容',`is_deleted` tinyint unsigned NOT NULL DEFAULT '0' COMMENT '删除标志，默认0不删除，1删除',`send_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',`scid` bigint NOT NULL DEFAULT '1773322941155852290' COMMENT '发送人',`rcid` bigint NOT NULL DEFAULT '12345678' COMMENT '接收人',`type` varchar(50) NOT NULL DEFAULT 'text' COMMENT '消息类型',`read` int DEFAULT '0' COMMENT '是否已读',`voice_length` int DEFAULT '0' COMMENT '语音长度',PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2056 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci ROW_FORMAT=DYNAMIC COMMENT='ai对话历史记录';

CREATE TABLE `history_info` (`id` bigint NOT NULL AUTO_INCREMENT COMMENT '主键',`scid` char(11) DEFAULT NULL COMMENT '电话',`uid` varchar(50) DEFAULT NULL COMMENT '对话uid',`last_msg_time` varchar(50) DEFAULT '某个时间' COMMENT '最后一条消息的时间',`last_question_msg` text COMMENT '最后一条消息的内容',`is_deleted` tinyint NOT NULL DEFAULT '0',PRIMARY KEY (`id`),UNIQUE KEY `uid` (`uid`)
) ENGINE=InnoDB AUTO_INCREMENT=2147119107 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci ROW_FORMAT=DYNAMIC COMMENT='对话历史记录';

接着建立mapper：

@Mapper
public interface HistoryInfoMapper extends BaseMapper<HistoryInfo> {
}

@Mapper
public interface RoleContentPoMapper extends BaseMapper<RoleContentPo> {
}

service以及其impl：

public interface AiHistoryRecordsService{ResultData<AiHistoryRecords> getAiHistory(AiHistoryPage aiHistoryPage);
}

public interface HistoryInfoService extends IService<HistoryInfo> {
}

public interface PushService {void pushToOne(String uid, String text);void pushToAll(String text);//测试账号只有2个并发，此处只使用一个，若是生产环境允许多个并发，可以采用分布式锁ResultData pushMessageToXFServer(RoleContentPo roleContentPo, String background);ResultData<RoleContentPo> pushVoiceMessageToXFServer(RoleContentPo roleContentPo, String background,String voiceMessage);
}

public interface RoleContentPoService extends IService<RoleContentPo> {ResultData getAllMessage(AiMessagePage page);ResultData getAllMsgPageMax(AiMessagePage page);ResultData<RoleContentPo> voiceMsgToAi(RoleContentPo roleContentPo, String message);}

impl：

@Service
public class AiHistoryRecordsServiceImpl implements AiHistoryRecordsService {@Resourceprivate HistoryInfoMapper historyInfoMapper;@Overridepublic ResultData<AiHistoryRecords> getAiHistory(AiHistoryPage aiHistoryPage) {//分页构造器IPage<HistoryInfo> page = new Page<>(aiHistoryPage.getPageNum(),aiHistoryPage.getPageSize());LambdaQueryWrapper<HistoryInfo> lambdaQueryWrapper = new LambdaQueryWrapper<>();lambdaQueryWrapper.eq(HistoryInfo::getScid,aiHistoryPage.getScid());//将分页信息查询出来IPage<HistoryInfo> selectPage = historyInfoMapper.selectPage(page, lambdaQueryWrapper);List<HistoryInfo> records = selectPage.getRecords();ListUtil.reverse(records);ArrayList<HistoryInfo> historyInfos = ListUtil.toList(records);AiHistoryRecords aiHistoryRecords = new AiHistoryRecords();aiHistoryRecords.setHistory(historyInfos);aiHistoryRecords.setRecordsNum(records.size());return ResultData.success(aiHistoryRecords);}
}

@Service
public class HistoryInfoServiceImpl extends ServiceImpl<HistoryInfoMapper, HistoryInfo> implements HistoryInfoService {
}

之后主要调用的就是这个PushService的pushMessageToXFServer方法，只需要传入用户输入信息和提示词，就可以获得ai输出信息。

如果想要前端实时接收流式数据，要在todo处，进行和前端的webSocket连接。

@Slf4j
@Service
public class PushServiceImpl implements PushService {@Autowiredprivate XFConfig xfConfig;@Autowiredprivate XFWebClient xfWebClient;@Resourceprivate RoleContentPoMapper roleContentPoMapper;@Resourceprivate WebSocketServer webSocketServer;@Overridepublic void pushToOne(String uid, String text) {if (StringUtils.isEmpty(uid) || StringUtils.isEmpty(text)) {log.error("uid或text均不能为空");throw new RuntimeException("uid或text均不能为空");}ConcurrentHashMap<String, Channel> userChannelMap = NettyGroup.getUserChannelMap();for (String channelId : userChannelMap.keySet()) {if (channelId.equals(uid)) {Channel channel = userChannelMap.get(channelId);if (channel != null) {ResultBean success = ResultBean.success(text);channel.writeAndFlush(new TextWebSocketFrame(JSON.toJSONString(success)));log.info("信息发送成功：{}", JSON.toJSONString(success));} else {log.error("该id对于channelId不存在！");}return;}}log.error("该用户不存在！");}@Overridepublic void pushToAll(String text) {String trim = text.trim();ResultBean success = ResultBean.success(trim);NettyGroup.getChannelGroup().writeAndFlush(new TextWebSocketFrame(JSON.toJSONString(success)));log.info("信息推送成功：{}", JSON.toJSONString(success));}//测试账号只有2个并发，此处只使用一个，若是生产环境允许多个并发，可以采用分布式锁//    @Async@Overridepublic synchronized ResultData<RoleContentPo> pushMessageToXFServer(RoleContentPo roleContentPo, String background) {//******************  part1  ************************************************//从数据库中查询历史记录LambdaQueryWrapper<RoleContentPo> lambdaQueryWrapper = new LambdaQueryWrapper<>();lambdaQueryWrapper.eq(RoleContentPo::getUid,roleContentPo.getUid());    //查询对应uid的对话历史记录lambdaQueryWrapper.orderByAsc(RoleContentPo::getSendTime);   //要根据创建时间升序排序List<RoleContentPo> historyPoList = roleContentPoMapper.selectList(lambdaQueryWrapper);//判断是否历史记录的消息字数是否大于限制if(!canAddHistory(historyPoList) && historyPoList != null){beforeAddHistory(historyPoList);    //如果字数太多，对历史记录进行清理}List<RoleContent> historyList = null;if(historyPoList != null){//RoleContentPo是用来接收数据库的数据的，ai要使用的是RoleContent类型的数据，所以要使用stream流将类型转换一下。historyList = historyPoList.stream().map((item) -> {RoleContent roleContent = new RoleContent();roleContent.setRole(item.getRole());roleContent.setContent(item.getContent());return roleContent;}).collect(Collectors.toList());}else {historyList = new ArrayList<>(); //如果数据库中没有数据，就用一个空的历史记录给ai}ArrayList<RoleContent> questions = new ArrayList<>();
//**************************************  part2  ************************************//设置对话背景RoleContent system = new RoleContent();system.setRole("system");system.setContent(background);questions.add(system);//添加对话背景到要给ai的集合中//将历史消息加入其中if (historyList.size() > 0){questions.addAll(historyList);}//接收用户的输入信息RoleContent userRoleContent = RoleContent.createUserRoleContent(roleContentPo.getContent());//将用户的输入信息加入到要给ai的集合中questions.add(userRoleContent);//将用户的消息输入也加入到数据库RoleContentPo newUserRoleContent = new RoleContentPo();newUserRoleContent.setContent(userRoleContent.getContent());newUserRoleContent.setRole(userRoleContent.getRole());newUserRoleContent.setUid(roleContentPo.getUid());newUserRoleContent.setSendTime(new Date(System.currentTimeMillis()));//将当前时间存入数据库newUserRoleContent.setScid(roleContentPo.getScid());roleContentPoMapper.insert(newUserRoleContent);//将信息给ai，让ai处理后，结果会给到xfWebSocketListener上XFWebSocketListener xfWebSocketListener = new XFWebSocketListener();WebSocket webSocket = xfWebClient.sendMsg(roleContentPo.getUid(), questions, xfWebSocketListener);//*************************************  part3  *****************************************if (webSocket == null) {log.error("webSocket连接异常");ResultBean.fail("请求异常，请联系管理员");}try {int count = 0;int maxCount = xfConfig.getMaxResponseTime() * 5;while (count <= maxCount) {Thread.sleep(2000);log.info("大模型的回答是："+xfWebSocketListener.getAnswer());//这个地方是每两秒可以获得ai进行途中未完成的回答，可以进行异步返回给前端//todo:  在此处实现异步返回给前端（待实现）webSocketServer.sendToOne(roleContentPo.getScid().toString(),xfWebSocketListener.getAnswer());log.info("###############################################################################\n################################################################################################");if (xfWebSocketListener.isWsCloseFlag()) {break;}count++;}if (count > maxCount) {return ResultData.fail(ReturnCodeEnum.RC500.getCode(),"响应超时，请联系相关人员");}//获取ai的回答的总内容String totalAnswer = xfWebSocketListener.getAnswer();//ai的回答也要存进数据库中RoleContentPo roleContent = new RoleContentPo();roleContent.setRole("assistant");roleContent.setContent(totalAnswer);roleContent.setUid(roleContentPo.getUid());roleContent.setSendTime(new Date(System.currentTimeMillis()));roleContent.setScid(roleContentPo.getRcid());roleContent.setRcid(roleContentPo.getScid());roleContent.setType("text");roleContentPoMapper.insert(roleContent);return ResultData.success(roleContent);} catch (Exception e) {log.error("请求异常：{}", e);} finally {webSocket.close(1000, "");}return ResultData.success(null);}@Overridepublic synchronized  ResultData<RoleContentPo> pushVoiceMessageToXFServer(RoleContentPo roleContentPo, String background,String voiceMessage) {//******************  part1  ************************************************//从数据库中查询历史记录LambdaQueryWrapper<RoleContentPo> lambdaQueryWrapper = new LambdaQueryWrapper<>();lambdaQueryWrapper.eq(RoleContentPo::getUid,roleContentPo.getUid());    //查询对应uid的对话历史记录lambdaQueryWrapper.orderByAsc(RoleContentPo::getSendTime);   //要根据创建时间升序排序List<RoleContentPo> historyPoList = roleContentPoMapper.selectList(lambdaQueryWrapper);//判断是否历史记录的消息字数是否大于限制if(!canAddHistory(historyPoList) && historyPoList != null){beforeAddHistory(historyPoList);    //如果字数太多，对历史记录进行清理}List<RoleContent> historyList = null;if(historyPoList != null){//RoleContentPo是用来接收数据库的数据的，ai要使用的是RoleContent类型的数据，所以要使用stream流将类型转换一下。historyList = historyPoList.stream().map((item) -> {RoleContent roleContent = new RoleContent();roleContent.setRole(item.getRole());roleContent.setContent(item.getContent());return roleContent;}).collect(Collectors.toList());}else {historyList = new ArrayList<>(); //如果数据库中没有数据，就用一个空的历史记录给ai}ArrayList<RoleContent> questions = new ArrayList<>();
//**************************************  part2  ************************************//设置对话背景RoleContent system = new RoleContent();system.setRole("system");system.setContent(background);questions.add(system);//添加对话背景到要给ai的集合中//将历史消息加入其中if (historyList.size() > 0){questions.addAll(historyList);}//接收用户的输入信息RoleContent userRoleContent = RoleContent.createUserRoleContent(voiceMessage);//将用户的输入信息加入到要给ai的集合中questions.add(userRoleContent);//将信息给ai，让ai处理后，结果会给到xfWebSocketListener上XFWebSocketListener xfWebSocketListener = new XFWebSocketListener();WebSocket webSocket = xfWebClient.sendMsg(roleContentPo.getUid(), questions, xfWebSocketListener);//*************************************  part3  *****************************************if (webSocket == null) {log.error("webSocket连接异常");ResultBean.fail("请求异常，请联系管理员");}try {int count = 0;int maxCount = xfConfig.getMaxResponseTime() * 5;while (count <= maxCount) {Thread.sleep(2000);log.info("大模型的回答是："+xfWebSocketListener.getAnswer());//这个地方是每两秒可以获得ai进行途中未完成的回答，可以进行异步返回给前端//todo:  在此处实现异步返回给前端（待实现）log.info("###############################################################################\n################################################################################################");if (xfWebSocketListener.isWsCloseFlag()) {break;}count++;}if (count > maxCount) {return ResultData.fail(ReturnCodeEnum.RC500.getCode(),"响应超时，请联系相关人员");}//获取ai的回答的总内容String totalAnswer = xfWebSocketListener.getAnswer();//ai的回答也要存进数据库中RoleContentPo roleContent = new RoleContentPo();roleContent.setRole("assistant");roleContent.setContent(totalAnswer);roleContent.setUid(roleContent.getUid());roleContent.setSendTime(new Date(System.currentTimeMillis()));roleContent.setScid(roleContentPo.getRcid());roleContent.setRcid(roleContent.getScid());roleContent.setType("text");roleContentPoMapper.insert(roleContent);return ResultData.success(roleContent);} catch (Exception e) {log.error("请求异常：{}", e);} finally {webSocket.close(1000, "");}return ResultData.success(null);}public void beforeAddHistory(List<RoleContentPo> historyList){  // 如果历史记录过长，就删除5条记录for (int i = 0;i < 5;i++) {RoleContentPo roleContentPo = historyList.get(i);roleContentPoMapper.deleteById(roleContentPo);//数据库中的历史记录删除5条historyList.remove(i);//给ai的信息中也删除5条}}public boolean canAddHistory(List<RoleContentPo> historyList){  // 由于历史记录最大上线1.2W左右，需要判断是能能加入历史int history_length=0;for(RoleContentPo temp:historyList){history_length=history_length+temp.getContent().length();}if(history_length>11000){return false;}else{return true;}}
}

@Service
public class RoleContentPoServiceImpl extends ServiceImpl<RoleContentPoMapper, RoleContentPo> implements RoleContentPoService {@Resourceprivate RoleContentPoMapper roleContentPoMapper;@Resourceprivate PushService pushService;@Overridepublic ResultData<List<RoleContentPo>> getAllMessage(AiMessagePage page) {//构造分页构造器IPage<RoleContentPo> optionPage = new Page<>(page.getPageNum(),page.getPageSize());//构造条件构造器LambdaQueryWrapper<RoleContentPo> queryWrapper = new LambdaQueryWrapper<>();//当uid等于指定的值，将其记录查询出来queryWrapper.eq(RoleContentPo::getUid,page.getUid());//根据查询时间升序排列queryWrapper.orderByDesc(RoleContentPo::getSendTime);//分页查找roleContentPoMapper.selectPage(optionPage,queryWrapper);//查找的结果List<RoleContentPo> records = optionPage.getRecords();return ResultData.success(records);}@Overridepublic ResultData<Long> getAllMsgPageMax(AiMessagePage page) {//构造分页构造器IPage<RoleContentPo> optionPage = new Page<>(page.getPageNum(),page.getPageSize());//构造条件构造器LambdaQueryWrapper<RoleContentPo> queryWrapper = new LambdaQueryWrapper<>();//当uid等于指定的值，将其记录查询出来queryWrapper.eq(RoleContentPo::getUid,page.getUid());//根据查询时间升序排列queryWrapper.orderByAsc(RoleContentPo::getSendTime);//分页查找roleContentPoMapper.selectPage(optionPage,queryWrapper);return ResultData.success(optionPage.getPages());}@Overridepublic ResultData<RoleContentPo> voiceMsgToAi(RoleContentPo roleContentPo,String voiceMessage) {return pushService.pushVoiceMessageToXFServer(roleContentPo,"",voiceMessage);}
}

使用案例：

String answer = userAnswer.getAnswer()[0];RoleContentPo roleContentPo = new RoleContentPo();roleContentPo.setUid(userExamVo.getExamId());String content = """问题：""" + question.getText() + """参考答案：""" + question.getAnswer()[0] + """答案：""" + answer + """满分：""" + question.getScore() + """""";roleContentPo.setContent(content);roleContentPo.setRole("user");roleContentPo.setType("text");roleContentPo.setRcid(12345678L);roleContentPo.setScid(Long.parseLong(userExamVo.getUserId()));String background = """给你一个填空题的题目和答案，再告诉你这题的满分是多少，你只需要返回一个分数的数字，不要出现任何其他内容，我只要这个数字，不要出现任何其他内容。关于评分规则，如果一道比较灵活的题目，有多种作答，或者应该要分多个点，或者需要更多元化的答案，那么请分析用户的回答的质量来评分。参考答案只是一个用于参考的答案，不一定是唯一的正确答案，你可以根据你的判断来评分。你要分析的信息的格式为：问题：{问题内容}参考答案：{参考答案}答案：{用户的回答}满分：{满分}注意，请严格按照格式要求返回分数，不要出现任何其他内容，我只要这个数字，不要出现任何其他内容。""";ResultData resultData = pushService.pushMessageToXFServer(roleContentPo, background);RoleContentPo data1 = (RoleContentPo) resultData.getData();String data = data1.getContent();int resultScore = Integer.parseInt(data);score += resultScore;userAnswer.setScore(resultScore);

语音合成服务TTS：

oss工具类

这里建议使用阿里的Oss对象云存储来存取文件信息：

这里我还多做了一个可以传入File类的对象的存取，在TTS中会使用到

@Component
public class OssUtils {//读取配置文件的内容@Value("${aliyun.oss.file.endpoint}")private String ENDPOINT;@Value("${aliyun.oss.file.access-key-id}")private String ACCESS_KEY_ID;@Value("${aliyun.oss.file.serct-access-key}")private String ACCESS_KEY_SECRET;@Value("${aliyun.oss.file.bucket}")private String bucketName;private OSS ossClient;//初始化oss服务public void init() {ossClient = new OSSClientBuilder().build(ENDPOINT, ACCESS_KEY_ID, ACCESS_KEY_SECRET);}//上传文件// file:文件public String uploadFile(MultipartFile file) throws IOException {//获取文件的输入流InputStream inputstream = file.getInputStream();String filename = file.getOriginalFilename();//保证文件唯一性String uuid = UUID.randomUUID().toString().replaceAll("-", "");filename = uuid + filename;//按照类别进行分类// 判断文件类型String contentType = file.getContentType();String fileType = "unknown";if (contentType != null) {if (contentType.startsWith("image/")) {fileType = "image";    //图像文件} else if (contentType.startsWith("application/")) {fileType = "application"; //应用程序文件} else if (contentType.startsWith("text/")) {fileType = "text"; //文本文件 (包括 HTML、CSS、JavaScript)} else if (contentType.startsWith("video/")) {fileType = "video"; //视频文件} else if (contentType.startsWith("audio/")) {fileType = "audio"; //音频文件}} else {fileType = "other"; //其他文件}//文件路劲filename = fileType + "/" + filename;try {ossClient.putObject(bucketName, filename, inputstream);
//            // 设置 URL 过期时间
//            Date expiration = new Date(System.currentTimeMillis() + 60 * 1000 * 60);
//            URL url = ossClient.generatePresignedUrl(bucketName, filename, expiration);
//            return url.toString();} catch (OSSException oe) {System.out.println("Caught an OSSException, which means your request made it to OSS, "+ "but was rejected with an error response for some reason.");System.out.println("Error Message:" + oe.getErrorMessage());System.out.println("Error Code:" + oe.getErrorCode());System.out.println("Request ID:" + oe.getRequestId());System.out.println("Host ID:" + oe.getHostId());} finally {if (ossClient != null) {ossClient.shutdown();}}String url = "https://" + bucketName + "." + ENDPOINT + "/" + filename;//   return generatePresignedUrl(filename, 60 * 60);return url;}public String uploadFile(File file,String type) throws IOException {//获取文件的输入流InputStream inputstream = new FileInputStream(file);String filename = file.getName();if (file.exists() && file.isFile()) {long fileSizeInBytes = file.length();System.out.println("文件的字节大小为: " + fileSizeInBytes + " 字节");} else {System.out.println("文件不存在或不是一个常规文件。");}//保证文件唯一性String uuid = UUID.randomUUID().toString().replaceAll("-", "");filename = uuid + filename;//按照类别进行分类// 判断文件类型String contentType = type;String fileType = "unknown";if (contentType != null) {if (contentType.startsWith("image/")) {fileType = "image";    //图像文件} else if (contentType.startsWith("application/")) {fileType = "application"; //应用程序文件} else if (contentType.startsWith("text/")) {fileType = "text"; //文本文件 (包括 HTML、CSS、JavaScript)} else if (contentType.startsWith("video/")) {fileType = "video"; //视频文件} else if (contentType.startsWith("audio/")) {fileType = "audio"; //音频文件}} else {fileType = "other"; //其他文件}//文件路劲filename = fileType + "/" + filename;try {ossClient.putObject(bucketName, filename, inputstream);
//            // 设置 URL 过期时间
//            Date expiration = new Date(System.currentTimeMillis() + 60 * 1000 * 60);
//            URL url = ossClient.generatePresignedUrl(bucketName, filename, expiration);
//            return url.toString();System.out.println("上传成功，文件名为：" + filename);} catch (OSSException oe) {System.out.println("Caught an OSSException, which means your request made it to OSS, "+ "but was rejected with an error response for some reason.");System.out.println("Error Message:" + oe.getErrorMessage());System.out.println("Error Code:" + oe.getErrorCode());System.out.println("Request ID:" + oe.getRequestId());System.out.println("Host ID:" + oe.getHostId());} finally {if (ossClient != null) {ossClient.shutdown();}}String url = "https://" + bucketName + "." + ENDPOINT + "/" + filename;//   return generatePresignedUrl(filename, 60 * 60);return url;}//下载文件// objectName:oss中的相对路径// localPath:本地文件路径public void downloadFile(String objectName, String localPath) throws Exception {try {//获取文件的名称String fileName = new File(objectName).getName();// 调用ossClient.getObject返回一个OSSObject实例，该实例包含文件内容及文件元数据。
//            OSSObject ossObject = ossClient.getObject(bucketName, objectName);
//             调用ossObject.getObjectContent获取文件输入流，可读取此输入流获取其内容。
//            InputStream content = ossObject.getObjectContent();// 构建完整的文件路径File path = new File(localPath, fileName);// 检查并创建目录File parentDir = path.getParentFile();if (parentDir != null && !parentDir.exists() && !parentDir.mkdirs()) {throw new IOException("无法创建目录: " + parentDir.getAbsolutePath());}// 检查文件是否可以创建if (!path.exists() && !path.createNewFile()) {throw new IOException("无法创建文件: " + path.getAbsolutePath());}ossClient.getObject(new GetObjectRequest(bucketName, objectName), path);//流式下载
//            if (content != null) {
//                try (InputStream inputStream = content;
//                     OutputStream outputStream = new FileOutputStream(path)) {
//                    byte[] buffer = new byte[1024];
//                    int length;
//                     while ((length = inputStream.read(buffer)) > 0) {
//                        outputStream.write(buffer, 0, length);
//                    }
//                }
//            }} catch (OSSException oe) {System.out.println("Caught an OSSException, which means your request made it to OSS, "+ "but was rejected with an error response for some reason.");System.out.println("Error Message:" + oe.getErrorMessage());System.out.println("Error Code:" + oe.getErrorCode());System.out.println("Request ID:" + oe.getRequestId());System.out.println("Host ID:" + oe.getHostId());} finally {
//            if (ossClient != null) {
//                ossClient.shutdown();
//            }}}//列举指定目录的所有的文件/*** @param folderPath:文件夹路径* @return java.util.List<com.aliyun.oss.model.OSSObjectSummary>* @author zhang* @create 2024/10/31**///OSSobjectSummary存储了元数据public List<OSSObjectSummary> listAllObjects(String folderPath) {List<OSSObjectSummary> objectSummaries = null;try {// 创建 ListObjectsRequest 对象并设置前缀ListObjectsRequest listObjectsRequest = new ListObjectsRequest(bucketName);listObjectsRequest.setPrefix(folderPath);// ossClient.listObjects返回ObjectListing实例，包含此次listObject请求的返回结果。ObjectListing objectListing = ossClient.listObjects(listObjectsRequest);objectSummaries = objectListing.getObjectSummaries();//文件对应的相对路劲打印出来for (OSSObjectSummary objectSummary : objectListing.getObjectSummaries()) {System.out.println(" - " + objectSummary.getKey() + "  " +"(size = " + objectSummary.getSize() + ")");}while (true) {// 如果有下一批，继续获取if (objectListing.isTruncated()) {objectListing = ossClient.listObjects(String.valueOf(objectListing));objectSummaries.addAll(objectListing.getObjectSummaries());} else {break;}}return objectSummaries;} catch (OSSException oe) {System.out.println("Caught an OSSException, which means your request made it to OSS, "+ "but was rejected with an error response for some reason.");System.out.println("Error Message:" + oe.getErrorMessage());System.out.println("Error Code:" + oe.getErrorCode());System.out.println("Request ID:" + oe.getRequestId());System.out.println("Host ID:" + oe.getHostId());} finally {
//            if (ossClient != null) {
//                ossClient.shutdown();
//            }}return objectSummaries;}public List<String> getListUrl(String folderPath){ArrayList<String> listUrl = new ArrayList<>();List<OSSObjectSummary> summaries = listAllObjects(folderPath);for (OSSObjectSummary summary:summaries){String fileName = summary.getKey();String url = "https://" + bucketName + "." + ENDPOINT + "/" +fileName;System.out.println(url);listUrl.add(url);}return listUrl;}//删除文件//objectName:oss中的相对路径public void deleteFile(String objectName) {try {// 删除文件。ossClient.deleteObject(bucketName, objectName);} catch (OSSException oe) {System.out.println("Caught an OSSException, which means your request made it to OSS, "+ "but was rejected with an error response for some reason.");System.out.println("Error Message:" + oe.getErrorMessage());System.out.println("Error Code:" + oe.getErrorCode());System.out.println("Request ID:" + oe.getRequestId());System.out.println("Host ID:" + oe.getHostId());} finally {
//            if (ossClient != null) {
//                ossClient.shutdown();
//            }}}//查看文件是否已经存在:默认不存在  没怎么必要,上传是写入了uuid唯一标识/*** @param objectName:文件的相对路径* @return boolean* @author xinggang* @create 2024/10/31**/public boolean isExist(String objectName) {boolean found = false;try {// 判断文件是否存在。如果返回值为true，则文件存在，否则存储空间或者文件不存在。// 设置是否进行重定向或者镜像回源。默认值为true，表示忽略302重定向和镜像回源；如果设置isINoss为false，则进行302重定向或者镜像回源。//boolean isINoss = true;found = ossClient.doesObjectExist(bucketName, objectName);//boolean found = ossClient.doesObjectExist(bucketName, objectName, isINoss);return found;} catch (OSSException oe) {System.out.println("Caught an OSSException, which means your request made it to OSS, "+ "but was rejected with an error response for some reason.");System.out.println("Error Message:" + oe.getErrorMessage());System.out.println("Error Code:" + oe.getErrorCode());System.out.println("Request ID:" + oe.getRequestId());System.out.println("Host ID:" + oe.getHostId());} finally {}return found;}// 生成带签名的临时访问 URL,用于URL安全// filename:oss中的相对路径// expires:过期时间(分钟)public String generatePresignedUrl(String filename, long expirationInSeconds) {// 设置 URL 过期时间Date expiration = new Date(System.currentTimeMillis() + expirationInSeconds * 1000);// 生成带签名的 URLGeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucketName, filename);request.setExpiration(expiration);URL url = ossClient.generatePresignedUrl(request);// 返回 URL 字符串return url.toString();}public byte[] getObject(String pathUrl) {//初始化ossClient = new OSSClientBuilder().build(ENDPOINT, ACCESS_KEY_ID, ACCESS_KEY_SECRET);String key = pathUrl.replace(ENDPOINT + "", "").replaceAll(bucketName+".","").replaceAll("https://","");int index = key.indexOf("/");
//        String bucketName = key.substring(0, index);String filePath = key.substring(index + 1);InputStream inputStream = null;try {// 获取文件输入流inputStream = ossClient.getObject(bucketName, filePath).getObjectContent();} catch (Exception e) {e.printStackTrace();}// 读取文件内容ByteArrayOutputStream outputStream = new ByteArrayOutputStream();byte[] buffer = new byte[1024];int len;while (true) {try {if (!((len = inputStream.read(buffer)) != -1)) break;} catch (IOException e) {throw new RuntimeException(e);}outputStream.write(buffer, 0, len);}return outputStream.toByteArray();}//关闭oss服务public void shutdown() {if (ossClient != null) {ossClient.shutdown();}}}

aliyun: #阿里云的配置oss:file:endpoint: 。。。。。。。access-key-id: 。。。。。。serct-access-key: 。。。。。。。bucket: 。。。。。。

TTS工具类：

调用createVoice方法，传入想要合成的语音的内容，就会返回相应文件的oss的访问url。在websocket连接到星火服务之后，要sleep一小会，因为文件写出到指定文件需要一定时间。之前我没有sleep的时候，还没写出就给oss了，导致文件为0字节。所以这个sleep是有必要的，我这里有点久，可以适当短一点，能写出完成就行。

这里把它注册成了一个组件，在使用的时候注入，并且要设置其开启状态：案例：

@Resource
private WebTtsWs webTtsWs@PostMapping("/createVoice")public ResultData<String[]> createVoice(@RequestBody String voice) throws Exception {webTtsWs.setWsCloseFlag(false);String resultString = webTtsWs.createVoice(voice);Thread.sleep(3000);//等语音生成完成了再关闭，给一点等待时间webTtsWs.setWsCloseFlag(true);return ResultData.success(resultString);}

@Service
public class WebTtsWs {@Resourceprivate OssUtils ossUtils;// 地址与鉴权信息public static final String hostUrl = "https://tts-api.xfyun.cn/v2/tts";// 均到控制台-语音合成页面获取public static final String appid = "。。。。。";public static final String apiSecret = "。。。。。。。";public static final String apiKey = "。。。。。。。。";// 合成文本public static final String TEXT = "讯飞的文字合成语音功能，测试成功";// 合成文本编码格式public static final String TTE = "UTF8"; // 小语种必须使用UNICODE编码作为值// 发音人参数。到控制台-我的应用-语音合成-添加试用或购买发音人，添加后即显示该发音人参数值，若试用未添加的发音人会报错11200public static final String VCN = "aisjiuxu";// jsonpublic static final Gson gson = new Gson();public static boolean wsCloseFlag = false;public void setWsCloseFlag(boolean wsCloseFlag) {WebTtsWs.wsCloseFlag = wsCloseFlag;}public String createVoice(String text) throws Exception {ossUtils.init();String wsUrl = getAuthUrl(hostUrl, apiKey, apiSecret).replace("https://", "wss://");String basePath = System.getProperty("user.dir") + "\\voiceSource\\" + System.currentTimeMillis() + ".mp3";File file = new File(basePath);OutputStream outputStream = new FileOutputStream(file);websocketWork(wsUrl, outputStream,text);Thread.sleep(2500);if (file.exists() && file.isFile()) {long fileSizeInBytes = file.length();System.out.println("文件的字节大小为: " + fileSizeInBytes + " 字节");} else {System.out.println("文件不存在或不是一个常规文件。");}String fileString = ossUtils.uploadFile(file, "audio/mpeg");ossUtils.shutdown();return fileString;}// Websocket方法，为流式文件生成服务public static void websocketWork(String wsUrl, OutputStream outputStream,String text) {try {URI uri = new URI(wsUrl);WebSocketClient webSocketClient = new WebSocketClient(uri) {@Overridepublic void onOpen(ServerHandshake serverHandshake) {System.out.println("ws建立连接成功...");}@Overridepublic void onMessage(String text) {// System.out.println(text);JsonParse myJsonParse = gson.fromJson(text, JsonParse.class);System.out.println("---------------" + myJsonParse + "数据的值是这个");if (myJsonParse.code != 0) {System.out.println("发生错误，错误码为：" + myJsonParse.code);System.out.println("本次请求的sid为：" + myJsonParse.sid);}if (myJsonParse.data != null) {try {byte[] textBase64Decode = Base64.getDecoder().decode(myJsonParse.data.audio);outputStream.write(textBase64Decode);outputStream.flush();} catch (Exception e) {e.printStackTrace();}if (myJsonParse.data.status == 2) {try {outputStream.close();} catch (IOException e) {e.printStackTrace();}System.out.println("本次请求的sid==>" + myJsonParse.sid);// 可以关闭连接，释放资源
//                            wsCloseFlag = true;}}}@Overridepublic void onClose(int i, String s, boolean b) {System.out.println("ws链接已关闭，本次请求完成...");}@Overridepublic void onError(Exception e) {System.out.println("发生错误 " + e.getMessage());}};// 建立连接webSocketClient.connect();while (!webSocketClient.getReadyState().equals(WebSocket.READYSTATE.OPEN)) {//System.out.println("正在连接...");Thread.sleep(100);}MyThread webSocketThread = new MyThread(webSocketClient,text);webSocketThread.start();} catch (Exception e) {System.out.println(e.getMessage());}}// 线程来发送音频与参数static class MyThread extends Thread {WebSocketClient webSocketClient;String text;public MyThread(WebSocketClient webSocketClient,String text) {this.webSocketClient = webSocketClient;this.text = text;}public void run() {String requestJson;//请求参数json串try {requestJson = "{\n" +"  \"common\": {\n" +"    \"app_id\": \"" + appid + "\"\n" +"  },\n" +"  \"business\": {\n" +"    \"aue\": \"lame\",\n" +"    \"sfl\": 1,\n" +"    \"tte\": \"" + TTE + "\",\n" +"    \"ent\": \"intp65\",\n" +"    \"vcn\": \"" + VCN + "\",\n" +"    \"pitch\": 50,\n" +"    \"volume\": 100,\n" +"    \"speed\": 50\n" +"  },\n" +"  \"data\": {\n" +"    \"status\": 2,\n" +"    \"text\": \"" + Base64.getEncoder().encodeToString(text.getBytes(StandardCharsets.UTF_8)) + "\"\n" +//"    \"text\": \"" + Base64.getEncoder().encodeToString(TEXT.getBytes("UTF-16LE")) + "\"\n" +"  }\n" +"}";webSocketClient.send(requestJson);// 等待服务端返回完毕后关闭while (!wsCloseFlag) {Thread.sleep(200);}webSocketClient.close();} catch (Exception e) {e.printStackTrace();}}}// 鉴权方法public static String getAuthUrl(String hostUrl, String apiKey, String apiSecret) throws Exception {URL url = new URL(hostUrl);// 时间SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss z", Locale.US);format.setTimeZone(TimeZone.getTimeZone("GMT"));String date = format.format(new Date());// 拼接String preStr = "host: " + url.getHost() + "\n" +"date: " + date + "\n" +"GET " + url.getPath() + " HTTP/1.1";//System.out.println(preStr);// SHA256加密Mac mac = Mac.getInstance("hmacsha256");SecretKeySpec spec = new SecretKeySpec(apiSecret.getBytes(StandardCharsets.UTF_8), "hmacsha256");mac.init(spec);byte[] hexDigits = mac.doFinal(preStr.getBytes(StandardCharsets.UTF_8));// Base64加密String sha = Base64.getEncoder().encodeToString(hexDigits);// 拼接String authorization = String.format("api_key=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"", apiKey, "hmac-sha256", "host date request-line", sha);// 拼接地址HttpUrl httpUrl = Objects.requireNonNull(HttpUrl.parse("https://" + url.getHost() + url.getPath())).newBuilder().//addQueryParameter("authorization", Base64.getEncoder().encodeToString(authorization.getBytes(StandardCharsets.UTF_8))).//addQueryParameter("date", date).//addQueryParameter("host", url.getHost()).//build();return httpUrl.toString();}//返回的json结果拆解class JsonParse {int code;String sid;Data data;}class Data {int status;String audio;}
}

语音识别服务 IAT：

调用voiceMsgSendToAI方法，传入语音文件的MutiPartFile文件形式数据，就会返回识别结果。

iat工具类：

public class WebIATWS extends WebSocketListener {private static final String hostUrl = "https://iat-api.xfyun.cn/v2/iat"; //中英文，http url 不支持解析 ws/wss schema// private static final String hostUrl = "https://iat-niche-api.xfyun.cn/v2/iat";//小语种private static final String appid = "。。。。。。。。。"; //在控制台-我的应用获取private static final String apiSecret = "。。。。。。。。"; //在控制台-我的应用-语音听写（流式版）获取private static final String apiKey = "。。。。。。。。。"; //在控制台-我的应用-语音听写（流式版）获取private String filePath;public static final int StatusFirstFrame = 0;public static final int StatusContinueFrame = 1;public static final int StatusLastFrame = 2;public static final Gson json = new Gson();Decoder decoder = new Decoder();// 开始时间private static Date dateBegin = new Date();// 结束时间private static Date dateEnd = new Date();private static final SimpleDateFormat sdf = new SimpleDateFormat("yyy-MM-dd HH:mm:ss.SSS");private static String resultContent = "无";private MultipartFile multipartFile;public WebIATWS(MultipartFile multipartFile) {this.multipartFile = multipartFile;}//将录音文件发给aipublic String voiceMsgSendToAI(MultipartFile multipartFile) throws Exception {// 构建鉴权urlString authUrl = getAuthUrl(hostUrl, apiKey, apiSecret);OkHttpClient client = new OkHttpClient.Builder().build();//将url中的 schema http://和https://分别替换为ws:// 和 wss://String url = authUrl.toString().replace("http://", "ws://").replace("https://", "wss://");//System.out.println(url);Request request = new Request.Builder().url(url).build();// System.out.println(client.newCall(request).execute());//System.out.println("url===>" + url);System.out.println(filePath);String pathOption = StringEscapeUtils.escapeJava(filePath);System.out.println(pathOption);WebSocket webSocket = client.newWebSocket(request, new WebIATWS(multipartFile));Thread.sleep(1500);System.out.println("要发送的结果是："+resultContent);return resultContent;}@Overridepublic void onOpen(WebSocket webSocket, Response response) {super.onOpen(webSocket, response);new Thread(()->{//连接成功，开始发送数据int frameSize = 1280; //每一帧音频的大小,建议每 40ms 发送 122Bint intervel = 40;int status = 0;  // 音频的状态String basePath = System.getProperty("user.dir") + "\\voiceSource";//原始文件名String originalFilename = multipartFile.getOriginalFilename();//获得文件名末尾的文件格式，如：.jpgString suffix = originalFilename.substring(originalFilename.lastIndexOf("."));//生成一个带指定文件格式的不重复的字符串做文件命名String fileName = UUID.randomUUID() + suffix;//根据需求判断是否要新建一个目录File dir = new File(basePath);if (!dir.exists()){//如果不存在，则创建此目录dir.mkdirs();}try {//将临时文件转存到指定位置multipartFile.transferTo(new File(basePath + "\\" + fileName));} catch (IOException e) {e.printStackTrace();}System.out.println(fileName);String filePath = basePath + "\\" + fileName;try (FileInputStream fs = new FileInputStream(filePath)) {byte[] buffer = new byte[frameSize];// 发送音频end:while (true) {int len = fs.read(buffer);if (len == -1) {status = StatusLastFrame;  //文件读完，改变status 为 2}switch (status) {case StatusFirstFrame:   // 第一帧音频status = 0JsonObject frame = new JsonObject();JsonObject business = new JsonObject();  //第一帧必须发送JsonObject common = new JsonObject();  //第一帧必须发送JsonObject data = new JsonObject();  //每一帧都要发送// 填充commoncommon.addProperty("app_id", appid);//填充businessbusiness.addProperty("language", "zh_cn");//business.addProperty("language", "en_us");//英文//business.addProperty("language", "ja_jp");//日语，在控制台可添加试用或购买//business.addProperty("language", "ko_kr");//韩语，在控制台可添加试用或购买//business.addProperty("language", "ru-ru");//俄语，在控制台可添加试用或购买business.addProperty("domain", "iat");
//                            business.addProperty("accent", "mandarin");//中文方言请在控制台添加试用，添加后即展示相应参数值//business.addProperty("nunum", 0);//business.addProperty("ptt", 0);//标点符号
//                            business.addProperty("rlang", "zh-hk"); // zh-cn :简体中文（默认值）zh-hk :繁体香港(若未授权不生效，在控制台可免费开通)//business.addProperty("vinfo", 1);business.addProperty("dwa", "wpgs");//动态修正(若未授权不生效，在控制台可免费开通)business.addProperty("nbest", 5);// 句子多候选(若未授权不生效，在控制台可免费开通)business.addProperty("wbest", 3);// 词级多候选(若未授权不生效，在控制台可免费开通)//填充datadata.addProperty("status", StatusFirstFrame);data.addProperty("format", "audio/L16;rate=16000");data.addProperty("encoding", "lame");data.addProperty("audio", Base64.getEncoder().encodeToString(Arrays.copyOf(buffer, len)));//填充frameframe.add("common", common);frame.add("business", business);frame.add("data", data);webSocket.send(frame.toString());status = StatusContinueFrame;  // 发送完第一帧改变status 为 1break;case StatusContinueFrame:  //中间帧status = 1JsonObject frame1 = new JsonObject();JsonObject data1 = new JsonObject();data1.addProperty("status", StatusContinueFrame);data1.addProperty("format", "audio/L16;rate=16000");data1.addProperty("encoding", "lame");data1.addProperty("audio", Base64.getEncoder().encodeToString(Arrays.copyOf(buffer, len)));frame1.add("data", data1);webSocket.send(frame1.toString());// System.out.println("send continue");break;case StatusLastFrame:    // 最后一帧音频status = 2 ，标志音频发送结束JsonObject frame2 = new JsonObject();JsonObject data2 = new JsonObject();data2.addProperty("status", StatusLastFrame);data2.addProperty("audio", "");data2.addProperty("format", "audio/L16;rate=16000");data2.addProperty("encoding", "lame");frame2.add("data", data2);webSocket.send(frame2.toString());System.out.println("sendlast");break end;}Thread.sleep(intervel); //模拟音频采样延时}System.out.println("all data is send");} catch (FileNotFoundException e) {e.printStackTrace();} catch (IOException e) {e.printStackTrace();} catch (InterruptedException e) {e.printStackTrace();}}).start();}@Overridepublic void onMessage(WebSocket webSocket, String text) {super.onMessage(webSocket, text);//System.out.println(text);ResponseData resp = json.fromJson(text, ResponseData.class);if (resp != null) {if (resp.getCode() != 0) {System.out.println( "code=>" + resp.getCode() + " error=>" + resp.getMessage() + " sid=" + resp.getSid());System.out.println( "错误码查询链接：https://www.xfyun.cn/document/error-code");return;}if (resp.getData() != null) {if (resp.getData().getResult() != null) {Text te = resp.getData().getResult().getText();//System.out.println(te.toString());try {decoder.decode(te);System.out.println("中间识别结果 ==》" + decoder.toString());} catch (Exception e) {e.printStackTrace();}}if (resp.getData().getStatus() == 2) {// todo  resp.data.status ==2 说明数据全部返回完毕，可以关闭连接，释放资源System.out.println("session end ");dateEnd = new Date();System.out.println(sdf.format(dateBegin) + "开始");System.out.println(sdf.format(dateEnd) + "结束");System.out.println("耗时:" + (dateEnd.getTime() - dateBegin.getTime()) + "ms");System.out.println("最终识别结果 ==》" + decoder.toString());resultContent = decoder.toString();System.out.println(resultContent+"########");System.out.println("本次识别sid ==》" + resp.getSid());decoder.discard();webSocket.close(1000, "");} else {// todo 根据返回的数据处理resultContent = decoder.toString();}}}}@Overridepublic void onFailure(WebSocket webSocket, Throwable t, Response response) {super.onFailure(webSocket, t, response);try {if (null != response) {int code = response.code();System.out.println("onFailure code:" + code);System.out.println("onFailure body:" + response.body().string());if (101 != code) {System.out.println("connection failed");System.exit(0);}}} catch (IOException e) {// TODO Auto-generated catch blocke.printStackTrace();}}public static String getAuthUrl(String hostUrl, String apiKey, String apiSecret) throws Exception {URL url = new URL(hostUrl);SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss z", Locale.US);format.setTimeZone(TimeZone.getTimeZone("GMT"));String date = format.format(new Date());StringBuilder builder = new StringBuilder("host: ").append(url.getHost()).append("\n").//append("date: ").append(date).append("\n").//append("GET ").append(url.getPath()).append(" HTTP/1.1");//System.out.println(builder);Charset charset = Charset.forName("UTF-8");Mac mac = Mac.getInstance("hmacsha256");SecretKeySpec spec = new SecretKeySpec(apiSecret.getBytes(charset), "hmacsha256");mac.init(spec);byte[] hexDigits = mac.doFinal(builder.toString().getBytes(charset));String sha = Base64.getEncoder().encodeToString(hexDigits);//System.out.println(sha);String authorization = String.format("api_key=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"", apiKey, "hmac-sha256", "host date request-line", sha);//System.out.println(authorization);HttpUrl httpUrl = HttpUrl.parse("https://" + url.getHost() + url.getPath()).newBuilder().//addQueryParameter("authorization", Base64.getEncoder().encodeToString(authorization.getBytes(charset))).//addQueryParameter("date", date).//addQueryParameter("host", url.getHost()).//build();return httpUrl.toString();}public static class ResponseData {private int code;private String message;private String sid;private Data data;public int getCode() {return code;}public String getMessage() {return this.message;}public String getSid() {return sid;}public Data getData() {return data;}}public static class Data {private int status;private Result result;public int getStatus() {return status;}public Result getResult() {return result;}}public static class Result {int bg;int ed;String pgs;int[] rg;int sn;Ws[] ws;boolean ls;JsonObject vad;public Text getText() {Text text = new Text();StringBuilder sb = new StringBuilder();for (Ws ws : this.ws) {sb.append(ws.cw[0].w);}text.sn = this.sn;text.text = sb.toString();text.sn = this.sn;text.rg = this.rg;text.pgs = this.pgs;text.bg = this.bg;text.ed = this.ed;text.ls = this.ls;text.vad = this.vad==null ? null : this.vad;return text;}}public static class Ws {Cw[] cw;int bg;int ed;}public static class Cw {int sc;String w;}public static class Text {int sn;int bg;int ed;String text;String pgs;int[] rg;boolean deleted;boolean ls;JsonObject vad;@Overridepublic String toString() {return "Text{" +"bg=" + bg +", ed=" + ed +", ls=" + ls +", sn=" + sn +", text='" + text + '\'' +", pgs=" + pgs +", rg=" + Arrays.toString(rg) +", deleted=" + deleted +", vad=" + (vad==null ? "null" : vad.getAsJsonArray("ws").toString()) +'}';}}//解析返回数据，仅供参考public static class Decoder {private Text[] texts;private int defc = 10;public Decoder() {this.texts = new Text[this.defc];}public synchronized void decode(Text text) {if (text.sn >= this.defc) {this.resize();}if ("rpl".equals(text.pgs)) {for (int i = text.rg[0]; i <= text.rg[1]; i++) {this.texts[i].deleted = true;}}this.texts[text.sn] = text;}public String toString() {StringBuilder sb = new StringBuilder();for (Text t : this.texts) {if (t != null && !t.deleted) {sb.append(t.text);}}return sb.toString();}public void resize() {int oc = this.defc;this.defc <<= 1;Text[] old = this.texts;this.texts = new Text[this.defc];for (int i = 0; i < oc; i++) {this.texts[i] = old[i];}}public void discard(){for(int i=0;i<this.texts.length;i++){this.texts[i]= null;}}}
}

使用案例：

@PostMapping("/audioAnalysis")public ResultData<String> audioAnalysis(@RequestParam("file") MultipartFile file) throws Exception {WebIATWS webIATWS = new WebIATWS(file);String resultString = webIATWS.voiceMsgSendToAI(file);return ResultData.success(resultString);}

tuputech表情识别服务：

要多放两个util类：

public class FileUtil {/*** 读取文件内容为二进制数组* * @param filePath* @return* @throws IOException*/public static byte[] read(String filePath) throws IOException {InputStream in = new FileInputStream(filePath);byte[] data = inputStream2ByteArray(in);in.close();return data;}/*** 流转二进制数组* * @param in* @return* @throws IOException*/private static byte[] inputStream2ByteArray(InputStream in) throws IOException {ByteArrayOutputStream out = new ByteArrayOutputStream();byte[] buffer = new byte[1024 * 4];int n = 0;while ((n = in.read(buffer)) != -1) {out.write(buffer, 0, n);}return out.toByteArray();}/*** 保存文件* * @param filePath* @param fileName* @param content*/public static void save(String filePath, String fileName, byte[] content) {try {File filedir = new File(filePath);if (!filedir.exists()) {filedir.mkdirs();}File file = new File(filedir, fileName);OutputStream os = new FileOutputStream(file);os.write(content, 0, content.length);os.flush();os.close();} catch (FileNotFoundException e) {e.printStackTrace();} catch (IOException e) {e.printStackTrace();}}
}

public class HttpUtil {/*** 发送post请求* * @param url* @param header* @param body* @return*/public static String doPost1(String url, Map<String, String> header, byte[] body) {String result = "";BufferedReader in = null;try {// 设置 urlURL realUrl = new URL(url);URLConnection connection = realUrl.openConnection();HttpURLConnection httpURLConnection = (HttpURLConnection) connection;// 设置 headerfor (String key : header.keySet()) {httpURLConnection.setRequestProperty(key, header.get(key));}// 设置请求 bodyhttpURLConnection.setDoOutput(true);httpURLConnection.setDoInput(true);httpURLConnection.setRequestProperty("Content-Type", "binary/octet-stream");OutputStream out = httpURLConnection.getOutputStream();out.write(body);out.flush();out.close();if (HttpURLConnection.HTTP_OK != httpURLConnection.getResponseCode()) {System.out.println("Http 请求失败，状态码：" + httpURLConnection.getResponseCode());return null;}// 获取响应bodyin = new BufferedReader(new InputStreamReader(httpURLConnection.getInputStream()));String line;while ((line = in.readLine()) != null) {result += line;}} catch (Exception e) {return null;}return result;}}

表情识别工具类：

调用方法getFaceExpression传入文件名（任意），传入表情图片文件的MutiPartFile文件数据，即可返回一个表情识别数据。

/***人脸特征分析表情WebAPI接口调用示例接口文档(必看)：https://doc.xfyun.cn/rest_api/%E4%BA%BA%E8%84%B8%E7%89%B9%E5%BE%81%E5%88%86%E6%9E%90-%E8%A1%A8%E6%83%85.html*图片属性：png、jpg、jpeg、bmp、tif图片大小不超过800k*(Very Important)创建完webapi应用添加服务之后一定要设置ip白名单，找到控制台--我的应用--设置ip白名单，如何设置参考：http://bbs.xfyun.cn/forum.php?mod=viewthread&tid=41891*错误码链接：https://www.xfyun.cn/document/error-code (code返回错误码时必看)*@author iflytek
*/
public class face {// webapi 接口地址private static final String URL = "http://tupapi.xfyun.cn/v1/expression";// 应用ID(必须为webapi类型应用,并人脸特征分析服务,参考帖子如何创建一个webapi应用：http://bbs.xfyun.cn/forum.php?mod=viewthread&tid=36481)private static final String APPID = "。。。。。。。";// 接口密钥(webapi类型应用开通人脸特征分析服务后，控制台--我的应用---人脸特征分析---服务的apikeyprivate static final String API_KEY = "。。。。。。。";// 图片数据可以通过两种方式上传，第一种在请求头设置image_url参数，第二种将图片二进制数据写入请求体中。若同时设置，以第一种为准。// 此demo使用第二种方式进行上传图片地址，如果想使用第一种方式，请求体为空即可。// 图片名称private static final String IMAGE_NAME = "1.jpg";// 图片url//private static final String IMAGE_URL = " ";// 图片地址private static final String PATH = "image/1.jpg";/*** WebAPI 调用示例程序* * @param args* @throws IOException*/public String getFaceExpression(String imageName, MultipartFile file) throws IOException {Map<String, String> header = buildHttpHeader(imageName,filePath);byte[] imageByteArray = file.getBytes();String result = HttpUtil.doPost1(URL,header, imageByteArray);System.out.println("接口调用结果：" + result);return result;}/*** 组装http请求头*/private static Map<String, String> buildHttpHeader(String imageName) throws UnsupportedEncodingException {String curTime = System.currentTimeMillis() / 1000L + "";String param = "{\"image_name\":\"" + imageName +  "\"}";String paramBase64 = new String(Base64.encodeBase64(param.getBytes("UTF-8")));String checkSum = DigestUtils.md5Hex(API_KEY + curTime + paramBase64);Map<String, String> header = new HashMap<String, String>();header.put("Content-Type", "application/x-www-form-urlencoded; charset=utf-8");header.put("X-Param", paramBase64);header.put("X-CurTime", curTime);header.put("X-CheckSum", checkSum);header.put("X-Appid", APPID);return header;}
}

结果中信息的意思如下

参数字段说明：
rate ：介于0-1间的浮点数，表示该图像被识别为某个分类的概率值，概率越高、机器越肯定
rates：各个label分别对应的概率值的数组，顺序如label的大小从小到大的顺序
label：大于等于0时，表明图片属于哪个分类或结果；等于-1时，代表该图片文件有错误，或者格式不支持（gif图不支持）
name：图片的url地址或名称
review：本次识别结果是否存在偏差，返回true时存在偏差，可信度较低，返回false时可信度较高，具体可参考rate参数值
fileList：每张图片的识别结果
reviewCount：需要复审的图片数量
statistic：各个分类的图片数量
label的值及其对应的表情
0：其他(非人脸表情图片)
1：其他表情
2：喜悦
3：愤怒
4：悲伤
5：惊恐
6：厌恶
7：中性

使用案例：

@PostMapping("/expressionAnalysis")public String expressionAnalysis(@RequestParam("file") MultipartFile file) throws IOException {ossUtils.init();String url = ossUtils.uploadFile(file);ossUtils.shutdown();System.out.println(url);String[] split = url.split("/");String fileName = split[split.length - 1];face face = new face();String faceExpression = face.getFaceExpression(fileName, url,file);return faceExpression;}@PostMapping("/expressionAnalysisResult")public ResultData<String> expressionAnalysisResult(@RequestBody FileDetail[] fileDetail) throws IOException {String expressionData = JSON.toJSONString(fileDetail);Long userId = LearningThreadLocalUtil.getUser();RoleContentPo roleContentPo = new RoleContentPo();roleContentPo.setRole("user");roleContentPo.setRcid(12345678L);roleContentPo.setScid(111111111L);roleContentPo.setUid("expressionAnalysis" + System.currentTimeMillis() + "-" + userId);roleContentPo.setContent(expressionData);roleContentPo.setSendTime(new Date());String background = """你是一个面试建议小助手，你要根据用户的面试数据，温柔地鼓励用户，温柔地给用户一些建议。不要太死板，要温和。用户会给你一个表情数据的json格式的字符串，里面包含了面试过程中抓拍到的表情分析的结果集，你需要根据这个结果集，给出合理的分析和建议。这些是表情数据的json格式的说明：参数字段说明：rate ：介于0-1间的浮点数，表示该图像被识别为某个分类的概率值，概率越高、机器越肯定rates：各个label分别对应的概率值的数组，顺序如label的大小从小到大的顺序label：大于等于0时，表明图片属于哪个分类或结果；等于-1时，代表该图片文件有错误，或者格式不支持（gif图不支持）name：图片的url地址或名称review：本次识别结果是否存在偏差，返回true时存在偏差，可信度较低，返回false时可信度较高，具体可参考rate参数值label的值及其对应的表情0：其他(非人脸表情图片)1：其他表情2：喜悦3：愤怒4：悲伤5：惊恐6：厌恶7：中性注意：最终的结果中，不用包含整体分析，我只要得到给用户的本次面试的建议即可。记住你是一个平易近人的面试小助手。""";ResultData resultData = pushService.pushMessageToXFServer(roleContentPo, background);RoleContentPo data = (RoleContentPo) resultData.getData();String content = data.getContent();return ResultData.success(content);}