黑马头条_SpringCloud项目阶段五:openFeign服务接入以及接入腾讯云内容安全服务实现文章提交违规信息自动审核
gitee地址:https://gitee.com/whltaoin_admin/hmtt_cloud-project
版本:1c600acd20d6787a187db4406befd60d7df92983
知识点一:文章自动审核流程
知识点二:腾讯云文本内容安全服务接入
简介:
腾讯文本内容安全(Text Moderation System,TMS)是一款文本内容智能识别审核服务,针对用户上传的文本进行内容安全识别并审核的安全服务,文本内容安全服务能够做到识别准确率高、召回率高,多维度覆盖不同类型的文本内容,同时产品将会持续更新审核服务识别标准及产品服务能力。
内容安全控制台
地址:https://console.cloud.tencent.com/cms/clouds/package
接入方式
本文选用SDK接入方法:https://cloud.tencent.com/document/product/1124/100983
API
云API示例中心:https://console.cloud.tencent.com/api/explorer?Product=tms&Version=2020-12-29&Action=TextModeration
SDK示例
SDK示例:https://console.cloud.tencent.com/api/explorer?Product=tms&Version=2020-12-29&Action=TextModeration
参数说明
参数说明地址:https://cloud.tencent.com/document/api/1124/51860
集成项目
- 申请密钥:
地址:https://console.cloud.tencent.com/cam/capi
- 倒入依赖
SDK包名称:tencentcloud-sdk-java-tms
版本:3.1.1321
使用条件:Java 7+
# 版本在maven生效需要时间,如获取不到对应的版本,可以调低版本号<dependency><groupId>com.tencentcloudapi</groupId><artifactId>tencentcloud-sdk-java-common</artifactId><version>LATEST</version></dependency><dependency><groupId>com.tencentcloudapi</groupId><artifactId>tencentcloud-sdk-java-tms</artifactId><version>LATEST</version></dependency><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId></dependency>
- 定义返回实体
JSON格式示例
{"Response": {"BizType": "0","DataId": "123","DetailResults": [{"Keywords": ["色情"],"Label": "Porn","LibId": "12","LibName": "Porn","LibType": 0,"Score": 72,"Suggestion": "Review"},{"Keywords": ["色情"],"Label": "","LibId": "1","LibName": "Porn","LibType": 2,"Score": 0,"Suggestion": "Block"}],"Extra": "xx","Keywords": ["加我好友,给你发优惠券"],"Label": "Ad","RequestId": "x2123-123123-123","RiskDetails": [{"Label": "RiskAccount","Level": 2}],"Score": 87,"Suggestion": "Block"}
}
package cn.varin.tencent.entity;import lombok.Data;@Datapublic class TencentContentCheckResponseData {private String BizType;private String DataId;private String Extra;private String[] Keywords;private String Label;private String RequestId;private Integer Score;private String Suggestion;}
- 具体方法类定义
注意:腾讯云文本安全检测时,需要将文本内容转成base64编码格式
package cn.varin.tencent;import cn.varin.tencent.entity.TencentContentCheckResponseData;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.exceptions.ServerException;
import com.aliyuncs.green.model.v20180509.TextScanRequest;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.HttpResponse;
import com.aliyuncs.profile.DefaultProfile;
import com.aliyuncs.profile.IClientProfile;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
import com.tencentcloudapi.common.AbstractModel;import java.util.*;import com.tencentcloudapi.common.Credential;
import com.tencentcloudapi.common.profile.ClientProfile;
import com.tencentcloudapi.common.profile.HttpProfile;
import com.tencentcloudapi.common.exception.TencentCloudSDKException;
import com.tencentcloudapi.tms.v20201229.TmsClient;
import com.tencentcloudapi.tms.v20201229.models.*;@Getter
@Setter
@Component
@ConfigurationProperties(prefix = "tencent")
public class GreenTextScan {private String accessKey;private String accessId;public TencentContentCheckResponseData greeTextScan(String content) throws Exception {TencentContentCheckResponseData responseData =null;try{// 密钥信息从环境变量读取,需要提前在环境变量中设置 TENCENTCLOUD_SECRET_ID 和 TENCENTCLOUD_SECRET_KEY// 使用环境变量方式可以避免密钥硬编码在代码中,提高安全性// 生产环境建议使用更安全的密钥管理方案,如密钥管理系统(KMS)、容器密钥注入等// 请参见:https://cloud.tencent.com/document/product/1278/85305// 密钥可前往官网控制台 https://console.cloud.tencent.com/cam/capi 进行获取Credential cred = new Credential(accessId, accessKey);// 使用临时密钥示例// Credential cred = new Credential("SecretId", "SecretKey", "Token");// 实例化一个http选项,可选的,没有特殊需求可以跳过HttpProfile httpProfile = new HttpProfile();httpProfile.setEndpoint("tms.tencentcloudapi.com");// 实例化一个client选项,可选的,没有特殊需求可以跳过ClientProfile clientProfile = new ClientProfile();clientProfile.setHttpProfile(httpProfile);// 实例化要请求产品的client对象,clientProfile是可选的TmsClient client = new TmsClient(cred, "ap-shanghai", clientProfile);// 实例化一个请求对象,每个接口都会对应一个request对象TextModerationRequest req = new TextModerationRequest();String base64Content = Base64.getEncoder().encodeToString(content.getBytes());req.setContent(base64Content);// 返回的resp是一个TextModerationResponse的实例,与请求对象对应TextModerationResponse resp = client.TextModeration(req);// 输出json格式的字符串回包System.out.println(AbstractModel.toJsonString(resp));responseData = JSON.parseObject(AbstractModel.toJsonString(resp), TencentContentCheckResponseData.class);
// System.out.println(o.getResponse());} catch (TencentCloudSDKException e) {System.out.println(e.toString());}return responseData;}}
- 调试返回结果示例
TencentContentCheckResponseData(BizType=0, DataId=, Extra=, Keywords=[], Label=Normal, RequestId=53d026e7-addf-45f9-80f0-50996afebada, Score=0, Suggestion=Pass)
知识点三:腾讯云图片内容安全服务接入
简介:
腾讯图片内容安全(Image Moderation System,IMS)是一款采用前沿的图像识别算法,结合海量的违规图片数据进行训练建模,针对用户上传的图片进行内容安全识别并审核的安全服务,图片内容安全服务能够做到识别准确率高、召回率高,多维度覆盖不同类型的内容,同时产品将会持续更新审核服务识别标准及产品服务能力。
内容安全控制台
地址:https://console.cloud.tencent.com/cms/clouds/package
接入方式
本文选用SDK接入方法:https://cloud.tencent.com/document/product/1125/100989
API
云API示例中心:https://console.cloud.tencent.com/api/explorer?Product=ims&Version=2020-12-29&Action=ImageModeration
SDK示例
SDK示例:https://console.cloud.tencent.com/api/explorer?Product=ims&Version=2020-12-29&Action=ImageModeration
参数说明
参数说明地址:https://cloud.tencent.com/document/api/1125/53273
集成项目
- 申请密钥:
地址:https://console.cloud.tencent.com/cam/capi
- 倒入依赖
SDK包名称:tencentcloud-sdk-java-ims
版本:3.1.1321
使用条件:Java 7+
# 版本在maven生效需要时间,如获取不到对应的版本,可以调低版本号<dependency><groupId>com.tencentcloudapi</groupId><artifactId>tencentcloud-sdk-java-common</artifactId><version>LATEST</version></dependency><dependency><groupId>com.tencentcloudapi</groupId><artifactId>tencentcloud-sdk-java-ims</artifactId><version>3.1.1312</version>
</dependency><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId></dependency>
- 定义返回实体
JSON格式示例
{"Response": {"BizType": "0","DataId": "123","DetailResults": [{"Keywords": ["色情"],"Label": "Porn","LibId": "12","LibName": "Porn","LibType": 0,"Score": 72,"Suggestion": "Review"},{"Keywords": ["色情"],"Label": "","LibId": "1","LibName": "Porn","LibType": 2,"Score": 0,"Suggestion": "Block"}],"Extra": "xx","Keywords": ["加我好友,给你发优惠券"],"Label": "Ad","RequestId": "x2123-123123-123","RiskDetails": [{"Label": "RiskAccount","Level": 2}],"Score": 87,"Suggestion": "Block"}
}
package cn.varin.tencent.entity;import lombok.Data;@Datapublic class TencentContentCheckResponseData {private String BizType;private String DataId;private String Extra;private String[] Keywords;private String Label;private String RequestId;private Integer Score;private String Suggestion;}
- 具体方法类定义
注意:腾讯云文本安全检测时,需要将文本内容转成base64编码格式
package cn.varin.tencent;import cn.varin.tencent.entity.TencentContentCheckResponseData;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.exceptions.ServerException;
import com.aliyuncs.green.model.v20180509.TextScanRequest;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.HttpResponse;
import com.aliyuncs.profile.DefaultProfile;
import com.aliyuncs.profile.IClientProfile;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
import com.tencentcloudapi.common.AbstractModel;import java.util.*;import com.tencentcloudapi.common.Credential;
import com.tencentcloudapi.common.profile.ClientProfile;
import com.tencentcloudapi.common.profile.HttpProfile;
import com.tencentcloudapi.common.exception.TencentCloudSDKException;
import com.tencentcloudapi.tms.v20201229.TmsClient;
import com.tencentcloudapi.tms.v20201229.models.*;@Getter
@Setter
@Component
@ConfigurationProperties(prefix = "tencent")
public class GreenTextScan {private String accessKey;private String accessId;public TencentContentCheckResponseData greeTextScan(String content) throws Exception {TencentContentCheckResponseData responseData =null;try{// 密钥信息从环境变量读取,需要提前在环境变量中设置 TENCENTCLOUD_SECRET_ID 和 TENCENTCLOUD_SECRET_KEY// 使用环境变量方式可以避免密钥硬编码在代码中,提高安全性// 生产环境建议使用更安全的密钥管理方案,如密钥管理系统(KMS)、容器密钥注入等// 请参见:https://cloud.tencent.com/document/product/1278/85305// 密钥可前往官网控制台 https://console.cloud.tencent.com/cam/capi 进行获取Credential cred = new Credential(accessId, accessKey);// 使用临时密钥示例// Credential cred = new Credential("SecretId", "SecretKey", "Token");// 实例化一个http选项,可选的,没有特殊需求可以跳过HttpProfile httpProfile = new HttpProfile();httpProfile.setEndpoint("tms.tencentcloudapi.com");// 实例化一个client选项,可选的,没有特殊需求可以跳过ClientProfile clientProfile = new ClientProfile();clientProfile.setHttpProfile(httpProfile);// 实例化要请求产品的client对象,clientProfile是可选的TmsClient client = new TmsClient(cred, "ap-shanghai", clientProfile);// 实例化一个请求对象,每个接口都会对应一个request对象TextModerationRequest req = new TextModerationRequest();String base64Content = Base64.getEncoder().encodeToString(content.getBytes());req.setContent(base64Content);// 返回的resp是一个TextModerationResponse的实例,与请求对象对应TextModerationResponse resp = client.TextModeration(req);// 输出json格式的字符串回包System.out.println(AbstractModel.toJsonString(resp));responseData = JSON.parseObject(AbstractModel.toJsonString(resp), TencentContentCheckResponseData.class);
// System.out.println(o.getResponse());} catch (TencentCloudSDKException e) {System.out.println(e.toString());}return responseData;}}
- 调试返回结果示例
TencentContentCheckResponseData(BizType=0, DataId=, Extra=, Keywords=[], Label=Normal, RequestId=53d026e7-addf-45f9-80f0-50996afebada, Score=0, Suggestion=Pass)
注意点:因为图片检测和内容检测的模块存在于comment中,但是他又需要注入,所以需要再****spring.factories
配置文件中,添加上这两个类的全限定类路径。
知识点四:app端文章保存
涉及到的数据库表
ap_article 文章信息表
ap_article_config 文章配置
ap_article_content 文章内容
分布式ID-雪花算法使用
- 使用场景:在使用分库时,相同的表id可以出现重复,所有需要一个不会重复的id生成算法
- 特点:
- Long类型的ID
- 组成:
- 符号位:0
- 41位的毫秒数
- 10位工作机器
- 前五位是机房id,后五位时机器id
- 12位的序列化
实现思路
接口信息
返回信息
实现步骤
在feign模块中定义接口
- 倒入依赖
<dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-openfeign</artifactId></dependency>
- 建立apArticleDto实体
package com.heima.model.article.dtos;import com.heima.model.article.pojos.ApArticle;
import lombok.Data;@Data
public class ArticleDto extends ApArticle {/*** 文章内容*/private String content;
}
- feign Article客户端
package cn.varin.apis.article;import com.heima.model.article.dtos.ArticleDto;
import com.heima.model.article.pojos.ApArticleConfig;
import com.heima.model.common.dtos.ResponseResult;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;@FeignClient(value = "leadnews-article")
public interface IArticleClient {@PostMapping("/api/v1/article/save")ResponseResult save(@RequestBody ArticleDto dto);
}
- 在article-service模块下田间feign包并实现IArticleClient
package com.heima.article.feign;import cn.varin.apis.article.IArticleClient;
import com.heima.article.mapper.ApArticleConfigMapper;
import com.heima.article.service.ApArticleService;
import com.heima.model.article.dtos.ArticleDto;
import com.heima.model.common.dtos.ResponseResult;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;import javax.annotation.PostConstruct;@RestController
public class ArtcleClient implements IArticleClient {@Autowiredprivate ApArticleService apArticleService;@PostMapping("api/v1/article/save")@Overridepublic ResponseResult save(ArticleDto dto) {return apArticleService.saveArticle(dto);}
}
- 实现Article模块中的service方法
package com.heima.article.service.impl;import com.baomidou.mybatisplus.core.toolkit.Wrappers;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import com.heima.article.mapper.ApArticleConfigMapper;
import com.heima.article.mapper.ApArticleContentMapper;
import com.heima.article.mapper.ApArticleMapper;
import com.heima.article.service.ApArticleService;
import com.heima.model.article.dtos.ArticleDto;
import com.heima.model.article.dtos.ArticleHomeDto;
import com.heima.model.article.pojos.ApArticle;
import com.heima.model.article.pojos.ApArticleConfig;
import com.heima.model.article.pojos.ApArticleContent;
import com.heima.model.common.dtos.ResponseResult;
import com.heima.model.common.enums.AppHttpCodeEnum;
import lombok.extern.slf4j.Slf4j;
import org.checkerframework.checker.units.qual.Temperature;
import org.springframework.beans.BeanUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;@Service
@Slf4j
@Transactional
public class ApArticleServiceImpl extends ServiceImpl<ApArticleMapper, ApArticle> implements ApArticleService {@Autowiredprivate ApArticleMapper apArticleMapper;@Overridepublic ResponseResult load(ArticleHomeDto dto, short type) {return ResponseResult.okResult(apArticleMapper.loadArticleList(dto,type));}@Autowiredprivate ApArticleConfigMapper apArticleConfigMapper;@Autowiredprivate ApArticleContentMapper apArticleContentMapper;;@Overridepublic ResponseResult saveArticle(ArticleDto dto) {// 检查参数if (dto==null) {return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);}ApArticle apArticle = new ApArticle();// 拷贝属性BeanUtils.copyProperties(dto, apArticle);// 判断是新增还是修改if (dto.getId()==null) {save(apArticle);// 文章配置信息保存ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId());apArticleConfigMapper.insert(apArticleConfig);// 文章内容保存ApArticleContent apArticleContent = new ApArticleContent();apArticleContent.setArticleId(apArticle.getId());apArticleContent.setContent(dto.getContent());apArticleContentMapper.insert(apArticleContent);// 新增}else{// 修改updateById(apArticle);ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers.<ApArticleContent>lambdaQuery().eq(ApArticleContent::getArticleId, dto.getId()));apArticleContent.setContent(dto.getContent());apArticleContentMapper.updateById(apArticleContent);}return ResponseResult.okResult(apArticle.getId());}
}
- PostMan测试-新增
URL:http://127.0.0.1:51802/api/v1/article/save
注意点:在进行新增测试时,将属性id去除,因为在service层中的判断条件仅为:id==null,
存在id会直接进入到修改
- JSON
{"title":"varin","authoId":1102,"layout":1,"labels":"hmtt","publishTime":"2028-03-14T11:35:49.000Z","images": "http://192.168.200.130:9000/leadnews/2021/04/26/5ddbdb5c68094ce393b08a47860da275.jpg","content":"2vaffffff"
}
- PostMan测试-修改
URL:http://127.0.0.1:51802/api/v1/article/save
- JSON
{"id":"1969663614994386945","title":"whltaoin","authoId":1102,"layout":1,"labels":"hmtt","publishTime":"2028-03-14T11:35:49.000Z","images": "http://192.168.200.130:9000/leadnews/2021/04/26/5ddbdb5c68094ce393b08a47860da275.jpg","content":"whltaoin"
}
知识点五:文章审核实现
- 具体步骤:
- 在自媒体模块中建立一个处理类,WmNewsAutoScanSerivce
- 通过id获取到每条文章的具体信息
- 从getContent属性中,将文本和图片分别提取到对应的map集合中。
- 再分别建立图片和文本审核的方法进行审核。
实现思路:
判断文章是否存在
判断文章状态是否为待审核
提取文本和图片
文本审核
图片审核
修改自媒体文章审核状态
package com.heima.wemedia.service.impl;import cn.varin.tencent.GreenImageScan;
import cn.varin.tencent.GreenTextScan;
import cn.varin.tencent.entity.TencentContentCheckResponseData;
import com.alibaba.fastjson.JSONArray;
import com.heima.model.wemedia.pojos.WmNews;
import com.heima.wemedia.mapper.WmNewsMapper;
import com.heima.wemedia.service.WmNewAutoScanService;
import com.heima.wemedia.service.WmNewsService;
//import com.heima.wemedia.tencent.GreenTextScan;
import com.tencentcloudapi.common.exception.TencentCloudSDKException;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.kafka.common.protocol.types.Field;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;import java.util.*;@Service
@Slf4j
@Transactional
public class WmNewAutoScanServiceImpl implements WmNewAutoScanService {@Autowiredprivate WmNewsMapper wmNewsMapper;@Overridepublic void AutoScanWmNews(Integer id) {// 1. 判断文章是否存在WmNews wmNews = wmNewsMapper.selectById(id);if(wmNews == null){throw new RuntimeException("WmNewAutoScanServiceImpl-文章不存在");}// 判断文章状态是否为待审核if (wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())) {// 提取文本和图片Map<String, Object> map = handleTextAndImage(wmNews);// 文本审核Boolean textStatus = textAutoScan(map,wmNews);if (textStatus) {// 文字已经违规,不需要再审核图片了return;}// 图片审核Boolean imageStatus = imageAutoScan(map,wmNews);if (imageStatus) {return;}System.out.println("suceess");// 修改自媒体文章审核状态wmNews.setStatus((short)8);wmNewsMapper.updateById(wmNews);}}@Autowiredprivate GreenImageScan greenImageScan;// 审核图片private Boolean imageAutoScan(Map<String, Object> map, WmNews wmNews) {boolean flag = false;List<String> images =(List<String>) map.get("images");for (String image : images) {try {TencentContentCheckResponseData responseData = greenImageScan.checkImageScan(image);if (!responseData.getSuggestion().equals("Pass")) {flag = true;wmNews.setStatus(WmNews.Status.FAIL.getCode());wmNewsMapper.updateById(wmNews);break;}} catch (TencentCloudSDKException e) {throw new RuntimeException(e);}}return flag;}@Autowiredprivate GreenTextScan greenTextScan;// 审核文本private Boolean textAutoScan(Map<String, Object> map, WmNews wmNews) {Boolean flag = true;String result =map.get("text").toString();try {TencentContentCheckResponseData responseData = greenTextScan.greeTextScan(result);String suggestion = responseData.getSuggestion();if (suggestion.equals("Pass")) {flag = false;}else{flag = true;wmNews.setStatus(WmNews.Status.FAIL.getCode());wmNewsMapper.updateById(wmNews);}System.out.println(suggestion);} catch (Exception e) {throw new RuntimeException(e);}return flag;}// 提取文本和图片到map 中private Map<String, Object> handleTextAndImage(WmNews wmNews) {Map<String, Object> map = new HashMap<>();// 存储字符StringBuffer sb = new StringBuffer();sb.append(wmNews.getTitle());sb.append(wmNews.getLabels());// 存储图片List<String> images = new ArrayList<>();if (StringUtils.isNotBlank(wmNews.getContent())) {List<Map> maps = JSONArray.parseArray(wmNews.getContent(), Map.class);maps.forEach(item -> {if (item.get("type").equals("image")) {images.add(item.get("value").toString());}else if (item.get("type").equals("text")) {sb.append(item.get("value").toString());}});if (StringUtils.isNotBlank(wmNews.getImages())) {// 封面String[] split = wmNews.getImages().split(".");images.addAll(Arrays.asList(split));}}map.put("images", images);map.put("text", sb.toString());return map;}
}
- 测试类:
package cn.varin;import com.heima.wemedia.WemediaApplication;
import com.heima.wemedia.service.WmNewAutoScanService;
import com.tencentcloudapi.common.exception.TencentCloudSDKException;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runner.Runner;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;@SpringBootTest(classes = WemediaApplication.class)
@RunWith(SpringRunner.class)
public class handleScanTest {@Autowiredprivate WmNewAutoScanService wmNewAutoScanService;@Testpublic void wmNewAutoScanTest() {wmNewAutoScanService.AutoScanWmNews(6232);}
}
- 测试结果:
今天也要加油呀⛽️🎆