深圳分销网站设计哪家好专业 网站设计公司价格
一.赛事目标
基于星火大模型Spark 4.0 Ultra，对视频和评论的数据进行商品识别，情感分析，归类分析，最终为带货效果进行评价。并通过优化模型来提高评价准确度
二.赛事环境
1.基础平台：星火大模型Spark 4.0 Ultra
2.赛事数据：视频，视频弹窗评论
包含85条脱敏后的带货视频数据及6477条评论文本数据
包括少量有人工标注结果的训练集（仅包含商品识别和情感分析的标注结果）以及未标注的测试集。
- 带货视频内容文本信息的数据格式
| 序号 | 变量名称 | 变量格式 | 解释 |
|---|---|---|---|
| 1 | video_id | string | 视频id |
| 2 | video_desc | string | 视频描述 |
| 3 | video_tags | string | 视频标签 |
| 4 | product_name | string | 推广商品名称 |
- 评论区文本信息的数据格式
| 序号 | 变量名称 | 变量格式 | 解释 |
|---|---|---|---|
| 1 | video_id | string | 视频id |
| 2 | comment_id | string | 评论id |
| 3 | comment_text | string | 评论文本 |
| 4 | sentiment_category | int | 关于商品的情感倾向分类 |
| 5 | user_scenario | int | 是否与用户场景有关，0表示否，1表示是 |
| 6 | user_question | int | 是否与用户疑问有关，0表示否，1表示是 |
| 7 | user_suggestion | int | 是否与用户建议有关，0表示否，1表示是 |
| 8 | positive_cluster_theme | string | 按正面倾向聚类的类簇主题词 |
| 9 | negative_cluster_theme | string | 按负面倾向聚类的类簇主题词 |
| 10 | scenario_cluster_theme | string | 按用户场景聚类的类簇主题词 |
| 11 | question_cluster_theme | string | 按用户疑问聚类的类簇主题词 |
| 12 | suggestion_cluster_theme | string | 按用户建议聚类的类簇主题词 |
三.赛事任务
【商品识别】从视频的数据里精准识别推广商品；
【情感分析】对评论文本进行多维度情感分析，涵盖维度见数据说明；
【评论聚类】按商品对归属指定维度的评论进行聚类，并提炼类簇总结词。
四.赛事目标
基于给定的赛事步骤
1.加载数据
import pandas as pd
video_data = pd.read_csv("origin_videos_data.csv")
comments_data = pd.read_csv("origin_comments_data.csv")2.取样表条数10条
video_data.sample(10)3.提取表头
comments_data.head()4.将视频数据的两个字段合并到一个字段，用来提取商品信息
video_data["text"] = video_data["video_desc"].fillna("") + " " + video_data["video_tags"].fillna("")5.加载分词器，情感分析，聚类分析等工具包
import jieba
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC
from sklearn.cluster import KMeans
from sklearn.pipeline import make_pipeline6.进行分词，从视频数据里获取到商品名product_name的集合
product_name_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut, max_features=50), SGDClassifier()
)
product_name_predictor.fit(video_data[~video_data["product_name"].isnull()]["text"],video_data[~video_data["product_name"].isnull()]["product_name"],
)
video_data["product_name"] = product_name_predictor.predict(video_data["text"])max_features=50：表示在转换过程中只保留前50个最常见的词汇特征。这有助于减少特征的维度，并提高计算效率。
7.加载评论数据
comments_data.columns8.对评论数据进行情感分析
for col in ['sentiment_category','user_scenario', 'user_question', 'user_suggestion']:predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), SGDClassifier())predictor.fit(comments_data[~comments_data[col].isnull()]["comment_text"],comments_data[~comments_data[col].isnull()][col],)comments_data[col] = predictor.predict(comments_data["comment_text"])9.聚类提取关键词数量
top_n_words = 20top_n_words 的意义
主题表示：top_n_words 决定了你从每个聚类中提取多少个关键词来代表该聚类的主题。例如，如果 top_n_words=20，那么每个聚类主题将包含 20 个关键词，这些关键词是根据它们在聚类中心的贡献度排名选出的。
对理解聚类的影响：选择不同数量的关键词会影响你对聚类主题的理解。更多的关键词可以提供更全面的主题描述，但也可能引入噪声；较少的关键词则可能会导致主题描述不够完整。
总结来说，top_n_words 是一个关键的参数，它帮助定义在每个聚类中提取的最重要词汇的数量，进而影响对聚类主题的理解。
10.情感分析-1
kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["sentiment_category"].isin([1, 3])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["sentiment_category"].isin([1, 3])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["sentiment_category"].isin([1, 3]), "positive_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]11.情感分析-1
kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["sentiment_category"].isin([2, 3])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["sentiment_category"].isin([2, 3])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["sentiment_category"].isin([2, 3]), "negative_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]14.情感分析-3
kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["user_scenario"].isin([1])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["user_scenario"].isin([1])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["user_scenario"].isin([1]), "scenario_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]15.情感分析-4
kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["user_question"].isin([1])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["user_question"].isin([1])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["user_question"].isin([1]), "question_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]16.情感分析-5
kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["user_suggestion"].isin([1])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["user_suggestion"].isin([1])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["user_suggestion"].isin([1]), "suggestion_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]17.创建目录
mkdir submit18.打压缩包
video_data[["video_id", "product_name"]].to_csv("submit/submit_videos.csv", index=None)
comments_data[['video_id', 'comment_id', 'sentiment_category','user_scenario', 'user_question', 'user_suggestion','positive_cluster_theme', 'negative_cluster_theme','scenario_cluster_theme', 'question_cluster_theme','suggestion_cluster_theme']].to_csv("submit/submit_comments.csv", index=None)五.操作步骤
为了提高赛事提高分数，本文从一下几个方面进行微调
1.修改n_clusters=5，默认抽取10个视频样本
结果：
分数详情
2.20个关键字，仍然保持n_clusters=5
最终结果
分数详情
3.增加样本，选择20个样本，其他保持和2一致
分数更低，因为视频数据有一些是空的，没有手工标注的。可能手工标注之后会提高
总结：
赛事最高微调到228份多点，后面可以再手工标注后优化