当前位置：首页 > news >正文

Datawhale AI夏令营-基于带货视频评论的用户洞察挑战赛

news 2025/7/15 5:23:58

一.赛事目标

基于星火大模型Spark 4.0 Ultra，对视频和评论的数据进行商品识别，情感分析，归类分析，最终为带货效果进行评价。并通过优化模型来提高评价准确度

二.赛事环境

1.基础平台：星火大模型Spark 4.0 Ultra

2.赛事数据：视频，视频弹窗评论

包含85条脱敏后的带货视频数据及6477条评论文本数据

包括少量有人工标注结果的训练集（仅包含商品识别和情感分析的标注结果）以及未标注的测试集。

带货视频内容文本信息的数据格式

序号	变量名称	变量格式	解释
1	video_id	string	视频id
2	video_desc	string	视频描述
3	video_tags	string	视频标签
4	product_name	string	推广商品名称

评论区文本信息的数据格式

序号	变量名称	变量格式	解释
1	video_id	string	视频id
2	comment_id	string	评论id
3	comment_text	string	评论文本
4	sentiment_category	int	关于商品的情感倾向分类
5	user_scenario	int	是否与用户场景有关，0表示否，1表示是
6	user_question	int	是否与用户疑问有关，0表示否，1表示是
7	user_suggestion	int	是否与用户建议有关，0表示否，1表示是
8	positive_cluster_theme	string	按正面倾向聚类的类簇主题词
9	negative_cluster_theme	string	按负面倾向聚类的类簇主题词
10	scenario_cluster_theme	string	按用户场景聚类的类簇主题词
11	question_cluster_theme	string	按用户疑问聚类的类簇主题词
12	suggestion_cluster_theme	string	按用户建议聚类的类簇主题词

三.赛事任务

【商品识别】从视频的数据里精准识别推广商品；
【情感分析】对评论文本进行多维度情感分析，涵盖维度见数据说明；
【评论聚类】按商品对归属指定维度的评论进行聚类，并提炼类簇总结词。

四.赛事目标

基于给定的赛事步骤

1.加载数据

import pandas as pd
video_data = pd.read_csv("origin_videos_data.csv")
comments_data = pd.read_csv("origin_comments_data.csv")

2.取样表条数10条

video_data.sample(10)

3.提取表头

comments_data.head()

4.将视频数据的两个字段合并到一个字段，用来提取商品信息

video_data["text"] = video_data["video_desc"].fillna("") + " " + video_data["video_tags"].fillna("")

5.加载分词器，情感分析，聚类分析等工具包

import jieba
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC
from sklearn.cluster import KMeans
from sklearn.pipeline import make_pipeline

6.进行分词，从视频数据里获取到商品名product_name的集合

product_name_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut, max_features=50), SGDClassifier()
)
product_name_predictor.fit(video_data[~video_data["product_name"].isnull()]["text"],video_data[~video_data["product_name"].isnull()]["product_name"],
)
video_data["product_name"] = product_name_predictor.predict(video_data["text"])

max_features=50：表示在转换过程中只保留前50个最常见的词汇特征。这有助于减少特征的维度，并提高计算效率。

7.加载评论数据

comments_data.columns

8.对评论数据进行情感分析

for col in ['sentiment_category','user_scenario', 'user_question', 'user_suggestion']:predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), SGDClassifier())predictor.fit(comments_data[~comments_data[col].isnull()]["comment_text"],comments_data[~comments_data[col].isnull()][col],)comments_data[col] = predictor.predict(comments_data["comment_text"])

9.聚类提取关键词数量

top_n_words = 20

top_n_words 的意义
主题表示：top_n_words 决定了你从每个聚类中提取多少个关键词来代表该聚类的主题。例如，如果 top_n_words=20，那么每个聚类主题将包含 20 个关键词，这些关键词是根据它们在聚类中心的贡献度排名选出的。

对理解聚类的影响：选择不同数量的关键词会影响你对聚类主题的理解。更多的关键词可以提供更全面的主题描述，但也可能引入噪声；较少的关键词则可能会导致主题描述不够完整。

总结来说，top_n_words 是一个关键的参数，它帮助定义在每个聚类中提取的最重要词汇的数量，进而影响对聚类主题的理解。

10.情感分析-1

kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["sentiment_category"].isin([1, 3])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["sentiment_category"].isin([1, 3])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["sentiment_category"].isin([1, 3]), "positive_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]

11.情感分析-1

kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["sentiment_category"].isin([2, 3])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["sentiment_category"].isin([2, 3])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["sentiment_category"].isin([2, 3]), "negative_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]

14.情感分析-3

kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["user_scenario"].isin([1])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["user_scenario"].isin([1])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["user_scenario"].isin([1]), "scenario_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]

15.情感分析-4

kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["user_question"].isin([1])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["user_question"].isin([1])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["user_question"].isin([1]), "question_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]

16.情感分析-5

kmeans_predictor = make_pipeline(TfidfVectorizer(tokenizer=jieba.lcut), KMeans(n_clusters=5)
)kmeans_predictor.fit(comments_data[comments_data["user_suggestion"].isin([1])]["comment_text"])
kmeans_cluster_label = kmeans_predictor.predict(comments_data[comments_data["user_suggestion"].isin([1])]["comment_text"])kmeans_top_word = []
tfidf_vectorizer = kmeans_predictor.named_steps['tfidfvectorizer']
kmeans_model = kmeans_predictor.named_steps['kmeans']
feature_names = tfidf_vectorizer.get_feature_names_out()
cluster_centers = kmeans_model.cluster_centers_
for i in range(kmeans_model.n_clusters):top_feature_indices = cluster_centers[i].argsort()[::-1]top_word = ' '.join([feature_names[idx] for idx in top_feature_indices[:top_n_words]])kmeans_top_word.append(top_word)comments_data.loc[comments_data["user_suggestion"].isin([1]), "suggestion_cluster_theme"] = [kmeans_top_word[x] for x in kmeans_cluster_label]

17.创建目录

mkdir submit

18.打压缩包

video_data[["video_id", "product_name"]].to_csv("submit/submit_videos.csv", index=None)
comments_data[['video_id', 'comment_id', 'sentiment_category','user_scenario', 'user_question', 'user_suggestion','positive_cluster_theme', 'negative_cluster_theme','scenario_cluster_theme', 'question_cluster_theme','suggestion_cluster_theme']].to_csv("submit/submit_comments.csv", index=None)