零样本学习 zero-shot

1 是什么

2 如何利用零样本学习进行跨模态迁移？

demo代码 

安装clip

pip install ftfy regex tqdm

pip install git+https://github.com/openai/CLIP.git

import torch
import clip
from PIL import Image

# 加载 CLIP 模型
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

# 定义文本标签
text_labels = ["a photo of a cat", "a photo of a dog"]
text_inputs = clip.tokenize(text_labels).to(device)

# 加载图像
image = Image.open("test_image.jpg")
image_input = preprocess(image).unsqueeze(0).to(device)

# 计算图像和文本的特征
with torch.no_grad():
    image_features = model.encode_image(image_input)
    text_features = model.encode_text(text_inputs)

# 计算相似度
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)

# 预测结果
predicted_index = similarity.argmax().item()
predicted_label = text_labels[predicted_index]

print(f"Predicted label: {predicted_label}")

3 零样本学习有哪些应用场景？

4 零样本学习跨模态迁移的评估