MLU实现 comfyui+wan2.1 完成图生视频创作
文章目录
- 前言
- 一、平台环境准备
- 二、环境配置
- 1.comfyui安装
- 三、模型下载
- 四、工作流导入
- 五、开始运行
- 六、效果展示
前言
Wan2.1，是阿里云开源的一款AI视频生成大模型，支持文生视频和图生视频任务，具备强大的视觉生成能力。无论是复杂的运动生成，还是物理规律的精准模拟，Wan2.1都能轻松应对。
14B参数的专业版Wan2.1模型在Vbench评测中以86.22%的成绩超越Sora、Luma等国内外模型，稳居榜首。1.3B参数的极速版则能在消费级显卡上运行，显存需求低，适合二次开发和学术研究
一、平台环境准备
镜像选择：pytorch:v24.10-torch2.4.0-torchmlu1.23.1-ubuntu22.04-py310
【请注意仔细查看自己的镜像版本，老版本改法，请查阅之前文章】
卡选择：任意一款MLU3系列及以上卡
二、环境配置
1.comfyui安装
私信获取 comfyui-v0.6.0.tar.gz 插件版
git clone -bv0.3.27 https://githubfast.com/comfyanonymous/ComfyUI.git #直接下载官方的diffusers
tar -zxvf comfyui-v0.6.0.tar.gz
git apply comfyui_0.3.27_mlu0.6.0.patch
pip install -r requirements.txt #至此环境安装完成
三、模型下载
git-lfs clone https://hf-mirror.com/Comfy-Org/Wan_2.1_ComfyUI_repackaged
#这个下载很久请科学上网，或者从魔搭找相应模型
将下图中的所split_files的模型放置与comfyui代码中的指定位置
ComfyUI/models/clip_vision
ComfyUI/models/diffusion_models
ComfyUI/models/text_encoders
ComfyUI/models/vae
四、工作流导入
{"id": "7722c9f1-95c7-4e3e-ac98-2a42f04a4883","revision": 0,"last_node_id": 13,"last_link_id": 16,"nodes": [{"id": 12,"type": "CLIPLoader","pos": [742.6447143554688,67.46834564208984],"size": [315,98],"flags": {},"order": 0,"mode": 0,"inputs": [],"outputs": [{"name": "CLIP","type": "CLIP","links": [9,10]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "CLIPLoader"},"widgets_values": ["umt5_xxl_fp16.safetensors","wan","default"]},{"id": 5,"type": "CLIPVisionLoader","pos": [723.444580078125,467.7684326171875],"size": [315,58],"flags": {},"order": 1,"mode": 0,"inputs": [],"outputs": [{"name": "CLIP_VISION","type": "CLIP_VISION","links": [2]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "CLIPVisionLoader"},"widgets_values": ["clip_vision_h.safetensors"]},{"id": 1,"type": "UNETLoader","pos": [746.2453002929688,278.0558166503906],"size": [315,82],"flags": {},"order": 2,"mode": 0,"inputs": [],"outputs": [{"name": "MODEL","type": "MODEL","links": [12]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "UNETLoader"},"widgets_values": ["wan2.1_vace_1.3B_preview_fp16.safetensors","default"]},{"id": 8,"type": "VAEDecode","pos": [2216.811767578125,276.6905517578125],"size": [210,46],"flags": {},"order": 10,"mode": 0,"inputs": [{"name": "samples","type": "LATENT","link": 11},{"name": "vae","type": "VAE","link": 5}],"outputs": [{"name": "IMAGE","type": "IMAGE","links": [6]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "VAEDecode"},"widgets_values": []},{"id": 7,"type": "VAELoader","pos": [1994.1451416015625,567.4683837890625],"size": [315,58],"flags": {},"order": 3,"mode": 0,"inputs": [],"outputs": [{"name": "VAE","type": "VAE","links": [5,16]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "VAELoader"},"widgets_values": ["Wan2.1_VAE.pth"]},{"id": 9,"type": "SaveAnimatedWEBP","pos": [2561.8115234375,277.4683532714844],"size": [315,196],"flags": {},"order": 11,"mode": 0,"inputs": [{"name": "images","type": "IMAGE","link": 6}],"outputs": [],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "SaveAnimatedWEBP"},"widgets_values": ["ComfyUI",16.000000000000004,true,80,"default"]},{"id": 6,"type": "WanImageToVideo","pos": [1726.144775390625,819.4683837890625],"size": [342.5999755859375,210],"flags": {},"order": 8,"mode": 0,"inputs": [{"name": "positive","type": "CONDITIONING","link": 8},{"name": "negative","type": "CONDITIONING","link": 7},{"name": "vae","type": "VAE","link": 16},{"name": "clip_vision_output","shape": 7,"type": "CLIP_VISION_OUTPUT","link": 3},{"name": "start_image","shape": 7,"type": "IMAGE","link": 4}],"outputs": [{"name": "positive","type": "CONDITIONING","links": [13]},{"name": "negative","type": "CONDITIONING","links": [14]},{"name": "latent","type": "LATENT","links": [15]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "WanImageToVideo"},"widgets_values": [832,1280,81,1]},{"id": 13,"type": "KSampler","pos": [1755.1324462890625,107.07707977294922],"size": [315,262],"flags": {},"order": 9,"mode": 0,"inputs": [{"name": "model","type": "MODEL","link": 12},{"name": "positive","type": "CONDITIONING","link": 13},{"name": "negative","type": "CONDITIONING","link": 14},{"name": "latent_image","type": "LATENT","link": 15}],"outputs": [{"name": "LATENT","type": "LATENT","links": [11]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "KSampler"},"widgets_values": [546683418730044,"randomize",20,8,"euler","normal",1]},{"id": 4,"type": "CLIPVisionEncode","pos": [1186.0400390625,639.8672485351562],"size": [380.4000244140625,78],"flags": {},"order": 7,"mode": 0,"inputs": [{"name": "clip_vision","type": "CLIP_VISION","link": 2},{"name": "image","type": "IMAGE","link": 1}],"outputs": [{"name": "CLIP_VISION_OUTPUT","type": "CLIP_VISION_OUTPUT","links": [3]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "CLIPVisionEncode"},"widgets_values": ["center"]},{"id": 10,"type": "CLIPTextEncode","pos": [1167.561767578125,23.462318420410156],"size": [400,200],"flags": {},"order": 5,"mode": 0,"inputs": [{"name": "clip","type": "CLIP","link": 9}],"outputs": [{"name": "CONDITIONING","type": "CONDITIONING","links": [8]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "CLIPTextEncode"},"widgets_values": ["A dog jumped from the grass into the river High quality, ultrarealistic detail and breath-taking movie-like camera shot."]},{"id": 11,"type": "CLIPTextEncode","pos": [1158.2017822265625,287.580810546875],"size": [400,200],"flags": {},"order": 6,"mode": 0,"inputs": [{"name": "clip","type": "CLIP","link": 10}],"outputs": [{"name": "CONDITIONING","type": "CONDITIONING","links": [7]}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "CLIPTextEncode"},"widgets_values": ["Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"]},{"id": 3,"type": "LoadImage","pos": [740.4271850585938,669.6016845703125],"size": [315,314],"flags": {},"order": 4,"mode": 0,"inputs": [],"outputs": [{"name": "IMAGE","type": "IMAGE","links": [1,4]},{"name": "MASK","type": "MASK","links": null}],"properties": {"cnr_id": "comfy-core","ver": "0.3.27","Node name for S&R": "LoadImage"},"widgets_values": ["1.jpg","image",""]}],"links": [[1,3,0,4,1,"IMAGE"],[2,5,0,4,0,"CLIP_VISION"],[3,4,0,6,3,"CLIP_VISION_OUTPUT"],[4,3,0,6,4,"IMAGE"],[5,7,0,8,1,"VAE"],[6,8,0,9,0,"IMAGE"],[7,11,0,6,1,"CONDITIONING"],[8,10,0,6,0,"CONDITIONING"],[9,12,0,10,0,"CLIP"],[10,12,0,11,0,"CLIP"],[11,13,0,8,0,"LATENT"],[12,1,0,13,0,"MODEL"],[13,6,0,13,1,"CONDITIONING"],[14,6,1,13,2,"CONDITIONING"],[15,6,2,13,3,"LATENT"],[16,7,0,6,2,"VAE"]],"groups": [],"config": {},"extra": {"ds": {"scale": 0.6209213230591555,"offset": [-698.7441833037851,99.65105349022558]}},"version": 0.4
}
导入以上json工作流之后
五、开始运行
python main.py --listen --port 8998 --force-fp16 --tmo --fp16-vae #打开服务后
浏览器 http://0.0.0.0:8998
--tmo就可以用到mlu 不加则可以则是英伟达的用法
六、效果展示
### Loading: ComfyUI-Manager (V3.31.13)
[ComfyUI-Manager] network_mode: public
### ComfyUI Revision: UNKNOWN (The currently installed ComfyUI is not a Git repository)
/workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-Manager/glob/manager_util.py:139: DeprecationWarning: verify_ssl is deprecated, use ssl=False insteadasync with aiohttp.ClientSession(trust_env=True, connector=aiohttp.TCPConnector(verify_ssl=False)) as session:
------------------------------------------
Comfyroll Studio v1.76 : 175 Nodes Loaded
------------------------------------------
** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md
** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki
------------------------------------------
[ComfyUI-Easy-Use] server: v1.2.8 Loaded
[ComfyUI-Easy-Use] web root: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-Easy-Use/web_version/v2 Loaded
WAS Node Suite: OpenCV Python FFMPEG support is enabled
WAS Node Suite Warning: `ffmpeg_bin_path` is not set in `/workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/was-node-suite-comfyui/was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
WAS Node Suite: Finished. Loaded 220 nodes successfully."Be the change that you wish to see in the world." - Mahatma GandhiImport times for custom nodes:0.0 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/websocket_image_save.py0.0 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-Florence20.0 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI_essentials0.0 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/comfyui-custom-scripts0.1 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/comfyui-kjnodes0.1 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-Manager0.1 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI_Comfyroll_CustomNodes0.2 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/comfyui_segment_anything0.2 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-PuLID-Flux-master0.3 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/comfyui-tensorops0.5 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/comfyui_layerstyle0.7 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-Easy-Use2.4 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/was-node-suite-comfyui7.1 seconds: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/pulid_comfyui/torch/venv3/pytorch/lib/python3.10/site-packages/aiohttp/web_urldispatcher.py:202: DeprecationWarning: Bare functions are deprecated, use async oneswarnings.warn(
**************************enable torch_mlu_ops success****************************
Starting serverTo see the GUI go to: http://0.0.0.0:8090
To see the GUI go to: http://[::]:8090
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
/workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-Manager/glob/manager_util.py:139: DeprecationWarning: verify_ssl is deprecated, use ssl=False insteadasync with aiohttp.ClientSession(trust_env=True, connector=aiohttp.TCPConnector(verify_ssl=False)) as session:
FETCH ComfyRegistry Data: 5/84
FETCH ComfyRegistry Data: 10/84
FETCH ComfyRegistry Data: 15/84
FETCH ComfyRegistry Data: 20/84
FETCH ComfyRegistry Data: 25/84
FETCH ComfyRegistry Data: 30/84
FETCH ComfyRegistry Data: 35/84
Cannot connect to comfyregistry.
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json[ComfyUI-Manager] Due to a network error, switching to local mode.
=> custom-node-list.json
=>
FETCH DATA from: /workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/custom_nodes/ComfyUI-Manager/custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: mlu:0, offload device: cpu, dtype: torch.float16
Requested to load CLIPVisionModelProjection
loaded completely 41176.8 1208.09814453125 True
/torch/venv3/pytorch/lib/python3.10/site-packages/torch_mlu/utils/gpu_migration/migration.py:352: DeprecationWarning: at::autocast::is_privateuseone_enabled() is deprecated. Please use at::autocast::is_autocast_enabled(at::kPrivateUse1) instead. (Triggered internally at /opt/python/cp310-cp310/lib/python3.10/site-packages/torch/include/ATen/autocast_mode.h:125.)return fn(*args, **kwargs)
CLIP/text encoder model load device: mlu:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load WanTEModel
loaded completely 39876.3849609375 10835.4765625 True
/workspace/volume/guojunceshi2/yanshicomfyui/comfyui0330/ComfyUI-0.3.27/comfy/text_encoders/t5.py:114: UserWarning: Casting input of dtype int64 to int32, maybe overflow! (Triggered internally at /torch_mlu/torch_mlu/csrc/aten/utils/cnnl_util.cpp:128.)relative_position = torch.abs(relative_position)
Requested to load WanVAE
loaded completely 17607.60791015625 242.02829551696777 True
ezgif-7f484a9ed12e97