在SCNet使用异构海光DCU 部署文心21B大模型报错HIP out of memory（未调通）
使用命令
vllm serve baidu/ERNIE-4.5-21B-A3B-Base-PT --tensor-parallel-size 4 --trust-remote-code --block-size 8 --max-model-len 4096 --gpu-memory-utilization 0.85 --dtype float --kv_cache_dtype fp8
报错：
torch.OutOfMemoryError: HIP out of memory. Tried to allocate 16.00 MiB. GPU 0 has a total capacity of 63.98 GiB of which 0 bytes is free. Of the allocated memory 63.61 GiB is allocated by PyTorch, and 896.00 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
现在用这条命令
vllm serve baidu/ERNIE-4.5-21B-A3B-Base-PT --gpu-memory-utilization 0.92 --tensor-parallel-size 2 --max-num-seqs 32 --max-model-len 2000 --tensor-parallel-size 2 --dtype float16报错
(VllmWorkerProcess pid=31673) INFO 10-15 15:16:56 [model_runner.py:1156] Model loading took 40.3319 GiB and 63.837660 seconds
INFO 10-15 15:16:56 [model_runner.py:1156] Model loading took 40.3348 GiB and 63.986810 seconds
(VllmWorkerProcess pid=31673) /usr/local/lib/python3.10/dist-packages/vllm/attention/backends/rocm_flash_attn.py:1025: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:627.)
(VllmWorkerProcess pid=31673) sub_out = torch.nn.functional.scaled_dot_product_attention(
/usr/local/lib/python3.10/dist-packages/vllm/attention/backends/rocm_flash_attn.py:1025: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:627.)sub_out = torch.nn.functional.scaled_dot_product_attention(
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks.
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] Traceback (most recent call last):
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
ERROR 10-15 15:17:04 [engine.py:453] expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
ERROR 10-15 15:17:04 [engine.py:453] Traceback (most recent call last):
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 441, in run_mp_engine
ERROR 10-15 15:17:04 [engine.py:453] engine = MQLLMEngine.from_vllm_config(
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 133, in from_vllm_config
ERROR 10-15 15:17:04 [engine.py:453] return cls(
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 87, in __init__
ERROR 10-15 15:17:04 [engine.py:453] self.engine = LLMEngine(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 280, in __init__
ERROR 10-15 15:17:04 [engine.py:453] self._initialize_kv_caches()
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 427, in _initialize_kv_caches
ERROR 10-15 15:17:04 [engine.py:453] self.model_executor.determine_num_available_blocks())
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 104, in determine_num_available_blocks
ERROR 10-15 15:17:04 [engine.py:453] results = self.collective_rpc("determine_num_available_blocks")
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 334, in collective_rpc
ERROR 10-15 15:17:04 [engine.py:453] return self._run_workers(method, *args, **(kwargs or {}))
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
ERROR 10-15 15:17:04 [engine.py:453] driver_worker_output = run_method(self.driver_worker, sent_method,
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2595, in run_method
ERROR 10-15 15:17:04 [engine.py:453] return func(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 10-15 15:17:04 [engine.py:453] return func(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 249, in determine_num_available_blocks
ERROR 10-15 15:17:04 [engine.py:453] self.model_runner.profile_run()
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 10-15 15:17:04 [engine.py:453] return func(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1253, in profile_run
ERROR 10-15 15:17:04 [engine.py:453] self._dummy_run(max_num_batched_tokens, max_num_seqs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1379, in _dummy_run
ERROR 10-15 15:17:04 [engine.py:453] self.execute_model(model_input, kv_caches, intermediate_tensors)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 10-15 15:17:04 [engine.py:453] return func(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1796, in execute_model
ERROR 10-15 15:17:04 [engine.py:453] hidden_or_intermediate_states = model_executable(
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 172, in __call__
ERROR 10-15 15:17:04 [engine.py:453] return self.forward(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/transformers.py", line 422, in forward
ERROR 10-15 15:17:04 [engine.py:453] model_output = self.model(input_ids, positions, intermediate_tensors,
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
ERROR 10-15 15:17:04 [engine.py:453] return self._call_impl(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
ERROR 10-15 15:17:04 [engine.py:453] return forward_call(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/transformers.py", line 329, in forward
ERROR 10-15 15:17:04 [engine.py:453] hidden_states = self.model(
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
ERROR 10-15 15:17:04 [engine.py:453] return self._call_impl(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
ERROR 10-15 15:17:04 [engine.py:453] return forward_call(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py", line 1064, in wrapper
ERROR 10-15 15:17:04 [engine.py:453] outputs = func(self, *args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 558, in forward
ERROR 10-15 15:17:04 [engine.py:453] hidden_states = decoder_layer(
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_layers.py", line 94, in __call__
ERROR 10-15 15:17:04 [engine.py:453] return super().__call__(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
ERROR 10-15 15:17:04 [engine.py:453] return self._call_impl(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
ERROR 10-15 15:17:04 [engine.py:453] return forward_call(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
ERROR 10-15 15:17:04 [engine.py:453] return func(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 459, in forward
ERROR 10-15 15:17:04 [engine.py:453] hidden_states = self.mlp(hidden_states)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
ERROR 10-15 15:17:04 [engine.py:453] return self._call_impl(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
ERROR 10-15 15:17:04 [engine.py:453] return forward_call(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 344, in forward
ERROR 10-15 15:17:04 [engine.py:453] router_logits = self.gate(hidden_states.float())
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
ERROR 10-15 15:17:04 [engine.py:453] return self._call_impl(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
ERROR 10-15 15:17:04 [engine.py:453] return forward_call(*args, **kwargs)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 352, in forward
ERROR 10-15 15:17:04 [engine.py:453] output = self.quant_method.apply(self, x, bias)
ERROR 10-15 15:17:04 [engine.py:453] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 222, in apply
ERROR 10-15 15:17:04 [engine.py:453] return dispatch_unquantized_gemm()(x, layer.weight, bias)
ERROR 10-15 15:17:04 [engine.py:453] RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2595, in run_method
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 249, in determine_num_available_blocks
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] self.model_runner.profile_run()
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1253, in profile_run
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] self._dummy_run(max_num_batched_tokens, max_num_seqs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1379, in _dummy_run
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] self.execute_model(model_input, kv_caches, intermediate_tensors)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1796, in execute_model
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] hidden_or_intermediate_states = model_executable(
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 172, in __call__
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return self.forward(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/transformers.py", line 422, in forward
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/transformers.py", line 329, in forward
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] hidden_states = self.model(
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py", line 1064, in wrapper
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] outputs = func(self, *args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 558, in forward
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] hidden_states = decoder_layer(
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_layers.py", line 94, in __call__
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return super().__call__(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 459, in forward
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] hidden_states = self.mlp(hidden_states)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 344, in forward
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] router_logits = self.gate(hidden_states.float())
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 352, in forward
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] output = self.quant_method.apply(self, x, bias)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 222, in apply
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] return dispatch_unquantized_gemm()(x, layer.weight, bias)
(VllmWorkerProcess pid=31673) ERROR 10-15 15:17:04 [multiproc_worker_utils.py:238] RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
INFO 10-15 15:17:04 [multiproc_worker_utils.py:124] Killing local vLLM worker processes
Process SpawnProcess-1:
Traceback (most recent call last):File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrapself.run()File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in runself._target(*self._args, **self._kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 455, in run_mp_engineraise eFile "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 441, in run_mp_engineengine = MQLLMEngine.from_vllm_config(File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 133, in from_vllm_configreturn cls(File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 87, in __init__self.engine = LLMEngine(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 280, in __init__self._initialize_kv_caches()File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 427, in _initialize_kv_cachesself.model_executor.determine_num_available_blocks())File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 104, in determine_num_available_blocksresults = self.collective_rpc("determine_num_available_blocks")File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 334, in collective_rpcreturn self._run_workers(method, *args, **(kwargs or {}))File "/usr/local/lib/python3.10/dist-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workersdriver_worker_output = run_method(self.driver_worker, sent_method,File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2595, in run_methodreturn func(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_contextreturn func(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 249, in determine_num_available_blocksself.model_runner.profile_run()File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_contextreturn func(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1253, in profile_runself._dummy_run(max_num_batched_tokens, max_num_seqs)File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1379, in _dummy_runself.execute_model(model_input, kv_caches, intermediate_tensors)File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_contextreturn func(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1796, in execute_modelhidden_or_intermediate_states = model_executable(File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 172, in __call__return self.forward(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/transformers.py", line 422, in forwardmodel_output = self.model(input_ids, positions, intermediate_tensors,File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_implreturn forward_call(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/transformers.py", line 329, in forwardhidden_states = self.model(File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_implreturn forward_call(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py", line 1064, in wrapperoutputs = func(self, *args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 558, in forwardhidden_states = decoder_layer(File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_layers.py", line 94, in __call__return super().__call__(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_implreturn forward_call(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/utils/deprecation.py", line 172, in wrapped_funcreturn func(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 459, in forwardhidden_states = self.mlp(hidden_states)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_implreturn forward_call(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py", line 344, in forwardrouter_logits = self.gate(hidden_states.float())File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_implreturn forward_call(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 352, in forwardoutput = self.quant_method.apply(self, x, bias)File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 222, in applyreturn dispatch_unquantized_gemm()(x, layer.weight, bias)
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
Traceback (most recent call last):File "/usr/local/bin/vllm", line 8, in <module>sys.exit(main())File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 53, in mainargs.dispatch_function(args)File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 27, in cmduvloop.run(run_server(args))File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 82, in runreturn loop.run_until_complete(wrapper())File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_completeFile "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 61, in wrapperreturn await mainFile "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_serverasync with build_async_engine_client(args) as engine_client:File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__return await anext(self.gen)File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_clientasync with build_async_engine_client_from_engine_args(File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__return await anext(self.gen)File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_argsraise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
root@notebook-1978259446016311297-ac7sc1ejvp-24335:/# /usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdownwarnings.warn('resource_tracker: There appear to be %d '
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdownwarnings.warn('resource_tracker: There appear to be %d 'root@notebook-1978259446016311297-ac7sc1ejvp-24335:/# echo $HIP_VISIBLE_DEVICES改成bfloat试试
vllm serve baidu/ERNIE-4.5-21B-A3B-Base-PT --gpu-memory-utilization 0.92 --tensor-parallel-size 2 --max-num-seqs 32 --max-model-len 2000 --tensor-parallel-size 2 --dtype bfloat16现在出来了新的报错：
(VllmWorkerProcess pid=2441) INFO 10-15 15:23:03 [model_runner.py:1156] Model loading took 40.3319 GiB and 24.727362 seconds
/usr/local/lib/python3.10/dist-packages/vllm/attention/backends/rocm_flash_attn.py:1025: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:627.)
sub_out = torch.nn.functional.scaled_dot_product_attention(
(VllmWorkerProcess pid=2441) /usr/local/lib/python3.10/dist-packages/vllm/attention/backends/rocm_flash_attn.py:1025: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:627.)
用deepseek 7b模型试试
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --dtype float16 --trust-remote-code --gpu-memory-utilization 0.98 --tensor-parallel-size 2 --max-num-seqs 32 --max-model-len 40000你看看，人家这就好好的啊
INFO 10-15 15:38:13 [serving_completion.py:61] Using default completion sampling params from model: {'temperature': 0.6, 'top_p': 0.95}
INFO 10-15 15:38:13 [api_server.py:1090] Starting vLLM API server on http://0.0.0.0:8000
INFO 10-15 15:38:13 [launcher.py:28] Available routes are:
INFO 10-15 15:38:13 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD显存占到90% ,这样看，是不是2卡不行，需要4卡啊，问题是我四卡也没跑成啊！
没搞定，1卡 2卡 4卡都没跑通....