当前位置：首页 > news >正文

大模型量化框架GPTQModel的基本使用方法

news 来源：原创 2025/5/17 6:35:33

接上一篇博客：AutoGPTQ报torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed的解决办法-CSDN博客

如果Llama factory量化一直报错，可以改用其他的量化框架，例如GPTQ：https://github.com/ModelCloud/GPTQMode

主要注意的是校准数据集的构造，可以参考这里：使用LLaMA-Factory进行模型量化_llamafactory 量化-CSDN博客

最简单的方法就是直接通过微调数据集构造校准数据集，然后再参考一下GPTQModel的主页例子。我把GitHub上的例子稍微修改了一下：

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig

model_id = "/home/admin90601/ProjectsQuYuNew/LLaMA-Factory/models/qwen2.5-32b_lora_sft"
quant_path = "/home/admin90601/ProjectsQuYuNew/QuantiModels/Qwen2.5-32B"

calibration_dataset = load_dataset(
    "json",
    data_files="private_train_quanti.json",
    split="train"
  ).select(range(3913))["text"]

print(len(calibration_dataset))

quant_config = QuantizeConfig(bits=4, group_size=128)

model = GPTQModel.load(model_id, quant_config)

# increase `batch_size` to match gpu/vram specs to speed up quantization
model.quantize(calibration_dataset, batch_size=2)

model.save(quant_path)

# test post-quant inference
model = GPTQModel.load(quant_path)
result = model.generate("Uncovering deep insights begins with")[0] # tokens
print(model.tokenizer.decode(result)) # string output

主要需要注意的就是load_dataset这个函数，其中第一个和第三个参数的说明可以看其源码注释：

    Args:

        path (`str`):
            Path or name of the dataset.

            - if `path` is a dataset repository on the HF hub (list all available datasets with [`huggingface_hub.list_datasets`])
              -> load the dataset from supported files in the repository (csv, json, parquet, etc.)
              e.g. `'username/dataset_name'`, a dataset repository on the HF hub containing the data files.

            - if `path` is a local directory
              -> load the dataset from supported files in the directory (csv, json, parquet, etc.)
              e.g. `'./path/to/directory/with/my/csv/data'`.

            - if `path` is the name of a dataset builder and `data_files` or `data_dir` is specified
              (available builders are "json", "csv", "parquet", "arrow", "text", "xml", "webdataset", "imagefolder", "audiofolder", "videofolder")
              -> load the dataset from the files in `data_files` or `data_dir`
              e.g. `'parquet'`.

            It can also point to a local dataset script but this is not recommended.
        name (`str`, *optional*):
            Defining the name of the dataset configuration.
        data_dir (`str`, *optional*):
            Defining the `data_dir` of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and `data_files` is `None`,
            the behavior is equal to passing `os.path.join(data_dir, **)` as `data_files` to reference all the files in a directory.

简单来说，当data_dir这个参数指定的时候，第一个参数path可以用来指定数据类型，如json，csv，text，xml等。相信大家看一下我给的例子就能明白。