OpenCompass模型评估

OpenCompass面向大模型的开源方和使用者， 提供开源、高效、全面的大模型评测开放平台。

一、OpenCompass文档

1.基础安装
使用Conda准备 OpenCompass 运行环境：

conda create --name opencompass python=3.10 -y
conda activate opencompass

2. 安装 OpenCompass

git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .

root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace# conda create --name opencompass python=3.10 -y
root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace# conda activate opencompass
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace# cd opencompass
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# pip install -e .
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# pip install -e .
### 报错如下
Looking in indexes: https://mirrors.cloud.aliyuncs.com/pypi/simple
Obtaining file:///mnt/workspace/opencompassPreparing metadata (setup.py) ... errorerror: subprocess-exited-with-error× python setup.py egg_info did not run successfully.│ exit code: 1╰─> [56 lines of output]/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.ymlwarnings.warn(/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.!!********************************************************************************Requirements should be satisfied by a PEP 517 installer.If you are using pip, you can try `pip install --use-pep517`.********************************************************************************!!dist.fetch_build_eggs(dist.setup_requires)WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/ERROR: Could not find a version that satisfies the requirement nltk==3.8 (from versions: none)ERROR: No matching distribution found for nltk==3.8Traceback (most recent call last):File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/installer.py", line 107, in _fetch_build_egg_no_warnsubprocess.check_call(cmd)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/subprocess.py", line 369, in check_callraise CalledProcessError(retcode, cmd)subprocess.CalledProcessError: Command '['/mnt/workspace/Anaconda3/envs/opencompass/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpvvttj4xe', '--quiet', 'nltk==3.8']' returned non-zero exit status 1.The above exception was the direct cause of the following exception:Traceback (most recent call last):File "<string>", line 2, in <module>File "<pip-setuptools-caller>", line 34, in <module>File "/mnt/workspace/opencompass/setup.py", line 164, in <module>do_setup()File "/mnt/workspace/opencompass/setup.py", line 108, in do_setupsetup(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py", line 116, in setup_install_setup_requires(attrs)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py", line 89, in _install_setup_requires_fetch_build_eggs(dist)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py", line 94, in _fetch_build_eggsdist.fetch_build_eggs(dist.setup_requires)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/dist.py", line 768, in fetch_build_eggsreturn _fetch_build_eggs(self, requires)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/installer.py", line 44, in _fetch_build_eggsresolved_dists = pkg_resources.working_set.resolve(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 893, in resolvedist = self._resolve_dist(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 929, in _resolve_distdist = best[req.key] = env.best_match(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1267, in best_matchreturn self.obtain(req, installer)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1303, in obtainreturn installer(requirement) if installer else NoneFile "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/installer.py", line 109, in _fetch_build_egg_no_warnraise DistutilsError(str(e)) from edistutils.errors.DistutilsError: Command '['/mnt/workspace/Anaconda3/envs/opencompass/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpvvttj4xe', '--quiet', 'nltk==3.8']' returned non-zero exit status 1.[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed× Encountered error while generating package metadata.
╰─> See above for output.note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
改用如下安装成功
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# pip install -r /mnt/workspace/opencompass/requirements/runtime.txt

3.数据集准备
在 OpenCompass 项目根目录下运行下面命令，将数据集准备至 ${OpenCompass}/data 目录下：

wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip

4.配置评估任务
命令行（自定义HF模型）
对于 HuggingFace 模型，用户可以通过命令行直接设置模型参数，无需额外的配置文件。例如，对于
internlm/internlm2-chat-1_8b 模型，可以使用以下命令进行评估：
命令行（自定义hf类型）

windows环境好像不太行

 python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen  --hf-type chat --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\model\Qwen\Qwen1___5-0___5B-Chat" --debug

 python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo07\models\Qwen\Qwen2___5-1___5B-Instruct" --debug

报错如下：

signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
04/11 15:38:55 - OpenCompass - INFO - Loading demo_gsm8k_chat_gen: D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\configs\./datasets\demo\demo_gsm8k_chat_gen.py
04/11 15:38:55 - OpenCompass - INFO - Loading demo_math_chat_gen: D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\configs\./datasets\demo\demo_math_chat_gen.py
04/11 15:38:55 - OpenCompass - INFO - Loading example: D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\configs\./summarizers\example.py
04/11 15:38:55 - OpenCompass - INFO - Current exp folder: outputs\default\20250411_153855
04/11 15:38:56 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
demo_gsm8k train 7473
demo_gsm8k test 1319
demo_math test 5000
demo_math train 5000
04/11 15:38:59 - OpenCompass - INFO - Partitioned into 1 tasks.
Traceback (most recent call last):File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\run.py", line 4, in <module>main()File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\cli\main.py", line 339, in mainrunner(tasks)File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\runners\base.py", line 38, in __call__status = self.launch(tasks)File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\runners\local.py", line 102, in launchassert len(all_gpu_ids) >= num_gpus
AssertionError

AssertionError: assert len(all_gpu_ids) >= num_gpus。这表明系统检测到的可用GPU数量不足以满足任务所需的GPU数量。

安装cuda

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

 python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo07\models\Qwen\Qwen2___5-1___5B-Instruct" --debug --hf-num-gpus 1  #指定使用gpu的数量

或者直接使用cpu

 python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo07\models\Qwen\Qwen2___5-1___5B-Instruct" --debug --hf-num-gpus 0

运行成功以后有outputs文件

没有结果，不知道什么原因

日志如下：

D:\envs\opencompass\python.exe: can't open file 'D:\\Program': [Errno 2] No such file or directory
D:\envs\opencompass\python.exe: can't open file 'D:\\Program': [Errno 2] No such file or directory
D:\envs\opencompass\python.exe: can't open file 'D:\\Program': [Errno 2] No such file or directory

但是不管怎么改都是一样的问题，所以改用linux环境

linux 环境没问题

(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen  --hf-type chat --hf-path "/mnt/workspace/llm/Qwen/Qwen1.5-0.5B-Chat" --debug

正常启动

有正常的outputs文件输出

有评测结果

请注意，通过这种方式，OpenCompass 一次只评估一个模型，而其他方式可以一次评估多个模型。
命令行
用户可以使用 --models 和 --datasets 结合想测试的模型和数据集。

models 名为以下两个文件名，不然会报错

python run.py --models hf_qwen2_5_0_5b_instruct hf_qwen1_5_0_5b_chat --datasets demo_gsm8k_chat_gen demo_math_chat_gen --debug

在文件opencompass/opencompass/configs/models/qwen2_5或者qwen


最后看到如下得分:opencompass/outputs/default/20250411_174951/summary/summary_20250411_174951.md

模型和数据集的配置文件预存于 configs/models 和 configs/datasets 中。用户可以使用
tools/list_configs.py 查看或过滤当前可用的模型和数据集配置。

# 列出所有配置
python tools/list_configs.py
# 列出与llama和mmlu相关的所有配置
python tools/list_configs.py llama mmlu

运行 python tools/list_configs.py llama mmlu 将产生如下输出：

+-----------------+-----------------------------------+
| Model | Config Path |
|-----------------+-----------------------------------|
| hf_llama2_13b | configs/models/hf_llama2_13b.py |
| hf_llama2_70b | configs/models/hf_llama2_70b.py |
| ... | ... |
+-----------------+-----------------------------------+
+-------------------+---------------------------------------------------+
| Dataset | Config Path |
|-------------------+---------------------------------------------------|
| cmmlu_gen | configs/datasets/cmmlu/cmmlu_gen.py |
| cmmlu_gen_ffe7c0 | configs/datasets/cmmlu/cmmlu_gen_ffe7c0.py |
| ... | ... |
+-------------------+---------------------------------------------------+

用户可以使用第一列中的名称作为 python run.py 中 --models 和 --datasets 的输入参数。对于数据集，同一名称的不同后缀通常表示其提示或评估方法不同。

二、配置文件

除了通过命令行配置实验外，OpenCompass 还允许用户在配置文件中编写实验的完整配置，并通过run.py 直接运行它。配置文件是以 Python 格式组织的，并且必须包括 datasets 和 models 字段。

from mmengine.config import read_basewith read_base():from .datasets.demo.demo_gsm8k_chat_gen import gsm8k_datasetsfrom .datasets.demo.demo_math_chat_gen import math_datasetsfrom .models.qwen.hf_qwen1_5_0_5b_chat import models as hf_qwen1_5_0_5b_chatfrom .models.qwen2_5.hf_qwen2_5_1_5b_instruct import models as hf_qwen2_5_1_5b_instructdatasets = gsm8k_datasets + math_datasets
models = hf_qwen1_5_0_5b_chat + hf_qwen2_5_1_5b_instruct

运行任务时，我们只需将配置文件的路径传递给 run.py ：

python run.py configs/eval_chat_demo.py --debug

三、自定义数据集

数据集格式

选择题 ( mcq )

对于选择 ( mcq ) 类型的数据，默认的字段如下：

• question : 表示选择题的题干
• A , B , C , …: 使用单个大写字母表示选项，个数不限定。默认只会从 A 开始，解析连续的字母作
  为选项。
• answer : 表示选择题的正确答案，其值必须是上述所选用的选项之一，如 A , B , C 等。
  对于非默认字段，我们都会进行读入，但默认不会使用。如需使用，则需要在 .meta.json 文件中进行
  指定。
  .jsonl 格式样例如下：

{"question": "165+833+650+615=", "A": "2258", "B": "2263", "C": "2281", "answer":
"B"}
{"question": "368+959+918+653+978=", "A": "3876", "B": "3878", "C": "3880",
"answer": "A"}
{"question": "776+208+589+882+571+996+515+726=", "A": "5213", "B": "5263", "C":
"5383", "answer": "B"}
{"question": "803+862+815+100+409+758+262+169=", "A": "4098", "B": "4128", "C":
"4178", "answer": "C"}

• .csv 格式样例如下:

question,A,B,C,answer
127+545+588+620+556+199=,2632,2635,2645,B
735+603+102+335+605=,2376,2380,2410,B
506+346+920+451+910+142+659+850=,4766,4774,4784,C
504+811+870+445=,2615,2630,2750,B

问答题 ( qa )

对于问答 ( qa ) 类型的数据，默认的字段如下：

• question : 表示问答题的题干
• answer : 表示问答题的正确答案。可缺失，表示该数据集无正确答案。
  对于非默认字段，我们都会进行读入，但默认不会使用。如需使用，则需要在 .meta.json 文件中进行
  指定。
  .json格式样例如下：

{"question": "752+361+181+933+235+986=", "answer": "3448"}
{"question": "712+165+223+711=", "answer": "1811"}
{"question": "921+975+888+539=", "answer": "3323"}
{"question": "752+321+388+643+568+982+468+397=", "answer": "4519"}

.csv 格式样例如下:

question,answer
123+147+874+850+915+163+291+604=,3967
149+646+241+898+822+386=,3142
332+424+582+962+735+798+653+214=,4700
649+215+412+495+220+738+989+452=,4170

6.命令行列表
自定义数据集可直接通过命令行来调用开始评测。

python run.py --models hf_llama2_7b --custom-dataset-path xxx/test_mcq.csv --custom-dataset-data-type mcq --custom-dataset-infer-method ppl

python run.py \
--models hf_llama2_7b \
--custom-dataset-path xxx/test_qa.jsonl \
--custom-dataset-data-type qa \
--custom-dataset-infer-method gen

在绝大多数情况下， --custom-dataset-data-type 和 --custom-dataset-infer-method 可以省略，

• OpenCompass 会根据以下逻辑进行设置：
  如果从数据集文件中可以解析出选项，如 A , B , C 等，则认定该数据集为 mcq ，否则认定为
  qa 。
• 默认 infer_method 为 gen 。