Skip to content

Releases: open-compass/opencompass

OpenCompass v0.3.1

23 Aug 03:00
@MaiziXiao MaiziXiao
0.3.1
5485207
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.3.1

The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.1!

🌟 Highlights

  • 🚀 Support pip installation, update Readme and evaluation demo
  • 🐛 Fixed various dataset loading issues.
  • ⚙️ Enhanced auto-download features for datasets.

🚀 New Features

  • 🆕 Introduced support for Ruler datasets.
  • 🆕 Enhanced model compatibility.
  • 🆕 Improved dataset handling, support auto-download for various datasets

📖 Documentation

  • 📚 Updated README to reflect the latest changes.
  • 📚 Improved documentation for dataset loading procedures.

🐛 Bug Fixes

  • 🐞 Resolved modelscope dataset load issues.
  • 🐞 Corrected evaluation scores for the Lawbench dataset.
  • 🐞 Fixed dataset bugs for CommonsenseQA and Longbench.

⚙ Enhancements and Refactors

  • 🔧 Retained first and last halves of prompts to avoid max_seq_len issues.
  • 🔧 Updated Compassbench to v1.3.
  • 🔧 Switched to Python runner for single GPU operations.

🎉 Welcome New Contributors

  • 🙌 @Yunnglin for fixing modelscope dataset load problem.
  • 🙌 @changyeyu for addressing max_seq_len issues with prompt handling.
  • 🙌 @seetimee for updates to openai_api.py.
  • 🙌 @HariSeldon0 for adding the scicode dataset.

What's Changed

Full Changelog: 0.3.0...0.3.1

Thank you for your continued support and contributions to OpenCompass!

Contributors

tonysy, xu-song, and 9 other contributors
Assets 2
Loading

OpenCompass v0.3.0

06 Aug 17:34
@tonysy tonysy
0.3.0
264fd23
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.3.0

The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.0! This release brings a variety of new features, enhancements, and bug fixes to improve your experience.

🌟 Highlights

  1. Support for OpenAI ChatCompletion
  2. Updated Model Support List
  3. Support Dataset Automatic Download
  4. Support pip install opencompass

🚀 New Features

  1. Support for CompassBench Checklist Evaluation
  2. Adding support for Doubao API
  3. Support for ModelScope Datasets

📖 Documentation

  1. Update NeedleBench Docs
  2. Update Documentation

🐛 Bug Fixes

  1. Fix Typing and Typo
  2. Fix Lint Issues
  3. Fix Summary Error in subjective.py

⚙ Enhancements and Refactors

  1. Upgrade Default Math pred_postprocessor
  2. Fix Path and Folder Updates
  3. Update Get Data Path for LCBench and HumanEval

🔗 Full Change Logs

🎉 Welcome New Contributors

Full Changelog: 0.2.6...0.3.0

Contributors

tonysy, wangxingjun778, and 16 other contributors
Assets 2
Loading

OpenCompass v0.2.6

05 Jul 16:36
@Leymore Leymore
0.2.6
a62c613
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.6

The OpenCompass team is thrilled to announce the release of OpenCompass v0.2.6!

🌟 Highlights

  • No noteworthy highlights.

🚀 New Features

  1. #1215 #1224 #1266 Add Datasets MT-Bench-101, Fofo, wildbench
  2. #1286 Add Models InternLM2.5-7B

📖 Documentation

  1. #1252 Add doc for accelerator function
  2. #1263 Update quick start guide

🐛 Bug Fixes

  1. #1221 Resolve release version installation and import issues
  2. #1228 Fix pip version issues
  3. #1282 Update MathBench summarizer & fix cot setting

⚙ Enhancements and Refactors

  1. #1284 Reorganize subjective eval

🎉 Welcome New Contributors

🔗 Full Change Logs

Full Changelog: 0.2.5...0.2.6

Contributors

Zor-X-L, xingyuanbu, and 8 other contributors
Assets 2
Loading

OpenCompass v0.2.5

29 May 16:35
@Leymore Leymore
0.2.5
a77b8a5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.5

The OpenCompass team is thrilled to announce the release of OpenCompass v0.2.5!

🌟 Highlights

  • Simplify the huggingface / vllm / lmdeploy model wrapper. meta_template is no longer needed to be hand-crafted in model configs
  • Introduce evaluation results README in ~20 dataset config folders.

🚀 New Features

  1. #1065 Add LLaMA-3 Series Configs
  2. #1048 Add TheoremQA with 5-shot
  3. #1094 Support Math evaluation via judgemodel
  4. #1080 Add gpqa prompt from simple_evals, openai
  5. #1074 Add mmlu prompt from simple_evals, openai
  6. #1123 Add Qwen1.5 MoE 7b and Mixtral 8x22b model configs

📖 Documentation

  1. #1053 Update readme
  2. #1102 Update NeedleInAHaystack Docs
  3. #1110 Update README.md
  4. #1205 Remove --no-batch-padding and Use --hf-num-gpus

🐛 Bug Fixes

  1. #1036 Update setup.py install_requires
  2. #1051 Fixed the issue caused
  3. #1043 fix multiround
  4. #1070 Fix sequential runner
  5. #1079 Fix Llama-3 meta template

⚙ Enhancements and Refactors

  1. #1163 enable HuggingFacewithChatTemplate with --accelerator via cli
  2. #1104 fix prompt template
  3. #1109 Update performance of common benchmarks

🎉 Welcome New Contributors

🔗 Full Change Logs

Read more

Contributors

lvhan028, tonysy, and 24 other contributors
Assets 2
Loading

OpenCompass v0.2.5.rc1

23 Apr 09:21
@Leymore Leymore
0.2.5.rc1
81d0e4d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.5.rc1 Pre-release
Pre-release 
[Feature] Add lmdeploy tis python backend model (#1014)

* add lmdeploy tis python backend model

* fix pr check

* update
Assets 3
Loading

OpenCompass v0.2.4

09 Apr 10:06
@liushz liushz
0.2.4
b39f501
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.4

The OpenCompass team is thrilled to announce the release of OpenCompass v0.2.4!

🌟 Highlights

  • Enhanced support for multiple datasets including QuALITY, APPS and TACO.
  • Introducing multi-model judging for subjective test.
  • Bug fixes and improvements in configurations and documentation.

🚀 New Features

🌐 General

  1. Feat #963 - Support for APPS dataset.
  2. Feature #976 - Add the implementation of QuALITY datasets.
  3. Feature #984 - Add support for setting prediction paths.
  4. Feature #1006 - Support alpacaeval_v2.
  5. Feature #1016 - Add multi-model judge.
  6. Feature #1019 - Add ATC Choice Version.

📖 Documentation

  1. Updates docs #1015 - General documentation updates and improvements.

🐛 Bug Fixes

  1. Fix #964 - Fix the config's name of deepseek-coder.
  2. Fix #890 - Update links and link checkers.
  3. Fix #977 - Fix a bug in internlm2 series configs.
  4. Fix #975 - Fix documentation issues.
  5. Fix #992 - Fix running issues in turbomind_tis.
  6. Fix #994 - Change status to list in base.py.
  7. Fix #995, Fix #1020 - Quick fixes and refactors for configs.

⚙ Enhancements and Refactors

  1. Modify requirements/runtime.txt #983 - Update numpy version requirement.
  2. Update Needlebench and configs #986 - Enhancements in Needlebench configurations.
  3. Simplify needlebench summarizer #1024 - Streamline Needlebench summarizer for better efficiency.

🎉 Welcome New Contributors

🔗 Full Change Logs

[Fix] fix the config's name of deepseek-coder by @jingmingzhuo in #964
[Fix] Update links and link checkers by @Leymore in #890
[Feat] support apps by @Connor-Shen in #963
fix doc problem by @seanzhang-zhichen in #975
[Fix] fix a bug in internlm2 series configs by @jingmingzhuo in #977
[Feature] Add the implement of QuALITY datasets by @jingmingzhuo in #976
modify the requirements/runtime.txt: numpy==1.23.4 --> numpy>=1.23.4 by @kleinzcy in #983
[Feature] add support for set prediction path by @bittersweet1999 in #984
[Feat] Support TACO by @Connor-Shen in #966
[Feature] update apps by @Connor-Shen in #985
[Fix] update apps/taco by @Connor-Shen in #988
[Feature] add one script for subjective by @bittersweet1999 in #993
Fix running issues in turbomind_tis by @ispobock in #992
[Fix] base.py change status into list by @Chaseldot in #994
[Fix] quick fix for configs by @bittersweet1999 in #995
[Feature] update needlebench and configs by @DseidLi in #986
[Feature] support alpacaeval_v2 by @bittersweet1999 in #1006
updates docs by @Y0oMu in #1015
[Feature] Add multi-model judge and fix some problems by @bittersweet1999 in #1016
[Fix] Refactor Needlebench Configs for CLI Testing Support by @DseidLi in #1020
[Feature] Add ATC Choice Version by @DseidLi in #1019
[Fix] Simplify needlebench summarizer by @DseidLi in #1024

For a detailed overview of all changes, check out our Full Changelog.

Contributors

Leymore, ispobock, and 8 other contributors
Assets 2
Loading

OpenCompass v0.2.4.rc1

25 Mar 10:15
@Leymore Leymore
0.2.4.rc1
0a6a03f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.4.rc1 Pre-release
Pre-release

Provide with more parsed datasets:

OpenCompassData-complete-20240325.zip

Important updates compared to previous version are as follow:

Subjective: Add MTBench
LongText: Support Needle-In-Haystack Test Dataset
Code: Update generation version of CIBench

Assets 5
Loading

OpenCompass v0.2.3

12 Mar 03:53
@Leymore Leymore
0.2.3
ab6cdb2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.3

The OpenCompass team is thrilled to announce the release of OpenCompass v0.2.3! This version is packed with new features, crucial fixes, and documentation updates to improve your experience. We're continuously working to enhance OpenCompass, making it more robust and versatile for all users.

🌟 Highlights:

  • Enhanced Model Support: Introduction of new models and configurations, including support for the LightllmApi, lmdeploy pytorch engine, and more.
  • New Datasets and Benchmarks: Expanding our dataset repository with additions like OpenFinData, lveval benchmark, and an upgrade to Needlebench.
  • Documentation and Sync Improvements: Updated dataset pack URLs, fixed documentation errors, and synchronized with internal codes for consistency.

Explore the key updates in this release:

🌟 New Features:

  • 📦 Dataset and Benchmark Expansion:

    • Support for new datasets like OpenFinData and an upgrade to Needlebench, offering broader evaluation capabilities (#896, #913).
    • Introduction of the lveval benchmark to enrich the evaluation landscape (#914).

  • 🛠 Model and API Integrations:

    • Enhanced functionality with support for LightllmApi input_format and prompt templates, alongside the introduction of get_ppl for TurbomindModel (#888, #878).
    • New model configurations added, including support for gemini and deepseek-coder, further broadening the tools available for users (#931, #943).

  • 📖 Documentation and Sync Updates:

    • Updated dataset pack URLs and rank link in README to ensure users have access to the latest resources (#922, #911).
    • Several syncs with internal codes and GitHub blacklist update to maintain consistency and integrity (#929, #953).

🐛 Bug Fixes:

  • Addressed various configuration and template issues to ensure smoother operation across different models and benchmarks (#894, #893).
  • Fixed issues related to IFEval, including type hints and config bugs, enhancing evaluation accuracy and functionality (#906, #915).

🎉 Welcome New Contributors:

🔗 Full Changelog

For a detailed overview of all changes, check out our Full Changelog.

Contributors

x22x22, tonysy, and 10 other contributors
Assets 2
Loading

OpenCompass v0.2.2

08 Feb 06:04
@Leymore Leymore
0.2.2
08133e0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.2

Welcome to OpenCompass v0.2.2, a release brimming with new features, essential fixes, and significant improvements across the board. With a focus on enhancing functionality and expanding dataset support, this update underscores our commitment to providing a robust platform for our users.

🌟 Highlights:

  • Broadened Dataset Support: Introduction of diverse datasets like T-Eval, CIBench, IFEval, and NPHardEval, and more, broadening the horizons for research and evaluation.
  • API Integrations and Updates: New support for APIs like Nanbeige and updates to existing ones such as Zhipu and Sensetime, enhancing model interaction capabilities.
  • Dataset Collection Release: Integrated dataset collection is availabe in 0.2.2.rc1. Dataset used in OpenCompass 2.0 leaderboard is NOT included in this collection.

Dive into what's new and improved:

🌟 New Features:

  • 📦 Datasets Expansion:

    • Addition of multiple new datasets and evaluations, including T-Eval, CIBench, IFEval, and NPHardEval, offering more versatility for users (#813, #829, #809, #835).

  • 🛠 API and Model Enhancements:

    • Support for new APIs like Nanbeige and updates to enhance the functionality of existing ones (#786, #847, #834).
    • Configurations and support for models and evaluators have been improved and expanded (#791, #812, #845).

  • 📖 Documentation and CI Enhancements:

    • Updated FAQs, contribution guides, and added new test runners to improve CI/CD processes (#830, #751, #874).

🐛 Bug Fixes:

  • Various fixes have been applied to address issues across datasets, evaluators, and configurations, ensuring a smoother experience for all users (#787, #788, #789).

🎉 Welcome New Contributors:

🔗 Full Changelog

For a full list of updates, visit our Full Changelog.

Thank you to every contributor, old and new. Your dedication is shaping OpenCompass into a more robust and versatile tool. 🙌 🎉

Remember to star 🌟 our GitHub repository if OpenCompass aids your research and development! Your support and feedback are crucial for our continuous improvement.

Contributors

tonysy, QipengGuo, and 17 other contributors
Assets 2
Loading

OpenCompass v0.2.2.rc1

06 Feb 17:33
@Leymore Leymore
0.2.2.rc1
e257254
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.2.rc1 Pre-release
Pre-release

Provide with more parsed datasets:

OpenCompassData-core-20240207.zip
OpenCompassData-complete-20240207.zip

Important updates compared to previous version are as follow:

  • Subjective: Add AlignBench, MTBench
  • Agent: Add T-Eval
  • Medicine: Add MedBench
  • Code: Add HumanEval-X, DS-1000
  • Finance: Add FinanceIQ
  • Law: Update LawBench Evaluation Assets

OpenCompassData-core-20240207.zip
AGIEval ARC BBH ceval CLUE cmmlu
commonsenseqa drop FewCLUE flores_first100 GAOKAO-BENCH gsm8k
hellaswag humaneval lambada LCSTS math mbpp
mmlu nq openbookqa piqa race siqa
strategyqa summedits SuperGLUE TheoremQA triviaqa tydiqa
winogrande xstory_cloze Xsum

OpenCompassData-complete-20240207.zip
AGIEval anli ARC BBH CDME ceval
cibench_dataset cleva clozeTest-maxmin CLUE CMB cmmlu
commonsenseqa commonsenseqa_cn crowspairs_cn drop ds1000_data FewCLUE
FinanceIQ flores200_dataset flores_first100 FunctionalMT game24 GAOKAO-BENCH
gpqa gsm8k hellaswag humaneval humaneval_cn humaneval_multipl-e
humanevalx HungarianExamMath InfiniteBench lambada lanQ lawbench
LCSTS math math401 mbpp mbpp_cn mbpp_plus
MedBench mmlu MNIST NPHardEval nq nq_cn
nq-open openbookqa piqa py150 qabench race
scibench siqa SQuAD2.0 strategyqa alignment_bench mtbench
summedits SuperGLUE svamp teval TheoremQA triviaqa
tydiqa winogrande xiezhi xlsum xstory_cloze Xsum
Assets 4
Loading