~/pushd/gh$pushd read huggingface/lightevalnot on the watchlist
huggingface/lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
stars
2,572
forks
593
watchers
25
open issues
464
created
26 Jan 2024
last push
58m ago
README
the first lines Your go-to toolkit for lightning-fast, flexible LLM evaluation, from Hugging Face's Leaderboard and Evals Team.
[](https://github.com/huggingface/lighteval/actions/workflows/tests.yaml?query=branch%3Amain)
[](https://github.com/huggingface/lighteval/actions/workflows/quality.yaml?query=branch%3Amain)
[](https://www.python.org/downloads/)
[](https://github.com/huggingface/lighteval/blob/main/LICENSE)
[](https://pypi.org/project/lighteval/)
Lighteval is your *all-in-one toolkit* for evaluating LLMs across multiple
backends-whether your model is being served somewhere or already loaded in memory.
Dive deep into your model's performance by saving and exploring *detailed,
sample-by-sample results* to debug and see how your models stack-up.
*Customization at your fingertips*: letting you either browse all our existing tasks and metrics or effortlessly create your own custom task and custom metric, tailored to your needs.
## Available Tasks
Lighteval supports 1000+ evaluation tasks across multiple domains and
Ask a scout
a deep read
DeepSeekDeepSeek . DeepSeek V4 Pro for the deep read
huggingface is not in a camp, so the read goes to DeepSeek, the open source scout.
10 of 10 public reads left today. Holders get 3 a day.
No read kept yet.A read is kept for a few hours, so the next visitor sees it without a new call.
Scout calls and heat are information, not advice. They describe public GitHub activity and are never a signal to trade.
Recent commits
default branch- 58mBump nltk from 3.10.3 to 3.10.4 (#1455) . Muhammad Naufal Mahdi
- 2htests: make harness metric comparison work with pytest 9.2 (#1456) . Nathan Habib
- 1dfix: exclude compromised LiteLLM 1.107.0 (#1445) . Guillaume Salou
- 23dBump doc-builder workflow pin (#1417) . Pauline Bailly-Masson
- 28dRun CI in prebuilt containers (pytorch / vllm nightly) and support vllm 0.30 / transformers 5 (#1403) . Nathan Habib
- 31dFix DetailsLogger hashing with xxhash 4.0 (#1402) . Nathan Habib
- 36dPin GitHub Actions to commit SHAs (#1394) . hf-security-analysis[bot]
- 36dScope GITHUB_TOKEN permissions per job (#1393) . hf-security-analysis[bot]
- 42dchore: enable Dependabot weekly GitHub Actions bumps (#1246) . hf-dependantbot-rollout[bot]
- 50ddeps: bump nltk to 3.10.3 (#1380) . Muhammad Naufal Mahdi