Localized reading guide
huggingface-community-evalsPurpose, usage and source
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
Source retrieved:
- Source repository
- huggingface/skills
- SKILL.md
skills/huggingface-community-evals/SKILL.md- Repository stars
- 11,241
- Repository updated
- 2026-10-30
- Categories
- Automation
- Last collected
License: Apache-2.0
Workflow excerpts are reading aids, not executable instructions from Codez. No third-party Skill was run to produce this page.
The purpose and workflow passages below are upstream material, organized and translated for reading. They are distinct from the separately labeled Codez editorial analysis.
What this Skill does
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
How the upstream workflow works
Core Workflow
Choose the evaluation framework. Use inspect-ai when you want explicit task control and inspect-native flows. Use lighteval when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks. Choose the inference backend.
Overview
This skill is for running evaluations against models on the Hugging Face Hub on local hardware. inspect-ai with local inference lighteval with local inference choosing between vllm, Hugging Face Transformers, and accelerate
Remote Execution Boundary
This skill intentionally stops at local execution and backend selection. If the user wants to: run these scripts on Hugging Face Jobs pick remote hardware
Upstream passage · L136 ↗Requirements and setup
Prerequisites
Prefer uv run for local execution. Set HF_TOKEN for gated/private models.
uv --version
printenv HF_TOKEN >/dev/null
nvidia-smiLimits and quality checks
No separate limits passage was selected from this file. This does not mean the Skill is safe for every input or environment. Review the complete source and validate your own result.
Start with a safe trial
- Check the supported host and the repository installation instructions. Keep the Skill directory and its referenced scripts, assets and documentation together; SKILL.md alone may omit dependencies.
- Give the agent your real input, desired output and acceptance criteria. Invocation syntax and available tools depend on the host; this site does not assume a universal install command.
- Try the source workflow in a disposable project or on a copy. Review operations that change files, remote services or billable resources before allowing them.
- Validate the output against the source quality checks and your own requirements. A successful trigger or a high repository Star count does not establish a correct result.
Read further in the source
Related Skills
Translation and scope
This reading guide selects passages from the pinned upstream SKILL.md. Translation is assisted. It is not a complete manual or a claim of tested compatibility. Read the original and its referenced files before use.
This introduction refers to the pinned version below; current repository activity and collection time are shown separately.
- Pinned source
- Read the original ↗
- Source retrieved
- Source commit
fabc634a893f62ad5c2039830397efa35fa62715- Source SHA-256
a97f1c703f55b72427453a76af858237e6392a447fcafa9eeb85f7ac67f0155d