hugging-face-evaluation

Overview

This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data:

• Extracting existing evaluation tables from README content
• Importing benchmark scores from Artificial Analysis
• Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai)

Detailed Guide

Read the detailed guide before executing this skill. It retains the complete procedure and reference material. Treat its safety, prerequisites, and validation requirements as mandatory. For focused work, load the relevant sections; for end-to-end work, read the guide completely.

When to Use

• You need to add structured evaluation results to a Hugging Face model card.
• You want to import benchmark data or run custom evaluations with vLLM, lighteval, or inspect-ai.
• You are preparing leaderboard-compatible model-index metadata for a model release.

Limitations

• Use this skill only when the task clearly matches the scope described above.
• Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
• Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.

Evaluation provenance and publication limits

Importing a number or extracting YAML is not running an evaluation, reproducing a benchmark or establishing leaderboard acceptance. Preserve model revision, task/dataset revision and split, prompt/chat template, shots, decoding parameters, metric definition/scale, date, evaluator version and original source. Never attach another hosted model’s scores to a different downloadable checkpoint merely because names resemble one another. Missing context remains unknown. Separate different benchmark datasets/results rather than putting unrelated scores under one dataset entry.

Check only relevant existing evaluation PRs for duplication; an unrelated open PR does not by itself block legitimate work. Prepare a local diff and publish only to the explicitly authorized repo/branch, preserving unrelated model-card metadata. The helper example ignores readme_content and lacks the described duplicate-PR check; do not treat it as a complete integration. Hardware size tables are rough leads, not memory guarantees, and the shown 70B Llama-3.2 identifier must be resolved against an actual model repository rather than invented. Bound GPU/API cost, execution time, data transfer and code-evaluation sandbox permissions. No dependencies installed, model loaded, job submitted, evaluation run or model card published here.