How Agent Skills Are Evaluated: The Verify AI Skills Methodology
Verify AI Skills evaluates each Agent Skill by reading its files at a pinned commit. It weighs six months of dated public evidence. An Agent Skill is a SKILL.md package that an AI agent loads for one job. Our guide to what an Agent Skill is covers the format. The rules below cover every label, score, safety grade and rank.
How a verdict is made:
- Pin the skill's commit and record the version under review.
- Read SKILL.md and every script, hook and reference file in the package.
- Gather dated evidence from issues, pull requests, independent evals, Hacker News and targeted searches.
- Score each measure, apply the published caps, then set the label and confidence.
- Check the verdict against the published rules, then pass it to a human editor before publishing.
How are AI agent skills evaluated on Verify AI Skills?
Verify AI Skills evaluates AI agent skills by reading each skill's files at a pinned commit. Those files are weighed against six months of dated public evidence. The verdict covers that pinned version only.
- Pinned version: a repository's default branch keeps moving, so each verdict covers one recorded commit.
- Files read: SKILL.md, scripts, hooks and references are opened and read.
- Code examples checked: API names and import paths are compared with current docs. A stale example counts as evidence.
Every current verdict dates from a review on 2026-09-30.
How does a file-and-evidence review differ from running skill evals?
A file-and-evidence review reads the skill and its public record, while an eval harness runs the skill on tasks. When a third party has measured a skill, as Tessl does, that result counts as evidence.
Eval harnesses run a skill against tasks and grade each trial. Anthropic's guide Demystifying evals for AI agents defines an eval task. It is "a single test with defined inputs and success criteria." Our review covers a different question: what the public record says about a skill in use.
| Eval harness | Our review |
|---|---|
| Runs the skill on tasks | Reads the files and the public record |
| Compares runs with and without the skill | Uses a third-party with/without result when one exists |
| Measures activation directly | Reports activation only when third-party evidence measured it |
| Grades each trial | Scores seven universal measures from evidence |
| Outputs a pass rate | Outputs a label, score, safety grade and confidence |
Effectiveness is capped at 85 without a measured result. High confidence needs a live benchmark, so no verdict holds it. Third-party runs still enter as evidence, such as Tessl evals and arXiv papers.
Which Agent Skills were selected, and what evidence counts?
We select Agent Skills from the agenticskills.io catalogue. Evidence counts when it is dated in the six months before review and links to a source we opened. Most evidence comes from GitHub.
The agenticskills.io catalogue is a public directory of Agent Skills. Our picks are mostly popular listed skills, not the long tail. The evidence window runs from 2026-03-30 to 2026-09-30.
Evidence comes from GitHub issues, pull requests and repositories, skills.sh, Tessl, Hacker News, arXiv and other web pages.
- One source counts as one item, so a single issue never becomes two lines.
- Listing pages that restate the publisher's description never count.
- Publisher claims sit in a separate list.
- Negative evidence is searched for on every skill.
Each item carries a paraphrase, a source URL, a month, a sentiment and a strength. Issue trackers over-represent problems, so the negative share is not a quality measure. A defect that was reported and fixed counts as a maintenance positive.
What does the safety grade mean?
The safety grade rates what a skill's files can do, from A to F. A means no executable code, and F means secrets, undisclosed endpoints, telemetry or unsafe behaviour. It is a file reading, not a penetration test.
| Safety grade | What the files show |
|---|---|
| A | No executable code |
| B | Scripts present, nothing alarming |
| C | Needs credentials or broad permissions |
| F | Secrets, undisclosed endpoints, telemetry or unsafe behaviour |
A C grade describes the access a skill asks for. It says nothing about intent. The grade covers the files at the pinned commit only.
How are verdict scores and labels set?
Verdict scores are set from seven universal measures and four to eight job-specific ones, weighted by the worst failure. Published caps limit scores without proof. The score band then sets the verdict label.
The seven universal measures are effectiveness, reliability, safety, efficiency, compatibility, maintainability and setup friction. Job measures include the worst failure, the cost of adoption and instruction weight against value. The overall score is weighted by the worst failure instead of averaged. Each measure uses the same anchors.
| Measure score | Anchor |
|---|---|
| 90+ | Consistent measured advantage, no notable failures |
| 75 to 89 | Clearly positive, minor trade-offs |
| 60 to 74 | Works with real costs or limits |
| 40 to 59 | Mixed |
| 20 to 39 | Mostly negative |
| 0 to 19 | Broken or unsafe |
A measure with no evidence stays blank, never a guessed low number.
Three published caps limit a score without proof:
- Effectiveness stays at 85 or below without a measured result.
- A writing skill that fabricates facts scores 50 or below. It applies to Landing Page Design, 50.
- A data-access tool that reports false success scores 60 or below. It applies to Skill Creator, 57.
| Verdict label | Score band and rule |
|---|---|
| Highly Recommended | 85+ with Strong evidence |
| Recommended | 75 to 84 |
| Recommended with caveats | 70 to 74, caveat stated |
| Conditional | 55 to 69 |
| Not Recommended | Below 55 |
| Insufficient Evidence | Fewer than four evidence items, any score |
What do confidence and verified mean in a verdict?
Confidence measures how much evidence sits behind a verdict, not how good the skill is. Verified means an evidence item shows the skill running in that named agent. A claim or a model harness does not count.
- Low confidence: thin evidence or documentation only.
- Moderate confidence: eight or more items, four or more of them strong.
- High confidence: a live benchmark of the skill. No verdict holds it.
An agent counts as verified when dated evidence shows the skill running in it. Claude Code, Codex and Cursor are examples. A publisher's compatibility claim stays a claim. A model run in a test harness is labelled "verified (model run)" and stays out of agent totals. A GPT model run in a "Codex-style client" is not a Codex run.
How are ranked skill lists and skill packs ordered?
Ranked skill lists are ordered by verdict score. Equal scores go by confidence, then safety grade, then name. A skill pack sits in a separate block and is never ranked beside its own member skills.
Our skills ranked by verdict follow three rules:
- Order rows by score, then break ties by confidence, safety grade and name.
- Rank installable pack members as their own rows. The pack sits in a Skill packs block when its members hold verdicts.
- Rank only skills whose main job is the hub's work, and list the rest as "also useful".
Two real ties apply the rule. Remotion Best Practices (77, Moderate) ranks above Animate (77, Low) on confidence. Emil Design Engineering ranks before Frontend Design on name; both score 76, Moderate, grade A. A bundle with no evaluated member ranks as a normal row. Designer Skills Collection and Remotion Best Practices are two such bundles.
Who writes the verdicts, and what checks run before publishing?
We use AI to collect evidence and write a first draft of each verdict from a fixed checklist. Automated checks run on every draft, and a failed check blocks it. A human editor reviews every verdict before it is published.
Each draft is based on the skill's files at a pinned commit and dated public evidence. The automated checks cover eight rules:
- Label matches the score band.
- Fewer than four evidence items force Insufficient Evidence.
- Moderate confidence needs eight or more items, with four or more strong.
- High confidence is rejected without a live benchmark.
- Every evidence date sits inside the window.
- Every cited URL returned HTTP 200 at check time.
- No source is counted twice.
- Promotional phrases such as "the best", "leading" and "most popular" are rejected.
What does a verdict not claim, and when is it re-reviewed?
A verdict does not claim a security audit, a certification, a runtime test or a guarantee for your setup. It is re-reviewed when a new release or a newly reported defect appears.
- Token and cost figures are estimates for a stated typical task.
- Material errors are corrected in place and recorded in the public corrections log.
- Each verdict covers its pinned commit and the evidence window, nothing later.
Common questions
Why does no skill reach Highly Recommended?
Highly Recommended needs 85+ with Strong evidence, and effectiveness caps at 85 without a measured result. The top score is 78.
Can a skill publisher change a verdict?
No. Publishers submit evidence and never approve a verdict, as our page on who can influence a verdict explains.
How can I evaluate an Agent Skill myself?
Readers can vet a skill with the same checks before installing it. Skill authors can follow the guide to build and test your own skill with their own evals.