Semgrep Rule Creator skill review: verdict, safety and alternatives

Semgrep Rule Creator is Trail of Bits' Agent Skill for writing one tested, custom Semgrep rule at a time. Our verdict is Recommended, 76/100, at Low confidence, with safety grade A.

Get the skill: semgrep-rule-creator folder on GitHub · SKILL.md at the reviewed commit · skills.sh listing

What is the Semgrep Rule Creator skill?

The Semgrep Rule Creator skill is Trail of Bits' instruction-only Agent Skill for writing one custom Semgrep rule. It writes tests first, reads the AST, and prefers taint mode. It loops semgrep --test until every case passes.

Semgrep is an open-source static analysis tool whose rules are written in YAML. Trail of Bits is a security firm that publishes this skill in its trailofbits/skills repository. An Agent Skill is a SKILL.md package that an agent loads on demand. Our page with Agent Skills explained covers the format. This skill holds no code. It is a playbook that tells the agent how to build and prove one rule.

The repo is the source to trust. Directory pages restate its description, and some list copies under other authors.

Is the Semgrep Rule Creator skill worth installing?

Yes, the Semgrep Rule Creator skill is worth installing for Semgrep users: 76/100, Recommended, Low confidence. A contributor benchmark passed every check in four of five runs. Each run fetches seven documentation pages, at about $0.48 per rule.

Its strength is discipline. It enforces ruleid and ok test annotations, a 100% semgrep --test pass, AST inspection and taint-first design. All seven doc links and every CLI command were current at the reviewed commit. Its weakness is cost. The mandatory fetch of seven external docs adds network dependence. Local references lack advanced taint features such as pattern-propagators.

It suits security engineers, auditors and developers who maintain custom Semgrep rule packs. The job is narrow: turn one described bug pattern into one rule and one test file. That test file must pass semgrep --test with no false positives on safe cases.

Skip it in two cases. It does not run existing rulesets or general scans, and it fails on offline or network-restricted agents.

Read our scoring method for how each measure works.

Reliability scores 76. We found no open bug reports against this skill. The only defect found, five dead doc links, was fixed in PR #179. Maintainability scores 80. The repository was active at review, with a push on 2026-09-28.

The score leans on the worst failure: a rule that passes its own tests but misses real variants. Compare it with the best Agent Skills ranked by verdict.

How many tokens does the Semgrep Rule Creator skill use?

The Semgrep Rule Creator SKILL.md is about 1,830 tokens, and its references add roughly 3,300 by our estimate. The mandatory doc fetches dominate: PR #322 measured about 944,000 billable context tokens per rule.

Our two folder figures differ because the measured count misses files linked through {baseDir}. We show both. At three rules a week, the doc fetches cost far more than the skill text. The hidden cost is network access: the skill fails or degrades without it.

What changed since our review?

PR #322 closed without merge on 2026-10-05. The skill folder has had no commits since 2026-09-16. Our verdict stays Recommended at 76.

• 2026-10-05: PR #322 closed unmerged. It proposed a shorter entrypoint with no mandatory doc fetch. The seven-fetch rule stays.
• 2026-09-16: last commit to the skill folder, 123037e (#310, ChatGPT import metadata). Our reviewed commit already includes it.
• 2026-10-21: SKILL.md is still 7,320 bytes, and plugin.json still says 1.2.6.
• 2026-10-21: stars rose from 7,319 to 7,661, and forks from 624 to 665.

Listings disagree on the version. Agenticskills.io shows "v1.7.0", while plugin.json says 1.2.6. We use 1.2.6. Our page on how we log re-reviews explains how we handle changes like this.

Is the Semgrep Rule Creator skill safe to install?

Yes, the Semgrep Rule Creator skill is safe to install, with safety grade A. It contains Markdown only, with no scripts, hooks or credentials. It does fetch seven fixed public documentation pages on every run.

The grade comes from reading the files at commit 82fe822. For the checks to run on any skill, see how to vet a skill before you install it. The files show:

• Allowed tools: Bash, Read, Write, Edit, Glob, Grep and WebFetch.
• Executable code: none in the folder.
• Commands run: the local semgrep CLI with --test, --validate, --dump-ast, --dataflow-traces and --config.
• Writes: exactly two files in a rule-id directory, the rule YAML and an annotated test file.
• Network: six semgrep-docs raw GitHub pages and one Trail of Bits Testing Handbook page.
• Status: all seven fetch URLs returned HTTP 200 on 2026-09-30.
• Prompt injection: no handling for fetched docs, but the URLs are fixed, not user-supplied.

The Testing Handbook is Trail of Bits' public guide to tools such as Semgrep, hosted at appsec.guide. On skills.sh, Gen Agent Trust Hub and Socket show Pass. Snyk shows Warn with no published detail. Safety scores 88.

What does the community report about the Semgrep Rule Creator skill?

The community reports about the Semgrep Rule Creator skill are thin but positive. Evidence confidence is Thin, with six items. The strongest are PR #322's benchmark and Semgrep staff praise in PR #179.

The split is three positive, one neutral and two negative. We found no independent negative reports in GitHub issues or web searches. The PR #322 benchmark is a contributor's work, not independently reproduced.

Tessl rated conciseness 4/5, with some redundancy. It scored distinctiveness 5/5 but description specificity 3/5. The description misses taint and SAST trigger terms.

What are the Semgrep Rule Creator skill's known failures?

The Semgrep Rule Creator skill has four known limits at commit 82fe822. It fetches seven external docs per run, and local references omit advanced taint features. One benchmark run missed held-out cases; semgrep --test can pass over zero tests.

1. Seven mandatory doc fetches run on every task. They drive cost and break offline use.
2. Local references lack pattern-propagators, labels and requires. Complex taint flows depend on the fetched docs.
3. One of five benchmark runs missed held-out cases, on Semgrep 1.156.0. That is the worst failure: false confidence in coverage.
4. A bot review on PR #322 notes semgrep --test can pass over zero graded tests. Nothing in the skill checks for that.

One older defect is fixed. Five of the seven doc links returned 404 after a Semgrep docs migration. PR #179 repaired them, merged on 2026-06-10.

Some gaps remain unknown. No published report covers rule quality in languages other than Python. None covers complex taint flows that need propagators, or runs with no network.

How does the Semgrep Rule Creator skill differ from Trail of Bits' other Semgrep tools?

The Semgrep Rule Creator skill writes one new rule with tests. Trail of Bits' static-analysis plugin runs Semgrep scans instead. The semgrep-rule-variant-creator ports existing rules to other languages, and the trailofbits ruleset holds ready-made rules.

Trail of Bits' own Semgrep Security Scan page routes custom rule work here. Its "When NOT to Use" list says to use the semgrep-rule-creator skill for that. The claudecowork.im listing writes the variant creator's skill name in a different form.

Two directories list plurigrid/asi under this name. Those listings are copies, not the Trail of Bits folder. GitHub code search counts 305 exact copies of the SKILL.md. Install from trailofbits/skills.

Which agents does the Semgrep Rule Creator skill work with?

The Semgrep Rule Creator skill is verified in Claude Code. It ships a Codex openai.yaml manifest, and PR #322 reports Codex loadability checks pass. No Codex rule-writing run is documented anywhere we checked.

Every agent needs three tools: the semgrep CLI, Bash and WebFetch. An agent without web fetch cannot finish the workflow. Compatibility scores 74. Automatic triggering is unknown: no evidence shows whether an agent picks the skill unprompted. Browse more Claude Code skills ranked by verdict.

How do you install the Semgrep Rule Creator skill?

Install the Semgrep Rule Creator skill with npx skills add https://github.com/trailofbits/skills --skill semgrep-rule-creator. Then install the semgrep CLI with uv or Homebrew. No API keys are needed, but network access is.

The skills.sh route installs the full folder:

npx skills add https://github.com/trailofbits/skills --skill semgrep-rule-creator

Agenticskills.io lists a shorter form of the same command:

npx skills add trailofbits/skills@semgrep-rule-creator

In Claude Code, the trailofbits/skills README adds the marketplace first:

/plugin marketplace add trailofbits/skills

Then install the semgrep-rule-creator plugin from that marketplace. The README lists a Codex route too. No published run confirms it:

codex plugin marketplace add trailofbits/skills

The skill needs a local semgrep binary. Install it with uv tool or Homebrew:

uv tool install semgrep
brew install semgrep

Avoid copying SKILL.md alone, as one directory's curl route does. The skill links its references through {baseDir}, so a lone file loses them. Our page on what a SKILL.md file contains explains that layout. For project folders and scopes, see the full Claude Code skills setup. Setup friction scores 80.

How do you use the Semgrep Rule Creator skill?

Use the Semgrep Rule Creator skill by describing one bug pattern, such as request input reaching os.system. It writes vulnerable and safe tests first, then drafts a taint rule. It reruns semgrep --test until every case passes.

The skill runs five steps:

1. Write the test file first, with ruleid lines for vulnerable code and ok lines for safe code.
2. Dump the AST with --dump-ast to see how Semgrep parses the target code.
3. Choose taint mode for data-flow bugs, with sources, sinks and sanitizers.
4. Loop semgrep --test until every annotated case passes.
5. Run --validate on the final YAML.

Taint mode tracks untrusted data from a source to a sink, unless a sanitizer cleans it first. The AST is the parsed tree of the code that Semgrep patterns match against. The ruleid and ok comments mark which lines the rule must flag or skip. The skill bans todook and todoruleid annotations, so every test case counts. Its severity value HIGH matches current rule-syntax docs.

The evidence names three tasks. The primary one is a taint rule for user input reaching a command sink. It runs weekly for AppSec engineers. The second is a pattern rule for one dangerous API. Encoding internal secure-coding standards is a publisher claim only.

Test it yourself:

1. Ask for a Python taint rule for request input reaching os.system.
2. Name shlex.quote as the sanitizer.
3. Write a few unseen vulnerable and safe snippets.
4. Run the rule against them.

The last step matters most. The skill's own tests pass by design, so unseen cases show real coverage.

What are the alternatives to the Semgrep Rule Creator skill?

The alternatives to the Semgrep Rule Creator skill depend on the job. Trail of Bits' Static Analysis plugin runs existing scans. OpenAI's Security Best Practices and Security Threat Model cover broader reviews.

None of these writes a custom rule with tests. For that job, Semgrep Rule Creator has no direct rival in our reviews.

Common questions

Does the Semgrep Rule Creator skill work offline?

No. The seven doc fetches are mandatory on every run. PR #322 proposed removing them, but it closed unmerged on 2026-10-05.

Do you need Semgrep installed?

Yes. The skill calls the local semgrep CLI for tests, validation and AST dumps.

Can it run my existing Semgrep rulesets?

No. It writes one new rule at a time. Trail of Bits' Static Analysis plugin handles scans with existing rules.

Do you need Semgrep experience to use it?

No for drafting, since the skill supplies the workflow. Reviewing a taint rule's sources and sinks still takes Semgrep knowledge.

Is the Semgrep Rule Creator skill free?

Yes. It costs nothing to install, and PR #322 measured about $0.48 per rule in tokens. GitHub lists CC-BY-SA-4.0, while agenticskills.io shows AGPL-3.0.

What does the Semgrep Rule Creator plugin add?

It adds one /semgrep-rule command, 554 bytes long, that invokes the skill. The plugin.json version is 1.2.6.

Where are Trail of Bits' ready-made Semgrep rules?

They sit in the Trail of Bits ruleset on the Semgrep Registry, at semgrep.dev/p/trailofbits. That ruleset is separate from this skill.