Threat model

What we catch, what we miss

AiSkillsGuard reads an agent skill before you install it. This page explains each detection layer, the limits of each one, and how accurate the static scanner is on our own regression corpus.

Static scanner accuracy

Measured 2026-10-22
Precision
100%
Recall
100%
Entries
25
14 malicious · 11 benign
False positives
0

This is a small, 25-entry regression suite we wrote alongside the rules — not an independent benchmark, and a 100% score on it is a reflection of that, not a claim about real-world traffic. Treat these numbers as a floor that stops regressions, not as a measure of real-world accuracy. Run it yourself with npm run eval.

Detection layers

  1. Static rules

    Pattern rules over the full skill and its natural-language instructions. Text is normalized first (Unicode folding, invisible characters removed, whitespace padding collapsed) so simple evasion does not hide patterns. Rules distinguish capability from action where it matters — e.g. a literal, hardcoded shell command scores lower than one built from a variable or template interpolation.

    Limit: Pattern-based. A determined author can rephrase around any single rule.

  2. Scanner-directed text

    Detects instructions that try to steer scanners or reviewers into approving a skill.

    Limit: Only catches known phrasings.

  3. Supply-chain / typosquat check

    Extracts imported package names (require/import, JS and Python) and flags ones within one or two edits of a well-known package name they are not — the classic typosquat shape, including adjacent-letter swaps and separator removal (e.g. a real 2022 npm incident used "crossenv" for "cross-env").

    Limit: Checked against a curated list of popular packages, not a live registry. Cannot confirm a package is actually malicious, only that its name is suspicious.

  4. Within-skill attack-chain detection

    Looks at the full set of findings together, not just one line at a time. When a skill both accesses sensitive data (credentials, SSH keys, cookies, clipboard, etc.) and makes an outbound network call elsewhere in the content, it is flagged even though the two halves never appear in the same snippet.

    Limit: Scoped to a single skill. Does not yet track chains across multiple installed skills.

  5. LLM review

    A language model reads the skill for intent and paraphrased injection that rules miss. Its verdict is checked against the deterministic findings: a SAFE verdict is downgraded when HIGH or CRITICAL rules fired.

    Limit: The model can be wrong or persuaded. It is one layer, not the final word.

  6. Vulnerability intelligence

    Checks packages and CVEs against OSV, NVD, the CISA Known Exploited Vulnerabilities feed, and GitHub advisories.

    Limit: Covers known issues only. Unknown or unpublished vulnerabilities are not caught.

  7. Change monitoring

    Hashes a published skill and alerts when its content changes after you reviewed it.

    Limit: Detects change, not whether the change is malicious.

  8. Cross-skill exposure (dashboard, signed in)

    Looks across your own saved scan history for the same secret-access + egress combination split across different skills, not just within one. If you've scanned a skill that reads credentials and a separate one that calls out to the network, your dashboard shows the aggregate count.

    Limit: Not a traced data-flow connection — installed skills don't share an observable flow we can follow. It's an exposure count to review, not proof two skills interact. Requires a signed-in account with scan history.

What we do not detect yet

  • Runtime behavior. Skills are not executed in a sandbox, so a payload assembled at run time is only caught if the static rules see it.
  • A proven connection between skills. The cross-skill layer above counts exposure across your scan history, but cannot trace whether one skill's output actually reaches another.
  • Obfuscation beyond the normalization above, such as custom encoding schemes.
  • Malicious intent that is expressed only in natural language with no suspicious technical signal.
  • Typosquats of packages outside our curated popular-package list, or in ecosystems we don't check (only npm- and Python-style imports are parsed today).

Known false positives

  • The attack-chain and typosquat checks are heuristics over text, not proof of intent — a skill that happens to read a cookie for one feature and call an unrelated API for another can still trigger the chain warning. Read the "review whether" wording as exactly that: a prompt to look closer, not a verdict.

What the verdicts mean

Verdicts describe what was found. They are advice, not a guarantee. A safe verdict means no HIGH or CRITICAL signal was found by any layer, not that the skill is proven harmless.

Run it locally or in CI

By default the command-line scanner runs only the deterministic rules. It makes no LLM call and no network request, so it works in CI with no API key and no cost.

npm run aiskillsguard -- ./my-skill --fail-on high
npm run aiskillsguard -- ./skills --json

Pass --llm to additionally run the same LLM review the web scanner uses, per file. This needs ANTHROPIC_API_KEY set (or in .env.local) and calls the Anthropic API, so it costs API credits and is opt-in — never turned on by the base command, so a CI job that doesn't ask for it never makes a network call or spends anything.

npm run aiskillsguard -- ./my-skill --llm

Exit code 0 means nothing at or above the threshold. 1 means findings were found. 2 means a usage or read error — including a missing API key when --llm is passed.