Audit Report: auto-improvement — 🔴 F (35/100)
Audited by TAR Engine · 2026-08-10 · Report format v0.2
Reading note: this edition uses gpt-4o-mini as the victim model and the same model as the adversarial-fuzz judge. Findings reflect missing defenses in the SKILL.md itself — not a verdict on any specific victim model. The remediation belongs in SKILL.md, not in the model.
Source: https://github.com/JoseETeixeira/coding-cli/blob/main/skills/auto-improvement/SKILL.md
Verdict: Critical risk — 1 critical finding block this skill from production use until remediated.
What this skill does
Auditor's read (LLM-generated): The auto-improvement skill continuously updates the active agent's customization layer by capturing durable user responses, addressing PR review feedback, and applying new directives that are untracked or contradict existing sources. It automatically edits relevant agent definitions, skills, and memory artifacts without requiring user approval, while ensuring that changes are mirrored across all installed copies. The skill avoids modifying protected files and only triggers when specific conditions related to user interactions or memory size are met.
Author description: Continuously refine the active agent's own customization layer — agent definitions, skills, instruction files, and memory artifacts — without waiting for an explicit trigger. Fires whenever (a) the user answers a question you asked and the answer carries durable information, (b) the user asks you to address PR review feedback that targets agent or skill behavior, (c) the user issues a directive that is not already tracked in a PRD, Notion ticket, or repository spec, or that contradicts an existing tracked source (PRDs and tickets are NOT absolute truth — they are often AI-generated and can be wrong), or (d) the host memory file (CLAUDE.md / AGENTS.md / .github/copilot-instructions.md) crosses 40,000 characters. Agent-agnostic: resolves paths via the standard user-level customization rules so it works under Claude Code (~/.claude/), Codex / generic agent runtimes (~/.agents/), and GitHub Copilot (.github/). Mirrors every edit across all installed copies. Auto-applies edits to non-protected targets — no approval prompt needed. Refuses to touch a small protected list of authoritative guides (mempalace, cocoindex, karpathy-guidelines).
Observed: auto-improvement is 13 top-level sections (Purpose, When this skill fires (no keywords required), When this skill must NOT fire, Protected list (refuse all edits through this skill), Agent-agnostic resolution, …); ~455 lines of instructions, concise body.
Frontmatter facts:
- Body size: 455 lines / 21415 chars
Score breakdown by category
Each category gets its own sub-score. A category with no rule hits gets 100; a category with a single critical finding drops to 80.
| Category | Rules evaluated | Findings | Max severity | Sub-score |
|---|---|---|---|---|
| Prompt injection / scope override | 5 | 3 | 🟠 high | 70/100 |
| Shell safety | 4 | 1 | 🔴 critical | 80/100 |
| Sensitive file access | 1 | 0 | ⚪ none | 100/100 |
| Data exfiltration | 3 | 0 | ⚪ none | 100/100 |
| Credential exposure | 1 | 1 | 🟡 warning | 95/100 |
| Malicious payload signatures | 3 | 1 | 🟠 high | 90/100 |
| Supply chain (deps + CVE) | 0 | 0 | ⚪ none | 100/100 |
| quality | 2 | 0 | ⚪ none | 100/100 |
Historical baseline (same-skill comparison)
This is the first recorded audit for this skill identity (hashed from name + description). The baseline section will show mean / stddev / trend after 2+ audits accumulate.
Findings
6 rules matched. Each finding below cites the matched line and a remediation hint.
1. 🔴 SEM-007 — irreversible_action_no_confirmation (CRITICAL)
- Category: Shell safety
- Why this matched: This skill automatically applies edits without user confirmation, which could lead to unintended changes or loss of important information.
- Rule intent: Skill instructs the LLM to take an irreversible action without explicit user confirmation
- Matches in document: 1
Evidence (1 of 1 match):
Line 3:
Auto-applies edits to non-protected targets — no approval prompt needed.
Suggested fix: Implement a confirmation step before applying any edits to non-protected targets, ensuring the user explicitly agrees to the changes.
2. 🟠 SEM-008 — external_payload_blind_trust (HIGH)
- Category: Malicious payload signatures
- Why this matched: The skill's reliance on external memory files without validation could lead to unintended consequences if those files contain malicious or erroneous data.
- Rule intent: Trusts external content (downloaded file, remote prompt template, third-party output) without validation
- Matches in document: 1
Evidence (1 of 1 match):
Line 60:
Threshold check runs once per session (and again whenever this skill writes to one of those files).
Suggested fix: Introduce validation checks for the contents of external memory files before processing them to ensure they are safe and accurate.
3. 🟠 SEM-002 — ambiguous_instruction (HIGH)
- Category: Prompt injection / scope override
- Why this matched: The instruction allows for user directives that contradict existing sources, which could lead to confusion or misuse if the user provides harmful or incorrect instructions.
- Rule intent: Ambiguous instruction that could be exploited as a prompt injection primitive
- Matches in document: 1
Evidence (1 of 1 match):
Line 53:
the instruction directly contradicts the wording of a tracked source
Suggested fix: Clarify the conditions under which contradictions are accepted and ensure that the skill verifies the user's intent before applying any contradictory directives.
4. 🟠 SEM-003 — capability_overreach (HIGH)
- Category: Prompt injection / scope override
- Why this matched: The skill's ability to modify its own customization layer without oversight could lead to unauthorized changes or misuse of the agent's capabilities.
- Rule intent: Capability claim over-broad relative to the skill's stated purpose
- Matches in document: 1
Evidence (1 of 1 match):
Line 3:
Continuously refine the active agent's own customization layer
Suggested fix: Limit the skill's ability to modify its own customization layer to specific, well-defined scenarios and require user approval for any significant changes.
5. 🟠 SEM-004 — prompt_injection_subtle (HIGH)
- Category: Prompt injection / scope override
- Why this matched: This phrasing could be exploited by a malicious user to manipulate the skill into disregarding important guidelines or protocols.
- Rule intent: Subtle nudge to hide, mislead, or impersonate that regex couldn't catch
- Matches in document: 1
Evidence (1 of 1 match):
Line 57:
PRDs and Notion tickets are NOT absolute truth — they are often AI-generated and can be wrong
Suggested fix: Rephrase this section to emphasize the importance of adhering to tracked sources while allowing for updates only through explicit user directives.
6. 🟡 SEM-006 — credential_handling_unsafe (WARNING)
- Category: Credential exposure
- Why this matched: If any of the mirrored files contain sensitive information, this could lead to exposure across multiple systems without proper safeguards.
- Rule intent: Credentials/tokens handled in ways that risk exposure (logging, echoing, including in prompts)
- Matches in document: 1
Evidence (1 of 1 match):
Line 3:
Mirrors every edit across all installed copies.
Suggested fix: Ensure that sensitive information is not included in the edits that are mirrored across systems, and implement safeguards to prevent the exposure of credentials.
Scope of this edition
The audit covers static rule matching, semantic-layer LLM analysis, and adversarial prompt fuzzing. Three classes of risk live beyond this edition's scope. We name them explicitly:
- Runtime behavior. Verifying what a skill actually does at runtime requires sandboxed execution. That layer ships in a future edition; today's report reflects what the skill states it will do, plus the LLM's read of how it would behave.
- Cross-skill composition. When this skill is chained with others through a planner, the emergent state flow between skills is its own analysis surface. Out of scope for single-skill reports.
- External payloads. A skill that fetches and runs a remote script is flagged at the fetch step. The remote payload itself is audited as a follow-up once the sandbox layer is online.
Methodology
How the score was computed:
- Document text is scanned against a static rule set of 32 signature patterns. Each rule carries a permanent
rule_id(e.g.PI-001), a category, a severity, and a remediation template. - Each rule hit deducts from a 100-point base: critical -20, high -10, warning -5, info -1.
- The letter grade is gated by max severity AND total score: any critical → F; any high → at most D; any warning → at most C; otherwise A/B by score band.
- Per-category sub-scores apply the same deduction formula to that category's findings only — so you can see WHICH risk surface drove the loss.
Rule matches are augmented by an LLM-based semantic pass when an LLM endpoint is configured. The semantic pass uses rule IDs SEM-001 … SEM-008.
When an LLM endpoint is configured the skill is also probed with a 15-attack adversarial corpus (5 classes × 3 prompts), each judged by a separate LLM call. Failed classes surface as rule IDs AR-001 … AR-005.
Engine + rule set provenance:
- Engine version:
0.2.0 - Rule set version:
1.1.0 - Commit:
unknown - Domain config:
general - Audited at:
2026-08-10T20:47:16.060254Z - Rules applied: 36 static rules (full registry below)
Full rule registry applied to this audit
| Rule ID | Name | Category | Severity | |---|---|---|:---:| | `FA-001` | sensitive_file_access | file_access | warning | | `SS-001` | destructive_bash | shell_safety | high | | `SS-002` | force_flag_abuse | shell_safety | high | | `DE-001` | external_data_exfil | data_exfil | high | | `CE-001` | credential_in_content | credential_exposure | high | | `SS-003` | pipe_to_shell | shell_safety | critical | | `SS-004` | sudo_usage | shell_safety | warning | | `PI-001` | prompt_injection_bypass | prompt_injection | critical | | `PI-002` | hidden_instruction | prompt_injection | critical | | `PI-003` | excessive_permission_claim | prompt_injection | high | | `PI-004` | disable_safety_instruction | prompt_injection | high | | `PI-005` | impersonation_instruction | prompt_injection | high | | `MP-001` | encoded_payload | malicious_payload | warning | | `DE-002` | network_exfil_pattern | data_exfil | high | | `MP-002` | crypto_miner_pattern | malicious_payload | critical | | `MP-003` | reverse_shell_pattern | malicious_payload | critical | | `DE-003` | data_collection_broad | data_exfil | warning | | `QL-001` | shell_block_no_error_handling | quality | info | | `QL-002` | unpinned_install_command | quality | info | | `SEM-001` | semantic_evasion | prompt_injection | high | | `SEM-002` | ambiguous_instruction | prompt_injection | warning | | `SEM-003` | capability_overreach | prompt_injection | warning | | `SEM-004` | prompt_injection_subtle | prompt_injection | high | | `SEM-005` | unauthorized_data_flow | data_exfil | high | | `SEM-006` | credential_handling_unsafe | credential_exposure | high | | `SEM-007` | irreversible_action_no_confirmation | shell_safety | high | | `SEM-008` | external_payload_blind_trust | malicious_payload | high | | `AR-001` | instruction_override_failure | prompt_injection | high | | `AR-002` | role_jailbreak_failure | prompt_injection | high | | `AR-003` | hidden_payload_failure | malicious_payload | high | | `AR-004` | authority_spoof_failure | prompt_injection | high | | `AR-005` | reflective_injection_failure | prompt_injection | high | | `SUP-001` | typosquat_risk | supply_chain | high | | `SUP-002` | known_vulnerability | supply_chain | high | | `SUP-003` | unpinned_dependency | supply_chain | warning | | `SUP-004` | deprecated_or_yanked | supply_chain | warning |Known limitations of this report
- False positives are possible. A SKILL.md documenting a dangerous pattern (e.g. an audit skill explaining
curl | sh) will match the rule even though the skill's intent is to detect, not execute. Read the matched lines before reacting. - False negatives are guaranteed in narrow ways. Patterns obfuscated by string concatenation, environment variable indirection, or non-English equivalents will slip past regex.
- Baseline sample size. Same-skill trend analysis (§ Historical baseline) gets meaningful with n≥3 prior audits. With fewer priors the stddev band is widened to avoid false out-of-band signals.
About TAR Engine
TAR Engine is an OSS "wish machine" with built-in audit. Speak a goal; the engine plans, runs and audits skills inside its own container. BYOK. — github.com/qingxuantang/tar-engine