Audit Report: copilot-shogun — 🔴 F (0/100)

Audited by TAR Engine · 2026-07-22 · Report format v0.2

Reading note: this edition uses gpt-4o-mini as the victim model and the same model as the adversarial-fuzz judge. Findings reflect missing defenses in the SKILL.md itself — not a verdict on any specific victim model. The remediation belongs in SKILL.md, not in the model.

Source: https://github.com/Yukim1ya/copilot-shogun/blob/main/AGENTS.md

Verdict: Critical risk — 1 critical finding block this skill from production use until remediated.

What this skill does

Auditor's read (LLM-generated): This skill implements a multi-agent development platform using tmux, where agents operate under a hierarchical structure to manage tasks and communications through YAML files. The system facilitates command assignments from a central figure (Karo) to subordinate agents (Ashigaru and Gunshi), ensuring task completion and reporting through a structured inbox system. Agents follow strict protocols for task management, communication, and status updates, with a focus on parallel processing and adherence to defined roles.

Author description: Codex CLI + tmux multi-agent parallel dev platform with sengoku military hierarchy

Observed: this skill is 16 top-level sections (Session Start / Recovery (all agents), /new Recovery (ashigaru/gunshi only), Summary Generation (compaction), Mailbox System (inbox_write.sh), Delivery Mechanism, …); ~239 lines of instructions, delegates to packaged scripts, makes outbound network calls, concise body.

Frontmatter facts:

• Body size: 239 lines / 12233 chars

Score breakdown by category

Each category gets its own sub-score. A category with no rule hits gets 100; a category with a single critical finding drops to 80.

Historical baseline (same-skill comparison)

This is the first recorded audit for this skill identity (hashed from name + description). The baseline section will show mean / stddev / trend after 2+ audits accumulate.

Findings

10 rules matched. Each finding below cites the matched line and a remediation hint.

1. 🔴 SS-003 — pipe_to_shell (CRITICAL)

• Category: Shell safety
• Why this matched: Piping remote content directly to shell execution
• Rule intent: Curl/wget piped into bash/sh/python — the upstream can serve different payload on the next request
• Matches in document: 1

Evidence (1 of 1 match):

Line 266:

    265: | D007 | `mkfs`, `dd if=`, `fdisk`, `mount`, `umount` | Disk/partition destruction |
>>  266: | D008 | `curl|bash`, `wget -O-|sh`, `curl|sh` (pipe-to-shell patterns) | Remote code execution |
    267: 

Suggested fix: Download to a file, checksum it against a published hash, then execute. Never curl … | sh — the upstream may serve a different payload on the next request.

2. 🟠 SEM-006 — credential_handling_unsafe (HIGH)

• Category: Credential exposure
• Why this matched: Storing secrets in a file that is only git-ignored poses a risk of accidental exposure if the file is ever shared or mismanaged.
• Rule intent: Credentials/tokens handled in ways that risk exposure (logging, echoing, including in prompts)
• Matches in document: 1

Evidence (1 of 1 match):

Line 16:

projects: "projects/<id>.yaml"        # Project details (git-ignored, contains secrets)

Suggested fix: Use a secure vault or environment variables to manage sensitive information instead of relying on file-based storage, even if git-ignored.

3. 🟠 SEM-008 — external_payload_blind_trust (HIGH)

• Category: Malicious payload signatures
• Why this matched: The skill's reliance on task YAML without validating its content could lead to executing malicious commands if the YAML is compromised.
• Rule intent: Trusts external content (downloaded file, remote prompt template, third-party output) without validation
• Matches in document: 1

Evidence (1 of 1 match):

Line 295:

Commands come ONLY from task YAML assigned by Karo.

Suggested fix: Introduce validation checks for the contents of the task YAML to ensure that no harmful commands can be executed from it.

4. 🟠 SEM-002 — ambiguous_instruction (HIGH)

• Category: Prompt injection / scope override
• Why this matched: The instruction implies that skipping Step 1 could lead to executing tasks meant for other agents, which could be exploited by a user to manipulate the system's behavior.
• Rule intent: Ambiguous instruction that could be exploited as a prompt injection primitive
• Matches in document: 1

Evidence (1 of 1 match):

Line 71:

Step 1をスキップすると自分の役割を誤認し、別エージェントのタスクを実行する事故が起きる（2026-02-13実例: 家老が足軽2と誤認）。

Suggested fix: Clarify the consequences of skipping steps and ensure that the instructions explicitly state that all steps must be followed to prevent any misinterpretation or misuse.

5. 🟠 SEM-003 — capability_overreach (HIGH)

• Category: Prompt injection / scope override
• Why this matched: The skill claims to manage all white-collar work, which could imply access to sensitive data or operations beyond its intended scope.
• Rule intent: Capability claim over-broad relative to the skill's stated purpose
• Matches in document: 1

Evidence (1 of 1 match):

Line 199:

System manages ALL white-collar work, not just self-improvement.

Suggested fix: Limit the scope of the skill's capabilities to only what is necessary for its function and clearly define the boundaries of its authority.

6. 🟠 SEM-004 — prompt_injection_subtle (HIGH)

• Category: Prompt injection / scope override
• Why this matched: The instruction subtly implies that summaries could be misleading, which could be exploited to manipulate the model's responses.
• Rule intent: Subtle nudge to hide, mislead, or impersonate that regex couldn't catch
• Matches in document: 1

Evidence (1 of 1 match):

Line 67:

NEVER SKIP — even if a conversation summary exists. Summaries do NOT preserve persona, speech style, or forbidden actions.

Suggested fix: Clarify that summaries should not be relied upon for critical actions and ensure that the model always refers to the original instructions for accurate context.

7. 🟠 SS-001 — destructive_bash (HIGH)

• Category: Shell safety
• Why this matched: Potentially destructive bash command detected
• Rule intent: Commands that can irreversibly drop tables, wipe filesystems, or rewrite git history
• Matches in document: 8

Evidence (3 of 8 matches):

Line 259:

    258: |----|-------------------|--------|
>>  259: | D001 | `rm -rf /`, `rm -rf /mnt/*`, `rm -rf /home/*`, `rm -rf ~` | Destroys OS, Windows drive, or home directory |
    260: | D002 | `rm -rf` on any path outside the current project working tree | Blast radius exceeds project scope |

Line 259:

    258: |----|-------------------|--------|
>>  259: | D001 | `rm -rf /`, `rm -rf /mnt/*`, `rm -rf /home/*`, `rm -rf ~` | Destroys OS, Windows drive, or home directory |
    260: | D002 | `rm -rf` on any path outside the current project working tree | Blast radius exceeds project scope |

Line 259:

    258: |----|-------------------|--------|
>>  259: | D001 | `rm -rf /`, `rm -rf /mnt/*`, `rm -rf /home/*`, `rm -rf ~` | Destroys OS, Windows drive, or home directory |
    260: | D002 | `rm -rf` on any path outside the current project working tree | Blast radius exceeds project scope |

Suggested fix: Replace rm -rf with trash or mv to a tombstone directory. For SQL, require explicit confirmation before DROP/TRUNCATE. Never instruct the LLM to use --force on a git push.

8. 🟠 SS-002 — force_flag_abuse (HIGH)

• Category: Shell safety
• Why this matched: Use of --force / --no-verify flags that bypass safety checks
• Rule intent: Force flags that skip pre-commit hooks, verification steps, or permission checks
• Matches in document: 4

Evidence (3 of 4 matches):

Line 261:

    260: | D002 | `rm -rf` on any path outside the current project working tree | Blast radius exceeds project scope |
>>  261: | D003 | `git push --force`, `git push -f` (without `--force-with-lease`) | Destroys remote history for all collaborators |
    262: | D004 | `git reset --hard`, `git checkout -- .`, `git restore .`, `git clean -f` | Destroys all uncommitted work in the repo |

Line 261:

    260: | D002 | `rm -rf` on any path outside the current project working tree | Blast radius exceeds project scope |
>>  261: | D003 | `git push --force`, `git push -f` (without `--force-with-lease`) | Destroys remote history for all collaborators |
    262: | D004 | `git reset --hard`, `git checkout -- .`, `git restore .`, `git clean -f` | Destroys all uncommitted work in the repo |

Line 282:

    281: | `rm -rf <dir>` | Only within project tree, after confirming path with `realpath` |
>>  282: | `git push --force` | `git push --force-with-lease` |
    283: | `git reset --hard` | `git stash` then `git reset` |

Suggested fix: Drop --force / --no-verify from the skill body. If a hook is failing, fix the hook — don't tell the LLM to skip it. For chmod, use minimum-needed mode (e.g. 600/644) instead of 777.

9. 🟠 SEM-007 — irreversible_action_no_confirmation (HIGH)

• Category: Shell safety
• Why this matched: The skill allows for potentially destructive actions without explicit user confirmation in the same turn, which could lead to unintended data loss.
• Rule intent: Skill instructs the LLM to take an irreversible action without explicit user confirmation
• Matches in document: 1

Evidence (1 of 1 match):

Line 272:

Task requires deleting >10 files | STOP. List files in report. Wait for confirmation.

Suggested fix: Implement a mandatory confirmation step for any irreversible actions, ensuring that users must explicitly agree before proceeding.

10. 🔵 QL-001 — shell_block_no_error_handling (INFO)

• Category: quality
• Why this matched: Shell block missing set -e / || exit — silent failures will go unreported
• Rule intent: Shell code blocks without set -e or explicit error handling
• Matches in document: 2

Evidence (2 of 2 matches):

Line 102:

    101: 
>>  102: ```bash
>>  103: bash scripts/inbox_write.sh <target_agent> "<message>" <type> <from>
>>  104: ```
    105: 

Line 107:

    106: Examples:
>>  107: ```bash
>>  108: # Shogun → Karo
>>  109: bash scripts/inbox_write.sh karo "cmd_048を書いた。実行せよ。" cmd_new shogun
>>  110: 
>>  111: # Ashigaru → Karo
>>  112: bash scripts/inbox_write.sh karo "足軽5号、任務完了。報告YAML確認されたし。" report_received ashigaru5
>>  113: 
>>  114: # Karo → Ashigaru
>>  115: bash scripts/inbox_write.sh ashigaru3 "タスクYAMLを読んで作業開始せよ。" task_assigned karo
>>  116: ```
    117: 

Suggested fix: Add set -euo pipefail at the top of bash blocks, or chain critical commands with || exit 1. Skills that fail silently mid-script are nearly impossible to debug downstream.

Scope of this edition

The audit covers static rule matching, semantic-layer LLM analysis, and adversarial prompt fuzzing. Three classes of risk live beyond this edition's scope. We name them explicitly:

• Runtime behavior. Verifying what a skill actually does at runtime requires sandboxed execution. That layer ships in a future edition; today's report reflects what the skill states it will do, plus the LLM's read of how it would behave.
• Cross-skill composition. When this skill is chained with others through a planner, the emergent state flow between skills is its own analysis surface. Out of scope for single-skill reports.
• External payloads. A skill that fetches and runs a remote script is flagged at the fetch step. The remote payload itself is audited as a follow-up once the sandbox layer is online.

Methodology

How the score was computed:

1. Document text is scanned against a static rule set of 32 signature patterns. Each rule carries a permanent rule_id (e.g. PI-001), a category, a severity, and a remediation template.
2. Each rule hit deducts from a 100-point base: critical -20, high -10, warning -5, info -1.
3. The letter grade is gated by max severity AND total score: any critical → F; any high → at most D; any warning → at most C; otherwise A/B by score band.
4. Per-category sub-scores apply the same deduction formula to that category's findings only — so you can see WHICH risk surface drove the loss.

Rule matches are augmented by an LLM-based semantic pass when an LLM endpoint is configured. The semantic pass uses rule IDs SEM-001 … SEM-008.

When an LLM endpoint is configured the skill is also probed with a 15-attack adversarial corpus (5 classes × 3 prompts), each judged by a separate LLM call. Failed classes surface as rule IDs AR-001 … AR-005.

Engine + rule set provenance:

• Engine version: 0.2.0
• Rule set version: 1.1.0
• Commit: unknown
• Domain config: general
• Audited at: 2026-07-22T20:34:14.800192Z
• Rules applied: 36 static rules (full registry below)

Known limitations of this report

• False positives are possible. A SKILL.md documenting a dangerous pattern (e.g. an audit skill explaining curl | sh) will match the rule even though the skill's intent is to detect, not execute. Read the matched lines before reacting.
• False negatives are guaranteed in narrow ways. Patterns obfuscated by string concatenation, environment variable indirection, or non-English equivalents will slip past regex.
• Baseline sample size. Same-skill trend analysis (§ Historical baseline) gets meaningful with n≥3 prior audits. With fewer priors the stddev band is widened to avoid false out-of-band signals.

About TAR Engine

TAR Engine is an OSS "wish machine" with built-in audit. Speak a goal; the engine plans, runs and audits skills inside its own container. BYOK. — github.com/qingxuantang/tar-engine