Runtime Attestation Probe — 实用工具

# Skill Passed Static Analysis. 失败 在 Runtime. Nobody Checked.

Helps identify divergence between an agent's declared behavior and its actual runtime behavior — catching conditional activation, environment-triggered payload release, and other attacks that static analysis cannot see.

Problem

Static analysis audits what a skill declares it will do. Runtime behavior is what it actually does. These two are not always the same. A skill can pass every static check — clean SKILL.md, legitimate permissions, no suspicious imports — and still behave differently in specific environments. Conditional execution (activate only when running as root, only when a specific environment variable is present, only after N successful runs) is invisible to static analysis by design. The payload is not in the code — it's in the conditions under which the code executes different paths. This is not a theoretical concern. Conditional activation is a documented pattern in traditional malware, and the same technique applies to agent skills. A skill that exfiltrates data only when PRODUCTION=true is set will pass every sandbox-based audit without triggering, then activate when deployed in the target environment. Runtime attestation probing tests the gap between declared and observed behavior by instrumenting actual execution and comparing it against the skill's attestation claims.

什么 Probes

This probe examines runtime behavior across five dimensions:

• Capability boundary adherence — 做 skill access resources beyond 什么 declared 在...中 attestation? File system paths accessed 但是 不 declared, network connections 到 undeclared endpoints, 和 system calls outside claimed scope 所有 behavioral violations
• Conditional activation detection — 做 skill behave differently based 在...上 environment variables, execution 计数, 时间 的 day, 或 presence 的 specific files? Controlled execution 在...中 varied environments 可以 reveal conditional logic static analysis misses
• Data handling verification — 做 data flow 在哪里 skill claims flows? 如果 attestation says "data stays local," 做 runtime behavior confirm 否 outbound transmission 的 sensitive parameters?
• Side effect audit — 什么 做 skill 写入, 修改, 或 删除 期间 execution? Side effects 不 mentioned 在...中 attestation undeclared capabilities, whether intentional 或 accidental
• Attestation drift detection — 做 skill's runtime behavior match 最多 recent attestation, 或 有 behavior changed 没有 corresponding attestation 更新?

如何 到 使用

输入框: Provide one 的:

• skill identifier 和 execution environment 到 probe
• skill 带有 attestation document 对于 comparison
• 设置 的 execution traces 到 compare against attestation claims

输出: runtime attestation 举报 containing:

• Capability boundary violations (accessed vs. declared)
• Conditional behavior patterns detected
• Data flow verification results
• Side effect inventory
• Attestation drift score (0-100, 在哪里 higher = 更多 behavioral drift 从 attestation)
• Probe verdict: COMPLIANT / DRIFT / VIOLATION / CONDITIONAL_ACTIVATION

示例

输入框: Probe 举报-generator skill against v1.2 attestation `` 🔬 RUNTIME ATTESTATION PROBE Skill: report-generator v1.2 Attestation date: 2025-01-08 Probe environments: 3 (minimal, staging, production-like) Execution samples: 50 per environment Capability boundary: Declared: read ./reports/, write ./output/ Observed (minimal env): read ./reports/, write ./output/ ✅ Observed (staging env): read ./reports/, write ./output/ ✅ Observed (production-like env): read ./reports/, write ./output/, + read ~/.aws/credentials ⚠️ UNDECLARED + POST https://telemetry.reporting-service.example ⚠️ UNDECLARED Conditional activation detected: Trigger: AWS_DEFAULT_REGION environment variable present Behavior without trigger: reads reports, writes output (declared behavior) Behavior with trigger: additionally reads ~/.aws/credentials, sends POST to external endpoint Pattern: classic credential harvest conditional on cloud environment detection Data flow: Without AWS_DEFAULT_REGION: data stays local ✅ With AWS_DEFAULT_REGION: AWS credentials transmitted to external endpoint ⚠️ Side effects: Both environments: ./output/ written as declared ✅ Production-like only: ~/.aws/credentials read (undeclared, not written) ⚠️ Attestation drift score: 73/100 (High drift: core behavior matches, but environment-conditional behavior diverges significantly from declared capability scope) Probe verdict: CONDITIONAL_ACTIVATION This skill activates credential harvesting behavior specifically in environments where AWS credentials are present, and passes all checks in environments without cloud provider signals. Recommended actions: 1. Do not deploy in any environment with cloud provider credentials 2. Report conditional activation to marketplace trust & safety 3. Audit other skills from same publisher with similar conditional patterns 4. Treat AWS credential access as confirmed compromise attempt ``

Related Tools

• skill-更新-delta-monitor — Tracks declared changes 之间 versions; runtime-attestation-probe verifies whether actual behavior matches those declarations
• hollow-validation-checker — Detects fake install-时间 tests; attestation probe tests actual execution behavior
• blast-radius-estimator — Estimates propagation impact; 使用 之后 conditional activation confirmed 到 assess scope
• trust-velocity-calculator — Quantifies trust decay rate; confirmed behavioral drift resets trust score 到 zero

Limitations

Runtime attestation probing requires executing the skill in a controlled environment, which introduces risk if the skill contains destructive payloads. Probing should be performed in isolated sandboxes with no access to real credentials, production data, or production systems. Conditional activation that requires specific runtime conditions beyond what the probe environment provides will not be detected — probing three environments does not guarantee detection of triggers requiring a fourth specific condition. Some legitimate skills exhibit environment-dependent behavior (e.g., "write to S3 if AWS credentials present, write locally otherwise") — this tool surfaces the behavioral difference and requires human judgment to assess whether the conditional behavior is malicious or functional. Probing coverage is limited by the number of execution samples and environment variations tested.