Skip to main content Home Creators adu2021 skillxiv agent-skills-security-analysis
agent-skills-security-analysis Empirically analyzes 31,132 agent skills to identify 14 distinct vulnerability patterns, finding 26.1% contain security flaws including data exfiltration, privilege escalation, and malicious intent risks that require mandatory vetting.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill agent-skills-security-analysisThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name agent-skills-security-analysis title Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2601.10338 keywords ["agent-security","skill-vetting","vulnerability-detection","threat-analysis","skill-marketplace"] description Empirically analyzes 31,132 agent skills to identify 14 distinct vulnerability patterns, finding 26.1% contain security flaws including data exfiltration, privilege escalation, and malicious intent risks that require mandatory vetting.
Overview
Conduct security analysis of modular agent skills before deployment. Use multi-stage detection combining static analysis and LLM-based semantic classification to identify vulnerabilities such as data exfiltration, privilege escalation, prompt injection, and supply chain risks.
When to Use
When integrating third-party agent skills into your systems
For skill marketplaces or skill package management
To build automated vetting pipelines before skill deployment
When auditing existing skill collections for vulnerabilities
When NOT to Use
For single-use, internally-developed skills
In fully sandboxed environments with no skill communication
For read-only skills with no side effects
In low-risk applications where skill compromise has minimal impact
Key Technical Components
SkillScan Detection Framework
Implement multi-stage detection pipeline combining static and semantic analysis.
class SkillScan :
def __init__ (self ):
self .static_analyzer = StaticAnalyzer()
self .semantic_classifier = SemanticClassifier()
def scan_skill (self, skill_code, skill_metadata ):
"""Comprehensive vulnerability detection"""
results = {
"static_findings" : self .static_analyzer.analyze(skill_code),
"semantic_findings" : self .semantic_classifier.classify(skill_code),
"metadata_issues" : self .check_metadata(skill_metadata),
"vulnerability_patterns" : []
}
results["vulnerability_patterns" ] = self .consolidate_findings(results)
results["risk_score" ] = self
return
def
consolidate_findings
self, findings
"""Identify distinct vulnerability patterns"""
for
in
"static_findings"
self
return
list
set
Static Analysis Component Detect vulnerabilities through code pattern matching.
class StaticAnalyzer :
VULNERABILITY_PATTERNS = {
"credential_exposure" : [
r"api[_]?key\s*=" ,
r"password\s*=" ,
r"token\s*="
],
"file_access_risk" : [
r"open\s*\(" ,
r"read\s*file" ,
r"write\s*file" ,
r"os\.remove"
],
"network_calls" : [
r"requests\." ,
r"urllib" ,
r"socket\." ,
r"send.*http"
],
"process_execution" : [
r"subprocess\." ,
r"os\.system" ,
r"popen"
]
}
def analyze (self, code ):
"""Find suspicious patterns in code"""
findings = []
for pattern_type, patterns in self .VULNERABILITY_PATTERNS.items():
for pattern in patterns:
matches = re.findall(pattern, code, re.IGNORECASE)
if matches:
findings.append({
"type" : pattern_type,
"pattern" : pattern,
"match_count" : len (matches),
"severity" : self .estimate_severity(pattern_type)
})
return findings
def estimate_severity (self, pattern_type ):
"""Assess severity of vulnerability pattern"""
severity_map = {
"credential_exposure" : "critical" ,
"process_execution" : "high" ,
"network_calls" : "medium" ,
"file_access_risk" : "medium"
}
return severity_map.get(pattern_type, "low" )
LLM-Based Semantic Classification Use language models to understand vulnerability intent.
class SemanticClassifier :
def classify (self, code ):
"""Identify vulnerability intent through semantic analysis"""
intent_categories = [
"data_exfiltration" ,
"privilege_escalation" ,
"prompt_injection" ,
"supply_chain_risk" ,
"benign_risky_pattern"
]
classifications = {}
for category in intent_categories:
prompt = f"""Analyze this code for {category} intent:
{code}
Is there evidence of {category} ? (yes/no/unclear)
Confidence: 0-1
"""
result = self .llm_classify(prompt)
classifications[category] = {
"detected" : result["answer" ],
"confidence" : result["confidence" ]
}
return classifications
def llm_classify (self, prompt ):
"""LLM-based semantic analysis"""
pass
Vulnerability Category Framework Organize vulnerabilities into actionable categories.
class VulnerabilityCategory :
CATEGORIES = {
"data_exfiltration" : {
"description" : "Attempt to send data outside system" ,
"prevalence" : 0.133 ,
"examples" : ["send_logs" , "collect_files" , "api_exfil" ]
},
"privilege_escalation" : {
"description" : "Attempt to gain higher permissions" ,
"prevalence" : 0.118 ,
"examples" : ["sudo_access" , "admin_check" , "permission_bypass" ]
},
"malicious_intent" : {
"description" : "Clear evidence of deliberate harm" ,
"prevalence" : 0.052 ,
"examples" : ["backdoor" , "ransomware_pattern" , "botnet" ]
},
"prompt_injection" : {
"description" : "Vulnerability to LLM prompt injection" ,
"prevalence" : 0.045 ,
"examples" : ["unescaped_input" , "eval_user_input" ]
},
"supply_chain" : {
"description" : "Risk to dependency management" ,
"prevalence" : 0.035 ,
"examples" : ["typosquatting" , "dependency_confusion" ]
}
}
@staticmethod
def get_risk_level (vulnerability ):
"""Map vulnerability to risk level"""
if vulnerability in ["data_exfiltration" , "privilege_escalation" , "malicious_intent" ]:
return "high"
elif vulnerability in ["prompt_injection" ]:
return "medium"
else :
return "low"
Risk Scoring System Assign quantitative risk scores to skills.
class RiskScorer :
def compute_score (self, findings ):
"""Compute overall risk score 0-1"""
if not findings:
return 0.0
critical_count = len ([f for f in findings if f.get("severity" ) == "critical" ])
high_count = len ([f for f in findings if f.get("severity" ) == "high" ])
medium_count = len ([f for f in findings if f.get("severity" ) == "medium" ])
score = (
critical_count * 0.5 +
high_count * 0.3 +
medium_count * 0.1
) / len (findings)
return min (1.0 , score)
def requires_vetting (self, risk_score ):
"""Determine if skill requires manual review"""
return risk_score > 0.3
Skill Bundling Analysis Identify that executable scripts increase vulnerability risk.
class BundlingAnalysis :
def assess_bundling_risk (self, skill ):
"""Check if skill includes executable scripts"""
has_scripts = any (
script_ext in skill["files" ]
for script_ext in [".py" , ".sh" , ".js" , ".exe" ]
)
if has_scripts:
return {"has_scripts" : True , "risk_multiplier" : 2.12 }
else :
return {"has_scripts" : False , "risk_multiplier" : 1.0 }
Performance Characteristics
Detection coverage: 31,132 skills analyzed
Vulnerability prevalence: 26.1% contain at least one vulnerability
Detection precision: 86.7%
Detection recall: 82.5%
14 distinct vulnerability patterns identified
Deployment Recommendations
Scan all incoming skills before adding to marketplaceUse tiered trust levels : vetted, pending-review, blockedImplement capability-based permissions to limit vulnerable skillsMaintain signature database of known vulnerable patternsRequire security attestation from skill publishersArchive and track all security findings for auditing
Vetting Checklist
References
26.1% of agent skills contain exploitable vulnerabilities
Executable script bundling increases vulnerability risk 2.12x
Multi-stage detection (static + semantic) required for comprehensive coverage
Mandatory vetting essential before production deployment
