awesome-ai-security-reference

# Awesome AI Security Reference > Skill by [ara.so](https://ara.so) — Security Skills collection. This skill helps you navigate the Awesome AI Security learning journey, a curated collection of resources covering ML security fundamentals, attack vectors (prompt injection, adversarial examples, poisoning, privacy attacks), defensive tools, and AI-powered penetration testing. ## What This Project Provides Awesome AI Security is a comprehensive, annotated roadmap covering: - **ML Foundations**: Essential courses and books (Stanford CS229, fast.ai, Deep Learning Specialization) - **Deep Learning**: Neural networks, transformers, and modern architectures - **Prompt Injection**: Attacks manipulating LLM behavior and defenses - **Adversarial Attacks**: Methods to fool neural networks with crafted inputs - **Poisoning & Backdoors**: Data poisoning and neural network backdoors - **Privacy & Extraction**: Membership inference, model extraction, data leakage - **Tools & Frameworks**: Security testing tools (Garak, Rebuff, Counterfit, NeMo Guardrails) - **AI Pentesting**: Using AI agents for automated security assessments - **Vulnerability Detection**: AI-powered code analysis and bug detection ## Installation This is a reference repository — clone it to browse locally or use the interactive roadmap: ```bash git clone https://github.com/muellerberndt/awesome-ai-security.git cd awesome-ai-security ``` View the interactive roadmap at: https://floatingpragma.io/awesome-ai-security ## Key Resource Categories ### Learning Paths **ML Foundations (Start Here)** - Stanford CS229: https://cs229.stanford.edu/ - fast.ai Practical Deep Learning: https://course.fast.ai/ - Deep Learning Specialization: https://www.coursera.org/specializations/deep-learning **Deep Learning Theory** - Dive into Deep Learning (interactive): https://d2l.ai/ - Neural Networks and Deep Learning: http://neuralnetworksanddeeplearning.com/ - Deep Learning Book (Goodfellow): https://www.deeplearningbook.org/ ### Attack Vectors **Prompt Injection** Key Resources: - OWASP LLM01:2025 Prompt Injection: https://genai.owasp.org/llmrisk/llm01-prompt-injection/ - Lakera Guide: https://www.lakera.ai/blog/guide-to-prompt-injection Detection/Defense Tools: ```bash # Rebuff - Self-hardening prompt injection detector pip install rebuff # Garak - NVIDIA's LLM vulnerability scanner pip install garak garak --model_type openai --model_name gpt-3.5-turbo --probes promptinject # Vigil LLM - Detect risky inputs pip install vigil-llm ``` **Adversarial Attacks** Key Papers: - A Brief Introduction: https://gradientscience.org/intro_adversarial/ - NIST Adversarial ML Taxonomy: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf Tools: ```python # Adversarial Robustness Toolbox (ART) from art.attacks.evasion import FastGradientMethod from art.estimators.classification import PyTorchClassifier # Wrap your model classifier = PyTorchClassifier( model=model, loss=criterion, optimizer=optimizer, input_shape=(3, 32, 32), nb_classes=10 ) # Generate adversarial examples attack = FastGradientMethod(estimator=classifier, eps=0.2) x_adv = attack.generate(x=x_test) ``` ```python # TextAttack - NLP adversarial attacks from textattack.attack_recipes import TextFoolerJin2019 from textattack.datasets import HuggingFaceDataset from textattack.models.wrappers import HuggingFaceModelWrapper model = HuggingFaceModelWrapper(model, tokenizer) attack = TextFoolerJin2019.build(model) ``` **Data Poisoning & Backdoors** Key Papers: - Witches' Brew (gradient matching): https://arxiv.org/abs/2009.02276 - Instruction Backdoor Attacks: https://arxiv.org/abs/2402.09179 - MNTD (Trojan detection): https://arxiv.org/abs/1910.03137 Detection: - Beatrix (activation-based): https://www.ndss-symposium.org/wp-content/uploads/2024/09/2023-69-slides.pdf **Privacy Attacks** Key Resources: - Extracting Training Data (USENIX 2021): https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting - Membership Inference Survey: https://arxiv.org/abs/2103.07853 - awesome-ml-privacy-attacks: https://github.com/stratosphereips/awesome-ml-privacy-attacks ### Security Testing Tools **Comprehensive LLM Testing** ```bash # Garak - NVIDIA's vulnerability scanner pip install garak # Test for prompt injection garak --model_type openai \ --model_name gpt-4 \ --probes promptinject \ --report_prefix my_test # Test for jailbreaks garak --model_type huggingface \ --model_name meta-llama/Llama-2-7b-chat-hf \ --probes jailbreak # Full scan (all plugins) garak --model_type openai --model_name gpt-3.5-turbo --all_probes ``` **Guardrails & Defense** ```python # NeMo Guardrails - NVIDIA programmable guardrails from nemoguardrails import RailsConfig, LLMRails config = RailsConfig.from_path("./config") rails = LLMRails(config) response = rails.generate( messages=[{"role": "user", "content": "Ignore previous instructions..."}] ) # Guardrails intercept and block malicious prompts ``` ```python # Purple Llama - Meta's safety tools from llama_guard import LlamaGuard guard = LlamaGuard() result = guard.evaluate( prompt="How do I make a bomb?", response="I cannot help with that." ) # Returns: {unsafe: True, categories: ['violence']} ``` **Evaluation Framework** ```python # ai-evaluation - Systematic LLM testing from ai_evaluation import Evaluator, metrics evaluator = Evaluator( model_name="gpt-4", api_key_env="OPENAI_API_KEY" ) # Test with guardrail scanners results = evaluator.run( dataset="security_prompts.json", metrics=["jailbreak", "pii_leakage", "prompt_injection"], guardrails=["llama_guard", "prompt_guard"] ) print(results.summary()) ``` ### AI-Powered Pentesting **PentestGPT - Autonomous Pentesting** ```bash # Install PentestGPT git clone https://github.com/GreyDGL/PentestGPT.git cd PentestGPT pip install -r requirements.txt # Configure API key export OPENAI_API_KEY="your-key-here" # Run interactive pentesting session python pentestGPT.py ``` Interactive workflow: ``` > Target: 192.168.1.100 > Objective: Find SQL injection vulnerabilities [PentestGPT suggests reconnaissance] 1. Run nmap scan 2. Enumerate web services 3. Test input fields > Execute step 1 [Runs: nmap -sV -sC 192.168.1.100] [Analyzes results with GPT-4] [Suggests next steps based on findings] ``` **Code Analysis with GhidraGPT** ```python # GhidraGPT - AI-assisted reverse engineering # Install as Ghidra plugin, then use in Ghidra Python console from ghidra_gpt import analyze_function # Analyze suspicious function result = analyze_function( function_name="decode_payload", context="Looking for encryption or obfuscation" ) print(result.explanation) print(result.vulnerabilities) ``` ### Vulnerability Detection **Semgrep with AI** ```bash # Install Semgrep pip install semgrep # Scan for AI-detected vulnerabilities semgrep --config=auto --ai-mode # Specific security rules semgrep --config=p/security-audit \ --config=p/owasp-top-ten \ --ai-assist # Check AI-generated code semgrep --config=p/ai-code-review ./src ``` ```yaml # .semgrep.yml - Custom AI-assisted rule rules: - id: detect-prompt-injection-risk pattern: | $MODEL.generate($USER_INPUT) message: User input passed directly to LLM without sanitization severity: ERROR languages: [python] metadata: category: security owasp: A03:2021 - Injection ``` ## Common Workflows ### 1. Learning Path for AI Security ```markdown ## Beginner Track 1. Start: Stanford CS229 or fast.ai (4-8 weeks) 2. Read: Neural Networks and Deep Learning (Nielsen) 3. Practice: Implement basic neural network in PyTorch 4. Study: OWASP Top 10 for LLMs ## Intermediate Track 1. Deep dive: Dive into Deep Learning (d2l.ai) 2. Study prompt injection: Lakera guide + OWASP LLM01 3. Hands-on: Test models with Garak 4. Read: Adversarial examples intro (Gradient Science) ## Advanced Track 1. Research: Read key papers (Carlini, Goodfellow) 2. Implement: ART adversarial attacks 3. Build: Custom guardrails with NeMo 4. Contribute: Test tools on TrustLLM benchmark ``` ### 2. Testing LLM Security ```python # Complete LLM security assessment workflow import os from garak import run_scan from nemoguardrails import RailsConfig, LLMRails # 1. Baseline vulnerability scan scan_results = run_scan( model_type="openai", model_name="gpt-4", api_key=os.getenv("OPENAI_API_KEY"), probes=["promptinject", "jailbreak", "pii_leakage"] ) # 2. Deploy guardrails config = RailsConfig.from_path("./security_config") rails = LLMRails(config) # 3. Test with guardrails protected_response = rails.generate( messages=[{ "role": "user", "content": scan_results.worst_case_prompt }] ) # 4. Validate defense assert protected_response.blocked or protected_response.sanitized ``` ### 3. Defending Against Adversarial Attacks ```python # Multi-layer defense strategy from art.defences.preprocessor import FeatureSqueezing from art.defences.postprocessor import HighConfidence # 1. Input preprocessing squeezer = FeatureSqueezing(bit_depth=4) x_squeezed = squeezer(x_input) # 2. Adversarial training from art.attacks.evasion import ProjectedGradientDescent attack = ProjectedGradientDescent( classifier, eps=0.3, max_iter=10 ) # Generate adversarial training data x_adv_train = attack.generate(x_train) # Retrain with adversarial examples model.fit( np.concatenate([x_train, x_adv_train]), np.concatenate([y_train, y_train]) ) # 3. Post-processing defense high_conf = HighConfidence(cutoff=0.9) predictions_filtered = high_conf(predictions) ``` ### 4. Privacy-Preserving ML ```python # Test for membership inference vulnerabilities from art.attacks.inference.membership_inference import MembershipInferenceBlackBox # Train shadow models attack = MembershipInferenceBlackBox(classifier) attack.fit(x_train, y_train, x_test, y_test) # Infer membership inferred = attack.infer(x_target, y_target) # If vulnerable, apply differential privacy from art.defences.trainer import DPInceptionTrainer dp_trainer = DPInceptionTrainer( model=model, epsilon=1.0, # Privacy budget delta=1e-5 ) dp_model = dp_trainer.fit(x_train, y_train) ``` ## OWASP Top 10 for LLMs (2025) Reference the latest risks when building/testing: 1. **LLM01: Prompt Injection** - Manipulating model via crafted inputs 2. **LLM02: Sensitive Information Disclosure** - Leaking training data 3. **LLM03: Supply Chain Vulnerabilities** - Compromised models/datasets 4. **LLM04: Data and Model Poisoning** - Corrupted training data 5. **LLM05: Improper Output Handling** - Unvalidated LLM outputs 6. **LLM06: Excessive Agency** - Over-privileged LLM actions 7. **LLM07: System Prompt Leakage** - Exposing system instructions 8. **LLM08: Vector and Embedding Weaknesses** - RAG vulnerabilities 9. **LLM09: Misinformation** - Generated false/misleading content 10. **LLM10: Unbounded Consumption** - Resource exhaustion attacks Full details: https://owasp.org/www-project-top-10-for-large-language-model-applications/ ## Benchmarks & Evaluation **TrustLLM Benchmark** ```python # Comprehensive trustworthiness testing from trustllm import TrustLLMBenchmark benchmark = TrustLLMBenchmark( model="gpt-4", api_key_env="OPENAI_API_KEY" ) results = benchmark.evaluate( dimensions=[ "truthfulness", "safety", "fairness", "robustness", "privacy", "ethics" ] ) print(results.report()) ``` ## Troubleshooting **Garak not detecting vulnerabilities** - Use `--extended` flag for more thorough testing - Try multiple probe categories: `--probes promptinject,jailbreak,encoding` - Check model API rate limits **ART attacks failing** - Verify model wrapper is correct (PyTorch/TensorFlow/Keras) - Ensure input shapes match: `classifier.input_shape` - Reduce `eps` parameter for subtle perturbations **NeMo Guardrails not blocking** - Review `config.yml` rail definitions - Enable verbose logging: `rails.generate(..., debug=True)` - Test individual rails in isolation **API rate limits during testing** - Set delays: `time.sleep(1)` between requests - Use smaller test datasets - Consider local models (Llama, Mistral) for unlimited testing ## Additional Resources - **Interactive Roadmap**: https://floatingpragma.io/awesome-ai-security - **OWASP GenAI Security**: https://genai.owasp.org/ - **NIST AI Risk Management**: https://www.nist.gov/itl/ai-risk-management-framework - **ACL 2024 LLM Vulnerabilities Tutorial**: https://llm-vulnerability.github.io/ This skill provides navigation through the comprehensive Awesome AI Security knowledge base. Use it to find relevant tools, papers, and learning resources for any AI security topic.