prompt injection
A security risk where malicious instructions manipulate agent behavior. The newsletter references it in the context of skill/repo scanning tools.
Key Highlights
- Prompt injection is a core agent-security risk where malicious content overrides intended instructions and can trigger unsafe actions.
- Newsletter examples connect prompt injection to supply-chain attacks, data exfiltration, malicious websites, and unsafe enterprise connectors.
- The strongest pattern across mentions is defense in depth: model safeguards plus sandboxing, permissions, egress controls, and content isolation.
- Anthropic-cited results suggest low single-attack success is still not enough, because adaptive attacks can materially raise compromise rates.
- Tools like NVIDIA SkillSpector show growing demand for scanning skills and repos for prompt injection and related hidden-instruction risks.
Prompt injection
Overview
Prompt injection is a security failure mode in which malicious or untrusted content causes an AI system to follow attacker-supplied instructions instead of the product’s intended rules, user intent, or system constraints. It can happen directly through user input or indirectly through external content such as web pages, emails, documents, GitHub issues, or tool outputs. In agentic products, the risk is especially important because the model may not just generate text—it may also read sensitive data, call tools, send messages, modify code, or take actions in real environments.For AI Product Managers, prompt injection matters because it is not just a model-quality issue; it is a product, platform, and trust issue. The newsletter coverage frames it as a recurring risk across coding agents, enterprise copilots, browser agents, and repo/skill ecosystems. The practical lesson is that model robustness alone is insufficient: strong defenses usually combine model-level mitigations with permissioning, sandboxing, egress controls, content isolation, and careful tool design to reduce blast radius when attacks succeed.
Key Developments
- 2026-03-07: Adnan Khan’s attack on Cline’s issue triage workflow showed how a prompt injection embedded in a GitHub issue title could trigger a broader supply-chain compromise, ultimately enabling malicious NPM releases.
- 2026-03-12: OpenAI highlighted design techniques for building AI agents that resist prompt injection, emphasizing architectural and behavioral mitigations rather than relying on prompting alone.
- 2026-05-26: Reporting on Microsoft Copilot Cowork described a prompt-injection exfiltration path involving agent-sent emails, externally rendered images, and pre-authenticated OneDrive links.
- 2026-06-01: Anthropic argued that prompt injection cannot be solved by model defenses alone, citing Claude Opus 4.7 results of about 0.1% success on single attempts and about 5–6% after 100 adaptive attempts, and stressing containment controls.
- 2026-06-19: Anthropic reiterated those containment lessons across Claude products, pairing environment controls with system prompts, classifiers, probes, training, and limits on access to external content and tools.
- 2026-06-23: Simon Willison highlighted research on role confusion, where models mistake privileged role-tagged text for user text based on style; the paper also showed that destyling can sharply reduce attack success.
- 2026-08-08: Boris Cherny said that combining model training, input probes, and an intent-checking classifier could drive indirect prompt injection close to zero on unseen attacks in testing.
- 2026-08-10: Boris Cherny described prompt injection as one of the most common ways scammers target both people and agents, including through malicious website text aimed at stealing credentials.
- 2026-08-24: Claire Vo raised prompt injection as a core product question for agents, alongside how source-data indexing, ephemeral connectors, and stored data should be communicated to users.
- 2026-09-03: NVIDIA SkillSpector was noted as a tool that scans skills or GitHub repos for prompt injection, hidden instructions, data exfiltration, supply-chain risk, and MCP-related issues.
Relevance to AI PMs
1. Design for compromised inputs, not perfect obedience. Assume websites, documents, emails, repo metadata, and tool outputs may contain hostile instructions. Product requirements should specify what the agent is allowed to read, what actions require confirmation, and what must remain inaccessible even if the model is manipulated.2. Treat prompt injection as a systems risk with measurable blast radius. PMs should define security boundaries around connectors, file access, network egress, messaging, and code execution. Success metrics should include not only task completion but also exfiltration resistance, unsafe action rates, and recovery behavior under adversarial inputs.
3. Invest in layered defenses and evaluation. The mentions consistently point to multi-layer mitigations: model training, classifiers, probes, content filtering, sandboxing, permission controls, and repo/skill scanning. PMs should ensure red-teaming and benchmark coverage include indirect prompt injection, adaptive attacks, and supply-chain pathways—not just benign QA prompts.
Related
- OpenAI: Covered agent design practices for resisting prompt injection, reinforcing the need for architectural mitigations.
- Cline / Claude Code: Coding-agent contexts where prompt injection can affect automated triage, code execution, and release workflows.
- Adnan Khan: Documented a high-impact real-world attack chain against Cline’s production release process.
- Microsoft Copilot Cowork / OneDrive: Example of enterprise-agent exfiltration risk when prompt injection intersects with connectors, email, and file-sharing mechanisms.
- Anthropic / Claude / Claude Opus 4.7 / Mythos Preview: Frequently referenced in discussions of layered defenses, empirical attack success rates, and blast-radius containment.
- Role confusion / destyling: Research concepts tied to why prompt injection works and how formatting/style-based mitigations may help.
- Boris Cherny / Claire Vo: Commentary from operators and builders on the prevalence of the problem and the need for practical defenses.
- NVIDIA SkillSpector: Security scanning tool focused on prompt injection and adjacent risks in skills and repositories.
- browser-use/video-use: Relevant as an example of agentic workflows that process external content, where prompt-injection risk can emerge.
Frequently asked questions
What is prompt injection?
A security risk where malicious instructions manipulate agent behavior. The newsletter references it in the context of skill/repo scanning tools.
What should AI product managers know about prompt injection?
Prompt injection is a core agent-security risk where malicious content overrides intended instructions and can trigger unsafe actions. Newsletter examples connect prompt injection to supply-chain attacks, data exfiltration, malicious websites, and unsafe enterprise connectors. The strongest pattern across mentions is defense in depth: model safeguards plus sandboxing, permissions, egress controls, and content isolation.
Newsletter Mentions (10)
“NVIDIA SkillSpector scans skills or GitHub repos for prompt injection, data exfiltration, supply-chain risk, hidden instructions, and MCP-related risks, while Phone Harness controls iPhones through Mac iPhone mirroring and Android devices through ADB.”
Five free, open-source GitHub repos—Peter Yang’s No AI Slop Skill, TryComp AI CRM, browser-use/video-use, NVIDIA SkillSpector, and ShawnPana/phone-harness—cover AI writing cleanup, agent-managed CRM workflows, agentic video editing, skill security scanning, and real-phone automation. No AI Slop Skill installs with npx skills add plus the GitHub link; it removes AI-writing patterns while retaining a human-written outline, voice, and core points. TryComp AI CRM requires Bun and Docker; its local setup uses bun install , docker compose up -d , bun run db deploy , bun run db seed , and bun run dev , then runs on localhost:3000 with its API on localhost:3001 . browser-use/video-use uses an agent workflow with FFmpeg and requests an 11 Labs API key when needed; NVIDIA SkillSpector scans skills or GitHub repos for prompt injection, data exfiltration, supply-chain risk, hidden instructions, and MCP-related risks, while Phone Harness controls iPhones through Mac iPhone mirroring and Android devices through ADB.
“#6 𝕏 claire vo 🖤 questioned whether agents can work without indexing source data, how ephemeral connectors and stored data should be presented to users, and how to prevent prompt injection.”
#6 𝕏 claire vo 🖤 questioned whether agents can work without indexing source data, how ephemeral connectors and stored data should be presented to users, and how to prevent prompt injection. She characterized an unnamed harness as immature and lacking user-empathetic evaluation, while noting that self-serve data deletion shipped but promised email follow-up did not occur.
“He described prompt injection as the most common way scammers attack people and agents, including by using malicious website text to trick agents into disclosing credentials.”
𝕏 Boris Cherny shared a prompt-injection benchmark created by an unnamed independent researcher and said Anthropic has largely solved the threat in practice for Claude models, with red-team testing producing similar results. He described prompt injection as the most common way scammers attack people and agents, including by using malicious website text to trick agents into disclosing credentials.
“Boris Cherny said stacking model training, input probes, and an intent-checking classifier can bring indirect prompt injection to ~0 on unseen attacks—a result he did not expect a year ago.”
#6 𝕏 Boris Cherny said stacking model training, input probes, and an intent-checking classifier can bring indirect prompt injection to ~0 on unseen attacks—a result he did not expect a year ago. He also announced that auto mode would become the default in Claude Code the following week.
“Simon Willison Prompt Injection as Role Confusion - Discussion of a paper showing models confuse privileged role-tagged text with user text based on style, enabling serious jailbreaks; the authors demonstrate that 'destyling' text can dramatically reduce attack success.”
This item discusses prompt injection defenses and a specific failure mode called role confusion.
“Telemetry showed users approved ~93% of permission prompts, Claude Code auto mode blocks roughly 83% of overeager behaviors before execution, and Claude Opus 4.7 resists prompt-injection with about 0.1% success on single attempts and ~5–6% after 100 adaptive attempts.”
📝 Anthropic Engineering How we contain Claude across products - Anthropic has deployed Claude across claude.ai, Claude Code, and Claude Cowork while containing blast radius via environment controls (sandboxes, VMs, filesystem/egress limits), model-layer controls (system prompts, classifiers, probes, training), and restricting external-content/tool access, noting Claude Mythos Preview was judged too risky to ship in April 2026. Telemetry showed users approved ~93% of permission prompts, Claude Code auto mode blocks roughly 83% of overeager behaviors before execution, and Claude Opus 4.7 resists prompt-injection with about 0.1% success on single attempts and ~5–6% after 100 adaptive attempts.
“They acknowledge model defenses aren’t perfect—Claude Opus 4.7 shows ≈0.1% attack success on single prompt-injection attempts and ≈5–6% after 100 adaptive attempts—cited Mythos Preview as too high a blast radius to ship in April 2026, and argue combined environment, model, and external-content controls are necessary to cap agents’ blast radius.”
Anthropic ships Claude Code auto mode #1 📝 Anthropic Engineering How we contain Claude across products - Anthropic says it has shipped claude.ai, Claude Code, and Claude Cowork and moved from human-in-the-loop approvals—which users accepted about 93% of the time, producing approval fatigue—toward containment (sandboxes, VMs, egress controls) and automated defenses like Claude Code auto mode, which catches roughly 83% of overeager behaviors. They acknowledge model defenses aren’t perfect—Claude Opus 4.7 shows ≈0.1% attack success on single prompt-injection attempts and ≈5–6% after 100 adaptive attempts—cited Mythos Preview as too high a blast radius to ship in April 2026, and argue combined environment, model, and external-content controls are necessary to cap agents’ blast radius.
“#8 📝 Simon Willison Microsoft Copilot Cowork Exfiltrates Files - A report describes how Microsoft Copilot Cowork allowed agent-sent emails to leak data via externally rendered images and pre-authenticated OneDrive links, creating a path for prompt-injection exfiltration.”
#8 📝 Simon Willison Microsoft Copilot Cowork Exfiltrates Files - A report describes how Microsoft Copilot Cowork allowed agent-sent emails to leak data via externally rendered images and pre-authenticated OneDrive links, creating a path for prompt-injection exfiltration. The post highlights the continued challenge of designing agentic systems that don't enable attackers to extract sensitive data.
“Designing AI agents to resist prompt injection - This post describes techniques for designing AI agents that are robust against prompt injection attacks, outlining security practices and mitigations.”
Today's top 25 insights for PM Builders, ranked by relevance from Blogs, X, LinkedIn, and YouTube. #10 𝕏 Google Research partnered with @BIDMC_Medicine to pilot AMIE, a conversational AI for clinical reasoning, and in a real-world study found it to be safe, feasible, and well-received by patients. #11 📝 OpenAI News Designing AI agents to resist prompt injection - This post describes techniques for designing AI agents that are robust against prompt injection attacks, outlining security practices and mitigations. It focuses on architecture and behavioral approaches to reduce the risk of maliciously crafted inputs influencing agent behavior.
“#19 📝 Simon Willison Clinejection — Compromising Cline’s Production Releases just by Prompting an Issue Triager - Adnan Khan details an attack chain where a prompt injection in a GitHub issue title against an AI-powered triage workflow led to a cache poisoning attack that allowed publishing malicious NPM releases.”
GenAI PM Daily March 07, 2026 GenAI PM Daily 🎧 Listen to this brief 3 min listen Today's top 25 insights for PM Builders, ranked by relevance from LinkedIn, YouTube, X, and Blogs. #18 📝 Simon Willison Agentic manual testing - A guide explaining that coding agents' defining capability is executing the code they write, and emphasizing the necessity of running generated code to verify correctness. The post argues that agents can iterate until code works, but humans should not assume generated code functions without execution. #19 📝 Simon Willison Clinejection — Compromising Cline’s Production Releases just by Prompting an Issue Triager - Adnan Khan details an attack chain where a prompt injection in a GitHub issue title against an AI-powered triage workflow led to a cache poisoning attack that allowed publishing malicious NPM releases.
Related
An AI company mentioned for releasing a report on model behaviors and for internal tool use in later context. Relevant for PMs tracking safety reporting and frontier model operations.
Anthropic’s coding agent/environment used here to control physical tools in a long-running home experiment.
OpenAI is an AI company building foundation models and consumer products like ChatGPT. In this newsletter it is discussed in connection with GPT-6, Dots, ChatGPT for Teens, and College Planner.
Anthropic’s AI assistant, referenced here through new dashboard, motion, docs, slides, and design capabilities.
A creator and product thinker sharing AI usage tracking and product documentation workflows. Relevant to PMs focused on measurement and prompt-to-spec workflows.
A builder or AI practitioner mentioned for using Claude to automate Slack outreach and response tracking. This is relevant to PMs as an example of agent-driven workflow automation.
A Claude model version referenced for its prompt-injection resistance metrics. It serves as a benchmark example of model-layer defenses being strong but not sufficient on their own.
Stay updated on prompt injection
Get curated AI PM insights delivered daily — covering this and 1,000+ other sources.
Subscribe Free