PromptInject

Why we included this project

At some point, anyone running a customer-facing LLM has to wonder how easily a stranger can talk the model out of its instructions. PromptInject turns that worry into a measurable test: it builds base prompts from modular pieces, then feeds in malicious user inputs to exercise two attack families, goal hijacking and prompt leaking. Security engineers and red-teamers can reuse it to get a number for how robust their own system prompt really is to 'ignore previous instructions' style inputs, instead of trusting a gut feeling. The attacks come from a NeurIPS ML Safety paper, so the definitions are published research rather than ad hoc tricks. Do expect to adapt the code to newer models, since it was built around GPT-3-era APIs; treat it as the starting point for an ongoing evaluation harness rather than a plug-and-play tool.