Promptfoo Is Joining OpenAI

Happy to answer questions.

The one I'd ask if I were reading this: what happens to Promptfoo open source? We're going to keep maintaining it. The repo will stay public under the same license, we will continue to support multiple providers, and we'll keep reviewing PRs and cutting releases.

We started Promptfoo because there was no good way to test AI systems before shipping them. That turned into evals, then red teaming, then a broader security platform. We're joining OpenAI because this work has more impact closer to the model and infrastructure layers.

Ask me anything.

I'm curious about one thing though: the blog post mentions integration at the 'model and inference layer,' not just bolting it onto Frontier as a feature. That's a deeper integration than most acquisitions.

In practice, does that mean the security testing becomes invisible for OpenAI-hosted models? Because if so, that's great for OpenAI customers but creates an interesting gap for teams building on Claude, Gemini, or local models. They lose their independent testing tool.

The other thing I keep thinking about: promptfoo solves pre-deployment testing really well. But once agents are running in production making tool calls, the attack surface changes. A prompt that passes red-teaming can still be exploited through indirect injection via tool outputs.

Has anyone been working on runtime monitoring for agents? Not just testing before you ship, but watching what agents actually do during execution?