A new dataset, LLMail-Inject, is introduced to evaluate the effectiveness of LLMs against prompt injection attacks. The dataset contains 208,095 human-written attack prompts collected through a public red-teaming challenge. Existing benchmarks are shown to be overfit to static attacks, and models that perform well on them struggle to achieve low ASR on LLMail-Inject.