This paper proposes a framework called LEST (LLM Evaluation through Specification-driven Testing) to evaluate the safety claims of large language models. It uses documented safety claims as specifications for black-box testing, mutating prompts to search for counterexamples, and manually validating each candidate. The authors evaluate LEST using two models and find that automated evaluation overestimates behavioral failures, highlighting the need for manual validation.