MAGS, a multi-agent framework, generates executable programs with formal safety guarantees for LLM-generated code. It uses Dafny as a verification-aware intermediate representation, formalizes APIs and safety requirements, and achieves a 100% success rate in producing safe programs for 220 examples.