Skip to content
ai

MMLU

Massive Multitask Language Understanding

Definition

MMLU is a benchmark consisting of approximately 15,000 multiple-choice questions across 57 academic and professional subjects, from elementary mathematics to professional law and medicine. It measures a model's breadth of world knowledge and reasoning ability.

MMLU scores are widely reported in model release papers, though models trained on instruction-following datasets can achieve high scores without genuine understanding.

Can this ship? Are we exposed? Can you prove it?

Every CISO is asking these three questions. We answer them with facts, not guesses, from repository to runtime.