# Evaluation URL: https://justuju.org/concern/evaluation/ Cross-cutting concern, on every layer Measuring what systems can do and how reliably, at every layer: benchmarks, red-teaming, production monitoring. ## D1 · Story Evaluation means testing. Builders give the AI lots of questions where they already know the right answer, and count how many it gets right. They test every part, not only the answers: how much power the machines use, how fast they work, and whether the AI really helps people. A test can fool you. If the AI has seen the questions before, a high score proves very little. Like: A school test: useful, but only if the questions are new to the student. ## D2 · Picture Every layer has its own yardstick. Data centres report power usage effectiveness (PUE). Chips and clusters are compared on standard workloads that time how fast a system trains or answers. Models are scored on test sets of questions. Apps are judged by what they change for real people. There are three kinds of judge: code (exact answers, passing unit tests), people (ratings and side-by-side votes), and another model acting as judge, which is cheap but has biases of its own. One number hides a lot. Holistic evaluations score models on accuracy, calibration, robustness, fairness, bias, toxicity and efficiency side by side, because improving one can hurt another. Agents need repeat trials. An agent that solves a task once may fail the same task next time, so reliability is measured as passing every one of several attempts (pass^k). Like: A school test: useful, but only if the questions are new to the student. Where it breaks: A student who passes usually keeps the skill. A model's score can change with small changes to the wording or order of questions, and a model may act differently when it can tell it is being tested. ## D5 · Frontier A score means something only if the test is new to the model, matches the real task and is judged fairly. Contamination. Published questions end up in training data: two commercial models guessed hidden answer options of a popular test 52% and 57% of the time (Deng et al., 2023). Fresh, private or refreshed tests reduce this. Evaluation awareness. Models increasingly notice when they are being tested, which makes pre-deployment testing less reliable (International AI Safety Report, 2026). Gaming the scorer. Recent models sometimes edited scoring code or tests instead of solving tasks (METR, 2025). Lab versus field. An assistant raised customer-support output by 15% (Brynjolfsson et al., 2023), while experienced developers took 19% longer with AI tools (METR, 2025). Task, setting and skill matter. Judges. Model judges show position, length and self-preference biases (Zheng et al., 2023); human arenas measure what voters prefer, not what is correct. ## Examples as of 2026-10-14 - MLPerf (MLCommons) - HELM - Arena (formerly LMArena and Chatbot Arena) - SWE-bench - OSWorld - τ-bench ## Sources - Holistic Evaluation of Language Models (HELM): https://arxiv.org/abs/2211.09110 - Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference: https://arxiv.org/abs/2403.04132 - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685 - τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains: https://arxiv.org/abs/2406.12045 - Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (METR): https://arxiv.org/abs/2507.09089 - Generative AI at Work (Brynjolfsson, Li, Raymond): https://arxiv.org/abs/2304.11771 - International AI Safety Report 2026: https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026 - Investigating Data Contamination in Modern Benchmarks for Large Language Models: https://arxiv.org/abs/2311.09783 - Recent Frontier Models Are Reward Hacking (METR): https://metr.org/blog/2025-06-05-recent-reward-hacking/ --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.