book a demo
book a demo
For the past three years, the same conversation has been had at every localization conference: “We tested an LLM for quality evaluations, and it seemed to work pretty well.”
Seemed. Compared to what? On which content? Who tested it, and what did they use as the answer key? Most of us have been improvising our own answers to these questions, one spreadsheet at a time.
In August 2026, the AMTA Quality Estimation Users Working Group — a group that includes people from Carnegie Mellon, the MQM Council, the European Commission's DG Translation, the IMF, and AWS — published something the industry has needed for a while: a shared, practical methodology for evaluating quality estimation (QE) systems. I want to walk through what the framework actually says here, and show how it maps directly onto how loc teams can benchmark AI in practice.
The central claim of the AMTA Principles is almost anticlimactic: QE performance is use-case and context-specific. A model that scores well on English-to-Spanish marketing copy can fall apart on German legal disclaimers translated by a different MT engine. Performance shifts by language pair, domain, content type, and even by which system produced the translation in the first place — and it also drifts over time as models and prompts get updated.
This sounds obvious once you say it out loud, but it cuts against how most of us have actually been testing AI. We run one pilot, on one dataset, and generalize from it to every language and content type we own. AMTA is explicit that this is the wrong move: avoid applying a result or a threshold broadly without validating it first. Test the model on your language pairs, your content types, your risk profile — not on a generic public benchmark someone else built.
The framework splits QE into three distinct use cases, and conflating them is where a lot of testing goes wrong:
Most enterprise loc teams live mostly in UC2 and UC3 — that's literally what "AI LQA" or "AI-assisted review" means in practice, and it's why the Recipe itself focuses there. Each use case needs its own gold standard and its own success metric. A model can annotate errors precisely and still be a mediocre routing tool, or vice versa — the two capabilities don't automatically transfer.
Strip out the statistics and the six-step Recipe is basically what an experienced Language Quality Manager already does intuitively — it just makes things rigorous and repeatable:
Step 4 is where the framework earns its keep, and it's worth a concrete example straight from the document. Say a model scores every segment 0–100. If your review budget only covers 60% of segments, the data might show a threshold of 91 gets you there — but around 9 error-containing segments per 100 will still slip through uncaught. Push the threshold to 93, and you catch more errors, but now you're reviewing 71% instead of 60%. Neither number is "correct" — it depends on whether those 9 slipped-through errors are typos in a blog post or a wrong dosage instruction in a pharma insert.
The framework calls out two failure modes that matter a lot now that most "QE" in practice is an LLM being prompted to judge translation quality:
There's also a subtler trap in the appendices worth flagging: content that's already very good is often the hardest to QE. When a translation is 90%+ error-free to start with, most QE systems' detection curve looks barely better than a coin flip, simply because there's so little signal left to find.
Meanwhile, heavily post-edited content can make a QE system look like it's failing when the real issue is over-editing or a genuinely bad MT engine upstream. Both are diagnosis traps, not model failures, and the Recipe walks through how to tell them apart.
We’ve built ContentQuo Test before AMTA has published this framework, so reading the final Recipe felt less like discovering something new and more like getting a formal write-up of a process we'd already converged on in our automated benchmarking tool:
The bigger point AMTA is making — and the one I'd underline for any Localization leader reading this in 2026 or 2027 — is that evaluating AI quality is now a core competency, not a one-time vendor bake-off. Models change, prompts age, content evolves, and a QE solution that was accurate in Q1 can be stale by Q3. Owning that evaluation loop, on your own data, with your own risk tolerances, is what separates teams that can explain why they're using a given model from teams that are just praying the vendor's demo held up.
That's really the core philosophy behind ContentQuo: give localization teams the infrastructure to build, test, and run this kind of rigorous, repeatable, highly multilingual AI evaluation programs themselves, instead of outsourcing the judgment call to whoever's selling the model. And now that we’ve got cool new tools that enable your team to easily automate both the benchmarking and the development of AI QE, what’s stopping you from trying those out?
Choose the best QE vendor for your Loc program.
Automated, AMTA-compliant benchmarking of AI QE, AI LQA, and AI Translation on your own content and with your own quality standards.
