Output auditing

Output auditing is checking what the system actually generates, not just the code that renders it. Because the output is produced at runtime and changes with every model and prompt change, a one-time review is not enough. Auditing the output is how you catch accessibility and bias problems that only appear in what the model says.

Key points

Output auditing checks what the system actually generates, not just the code that renders it.
The interface can be perfectly accessible while the generated content is not, so audit the real output.
Sample deliberately, because the output is a population of varied answers.
Check two layers, accessibility and fairness toward disabled people, and judge alt-text quality by hand.
Make it continuous, with a model or prompt change triggering a re-audit.

Why audit the output

The interface can be perfectly accessible while the generated content is not. A model defaults to inaccessible structure unless it is told otherwise, which a Microsoft evaluation measured as only 12 percent of unguided output passing automated checks. Auditing samples the real output and asks whether it is structured, described, and fair, rather than assuming the template guarantees it.

Sample, because output is a population

You cannot audit every possible answer, so sample deliberately. Cover representative prompts, long and short outputs, error and refusal states, and outputs that include images, tables, or code. This is the same idea as in Testing methodology, applied to the content the model produces.

What to check in the output

There are two layers. The first is accessibility. Check whether generated text has real structure, whether generated images carry a reviewed and context-aware description, and whether generated links and files are real elements. AI-written alt text in particular tends to be generic and formulaic and to miss the purpose of the image, and a stored description is not automatically right for the context. The second layer is fairness. Check whether the output is biased against or demeaning toward disabled people, which ties back to bias and representation.

Use AI to help, but verify

AI can help review its own output at scale, for example coaching better audio descriptions or drafting alt text for a person to check. But AI auditing of AI is not self-sufficient. Audit tools and LLM reviewers miss context and produce confident but wrong findings, so a human stays in the loop on quality, as the automated tools page explains.

Make it continuous

Wire output auditing into a schedule rather than running it once. A model change or a prompt change is a trigger to re-sample and re-audit, which connects to feedback loops and the evaluation loop in Foundations.

Audience: Engineer

Build a small harness that audits a sample of real prompts on a schedule, runs the automated structure checks, and queues anything subjective for a person. The shape is simple.

// Audit a sample of real prompts, not a single example
const prompts = loadSampledPrompts(); // long, short, error, table, and image cases

for (const prompt of prompts) {
  const html = await renderAssistantReply(prompt);
  const violations = await runAxe(html);          // structure, labels, contrast
  const images = extractImages(html);
  queueForHumanReview(prompt, violations, images); // a person judges alt quality and fairness
}

The automated step catches missing structure and labels. The human step is where alt-text quality and fairness get judged, because those are exactly what the tools cannot decide.

Audience: Accessibility Specialist

Audit the output in two layers and keep them separate in your report. The accessibility layer asks whether the generated content is structured, labeled, and navigable. The fairness layer asks whether it treats disabled people equitably, which needs disability-related cases in the sample, not just a generic spread. Judge alt-text quality by hand on a sample, since an automated check only confirms an alt attribute exists.

Audience: Product Manager

Output auditing is the practice that keeps a launched AI product honest, so schedule it rather than treating launch as the end. Tie an audit to every model and prompt change, and budget for the human review time, because the parts that matter most for accessibility and fairness are the parts a tool cannot sign off. A dated audit with a stated sample is the artifact to ask for.