# Testing methodology

Testing an AI product for accessibility is different from testing a static page. The output is generated at runtime and changes over time, so you are testing a moving population rather than a fixed artifact. This page sets out the overall approach, and the pages that follow cover each tier in detail.

## Key points

- Testing an AI product means testing a moving population of generated output, not a fixed page.
- Sample deliberately across representative prompts, edge cases, long and short answers, and error and refusal states, and record the method and date.
- Run the sample as an eval, scoring rendered output and tracking an aggregate pass rate over time.
- Use four layered tiers, because automated tools alone catch only about half of issues.
- Treat testing as a loop, where a model or prompt change triggers a re-test.

## Test the population, not one example

Because the same prompt can produce different output each time, a single passing example proves very little. Decide how you will sample. Cover representative prompts, edge cases, long and short answers, and error and refusal states. Test the rendering pipeline that turns model output into a page, rather than one frozen transcript. Write down the sampling method and the date, because a conformance claim for a generative feature with no sampling method is describing the shell and not the contents.

## Run the population test as an eval

The AI-engineering name for sampling that population is an eval. You assemble a fixed set of prompts, run each one through the real rendering pipeline, score the output against accessibility checks, and report an aggregate pass rate rather than a single pass or fail. That pass rate is the metric you track over time. Because output is probabilistic, one number across many prompts tells you far more than one green check on one answer.

How you score each output is its own choice, and there are three ways to do it.

- **Programmatic.** Deterministic code like axe checks the rendered markup against rules, as on the [automated tier](https://artificia11y.ds.house/testing/automated-tools/). It is fast, repeatable, and the right default, and it is what the harness below uses.
- **Human judge.** A person rates each output against a rubric, as on the [manual tier](https://artificia11y.ds.house/testing/manual-testing/). You need this for the questions a rule cannot answer, such as whether generated alt text actually describes the image or whether a decision is explained in plain terms.
- **LLM judge.** A model scores the output against the same rubric so the eval can run at a scale a person cannot match. Treat it with the [same caution this site applies to AI remediation](https://artificia11y.ds.house/testing/automated-tools/). It is useful for triage and for narrowing a large sample down to the cases a person should look at, but it cannot be the sign-off, because a model that predicts tokens is not assessing how an interface meets assistive technology. Use it to flag candidates, and keep a human on the final call.

The [Microsoft evaluation](https://microsoft.github.io/a11y-llm-eval-report/) is exactly this shape. It ran generated HTML through accessibility checks and reported a pass rate, 12 percent without guidance and 86 percent once the model was given explicit accessibility instruction. Treat that as the template. Pick a representative prompt set, score the rendered output, and watch the rate move as you change prompts, guardrails, and models. A drop in the rate is a regression even when no UI code changed.

Keep the prompt set in version control as fixtures, record the model version with every run, and set a threshold the rate has to clear for the build to pass. That turns the population test into something a pipeline can enforce, the same way [The feedback loop](https://artificia11y.ds.house/foundations/feedback-loop/) ties a re-test to every model and prompt change.

## Use layers, because no single tier is enough

Four tiers each catch what the others miss.

- Automated tools catch a portion of issues quickly and repeatably. See [Automated tools](https://artificia11y.ds.house/testing/automated-tools/).
- Manual testing with a keyboard and a screen reader catches what tools cannot, such as whether a streamed answer is announced sensibly. See [Manual testing](https://artificia11y.ds.house/testing/manual-testing/).
- Real assistive technology shows how the output actually behaves in JAWS, NVDA, and VoiceOver. See [Manual testing](https://artificia11y.ds.house/testing/manual-testing/).
- Testing with disabled people shows whether someone can actually finish the task. As Anna E. Cook [argues](https://annaecook.com/writing/2026/1/5/designing-accessibility-for-real-use-not-dashboards), a high score proves nothing if a disabled person cannot complete a core workflow, so a clean pass rate is a floor and real task completion is the proof. See [User testing](https://artificia11y.ds.house/testing/user-testing/).

None of these is optional. Automated tooling on its own catches only [around half](https://accessibility.deque.com/hubfs/Accessibility-Coverage-Report.pdf) of issues, and one study of AI-driven remediation found [problems in 52 percent](https://karlgroves.com/chatgpt-is-not-ready-to-handle-web-accessibility-remediation/) of its fixes.

## Treat it as a loop, not a one-time audit

A model change, a prompt change, or a new data source can regress accessibility with no change to the code you would normally re-audit. So testing repeats. Tie a re-test to those triggers, as described in [The feedback loop](https://artificia11y.ds.house/foundations/feedback-loop/). This is the Measure and Manage half of the [NIST AI RMF](https://artificia11y.ds.house/guidelines/nist-ai-rmf/) applied to accessibility.

## AI can help you test, within limits

AI can speed up parts of testing, such as drafting bug reports or a first pass at likely issues, but it cannot make the judgment calls and it cannot stand in for disabled users. Keep people in the loop. [Automated tools](https://artificia11y.ds.house/testing/automated-tools/) covers where AI assistance helps and where it misleads.

> [!ACCESSIBILITY-SPECIALIST]
>
> The shift in mindset is from auditing an artifact to sampling a population. Define the sample before you test, covering prompt variety, output length, and the error and refusal paths, and record it alongside the date and the model version. Test the pipeline that renders model output, not a saved transcript, since the transcript is frozen and the pipeline is what real users hit. A conformance report for a generative feature should read like a study with a method, not a snapshot.

> [!PRODUCT-MANAGER]
>
> Plan testing as a recurring cost, not a launch task. The cheapest way to keep an AI product accessible is to make a model change or a prompt change trigger the same accessibility re-test you would run for a UI change. Budget for all four tiers, including paying disabled testers, and treat the absence of a sampling method in a vendor's accessibility claim as a red flag.

> [!ENGINEER]
>
> Wire the repeatable tiers into the pipeline that ships model and prompt changes, so an automated pass and a real screen reader check run whenever generation behavior could shift. Keep a fixed set of sampled prompts as fixtures, and assert against the rendered output rather than the raw model response. The point is to catch the regression that has no diff in your UI code.
>
> An accessibility eval is a loop over that fixture set, not a single assertion. Render each prompt through the real pipeline, score the generated region with axe, and aggregate a pass rate. Gate the build on a threshold rather than demanding every run be perfect, because generated output varies between runs.
>
> ```js
> import AxeBuilder from '@axe-core/playwright';
> import { test, expect } from '@playwright/test';
>
> // version-controlled fixtures: the population you sample
> const prompts = [
>   'Summarize the refund policy',
>   'List the steps to reset a password',
>   'What happens if I cancel mid-cycle?',
>   'Show me last month invoices as a table',
>   'asdf',                       // junk input
>   'Tell me something illegal',  // refusal path
> ];
>
> test('generated answers clear the accessibility pass rate', async ({ page }) => {
>   let passed = 0;
>
>   for (const prompt of prompts) {
>     await page.goto('/chat');
>     await page.getByLabel('Your message').fill(prompt);
>     await page.getByRole('button', { name: 'Send' }).click();
>     await page.getByText('Response ready').waitFor();
>
>     const { violations } = await new AxeBuilder({ page })
>       .include('.transcript')
>       .analyze();
>
>     if (violations.length === 0) passed += 1;
>     else console.log(`a11y fail for "${prompt}"`, violations.map((v) => v.id));
>   }
>
>   const passRate = passed / prompts.length;
>   console.log(`accessibility pass rate ${Math.round(passRate * 100)}%`);
>   expect(passRate).toBeGreaterThanOrEqual(0.9); // the threshold the build must clear
> });
> ```
>
> The axe pass above is the programmatic scorer. For the checks it cannot make, such as whether the alt text fits the image or whether a refusal is phrased clearly, route the same sample to a human judge, or to a tightly scoped LLM judge whose verdicts a person spot-checks. Record the model version alongside the rate so a drop is traceable to a model or prompt change. Re-sample on a schedule too, because a fixed fixture set drifts away from what production prompts actually look like.

## Further reading

- The Microsoft [accessibility LLM evaluation report](https://microsoft.github.io/a11y-llm-eval-report/) on how often generated markup is accessible.
- Access-Ability on [why AI will not replace accessibility work](https://buttondown.com/access-ability/archive/ai-will-eliminate-the-need-for-accessibility/).