Automated tools

Automated tools are the fast, repeatable tier. They check rendered markup against rules and flag clear violations. They are necessary, and they are limited. The limits matter more for AI output, because so much of what breaks is contextual rather than a simple rule violation.

Key points

Automated tools are the fast, repeatable tier, and they are necessary but limited.
Rule-based checkers reliably catch issues like missing labels, low contrast, and invalid ARIA, pointed at the rendered output.
Automation tops out around half of issues, so a clean run is a starting point, not a passing grade.
You can script interactions to push more checks into automation, like testing a chat widget open and closed.
AI remediation is unreliable, often introducing new issues or missing related ones, so use AI output as a draft a person checks.

What rule-based checkers catch

Tools like axe check the page against WCAG rules and find issues such as missing alt attributes, missing form labels, low contrast, and invalid ARIA. They are reliable for these and they run in a pipeline without a human. For an AI product, point them at the rendered output and not only at the surrounding interface, because the generated content is where structure tends to fail.

The automation ceiling

Automated tools only find a portion of accessibility issues. Deque’s own coverage figures put automated detection at around 57 percent of issues, and many practitioners report lower in practice. The rest need a human, because a tool can confirm that an alt attribute exists but not that the description is correct, and it cannot simulate how a screen reader actually behaves. Treat a clean automated run as a starting point, not a passing grade.

Push automation past the rule scan

A rule scan only sees the page in the state you hand it, so a lot of what people file under manual testing can still be automated by scripting the interaction first. You can confirm the skip link is the first thing that takes focus, that a toggle’s accessible name changes after it is pressed, and that focus returns to the trigger when a dialog closes. For a chat product, run the scan in both the open and the closed state, because the scanner only checks the state it is given and the launcher and the panel are different markup. This does not close the whole gap, but it moves real checks out of the slow tier and into the pipeline. David Mello’s account of what axe and Lighthouse miss walks through the failures a rule scan passes over and which of them a Playwright test can still catch.

AI-enhanced checkers

A newer class of tools adds machine learning or large language models to the rule engine. Deque’s axe uses computer vision to read visual relationships, Accessibility Cloud’s ACAI combines AI with rules for wider coverage, and GitHub’s accessibility scanner files issues and suggests fixes through Copilot. These extend what automation can reach. The vendors themselves caution against treating the AI output as a source of truth.

AI remediation is unreliable

Asking a model to fix accessibility issues is tempting and risky. Karl Groves tested 79 real fixes and found problems in 52 percent, including adding new issues and missing related ones. He also warns that AI audits produce counterfeit expertise, confidently citing standards that do not apply and sometimes suggesting a fix identical to the broken code. Equalify argues that LLMs cannot reliably review code for accessibility because they predict tokens rather than analyze how an interface meets assistive technology. Use AI output as a draft a person checks, never as the fix itself.

Where AI genuinely helps testers

Used as an assistant rather than an authority, AI speeds real work. Practitioners use it to draft structured bug tickets and remediation snippets and to turn captions into transcripts or draft alt text for review. A useful shift-left tactic is an enforcement prompt that makes an AI coding tool check keyboard access, focus, and labeling as it generates, as in this open enforcement loop.

Audience: Engineer

Run a rule-based checker against the rendered generated output, not just the shell, and do it across several sampled prompts. With Playwright and axe-core that looks like this.

import AxeBuilder from '@axe-core/playwright';

test('a generated answer is accessible', async ({ page }) => {
  await page.goto('/chat');
  await page.getByLabel('Your message').fill('Summarize the refund policy');
  await page.getByRole('button', { name: 'Send' }).click();
  await page.getByText('Response ready').waitFor();

  const results = await new AxeBuilder({ page }).include('.transcript').analyze();
  expect(results.violations).toEqual([]);
});

Push past the rule scan where you can. The scanner only sees the state you give it, so for a chat widget test both the closed launcher and the open panel, and add behavioral assertions a rule check cannot make.

test('the chat launcher and panel are both accessible', async ({ page }) => {
  await page.goto('/chat');

  // closed: only the launcher button is rendered
  let results = await new AxeBuilder({ page }).include('.chat-widget').analyze();
  expect(results.violations).toEqual([]);

  await page.getByRole('button', { name: 'Open chat' }).click();

  // open: the panel, input, and transcript are now in the DOM
  results = await new AxeBuilder({ page }).include('.chat-widget').analyze();
  expect(results.violations).toEqual([]);

  // a check no rule scan makes: focus moves into the panel on open
  await expect(page.getByLabel('Your message')).toBeFocused();
});

You can push automation all the way into the screen reader tier. This site uses guidepup to drive real NVDA and VoiceOver in the pipeline, which lets you assert on what was actually spoken.

import { nvda } from '@guidepup/guidepup';

await nvda.start();
// jump to the first heading in the assistant's reply
await nvda.perform(nvda.keyboardCommands.moveToNextHeadingLevel1);
const spoken = await nvda.lastSpokenPhrase();
// a structured reply announces as a heading, not a flat wall of text
expect(spoken).toContain('Assistant');
await nvda.stop();

Automated screen reader checks are a strong regression guard, but they do not replace a human listening to a full streamed answer on the manual tier, so keep both.

If you let an AI coding tool generate or fix UI, give it a standing instruction to check accessibility as it works, then review the result rather than trusting it.

Before returning code, review the critical user flows.
Confirm keyboard access, visible focus, semantic structure, and labeled controls.
List what you changed and any remaining accessibility risks.

Audience: Accessibility Specialist

A clean automated run is the floor, not conformance. State plainly which checks were automated and which still need manual and assistive-technology testing, so a green pipeline is not mistaken for a pass. Be skeptical of any product sold as an “AI audit” or one-click fix. The overlay industry made the same promise, and the lesson is that contextual judgment and real assistive-technology behavior cannot be automated away.

Audience: Product Manager

Do not accept “passes our AI accessibility scanner” as evidence of compliance. Automated coverage tops out well short of full conformance, and AI remediation has a high error rate, so a scan is a triage tool and not a sign-off. Budget for the human tiers, and treat vendor claims of automatic accessibility with the same caution the industry learned to apply to overlays.