# User testing

The final tier is testing with disabled people. Automated and manual testing tell you whether the product meets the rules and behaves in assistive technology. Only real users tell you whether they can actually accomplish the task, and whether the experience respects them. For AI products this matters even more, because the hardest moments, such as a confidently wrong answer, are exactly the ones a checklist does not capture.

## Key points

- Testing with disabled people is the only tier that tells you whether the task can actually be done.
- A technically conformant interface can still be unusable, especially when the model is confidently wrong.
- Recruit across disabilities and assistive technology, and pay participants for their time and expertise.
- Test whole tasks including the failure moments, not isolated happy-path screens.
- Bring disabled people in early to shape the product, not only at the end to sign off.

## Why it is not optional

A technically conformant interface can still be unusable, a point accessibility practitioners make [again and again](https://buttondown.com/access-ability/archive/ai-will-eliminate-the-need-for-accessibility/). When one AI startup claimed a fully accessible site, testing it [with a screen reader user](https://axesslab.com/webinar-lovables-ai-built-a-100-accessible-site-or-did-it/) showed the gap between the claim and reality. Disabled researchers studying their own use of generative AI [documented](https://dl.acm.org/doi/abs/10.1145/3597638.3614548) both real benefits and real harms that no automated tool would have surfaced.

## Synthetic users are not real users

A newer version of the shortcut is to skip recruiting and use synthetic users instead, AI personas prompted to react to a design the way a person would. The appeal is obvious, since you avoid the cost and time of finding testers. The evidence does not support using them this way. A systematic review of 182 studies [found](https://www.researchsquare.com/article/rs-9057643/v1) that AI-generated participants fail in consistent ways and cannot replace human ones, and that better prompting moves the result only a little because the underlying problem stays the same. Even the team that built one [agent-based testing system](https://arxiv.org/abs/2502.12561) frames it as a way to rehearse a study design before running it with real people, not as a substitute for the study.

The gap is widest for exactly the people this tier exists to protect. A model trained on web data represents disabled people least well, so a synthetic panel tends to miss them. See [Bias and representation](https://artificia11y.ds.house/inclusive-development/bias-and-representation/). Use these tools to pilot a script if you find them useful, then run the real session.

## Recruit across disabilities and assistive technology

Include people who use screen readers, screen magnification, voice control, and switch devices, and include people with cognitive and learning disabilities. Different assistive technology exposes different failures, so one tester is not enough. Recruit for the range of your real users, and pay participants for their time and expertise.

## Test the whole task, including failure

Have people complete real tasks from start to finish, not isolated screens. Include the moments where the model is uncertain or wrong, and watch whether the person can tell, recover, and stay in control. These are the situations where AI products tend to fail disabled users first, and they are easy to miss if you only test the happy path.

## Nothing about us without us

Bring disabled people in early, not only at the end to sign off. Microsoft's exclusion brainstorming method [is explicit](https://inclusivetechlab.github.io/web/ITL_AI_RecognizeExclusion.html) that an AI-generated list of barriers is a starting point to validate with real people, not a substitute for them. Center agency, and let participants tell you what works rather than telling them. The European Disability Forum's [work on AI literacy](https://www.edf-feph.org/projects/empowered-by-ai/) is about exactly this shift, from disabled people receiving AI to shaping it.

> [!PRODUCT-MANAGER]
>
> Schedule and budget user testing as a real line item, including fair payment for disabled participants, and run a round early rather than only before launch. Weigh the findings heavily. If a clean audit says pass but a screen reader user could not finish the task, the product is not done. Recruiting through disability organizations helps you reach a genuine range of assistive technology rather than the same few testers.

> [!SELF-ADVOCATE]
>
> Useful sessions are the ones where you do a real task your own way and say out loud where it breaks, rather than being walked through a script. The most valuable feedback is specific, naming your assistive technology, the step that failed, and what you expected. Your time is expertise, so expect to be paid for it, and expect to be brought in while decisions can still change, not just to rubber-stamp the result.

> [!ACCESSIBILITY-SPECIALIST]
>
> Treat user testing as the tier that overrides the others when they disagree. Combine it with the automated and manual findings, and when a participant cannot complete a task that passed every rule-based check, that is a finding about the product and not about the user. Capture the assistive technology, the task, and the exact point of failure so the team can reproduce it.

## Further reading

- Axess Lab on [testing an AI-built "accessible" site with a real user](https://axesslab.com/webinar-lovables-ai-built-a-100-accessible-site-or-did-it/).
- Glazko and colleagues, [an autoethnographic study of generative AI for accessibility](https://dl.acm.org/doi/abs/10.1145/3597638.3614548).
- The Microsoft Inclusive Tech Lab [exclusion brainstorming method](https://inclusivetechlab.github.io/web/ITL_AI_RecognizeExclusion.html).
- A systematic review of [synthetic participants generated by large language models](https://www.researchsquare.com/article/rs-9057643/v1).
- Lu and colleagues, [an LLM agent-based usability testing framework](https://arxiv.org/abs/2502.12561).