Training data

A model can only reflect what it learned from. The training data decides, before any interface is built, who the product works well for and who it fails. For accessibility this is often where exclusion starts, because disabled people are under-represented in the data and their language and behavior get treated as edge cases.

Key points

Training data decides who a product works for before any interface exists, and disabled people are often under-represented.
Statistical models push the uncommon to the margins, so disability gets treated as outlier data and left out.
Better representation has to be built in deliberately, by collecting and labeling data with disabled people and their consent.
Provenance and consent matter, since much training data was scraped without permission, including disabled people’s accounts.
Models increasingly train on AI output, which can amplify bias and erase the variety disabled users depend on.

Disability is under-represented and treated as an outlier

Statistical models learn the common case and push the uncommon to the margins. The W3C’s symposium on AI and accessibility opened with the point that this makes disabled people most vulnerable to AI harms. Researchers at Harvard describe how disability data is so varied that it gets treated as outlier data and left out, which designs the people most affected out of the system before training even finishes. The W3C’s report on AI and the web makes the same point, that bias disadvantages users whose input or output is less well represented in the data.

What gets missed

In concrete terms, training data tends to under-represent disabled speech and writing, such as the patterns of someone who stutters or composes with a switch, images and video that include assistive technology or disabled bodies, and the way disabled people describe their own experience. When these are missing, the model performs worse for those users and can score their language as unusual or negative. The page on bias and representation covers that effect in detail.

Representation has to be built in, not assumed

Better data does not appear on its own. It comes from deliberately collecting and labeling data with disabled people, with their consent, rather than scraping whatever is most common. Projects that build dedicated datasets, such as one assembling a large sign-language corpus for translation, show both the effort this takes and the payoff. Document what is in the data and what is missing, so the gaps are known up front rather than discovered later by users.

Where the data came from is an accessibility and an ethics question. A great deal of training data was scraped without consent, and publishers have started to block AI crawlers in response. For disabled people, whose personal accounts and health information may sit in that data, consent matters all the more. Track provenance so you can answer where the data came from and whether it was used with permission.

The web is filling with AI output

A growing share of the web is now AI-generated, including low-quality AI content farms. Models increasingly train on the output of earlier models, which can amplify existing bias and erase the long-tail variety that disabled users depend on. The W3C symposium warned of a monoculture of identical descriptions. Curating human, varied, representative data is part of keeping a product accessible over time.

This loop has a measurable cost for accessibility. The 2026 WebAIM Million analysis found that 95.9 percent of home pages had detectable WCAG failures, up from 94.8 percent the year before, and that the average page carried 56.1 errors, a 10.1 percent rise in a single year. It reversed six years of slow improvement, and WebAIM points to heavier use of frameworks and to automated or AI-assisted coding as part of the cause. Anna E. Cook draws out the consequence, that AI does not repair a broken structure but inherits it and reproduces it at scale. As more of the web is generated this way, the inaccessible patterns grow more common in the very data the next models learn from, so the gap compounds rather than corrects.

Audience: Product Manager

Data choices are accessibility choices, made before any UI exists, so put them on your radar early. Ask what disabled populations the training and evaluation data actually represents, and treat “we used a general dataset” as a sign that disability was probably averaged out. Budget for documenting datasets and for the consent and provenance questions, because both are becoming procurement and legal concerns, not just ethical ones.

Audience: Accessibility Specialist

Ask for documentation of the data behind a model the way you would ask for a VPAT for an interface. A datasheet that states what populations are represented, how disability-related data was handled, and what is known to be missing tells you where to expect failures. Where that documentation does not exist, the representation gaps are unknown, which is itself a finding worth recording.

Audience: Self-advocate

Being an edge case in the data is why a tool can feel like it was not built for you, mishearing your speech or misreading your writing. It is also personal, because accounts of disabled life and health are part of what these systems were trained on, often without anyone asking. Both the performance gap and the consent question are fair to raise, and specific examples of where a tool fails your input help teams find the gap.