Overlay Effectiveness Study: Research Protocol

This page describes a proposed study. A reproducible dataset, completed analysis, and results have not been published here. It should not be cited as evidence that a 100,000-site study has been completed.

Proposed research question

Does the presence of an accessibility overlay correlate with differences in detectable accessibility barriers after accounting for site characteristics? An observational association would not, by itself, establish that the overlay caused a difference.

What a publishable study needs

  1. A dated sampling frame, inclusion criteria, actual sample count, and a request-failure log.
  2. A documented overlay-detection method, manual validation sample, and treatment of uncertain classifications.
  3. Identical browser, viewport, scanner version, timing, and page-state rules across comparison groups.
  4. Matching or adjustment for relevant site differences, with exclusions and missing data reported.
  5. Separate reporting of automated findings and manual accessibility evaluation.
  6. An analysis plan, uncertainty estimates, limitations, reproducible outputs, and review before results are announced.

Useful guidance available now

For a practical review of your own site, use the audit process guide and audit scope checklist. W3C explains the role of structured evaluation in its conformance evaluation guidance.