4. Evaluating deliverables

The contract documents should have clearly stated the accessibility targets to be achieved (see writing an RFT). But how do you know if what you have received from the supplier actually meets these targets? accessibility evaluation and testing of the deliverables is essential.

Evaluations should preferably be carried out by independent evaluators. It is very difficult to objectively evaluate a product that you have been involved in creating or otherwise have a vested interest in. If an independent accessibility expert was used to assist with the selection of the candidate or tenderer they could now be used to assist with the evlauation of the deliverables.

You will need to factor in the time required to evaluate the delivered product and the time required for the supplier to take remedial action to fix any problems that are identified by the evaluation. The supplier may have engaged as accessibility expert during the development process. If the supplier carries out accessibility auditing and user testing as part of its quality control procedures, the product or service should have a high level of accessibility on delivery. The independent auditing or testing should then reveal fewer and less serious problems, which will reduce the time and effort required for remedial work.

Scope of the evaluation

The evaluation should cover not only the technical accessibility of the product or service against the applicable standard or guidelines, but its usability for people with disabilities. This will enable you to assess important practical issues such as efficiency of use with assistive technologies and identify any aspects of the system that cause confusion and frustration for the user.

The evaluation of the deliverables will be the final arbiter in deciding whether the accessibility requirements have been met. It should be done rigorously and may go well beyond the supplier’s internal quality assurance process.

Evaluation methods

There are two basic accessibility evaluation methods – expert auditing and user testing. The guidance on web accessibility auditing contains a range of information on both these methods. These describe an appropriate process for carrying out an accessibility audit of a website. A similar process can be employed for other technologies. Read more about user testing.

Two things to note about auditing and user testing:

• For auditing, you cannot rely on automated accessibility
  checking tools. human accessibility experts are needed;
• Formal user testing is a highly skilled activity that involves careful preparation and management if it is to produce results that are reliable and useful.