top of page

Usability Testing

View the entire case study on the desktop version of this webpage joshuadema.com

​

This case study oulines Usability Testing client projects which I have delivered to clients recently. 

I've detailed the UX Testing process, as well as providing an accessibility lens at each step.

The nature of my recent client work in the usability testing space is confidential so i'll be speaking in general terms here.

​

​

Client Usability Testing

Usability testing with People with Disabilities (PwD) is done to ensure that a product or service is accessible and usable for all individuals, including those with disabilities. This testing helps to identify any issues that may not have been apparent to designers or developers and ensures that the product is inclusive and accessible to all users.

The Process

Frame 60.png

Project Scoping

Meeting with the client

Project scoping involves defining the objectives and parameters of the testing process.
 

I would typically take clients though the following steps:

 

Define the project goals: Identify the purpose of the testing and what we want to achieve through the process. For example, is the goal to ensure accessibility compliance, identify usability issues, or both?
​

Identify the target audience: Determine the demographic requirements for the project. This might include specific disabilities and Assistive technology users.
 

Develop the test plan: Determine the methodology for testing, (for example, moderated usability testing, surveys, interviews, etc). What tasks or flows might participants be asked to complete, and what metrics will be used to evaluate the testing?

Determine the resources required: Identify the resources that will be required for the testing process, such as assistive technology devices, testing software, and testing facilities.
 

Set the timeline and budget: Determine the timeline and budget for the testing process, including the duration of the testing process and the resources required.

First contact with the client is also the time to start talking about the value of Accessibility.

 

It's a good time to learn where the client is at with their understanding of accessibility, some questions to ask would include:

 

  • What sparked the desire to test with people with Disabilities?

  • Are they looking to achieve WCAG compliance?

  • Have they done any accessibility work on their product before?

  • Do they have a specific audience in mind?

​

​

Checking Bias

I consider it part of my responsibility to help clients understand any inherent biases they have towards their product or to the test participant. 

​

If time allows I'll incorporate a bias-awareness activity with stakeholders. The idea is to ask stakeholders to consider how their environment, socio-economic status, race, and exposure to the product might impact their assumptions about usability testing. 

​

​

​

​

Accessibility Lens

Participant Recruitment

Who will test the product?

Participant recruitment is the process of finding and selecting suitable individuals to take part in a usability testing study.
 

Recruitment can be challenging at the best of time, particularly when the required demographics are small or hard-to reach.

 

While working at Intopia I have been able to leverage the wonderful Intopia Connect participant database.

When I need to recruit outside of an existing database, I have experience in:

 

  • Writing social media posts to attract new participants (for example, designing accessible Facebook posts, tweets, and LinkedIn posts)

  • Networking with organisations to reach new participants.

  • Forming connections with existing Facebook/reddit groups to advertise usability testing opportunities.

  • Using my own network to advertise opportunities.

 


Planning the experience
Planning out the participant experience from the first point of contact to the final thank-you note is the key.

My skills in providing clear and concise information are utilised here. I prioritise the participant experience and ensure they are comfortable with the process.

Considerations include:

 

  • How much information should I reveal in the initial recruitment, to inform but not overwhelm?

  • What kind of reimbursement can I offer the participant?

  • How will I explain what a Usability Testing Session is?

  • Remote testing: How will I make the experience with technology as smooth as possible?

  • In-person testing: How will I ensure a smooth experience on the day of testing?

  • In session: How will I make the participant feel comfortable in expressing their thoughts and opinions?

  • Post session: What kind of timely thank-you note and follow up will I send to participants?


 

Working with the right mix of participants is crucial to improving the accessibility and inclusivity of a product.


I'm passionate that research should represent a diverse range of individuals with different abilities and experiences. 

 

I have worked with participants from the following disbability demographics, and who use a variety of Assistive technologies:
 

People who are Blind

  • JAWS Screen reader (Windows)

  • NVDA Screen Reader (Windows)

  • Talkback Screen reader (Android)

  • Voiceover Screen Reader (iOS/MacOS)
     

People who are Deaf

  • Sign-Language interpreter
     

People with Low Vision

  • Screen Magnifier technology
     

People with Limited Dexterity

  • Dragon Naturally Speaking 

  • Head Tracking Device

  • Hands free assistive settings

  • Other alternate input methods

​

Neurodiverse people (for example, ADHD, ASD, Dyslexia)

  • Voice to text

  • Alternate adjustments

​

Accessibility Lens

Script Writing

Maximising potential findings

Script writing is the process of creating a detailed plan for conducting a usability test.

My approach to script writing involves the following steps:

 

Define the flows: I'll engage my the client to understand the flows that they would like to test. I'll also be checking that the proposed flows will be accessible to assistive technology users. (for example, a Figma prototype may not be accessible to a screen reader user, so i'd advise alternate testing methods)
 

Draft the testing script: I'll create specific tasks that participants will be asked to complete during the testing process. My focus is to ensure tasks are clear, concise, and easy to understand. I'll refrain from asking leading questions, and build in time for the participant to approach the product in their own way. 
 

Determine the metrics to be used: Define the metrics that will be used to evaluate the success of the testing process. This might include completion rates, success rates, error rates, and time-on-task.
 

Test the script: Where possible I'll organise a session to test the script with a colleagues or stakeholder.

Incorporating Accessibility into the script.

 

At this stage i'll take the following considerations on board

 

  • Will the tasks in the script be compatible with any assistive technologies in use?
     

  • Will any of the participants benefit from a plain-language script?
     

  • If I the participant is Deaf or Non-Verbal, will I need to present the script in an alternate format? (for example, via text chat)
     

  • If the participant is Deaf, will I need to adjust the script to work with a sign language interpreter?

Accessibility Lens

Technology and Session Set-up

When preparing technology for usability testing, there are several important considerations to keep in mind to ensure that the testing is accessible and inclusive.
 

Remote Testing

 

  • Technology

    • Which platform am I hosting the sessions on (Popular options include Zoom and MS Teams)​

    • Can I meet with the participant before the day of testing to introduct myself, and ensure they are comfortable with the technology.



In-Person Testing

 

  • Is the physical environment accessible to the particiapnt?

    • Clear signage

    • Appropriate lighting and sound levels

    • Accessible bathrooms

    • Manual and motorised Wheelchairs​

      • Elevators​

      • Door frame sizes

      • Flooring and carpet condition




,

It is important to ensure that the technology itself is accessible and user-friendly for people with disabilities.

Before the session, i'll be testing that the assistive technologies are compatible with the product we are testing, and that the user is confident in using their assistive technology with the testing setup.

Some common problems encoutnered in Accessibility testing:

 

  • Screen readers (e.g. JAWS, Voiceover) sometimes have issues with sharing sound over video-conferencing platforms.

  • Unreleased products may not be compatible with system accessibility settings

  • Prototypes may not be compatible with screen reader software

  • Participant may not be familiar with screen-sharing ov video platforms.

​

Accessibility Lens

Facilitation & Note Taking

Facilitation and Note-Taking

Facilitation is my favourite part of the Usability Testing process, it's the culmination of all the preparation work, time to start gathering product insights.
 

To sessions successful, i'll ensure the following happen:

 

Communicate clearly and effectively: I'll use clear and concise language, and communicate in a way that is accessible and easy to understand.
 

Create a comfortable environment: Whether the session is in-person or online, i'll ensuring that the testing environment is comfortable and welcoming for participants.

​

Encourage honest feedback: Participants will be invited to provide honest and detailed feedback about their experience with the product or service being tested.

​

Remain flexible: Problems might happen on the day, so i'll be prepared to adapt to the needs of each individual participant. This may involve adjusting tasks on the fly, or providing additional support or guidance as needed.

​

Good facilitation technique is especially important in usability testing with people with disabilities. Considerations may be made to ensure that participants can fully engage with the testing process.
 

As a facilitator I'll be reminding myself to:
 

  • Be aware of any potential communication barriers and adjust my communication style accordingly.
     

  • Take time for participant to process instructions and give feedback, it's important not to rush people along a flow.
     

  • Allow users to navigate in their own way. Different assistive technologies come with varying use
    patterns

Accessibility Lens

Analysis

Organising the findings

Analysis of usability testing results provides insight into the problems or challenges that users faced while interacting with a product or service.
 

This step involves examining the data collected during testing to identify recurring issues, patterns, and opportunities for improvement.
 

Here are the steps involved in my analysis:
 

Data organization: I aim to organise data in a meaningful and manageable way. I'll start noticing themes and group similar feedback together. This step is made easier using spreadsheets or online tools to organize and filter the data.
 

Recognise Recurring issues: During analysis, products might have recurring issues or patterns in the feedback.
 

Categorizing issues: The next step is to categorize the issues into specific areas of the product or service.

Depending on the project, the some or all of the following might be useful to present back to stakeholders:
 

  • Issues categorised into Usability, Accessibility, or Functional problems.

  • Issues categories by Task

  • Issues categorised by product page or screen

  • Issues categorised by Assistive Technology

  • Issues categories by Disability Type
     

Prioritizing issues: After categorizing the issues, the next step is to prioritize the issues based on their severity, impact on the user experience, and the ease of implementing a solution.

A MVP (most viable process) mapping exercise would be useful here.

At this point in the process I'll be very familiar with the product and it's issues. This knowledge, combined with my experience of accessibility issues and usability heuristics allows me to differentiate between:
 

  • User error

  • Individual user preferences

  • Functional issues with the product

  • Assistive Technology competency

  • Usability Issues

  • Issues caused by the synthetic nature of a testing environment
     

My understanding of the WCAG (Web Content Accessibility Guidelines) helps me determine which issues might fall into the four key categories of POUR
 

Perceivable - Information and user interface components must be presentable to users in ways they can perceive.
 

Operable - User interface components and navigation must be operable
 

Understanding - Information and the operation of user interface must be understandable.
 

Robust - Content must be robust enough that it can be interpreted reliably by a wide variety of user agents, including assistive technologies.

Accessibility Lens

Reporting

Reporting and Recommendations

Reporting
 

The final step is to report the findings and recommendations to the relevant stakeholders, such as designers, developers, or product managers.
 

I'll endeavour to write a report that is clear, concise, and actionable. Reports will provide specific guidance on how to address the issues identified during usability testing.

​

Recommendations
 

Based on the analysis, i'll develop recommendations to improving the product or service. This may involve suggesting:
 

  • Accessibility fixes

  • Usability fixes and design changes

  • Functionality bugs

  • Suggestions for feature enhancements

  • Broader recommendations for improving the overall user experience
     

Overall, analysis of usability testing issues requires a systematic and thorough approach to ensure that all issues are identified and addressed.

Delivering accessibility recommendation in a way which is digestible and actionable takes a careful approach. Adjusting to my stakeholder audience i'll be specific about"

 

  • What the issue is

  • Where in the product the issue occurred

  • When in the testing session it occurred (what was the user attempting to achieve)

  • Who it impacts

  • Why it is important to address the issue

​

Accessibility Lens

Frame 9 (1).png
bottom of page