Alaska airlines

Overview

Usability testing evaluating the accessibility of the Alaska Airlines website.

About the client:

Alaska Airlines is a major American airline headquartered in the Seattle metropolitan area, while Alaskaair.com is the official website and a major touchpoint between Alaska Airlines and its customers.

Goal:

Our goal of this study is to assess the accessibility of Alaskaair.com for the low-vision user group. We planned and carried out our usability study to understand low-vision participants' experience navigating the website and made design recommendations based on the study result to help make Alaskaair.com more accessible.  ‍

Timeline:

Research details

Research Question

Study objective:

• Understand how is the current experience navigating through Alaskaairs.com like for the low-vision users.
• Discover accessibility issues as low-vision users work through the following task scenarios: booking a flight ticket, checking flight status.

Research Question

1. What are the users’ experiences navigating through the website with assistive tools?
2. How UI elements present on the website hinder or help the user from using the website?
3. Are users able to understand and interact with different UI elements present on the website?
4. Are users able to complete the tasks of booking a flight and checking flight status?
5. What are users’ attitudes and reactions about the current flow for booking a flight?
6. What are users’ attitudes and reactions about the current flow for checking the flight status?

Participant

Participant Characteristics

1. The participant must be at least 18 years old.
2. The participants will be people with low vision.
3. The participant must have substantial experience navigating through websites on the desktop with assistive tools of their choice.
4. The participant should be comfortable in general about making online purchases using a desktop/computer.

‍Participant Profile

We have 7 participants in total, including the participant from the pilot study. Here is the list of participant profiles:

Test Environments and Equipment

Due to Covid restrictions, this usability test is conducted in the remote world. Our participants received tasks through zoom calls while we were observing them accomplishing those tasks. Participants are able to choose their preference among various accessibility magnifier tools, browsers and operating systems.

Researchers use Zoom (cloud meeting tool) to moderate the test remotely, monitor each participant’s process, and document all think-aloud feedback and facial expressions. Researchers also run pre-test technical checks with different accessibility magnifier tools, browsers and operating systems to anticipate any technical hiccups.

Data Collection and Analysis

Data collection:

During the test session, as the participant goes through the tasks, the assigned note-taker notes down the conversation and observations on the Data Logging Excel Sheet.

Data analysis:

After all testing sessions, the team gathered together and transferred all observations from the excel sheet to sticky notes on a Miro board. And then an affinity diagram is created to find out common occurrences and themes. We decided upon a severity rating scale and assigned severity rating to each of our 12 major areas of findings. Because the main purpose of this study is to improve the accessibility and overall experience navigating the website, our analysis focus was on qualitative data and participant feedback.

Heuristic Evaluation

Methodology

We are planning to conduct 6 rounds of moderated online interviews over the course of one  week with the 1st one being the pilot test. Interviews will take place via Zoom with a facilitator and observer and a note taker. Each session is expected to be an hour long. Detailed process is as follows:

1. Pre-test survey

We created a screener and provided it to our clients to recruit participants who match the requirements of our study. The screener consists of questions regarding participants' proficiency to use and website and make online purchases. We are also hoping to learn more about the participants through this screener.

2. Pilot Interview

Prior to conducting our first test, we planned to conduct a pilot study session with one of the participants. We will conduct the pilot study with our scripts and are hoping to modify based on our learning from the pilot study.

3. Moderate Zoom Interviews

We are planning to test the following scenarios:

Task Scenario 1: You are planning to meet your friend who is currently in New York. You visit the Alaska Airlines website as a guest to book your flight from [City A] to [City B] on [Date D1] and return on [Date D2].

Task Scenario 2: Your friend is returning to [City A] from [City B] tomorrow, which is [test date + 1] from the earliest flight that day and you need to check the status of his flight so that you can plan to pick him up from the airport. His flight number is [Alaska ###].

Task Scenario 3 (Conditional): You are deciding if you want to travel to New York to visit your family. You heard that there are travel restrictions for inter state travel. Now you are on the Alaska Airline website trying to find what travel restriction for your destination.

4. Post-Test Questionnaire
   We plan to ask participants to rate the overall experience of the test and to provide subjective feedback and impressions.

‍

Severity Rating Scale

We decided upon a severity rating scale and analyzed each finding in order to give it a rating.

Rating based on:

• Usability Catastrophe (5) = The user is not able to or will not want to use a particular part of the product (Imperative to fix this before product can be released.
• Major Problems (4) = The user will probably use or attempt to use the product, but will be severely limited in his or her ability to do so.
• Moderate Problems (3) = The user will be able to use the product in most cases, but will have to undertake some moderate effort in getting around the problem.
• Minor Problems (2) = The problem occurs only intermittently, can be circumvented easily, or is dependent on a standard that is outside the product’s boundaries
• Cosmetic Problems (1) = Need not be fixed unless extra time is available on project
  

What's next?

What can do better next time

• ‍Conducting tech checks before testing sessions could help prevent technical hiccups during the formal session. We find that some assistive tools don’t work well with Zoom screen sharing but work well on Skype. We switch to Skype for those participants whose assistive tools don’t work along with Zoom screen sharing.

• Be prepared to solve technical problems during sessions even with tech checks done before.  One participant has a problem sharing the screen during the session because she is not familiar with the tool and has severe low vision. Since it’s a remote session, we couldn’t really help her directly but could only guide her to explore using voice and be extra patient.

• Observe participants’ facial expressions and body language even during remote sessions. When conducting virtual usability testing sessions, it’s easy to focus purely on what's going on over the screen. However, participants sometimes don’t say things out loud but you can get feedback from their behaviors and facial expressions. For example, a participant may lean into the computer screen or squint his eyes when reading certain information on the screen. They may not even notice these behaviors themselves but these are some cues of potential problems.

• Be mindful of your wordings when communicating with participants with disabilities. Before conducting tests, we prepared ourselves by reading documents like Words With Dignity and Disability Etiquette. For our study, it’s better to prevent using descriptions like “normal vision people” or “vision problem”. Although our participants may not care or be offended, we should do our best to act respectfully.

Future Studies

• Conduct usability testing with groups of participants divided into smaller ranges of zoom levels and by different assistive tools. When conducting usability testing with a wide range of participants who zoom in from 110% to 1300%, we learn that participants’ experience may differ a lot based on their zoom level and assistive tools they use. Although this study helps us uncover a wide range of problems in general, it would be helpful to narrow down the user group and conduct more in-depth research for each group.

• Conduct usability testing with people who mainly rely on screen reading tools to navigate the website.  In our study, our participants mainly rely on assistive tools like magnifier or zoom due to limitation of conducting usability testing remotely. We want to put more emphasis on the visible part of the websites rather than considering visuals and voice tools at the same time. However, we do realize that there are severe low vision people and blind people who mainly rely on screen reading tools to navigate the website. Thus, it’s important to consider and advocate for their experience as well.

• Test the websites via tablets. Before running our formal testing sessions, we conducted a pilot session with a severe low vision volunteer who mainly uses iPad to browse websites. Our goal for this informal session is to estimate the approximate number of tasks we can go through in a formal session and to have more understanding of how low vision users’ experience would be like on the website. Since she didn’t really fulfill our requirements, we choose not to include her data in our findings. However, we still receive a lot of insights from her session. We learn that some of the findings are unique to the website experience of tablet users. Since people ranging from severe low vision to no vision mainly rely on voice reading tools to navigate websites, we recommend Alaska Airlines to conduct usability testing on the website via tablets for them.

• Evaluate the website experience through other task scenarios Due to time limitations, we are only able to conduct the usability testing using two test scenarios. However, there are other task scenarios that can be frequently used by low vision users like checking in, finding accessible services and more.

‍