Usability Study Project
January - March 2017
As part of the Usability Studies (HCDE 517) course in my Human Centered Design & Engineering Masters program, my teammates and I partnered with Microsoft Outlook's Design Research team to assist their new time management mobile app project.
In most companies, Microsoft dominates as the sole source of planning and communicating among and within teams. With the younger generation however, Microsoft applications such as Outlook are rarely used. To help with their effort to better target the college students, our team aimed to study two things -- (1) understand why a competitor app is successful or unsuccessful with the students, and (2) gain insights on what tools & resources students use to manage their time and why.
As the Usability Researcher & the Project Manager of the team:
- Communicated closely with stakeholders at Microsoft to establish the scope, goals, and the logistic details of the project
- Identified the target user group and recruited participants for user research
- Designed the usability studies and interviews by working collaboratively with the teammates
- Conducted the studies with the recruited participants
- Collected and analyzed participant data based on objective & subjective metrics
- Reported the findings back to the stakeholders in written and verbal presentation format at Microsoft office
At the onset of the project, we first met up with Microsoft Outlook team's design researchers to understand the project goals. Understanding that iCal (the native calendar app on iPhone) is a competitor for their new time management application, we aimed to assess the usability of the calendar portion of iCal with college students.
Specifically, we sought to investigate the following research areas:
1. What are the successful features and/or issues of iCal?
- What issues do users encounter while creating, editing, deleting, and sharing events?
- How easily can users create, edit, delete, and share events?
- How often do users sync their calendar with other apps? If so, how easily can they do it on iCal?
- Can users easily view their daily, weekly, monthly, and yearly schedules?
- What features of iCal are valuable to the users and why?
- Based on the usability findings of iCal, what recommendations can we make to Microsoft Outlook team to improve their new app?
2. How do college students track their tasks and manage their time?
- What are the tools (non-digital and digital) and resources that college students use?
- Why do students use those tools and resources? What makes them appealing?
- What are their current behaviors of time management?
- What things/events do they choose to track?
- What are the users' needs and wants for an ideal time management app?
Research Methods + Design
After identifying the project goals and research questions, our team started recruiting the participants. The desired demographics would be college students between 18-24 years old, who would use iPhones but do not use iCal as their primary time management tool. Based on these criteria, we recruited 8 participants: 4 participants who use non-digital tools and 4 others who use digital tools for their time management. The reasoning behind this decision was to compare and contrast behavioral patterns between the two user groups and find common valuable features in the tools they use. We also recruited current iPhone users since we did not want to test users on the operating system they're unfamiliar with.
Then we went on to design the study session. Each 1 hour session would be held in-person at Microsoft's facility, and would consist of two parts. First 30 minutes would be a contextual interview where participants were to bring their current time management tools and answer questions about their current time management behaviors, the pros/cons in the tools they use, and what they value the most in schedule tracking tools. The next 30 minutes would be task-oriented usability testing with iCal where participants were taken through 6 different major user tasks. These tasks included creating, editing, and deleting an event, accepting an invitation, navigating different schedule views, and syncing another calendar app to iCal. After the tasks, participants were asked follow-up questions regarding their overall experience with iCal. These questions involved participants rating the application’s ease of use and its usefulness, comparing it to their current resources, and the Net Promoter Score.
During the contextual interview, a moderator and an observer were present to document participant resources and to take notes. Throughout the usability study, the moderator took notes of important behaviors and feedbacks. All sessions were video and audio recorded for further review. After each session, we thanked the participant and gifted them with gratuity ($50 Microsoft Store gift card).
Data Collection + Analysis
After the testing day, the teammates and I reviewed the recordings, filled in the missing notes for each session, and documented noteworthy verbatim quotes. We walked through each session together to gain a full picture of how each of them looked like and what the highlights were.
For contextual interview data, we created several affinity diagrams to sort our information into summative categories, trying to find similar patterns and commonalities across the participants’ current use of time management tools. At the end, we were able to come up with a list of tools they use, the pros and cons to each tool, and what they would like and wouldn’t like to see in an ideal time management application.
For iCal usability data, we rated how successful each participant was in completing each user task, as well as things to take note of from their words and behaviors. We also created affinity diagrams to sort out the big pain points that participants experienced, and categorized them into digestible, key takeaways.
Combining the data from both the interviews and the usability, we were able to identify 5 major user needs when college students choose and use time management tools.
Personalization: Users want to personalize their calendars and schedules with colors, labels, fonts, stickers, and notification sounds to help prioritize things in the way they prefer. Personal relevance is key.
Accessibility and Discoverability: Users want to access and view quick "snapshots" of their days, weeks, and months. They want to be able to find and see their tasks and to-do lists in a fast and visually clean manner. This helps to efficiently manage their time.
Autonomy: Users want to feel like they got their stuff under their belt. Consider implementing features that empower users to feel like they are in control of their schedule and time management.
Sense of Organization and Order: Help users feel organized -- it doesn't have to be objectively organized. If users can feel like they have things in order, then it is a successful time management tool.
Sense of Accomplishment and Success: Allow users to visually check off, cross out, or delete finished tasks and events. This empowers users and gives them a sense of accomplishment.
Our study's findings led to concrete design recommendations for Microsoft Outlook team's newly launching time management app, and shed light on possible future directions for the prototype. We were able to gain in-depth insights on the wants and needs of college students, which would be highly valuable to the stakeholders in understanding their target demographics.
After submitting the final written report and giving a Powerpoint presentation in class, we were invited to present our quarter-long project to the Outlook team. On March 13, 2017, our team visited Microsoft's office at The Bravern (located in Bellevue, WA) to present our findings to 15+ stakeholders, ranging from design researchers, product managers, engineers, and product designers.
Challenges + Constraints
At the beginning of the project, the goal was to assess the usability of the new mobile app that Microsoft Outlook team had been working on. However, due to the back-end issues of the prototype, we had to change the scope of the project to assess the usability of their competitor product. If situation allowed it would have been helpful to do a comparative study on the prototype vs. iCal. However, we were still able to experience first-hand how project goals in the industry can change based on team needs and technical difficulties.
Also on the day of the study, we had one participant cancel the testing. Though we were still able to conduct our study with 7 other participants, the study would have benefited from more user input. If we had more time and resources, it would have been the most ideal to recruit 6 non-digital tool users and 6 digital tool users to do a fuller comparison of user behaviors and patterns.
What I Learned
- Practice makes perfect: Although we had practiced moderating the user study beforehand by running a couple of pilots, we quickly realized it was not enough when the wordings on our scripts had to constantly be modified on the testing day. Further practice would have helped us staying away from asking leading questions, lessening the chances of contaminating the data.
Working in collaboration with industry leaders: The design researchers from Microsoft have been a tremendous resource and help as we planned and conducted our user studies. On the testing day they stayed with us at the usability lab throughout all sessions, and did not hesitate to give us valuable feedback and help out with the logistics. Their guidance and mentorship inspired me to become a UX professional who can offer the same!