Some UX Research Methodologies (and when to use them)
For many organizations, user experience research (UXR) methodologies can seem like a daunting set of tasks, and may not be worth the effort. This risk avoidance is common. However, correctly utilizing UXR methodologies at the right time, can save the organization time and money, ensuring that the right product is being built for the right audience.
This article will examine a few methodologies, when to use them, and how they can help a product development org.
Generative Research
Generative research provides insight into the problem (or problems) that users have. This is a commonly-skipped step in the corporate world, as it’s not as well understood as other evaluative methods. Critically important, this foundational knowledge of the user problem not only helps to build the right product, but will also have a positive revenue impact — build the right MVP, once.
Generative Methodology #1 — Contextual Inquiries
What Are Contextual Inquiries?
Contextual inquiries are an observational methodology that allows researchers to learn what participants do in typical situations (at work, performing certain tasks, during their day in general), in order understand what problems they experience. Researchers observe participants in a live environment, whether in-person or over video, and guide the session by asking specific questions designed to prompt the participant into explaining why they make certain decisions. Participants also think out loud as they work through their tasks during the entire session.
Interviews are amazing, but can yield inaccurate results. Because individuals may not accurately recall how they perform a task, researchers will get more insight by watching how tasks (or jobs) are performed. Findings from contextual inquiries can be used to generate a number of UX artifacts, including personas, user journeys, and jobs-to-be-done frameworks.
Deliverables should include insights like lists of tasks, communication tools, how participants schedule their day, how they work in the space the company is researching, etc.
When To Use Contextual Inquiries
After intake, if the team realizes they do not understand the problem a proposed product or solution is trying to solve, this is a good time to use contextual inquiries. Other signs that this methodology is need will include:
If the team is making a lot of assumptions about what users need
If assumptions/hypotheses based on one user type are not working for a new one — this can be with full-built products as well
If a number of solutions are being proposed, but no one can decide which one will address user needs
If the company has different user types, but doesn’t have a deep understanding of the differences between each group
Generative Methodology #2 — Diary Studies
What Are Diary Studies?
Similar to contextual inquiries, diary studies are very useful tools when there is not a deep understanding of users and their days. This is an unmoderated research type, where participants take notes of what they’re doing during their days, certain tasks, etc. This can also be accomplished through video studies.
These studies should also provide context for UX artifacts such as personas and user journeys.
When To Use Diary Studies
The signs will be similar as to when you do contextual inquiries. However, diary studies are very helpful when you need to document extended periods of observation.
Generative Methodology #3 — Data Mining
What Is Data Mining?
Venturing from qualitative methodologies, quantitative studies can also give very important insights into where problems may be occurring, including things like abandonment, time on task, rage clicks, etc. Data mining is a great opportunity to not only understand exactly what users are doing in a product, but also build critical relationships with colleagues in data science and engineering.
When To Use Data Mining
Data mining is a good step to take, regardless of which qualitative methodologies might be utilized. Taking the time to go through quantitative data — behavioral, event tracking, etc. — will provide valuable insight into whether product hypotheses are valid.
However, the most powerful insights are built by pairing qualitative and quantitative data. When the “what” (quantitative), is validated by the “why” (qualitative), the team can feel confident in moving to the next step.
Evaluative Research
Evaluative research occurs later in the design process — when designers have a concept that can be tested. At this point, they will have gone through the generative process, worked with colleagues to understand business and tech constraints, and come up with wireframes or mockups to test.
Evaluative Methodology #1 — Usability Testing
What Is Usability Testing?
A commonly used methodology, usability testing gives researchers and designers an opportunity to work with “real” users and determine any usability violations that may cause friction in the experience. This can be done through moderated or unmoderated methodology, and involves testing with actual participants. Ideally, researchers will be able to recruit a random sample of participants matching target users, in order to make sure mental models are met by the product.
Tools like UserTesting and UserZoom make this type of testing incredibly simple and powerful.
When To Use Usability Testing
In general, some form of usability testing should always happen before releasing a product, whether that’s traditional testing as discussed above, or internal reviews (next in the list). This should occur once there is enough of a design to determine if the direction is accurate, or if there are major gaps in understanding that need to be addressed.
This will be completed after the user problem has been identified in generative research, and before engineering finishes building the product.
Evaluative Methodology #2 — Heuristic Evaluations
What Are Heuristic Evaluations?
Heuristic evaluations are a way to perform usability testing using internal experts. The formal framework is based on a set of criteria from Jakob Nielsen. There are 10 heuristics, or mental models, that products need to meet in order to be considered usable. Doing this review with five experts, should expose ~80% of usability issues.
Experts review the product, or design, and provide a rankings for severity and impact for heuristic violations, which are then stack ranked to determine the work/design update sequence.
When To Use Heuristic Evaluations
In situations where recruiting is difficult or very regulated (think doctors, small business owners, etc.), using this methodology can be very effective.
When there is limited time to do usability testing, this can also provide insights quickly, as there is no need to create an interview script, schedule moderated sessions, or spend time recruiting external participants.
Evaluative Methodology #3 — A/B Testing
What Is A/B Testing?
Returning to the qualitative vs. quantitative paradigm, A/B testing involves implementing two versions of a product of feature to determine which performs the best. This will be done in conjunction with product management, engineering (probably), and will likely involve using a special tool to minimize development effort.
The two versions will be live for a pre-determined amount of time to collect enough data. Once there is a clear indication of improved performance, the “winning” solution will be fully implemented.
When To Use A/B Testing
This is a methodology that many business stakeholders are fond of, as it’s possible to get very good quantitative evidence. However, this should specifically be used during later phases of the design process, when user problems have already been determined, and UX has had an opportunity to identify possible solutions.
This should not be done as a replacement for generative research.
Bonus Methodologies
There are some bonus methodologies that can cross the boundary between generative and evaluative research. Depending on teams are trying to accomplish through research, these can be used at different points in the design process.
Bonus Methodology #1 — Card Sorting
Card sorting involves providing test participants with a list of items, or “cards” and asking them to sort those items into groups. Closed card sorts provide the groups ahead of time, and users will simply put items into the categories in which they fit. Open card sorts allow users to create and name their own groups, and can provide researchers with extra information about users would group items.
This can be used in the generative phase to understand how users would think about a problem space, and the cognitive groupings they’re already utilizing with different parts of a product. While this will yield quantitative results, researchers will get insight into psychological frameworks. Using card sorting during the generative phase can also help with building information architectures, or task flows.
In the evaluative phase, card sorting can be used to determine pieces like menu taxonomy. The quantitative results will allow users to understand how to literally group items in a menu, or filters in e-commerce. It has a very obvious application in these spaces.
Bonus Methodology #2 — Storyboarding
Different types of storyboarding methodologies can be utilize during either phase of research — generative or evaluative.
During the generative phase, researchers will provide a story (in the form of a storyboard) to test participants, and ask interview questions to understand how accurate the story is for that user. This methodology can be used to inform UX artifacts like user journeys or personas.
In the evaluative phase, a different storyboard methodology can be used for concept validation. Providing test participants with multiple “stories” to understand which resonates the most will allow researchers to understand the right design and product direction.
While this article only discussed a few, there are a huge number of methodologies to develop an understanding of user needs. Having a better grasp of when to use each and why has many benefits for organizations developing products. Properly utilizing the design and research team to get these insights is paramount to a company’s success, and gathering more information about users is never a bad thing.