Card sort to understand Activition, Blizzard, and King’s service desk information architecture
Executive Summary
In 2019, Blizzard Entertainment’s internal tools team made the decision to redesign their service desk. Users often complained that it was not intuitive and difficult to use. The reason for this was that the service desk had been organized in a way that only made sense to the people fulfilling the requests, instead of the employees making the requests. The purpose of this study was to make the service desk easier to use by gaining a better understanding Blizzard employees’ mental models of a service desk and reorganizing the information architecture of the page to reflect those mental models.
A second stage of the study was added to accommodate the consolidation of service desks across Activision, Blizzard, and King (ABK) and to ensure that the learnings from our first study applied to our other two business units.
Overarching Research Questions:
How do ABK employees organize service categories?
Does the organization of these categories differ across business units?
Key Findings:
Our team conducted a two-part study consisting of 1) an in-person card sort with a representative sample of 14 Blizzard employees and 2) a virtual card sort with 13 Activision and King employees.
Both rounds consisted of 43 cards and revealed that all three business units viewed service requests very similarly, allowing us to create a new, streamlined information architecture that met everyone’s needs.
Definition
Nielsen Norman Group defines card sorting as a UX research method in which study participants group individual labels written on note cards according to criteria that makes sense to them. This method uncovers how the target audience’s domain knowledge is structured, and it serves to create an information architecture that matches users’ expectations.
Research Questions
With this purpose in mind, design, PM, and research came together to discuss what major questions must be answered in order to design a service desk that serves our employees, which were as follows:
Will employees view access to digital and physical resources in the same way?
Will employees assume hardware and software can be found in a different place from infrastructure and networking?
What types of requests (IT, HR, legal, or security) are most commonly used and which do employees find most important?
Do employees feel that reporting something broken is more important than requesting something new?
Do the majority of employees think in terms of ‘problems’ and/or ‘needs’ instead of requests?
Methodology & Participants
This project occurred in 3 stages:
Once the project plan was approved by PM and leadership, I created a short demographic questionnaire to establish who our users were. Through stakeholder interviews, we had learned that the team informally created 5 proto-personas, who the product team generally considered the main service desk users. These proto-personas were used to recruit a representative sample for the study (N = 24).
New Hires (defined as 5 months or less):
Blizzard n = 2; Activision n = 1; King n = 1
Top Requestors (defined as people who submit multiple times per month)
Blizzard n = 3; Activision n = 3; King n = 1
Low Requestors (defined as people who submit every 6 months or so)
Blizzard n = 5; Activision n = 0; King n = 0
Executive assistants (defined by role)
Blizzard n = 1; Activision n = 2; King n = 2
Approvers (defined based on job responsibility)
Blizzard n = 2; Activision n = 1; King n = 0
The card sort was open, meaning participants were free to assign whatever names they wanted to the categories they created. After each session, the cards were randomized to control for any order effects. While participants were in their session, they were asked to think aloud as they went through their session in order to get an understanding of spots that felt unclear or areas where they felt the task could have fit in more than one category. All sessions were recorded to air with analysis.
Study Execution
Our first round of cart sorts were with Blizzard employees in person. Participants were asked to come into the lab and use the cards to create service categories that made sense to them. Our second round of card sorts were completely virtual. Participants were asked to use Microsoft Powerpoint Online to sort the blue tiles into the categories that made the most sense to them and then name to categories with the green tiles, just like in the in person session.
Analysis
Once I completed data collection, I entered the data into an Excel spreadsheet and organized by the card task and sorter to allow us to get a broader sense of the story. The categories were then standardized (giving those with similar names or concepts a consistent name). This makes analysis easier because it allowed me to combine groups where participants used the same basic concept but a slightly different label. I then read the data into R to create the dendrogram. In total there were 31 unique categories created. The 10 categories that emerged from the dendrogram were prioritized for the new information architecture of the service desk.
To check our research questions regarding whether Blizzard employees though about the service desk in the same way Activision and King employees did, I ran the dendrograms separately and found virtually identical categories, which is shown below with the matching color coded boxes.
Impact & Lessons Learned
Findings from the study were communicated to the PM, design, and engineering teams directly after analyses were complete in a formal readout. From our findings, we determined that some of the categories created by participants could map directly to our main navigation page, like “Access” and “Legal”, whereas other would inform subcategories, like specific office or facilities requests.
Findings from this study enabled the service desk team to completely overhaul the information architecture of our service desk and find a solution that served the needs of thousands of employees across 3 business units
This exercise also taught us some very important lessons as well.
First, in many cases, the differences that emerged between category expectations seemed to be dependent on thinking style. For instance, users who were more likely to be in a position to help other employees were more likely to think about access to both physical and digital things, whereas someone who is mostly technical and more of a solo worker was more likely to think of digital access first and physical as an afterthought.
It also became clear pretty quickly in the study that we could have made different card decisions. We had a few users who made comments such as , “I am not seeing many security related cards but if they were here, I would put such and such card with those”. Each time this happened, I made a not of it and added such cards for the following participants.
We also noticed differences even within each segment of the representative sample that we recruited, making us wonder if our proto-personas we had been useing to make product decisions need to be rethought. Because of this, we kicked off a persona building study, which you can find here.
