Posted on Leave a comment

Agile user research at the Metropolitan Museum of Art: an interview with Liz Filardi and Karen Plemons

When you think of an institution as storied and grand as the Metropolitan Museum of Art, terms such as “agile” and “DIY” might not be the first to come to mind. But at the Met, staff from different departments are working together to employ rapid, low-cost research methods to better understand the needs of museum visitors.

For this post, I interviewed two staff members from the Met, Liz Filardi and Karen Plemons, about how they are using rapid research methods to inform the development and design of apps, websites, and digital games. The methods and approaches they’re employing can be applied not only in large institutions like the Met, but in small museums as well.

Liz Filardi (left) and Karen Plemons (right)

Liz is in the Digital Media Department, and manages mobile projects in all stages of development. Karen is in the Education Department, and oversees educational research and evaluation efforts. Liz and Karen are their own “guerrilla team,” doing user research and usability evaluation on a shoestring. The methods they use include card sorting, think aloud user testing with clickable prototypes, interviews, surveys, and visitor observations.

Dana Mitroff Silvers (DMS): How did you start working together?

Karen Plemons (KP): Well, I kind of accosted a colleague (in the Digital Media department) in the elevator one evening! I said, “You know, I have some really good data on our younger visitors. And I know you’re trying to capture younger visitors. I can be a resource to you.” I explained that I can help support the work they’re doing in the Digital Media team.

Liz Filardi (LF): And then when I heard about Karen and her work, I started to think about the importance of user research in the projects we’re doing in Digital Media. There is a big emphasis in the Digital Media department on understanding users. We even have a new position in our group, a Digital Analyst.

Staff reviewing visitor data in an internal workshop. Image courtesy Metropolitan Museum of Art.
Staff reviewing visitor data in an internal workshop. Image courtesy Metropolitan Museum of Art.

Now, when I’m kicking off a new project, or we have to make a design decision and we know that there are a lot of stakeholders with vested interests, I’ll approach Karen and ask, “How would you go about this?” Then, Karen develops the testing instrument, and we both carry out the user research. Collaborating with Karen has been one of my best resources to better understand audiences and visitors.

KP: And now I’m working on three projects with the Digital Media department. There has been a real shift in how Education and Digital Media collaborate and work together. We’re really making an effort to break down the silos between departments. And the ironic thing is we work on the same floor. It’s not even like we’re in other parts of the building. There’s another team in Digital Media that is a 15-minute walk away, but we are on the same floor! So it’s not difficult for me to be in an elevator with someone and say, “Hey, I can help you with that project!”

Visitors performing a card sort activity.
Image courtesy Metropolitan Museum of Art.

DMS: Can you give me an example of how you’re using user research on a digital project at the Met?

KP: We used card sorting for the content in the Met App. A card sort is a great way to find out how users group things together and what content is important to them.

In the app, we have exhibition information, event listings, permanent collection objects, and our Twitter feed. To help determine the mix of content that was most useful, we did card sorts with museum visitors. It helped us understand how visitors made connections, and what was most important to them.

After we had an idea of the potential content in the app, we created laminated cards, and gave visitors tasks such as, “Prioritize the five most helpful or interesting things you see here” and “Prioritize the five least helpful or interesting things you see here.”  We would then have participants talk through why they made those choices.

Twitter content inside the Met App.

And we noticed a trend: that visitors were grouping the cards related to selfies and social media in the “least helpful” category, saying they didn’t want to use social media in the museum. Yet, we also noticed that the Twitter cards were testing really well because people were drawn to the images in combination with short, colloquial text (e.g., Tweets about artists’ birthdays). Visitors did not seem to notice or mind that the source of this content was social media.

It’s possible that if we had just done a survey, we may have concluded that users didn’t want social media in the app and left it out entirely. But through the card sort, we were able to understand nuances around different types of social content.

DMS: What does a typical project look like for you?

LF: I might call Karen on a Monday. Then we’ll meet on a Tuesday and put together our instrument and testing plan. We’ll think about what audiences we want to target, and how can we engage with that audience based on what programs are happening at the Museum. We’ll do testing on a Thursday or a Friday. On the following Monday, we will synthesize the results, and put together the design recommendations.

KP: User research and testing doesn’t have to be super expensive and lengthy. And you don’t have to have the biggest sample size. The nice thing about working inside a museum is that we can go downstairs and engage with visitors in the galleries—literally, go downstairs and do some research and testing. In an ideal world, we’ll also find time to go outside the museum too. For example, we’ve done usability testing at the Chelsea Market at lunch.

DMS: What does “design thinking” mean to you?

LF: As it relates to user research, it’s about putting yourself in someone else’s shoes, understanding their needs, and being willing to go from there. Designing from this place often means you have to set aside your preconceived ideas. It often also means making stronger decisions more quickly.

KP: We often talk about constraints as fueling creativity. When working on projects, we often have to be within a box (as determined by funding, or time, or other limitations). But we can be very creative within this box. Which is where we can bring in and leverage what we know about particular users and audiences, and craft the best solutions for them.  I tend to think of design thinking as a tension between constraint and creativity, where research becomes key. Through constant agile research and testing, we are always learning and integrating our findings.

Follow Karen on Twitter and Liz on Twitter.

Also, check out the slides from their workshop at the 2015 Museums and the Web conference, An Introduction to Agile User Research and Testing.

Posted on 1 Comment

Spreading design thinking throughout an organization: lessons from Atlassian

atlassian-headerFor this post, I interviewed Karen Cross, a Design Manager at Atlassian, about the internal design thinking program the company has been building up over the past year. Atlassian makes tools for software development, collaboration, and project management, and several museums and nonprofits use their products such as Confluence, Jira, and HipChat.

Readers may be wondering why I’m featuring an interview with someone from a software company, and the answer is simple: I’ve always looked outside the museum sector for models of new ways of working, thinking, and collaborating.

I was first introduced to Agile software development by web developers when I was working on the San Francisco Museum of Modern Art website, and to design thinking through an executive education program at the Stanford, and brought both of these approaches back to my work at SFMOMA. I believe museums can look to the private sector for new models of working, and adapt these processes to make museums smarter, more efficient, and more awesome.

What struck me in my conversation with Karen was how purposeful Atlassian has been about spreading design thinking throughout the organization. The three key components of Atlassian’s internal design thinking program are:

  1. Trainings and workshops for staff
  2. Digital resources available to all staff
  3. Intentionally designed spaces to foster new ways of working

The trainings and materials that Atlassian has developed blend methods from Agile software development, Lean methodologies, and design thinking, all with the goal of putting users front and center.

Karen Cross, design manager at Atlassian, running a design thinking workshop for staff.

Q: Karen, can you tell me about your role at Atlassian?

A: I came on board as part of the user experience team. One of my roles is to spread more design thinking throughout the company.

For example, I run an introductory design thinking workshop for all new hires every quarter. (It’s based on the virtual crash course created by the Stanford Anyone can take the training. This is about establishing a design thinking practice, regardless of people’s individual roles. We think that anyone can find value in applying user-centered design, and we encourage all staff to participate.

Q: What other resources have you developed for staff in addition to trainings?

A: We’ve developed what we call the Atlassian Playbook. With the Playbook, we’re using a football analogy.

In football, it’s not like you do the same thing every time. You pull from the playbook the appropriate tool, technique, or practice, depending on the problem you want to solve. The playbook is available to all staff via the intranet, and in it we cover such things as:

The Playbook describes what these are, the supplies you’ll need, and why you might want to use these tools. We also cover things like how much time to anticipate, how many collaborators you’ll need, and how difficult or easy it will be.

Q: Can you give me an example of another tool or method you share with staff via the online Playbook?

A: One of the tools we cover is a design wall. (Design walls are large, vertical surfaces on which ideas, data, and work in progress can be displayed, rearranged, and extended. Read more about design walls here.)

We believe very strongly in the notion of design walls. This is about making work visible. Design walls are our new desks. We want staff to collaborate with their peers as much as possible.

Q: Can you talk about how you are using dedicated spaces in your office to promote design thinking?

A: We have both a dedicated area in the San Francisco office, along with more casual drop-in spaces.

We’ve thought a lot about closed spaces (dedicated conference rooms) vs. open spaces (drop-in spaces for stand-up meetings or design walls) and we’ve learned that closed spaces enable heads-down work time, while open spaces are best for impromptu discussions and foster a sense of community and sharing with non-designers.

An open, drop-in design space in the Atlassian office.

Ideally, spaces should be a mix of closed spaces, open spaces, and design walls. It’s a small thing, but having loads of markers, post-its, blue tape (for putting stuff up on the walls) and other materials available in the room is a time-saver, and encourages people to create rather than just talk.

We also use furniture to evoke a “this is where work gets done” vibe. This includes separate moveable tables of different heights (rather than one big conference table), stools, and free wall space to draw on and stick stuff on.

Q: Why train staff in design thinking?

A: Well, we see projects as having three phases:

  1. Envision It
  2. Make It
  3. Improve It

And we know we’re really good in the Make It phase. But we’re not so good in the Envision phase. We want to encourage staff to spend more time up-front so that we’re not just jumping into building stuff. We also want to encourage staff to spend more time in the Improve It phase so that we can answer the questions, “How did we do?” and “Should we pivot?”

In the end, this is about the ability to scale great experiences. Everyone should be empowered to ask, “Is this the right thing for our users? Are we solving the right problem here? Are we sure this makes sense?”

Follow Karen on Twitter at @karenmcross and Atlassian at @Atlassian.

All images courtesy of Atlassian.

Posted on 3 Comments

Using design thinking to connect the physical and digital at the Rijksmuseum: an interview with Shailoh Philips

shailoh-headerLast week I had the honor of interviewing Shailoh Philips, who worked for the last two years setting up the Media Lab at the Rijksmuseum in Amsterdam, The Netherlands. The Rijksmuseum is the largest national Dutch museum, and recently underwent a 10-year, multimillion-euro renovation and reopened in 2013.

I spoke with Shailoh about a project titled Augmenting Masterpieces. The project explores connections between the physical and digital within the gallery space, and examines how digital technologies can be integrated into the Rijksmuseum to deepen visitors’ on-site experience. Continue reading Using design thinking to connect the physical and digital at the Rijksmuseum: an interview with Shailoh Philips