Five years ago, Beth Devine was enrolled in a PhD program training to become a professor. She didn’t even know User Research existed, let alone that she would find herself working in the profession one day. Fast forward to today - Beth now works as a user researcher at Stack Overflow, one of the most visited sites by software developers.
We had a chance to sit down with Beth and talk about her work. She was kind enough to share details about a new initiative she’s working on called Stack Overflow for Teams, as well as give us a behind the scenes look at how research is done at Stack Overflow. Let’s dive in:
Sofia: Why don't we start by talking a bit about what you do at Stack Overflow?
Beth: Stack Overflow is among the top 100 most visited websites in the world, but if you're not a developer, you probably haven't heard of it. It's an online developer community -- kind of like a combination of Wikipedia and Reddit for software developers. So if you're a developer, or are just getting into coding, and you search something in Google related to coding, chances are the first result will be Stack Overflow. That's how most people find us and we get millions of visits every month. We have very passionate users, which makes research fun and exciting.
I'm the sole user experience researcher at Stack Overflow, but I'm not the only person who does user research. My predecessor, Kristina Lustig, was the first researcher hired at Stack Overflow and is now my manager. She evangelized user research throughout the company – I'm grateful that she paved the way.
Sofia: What is a recent research project you’ve done?
Beth: Lately I’ve been focused on research for our newest product, Stack Overflow for Teams. Teams is an area of Stack Overflow where developers can have a private workspace to work on their own website or company. It’s a place to store information the same way you would in a wiki, but because developers are already familiar with our site, they are more comfortable working there. Teams launched six months ago and users are still learning about it. I'm learning with them, understanding their process through their life cycle and how we can best meet their expectations.
Currently, we’re finding different patterns of user behavior within Teams, and using user research to tell us why. For example, if we have a feature that people aren’t using right away, user research can answer that question. I can speak to users and find out how they find their way to that feature, if they know about it, what they like and dislike, and answer a lot of the questions that we didn't have time or the bandwidth to tackle during the early stages of product development. It's rewarding to be part of this early phase of product refinement.
Sofia: What do you like about being an in-house researcher?
Beth: One of the great benefits to being in house is that I learn the product strategy, which helps me understand what pieces of feedback we can and can't integrate. These decisions are made based on factors such as user demand, our product goals, and team resources. Some changes may only take a few days while others might take weeks, and that's just not realistic against other priorities. It's interesting to be at the nexus of what's possible and realistic, and what's really going to make the biggest difference for users.
When I talk to users, I encourage them to think out of the box, to be creative. "What would you like to see? What's a blue sky idea?" It's fun to let users go wild, because sometimes they come up with neat solutions and we're like, "Why didn't we think of that!?" That's what's awesome about user research. Of course, there'll always be feedback that you can't accommodate. I love compiling those insights, and then working together with my team to figure it out and draw a good roadmap for moving forward into the next phase.
Sofia: Why don't you tell me a little bit about what research looks like at Stack Overflow?
Beth: Because we’re a small company, research is customized to fit the goals of a project. Some projects are large-scale investigations on topics that affect users on every part of the site. Other projects are more focused. Within a single product development cycle, we’ll run recurring research sessions every two weeks or so. I really like setting up rolling research like this, because I think that the more you make it an integrated part of product development, the more value you get back. The product managers can tell more quickly if the feedback has changed and if we’re headed in a good direction. And unlike one-off projects, rolling research is not a 0 to 60 jump that threatens to hold up an entire team's progress. It will simply complement it and help people move their work forward.
I can share some of the tools we use to get research done. For recruitment, we email research invitations via Iterable. It lets me select a random sample of users so we don’t contact the same people over and over again. Iterable is tied in with our research listserv and updates automatically as users join or leave. That’s very important because we don't want to take advantage of our users – we’re grateful for them, their time is valuable, and no one's going to be happy if they get emails from me every two weeks saying, "Hey, can we talk again?" As much as I would like that, I am not sure users would appreciate it! We use Calendly to schedule sessions. It enables people to click a link in the invite, see my calendar, and pick a window of time, and then automatically creates a calendar invite for both of us with a Google Meet link. Our test materials are hosted in Figma or InVision, depending on the designer’s preference.
I’m constantly wondering if we're using the best tools on the market. There’s costs and benefits to using what we know and are comfortable with, versus something new that might enhance productivity, insight sharing, and product development. I'm always interested to hear what tools other people use. We hire some of the best developers in the business, so we also have the option of building our own tools for things. Building a custom solution gets us what we want but takes up our developers’ time, whereas an existing solution might be sufficient but costly. I haven't gotten to work with the internal developer team to build anything yet, but it's an option we’re lucky to have.
We try to keep most research sessions at 30 minutes or less, which is not typical of user research. We know that developers have limited availability – they're busy and protective of their time – and they're very to the point. Since all of our products are web-based it's easy to hop on a call, share my screen, and walk through a couple different flows within that time span. Developers are good at saying, "I don't like this because X," or, "I like this because Y." Their natural mindset helps the research succeed, because they're opinionated and analytical. Even when it’s features we haven't rolled out publicly, or features that are brand new, they’ll have super useful feedback.
Once the interviews are complete and the data is analyzed, I write up the results in a Google doc and discuss the findings at a 30-minute “research roundtable” – a meeting where the product team members or other people interested in the research can listen in and share their thoughts. I find these meetings are key to drilling down to the key takeaways and creating momentum for next steps.
Sofia: How do you help other people do user research at Stack Overflow? Do they feel that user research is important and awesome, or do they struggle with it?
Beth: It’s key to enable other people to do research so the researcher doesn’t become a bottleneck. I’d say the struggle for non-researchers is finding time to do the research on top of their regular workload. Some people have done a few rounds while others are brand new – I just had a conversation with a colleague the other day who was preparing to do research for the first time.
To this end, we’ve made a lot of templates. This helps people through the basic process of setting up a study – those who have a question and an inkling to have some conversations with users. Once they fill in the details of a research plan, set aside time for sessions, and write a discussion guide, we’ll review it together and iron out final details. They’ve become familiar with research at different levels, so depending on their experience I am more or less hands on. If they’re new I might check for any leading questions, make sure that they give users the opportunity to answer in a way that doesn't feel biased, and confirm that they're not putting users in the position of just agreeing and saying something's great. I try to help them with that language to make sure that the session is as useful as possible.
Stack Overflow as a whole definitely sees the value of user research. Important findings get attention and responses from the highest levels of our company. One example of this is in our research on diversity and inclusion. Developer communities are notoriously homogenous and this can feel unwelcoming to underrepresented groups, so we’ve done a lot of research and work around improving that experience across all our sites. It isn’t an easy topic to research, but it is very rewarding.
Sofia: What do you think happens in other organizations – that is, what makes it so easy for you and your organization to have access to customers?
Beth: That's a great question. I think that it's a little easier for us for three reasons. The first reason is our users: we have many, many users constantly visiting and engaging with our site. Furthermore, many of these users are developers – they know how to build a great website and know what distinguishes good UX from functional UX. We even have a section of Stack Overflow called Meta, which is a public Q&A that hosts conversations about Stack Overflow itself. To be fair, that’s a subset of users, so it doesn’t capture the voices of everyone. But we’ve encouraged people to comment and to speak up about their experiences on Stack Overflow, and user research is just one form of that larger movement.
The second reason is internal support of user research. Stack Overflow values research internally and this matters enormously. My colleagues have gone above and beyond helping me make sure I have the tools I need. In companies where user research is less known or less respected, it can make people nervous. It's not a sales convo nor a marketing convo. The fact that it lacks that structure can make people nervous because they don't know what's going to happen. And they don't want users to feel upset or have a bad interaction. If you can’t get access to users or to tools without a lot of red tape, it slows down the process enormously.
Thirdly, our size – user research can be a little easier at smaller companies, because there tends to be less formality and it's easier to be directly in touch with users. You're also closer with colleagues, departments, products, and therefore have a better view of the user’s experience across the board. By talking to other product teams, I know what kind of changes are happening for different parts of our site. By talking to marketing, I know what newsletters or campaigns are going on. I might know a little bit about what sales conversations are happening. And sometimes I have a research session that turns into a different conversation when they say, "Hey, we never mentioned this before, but we might be interested in this other product. What’s it like? Can you put me in touch with the right person on your team?" A lot of cross pollination happens. In bigger companies with more separation it's harder to reap those benefits. At the same time, larger companies usually have larger research budgets and more researchers to share the load, so there are benefits and challenges at any size.
We have a few obstacles to research. Our biggest challenge is the thousands of users we can't reach because we don't know who they are – a lot of people visit our site but never create an account. They don't want to use the gamification and reputation systems that encourage people to contribute and refine knowledge on the site – things that makes Stack Overflow so successful. One reason might be because they code as a hobby or as a subset of their work and don't see the ROI of becoming an active user. Those people are really hard to capture.
And sometimes people don't show up. Sometimes I schedule seven people and only talk to three. Sometimes users don't have strong opinions to share. So it’s not a total walk in the park. But to be fair, the people who do respond to research invitations are lovely and we get great feedback. In my experience, users enjoy taking part in research because they get to preview new features. They get to give feedback. They feel like we care about their opinions, and we really do. Sometimes they even get to see their suggestions come to life. This makes the effort meaningful and worthwhile for them.
Sofia: Can small companies take advantage of Research Operations to scale their user research?
Beth: I don’t think it makes sense to wait until you have a whole research team and then say, "Oh, wait, we should optimize these things to scale because we're still running on chaos." That's not efficient or sustainable. When you’re the only researcher, or if you're on a small team, it can feel like you don't have the time to research new tools or change systems. I often sign up for product trials and then never log in. It's hard to structure time for those long term gains when prioritizing research sessions could impact a product the next day.
Ideally you try to build in time – say an hour per week – for operations-focused work. It’s much easier to enable others to do research and to scale your research team if those structures are in place. It's good to document what you're doing so somebody new can step in and emulate that as best they can from the get-go. Without process or documentation you're trying to teach people things that are only in your head, and you're taking the shortest path. But not doing it or waiting until later definitely brings more challenge.
The more you can show the long-term benefits of taking the time to set up those things, the better off you'll be. It can be hard to prove that return on investment at the get-go, if you're saying, "We're going to do one less study a month so we can devote time to the overall architecture of research." I'm not good at making that argument, so I'm grateful that my team already values it. I spend time thinking and talking about it as an inherent part of my role.
One current project of mine is centralizing and organizing research results for quick and easy access. We do research, write up beautiful reports, have quotes, sometimes even video clips, and amazing insights – but it doesn’t matter if no one reads it! I don’t want to hide findings within 10-page documents. The documentation is important to describe the research process and to understand what research has been done in the past, but it silos the information. There are many different products at Stack Overflow that might touch on the same user experience. For instance, a new user might enter Stack Overflow two or three different ways: to look for a developer job, to get information about a certain coding language, or to join a private team that their company has set up. It would be ideal to funnel all feedback on new user experience into a centralized database, and label it onboarding.
My dream would be to have a searchable database where if you queried, "What do people say about onboarding?" you’d see relevant results from every study. Every finding related to onboarding would be neatly organized and dated and referenced so that we could see, “Hey, for 6 months, across 12 studies, we’ve gotten this feedback from users. This issue has come up eight times.” If you only see it once or twice, here and there, you might not realize that there’s a bigger theme worth paying attention to. This stems from the idea of atomic research by Tomer Sharon.
Sofia: If you could go back in time, say five years, what do you wish you knew then that you know today?
Beth: Five years ago I was in a PhD program, training to become a professor and lead a life of the mind. I was so driven in that goal that I had never heard of user research. I didn't know that companies were talking to people and making meaningful decisions based on those conversations. That they were using the same tools and methods for a different purpose, and that it could be a gratifying form of work.
The academic environment is very different because publishing research is highly competitive and risky – a project could take years but have little to no impact. I love the collaborative work culture and immediate impact of the work that I do now. I am invested in learning users’ experiences, contributing value to my team, and seeing our products and our company succeed. If I could go back in time, I would have learned about this world sooner. Perhaps the path I took was correct, because I'm happy to be where I am now.
Sofia: What do you like most about this profession?
Beth: Sheer variety - a user researcher is a different role at a Fortune 500 company, a young startup, or a research agency. Our jobs differ based on level, industry, and a company’s internal culture. No two projects are the same. Each user is a unique person with their own worldview to share. It's exciting, fun, and privileged work. There's always something new to learn or discover.