A few weeks back we had the opportunity to chat with Bret Scofield, UX Research Team Lead at Sumo Logic. Sumo Logic is a cloud-based DevSecOps platform that helps companies build, run, and secure applications. The product is powered by great technology and used by some of the world’s leading companies such as Adobe and Delta Airlines.
Bret was kind enough to pull back the curtains on user research at Sumo Logic. From their implementation of Agile processes in research and design, to templatizing each step of the UX research process, you’re sure to learn something new from Bret and her supporting cast. Let’s dive in:
Sofia: When did you join Sumo Logic, and what do you do there?
Bret: I was hired three years ago as a product designer, and then we realized that we also needed to begin validating our ideas and designs. However, we didn't immediately hire a researcher. We decided that I'd do both for a while, and see how it went. It was a proof-of-concept before we committed to it. I was doing half product design and half research, and then transitioned completely to research about two and a half years ago. And now we're building out the team with plans to hire a few more more researchers in the next six months.
Sofia: From a research and product development perspective, what has changed since you joined Sumo Logic?
Bret: I think everyone in Silicon Valley follows some version of Agile. We all know the principles and we're all taking our own bits and making it fit into the culture and processes that exist. Last year in December, we had a few teams go through very strict Agile training, and they became the test beds. We were saying, "Okay, what does actual Agile look like? Not our modified version of it.” And so, there are two teams right now that are following strict agile processes.
Along with that, we said, "Okay, let's see what Design and Research look like if we embed them into an Agile team." Before, Design and Research were working on an agency type of model - we were getting requests that go in a big queue, and then we move items in the queue to get people what they need by the dates they want. The feedback was frequently, "Hey, why did my thing get deprioritized?" Or, "Why doesn't my feature get research for another six months?" You're always dealing with these competing demands, and always trying to be diplomatic and say, "We have many teams, and another team is releasing just a little ahead of you."
I think one of the big things that we've been noticing about embedding designers and researchers into Agile has been that they feel much closer to their teams. They sit with the engineers and the product managers, they do team events with them, and I think that's really great. They still come back to the Design and Research teams for our weekly design critiques and all that sort of stuff, so I don't think they feel a loss. It also brings the engineers and product managers closely into the design and research processes. Before, the design and research processes weren’t on display. We would definitely publicize the research and ask for inputs from the engineers and product managers. But with this new model, everyone is involved in everyone's business all the time. I think that's good because you can make better decisions when you have this intense clarity about things.
What Design has formalized here at Sumo Logic is possible because we have enough designers. We have designers embedded into all the scrum teams, and then we have a central DesignOps group. We haven't really done that with research yet, and I think that's eventually where we will go. We'll want our researchers all embedded in their scrum teams, and then someone at the core doing the ResearchOps stuff, doing the recruiting, handling the scheduling, handling our customer database, all those types of things. Right now that's a distributed task across all the researchers, and there's no one person designated to deal with it, which is a bit hard.
Sofia: Can you tell me how the DesignOps team came about, and what they normally do?
Bret: Our DesignOps group is focused on quality and efficiency. One large initiative that they've been focused on recently is called Yama. It's a component library, basically, and we saw, with our front-end and back-end teams, that there were all sorts of inconsistent components across the product.
We needed a team to create a library of all these components, and then to say, "These are the situations in which they should be used." They also follow the atomic design principles, where they build from the smallest units (atoms) to the largest units (pages). So, "What does the tiniest component look like, and then what are the larger patterns?”
The DesignOps team is very focused on the design system as a whole, from the library of components to guidelines to principles. We're in the process of building out that gigantic component library, which is a necessary centerpiece for all of this. Every design, before it goes into development, goes through a DesignOps review. This means that all of the components in the design are reviewed, to make sure we're following the standards for each component individually and the components together. This review also includes a discussion of voice and tone. We have guidelines for what we want to sound like when we're speaking with our customers about the product, and we want to make sure that that's consistent and that it's also following our UX writing principles.
The review before anything goes into development is those two chunks. I think ResearchOps would be slightly different because there's not necessarily an audit process or anything that we would do at the end of research. ResearchOps, to us, is more of the mechanics of making this research process flow more smoothly.
For ResearchOps, one of the big things that we're working on right now is our recruiting pipeline. As an enterprise product, we use our customer base very heavily for participation in UX research, and it's kind of a pain to recruit from this pool. A lot of designers and researchers that I have spoken with have the same pain where if you want to talk to customers you have to go through customer success or sales and exchange four to five emails to line the customer up for a session. That's extremely tedious, and to arrive at a body of 10 sessions you might spend two days on that.
So we've been working on a few things to smooth that out, namely working very closely with Sales to develop a process where they can nominate customers for UX feedback, and then also building out recruitment pools for specific capabilities. If we're focusing on a specific area of the product, then we recruit a pool that's going to be dedicated to feedback for six months at a time. We'll keep that pool, and then you don't have to go through the process of recruiting a brand new pool for every single research study. Of course, there are exceptions and it's necessary to recruit fresh eyes and experiences.
Sofia: With each researcher and each designer having to find their own customers, what is the process for speaking to customers?
Bret: We have a protocol that you go through when you're recruiting. The product manager usually has a list of people who have indicated that they're interested in the beta version of things, and usually that group of customers is quite excited to be involved in the evolution of the new product or feature. Our product management team will work with sales and customer success to find this group of customers.
Then, we will also use logs in the product. We might have indications that people are using dashboards heavily, or are using advanced features in dashboards. We'll reach out to those users to see if they have input or ideas for improvement. We also use EnjoyHQ. We'll look through all the mentions of dashboards, and we'll try and pick out customers who have, for instance, given us a Zendesk ticket like "Your dashboards suck," or "I would love to see this feature," etc. We have a checklist of places to look that you go through when you're recruiting a pool of participants for a study, but each new pool is the researcher's responsibility.
We're interviewing right now for a few different research positions, and people ask about that. They ask, "Oh, you know, I've been hit with this before, and I know what it's like to have to recruit 10 people. What do you have in place?" I think that researchers want to know exactly what they'll be getting into, as far as how much time will be spent with recruiting participants vs. executing on research.
Sofia: What do you think the future of ResearchOps will be? How can research become more unified within the whole product development process?
Bret: That’s one of the things that we think about a lot when hiring, too, because the idea of someone handling logistics and not doing research may not be very appealing to job-seekers. I don't know if people will get excited to do all of the scheduling, maintaining the databases, maintaining the findings, and all that sort of stuff. Maybe there's someone who is really delighted by that, but I think most of the researchers I know are so excited to interact with participants and to drive insights. You want to see the data, and you want to play with the data, or you want to talk with the human and see where they're coming from. There's magic in that.
I don't know if ResearchOps will become one role or one team, or if it stays distributed. As a hiring manager, I want to hire people into roles where they'll be successful, and I want to make sure that people are fulfilled by a ResearchOps career. From a craft perspective, people who are in DesignOps are still practicing design, and I’d be interested to learn how ResearchOps can support the quality and efficiency of research while executing on research, especially if the connection to research is the key to job fulfillment. I think ResearchOps will also fit into a specific point in a startup's maturity, and it will be very interesting to see where that is. Perhaps it's when you have four full-time researchers, or perhaps it's when you make the transition to embedding researchers into scrum teams. Finding where that lever that launches us into ResearchOps territory is interesting.
Sofia: Do you feel you’ve made a lot of progress, or no?
Bret: At times, with startups, it feels like it's fast and slow at the same time. Sometimes everything is going so fast, and then, other times you're like, "How do we not have this feature or process already? It's been years. How are we not working on this problem?" But I think that's why people stay in startups, too, because it's interesting. It's so interesting to be making those calls all the time, and to always have that sense of urgency, like, "This needs to get built." That's fascinating and a really fun working environment to be in.
Sofia: How are you standardizing your process?
Bret: For every step of the UX research process, we have templates. There are templates for all of the emails we send, as well as templates for the research plans and the reports. Perhaps most importantly, there are templates to guide each key meeting. These outline that the cross-functional team needs to answer before research can proceed.
We're also working on standardizing the research process across all of our scrum teams, from the ones that strictly follow agile and the ones that are a bit looser. We're also always delving into new types of research, and oftentimes we'll figure out a working model for a new type of research. Once we have that, we'll examine the current overarching research process and see where we can fit the new research process into the existing process, likely by generalizing the overarching research process a bit.
Sofia: And where are those templates? Are they Google Slides, or email templates?
Bret: Everything exists in a Google Drive, and it's all formatted in documents or slide decks. It's split into the steps of research - here are templates for before research begins, here are templates for executing on research, here are templates for after research is done.
It took us a while to get to this point. Aona, one of our founding researchers, came to us from Google, where there are amazing templates for everything. It was one of her first moves at Sumo Logic, to begin this process of templatizing everything possible. We are also frequently updating these templates or retiring ones that no longer suit our needs. However, the templates have been amazing for the team's efficiency.
Sofia: If you were going to go back to three years ago, what do you wish you knew then that would have helped on the journey that you’ve been on so far?
Bret: We're interviewing right now for researchers, and people ask, "What should I expect in the first three months of being at Sumo Logic?" One of the big things that I don't think people expect, especially if they're coming from larger companies, is the amount of evangelism that you have to do about UX research. Growing the UX research discipline involves a lot of talking with all the teams and being very clear about what your value is, what you're bringing to the table, and how you're influencing decisions, all that sort of stuff—that's huge, and you can't grow the team without doing that.
It's easy to think, "Oh, I'm growing the Research team, all we have to do is a lot of research to generate insights, and then people will see the importance of research." And it's not all that. It's more like, "Half the time I'm doing the actual research, and then half the time I'm doing the sell job," which sounds callous, but it's true. You have to sell it internally, and you have to be like, "This is the importance, and this is what we're trying to do to make a better product." Going back, I would have thought about it in those terms, instead of, "Oh, just do a ton of research." It would be more about buy-in and making sure that people understood what we as researchers were doing.