As part of our Customer-Driven Development series, we had the opportunity to chat with Teresa Torres. We’ve admired Teresa’s work for a long time. Her writing on ProductTalk.org has impacted product teams around the world and helped shape our current understanding of what it means to be truly customer-driven in the context of product management.
In this interview we talk about common misconceptions in product discovery, typical challenges that product teams face when trying to adopt better practices and practical tips to get you started.
This interview is a must read. Enjoy!
Sofia: Hi Teresa, can you tell us a bit about yourself and Producttalk.org?
Teresa: I’m currently working as a product discovery coach. I work with a cross-functional product team — a product manager, a design lead, and the tech lead. I work with a team for three months. I meet with them week over week, usually via video calls. It’s a continuous process. They’re meeting with customers every single week. We’re working on prototyping skills and experimentation skills. A big thing that I focus on is helping them develop critical thinking so that they can connect their research activity to their product decisions.
I see a lot of teams do all the right things from a research standpoint and then end up building the same thing they were always gonna build. It’s because our cognitive biases interfere. Most teams have the right intentions but they get excited by their own ideas. They fall in love with them. And it’s really easy to make mistakes when that happens. I like to help teams develop the muscle which enables them to visualize and examine their thinking.
I also blog at producttalk.org. I started doing that just to figure out my thinking and it happened to be very useful to other people as well which was serendipitous. Now, it’s a really nice way for me to test ideas and get feedback from the community.
Sofia: What are the typical problems product teams face when they get in touch with you?
Teresa: Usually, if you become a client, it’s because the head of product has reached out to me. Perhaps they’re not really confident that their team is building the right things. It could be that they’ve learned about continuous innovation and continuous delivery and they’re trying to figure out how to work that way. Or it could be that they’re already on that path and they just want to fine tune. But almost always, it starts with just some doubt about whether they’re building the right things.
I think our industry is shifting toward continuous discovery. A lot of our podcasts, blogs and conference talks address this subject. Typically, a head of product will start to get a window into this way of working. They get excited about it, they know their team needs to work that way and they’re looking for ways to get there.
Sofia: I talk to a lot of product teams that are very excited about the idea of implementing continuous product discovery but they are not sure where to start. How can they get started?
Teresa: One of the big shifts we’re going through is moving from project-based research to continuous research. Project-based research is what we’ve done in the past. It’s how user research/user experience grew up. A lot of it developed in external agencies, agencies which were selling big projects to companies. That means interviewing dozens of people, doing big persona projects and large-scale usability tests.
Those aren’t necessarily bad things and there’s always times where we want to do big project research but a product team really should be shifting their mindset to continuous research. The problem with project research is that we only reach out to customers when we think we’re done and then we’re using the customer as a sounding board for “Did we get it right?”
With continuous research, we’re changing the process from “Did we get it right?” to “Help create this with us”. Customer-driven development is really co-creating with our customers. It’s saying “Here’s what I’m thinking about. How does this work in your world? What would you do?” It means having the humility to work with customers to identify the most important problems and develop solutions with them.
An individual product manager or an individual product team that’s just getting started should look at their cadence of interacting with their customers. I push for at least a weekly cadence. That’s really foreign to a lot of teams. I have designers push back and say, “Hey, I need three weeks to get the design right.”
There’s a lot of hubris in that statement, right? That you’re gonna get the design exactly right on your own. Now just because I say you should get feedback from customers it doesn’t mean I don’t value your design skills. Your customer isn’t a designer. But your customer does have knowledge that you don’t have. They know about their own needs, their own problems and the context in which they work. We want to bring that knowledge into the process as early as possible.
Sofia: What are the typical challenges product teams face when trying to move from research projects to continuous research?
Teresa: Most teams, when they interview, they “recruit by hustling”. That’s what I call it. So they’re doing all these really inefficient things to get an interview. If you have to email 400 people to get five responses and three of them cancel, you’re not gonna interview every week. That’s not sustainable. If you’re interviewing once a month or once a quarter then hustling is fine but you really want to interview somebody every week and to do that you need to figure out how to automate the process of getting interviews.
A lot of the teams that I work with, if they’re consumer sites, are using something like Ethnio or Qualaroo to just pop up a question. These are one-question survey tools. And this question will be “Are you available in the next 15 minutes to do an interview?” They’re recruiting live off their sites. Intercom is great for this as well. They’re recruiting live off of their site to conduct an interview right then and there. And that removes a lot of the legwork which makes the process a lot more sustainable.
Some of the enterprise clients I work with are able to do the same thing. On the other hand, I work with a company that has on-premise software and they can’t recruit that way. They work with their sales teams or with their customer support teams and define triggers. They can give their customer support team rules like “If you see a customer that does this, then set up an interview for me to talk to them.”
There are lots of ways to do this but the underlying goal is to make it so that when you show up to work on Monday, there’s already an interview on your schedule.
After that, and this is the harder part, we need to get everybody in the organization to have doubt in their mindset. So many businesses are developed around the idea of rewarding people for being right and this goes all the way up the chain. Your director is a director because they were right more often than not. It’s the same with your C-level executives. Businesses build a culture around your value is being right.
This is really inconsistent with the new way of doing product development. Now that we have good feedback loops, we’re finding that we often think we’re right when we’re not. We need to keep doubt in the process. Instead of saying, “We know we should build this feature,” we need to say, “We think we should build this feature but we’re gonna test it.”
One of the biggest hurdles for an organization is that you still get people helicoptering in and saying, “Just build this feature.” A lot of work has to happen all the way up the hierarchy.
One of the best ways I have for doing that is to get the team to instrument every single thing they do. Then, when a director or a VP or a C-level comes in and says, “Build this feature” you measure the impact it has. If it doesn’t have the intended effect, you can go back to that person and say, “Hey, here’s what we learned”. That way they can start to see that they’re not always right.
Sofia: For some product teams, interviewing a customer per week can feel impossible. Specially in teams where research happens very sporadically, how they can do about it?
Teresa: This is the other thing I love about the continuous mindset. It’s really hard to fit project research into a busy schedule which is why we rarely do it but with a continuous mindset, we can make our discovery activities as small as they need to be. I teach people how to do five-minute interviews. I call them “Research in the wild.” “Go out into the world and find people that might use your products and just ask them a single question. Talk to them for five minutes.”
Most teams can do that over a lunch break. They can do it on the weekend. They can do it driving to and from work. Obviously, if you have an enterprise product, it can be a lot harder to reach your customers. We have some really good tools, though. You can use usertesting.com to recruit your target customers. There are services like GLG that recruit really specialized interview participants.
The interviews don’t have to be a full hour. If you’re working in an environment where delivery takes up 100% of your time and you’re barely keeping your head above water, do one 15-minute interview a week. You’re gonna get so much out of that interview that over time it’s gonna turn into 30 minutes and then 45 minutes and then it’s gonna turn into two 30-minute interviews.
I have a team I worked with last year who, when I told them at the very beginning of coaching that they were gonna be doing weekly interviews, they thought I was insane. They did not think they could keep up with that pace. Now they talk to customers every single day. And this is a team that was terrified by a weekly cadence. Now, they’re at a point where 80% of what they do is discovery.
This doesn’t mean they’re neglecting delivery because when the tech lead is involved in discovery and sees how much work is required to know what to build, the rest of the engineering team takes more responsibility for delivery. This frees up the product manager to spend more time in discovery and that’s really important.
It’s really fun to watch the transition. It feels like it’s gonna be impossible but if you can start with the smallest 15-minute activity, over time, it will grow. As long as every week, you say, “How can I do a little bit more?”
Sofia: As a product manager or product lead, how do I know it is time to move to continuous product discovery.
Teresa: I think it’s a couple of things. I think it’s a case of “Do you know if what you’re building is working?” A lot of teams still don’t know because they’re not measuring. Or they’re not measuring the right things. If that’s the case, that’s step one.
Thankfully, we’ve made a lot of progress there. We have a lot of really good instrumentation tools on the product side. That’s new. So I think a lot of teams are starting to get a good idea of what’s working and what’s not working.
Another symptom I see a lot of is “Our experiments never fail.” If your experiments never fail, you’re probably not good at running experiments. It’s not that you’re a genius and you always pick the right things. It’s probably that you have some problems with your experimentation mechanics and could probably use some help with adding some rigor into that process.
Another common thing I see is that experiments fail but they don’t roll back what they release. A lot of people think about experimentation purely as A/B testing. Even if it doesn’t outperform what was there, they keep it because they’re emotionally attached to it. If that’s the case your discovery process needs work.
I think it’s really hard for most organizations to have confidence in what they’re building. Most CEOs that are investing a lot of time and money into building software have doubts about whether they are building the right things. Often they don’t really know how those decisions are getting made. This makes them uncomfortable. A good technology team should be providing feedback all the way up the chain. This would tell them why they should have confidence in what they’re doing. And that that justification should not be “We think it’s a good idea. Our competitors are doing it.” It should be “We’ve been working with customers. We’ve been experimenting our way there”.
Sofia: What are the common misconceptions around product discovery?
Teresa: A lot of people equate usability testing with discovery and it is part of it. We need to know people can use it. But first we need to know if people want it? Are they willing to pay for it? Or, is it better than what is already out there? There are a lot of discovery questions besides “Is it usable?”
Similarly, people equate A/B testing with discovery. They think because they A/B test everything that they release, they do discovery just fine. Okay but if that’s 100% of your discovery, you are learning in the slowest, most expensive way possible. You’re building everything before you learn whether it’s right or wrong?
So I think those are the two biggest ones. And then I think that for teams that have strong user researchers or strong user experience designers who do their own research, the big one is this project mindset. They’ll say they do discovery but what they’re really doing is project-based user research. There are problems and questions that need project-based user research. I don’t think that world is going away. I think companies will always have longer time horizon research questions that a centralized user research team is perfect for but a product team has week over week research questions.
I think the big difference with discovery, and especially continuous discovery, is that teams should be asking “What do I need to learn today and what do I need to learn this week?” That’s the time frame that tells me you’re doing discovery well. If your research questions aren’t changing every week, you’re not learning every week. If you were learning every week, your questions would change.
That’s a big thing I look for. Do you have a weekly cadence of learning?
Sofia: Any advice for anybody that wants to understand a bit further of what it means to be customer-focused?
Teresa: I’m gonna reiterate something I said earlier. I really think if you’re not meeting with your customers every week, you can’t be customer-focused. We make product decisions every single day. And the longer you go between interacting with your customers, the less customer-focused your product decisions become.
Beyond that, we’re kind of in a golden age for customer-driven development. There are so many resources out there, right? Five years ago, we didn’t have very many product books and now there are dozens. Several are very good. We have a lot of really strong thought leaders in the continuous innovation/continuous discovery stage. It’s like drinking from the fire hose.
But here’s the thing: Most teams just read about it and don’t translate it to their work. It’s easy to get overwhelmed with ideal ways to work. I think read all the stuff if you’re interested but, most importantly, pick one or two things that you want to integrate back into your work. Once you have a handle on those, pick another one or two.
Because reading is not enough. We have to actually change the way that we work.