O’Reilly was a traditional book publishing company up until recently, when the company began to transition into an online learning platform for Enterprise teams. James is a Senior User Experience Designer who is helping the company with their digital transformation - he was kind enough to sit down with us to discuss O’Reilly’s venture into SaaS and the UX challenges presented by such a lofty endeavor. Let’s dive in:
Sofia: Let’s start by talking about what you do at O'Reilly.
James: I’m currently a Sr. User Experience Designer at O’Reilly (O'Reilly Media) in the Platform group. We have six full-time designers, one UX researcher and growing. We have a couple of open positions right now and looking to scale up even more next year.
O'Reilly is going through a significant digital transformation from a traditional book publishing company to a SaaS-based enterprise-level online learning platform. Along the way has been a merger of teams— a technical product company (Safari Books) coming together with a traditional media company (O’Reilly Media). With this comes a lot of up-skilling and education in design thinking throughout the organization.
The current product (learning.oreilly.com) doesn’t represent all the great thinking and research that's happening right now. O'Reilly as a company has always had a better understanding of the market and their customers—enterprise level companies like Google and Netflix—more than the actual end-user and how they use the product. There's now a strong focus towards user-centered design happening within the company to change that.
Sofia: What does User Research at O’Reilly look like? What’s the makeup of the team and how do researchers factor into product decision making?
James: We currently only have one UX Researcher (with another coming on board next month). But the one we have is amazing (Stefanie Owens, previously at IBM). Knowing that one person is just not going to be able to get the job done, I tend to work very closely with her nearly every day. We're doing a lot of different things to give people across the organization an understanding of how the user-centered design approach works. We’re saying, "Hey, look, this is not just a design thing. You can apply this to other areas of the business too."
We have cross-functional teams that work on different features and areas of the online learning platform, with a designer embedded in each one of those teams. Right now, research is spread out, and the researcher tackles what she can. We're beginning work on some significant projects that will affect the entire product, and we're trying to prioritize where the research happens. To balance that, each product manager and all the designers have been told, “you should initiate any research that needs to happen, and if research has already been started, you have the autonomy to continue with it.”
As a result, a lot of things like user testing end up falling into the hands of the designers and the product owners, so there’s mixed responsibility right now. Everyone's becoming much more aware of how you build SaaS-based products and how to gather as much information as we can from our users. But just like anything new, there are challenges. Everyone’s approaching things differently, maybe doing them incorrectly, and possibly not understanding the legal obligations of conducting research.
So everyone’s getting a baseline education on user research as they're doing it — here are the do's and don'ts if you're going to do this on your own. And then once you’ve gathered some inputs, what do you do with them? How do you take action on them? And where are those things kept for others to find – that’s super important. We have customer feedback coming in from multiple channels, but consolidated monthly reports have only recently started to get out to the different teams so they can stop and look at the input.
Along with surveys and things like that, we’re using an in-app service to collect specific metrics that we also reference. It's all coming in, and we need better organization of the data points to be able to make those actionable, too.
Another issue that's recently come up around research is how quickly you apply it. We had gone through this considerable marathon of synthesizing the output of dozens of design studios (Lean-UX exercise), and it all got put into a report for teams to run with while our researcher took maternity leave. When she got back, we had to refresh ourselves on everything that was there, and we realized that some of it was starting to grow stale. That brought a lot of awareness to the teams that you can't just do it once and think it's okay. You have to know what you intend to do with it before you get it, and then take action on it as soon as you can.
Sofia: When I talk to people about ResearchOps, there is this sense that it’s only for large organizations. What are your thoughts on that?
James: I've heard from friends working at other Enterprise companies that designers have to go through a formal request process to the research department when they have a research need. I'm curious about that "agency within a company" model, and I'm not sure it entirely works.
I like the distributed model, if you will, of research, knowing that one researcher is better than none. But I imagine that one definitely cannot handle the workload even in a small company. So how are you going to make that work? And how are you going to optimize that resource to make the most significant impact on the business? That's why it's important for researchers to spend time teaching others about best practices.
A lot of people defer from starting their research because they assume you have to be smart to do it. What many people don't realize is that there are a lot of low hanging fruit exercises you can do, and things you can integrate into your daily workflow, even if you're not a designer. I've been impressed with how our team of product owners and product managers accept that. And being part of the process has helped them respect research and become champions for it.
In our last company-wide meeting, two of our product managers presented roadmaps that go into next year. They also offered all of the user testing and research exercises that their teams were doing, showing everyone the results. I think that's a great way to make everyone aware of what's happening. It shows that the teams are not just coming up with these ideas on their own in a silo. And by presenting that process, product teams are becoming advocates for research without even realizing it.
I remember when we hired our first researcher. Everyone was asking, What is this person going to do? Why do we need this? And when she first started, her first couple of months were spent presenting pitch decks about the process to executive stakeholders to explain how it works. That first pitch for something like three months of research towards a specific project was like holding your breath: Am I going to get this? Am I going to get a lot of pushback? I can see the company thinking there's risk involved in that, but it's proved itself out.
Sofia: Why do you think the company was hesitant to embrace research?
James: I wish I understood it a little bit better myself. I think it's about not fully understanding what a design researcher or UX researcher does – not so much about assuming the role, but being told that the company needs to hire someone full-time to do it. I think that's part of it. The company might say: I've heard of market research, I've heard of all these other forms of research: that's a costly process, a time-consuming process, and you often get the feedback you don't know how to handle. You've been presented a report back, and you're like, "OK, I see you're pointing things out to me, but what are we going to do about it?"
I think that's possibly why UX research has worked better in our scenario, because it's not a standalone activity, it's not research done in isolation that’s presented as a report back to stakeholders. It's effectively like research in motion. Our researcher is initiating things, and then the teams are taking what she’s started and running with it. Or they are participating in parts of the research – they’re helping assist with the study as she does it so that they can learn to do it themselves, and taking it back into their teams and just doing it. I think that’s critical to spreading the knowledge and getting others to initiate research on their own.
Sofia: What is your role in helping the organization with the transition to SaaS, and what does the change mean for the product team?
James: It's multilayered, for sure. I've been working at O'Reilly a year and a half, and the digital transformation was already underway when I joined. A lot of the work is still happening in the background, but that will start to reveal itself to the public next year.
There's also a culture shift happening at the same time from very traditional employees who have never worked in software before to having a majority of the new hires being software-first, engineering and design, and all these other new areas. It’s not a culture clash, especially since we’re spreading the knowledge of innovators as our core mission, but now it's a lot of education, leading by example, and a lot of patience to get people to understand the agile process. It's also studying the behaviors of the end users, showing actual evidence of why something might not be working and making sure you understand why something sticks when it does.
Part of research is just discovering the people that work within the business and making connections. That's not talked about much – it’s usually all about the end user. But I’ve found that, from a researcher’s point of view, you end up talking to everyone in the business because you want to know what everyone does, what data they might have access to, and who they've spoken with. You're always in discovery mode.
Tactically, we're in an exciting stage that relies on research quite a bit. Our current version of the product and our future vision don't align yet. So we're trying to get a better understanding of how we’re going to get from where we are today to that vision. Without giving too much away, I can say that we're working on a significant information architecture project that goes above and beyond things like navigation of the product. It's mostly about trying to define core experiences for our users.
And for O'Reilly specifically, that means understanding how people learn. So a big part of our business is ramping up now, bringing in people like curriculum designers and data scientists to look at different learning frameworks along with studying the behaviors of our current users so that we're not excluding significant segments. We’re talking internally, on both sides of the house, to get a better understanding of who does what.
That all leads into the transformation I was talking about: there's a lot of groundwork set so we can turn something that is more or less flat and static into a very engaging, personalized, interactive experience. It's a tremendous amount of work that's all driven by research. It's a significant milestone to be able to tell your stakeholders, "You can't argue with this, this is what our users are doing and saying, and we have evidence that this solution will help solve our problem." That's going to help bring these teams together and make sure we're all moving forward on the same page.
Sofia: What do you know now, that you wish you’d known five years ago?
James: Several years ago I was working as a design partner in a digital agency here in San Diego. Our clients were startups and large companies like Qualcomm and Sony doing a mix of projects, but working with a lot of innovative technologies.
In an agency environment, you pitch the approach, but there's not a lot of research put into it. You're always skimming the surface. I found that if you're doing an end-to-end solution for someone on a deadline, you're probably not going to get the solution your clients business needs. Leaving the agency and being thrown into startups right after that and doing the same thing, it was always "push, push, push, ship it, let's get it out."
But it wasn't a message of, “Let’s get it out to hear from our users about it.” It was, “Get it out to get it done and move on to the next thing.” It wasn't about validating. And it was always this internal struggle with the product teams I was working with because we all knew what we should be doing and we weren't doing it. So my current self would go back five years ago and kick my ass for not saying, "Stop! All of this work and effort we're doing is not going to have any benefit if we don't take the time to dig deep into an area or two and test our hypotheses first."
That would be my advice to everyone; when you hear that voice in your head about what you should be doing, you should speak up. And you shouldn’t rely on having a dedicated researcher or using that as an excuse to say, "If we only had that person, we would do this." Pick up a book and watch some videos and learn the exercises yourself, and start to put it all into practice. Because the more you do it, the more it becomes a daily habit. It becomes how you think. So when you do have to produce something on a tight deadline, there's a much better chance that the work you do is going to be closer to right then if you hadn't.
Sofia: Do you remember when you finally started feeling like you had to speak up and say, "I'm not going to let this go through this way.”?
James: For me, it was a long time coming. Even early on, when I worked in other agencies, and I was working on digital products (not marketing websites), I had so many gut reactions or questions that were never answered. That continued to happen in many different places where I worked. It built up over time when I finally said to myself, "I'm not going to spend my whole career as a designer knowing that I'm not designing to the full extent, only touching a little bit of the process. I'm not doing the full work of what being a designer entails."
It just takes one person calling things out, and not placing blame. It's just, "Hey, I'm observing something, and there's likely a chance that you’ve observed it, too." Usually, you'll get some response back from the rest of the group that they’ve noticed the same things, but they haven’t done any research to validate if it's working or not. Now let's talk about the frameworks and stuff we can use to start solving the real problems.