How did Auth0 scale research 8x in 1 year? ‘We started with ops’

For the past few years Brad Orego has been hard at work scaling research at Auth0, a fast-growing tech company focused on the niche world of online log-ins.

“We’re a growing product organization with a growing design presence,” said Orego. “We needed to build a mature research function to support all of that.”

Auth0, based in Bellevue, Wash., handles more than 4 billion logins a day for software companies ranging from Pfizer and SoFi to Mazda and Siemens. Recently it became part of San Francisco, Calif.-based Okta, the world’s leading identity platform.

“Basically anything you sign into or sign up for, it’s probably powered by Okta or Auth0,” Orego said.

Orego joined as the company’s first full-time researcher in March 2020 and steadily grew his team, focusing on research operations first. It seemed counterintuitive to many that his team would devote all its early attention to setting up processes rather than doing actual research.

“I told the team, ‘It’s going to sound crazy, but it’s going to work long-term,’” Orego said.

By all accounts, it did. The company scaled its research 8x in Orego’s first full year on the job. He now heads a team of six people that supports hundreds of people across more than 15 product teams, marketing, and customer success in doing their own research.

Orego sat down with us to discuss:

• how he set up a research practice from scratch;

• Auth0's approach to incentives for global research participants; and

• his advice for setting up an incentives program.

Ian: Why is UX research important?

Brad: I pivoted to being a dedicated researcher in the early 2010s when I realized there was a lack of expertise and focus around research among tech companies.

Companies would raise a seed round and say, “We need you to design something.” I would say, “Well, I’ve got a bunch of questions. I need to understand the problem spaces and how people are solving this currently, and I need to test my designs before we build it.” They’d go, “We don’t have time for that. Just design something and we’ll build it.”

I’m like, “I’m twenty-three years old and you’re betting this million-dollar company on what I think is a good idea?” That was crazy to me. From there I focused more on research.
‍

Ian: Tell me about your role at Auth0.

Brad: My job includes everything from leading our research team to advising other teams on their research projects to setting our overall research strategy. I help shape how the research function fits into the overall goals of the company and product goals.

A big chunk of my time is spent reviewing requests and giving people feedback on research plans and making sure that they have everything they need to go out and do that research.
‍

Ian: What kind of research does Auth0 do?

Brad: The big reason that I wanted to come to Auth0 was that I wanted to work in this area of identity and access management. These are all really niche and specific, but every company needs our products.

We’re doing consumer testing now that’s going to come out as a white paper later this year. The big question we’re answering is, “What is the state of the world when it comes to authentication?” How do we get people to use multifactor authentication or biometrics or single sign on? What are the things that we can do to get people to move in this?

We do research all the time with consumers because we’re trying to understand how they interact with technology. We want to know what their fears are, what their behavior is like, what their preferences are. I love talking to people who work in the tech industry about this stuff because the way the general public interacts with technology is fundamentally different than how we do in tech.

One of the most common ways people manage their passwords is that they have a note in their phone or a physical notebook somewhere that they’ve written things down. It’s the number one thing they tell you not to do with the password—don’t write it down anywhere.

Ian: What current UX research are you excited about?

Brad: One thing we’ve learned from our marketing team is that more people want more flexibility when it comes to signin and signup. Right now there’s sort of a prescribed way to sign up for any sort of service. We are essentially trying to break the whole thing down such that you can do anything in any order.

Ian: Who do you recruit for research?

Brad: We have this interesting two-sided marketplace problem. Of course, we do research on the people that use our software—the software engineers, IT security teams, all these people that are implementing Auth0 and using it to build their apps. This is where we focus on democratized research. I feel confident giving someone some training and sending them into a conversation with a software engineer.

But we also have their end users, which is the general public. Pretty much anything that is consumer-facing we do ourselves because our research team has more training, practice, and expertise to interact with the general public. It can be difficult if you’re in a session talking to somebody about biometric authentication. It’s like, “What does that mean?” You have to have a bit more nuance.

Ian: Why did you focus so much on research operations when you started?

Brad: I joined Auth0 with this thesis that if you’re going to build a research team for scale, you should build operations first.

There are downsides. You don’t see a lot of progress at first because you’re putting all this effort into getting processes and tools stood up and making sure everything fits together. My team didn’t produce any research the first six months that I was at the company. At the one-year mark, we had increased the volume of research by about a factor of eight. We went from about one study per quarter across the organization to about one per team per quarter.

Ian: What’s your approach to research incentives?

Brad: I refuse to do research without some sort of reward for participants. We try to get creative with our customers. We want to provide a lot of different options.

Granted, it doesn’t have to be money. But even for swag items, you need a budget for it. We do in-person developer days in a handful of cities. We’ll set up a booth for research, and the reward for participating is a custom button you can only get by participating at that event. It’s a scarcity play.

We also have custom swag items like tote bags. But then you have to also weigh if participants are going to want to have this physical thing that took resources to produce. Do I need another water bottle? Probably not.

We give people the option to pick their reward and how they want to receive it—whether it’s an actual gift card to their favorite restaurant, a bank transfer, a prepaid Visa card, or some other option. Many of them work in highly regulated industries and can’t accept monetary gifts, so it’s helpful to have the option for charity.

We’re not a company that will give you a free month subscription to our product. I see startups do that. You’re making them do work and then rewarding them with your product? I don’t know about that.

Ian: Do these debates get more complicated when you’re doing research in other countries where $10 or $20 might be an astonishing amount?

Brad: This is a big, ongoing debate. Is it appropriate to pay somebody the same amount if they’re in a different country that has a different value associated with it?

Most of our users are software engineers. In the U.S., $20 is lunch. You filled out some questions and someone bought you lunch. That’s cool. But if you convert that $20 to local currencies in other parts of the world where we have customers, it’s a non-trivial amount. So then you start to get it into this interesting ethical gray area. Are you creating this incentive structure where people aren’t doing it because they want to share their opinion and help us out, but rather they’re doing it for the money?

At Auth0 we don’t localize the amount. We will pay in local currency—which is why we use something like Tremendous, so we don’t have to figure all that out ourselves—but we’re not going to pay someone less because they happen to live in a country that has a weaker economy. To me that seems unethical. It’s like, “We’re going to pay you less because you happen to live there.”

Ian: Are incentives more important for certain types of research projects?

Brad: There’s a difference when we talk about the research that we do with our customers versus the end users. I don't know if a member of the general public is going to care as much about an Auth0 sticker (laughs).

Ian: Do people use the charity option?

Brad: Yes. That’s really important for a lot of highly regulated industries with rules against accepting financial incentives.  

Ian: You mentioned that you had someone come to you today to talk for advice on an incentive. Can you tell us about that?

Brad: We’re working on deprecating some upcoming features. We want to understand what that path is going to look like— who’s using it, what are they using it for, and if we tell them it’s going away, what’s their failover?

We created a 10-question survey. Part of the debate that we’re having is essentially what’s the appropriate reward and how do we deliver it. Initially the team planned to offer $20 to the first 100 people who filled out the survey. But I see a number of reasons why we shouldn’t do that.

For one, it seems kind of cheap. Why not give everyone the reward?

It’s not good for our brand to offer an incentive where it looks like we don’t have enough money to do our research. Either you pay everyone or you pay no one. I don’t think there’s anything in between.

Also, we don’t want to incentivize people to act quickly as opposed to acting thoughtfully. We could end up with junkie data.

We also debated the reward amount. For a 10-question survey, $20 was too much. You don’t want people to fill this thing out just to get the easy money. We’ll probably drop that down to $10 or $15. We’re emailing 6,000 people, so I’m pretty sure we’ll get a decent enough sample.

Ian: How do you recommend people go about determining the right incentive?

Brad: There’s always a debate about how much you should pay people. That’s never going to go away. Ask your peers what they’re paying for different types of studies? For a lot of companies, there’s different pay grades for a general consumer, a professional, and an executive.

Ian: Do you have advice for managing relationships with legal and finance teams?

Brad: I think a lot of research teams view compliance as a burden. They don’t want to deal with it. I think that’s the completely wrong approach. It’s a huge missed opportunity.

Be proactive about building a relationship with your legal, finance, and compliance teams. They want you to be able to do your job. They want to work with you, but they have no idea what you do. They don’t have the context. The last thing you want to have happen is that you’re ready to get some research thing going and then somebody from legal or finance finds out about it at the last minute and you have a fire drill.

The research agreement form we give to participants is a one-page PDF with very plain language. I fought hard across many companies to get to this current evolution.

Also, we haven’t run into this often, but it’s important to tell your participants that there may be tax implications to participating in research. In the U.S., if you receive $600 in a year, it becomes taxable income. It varies by country and can be very complex. We include simple language in our recruitment emails letting people know there may be tax implications, and that we won’t manage that for them.

Ian: Do you have any advice for people setting up an incentive program?

Brad: If you have money to pay people for research, you have money to use a tool to manage your incentives. There’s no way I’ll ever go back to managing my own rewards and incentives.

I’ve done everything from buying Amazon gift cards and mailing them out to what we do now using something like Tremendous that makes it super easy. Just like I can train anyone how to do a usability test, anyone can type in an email address, select a dollar amount, and hit the “send” button.

There are providers out there that require you to sign a big yearly contract, and then you have to pay people on top of that. I made that mistake enough in my career.