ReOps CEO Holly Cole makes the case for research incentives

When Holly Cole joined a fledgling online community of research operations professionals as its ninth(ish) member, she couldn't have imagined it would grow into the standard bearer for the fast-growing research operations profession.

In a little over four years, the ReOps (short for “research operations”) community amassed nearly 12,000 members across the globe. They volunteer to help build processes and frameworks to bring standardization to new and growing teams of researchers. Cole, as co-chair and CEO of the community, has a pulse on the industry, the challenges it faces, and how strategies are evolving (or need to evolve) to adapt.

“I don't know any larger scale volunteer projects out there,” said Cole. “We put on workshops in over 100 countries. It’s intense how many data points we collect. And it’s all volunteer work. That’s something we're super proud of.”

Cole, a longtime researcher, linked up with people like Brigette Metzler and Emma Boulton to grow the organization from a message board with a handful of people to a global network of volunteers actively sharing experiences and knowledge.

The way Cole sees it, this was a community that was hungry to convene and find ways to help one another.

“The tank was already full of gas,” Cole said. “We really just kind of lit the match.”

One of the community’s first projects resulted in a stunning “flower graphic” — which resembles the unfurling of petals of an intricate flower — that visualizes all the functions that fall under research operations.

“We kept going back to this research and saying, ‘There’s more here.'” Cole said. “There’s a higher order of groupings and taxonomy here.”

The result of that work culminated in “The Eight Pillars of User Research,” a model and resources that have since been used as a blueprint for establishing a research operations team, Cole said.

One of the ReOps community's first projects resulted in a visually stunning graphic that's often referred to as the “flower graphic” because it resembles unfurling pedals of a flower. The graphic breaks down all the functions under research operations.

On June 8, the ReOps community is bringing its collective wisdom to the ReOpsConf 2022 in Brooklyn. Attend in-person or virtually to hear from the greatest minds in research, including a keynote presentation by Emma Boulton, head of research at Meta.

In advance of the conference, Cole sat down with Tremendous to discuss:

• the role of incentives in producing trustworthy research;

• her concerns about what budget cuts to incentives could mean for research quality; and

• the need for better resources to navigate the complexity of international incentives.

She also offered her top tip for running an effective incentive program (spoiler: Pay participants ASAP).

The following interview has been edited for length and clarity.

‘This is a community that at its core is about helping people at work.’

Russ: What does research operations entail?

Holly: Research operations is all about facilitating getting research in motion. It’s handling all the admin that otherwise bogs down researchers.

It’s a mix between people management and knowledge management.

• People management includes things like hiring and managing researchers and recruiting and rewarding research participants.

• Knowledge management includes things like tools and infrastructure, governance, and data privacy.

This is a community that at its core is about helping people at work, and helping people help people at work. They’re working for their coworkers. I think that’s something that makes the community somewhat unique and powerful.

Payments ‘the Holy Grail’ of research recruitment

Russ: What role do incentives play in research operations?

Holly: Incentives are a pretty key part of what we call “participant lifecycle management,” which basically means keeping participants happy from the first contact all the way through to the end. For some, that could mean several research projects over several years.

Things that go into participant lifecycle management include participant recruitment, panel management, as well as managing participant stipends and incentives. You need to figure out how you’re going to incentivize your participants, whether it’s with swag or donations or, if you’re lucky enough, paying them for their time, which is the Holy Grail.

Cutbacks to participant incentives will hurt research quality

Russ: What impact do you think current economic conditions will have on research incentives?

Holly: With everything that’s going on in the economy, I worry about cutbacks to participant incentives and what impact that can have on research quality. Cutting back on incentives is going to hurt feedback. It’s going to hurt companies’ ability to adjust and fix their products to address the way that people are reacting to the economy.

As a research leader, your first and primary focus is people. It’s headcount. And then it’s tools. You’re fighting for your budget and you've got to get your tools established somewhere in there. You’ve also got to find money to pay people to do the research. It’s a juggling act every single time. Unfortunately, incentives aren’t always high on the list. Often that’s the thing that gets put on the chopping block.

I’m hoping to get a project up and going to actually talk about that and talk about how important it is to put that higher up on the list.

‘Free information is horribly biased’

Russ: What’s the downside of asking people to participate in research for free?

Holly: If you’re not compensating people for their feedback, one, they are a lot less likely to give you any in the first place. So you’re fighting to get it. Two, the feedback that you do get is going to be lower quality, because free information is horribly biased.

When somebody gives you something for free, it usually means they’re complaining. It’s often going to be the squeaky wheel.

And how do you prioritize without decent feedback? The whole point of good user research is to know what to say no to. If you don’t have good solid research that’s balanced with not just quantitative research — A/B studies, all these other things — but with qualitative research as well, you end up going down a rapidly narrowing path that takes you in the wrong direction.

There’s plenty of examples of that in the past 10 years where it's like, “What were companies thinking?” They weren’t talking to people.

To talk to people, you need to compensate them for their time — especially as people’s time becomes more valuable because they’re having to work more in this economy.

Making a case for incentives

Russ: How can the ReOps community contribute to making the case for incentives?

Holly: Laws are constantly changing. Money is constantly changing. Exchange rates are constantly changing. At the same time, there’s something to be said for being able to give people confidence that they can walk into a meeting with their boss and their boss’s boss and whoever is holding the pursestrings at work and make a solid case for being able to pay participant stipends with a decent tool.

Give the reasons why it would be too much lift for them to do it on their own and have examples based on the kind of research that a massive multi-country volunteer effort can put behind it, so you can stand it up. The ReOps Community is well positioned to do that foundational research.

International research complicates incentives

Russ: More and more research is happening internationally. What challenges does that present for figuring out research incentives?

Holly: A lot of our questions come from researchers who have gone back to working at home and are spreading out their research into more global environments. They don’t have time to do extra research outside of their focused day-to-day work on the basics of incentivizing participants.

There are a lot of other related questions:

• How do I pay someone in South Africa, or Cambodia, or India?

• What’s the difference between paying a business and paying a consumer?

• What if it’s someone who’s just buying something at a retail store vs. compensating a database architect for an hour of their time?

• How do you incentivize people to talk to you in unstable places?

There was kind of a globalization where people were paying a uniform amount all the way around. You can’t really do that, because everywhere is a little bit different. You have to take into account the differences in currencies around the world.

It might be totally appropriate to pay someone in the United States $100 USD for an hour of intense conversation about your college plans or university plans. But if you’re talking to someone in India about their university plans for an hour, it’s probably not appropriate to pay them $100 USD. You need to know the conversion and the appropriate amount to pay them there, without having to ask every single person and then do a custom compensation for each person.

There’s plenty of places out there that are emerging markets where researchers really should be talking to people. But giving them an Amazon or Visa gift card isn’t really appropriate because it’s not an economy that’s trusting of traditional banking systems. A lot of NGOs and large engineering firms do research in places like this.

There’s complexity in domestic payments, too. So many people disregard the fact that one in sixteen people in the United States works for the government. Especially if you get higher and higher up, they can’t accept payment for their time if they’re talking about their job, because it could be considered biased.

It’s often that way with financial services, too. You can’t pay them. But can you make a donation? Can you do volunteer days?

That’s an entire research project to understand all of that.

No. 1 tip for incentives

Russ: What’s your advice for people setting up an incentives program?

Holly: My number one recommendation to keep participants happy is not to make them wait to get paid. Don’t wait until the end of that cycle of interviews or cycle of moderated testing or whatever it is before paying them. Don’t make them wait three weeks and have them go, “Wow, well, I guess they finally paid me.”

Try to pay them the same week — whether it’s a digital payment they get in their inbox, or an email verification of a donation to a charity of their choice, or an email confirmation that their swag is in the mail, with a tracking number.