Doing Research with Developers—the Art of the Science

Why We Do Research with Developers

Developers are the lifeblood of the technology industry. Their work drives innovation, brings us new products and services, creates and shapes customer experiences, and every so often, becomes a global phenomenon that fundamentally alters the fabric of our society. Small wonder that technology companies spend a considerable amount of time doing research with developers, trying to understand developers’ personas, assessing their sentiments, and gauging their view of the world.

But with 24 million developers globally and 4.4 million in the US, why is it so difficult to conduct research with them? Well first, they don’t hang around on regular consumer research panels, waiting to take a survey, so a researcher really needs to know where to find them. And, even if you can figure that part out, it requires deep industry knowledge and expertise to successfully engage them and get them to participate in your research. Then, after you’ve expended all of your best efforts, how can you be sure that you’re getting the developer insights that matter to your business?

It’s a daunting task, but with developer research becoming much more prevalent in the technology sector, I had a chat with a couple of colleagues who are experts in this area to dig into the intricacies:

• Erin Smith, the director on our Google account team
• Marie Boehmer, the lead analyst on our Google account team

Why do you think it’s important for technology companies to be conducting research with developers?

Erin: If you’re a tech company like Google, Apple, Amazon or Facebook, developers are building the digital bridges that take your products to market with millions of potential end users. You can’t get your product, platform, application or service in the hands of those end users if you don’t have developers on board supporting it. And, developers typically expect plenty of support in return from the tech companies whose tools they are using. We know developers will gravitate toward the more functional, successful or widely used tools, but they will also prefer those that are more collaborative and easier to work with. So, at a basic level, understanding developers and their needs is crucial to building the collaborative partnership to get to the end users, and that means doing plenty of research.

Marie: Let’s not forget that each of the companies Erin mentioned, and hundreds more, work in a hyper-competitive environment that moves faster than any other business sector. So it is really crucial to understand how you stack up against strong competitors and how that’s changing. Specifically, where you come up short, because you need to fix those issues quickly, but also where developers perceive you as having advantages, so that you can accentuate those in your messaging.

Erin: Remember, perception is not always reality. Developers are generally very smart, but they also get stuck in their ways. For example, we conducted a study where quantitative research showed that the biggest barrier to adoption of a new platform was developers having to learn a new language. They thought it would be too time-consuming or difficult, and felt it was just easier to stick with the languages they already know. But we did some qualitative deep dives, and found that when developers actually got into it, learning the new language was easy and well worth the effort. Developers felt more empowered and able to tackle new things they hadn’t been able to achieve before. So the research was critical to overcoming that barrier—allowing the client to effectively communicate the benefits of putting in the effort, while supporting it with testimonials.

Marie: That’s a great example. At the end of the day, our tech clients want to understand where developers spend their time, where developers see the value, and really get to know the environment developers are working in.

As you look to assess developer sentiment, what are the most important measures?

Marie: Well, measures of satisfaction and favorability are particularly useful. Satisfaction is most important because it relates directly to the developer experience, whereas favorability is more of a perception measure and doesn’t necessarily reflect personal experience with the brand or product. For example, an Android developer might be favorable toward other tools from Google that they haven’t used or haven’t used recently, simply because they exist in the same Google ecosystem.

Erin: I also think that some standard customer experience (CX) measures, such as likelihood to recommend and loyalty, need to be tailored more precisely for the developer experience (DX) space. Enterprise developers are rarely the final decision-makers, but they can be strong influencers. So you need to get to a deeper understanding of what developers are saying to people—beyond simply knowing they are “recommending” your tech—and whether they are a genuine advocate for something.

Marie: Then, if you want to understand your metrics on a deeper level, you need to benchmark. First, you need to compare yourself with competitors, like I said earlier. There’s no point in celebrating developers’ satisfaction with your product if they still see your competitor as the market leader with the best features and support. You also need to understand the relative importance of different attributes of satisfaction. After all, you want to ideally move the needle on the features or support services that matter most to developers. So something like a MaxDiff analysis—where the respondent chooses between their most and least preferred options—can play a very important role here.

That’s an interesting point. Do you generally see a lot of usage of marketing and data sciences in developer research?

Erin: Absolutely. Something like a MaxDiff is really useful to help understand the relative importance of product features. But the technology space is often way more complex than others, and developers have to factor in multiple variables such as features, time, costs, support and so on into a single decision. So, we also make extensive use of data sciences to help map out trade-off scenarios.

Marie: Another one that is common but often overlooked is a driver analysis—understanding the relative importance of the different attributes that drive satisfaction and favorability. It’s surprising how often the most important feature in a MaxDiff is not actually the key driver of the personal satisfaction of the developer. And let’s not forget segmentations. Developers are very diverse in type and motivation. Some of the best research I’ve seen has been when the client has segmented its target group of developers and really got its arms around the different personas and their role in the influence model of their organization.

It sounds like there’s more to this than just understanding developer sentiment?

Erin: For sure. DX is certainly a big part of what we do, but it totally depends on where your product is in its life cycle. You always want to understand the developer experience, but if your product is in beta or early release, your focus is going to be a bit different. You’ll want to know how developers become aware of your product, where they see its value and what makes them (or their organizations) decide to try it. We do a lot of qualitative research at this stage, going much deeper with a select group of developers to really get attuned to their initial thoughts and reactions.

Marie: A good way of thinking about DX is in comparison to regular consumer research. All companies want to understand CX, and all tech companies want to be doing DX. Similarly, most companies invest heavily in brand research to understand customer perceptions, and tech companies will do the same with developers.

Erin: But it’s also a bit different. The sort of “brand” research we do with developers gets very involved in the product detail, such as awareness of the features, how it can be used, how it compares to other products and how it can be compatible with other products. So, the research can be a lot more detailed than that top-of-the-funnel brand research you might be used to. Essentially, it’s more about measuring developers’ understanding than just their awareness.

What are some of the key considerations to take into account when doing research with developers?

Erin: The first consideration is getting a clear understanding of the types of developers that matter to the client. Are we talking enterprise developers only, or ones that work for agencies and are hobbyists in or out? Then, obviously, the developer’s area of focus, such as mobile, web, cloud, IoT and so on. If you don’t define the audience carefully up front, the data is going to be riddled with inconsistencies.

Marie: The client also needs to carefully consider whether the research is going to be branded or blinded. It seems easier and cheaper for tech companies to just send a branded survey out to their mailing lists; however, when developers know who the sponsor of the survey is, they invariably provide very skewed data. It’s much better to blind the survey and keep the developers more objective in their responses.

Erin: You also need to think through the scope. How many markets does it make sense to field in? If India, China and Germany really matter to you, do you want data from individual countries or is regional data like EMEA and LATAM just as good? You will also want to consider how often you want to field. Early-stage programs might want to field quarterly to gather early data quickly and track changes in a fast-moving market, whereas mature programs can often get by with once or twice a year.

Marie: Because I’m an analyst, I’m always thinking about the sort of data that we need to be gathering, and how to go about it. For example, going beyond basic firmographic data to get a deeper understanding of how the workplace functions, decision-making, team structures and so on. I’m also thinking about how best to frame the questions. Developers are an exceptionally intelligent and critical audience, so your survey needs to speak their language and show that you understand their world in order to build rapport and, thereby, obtain the best possible insights from them.

Thank you Erin and Marie for sharing your experience and insights!

If you would like to learn more about conducting research with developers, send us a note. We’d be happy to share more ideas on how you can go about gleaning insights from this critical stakeholder group.