A Look into Twilio’s Process for Studying Customers

“The name of the game when creating and growing a successful business is understanding and solving your customers’ biggest problems. To do so, it’s imperative to have efficient means to identify these problems early and fully.”

Laura Schaffer,Product Manager for Twilio’s Experimentation Platform

At Twilio, a Fierce Hunger to Learn Fuels Research as a Core Business Strategy

Twilio enables developers and businesses to build SMS, voice & messaging, and video applications using simple and powerful APIs. Twilio’s customers move quickly, so Twilio must too.

“To expand to an ever-evolving market like Twilio’s, you need to work quickly to stay in tune with the challenges customers face – both the ones they face today and the ones they will face tomorrow,”  Schaffer says. “Customer discovery efforts are critical. How do you best put yourself in the customer’s shoes, and do so in a way that is efficient and easy to do at all stages of product development?”

Understanding and anticipating customer needs is the backbone of Twilio’s product development strategy. All Twilio employees aim to identify and address these needs in some way as part of their jobs.


“…Customer discovery efforts are critical. How do you best put yourself in the customer’s shoes, and do so in a way that is efficient and easy to do at all stages of product development?…”

— Laura Schaffer

Customer Discovery Is a Part of Everyone’s Daily Responsibilities

One of the biggest challenges that companies face is that they lack time, budget, and resources to administer in-depth customer discovery programs. As a result, key customer insights are missed. Businesses can end up delivering products that don’t optimally solve the customer’s problem. In order to ensure they stay in step with customers’ needs, Twilio has adapted aspects of formal research to the constraints of fast-moving, aggressively evolving operations.

Schaffer, Product Manager of Twilio’s Experimentation Platform, explains how:

“At Twilio, the Experimentation Platform team provides services to enable Twilions to experiment at every stage of product development – from validating ideas in the early exploration phase, all the way through measuring the efficacy of post-launch optimizations. This empowers teams to deliver products and experiences that are data-proven to best serve our customers.”

Within a cross-functional operation like Twilio, ideas are abundant and opinions are strong. The question is how team members can identify the ideas that will hold water with customers and the ones that won’t. The process to do this identification begins with a customer insights program that makes it easy for teams to constantly experiment.

Over the last year, for instance, teams within Twilio have come together to improve onboarding experiences. The company offers a diverse product portfolio, which means that every customer has unique onboarding needs.

“There are so many ways to approach this challenge – you can try bucketing customers into different segments, you can try changing up your content, surface information at different times, etc.,”  Schaffer elaborates. “The hard part usually isn’t coming up with the hypotheses for what might work best – it’s knowing which are the right ones to try first.”

Qualaroo helps Twilio hone this process.

“Qualaroo allows us to connect with our customers’ questions directly and almost effortlessly,” Schaffer says. “The data we get from Qualaroo help us prioritize which hypotheses to formally run as an experiment first.”

“In less than 5 minutes, teams can go from having almost entirely opinion-backed ideas to having ones that are data-backed by targeted customer feedback. This step in the discovery process makes us more efficient and increases the chance that the experiments we run will yield a positive impact.”

Twilio Nudge

Qualaroo’s Question Engine Connects Fragmented Data Points

“As an organization, we’re believers in the small teams model in order to move quickly and deliver new functionality every week. Building platforms that each of our teams can use helps us stay coordinated despite how quickly we’re all delivering these experiences,” says Schaffer.

Qualaroo makes it easier for groups to collaborate. Across Twilio, 18 teams have access to the tool as part of their workflows. These teams continually use the data they receive from customers to shape their thinking and prioritize what they test. The simplicity and ease of this approach to information-gathering is especially important for an organization like Twilio that ships features as quickly as it can.

“Qualaroo immediately became a top choice for us because it is so easy to use,” says Schaffer. “When we first onboard new Twilio teams to Qualaroo, we immediately hear many different ways they want to use it – and they often start running these within the week. The speed with which teams can go from 0 to getting data is highly valuable to us. We operate and move quickly, so tools must be self-serve and easy to use in order to be viably added to a team’s development process.”

Here are some ways that teams across Twilio are using Qualaroo to quickly understand how to improve customer onboarding, engagement, and retention rates:

  • Collecting and scheduling targeted customers for user testing or product feedback interviews  
  • Asking targeted questions about what specific customers want or need help with achieving, to spark ideas and see if our expectations match what customers tell us
  • Providing step-by-step guidance to explore hypotheses about what kind of assistance will best help new customers. (When these are successful, they are formally built into the customer experience)
  • Gathering data to share between groups to help address differences in expectation or opinion

“These use cases help us gather the insights we need to make customer-based decisions in a fast-moving environment. Qualaroo gives us the insights we need to answer specific questions, connect with our customers, and in general feed ideation during early stages of product discovery.”

Do you want to improve conversions?

Use Qualaroo surveys to gather insights and improve messaging.