In 2016, Duolingo, the popular app that allows you to learn a language with just a few minutes of gamified practice a day, released an English language proficiency test, a common application requirement for international students applying to undergraduate and graduate programs. The test, which prospective undergraduate and graduate students can take online in just 45 minutes, provided an alternative to the major language proficiency players like TOEFL and IELTS.
But the Duolingo English Test team soon realized the new product had a problem: while the test gave students an overall score, it did not provide separate scores for subcategories of their language skills, or subscores. Survey data showed admissions professionals, who were used to receiving these subscores in reading, writing, listening, and speaking from other English language proficiency tests, needed Duolingo to offer the same subscores to accept the test at their institutions.
So Duolingo went to its admissions advisory board to ask: What questions were admissions professionals asking about applicants that they needed the subscores to answer? This stakeholder-sourced feedback enabled Duolingo to clearly define the problem: Admissions professionals wanted to know that an applicant could communicate with their instructors and classmates, contribute in class, and keep up with their class reading and assignments without extra academic support.
Armed with that data, “a giant light bulb flashed,” said Justin Goff, senior product operations manager at Duolingo. Instead of developing the traditional subscores, Duolingo introduced four categories (literacy, conversation, comprehension, and production), all designed to directly answer admissions officers’ questions.
The story of how Duolingo developed its English test subscores represents an approach to problem solving that is common among product marketers, who routinely face the challenge of understanding ever-evolving customer needs and business objectives, a constant pressure to differentiate their products from those of competitors, and the challenge of rallying a team to efficiently achieve that vision.
“Good product managers prioritize problems over solutions and obsess over the problem for a while before obsessing over a solution.” – Justin Goff, Duolingo
This type of thinking represents a shift from the traditional marketing adage that higher education marketers ascribed to for decades: “How do I get people to want what I’m selling?” Instead, product marketers begin with “What do people need that I can create?”
Higher education institutions are under intense pressure to adapt to the myriad ways that prospective students’ expectations, interests, and demands are changing — and even to combat what appears to be a waning appreciation among some demographic populations in the value of the college experience. Higher education institutions must adopt a user-centric approach driven by market research to continually refine or create offerings to achieve — and define — the institution’s mission. It’s a daunting challenge, but one for which higher education leaders can look to product marketers, who can offer important lessons.
Lesson 1: Pull insights with a good mix of research methods
At the heart of product marketing is data and research, said Helen Richardson Greenlea, senior product marketing manager at the software solutions company Jenzabar. “We really like to take a look at hard numbers and statistics and try to identify market trends and market problems and unaddressed pain points,” Richardson Greenlea said. “Looking at those gaps in our portfolio for example, those are the things that help us determine what new products we’re going to decide to develop or what new features we will be adding or implementing to our existing product line.”
To find opportunities for your institution to improve, Richardson Greenlea encourages higher education professionals to think of their schools like a piece of software. What programs are being utilized and which are being underutilized? “And when something is underutilized, is it a question at a school of retweaking a major?” Richardson Greenlea said. “Is it advertising for a major better? Is there something in the recruitment area? Is there a retention issue? Looking at the different degrees and subjects and course areas, considering them as a product, you’re able to remediate based on that data.”
Richardson Greenlea recommends that higher education institutions research skills gaps, and then create flexible programs that address the needs of growing markets and lifelong learners.
“That’s what makes tech a fun space to work in: No idea is bad, and everyone has a chance to contribute and decide what we’re going to build.” – Krystle Song, Zeemee
Using a mix of quantitative and qualitative research to pull insights is key. Duolingo uses user analytics tools, A/B testing and experimentation, surveying, user interviews, prototyping, usability tests, and academic and market research, Goff said. He recommends using quantitative methods to validate qualitative and vice versa.
Kim Moon, senior lecturer of marketing at the University of Illinois at Chicago, emphasizes human-centered research with her students, which allows the researcher to develop empathy by putting themselves in the consumer’s shoes. Though surveys and other ethnographic means can be helpful, keep in mind that users may not always be the most reliable in articulating their needs or behaviors, she said.
To analyze how long students would wait in line for coffee, for instance, she had students observe the line. “Students wouldn’t be able to tell you, ‘I waited five minutes before I aborted the line,’” Moon said. “But by observing, you can walk away with a lot of intel to help [the coffee shop] better understand at what point it is critical to engage the student so they won’t abort.”
Higher education, like coffee shops, can better analyze the barriers to prospective students submitting an application by observing their behaviors.
Creating an advisory board of important stakeholders works especially well in the higher education environment, Goff said. He refers to Duolingo’s advisory boards as their “special ingredient.”
“We assemble them very intentionally to be a good cross-section of the market they are looking at,” he said. “We bring the people involved in making the decisions about the product and have them help us make it.”
By providing real-time feedback, advisory boards can help you tighten the feedback loop and minimize disruptions during the year, Goff said. “[In higher education], a lot of constraints have to do with the academic calendar,” Goff said. “You don’t have time to take something to market and see if it works. We need to make sure it’s going to work before taking it to market.”
Lesson 2: Obsess over a problem before finding a solution
Once you start hearing the same information from multiple sources, you can start coalescing the data into a truth and move forward to addressing any problems, Goff said.
Good product managers will “prioritize problems over solutions and obsess over the problem for a while before obsessing over a solution,” Goff said. “I think that’s completely critical.”
If you do it right, Goff said, you may find that different issues may actually be symptoms of a common problem. Conversely, what may at the surface have seemed like similar problems are actually very different and require different solutions.
Having a better handle on the problem makes tactical decisions much easier and less painful, Goff said. “If everyone is focused on the same problem, it’s easier to avoid distractions and make good tactical decisions much faster.”
It’s also easier to assemble the right team, Goff said. “If you don’t understand the problem, it’s hard to find the right experts, and you’ll end up with a narrower range of options to consider,” he said.
Goff credits having the right people in the room for the outcome of Duolingo’s English test subscores.
“Because we defined the problem and had the right people in the room, we had the ‘aha’ moment that we don’t have to do the old four subscores,” he said. “I’m not sure if that option would have come up in another context.”
Lesson 3: Leverage everyone’s ideas regardless of role
In higher education, the responsibility of coming up with a great master plan typically comes from senior administration. Product managers, who typically work in the tech industry, often leverage ideas and perspectives from across the organization by getting everyone involved.
“We really like to take a look at hard numbers and statistics and try to identify market trends and market problems and unaddressed pain points.” – Helen Richardson Greenlea, Jenzabar
“In tech, it’s much more democratized, and leaders push that thinking across the company regardless of their role,” said Krystle Song, head of product at ZeeMee, a higher-education-specific social media platform.
Tech leaders are responsible to make their goals clear and well known. Then they encourage free-form thinking and ultimately organize everyone’s thoughts, Song said. Allowing all staff to contribute to developing solutions can increase diversity of thought and benefit the research, problem defining and solution development process.
“That’s what makes tech a fun space to work in,” Song said. “No idea is bad, and everyone has a chance to contribute and decide what we’re going to build.”
Higher education institutions face immense challenges in refining or creating academic programs: the restrictive timing of the academic school year, the time and research required to develop or change programs, and the current crisis and eventual legacy of COVID-19. But by following some of these lessons from product marketers, higher education professionals can make the most of the opportunities they do have to update their offerings to better serve their students and achieve their institutional missions.
