Building an Integrated Qualitative and Quantitative User Research Capability

by Jared M. Spool

A case study of how one team used quantitative and qualitative data to their advantage.

Sam Nordstrom had a big problem to solve. As a product manager for Intuit’s Quickbooks, Sam had learned that the new feature his team just shipped wasn’t used nearly as much as they’d hoped. He didn’t know why.

When they demoed their new feature, their users told them they loved it and would use it. Everyone agreed the feature offered huge advantages to QuickBooks users. Yet, now that it was out in the world, the users weren’t getting those benefits.

Sam’s team had implemented a new way for QuickBooks users to get their customer invoices paid. The software added a button to invoices labeled Pay Now. When customers pressed the button, the software set up an instant transaction that sent their invoice payment directly to the Quickbooks’ customer.

Early testing showed, when the customers paid this way, the Quickbooks user would get paid substantially sooner than when they used regular invoices. It was a huge advantage to every small business person using the product, as it increased their cash flow.

But users weren’t using it. Sam and his team didn’t know why.

Learning from deep hanging out

Sam and his team visited their users. Intuit has customer visits in their DNA, so this was not a special trip. It was common for the teams at Intuit to visit customers.

Yet, this time, Sam’s team was looking for specific answers. Why weren’t the users taking advantage of the speedy payment feature?

They hung out with several QuickBooks’ users. They watched those users send out their invoices and process their collections.

They saw it was awkward to use the Pay Now feature. Many of their users preferred Gmail to correspond with their customers. The QuickBooks users often had an existing correspondence thread, complete with price quotes and status updates.

The QuickBooks users would invoice their customers by replying with a Gmail message and attaching the invoice PDF. To these users, this made more sense than sending the invoice from within the QuickBooks user interface. But those users didn’t get the Pay Now capability, because saved PDFs can’t have the necessary button.

When users sent an invoice from inside the product, instead of in Gmail, their invoice message was disconnected to other messages in their thread. The users’ customers missed the Quickbooks-sent messages in their inbox, as they had a different ‘from address’ and ‘subject.’ This made collections more difficult.

Learning from deep data dives

Visiting a few users showed the new feature had a clear problem. Yet, Sam’s team wondered how widespread this issue was. Were other users avoiding the new Pay Now feature for the same reasons?

When Sam and his team returned to the office, they dove into the data QuickBooks already collected from their users. How many users were saving invoices as PDFs? It turns out it was a high percentage of regular users.

When those users saved invoice PDFs, were they attaching them to email threads in Gmail? The team asked a bunch more users and, sure enough, many of them were doing just that.

Were the users who were saving PDFs getting payments through the new Pay Now payment system? Their usage data said they weren’t.

Were the users who did get payments through the new payment system also saving PDFs? Rarely.

Their usage data suggested most users were saving PDFs and not using the payment system. More importantly, those users saving PDFs were paid substantially slower than those using the payment system.

Testing a hypothesis

Sam and his team formed a hypothesis: If they could get users to send invoices with a Pay Now button, those users would get paid faster. But how could they do that?

The team developed a Gmail extension that inserted an invoice with a Pay Now button directly into an email reply. They tried the extension out with a few customers and, sure enough, the customers found it easier than saving PDFs, adding them as attachments, and dealing with the collection headaches.

As Sam’s team rolled out the Gmail extension, they watched their usage data. Were users who installed the extension using it? Yes. Did they save fewer PDF invoices? Yes. Were they being paid faster? Yes.

Integrating Qualitative and Quantitative Data

Sam and his team solved their big problem. They created an innovative solution to a hard problem.

The team couldn’t have done it without their data. They formed their deep understanding by hanging out and observing a few customers. They learned how extensive the problem was by diving into the product’s usage data. They observed, in real-time, the changes in the usage data, as they rolled out the fix to the problem.

All of this data—qualitative and quantitative—guided the team to an ideal solution. It told a complete story that drove the team’s problem-solving process.

Sam’s team built a sophisticated approach to user research. They used data strategically, which delivered insight into their problem and its potential solutions. They couldn’t have achieved the improvement as quickly without it.

To deliver better designs, teams need to grow their own qualitative and quantitative data collection capabilities. They need to integrate these efforts into their user research processes. With this expanded capability, they’ll drive their organizations to deliver better-designed products and services.

Published here on January 22, 2022.