An online casual gaming company is testing a new piece of functionality. It did a standard A/B test where half of the players saw variant A and half saw variant B. The test concluded that variant A improved a critical element of the site’s conversion rate with a high confidence level, and the business moved all their players to the winning variant. Unfortunately, the income immediately began to decline. Only when the business looked further did it make a critical discovery. What is common in casual gaming is that usually only 1% of the visitor base spends any money. So, while variant A performed better overall, variant B performed better for visitors who actually spent money.
This example highlights two of the most common challenges businesses face when trying to improve their digital engagement with customers. One assumes that all customers are the same, and the other focuses on a siled digital outcome rather than a business outcome. It is true that variant A improves the conversion rate but, in this case, the conversion rate does not match the profit. As the saying goes, “The operation was successful, but the patient died.”
Another common challenge is around how to prioritize and focus on customer experiences. As businesses invest more in their digital engagement with consumers, there is an endless backlog of potential improvements driven by new ideas, improved device capabilities, and the relentless advances in AI – all of which continue to raise consumer expectations. Standing still is not an option, but knowing where to focus and what will make the biggest difference in your customers’ experience is not easy.
As consumers, we all know the importance of a seamless customer experience. The purpose of this article is to highlight the most common things that go wrong when developing digital customer experiences, and to suggest a method that will help get more opportunities to do it right.
Successful businesses focus on three main principles:
- Put the digital experience in a business context. While it’s tempting to focus on conversion rate and frame it as the goal of the digital experience, that’s rarely the route to success. It is important to understand how the digital experience drives profitability and focus on solving the business problem.
- Recognize that customers are not created equal. Leverage your customers’ digital experience and understand what’s important for different customer segments, especially your high-value and high-potential customers. Then personalize the experience as appropriate.
- Make zero-based decisions. Continuous improvement is good, but it’s important that you’re willing to think outside the box first principles, and to reassess and justify what you did and what you could have done differently. This helps ensure that the digital roadmap is prioritized and focused on the highest impact actions.
Put the Digital Experience in a Business Context
Digital experiences can be thought of as a funnel — customers enter a site or app with a goal in mind, such as booking a taxi, ordering a pizza, or buying shoes. Some succeeded and some failed. Most businesses that begin to manage this journey begin by measuring the various stages of the funnel. The overall conversion rate (the percentage of customers who reach their goal) is often seen as the ultimate measure of success.
However, any digital experience is (almost always) a window into the overall business proposition. The digital experience is strongly dependent on all aspects of a business and cannot be analyzed or optimized in isolation – just look at the spikes in the conversion rate during any sale or promotion. Obviously, individual elements of a digital experience can be improved – faster speed, bug fixes, better navigation – and these kinds of improvements are rarely bad, but often lost. the greater the opportunity.
A digital fashion retailer focused on increasing its “add to basket” rate by testing several different product page formats. But when they look at product conversion rates over time and across different products, they realize that the conversion rate for individual products is dominated by product quality, price, and availability. Great products, in-stock and competitively priced, have a conversion rate 10 times higher than the average product.
Their focus then shifted from page formats to delivering a better availability experience, by prompting customers to enter their size and then directing them to products available in their size. on. The retailer is also starting to add measurement of “add to basket” and conversion rates along with measurement of customer experience. Measuring and optimizing customer-perceived availability – that is, the products the customer sees in stock in their size – has become more important than turning on different button colors.
Another example further illustrates the challenge of finding real opportunities. A food delivery company is trying to optimize how expected delivery times are displayed on their app to drive customer satisfaction. However, when they talk to unhappy customers, they realize that the point of failure is a little more nuanced – the problem is less about the messaging and more about when the expected delivery time often varies between point of order and delivery. The expected arrival time is updated every few minutes and is thought to be “helpful” but actually brings a significant dissatisfaction. In addition, the delivery company only keeps order time and delivery time, making the issue challenging to diagnose. The company acknowledged that the cause of customer frustration had nothing to do with the digital journey but was driven by the algorithm for the predicted delivery time (which they later improved).
These examples highlight the need to unpack the “digital” element from interaction with the rest of the business. An observed failure of a digital journey often has nothing to do with the digital journey but a symptom of the overall business proposition. This trap is all too common when digital becomes a separate division, disconnected from the rest of the business. Digital experiences don’t respect organizational silos.
Customers Are Not Created Equal
A hotel chain identified a high bounce rate in search traffic from Google as an issue – the bounce rate is the percentage of visitors who enter and leave a site immediately, and only see one page . Their initial digitally focused solution was to conduct A/B testing of different landing page layouts. The effect on the overall bounce rate, while positive, is small. Further analysis of the data revealed an interesting insight: the bounce rate is driven by the customer’s device and the specificity of their search. For example, a customer searching for “hotels near Geneva” on a desktop prefers a large map and default radius to see all hotels in the Geneva area. But a customer searching for “hotels near Geneva airport” on a mobile device prefers a smaller default radius around the airport. Landing page design is not the issue. The real business opportunity is personalizing the default map radius based on customer search behavior which has a huge impact on both bounce rate and overall business performance.
Personalizing your customer experience expands understanding of what is important to your high value, and high potential value, customers. This same fashion retailer began analyzing its product availability for different types of customers, and discovered that its highest value customers had the worst availability experience. Further digging reveals that higher value customers are usually looking for either a smaller or larger clothing size. However, the seller makes decisions about size ratios based on old-fashioned rules of thumb, and does not adjust to the new online reality that changes the data available and also the mix of customer. They have changed their approach to optimizing size ratios.
As we saw in the opening example, A/B testing is seen as the gold standard but is full of complexity and misinterpretation. A critical challenge is the failure to look beyond the headline results. Only by analyzing the results at the atomic level — by devices, browsers, customers, or marketing sources — can one really understand the true effectiveness of a test, and if the best way is to choose in A vs. B, or personalization (as with the hotel example above). Oversimplification misses opportunities and at worst causes a lot of waste. Nothing about the test is easy.
Make Zero-Based Decisions
It’s common to have teams focused on continuously improving the digital experience. It’s less common to have zero-based thinking with teams willing to revisit a problem. But businesses that constantly challenge their priorities and actions from first principles make better decisions.
A retailer has developed a product recommendation system that offers cross-sells and upsells at various points throughout its site. The algorithm that powers it is intended to optimize “add to basket” and conversion rates. When the new algorithm went live, they were delighted with the results. It performed better than the previous recommendation.
However, they want to challenge taking success at face value. Further analysis reveals insights that all is not as it seems. Looking at all products and inventory, only about 12% of the inventory value is always recommended. It is known that the recommendation algorithm is determinedly focused on recommending bestsellers, which are often sold, and is less prone to recommending surplus stocks, new products, or products that are more difficult to sell. The metric chosen to optimize is misaligned with the business goals of sales, gross margin, and inventory efficiency.
Put It All On
Digitally led businesses are overwhelmed with opportunities. Identifying the top 100 ideas is easy, identifying the top 10 is difficult. Being deliberate about where to focus is a key management challenge and at its core it is an analytical challenge. What should be measured to drive the correct diagnosis? Which digital goal to optimize? How to understand the connection of the digital experience to the rest of the business? And as the examples above highlight, there is rarely a shortcut to doing analysis.
