S2 Ep. 5: How Do You Surface Assumptions in Product Discovery?
Subscribe: Apple Podcasts | Google Podcasts | Spotify | RSS
Hope Gurion: In the previous episode, we heard the horror stories of Product leaders and teams haunted by the risky assumptions that surfaced too late. Is this destined to be your fate as well? Not if I can help it! In this episode of Fearless Product Leadership, we learn how to mitigate those risky assumptions from seasoned product leaders who answer the question “How do you surface assumptions during product discovery?”
Welcome to the Fearless Product Leadership podcast. This is the show for new product leaders seeking to increase their confidence and competence. In every episode I ask experienced and thoughtful product leaders to share their strategies and tactics that have helped them tackle a tough responsibility of the product leader role. I love helping emerging product leaders shorten their learning curves to expedite their professional success with great products, teams and stakeholder relationships. I’m your host and CEO of Fearless Product, Hope Gurion.
Assumptions that never made the light of day frequently unsettle even the most customer-centric, evidence-driven product teams. Why? Because surfacing assumptions is not yet a widely practiced ritual for most product teams, even the advanced ones dedicating significant effort to customer discovery. In my quest to produce this episode, I found only a few product leaders willing to share their practices which is why I only have 3 guests on today’s episode which means you’re getting extra Hope time in this one.
Fearlessly tackling the question “How do you surface assumptions during product discovery?” are:
Marc Abraham, Head of Product-Engagement at ASOS, former CPO at Settled
Sean Murphy, former VP of Product at Target and VP of Product & Engineering at CustomInk
Stay tuned to the end of the episode for more examples and resources to help your teams surface assumptions when I share my recommended techniques to surface assumptions at all points on the product development journey from identifying market/TAM to delivering your initial product in a pilot or early access program.
First Marc Abraham of ASOS shares that he relies on that all-enlightening question, “Why?” to help his teams interrogate their ideas for their underlying assumptions. And Marc describes the technique he uses to mitigate risk in product development, the cross-functional pre-mortem.
Marc Abraham: When it comes to surfacing assumptions and particularly working with people who haven’t necessarily, for good reasons, have thought in that kind of way; maybe more used to focus on delivery or focusing on happy paths. To start thinking, assumptions is quite a big step. So, for me, the way I typically tried to bridge that step and making part of people's daily habit so almost, before it even becomes a more formal practice, if you like. It's just one simple word, which is: WHY.
I know it sounds super simple, we all know about the five WHY's, but getting into the habit of asking why is so important, if you want to really uncover any assumptions. What typically happens is that people will come to you, they can come from all kinds; they can be customers coming from within the business, can be external people with ideas. They come to you with solutions and in their minds, it's the best idea should definitely do it, it's a no-brainer. So, I appreciate, it's hard to say, “Where is the idea coming from? How do you know it's a good idea?” and the simple and gentlest way I feel, to do that, is to start uncovering some of the underlying assumptions, is to ask why. The way I sometimes do this, again with particular product people, are not necessarily used to that way of questioning, if you like, is to just do it with them. They come to me and say, “I want to do this” or “I want to implement this” or “I want to test this”. I'll say “Why?” they’ll give me a reason and then I'll say, “Why” again. It's classic five why and after the second ‘why’ it dawns on them what I'm doing, and they might find it a bit odd and say “Can you stop asking why and give me the answer and say yes or no? It's all I’m asking for” but equally, they start seeing the value of probing and really trying to uncover things.
So, that's the simplest way and that's typically where I start. Then, when he gets into more kind of formal practices of trying to unearth that and tactics that you can use, is when people start thinking about experiments; whether it's experiments or even just learning from customers. Again, if you want to do it in a behavioral test, if you want to do let's say an A/B test or you want to do more of kind of what they call attitudinal research; so you want to understand how customers feel about things. It's only worthwhile, in my humble opinion, if you have an understanding of what you're trying to learn, what kind of underlying assumptions are you trying to prove or disprove. Now, that’s the other key thing I sometimes fail to talk to people about, is like yes you might have this assumption and you test it through a prototype or a simple A/B test or customer conversation and it gets disproven very quickly and people are like, “No! Failure! I had this great assumption, it's great idea it's not working, or I've got it wrong!” and that's cool. I've learned that especially when you talk about assumption hypotheses, but those words in their own right, don’t necessarily resonate with everyone. Especially, if you're not a product person or you can get to it just bounce words.
So, I talk a lot more about risks and talking about uncovering these assumptions, even if they get disproven very quickly. It's a way of mitigating risk, because what if you didn't spend the time formulating the assumptions and then building a test or doing some research on top of it and you just went straight in and built that thing or you know created a feature launch that service? Think about the risk and think about the cost and the damage. To build on that kind of idea, one of the tactics that I use, is what they call a pre-mortem, originally created by a professor; I think he is called Gary Klein and it's really nice because it’s fun, but it's equally a very serious way in a safe space of saying, “Before we start writing any lines of code or building anything or designing anything, think of all the things that could go wrong in a year's time. What would need to happen for this project for this product or this feature to fail?” and it’s a really nice exercise, because you haven’t done anything yet. So, you can just freely brainstorm about all the things that could go wrong and why they could go wrong. Again, that's a nice way of almost flipping and saying what needs to go right and what do we so you need to get rights for this product or feature to succeed? Typically, when I do the exercise, I try to get really good kind of cross-section of people in the room, so not just engineers or designers but a marketers or salesperson; whoever has a stake in the particular product or feature and we look at that exercise of looking at all the things that could go wrong through a multitude of lenses as a result. So, we think about what could go wrong from a commercial or user adoption perspective, from an implementation perspective and legal perspective. Similarly, another tactic that I use is very similar to that pre-mortem exercise, is a scenario thinking. Where, again, very natural we always think about the happy path right of all the things that go right, then you think about one type of customer with one need, but let's play around with edge cases; do we need to cater for those edge cases? Do we need to cater for their specific customers? Sometimes you come to the conclusion that you don't, but even just widening your thinking about all the kind of different scenarios and related assumptions that worth considering is quite easily forgotten and very important to cover, if you want to really bring as much as you can on any of your assumptions going forward. Also, that gives you a baseline to go back to, because as you start developing, there will be new assumptions and it might be as you go, much more important or risky than some of the ones that you started off with, but the whole trick obviously, is to make those feedback loops between assumptions being tested or new assumptions come up as short as possible.
Hope Gurion: That's awesome! I never hear about teams doing pre mortems and I'm so excited that you do it like it’s so simple and getting those cross-functional perspectives in the room for it, is powerful. Like you said, it's it can be fun, it's not always fun to think about the downside risks but way more fun to think about it in that moment, then discover it later when it's really hard to recover.
Next, Audrey Cheng relies on using a lean canvas to capture assumptions early in product planning. Her product teams use that canvas of assumptions to check and recheck them with each other and stakeholders and continually revise it as they learn more about their customers’ needs and their potential solutions.
Audrey Cheng: I think with product discovery again, I think sometimes when you have to assume that you know certain things and sometimes you‘re using market data and thinking that your user group or your customers are also reflective of that market data and sometimes it might not be true. I think that's just one example, but I think the thing that we try to do as we go forward, is to say, “Assuming that’s true and this is true then these are the insights that we have and these are the things that we think we can realize if we go down this route” and we actually do that in a number of ways. So, we do, do our user testing, we do user interviews, we do our beta testing and that helps to bring a little bit more confidence and we hope to discover if some of our assumptions seem to be off during those periods, they may not always be highlighted. I think your metrics kind of fall into helping you to manage that as you go forward and hopefully you’re not building for 12 months before you release something or even six months before you release something, but I think those small incremental improvements that you make and monitoring and measuring help you to actually flesh out if your assumptions are actually wrong. The way that we actually would surface that in our discovery, is in a number of ways. So, when we complete a piece of research, we actually present out to our team to present the insights and findings from our research. As part of that we actually surface the assumptions there. We also use, so that everyone is aware of like what's our base, like what did we assume everyone is doing, what did we assume that everyone is feeling and whatever else. So, we have will surface it there.
The other thing that we do, is we actually do a lean canvas and those assumptions are also surfaced there. The lean canvas for us is really a high level, very super-high level, top view of what is it that we‘re building, what problem are we solving, how did we decide to approach it. In there, is also included as the assumptions. So, I can pick up that that lean canvas or I can bring up that lean canvas and show anybody, “Hey, this is the problem that we're solving, this is why we’re doing it, this is who it's for, it‘s how we decide to solve it.” We also made some assumptions here as to how we might go about solving it. So, we really try and bring that to the fore.
The other areas are we: work really closely with our customer success team, so if we have some assumptions we might actually talk to them about it too and sort of see if they have different feedback for us or have a different viewpoint, but we will surface those assumptions to them and quite often they may say, “Oh, I'm not too sure I've heard a couple cases here” and so that gives us an opportunity actually, to maybe get more confidence in areas where we thought we thought we knew something. So, sometimes you know things and sometimes you think you know things and I think the distinction there is to ensure that as we go forward and as we make those investments, that we feel comfortable enough with those assumptions and that the business teams that we're partnering with also feel comfortable with those assumptions.
Hope Gurion: Finally, Sean Murphy, former VP of Product at Target shares the tactics he uses to surface assumptions; like logical reasoning and customer interviews and his preferred techniques to test those assumptions.
Sean Murphy: How do I like to surface assumptions when we're doing discovery, understanding the opportunity space for a product and there's a few things that I go to and think through. So, first off, before jumping into anything, is just evaluating it logically. I know maybe this is too simple, but sometimes there's there needs to be a pause to say, “Is the assumption important?” and I think of an example about the tests of ‘would a consumer a guest pay for this product or service that you're trying to build’ hey, that’s an assumption but yet we've seen whole host of examples of where starting with free is the right way to go. So, even just working through the logic, “Is the assumption really necessary or make or break?” is an important first step, because sometimes there's not as many assumptions that you need to figure out, as you think needs to happen.
Second, is I like to interview people. I think a lot of product organizations like to run the testing, but I actually like to pause and usually five interviews and there's tremendous breakthroughs that can happen to validate or invalidate assumptions. I have become recently, a big fan of the jobs to be done method of interviewing; where you really get to the true motivations of your users, your consumers to understand if there's a need there.
Third, testing in the wild. If you've got a product and people are already using it and you have a chance to test or do a light prototype or a fake door test, of course, that's a great way to get a sense of it.
The fourth tactic I like to use for validating assumptions, is actually the Google AdWords test. This is whatever space you’re in, you can probably find some keywords and run an ad in Google and see will people even click on it. Can you draw people in and will they then, use that to validate whether this is a real problem or need or not?
Hope Gurion:
I’ve interviewed more than 30 product leaders for this podcast series and they each get to choose the questions from my backlog of what they are most interested in answering, and the reality is that while I think this is perhaps one of the most critical responsibilities of a product leader, this question wasn’t one of the more popular ones to answer. For the rest of the episode I’m going to share my recommendations on why, when and how you can bring critical assumptions to the light of day with your product teams and stakeholders to increase your alignment and bring focus and purpose to your product discovery.
So why don’t more teams surface assumptions or know how to do this?
First, there is an evolutionary spectrum of product teams. On one side of the spectrum are the delivery teams. They learn after delivering a solution whether they hit or missed customers’ expectations. Further evolved are the teams focus on problems and then develop an MVP to see if it solves the problem, but often these solutions are light in functionality and difficult to replace substitutes. Next on the evolutionary curve are the product teams that learn even earlier by observing how customers use existing solutions and often pair their discovery with a prototyped solution before they code. Even further evolved are the product teams that create hypotheses and describe the evidence that would prove their hypothesis true/false before they begin developing. There are many rituals and resources known and practiced by teams at these parts of the product team spectrum. But only the most advanced product teams make surfacing critical assumptions a must-do ritual in their product development. While Scrum is widely practiced, it has no ceremony for surfacing assumptions. It really will come down to how often in the past you or your product team members have worked on failed products burning in the ashes of risky assumptions and which of the many product discovery frameworks your teams use. Only 5/18 discovery frameworks prescribe surfacing assumptions. You can find out which do and which you’ll need to augment with a handy graphic I’ve posted in the show notes for this episode. Of course, I want to help you leverage the learning curves of others so you don’t have to experience the worst case scenarios of risky assumptions to encourage you to inject surfacing assumptions as a critical step in your product discovery process.
Why is it important to surface assumptions?
Every new business, every new product, every new feature, every UI change is a bet on future human behavior. Often when we are making such bets we have stars in our eyes of the optimism we feel believing that the desired behavior will take place. How many times have we heard the eye-rolling phrase: “Build it and they will come!” Our judgment and overconfidence bias clouds our thinking about the reality of how truly difficult it is to change human behavior. Our customers, our very human customers, just like us, have status quo bias, sunk cost bias and comfort with the devil we know. This aversion to change makes it challenging for the most effective product teams to drive adoption for new products and features. By shining a light on the assumptions that must be true for our products to succeed, we can identify the riskiest ones and examine our good intentions and best-laid plans against reality and increase our probability for success.
If you’ve made it this far, I’m going to assume (see what I did there?) that you believe in surfacing assumptions. Fantastic! So, what are some techniques you can use with your teams and stakeholders to do so? And when should you break out your “Surfacing Assumptions” bag of tricks? Let me share some scenarios and my recommended technique. Most details on each will be posted in the show notes for your reference.
Business Strategy and Model Decisions
Context: I can’t tell you how many product leaders I work with on product vision and strategy are working with companies who haven’t really defined their business strategy in a shareable, reference able artifact. Yet every day the leadership team is making decisions fraught with assumptions about their business, customers, competitors. Often it’s the product leader who shines a light on this by asking questions that try to externalize these assumptions for decisions being made with a leadership team trying to maximize their options to achieve their financial goals.
How-to: My go-to-technique to get these assumptions on the table is the Lean Canvas. Product leaders can either take the first stab at this and meet 1:1 with their leadership team to “correct” any mistakes. Or you can do this as a workshop (virtual or in-person), but give time to have each leader answer these questions individually first to reduce bias and surface good tension in the decisions required. You may need to create multiple versions if you’re operating in different markets. The results are especially helpful to the product leader and teams will be the value propositions, customer segments, relationships, channels and revenue streams. After you get a working version of these, you can ask your leadership team to dot vote with red and green dots which of the answers are known (green) vs unknown (red). Voila! You now are aligned with the assumptions that need to be validated before you risk investing too much.
Product & Business Decisions: Target Market, Go to market approach, pricing, customer support models
Context: Admittedly, this is kind of a catch-all, but everyday product teams and stakeholders are making decisions about choices that are fraught with assumptions. For small decisions that don’t impact many internal or external people (like customers, investors, partners) or are easily reversed you won’t need this level of ritual. But a quick test to know if it’s warranted to get those assumptions out of the darkness is to ask yourself two questions:
How far reaching will this decision be for our organization?
If we get it wrong, how costly will that be?
If it’s far reaching and if the wrong call would be costly, it’s likely a 1-way door decision and it’s worth a little assumption-surfacing exercise with your cross-functional partners.
How-to: My go-to-technique for product & business decisions like go-to-market, pricing, product support models and other cross-functional impacting decisions is a two-parter:
First, use the overconfidence- and blindspot-blaster, the Pre-Mortem (thanks Gary Klein!). This works best as a workshop (virtual or in-person) where you put the decision or recommendation on the center of the tombstone and have your product team and cross-functional partners murder that recommendation with all of the things that contributed to its untimely demise. This reverse-engineers the assumptions that must come true to ensure the recommendation/decision succeeds. Again, the participants should generate their “what went wrong” poison pills individually. Once they’re on display, you can group/dedupe them and behold your assumptions.
The next part leverages the assumption map (thanks David Bland/Precoil!) to identify the riskiest assumptions. Once you’ve identified the riskiest assumptions, you can ask yourselves “Is this truth knowable?” “How can we learn quickly if it’s true or false?” Capture ideas for how to test the validity of these assumptions. Voila! You now have your critical assumptions and a short-list of how to turn that assumption into evidence.
Quick plug—I have a helpful workshop to help facilitate cross-functional leadership teams working on new products to surface critical assumptions. I use a combination of techniques to help teams visualize the future in a product vision, murder that vision with a pre-mortem and mathematize the vision to surface all the assumptions that must be true to achieve their desired outcomes. If that’s of interest to you, please contact me for more info.
Product evolution decisions
Context: The best product teams are actively interviewing, prototyping and experimenting as part of their discovery practices, yet many still don’t surface and interrogate their assumptions throughout the process. This is easily corrected. I encourage product leaders to observe their teams discovery practices to see if they see assumptions coming to light early or late in their product development processes.
How-to: Whether your team is working on a new product or iterating on an existing product, odds are they are making many assumptions about the customer, their alternative solutions, their true level of need for a new solution, they’re willingness and motivators to adopt a new solution, the efficacy of the product’s design, the skill set of the product team to develop it, what the success criteria for any experiments. The list of potential assumptions can get long and overwhelming, but there are techniques that can help product teams break these assumptions into manageable chunks.
During the conception of a new feature or changing a user flow, you may try a Risky Assumption Test, which is a faster, less costly way to learn instead of an MVP.
If you use the design sprint method, you may capture your assumptions in an Assumption Board.
If you use user story maps to diagram your potential solutions, you may capture assumptions right on your story map.
During a pilot or early access program, you may try using either an Assumption Board or an Experiment Design worksheet to capture the assumptions that must come true during the pilot/early access to know it was successful (e.g. they used it, they preferred it to their alternative, they would be disappointed if they could know longer use it).
Another quick plug—I partner with Teresa Torres to coach product teams adopting Continuous Discovery Habits. Surfacing and testing assumptions is woven through every step of the curriculum and practice. If coaching your teams on how to adopt these practices and up-level their discovery habits is of interest to you, please contact me for more info. I’ve included links to some of Teresa’s resources on surfacing assumptions in her ProductTalk blog or refer to the resources in the show notes.
Helpful resources to refer to
How to Improve Your Experiment Design (And Build Trust in Your Product Experiments) | ProductTalk Blog (see Test Specific Assumptions, Not Ideas)
6 Guiding Principles for Effective Product Discovery | ProductTalk Blog (see Surface and Test Underlying Assumptions)
Testing Business Ideas Strategyzer series book by David Bland and Alex Osterwalder
Well that was a lot of how-to advice packed into this episode! I’ve included links and examples in my show notes you can find on my website, fearless-product.com. I hope that the stories in the previous episode, “How Have you been burned by risky assumptions?” and the advice and practices in this episode will help you and your product teams surface your riskiest assumptions and interrogate them fearlessly with effective product discovery.
———
If you’re a product leader feeling like your product team and stakeholders are in danger of being burned by risky assumptions, I’d love to be of help. Contact me on Linkedin or Twitter or schedule an initial consultation with me using the Contact Me page at fearless-product.com.