S2 Ep. 2: How Do You Balance Investing in Tech Debt vs Customer Value?
Subscribe: Apple Podcasts | Google Podcasts | Spotify | RSS
Hope Gurion: Knowing when and how to pay down Tech Debt is a blind spot for many product leaders. They’re often surprised by how much tech debt has accumulated and unsure of why or how or they’re not sure where to start, how to allocate bandwidth to retiring it or measure impact of doing so. In this episode of Fearless Product Leadership, I asked five experienced product leaders “How do you balance tech debt vs customer-facing investment?”
—
Welcome to the Fearless Product Leadership podcast. This is the show for new product leaders seeking to increase their confidence and competence. In every episode I ask experienced and thoughtful product leaders to share their strategies and tactics that have helped them tackle a tough responsibility of the product leader role. I love helping emerging product leaders shorten their learning curves to expedite their professional success with great products, teams and stakeholder relationships. I’m your host and CEO of Fearless Product, Hope Gurion.
—
Companies new and old face problems with tech debt slowing down their development. New product leaders in start-ups are often surprised how much tech debt has accumulated by companies who built first and then learned. Often engineering leaders face challenges retaining and attracting talent if tech debt goes unresolved so it’s in the best interest of the product and engineering leader to align on tech debt remediation strategy. For first time product leaders and inexperienced executive teams, tech debt can catch them by surprise. Even worse, tech debt often doesn’t rise to the radar of the executive team until it’s in a near crisis situation causing product and engineering team attrition and frustration with long-time to market or buggy experiences. Some companies halt all product development while they try to do a big-bang re-platforming effort only to exacerbate these frustrations they’re trying to resolve. Most product leaders will face the difficult choice of why and how to pay down tech debt to maximize the value of their products to their company and customers. That’s why in this episode we uncover strategies from five seasoned product leaders on HOW they tackle tech debt as part of their product development strategy.
In this episode you’ll learn:
WHEN tech debt is justified to accumulate.
WHO should be accountable for paying down the debt.
HOW to educate your leadership team on the impact of tech debt on the company, its goals and teams.
HOW to align with engineering on the allocation and priorities when paying down technical debt.
At the end we hear an enlightening perspective from Nate Walkingshaw on how they’ve managed to keep Tech Debt from becoming a burden through their engineering philosophy at Pluralsight.
Fearlessly tackling the question “How do you balance tech debt vs customer-facing investment?” are:
Sean Murphy, former VP of Product at Target, and former VP of Product and Engineering at CustomInk
Albert Lee, co-founder and former head of product at MyFitnessPal
First, Sean Murphy compares how he managed tech debt at an early stage startup vs a large Fortune 50 company and WHO, he believes, should determine when and how to pay down technical debt.
Sean Murphy: So, Hope, that's a great question. I mean, it’s a perennial challenge; you aspire to do great things for the customer, and you want to improve things for them, but yet, you've got all this technical debt. You need to have a good technical architecture. I've faced that challenge in a couple different scenarios: one at a startup, where we're struggling to find product market fit and then second at a big company like Target, where the product market fit is there and you're just trying to deliver value together. The answer, I think, is different in the two scenarios; when you're in the small company trying to find that product market bit, that is an accumulating bet, is just fine. I think you've got to find that fit, figure out how the consumer is going to respond to the product that you're building, but when you're in a larger environment, you figure that out what you have to maintain, is organizational technical agility. In those cases, I lean more towards getting the architecture and retiring that debt, is more important. So, when you're finding the entrepreneurial situation fact that is more okay, when you're in a bigger company, I think getting that technical architecture right for the long term, is the right call.
Hope Gurion: Just to follow up on that, like what is your practical approach when you are in fact investing in retiring tech debt? Is that a certain allocation? Is it dedicated teams? What practically do you feel is the best approach when you're trying to pay off technical debt?
Sean Murphy: Of course, there's no one-size-fits-all, but I do like to have the team that owns that product to both do the innovation and the retirement. You need to understand where things are headed to be able to retire, to be architect a system properly. I think the technical debt question becomes very easy to call when you know it's your road block. The business is hinging on making or breaking, or making the customer feature just right, but when the technical debt is slowing down the company, you're not able to achieve your ambitions. You need to slow down get that work done and you should have the team that's been working on all the great new capabilities for consumers, to actually retire that debt themselves.
Hope Gurion: Next, Laura Marino shares her techniques on HOW to expose the costs and impact of mounting tech debt to the executive team.
Laura Marino: The second situation was a little bit less about saying no to the CEO, but more about very much resetting the expectations of the CEO and the entire executive team. I had joined the company as VP of product, very recently. It was just kind of a couple of weeks into my job, when I started hearing a lot of complaints from the CEO and from the executive team about engineering not delivering, about my predecessor making promises they would never keep and about the speed of new functionality having gone down a lot. So, I started investigating what was going on and what I found out is something that unfortunately, is not uncommon for fast-growing b2b startups. In the early days, the engineering team in a start-up is really focused on delivering new functionality; they need to move fast, and they need to get out to the market, as much customer-facing functionality as they can. So, there's no time to pay attention for scalability; it is okay to make some kind of trade-offs and not build in the most robust way, as long as you can get things out quickly and of course there's no attention to building internal tools that would help an implementation team or a tech support team. Once, the company really starts growing and you start bringing in hundreds of customers, that doesn't work anymore, because the more customers you have, the more your engineering team is going to have to address escalation that happen. The more functionality you have put out there, the more there will be demand for maintaining and improving that functionality; because no feature is ever done after version one.
Of course, the more customers you’re supporting on your platform, the more those shortcuts that the team did on the technical side start surfacing and things start breaking. So, the team is going to be very distracted with fixing things. Finally, if you do have a complex product and you haven't taken the time to build tools for your implementation team, what will happen is that every new implementation, will pull resources from engineering to support it. So, this is very much what was happening at the time, but the problem was that both the engineering team and the product managers still thought that they were operating in this early stage, where 90% of the engineering bandwidth is actually going to new functionality. So, they kept promising that they would do all these things and of course they kept getting interested by all the other pieces. So, what I have to do was to step back with them and say, “Okay, let's really look realistically at capacity. Let's start reserving capacity for those things that we know are going to happen. Let's start proactively planning some bandwidth to address technical debt and now we can really create some more realistic estimates” and with that, I had to go back to the CEO and to the executive team and I had to give them the bad news that the reality was, no, they could not expect as many new features as quickly as they were accustomed to in the early days. So, I think that the message is that as the head of product, first of all you really need to understand where your team is, where is their capacity going? So, that you can help them really make sure they are giving you the right estimates; and then, you need to educate the executives and the rest of the company, because your VP of Sales won't even think about this idea that engineering has to pay for technical debt or if they have to build internal tools. They don't care about that. So, it is your role to be able to educate them why that's important and what it means for the roadmap.
Hope Gurion: Now, Troy Anderson shares his perspective on WHEN it’s ok to let tech debt accumulate and a technique to address it while benefitting customers.
Troy Anderson: So, on the topic of tech debt versus solving other customer facing initiatives. I found that the best approach is usually to understand what your companies trying to solve. I mean, that's the general rule anyway. If you're in a private equity company and you're looking sell and you're just putting lipstick on the pig, don't try and take a step. You need to have some runway; so really trying to understand where you are on that runway, is really important. If you have a long runway and you can get away with fixing your test, absolutely should have that. If you don't have a runway though, then you have to balance; pick step that or move with other improvements. It's a lot right here right, because then you’re going to have to keep some of the tech debt, because it takes time to fix. One of the things I really liked about the movie Chernobyl, was they realized some unpaid debt and that's exactly what tech debt is. It's kind of why you told yourself that this thing would work, and to unwind that, is really a tough challenge; but again, if you're looking to just sell then don't fix your tech debt, just put some more features on. If you have a long runway then absolutely start fixing your tech debt right away.
One of the things I found, over and over again is I've waited to fix my tech debt and I shouldn't have. I'll be like six months later and it's like if I would have fixed this before. So, I always have a bias towards fixing it as early as possible.
Hope Gurion: Let me ask you a follow-up question on that. When you would get an agreement that you do need to make an investment to retire some the tech debt that you've accumulated and getting features out to market, how do you think about allocating resources bandwidth to that? Is it a fixed period of time? Is it a across-all-team? How do you sort of think about the right way to start tearing down that debt?
Troy Anderson: One of the keys for paying down tech debt is understanding it, first off. Is it in control? If your tech does in control, that means people are maybe single points of failure or they have a very old technical one, let's say. If the people that run the monolith are kind of essential to its maintenance, then what you have to do, is really document how they're maintaining that model. Once you have kind of the tech debt under control, that is, you can swap one person for another; which means kind of eliminating single points feather. Once you have your kind of tech debt under control you understand what the kind of cost and labor is involved to maintain it; because obviously you have to maintain your tech debt while you’re fixing something else. Then you buy us all the resources towards either pulling it apart with micro services or writing API's or putting adapters and connectors in place to really go about and build things the right way. The key is understanding where your tech debt is; if it's completely out of control, then you might as well kind of start working on something new and don't try and pay down the tech debt just you know do greenfield. If you can get it under control and you can do part by part, then do a componentize replacement of that tech debt.
Hope Gurion: I think it's great that you added some of that context of what to use solving for. If you have the runway, maybe you don't need to pay it down and I found that some of the clients that I've been talking to are surprised; like new product leaders coming into it, like a startup that are surprised about how much tech debt they've already accumulated and all these sort of, “we thought this was the right thing to build” and now of course, it's not and they're trying to navigate the way things were either architected or the features they have to now work around. They don't have the runway, I think, to retire it and so for them, I'm like, “You don't have product market fit and so, now is not the time to pay down this tech debt. You've got to really try working on anything that you're doing, should be about gaining new customers and supporting work”
Troy Anderson: Yeah, I want to add one more anecdote on tech debt. The other about tech debt is you have to look at it from the perspective of the US Air Force; so, the US Air Force and the nuclear infrastructure, they just got off floppy disks like what two weeks ago? So, it's ridiculous how long tech debt can go on and be a very core part of your system. So, don't discard it just because it's old. Obviously, people are still using DOS and still using COBOL, but with regard to runway and resources to strengthen or control.
Hope Gurion: Next up we hear how Albert Lee aligned with his engineering leadership on how to keep tech debt at bay and when it required more significant investment.
Albert Lee: I kind of feel like this is one of those really eternal questions in software development. A technical debt: I have features that I'm trying to get out the door, I have business objectives I'm trying to beat. How do I ultimately balance these things and I think to be completely frank, I certainly think that there's no silver bullet in this case; it's always been very hard, every single place I've ever been to. I think there's some element, where I've been fortunate to sort of have seen a product like literally from the very beginning, all the way to like massive scale hundreds and billions of users and so I kind of feel like some of the things that I'm going to say now are probably not as applicable in early stages. Frankly, I think that's a little bit of like, you have to have a good team and sort of rely on their judgment there, but as you get much bigger, you just get these bigger and hairier tech debt type problems and so what I what I sort of felt like really worth well for our team or time, and I think this is probably maybe the most obvious thing, is to essentially have some allocation of time consistently addressing technical debt. So, it's kind of like I'm going to have somebody coming over every week and looking at all the different stuff around my house. It's the same kind of ideas, but if I'm doing that on a consistent basis, I can hopefully not let the weeds kind of overrun the garden.
I think we probably used like a 20% type idea around that, but there's things that I think you need to do to make sure that you know this sort of arbitrary level is actually working for you. One thing that we would do that I thought was pretty helpful, it's like after every quarter, we would sort of do this like more broad retrospective with engineering leadership, to sort of understand like how did we perform as a team this quarter and that would usually bubble up circumstances where you can really see how technical debt was actually impacting your ability to execute. So, that might surface something is more important or less important through that process and maybe that wouldn't change anything, that would just be the prioritization of what we work on in this 20% time, but maybe, that would be like, “Hey, I think we need to sort of bulk up on this for the next three to six months.”
Another thing that I thought was really helpful for us to do, was we would spend a decent amount of time with engineering leadership. Kind of looking at what would I sort of turn like the product ladder; I kind of got away from the notion of a road map, because I always felt like the road map had this clearer sort of time implication that always became sort of a little bit problematic. It always had to have this like quarter-to-quarter feel for anybody to feel like it was a real road map. I don't know that that's helpful in some circumstances, so instead, we would kind of look at what is this sequential idea of the things that we need to have, the capabilities we need to have as product, being able to deliver the experiences that we want and that's this ladder that we're building on. To the extent that we can show that even if the timeline is indeterminate; I think engineering leadership gets this idea, “Well hey, I can see ahead of time that in order for us to be able to do this really well, this one thing that we do or this one problem that we have had, for structure, needs to be addressed” and so that helps them think ahead about that. It helps us think about resourcing in a bigger way, which is not just how we allocate the fixed pile of resources that we have, but do we need to actually hire to address some of the stuff, to be able to meet this longer-term goal?
So, I think mixing sort of like, “Hey, we're going to have this one simple rule, that's going to allow us to hopefully, fix most of our problems and just make sure that we're making progress. We're going to have these checkpoints to think about this stuff more strategically, whether its quarterly or even you know more frequently good quarterly and we're going to resource and were going to strategize against them.” The kind of combination of that felt like you know we were able to sort of keep things going and never felt completely bogged down and you know I think obviously the biggest problem with this overall is it takes up a lot actually for tech debt, just to be this thing that feels like it's sinking business. It's like too late by the time appeals like that, so I think having a little bit of a proactive approach in my opinion makes a lot of sense.
Hope Gurion: Finally, we hear from Nate Walkingshaw, on how his experiences leading product and engineering teams at multiple companies influenced his approach to use the principles of Learning, Diversity, and Autonomy, at PluralSight to keep the product development teams nimble in delivering value.
Nate Walkingshaw: I want to talk to you a little bit about technical debt versus you know like value creation for customers. The way that I've tried to solve this problem again, this is my current best thinking; it could be completely different a quarter from now. My current best thinking is philosophically, you actually have to align the whole company and the team on your technology strategy, on the technical architecture that you're going to work in. There are a lot of components, but the reason why I care so much about this, I have a lot of passion around this this heated debate. On the on the surface, we hear things like, “Oh, it's 20% time for tech debt.” I don't even know how to describe how that lands over here. For me, it’s ‘good luck’, there is no way a 20%-time approach is going to solve your technology strategy.
The tech strategy is, you have to have a very strong opinion on what is the ongoing tech strategy for the company and then that tech strategy, that architecture, is definitely going to dictate the way that you work. A great use cases here, is when I got to Pluralsight, we were a .net and angular like shop. We were on prem. Why is that such a big deal? Well guess what? For hiring velocity, I can only hire .net or angular engineers. One, it's a mile that is code based; so, I can't work in continuous delivery and continuous discovery. We're going to branch, and then ship all that branch code into these commits and then it's going to tip over and then I have to do a rollback, then I have to do a root cause analysis; that doesn’t work. That's not the tech strategy. The tech strategy is, “Hey, we want to work in continuous delivery and continuous discovery. We want to talk to customers, we want diversity of thought on teams.” So, I just don't want .net and angular engineers, I want an African engineer that works in the learning product, to actually be a part of my team, to help me understand how people learn in the world or how people develop those products in the world. I want an early college graduate degree program, I want really senior level engineers, I want a different language framework tulip and so all of those things are actually kind of the secrets of building really good products; because the underlying architecture that sits behind there, the composition of the teams that you get to build, the way that you get to talk to customers and deliver that value, actually is the beauty of the decision that we're going to make.
So, like you have to be intentional; we decided, “Hey, we’re not going to be monolithic, we're going to move to micro services. We're not going to be on Prem we're going to move to the cloud.” Well, the reason wasn't because we want to be in a modern architecture, it was actually the people benefits, the product benefits, that that actually was the outcome that we wanted to achieve and that was really thoughtful. By the way, it was inspiring for people to want to do the work, because if we could hire different people on teams, if we could build different products for different diverse communities, for our social impact initiative, then that was really meaningful to the purpose of why people wanted to come and work here.
I guess what I'm trying to say, is if you can stitch as much of that technical debt, value creation discussion into the mission, the vision and the purpose of your company, it becomes like your superpower. It doesn't become this giant tech debt boat anchor that people were out on their neck and say, “Well I hope to get to 20% of the monolithic traffic today” I mean, they actually take it really serious and it's been awesome; I mean this has been the best rewrite I have ever done in my career and it mostly was because of all the failed attempts that I had made, in the in the previous years, to finally get it right once, felt pretty sweet.
Hope Gurion: I love the way that you answered that, because I do think I often think about it in terms of, “Well we're either doing it for the benefit of the development team or we're doing it for the benefit of customers.” What you basically did, it's like by having diversity of being able to attract and retain people with different skills and different code bases, programming languages, different points of view around the world and different diversity of perspectives and you make that like, “This will become our superpower” and that's what we want to solve for basically create this ongoing investment in making sure continue to fill your team with diversity of thought and perspectives and capabilities.
Nate Walkingshaw: Yeah, totally I mean dude, it's a cultural thing. If you're committed to learning, are we committed to it or not? Well if we are, you'd need to start, like you got to eat your own dog food here; this is it's not something that you can dance around. One of the very first thing we did, we built a TechRadar called adopt, avoid, assess, and so these were the languages, frameworks and tools we were going to adopt. These are the things we were assessing and then these things we’re going to avoid. So, what that did automatically, was just align all of the teams; in addition to that it also allowed the teams to provide input into what they wanted to adopt and assess. The architecture team, the platform team and the software development teams, that all got together, they created this really meaningful artifact, that actually kept new hires when they came into the company. Our engineering principles, engineering a Pluralsight, how we actually develop our products, it was just a straightforward document in the hiring process.
It's like, “Hey, so this is our engineering strategy, this is our technical strategy, these are the things we want to adopt, avoid, assess and this is the super-power it gives us” diversity of thought. Hiring velocity, gay autonomy, autonomy of teams is huge, so that means when you came in and you’re one of these developers. you owned the entire stack. So, when it's down at 2 a.m., there's no architecture platform team, dude that's you, so you’re going to have to own the codebase. So, it created this really awesome rigor, inside the company that is now shared and owned by those teams. So, it's not some tech debt thing, it's look we want clean code like we want to deliver a reliable experience.
Hope Gurion: I love that it's really instead of it becoming this sort of tax burden, that everybody's sort of dealing with, it became your strategy your philosophy and it made it constantly be investing in.
Nate Walkingshaw: I focus on the A not the F and people could criticize that, if you just heard that by itself, it's like well you just want to just want to talk about the positive, you're unwilling to talk about the negative. It's like, “Oh no! Look, we've got A’s and F’s every day, but if I actually just focus human behavior on F's and I keep telling you why you suck, and tell you to say, “Hey can you deploy the behaviors of the A on the F, the better the F’s get.” It's not that we're not committed to looking at them, we want to look at them and evaluate them, that's useful, but in order for you to actually lift those F’s up, we need to focus on what you do to produce A's; because you're doing something over there that's incredible, so like let's do that to this thing.
—
Hope Gurion: The executives and product leaders I work with usually fall into two categories. One they are blind to the pains and consequences felt by the product development teams from accumulated tech debt or they’re dealing with an impending “replatforming” initiative that everyone is fearful of because they aren’t experienced to have considered other options. While I admire what Nate has been able to achieve with the very intentional approach from the outset and through scaling at Pluralsight, this is the exception that proves the rule. If tech debt remediation is unavoidable, I encourage product and engineering leaders to take a similar approach as they would in product development which is to:
Define the unmet needs: velocity, team health and happiness, speed to customer value, code quality
Conceive of multiple ways to solve for each of those needs, at least 3 alternatives to “re-platform/re-architecture”
Experiment with these solutions to learn which achieves the most value soonest
If you’re a new product leader feeling out of your depth with tech debt, I’d love to be of help. Contact me on LinkedIn or Twitter or schedule an initial consultation with me using the Contact Me page at fearless-product.com.