{"url": "https://www.infoq.com/presentations/stripe-culture/", "page_url": "https://www.infoq.com/presentations/stripe-culture/", "title": "Building a Culture Where Software Projects Get Done", "show_or_channel": "InfoQ / QCon San Francisco 2013", "date": "2013-11-13", "duration_s": null, "type": "talk", "speakers": ["Greg Brockman"], "confidence": "medium", "notes": "QCon SF 2013 talk as Stripe CTO (InfoQ lists recording Dec 5, 2013 publication; QCon SF was Nov 2013, date approximate). Video hosted on InfoQ; duration not verified.", "people": ["greg-brockman"], "id": "2013-11-13_infoq-qcon-san-francisc-building-a-culture-where-software-projec", "audio_duration_s": 3146.18775, "worker": "jc-w17", "transcribe_s": 162.0, "model": "large-v3", "segments": [{"start": 0.69, "end": 29.98, "text": "This presentation was recorded at QCon San Francisco 2013. This audio file and others like it are available for free download at InfoQ.com. All right, so today I'm going to be talking about, you know, one of the single things in software engineering that just hasn't gotten any better in the past 30 years. That's kind of crazy when you think about it, right? Like every other aspect of building software has been completely revolutionized. But when it comes to actually getting projects done, we're basically exactly where we were back in the 1980s."}, {"start": 29.98, "end": 59.34, "text": "And most people in this room probably know about all the classic allegories in this space. You have the mythical man month, you have the second system effect, and things along those lines. But even though those stories have been around for a very long time, those same issues continue to plague us. And they're growing even more important as software continues to eat the world. Just take a look at the healthcare.gov fiasco. The difficulties of building software, that's now national headlines. And that's really just where it starts."}, {"start": 59.34, "end": 81.62, "text": "I think that the place to start is for us to admit that we have a problem. That we just do not know how to build software. And I think that we as a community need to get together and acknowledge that. Whether it's bugs, whether it's schedule delay, whether it's feature creep, all of these things start holding back the software development process. And these are all things that we just haven't gotten any better at."}, {"start": 81.81, "end": 108.21, "text": "Now, think about how reliable traditional engineering is compared to software engineering. And this is an analogy that a lot of people make. And you think about a bridge. As you're walking across that bridge, you don't worry that it's just about to fall down. You basically trust that the builders know what they're doing, that they were able to build this in a reliable way. When you think about software, you just assume it's going to be full of bugs. It's not a question of if there are bugs, it's just a question of how many and how long until they start ruining your life."}, {"start": 108.3, "end": 125.94, "text": "And there's certainly not a debate about just why that's true. Why are we so much better at building traditional stuff than we are at building software? And you could argue that, oh, well, software just hasn't been around for very long. Wait 100 years. We'll be fine. This happened in mathematics. This happened in engineering. We'll figure it out."}, {"start": 125.94, "end": 155.42, "text": "But I do think that there's something fundamentally different about software. Software is just far more complex than anything else that humans do. When you think about a single module and the interaction surface there, that that's far broader than any physical component. And when you think about the interaction between modules, that you just end up with more modules that touch each other. And so as a result, you end up with these far more complex systems that are just orders of magnitude more complex than anything else that we end up building."}, {"start": 155.42, "end": 177.26, "text": "One common problem is engineering timelines, right? That we just do not know how to actually hold ourselves to them. My usual rule of thumb is you take the timeline that you expect, you keep tripling it until you hit some insane number that you're like, there's no way it could possibly take that long. And the real timeline is probably somewhere between the disappointing point and that insane one."}, {"start": 177.26, "end": 204.88, "text": "And it's kind of crazy that this actually works. I recommend trying it in your next project, and you'll see that it's actually more reliable than not. And I think a lot of what happens here is that when you're planning something, when you're putting together a timeline, you're always just very optimistic, that you always kind of don't take into account all of the random interruptions that are going to come up. And it's kind of reasonable, because those are really hard to think about and to reason about, but at the same time, they're going to happen."}, {"start": 206.61, "end": 232.21, "text": "When was the last time you saw one of these? If you can't see it, that's a negative diff stat. And it probably wouldn't look very familiar anyway, because in practice, we just don't end up with these. Most engineering cultures, they measure productivity as new code produced to do new things. Changing old code to do old things in a better way or with less code, that's not really something that happens."}, {"start": 232.21, "end": 260.72, "text": "And it's really kind of surprising because one of the things that we do know about building software is that it's all about constraining complexity. And so if you can do something that will make your code base be less complex, you should absolutely take it. That's a very rare opportunity and you should try to exploit it for whatever it's worth. But that being said, it's also easy to approach that drive for simplicity from the wrong direction. And you can end up with basically the second system effect as a result. So as an example,"}, {"start": 260.72, "end": 284.24, "text": "Back when I was at Harvard, I was running the Harvard MIT Math Tournament's technology division. And I ended up inheriting a lot of code that had grown up over the years. There was a Java app that was used for entering results. There was a Perl script that did all the grading. There was a Python script that parsed the results of that, and it sent out all the results emails. And then there was a website that was all based on server-side includes. And so when I showed up..."}, {"start": 284.24, "end": 305.63, "text": "I looked at this and I was like, whoa, look at all this complexity. I know what I'll do. I will rewrite it all in Ruby on Rails. So it seemed like a good idea at the time. And I started writing. And there were several things that went wrong. And one was that whenever I came across some new functionality that I might want to add, that I would just go ahead and add it."}, {"start": 305.63, "end": 327.02, "text": "Being able to support multiple tournaments in one year, that seems like something someone might want. Okay, I'll do that. Adding in a CMS to actually manage all those static pages, yeah, that seems pretty useful. I guess I should do that one too. And over time, I noticed that the application was becoming very complex. And I did something that might be a little bit surprising. I rejoiced. I thought, this is amazing."}, {"start": 327.02, "end": 357.49, "text": "Think about all the value here. Look at that line count. That's continuing to grow. And the fact that there's all this complexity, that you're able to do all these things, that is what great software is, right? Now, there were some downsides here. As this thing grew, it became really hard to trace out the origin of any given behavior. There weren't really well-defined abstractions. And so I noticed that over time, I'd sort of start to forget how some of the innards worked and just layer things on top of it. But I was still able to get things done for the most part."}, {"start": 357.97, "end": 374.45, "text": "And the real problem was when I presented this to my co-maintainers and said, look at this rewrite, isn't this great? No one could figure out what was going on. And it was actually interesting because I had successfully rewritten this system, which is something that's pretty hard to do."}, {"start": 374.45, "end": 400.43, "text": "And that with the old system, there were just all these different tools. But one big difference was that with the old system, if you wanted to make a change, you just had to understand one component. You didn't have to understand how the parsing of the results worked if you wanted to be able to actually just deal with those results. You could deal with just the innards of that Perl script. With this massive Ruby on Rails rewrite, suddenly you had to understand the entire system before you could change any one component. And so even though..."}, {"start": 400.43, "end": 421.02, "text": "We had all this new functionality, even though we had successfully replaced everything, that we ended up just having to throw it out because no one could understand it in order to make changes to it. You should look at code as shackles. I think the way that I've seen this phrase before and like is that code is a liability, not an asset. Every additional feature you have"}, {"start": 421.02, "end": 448.8, "text": "That's something you're going to have to maintain. That's something that if someone re-implements, if someone ports it over, they're going to have to re-implement that. And this is something that's just going to interact with any new features that you add and you're going to have to think about. And even worse than the features that you actually mean to add are sort of the emergent behaviors that just kind of grow out of the way you happen to implement the system. And at some point, someone's going to start relying on those. And then at that point, it becomes incredibly hard to reason about..."}, {"start": 448.8, "end": 476.88, "text": "What your system actually is and to be able to figure out what you can break and what you can't. And given how bad we are at writing software, we really need to do everything we can to try to constrain complexity. But at the same time, you can't solve complexity by throwing more complexity at it. That really the only way to successfully do it is through this, through negative diff stats. And that really we as a community need to get better at figuring out how do you take an existing project"}, {"start": 476.88, "end": 504.13, "text": "Incrementally refactor it, incrementally make it better, rather than throw it out and start from scratch. So you can look at that story as just the follies of a novice programmer. Oh, he hadn't heard about the second system effect before, that other people, once they have experience, that would never happen to them. But the crazy thing is that there are so many stories just like that that come from industry, even from some of the best programmers out there."}, {"start": 504.13, "end": 533.15, "text": "And you've all probably heard about Netscape and how their massive rewrite basically killed the company. But the thing that I just find fascinating is that there are so many stories like this that continue to happen today. So to illustrate, one of my friend's companies is written in a scripting language. And as they grew, they needed more performance. Their web requests were too high latency. And they did some experiments and decided that they couldn't get that kind of performance out of their scripting language."}, {"start": 533.15, "end": 561.14, "text": "And so as a result, they turned to Scala. And they said, we're just going to rewrite the entire site in Scala. And so they sat down and they planned out a course of action that they would spend the next three months with a few people just porting over the core abstractions. Then they would do a feature freeze and spend two weeks with everyone doing a quick rewrite of all of the things in the scripting language over into Scala. At that point, they could just toss out the old one, flip the switch, and Scala would be serving requests."}, {"start": 561.14, "end": 588.91, "text": "The first sign of danger here was that the initial porting of abstractions took about twice as long as they'd expected. Rather than being three months, it turned out to be six months. And the delays were all these little things that you just wouldn't have thought about. The tool chain, it turned out, just was a little bit harder to integrate than they'd expected. Or they had to build out a stack to serve shadow requests from their new Scala stuff to make sure it was correct. But they got that done, and they got to the point where they were ready to do the feature freeze and start porting things."}, {"start": 589.1, "end": 618.11, "text": "So the two-week watermark was where they were supposed to be done. And at two weeks, just like that they barely made any progress, that it was a lot slower than they expected. People were just writing Scala for the first time. There was a lot of spin-up costs there. So they were like, okay, we can keep going. Time went on, and three months later, they were still going. It turned out that there was just a lot more complexity in their application logic than they'd expected, and there were just a lot of little bugs that they ran into along the way. At some point, they realized, okay, we just can't afford to leave our existing site."}, {"start": 618.11, "end": 644.19, "text": "We have to be building features that users want. We need to be responding to these issues. And so they put a few people back to maintaining their existing stuff. And suddenly they had a diverging code base. And that just slowed everything down further. And at some point it just became clear that they were not going to be able to complete the rewrite. Now, at that point, they basically threw out their entire new Scala thing. And they were like, what do we do?"}, {"start": 644.19, "end": 664.54, "text": "And they really doubled down on trying to figure out how can we make this scripting language work. And it turned out that with just some clever parallelization that they were able to get the latency wins that they needed. So there are a couple of interesting things in this story. One is that it's not like this company had bad engineers."}, {"start": 664.54, "end": 693.38, "text": "This company had, in fact, some of the best engineers that I know, but they still fell into this trap, which is not a new one. This is something that, if you look at Joel on software, he's been talking about this since the year 2000. This is stuff that we know about. Rewrites just will fail. And the second thing is that once the solution space was constrained, once they realized that, okay, rewrites are not okay, they were able to find solutions. And I think what happens there is that"}, {"start": 693.38, "end": 703.95, "text": "People are pretty good at finding a solution, but once you have one, then you sort of give up. You sort of don't look further. And once you've successfully solved the problem, that's where you sit."}, {"start": 703.95, "end": 728.11, "text": "And so I think that one thing that's really important is for us to realize that rewrites, they just never are the answer. That if you try them, you will fail. That you will possibly kill the company with you. That that is just something that you and your entire engineering team need to understand and need to believe. And once you believe that, you can solve all the problems that you want to solve through a rewrite. But if you don't have it, then that's the most obvious thing to do."}, {"start": 730.7, "end": 759.94, "text": "Perhaps the most important point here is this realization that you are not, in fact, special. You're not immune to any of the failure modes here. The things I'm going to talk about today, these are probably things that you've run into or will run into in the future. If you haven't run into them yet and you don't think that you will, that probably just means you haven't been doing software for long enough. That there have been lots of very smart people who have been thinking about all these problems and trying to avoid them and continue to run into them. And so I think that really the best approach..."}, {"start": 759.94, "end": 791.23, "text": "is to approach every problem cognizant of the fact that we are really bad at building software and constantly looking for all the warning signs that we're doing something wrong, that our product is doomed to failure, that this is a pattern that has been seen before. You'll still make mistakes, but this way you'll make fewer of them. So the most common reason that people fail to get things done is because they spend their time working on the wrong things. Shaping your culture so that people actually work on the most important things, that's a really hard problem."}, {"start": 791.23, "end": 810.59, "text": "But it's also really the only way to be successful. And so it's really worth spending a bunch of time trying to think about how do we make sure that people are actually working on the things that are most important. So I think one thing to think about is how do you decide when it's time to roll your own solution versus just using something off the shelf?"}, {"start": 810.59, "end": 837.98, "text": "As an engineer, I think it's always tempting to roll your own solution. And so NIH, or not invented here, is the name of the syndrome that goes with this. And I think as a manager, you have sort of the opposite inclination, that you just want to buy everything and then you can spend your engineer time on the things that actually matter for your business. And so given those two tensions, how do you actually decide when it's time to build, when it's time to buy? So the danger of building..."}, {"start": 837.98, "end": 850.5, "text": "It isn't actually that any of those problems are hard. And in fact, it's so easy to just get an MVP of some applicant tracking system out there. That's the easy part. The hard part, it's not that."}, {"start": 850.5, "end": 873.81, "text": "But it's that laundry list of 100 features that you're going to have to build in order to make this thing useful. That you're going to have to add user authentication. You're going to have to add email reminders and notifications and things like that. And that all of those are just incremental and small. And that your engineering time is probably better spent working on other things. Like, I mean, hopefully, right, that you have a business that is..."}, {"start": 873.81, "end": 901.68, "text": "Higher leverage than working on all those individual features. And so really when you're buying a solution, you're not paying for that core functionality, but you're paying for this whole ecosystem of just like 100 little features that actually make the product good to use. And so most cultures just lose sight of this, and they NIH all the easy problems. They think, okay, well, so we have these problems, these existing solutions, that they're missing some one little feature that we need or we think we need."}, {"start": 901.68, "end": 915.23, "text": "And wouldn't it be easy just to go and build it ourselves? And in contrast, when you see a hard problem, your first instinct might be, let's get someone else to solve this. This is hard. We want to go do other things."}, {"start": 915.23, "end": 944.11, "text": "And one good illustration of why you should watch out for outsourcing your hardest problem is a project that we worked on when we were first implementing sharding. And so for a long time, we just incrementally scaled our databases, doing the thing that everyone does where you've got a bunch of tables on a single machine, and you split those out into multiple machines, and look at that. You now can scale to 2x on the same physical hardware. And this is, of course, something that just is a stalling tech."}, {"start": 944.11, "end": 970.21, "text": "At some point, you know that you actually have to shard your databases. And so when we were reaching that point, we were like, okay, let's take a look around. Sharding seems hard. Let's see whether someone else can solve this problem for us. And so we use a lot of MongoDB, and so we looked at Mongo's sharding solution, and we rolled it out to our Logger archive cluster and started running it for a few months just to see what would happen and what that was like to run. And so..."}, {"start": 970.21, "end": 990.7, "text": "Over that time, we ran into a bunch of operational issues. There were just a lot of fires that came up. And whenever something went wrong, there were basically two things that we could do to try to solve it. One was go and read the source. Two was go and talk to MongoDB support, possibly do both. And so the thing about support, their support is actually quite good. We got very low latency answers."}, {"start": 990.7, "end": 1018.06, "text": "If someone takes 10 minutes to get back to you and your database is on fire and your site's down, that's actually a big deal. That's something that you just can't afford. And so we realized that if we were to roll this to production, that we just wouldn't be able to fix issues fast enough. The other thing was that even though we'd outsourced the code writing, we still had to understand exactly what this thing was doing. We had to understand all the code, all the semantics, all of the design, that we hadn't actually gotten that much of a win. And also that..."}, {"start": 1018.06, "end": 1047.54, "text": "We basically put ourselves in a world where we just couldn't patch the software. If it turned out it was doing something that was a little bit silly for our use case, that we just did not want to end up with a fork of our database. And so that we were effectively just constrained and unable to actually change our own core infrastructure. And so what we realized is that sharding is just something that we had to build in-house. It's sufficiently core that we couldn't just trust someone else to manage it for us, to build it for us. And we could also tune"}, {"start": 1047.54, "end": 1076.91, "text": "A lot of the sharding problems just for our own application, that we know these invariants, we know these constraints, that this is a way that we can make the problem solution for us be easier than the general case. So that being said, there are many hard problems that you should not try to solve yourself. And you should really only pick problems that are core to your business and where you have some reason to believe that the solution for you is easier than the general case. And so, for example, scaling your log pipeline, you have the exact same problem that everyone else does. Don't build that yourself."}, {"start": 1076.91, "end": 1106.56, "text": "Go and find someone else who has already written that solution. And as an aside, there's a really good paper on this subject, and it's called End-to-End Arguments in System Design, which is basically about the things that you should put in the application layer versus the things that you should allow other layers to implement. Perhaps one of the hardest things about building a rapidly growing company is figuring out how do you balance all the new things that you're building versus maintaining the old things."}, {"start": 1106.56, "end": 1130.02, "text": "Fixing the existing things, that is always the most urgent thing on your plate. If a user can't log in, if the site's down, if your database has been corrupted, you just have no choice. You have to drop everything you're doing and go and fix that. Being good at maintenance, that's how you avoid losing. But the really important stuff, the stuff that you're building, the stuff that's going to come out, that's how you win."}, {"start": 1130.02, "end": 1154.16, "text": "You really need to figure out how do you make it so that you can get all this new stuff done while not being bogged down by maintaining all the existing stuff. And nothing stalls a project more than the engineer involved being pulled into some meeting with a customer or a bunch of firefighting or helping diagnose some weird issues. And one structure that we've come up with that has worked really well for our organization is a build-run rotation."}, {"start": 1154.16, "end": 1184.43, "text": "Each week, it changes off. There's one person who's designated as the runner, and their job is basically to run the systems. And so they serve as a buffer between the operating systems and everyone else, that if something goes wrong, if someone else needs help with it, if there's something going on, that the runner will go and deal with that. And in their spare cycles, that they'll probably work on just quick wins because they just know that the week is going to be them constantly interrupted. And there are a couple of benefits here."}, {"start": 1184.43, "end": 1213.1, "text": "The main one is that this leaves the builders free to focus on whatever's most important to get done. And so that helps sort of solve that priority inversion that you end up with, with urgency versus importance. And a second thing is that everyone on the team, that they end up participating in the run rotation. And so that means they all have very good sense of what are the operational issues that we're dealing with? What are the problems? What are the things that are slowing us down? And that really helps focus figuring out what are the most important problems for us to be working on."}, {"start": 1214.22, "end": 1236.59, "text": "And along these lines, you should bake into your culture a desire to squash manual tasks. Otherwise, those tasks are just going to keep growing in number until that's all that you do. It's so easy for those things to accumulate, and it's always the easiest thing to do to just go and do the manual thing. Writing code is hard. Writing general solutions is hard. And so if you don't have a good framework in place to make that easy, people just won't do it."}, {"start": 1236.59, "end": 1265.15, "text": "So one investment that we've made is in a tool chain for automation. So we have a thing called Golem, which we use for things like rebuilding a database cluster or other things that at first glance seem like they'd actually be quite complex to automate. But if you get a good enough framework in place, you'll just notice that people will start automating things that you didn't even realize they were doing. And I think that that's really the sign of success there."}, {"start": 1265.15, "end": 1292.18, "text": "Another note sort of along these operational lines is that there's sort of this weird phenomenon that I've noticed, which is imagine you have some service outage due to a memory leak, and you track down the leak, and you see that it's due to some really rare, crazy corner case, and you have no idea how anyone was able to actually trigger it. So you have a choice. Do you immediately fix the issue, or do you just put it into your queue and say, this thing's rare, it's probably not going to come up again, I'll deal with it later?"}, {"start": 1292.53, "end": 1319.78, "text": "It's so tempting to go for the latter. It's so tempting to say, this is not something that I see how it'll be triggered, that I'll deal with it, doesn't seem urgent. But empirically, if you've hit an issue once, you'll probably just keep hitting it again on a much shorter time frame than you'd expect. And I think what's going on there is that if you've in fact managed to trigger it, that actually means, independent of what you think, it's actually fairly likely. That means you're now in a universe where this thing is..."}, {"start": 1319.78, "end": 1346.42, "text": "And the fact that it's plausible means that it's going to keep happening. And maybe there are some additional constraints that you didn't notice. Maybe there are some new customers sending you weird data. Maybe you've just crossed some volume threshold without you really noticing. But whatever it is, it's really important to fix your production bugs as soon as they're discovered. Because otherwise they just will keep biting you again. And that will mean that you end up spending a bunch of time working on the wrong things."}, {"start": 1347.89, "end": 1372.66, "text": "So kind of a second major theme here, I think, is that you should be investing in technology to support your rate of change. The single best thing you can do to get stuff done in your code base is provide good ways of ensuring that changes are correct. Ultimately, the hard thing about software is just actually building it, and then it's even harder to make sure that the thing you built actually does what you want it to do."}, {"start": 1372.66, "end": 1402.67, "text": "No code should be considered complete or shippable until you have a good way of ensuring that people will be able to make changes to it in the future. The most common way to accomplish this is through tests. So you should basically have tests. Those should be the ground truth for does this system work. And one common misconception about tests is that they're there for you. They're there for you to convince yourself that, hey, in fact, this code works. So that is not how you should be using tests. Some people do that, but..."}, {"start": 1402.67, "end": 1430.83, "text": "That's not the main value of them. The main value of tests is that these are a statement to whoever comes next about the functionality you care about. Anything that's not tested, you should basically assume that a future refactor is going to come along and break it. Because in reality, it probably will. It probably will make people sad. They'll roll back the code. Maybe they'll realize this is, in fact, an untested mess of code that they don't really understand. They don't know what interfaces can be changed, which ones can't. And as a result, you just will not be able to ever modify that code in the future."}, {"start": 1433.68, "end": 1454.82, "text": "And choosing what tests to actually write, that's a bit of an art. You should make sure that your tests are very explicit about what behavior they're actually testing because, again, it comes back to your tests are the contract that humans will also look at these and figure out, like, what does this code actually do? What are the behaviors that I'm able to change? What things do I need to think about if I, in fact, modify this? And importantly..."}, {"start": 1454.82, "end": 1479.94, "text": "Your test should really be statements about functionality rather than about implementation details. I sort of see this go wrong with some TDD methodologies where you start out, you write some empty function, and then you write an assertion that this function returns an array, and then you return an empty array, and then you write an assertion that, well, that array actually is of length two, and then you keep going from there."}, {"start": 1479.94, "end": 1508.7, "text": "Really, those tests that you end up with, those end up being implementation details. Those end up being things that maybe you'll change how you're representing this. Maybe this is a function which splits a name into a first name and a last name or something like that. Really, what you should be testing is that high-level behavior of what are you actually trying to do and not how are you doing it. One less obvious investment that you should be making is you should try as hard as possible, as hard as humanly possible, to stay on a unified stack."}, {"start": 1508.7, "end": 1538.26, "text": "As your company grows, it'll be tempting to introduce new technologies. Everyone who joins will probably come in with a set of things that they're familiar with and a set of ways that they know of solving problems. And the thing about a monoculture is it's really intellectually dissatisfying because it means you're no longer trying to find the best tool for the job, but you're trying to find the best plausible way of doing everything within your existing solution, no matter how kludgy it is. And for a purist, that can be really, really painful."}, {"start": 1538.26, "end": 1558.78, "text": "But the benefits that you get, those are totally worth the cost there. The thing you end up with is that if you have everything on the same stack, if you have everything built on top of some libraries that you've written, that a change in one part, if you fix some performance bug, if you figure out how to tune whatever virtual machine you're using, that those will suddenly be..."}, {"start": 1558.78, "end": 1572.59, "text": "Without any further effort, all that work will benefit all of your other code that you'll just be able to roll out to all of your other services with no additional code writing. And again, code writing is hard. You want to minimize how much of that you have to do."}, {"start": 1572.59, "end": 1600.5, "text": "There are some times where you do have to give up on this. There are times where if you... So we, for example, we're in a Ruby stack, and when it came time for us to start doing big data things, you kind of look at the solution space. And, well, I guess you could try to do everything in Ruby, but that there's already this existing ecosystem over in Java, and that Ruby probably, sort of in the scheme of wrong tool for the job, that that would be pretty high up there. And so..."}, {"start": 1600.5, "end": 1624.88, "text": "That in that case, you do have to just introduce an entirely new stack and figure out how to run it. But whenever you do that, that you should realize that that choice is going to be there with you forever. That forevermore, you're going to have to figure out how to run not just your original stack, but whatever this new thing is that you've introduced. And so you basically want to be able to count on one hand the number of times that you do that in effectively the entire life cycle of your company."}, {"start": 1626.48, "end": 1655.97, "text": "So technical debt is something that always comes up in these sorts of talks. And I think the best explanation of what technical debt is is this diagram. I can't claim credit for it. But the idea is that when you're building stuff, you sort of have four types of things that go into it. So you have the things that are visible, the user sees, and that are good. Those are what we call features. Then you have things the user sees and the bad shouldn't be there. That's what a bug is. On sort of the invisible side,"}, {"start": 1655.97, "end": 1686.21, "text": "No one really sees, but it's there. The good stuff, that's architecture. That's the things that we meant to be there. Those are the things we spent time thinking about. And the bad stuff, that's technical debt. This is the sort of soup of bad things that have entered your code that have made it hard for you to change it in the future that at some point you're going to have to deal with if you ever want to be able to get things done. The thing that most people ask about technical debt is, okay, I have a lot of it. It's slowing us down. How do we stop adding new technical debt?"}, {"start": 1686.21, "end": 1715.63, "text": "I think that's the wrong question. The thing that most people miss about technical debt is that if you manage it properly, if you have the right amount of it, it's actually good. It's just like real debt. It's something that in the future you're going to have to pay it off, that yes, that is something you'll have to do, but that for now it lets you get things done in a different way. And it means that the work of polishing your systems and of making things work as you'd like them to, that's something that you can spread out, that you can smooth, rather than having to pay everything up front right now."}, {"start": 1715.63, "end": 1744.69, "text": "And of course, like real debt, if you let too much of it accumulate, then that's it. You lose. There's nothing you can do. You're just not going to be able to make changes. You won't be able to get things done anymore. And so it's really just a question of not how do you stop accumulating technical debt or how do you stop creating it, but it's how do you manage it. How do you make sure that you're balancing your influx with the technical debt that you're paying down? How do you make sure that you're making a good balance?"}, {"start": 1744.69, "end": 1769.82, "text": "And that's a really hard question. I don't think there are any silver bullets there. I think there are some rule of thumbs. One rule of thumb I like to use is that you shouldn't do work that you're later going to undo. But that's definitely not 100%. I think that's probably about three quarters of the time that that turns out to be accurate. And I think it's also really important to make your technical debt explicit. There's nothing worse than you go into some system, you open it up, and you see, ah!"}, {"start": 1769.82, "end": 1796.03, "text": "That's how that was done and that no one realized that that was in fact what had happened. That people think from the outside that the system is much better than it actually is. And so I think it's really important to make sure that you're thinking about what are the corners that I'm cutting here? What are the things that I'm doing in order to get this thing out? So that at least you can make intelligent decisions about whether or not that makes sense. So given those statements, I think it's really important for you to pick."}, {"start": 1796.03, "end": 1819.86, "text": "Some properties that all the software you ship needs to have. You should basically assume that anything not on that list is going to be sacrificed in some permanent way in some project somewhere. And I think good testing, that should basically always be on this list. That you should never roll something out that isn't tested because you just will not be able to pay off your technical debt later. That's a way that you turn it into something permanent."}, {"start": 1819.86, "end": 1847.47, "text": "Security, that's actually a very interesting one, and that's actually a configuration option, which really depends on your company and your culture. Do you allow people to embed secrets in your code? That's really up to you. What kinds of third-party services do you allow? What kind of data do you give them? That really depends on your company, but it's important that you have very clear standards for what's okay and what's not. Quality is another thing that I think is really important."}, {"start": 1847.47, "end": 1872.74, "text": "One very surprising thing that I think a lot of us run into is just how long code that we write ends up staying out there. That actually my first project at Stripe was this thing called Password Vault, which was effectively a service for managing shared passwords. And it was pretty ugly that I figured I'd roll it out and make it better sometime in the next couple weeks. And no one has touched that code for the past three years."}, {"start": 1872.74, "end": 1897.81, "text": "There are a bunch of other standards that you might choose. What kind of monitoring do you want? Do you need to package up all your services? Things like that. And the choice that you make here, that really says a lot about who you are as a company and a culture. And one dissatisfying thing about having standards is you just can't have too many of them. That if you have all these requirements for things that all the software you ship needs to have, you're just not going to get anything done. That that won't work either."}, {"start": 1897.81, "end": 1927.92, "text": "And so you need to really think about what's important to you, what are the things that you're absolutely unwilling to compromise on, and make a decision based on that. So because humans are so bad at building software, it's really important to not just lean on your own intuitions and assumptions, that you should make sure that you have structures in place that sort of catch the fact that you're going to make mistakes and that are able to make sure that you're constantly reevaluating and looking at your priors and adapting the situation at hand."}, {"start": 1928.34, "end": 1946.38, "text": "So I think one really good rule of thumb is to try to minimize the distance to your first production use. And so basically anything you do should be as incremental as possible. You should trade off features, but don't trade off implementation quality. And so one of the great things about once you have a thing in production is that"}, {"start": 1946.38, "end": 1972.51, "text": "Suddenly, it can't go stale anymore. When people are making changes, they will have to go and make changes to the thing that you're working on. It's not even that much more work. If it's on a branch, no one's going to see it. No one's going to realize this is a thing that they should be thinking about. As well, it also really helps you guide which problems are important to solve. Maybe that thing you thought would be a bottleneck, it turns out that once you have this thing running on some percentage of data or something,"}, {"start": 1972.51, "end": 1992.98, "text": "It turns out just not to matter. Or maybe that web face that you thought you could punt on, it turns out people really need that to interact with the system. And in reality, I think what's going on is that if you get this thing into production, you're getting the universe decide what things are important versus you just trying to have to model this on your head and figure that out."}, {"start": 1992.98, "end": 2008.26, "text": "Really, once you're out there, that all the problems are nicely incremental, and it's way easier to solve incremental problems than it is to sort of build something from scratch, where it's not even clear exactly what problems are important and what ones aren't. And so I think that really your..."}, {"start": 2008.26, "end": 2025.58, "text": "Health as an engineering organization, your ability to get things done. It really comes down to how long does it take for you to get something from zero to production. The iteration cycle, that's the thing that I think most people focus on. And I think that that's actually pretty good at most companies. But the thing that I think most people just don't think about is this initial number."}, {"start": 2026.13, "end": 2052.54, "text": "And so I think the kind of impact that thinking about things in this way can have, I think, is illustrated again by our sharding project. So when we first sat down and came up with the design, we implemented the code in about three weeks to implement all the core sharding stuff. But we figured it would take us three months in order to get it fully comfortable rolling it out to production. Sharding is scary. You have distributed systems. You have your most critical data. These things should not go together."}, {"start": 2052.54, "end": 2078.86, "text": "But this is a place where, in fact, they have to. And so how do you get comfortable with it? Well, there are all these moving parts. You need to write something to split shards. You need to make sure that you're going to be turning correct results. You need to make sure you're not losing data. All of that, that takes time to get comfortable with, especially if you don't have anything to run it on. And so this was starting to sound like a massive project. And so we thought really hard about, OK, is there any way that we can make this more incremental or that we can be more comfortable with this than going out earlier?"}, {"start": 2078.86, "end": 2106.06, "text": "And finally, something she suggested, well, why don't we just roll it out to some internal collection that we don't really care about? We'll roll it out there. If we lose data, it would be a little bit sad, but we have snapshots, and this is something that just isn't production critical. And so we were able to roll that out immediately. That was a thing where it's just like, there's basically just an incremental week in order to get that going. And then once this thing was in production, suddenly it wasn't vaporware anymore. It was a real..."}, {"start": 2106.06, "end": 2134.99, "text": "A piece of software solving real problems that looking at it, that we were able to figure out, okay, what are the things that just aren't working that we need to solve? Suddenly all these hypothetical things that could go wrong, we were seeing which ones were going wrong, which ones weren't, and getting a lot more confidence with it in a way that we just would not have been able to if we'd continued to just sit around and think about it. So another part of that sharding project that was really useful was when we were trying to get it actually fully into production and to go from that..."}, {"start": 2134.99, "end": 2155.74, "text": "Okay, we have it on some collection we don't care about. How do we get to the one that we do care about? It wasn't conversations between the people who were really spearheading the project, who were really steeped in all the details, but it was actually a conversation between someone who was really steeped in it and working on it and someone who had the right background but was coming at it from a fresh perspective."}, {"start": 2155.74, "end": 2180.27, "text": "And we realized it was going to be a long time before we were really ready for it. That we had this shard splitter, which we hadn't really started implementing. That this was something that we just had to get right. There was no way to do it incrementally. We couldn't play that same trick. And so at one point, these two people were talking, and the engineer was going through and saying, here are all of the issues, and here's why we can't do it incrementally. And the person who had the background wasn't..."}, {"start": 2180.27, "end": 2195.46, "text": "Stephen, the project was sort of just asking questions and trying to figure out what was going on. And at one point it was like, wait, so the problem here is that we have the shard splitter that we won't be able to shard any of our existing data, but why can't we just start sharding all the new data?"}, {"start": 2195.46, "end": 2218.88, "text": "And sure enough, that was a great incremental solution, that we were able to just roll out charting immediately from there, and that you basically meant that our databases, while it wouldn't alleviate all the load, that they just would catch no more on fire. And so that was a pretty good win. And so now the interesting thing here is it wasn't like that person who was coming at this from a fresh perspective."}, {"start": 2218.88, "end": 2248.21, "text": "It wasn't like they were just a better engineer. It had nothing to do with their technical abilities. In fact, the people that we put on this were the best people at the company. But I think that there's something about when you're building a system, you're just making these thousands of little design decisions just constantly. What do you want to call this thing? All the way up to just how should the overall system work? And probabilistically, you're going to get a few wrong. That percentage probably varies per individual, but there just will be some decisions that you make incorrectly. And so if you have someone else come at it from a completely"}, {"start": 2248.21, "end": 2259.57, "text": "They're going to be able to have a different set of decisions that they see and feel like, wait a second, this one should be called out on."}, {"start": 2259.57, "end": 2286.35, "text": "I mean, I think that this is sort of what goes on in rubber ducking, right? That you actually are forced to re-examine all of your assumptions. And that it's actually, I think, even more useful to have someone else who understands what's going on, who's able to ask questions and actively press on things that don't make sense. And so one thing that we do for all major projects at Stripe at this point is make sure we have someone who is close enough to it that they can question all of the assumptions. So bus factor is another thing a lot of people talk about."}, {"start": 2286.35, "end": 2312.48, "text": "And most people think about bus factor as this is the number of people that can leave the company before we're in trouble, that we're unable to get our projects done. But I think actually the thing that bus factor is most useful for is not the how do we make this project survive, but getting a better project done. If you have people, if you have a collaboration, that again it comes back to the assumption questioning thing, where I..."}, {"start": 2312.48, "end": 2341.44, "text": "If you have someone else involved that they're able to say, hey, this thing doesn't make sense, press on it and ultimately make a better result. Forcing functions are an interesting one. They were something that I used to just not believe in at all. The thing about a forcing function is it feels very intellectually dissatisfying because it meant that you were unable to prioritize and that you had to sort of shell out to this external thing in order to get things done. And so why can't you just get better at prioritization?"}, {"start": 2341.44, "end": 2359.49, "text": "And I think the answer there is that prioritization is just actually ten times harder than any of us really understand. And that we're just not able to do it without setting up some external force that tells us what we need to do. And that in that way, the universe is just much better at computing what's important and what's not."}, {"start": 2359.49, "end": 2389.17, "text": "I think basically this is the best way of combating Parkinson's Law, that if you want to get something done, you should set a date for yourself. One thing that I found works very well is setting a soft launch deadline and a hard launch deadline, and space them out by a week or two weeks, depending on your project, and that you'll basically never make a soft one, but that at least that'll be a wake-up call saying, oh, wow, we really need to work harder in order to get done by the second one. And that gives you a checkpoint to actually figure out whether you'll be able to make it or not."}, {"start": 2389.17, "end": 2416.53, "text": "I think one really important thing, though, is that you shouldn't set externally facing deadlines. It's really tempting to say when a user comes and asks you when a feature is going to be done, oh, it'll be done by the end of the month. But what do you do when it's not done? That's a way that you end up with really unhappy users. And sort of the worst thing that you can do to an engineer on a project is to force some deadline on them that they're unable to move and they didn't really agree to or decide on. And so one thing that we do at Stripe is that"}, {"start": 2416.53, "end": 2446.26, "text": "Deadlines have to come from the engineers and that they get to decide exactly who those can be communicated to and in what way. So once you've gotten something ready to ship, how do you actually launch it? How do you actually get it out there? So at Stripe, a PM is a verb. It's not a person. For every project, we have someone PMing it. And this is the person who's responsible for making sure that all of the concerns associated with that launch get done."}, {"start": 2446.26, "end": 2473.54, "text": "And one of the things that happens is that as your company grows, as you are doing more things, that the launch process becomes more complicated. There are more things to think about. Maybe you need to think about tracking. Maybe you need to think about monitoring. It becomes really hard for any one person to think about that. And so the way that we do launches at Stripe is that we have a well-known list that you send an email to saying, I have this thing that's going to be launching. Give it a timeline. Give it a lot of context. Make sure that people are aware of what's happening. That you basically failed if..."}, {"start": 2473.54, "end": 2503.82, "text": "Anyone at your company finds out about a thing you're launching at the same time that the public does. Another thing that's really important is that your launch should not be viewed as the end goal. Your goal is to make a change. Your goal is to have some impact. And so you should make sure that after things are out that you're going back and saying, okay, did this actually do what we wanted it to do? And on the flip side, you also have things that are going to go wrong. That does happen. And it's really important to have a good post-mortem culture in place."}, {"start": 2503.82, "end": 2528.51, "text": "I think that there are a bunch of things that go into a good post-boredom culture. You should make sure that they're blameless, they're not about pointing fingers. You should make sure that they result in actionable changes that you'll actually make. I think one thing a lot of people do wrong is that they just end up with a lot of platitudes of, oh yeah, we should make the code here better, and then nothing changes as a result. If there's one thing your engineering culture needs to do well, it's collaboration."}, {"start": 2528.51, "end": 2545.68, "text": "If you're doing it right, your organization is made up of all these individually capable nodes. And then the main problem you have is how do you make all these nodes work together to produce something great? And if your collaboration is broken, basically everything else that I've mentioned doesn't matter. You're pretty much doomed."}, {"start": 2545.97, "end": 2559.49, "text": "So one thing we do at Stripe is we look for low-effort ways to make information accessible within the company. We do something we call email transparency, where if you're sending an email, you should CC it to a list."}, {"start": 2559.49, "end": 2588.78, "text": "List infrastructure, this means that people are able to effectively subscribe to the feed of everything happening at the company. And of course, there are some emails that you really shouldn't cc to a list if they're personal or if they're about people or those sorts of things. We leave that up to the discretion of the author. But the interesting piece here is that this is something that just doesn't require much effort. Ccing an email, that's easy. It's already in your flow. It's something that's a native part of your tool chain. And by just taking this one little hack, we're able to get a bunch of information out there."}, {"start": 2588.78, "end": 2615.66, "text": "They would just be siloed otherwise. Documentation is a very interesting communication channel. I think a lot of people write documentation the wrong way. A lot of people say that this is going to be the thing that will tell me about the code. But the important thing about docs is there are no tests. You can change the code, not change the docs, and everything will still appear to be fine. And so just the thing that happens, no matter, I've not seen an organization where this is false."}, {"start": 2615.66, "end": 2639.76, "text": "is that documentation just becomes out of date with your code. And so what you should view documentation as is as something that gives people context. It should be possibly a set of pointers to high-level concepts that people can figure out by reading the code. It should give people a place to start. And you should try to make it as abstract and high-level as possible in order to make sure that it doesn't go stale. Meetings are a communication channel to just get a bad rap."}, {"start": 2639.76, "end": 2664.64, "text": "I think the problem with meetings, it isn't that meetings are inherently flawed, but it's that people's usage of meetings tends to be pretty bad. There are basically two kinds of meetings that work. The first is the, let's kick a bunch of ideas around. The second is the, let's take these things we've been talking about and actually make a decision. But if you try to mix and match those two, if you try to start from zero and actually get something done, it's just not going to work. That will just result in sadness."}, {"start": 2664.64, "end": 2690.32, "text": "I think one other really important thing to note is that meetings have a very high fixed cost. You basically can't get any work done for the half hour leading up to a meeting because you know you're about to be interrupted, and so it's just not worth getting into the zone. And for the half hour after a meeting, you're spending that time ramping back up, figuring out what you were doing again. And so basically for every programmer, there's a fixed hour cost independent of how long the meeting actually is."}, {"start": 2691.41, "end": 2720.34, "text": "One of the biggest challenges with actually figuring out what to do when you have a bunch of great people is how do you resolve disputes? So if you have two designs and you're trying to figure out which one to pick, how do you figure out which one is the thing that you should do? And it's very easy to be paralyzed, to just be debating these back and forth and just to spend a ton of time on that. This was something that happened early on in Strikes history. We basically spent three weeks debating whether or not we should switch to a restful API. And at the end of the day..."}, {"start": 2720.34, "end": 2749.2, "text": "That it turns out that when you have fairly good alternatives, that you just need to make a decision. You need a good way for that to happen that feels fair to everyone and that people feel that in aggregate the right decisions will be made. And so the way that we tend to do this is that we segment out the various components at Stripe, that we have an owner for each component, and that they're the one person who ends up making all the decisions on is this a thing that we're doing here or not. So that happens for the API, for our website, and other components like that."}, {"start": 2749.2, "end": 2778.48, "text": "And that it's not always this glamorous job where you're just calling all the shots, right? That you're sort of responsible for the overall quality of the thing. And so that means that if there's maintenance that needs to be done, you don't necessarily need to do it yourself, but you're responsible for making sure that it gets done. One question that a lot of organizations are faced with is, should we hire remote engineers? Now, it's really very tempting to do this because there are probably a lot of great engineers that aren't within commute distance of your office. And so if you're able to hire them,"}, {"start": 2778.48, "end": 2805.6, "text": "Then you can successfully expand your talent pool. Hiring is hard. We all know that. Now, getting things done as a remote, it's basically this exercise in gathering information. You're coming from way behind everyone who's local. Locally, if there's a conversation going on in the office, you'll probably passively overhear it. You won't overhear all of them, but there's some chance that you will. As a remote, you have a 0% chance of hearing that. And so what this means is a good remote culture is one that ends up pushing everything into a written medium."}, {"start": 2805.6, "end": 2835.9, "text": "And the only way that that's going to happen is if there's a good forcing function there. And that a single remote or a few remotes, that that's not enough. That them saying, hey, I don't know what's going on. I'm not hearing enough. That's not a way that any of those problems get surfaced. You need to make sure that you have enough people who are working remotely that their problems are not considered second class. That those are something that people feel and that the only way that they can get work done is by communicating appropriately. So that's everything I have. Thanks a lot."}, {"start": 2835.9, "end": 2856.7, "text": "If you have any questions, I think we have a few minutes left. The question was, how do we choose the design dictators? So to some extent, there are only two properties that you need for a design dictator. One is that it needs to be someone whose judgment people generally trust. If you don't have people whose judgment is generally trusted, then you're probably doing something wrong."}, {"start": 2856.7, "end": 2886.9, "text": "The second thing is you need someone who cares. You need someone who's really passionate about whatever that thing is. And so for us, the person that we chose to be the dictator of API was the person who had started that three-week-long debate on whether or not to be restful. We didn't all agree with him, but that didn't matter. What mattered was here was someone who was going to care about it and who had good judgment and good ideas. So the question is, is there a technical debt limit? Is there a debt ceiling on technical debt?"}, {"start": 2886.96, "end": 2916.37, "text": "I mean, I think with all these things, it's a little bit too fuzzy to put a number to it. I think the answer is absolutely yes, that there are qualitative ways that you can tell, are you doing it wrong? If you find that, and I think a lot of this comes from just what the sentiment of people is, that if people feel like they're spending most of their time dealing with other people's design mistakes, that's clearly a world where you have too much. But at the same time, if you never have those murmurings, you're probably spending too much time polishing things, and you should instead be trying to move faster."}, {"start": 2916.82, "end": 2944.88, "text": "Okay, so two-part question, I guess. So first question is, how did you identify that you had a unique sharding problem? And then the second question is, how do you actually defend the maintenance cost versus something off the shelf? And I think the answer there is actually, it turns out that most people will have a unique sharding problem. And the way that you can tell is that if you look at what goes into a generalized sharding solution, that there are a bunch of assumptions about the data, right? That you have a..."}, {"start": 2944.88, "end": 2974.72, "text": "You need some shard key, and that what makes sense really depends on your application, right? So there's always some choice that you as an application developer need to make there. And the thing that we kept running into was that these operational issues were ones where it was like, well, if we just were able to tailor it to our application, that we know these assumptions about what the chunk range is, for example, and so we're able to just not even implement this case, which is going wrong. And so the kind of thing that happens is that"}, {"start": 2974.72, "end": 3000.05, "text": "Whenever there's a data dependency or whether there's something that depends on your application, that's usually a pretty good hint that this is something you should in-house. When it comes to defending the maintenance cost, you get to pick a very few number of these. You should not be implementing your own database from scratch. You should not be writing your own web server, maybe in some few cases. But the thing is that you only get a few of those."}, {"start": 3000.05, "end": 3029.26, "text": "And so I think it's actually not that hard to defend one of those in isolation, that it is just like sort of the way that the universe works, that there is no other choice. And really the question is, okay, so how do you defend doing lots of them? And the answer is you can't, right? But if you only have a few, then the fact that you're doing that should sort of be okay. So the question is to talk a little bit about what happens when you have sort of everyone involved in both defect work."}, {"start": 3029.26, "end": 3044.32, "text": "And the answer is that we played around a very little bit with people who are just a dedicated team for defect work. And the problem that we tend to run into is that you just..."}, {"start": 3044.32, "end": 3070.03, "text": "The people who are working on the defect stuff, it's really hard for them to think strategically about, OK, what are the problems we actually need to solve? And the people on the other side, they don't see any of the operational issues. And so what they're working on is, let's build this new feature, rather than, what are the things that are causing issues in our current one? And so actually, one place where we've done this back and forth is with support. So back in the day, we used to do, every engineer does support."}, {"start": 3070.03, "end": 3096.72, "text": "That over time, as we scaled, that we moved into a world where the support team just took care of that, blackboxed it, and that these days we actually now have people going back to the support rotation. And that's been something where there's just this qualitative change where you'd see that support would end up with a bunch of issues that were things that everyone was asking about, and that there wasn't always a good escalation path, or that for some of these things it wasn't obvious. Okay, so clearly people are having trouble with it, but how big of a..."}, {"start": 3096.72, "end": 3119.71, "text": "Fixes it, right? How big is that thing? And now there's this very clear communication channel, right? That people end up experiencing those issues themselves and if the engineers are doing it right, then they're able to go from there and actually implement the fixes. So the question is, with assumption questioning, does pair programming give you that? I think the answer is it gives you some of the benefits, but not all of them."}, {"start": 3119.71, "end": 3144.02, "text": "That one thing that's really important about assumption questioners is that they're not steeped in the details. That they're not someone who has been there as all these decisions have been made. That they're someone who comes from a completely blank slate, a completely fresh perspective. And I think we're at time. Are we at time? Someone who's authoritative? Okay, I guess yes we are. But feel free to come up and ask me questions afterwards."}]}