Ian Sherman is Head of Software at Formant, a company building cloud infrastructure for robotics. Prior to Formant, Ian led engineering teams at Google X and Bot & Dolly. The through line of his career has been tool building, for engineers and artists alike. He’s inspired by interdisciplinary collaboration of all types; currently this takes the form of applying patterns and practices from distributed systems operations to the relatively nascent field of robotics.

Links

https://formant.io

Transcript

Mike: This is the Real World DevOps Podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing the awesome work in the world of DevOps, from the creators of your favorite tools to the organizers of amazing conferences, from the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find. This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in.

Personally I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database, Influx DB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization and Kapacitor for real time streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of that out at Influxdata.com. My thanks to Influx Data for helping make this podcast possible.

Mike: Robots. You apparently are working at some company that does observability for robots and I'm a little confused because like what in the world is this all about? Do robots actually need observability?

Ian: Yeah. I work in a company called Formant. We are about a year and a half old and we're focused on a lot of problems in supported robots, but specifically observability for robotics is very important to us. I think it's representative of the type of concern that hasn't historically been important in robotics, but is increasing as we are shipping robots more and more to customers, deploying fleets of robot, deploying them in semi-structured environments, and generally seeing their numbers increase in the wild.

Mike: These robots, are these like Johnny 5 style robots or they're more like C3PO or The Terminator or Wall-E? Are these more Wall-E or maybe even the really terrifying stuff that General Dynamics is putting out?

Ian: Right. We like to maintain a flexible definition of a robot. I think that's maybe just a way of avoiding the definition question.

Mike: I'm sure the robots in the singularity will be very happy about your loose definition.

Ian: Yeah. The vast number of deployed robots in the world has sort of traditionally defines probably in the space of automotive manufacturing. That's where we see bolted down work cells of high payload position controls, heavy metal robots performing assembly and welding and applications like that. But the fastest growing part of the robotics market is actually in service robotics and in the deployment of robotics into less structured environments. That's environments like logistics and warehousing, retail, agriculture. This is where we have started focusing is in robots in semi-structured environments.

We do think that we have a lot to offer in industrial robotics as well, but it has some better focus to date.

Mike: I saw on your website there's this really interesting photo of a robot kind of strolling down the aisle at the grocery store.

Ian: Mm-hmm (affirmative).

Mike: Is that indicative of the kind of robots we're talking about primarily?

Ian: It is. We may have a little bit of an insight into the way things are going just from the customers we're talking to every day, but we are seeing more and more robots deployed into retail for example. It's just what that image shows. The applications at the moment are typically in things like floor cleaning, inventory scanning. Those are the front of house applications that we see the most often. Of course, in order fulfillment and logistics and warehousing, we see a lot of addition applications of robotics.

Mike: Got you. I want to take a little tangent here and ask how in the world did you get into this? I don't think anyone comes out of school and says, “You know what I'm going to do? I'm going to build robots and observability.”

Ian: I came to robotics through work at a company called Bot & Dolly about seven or eight years ago. It was focused on applying robotics to challenges in film and visual effects. I had an opportunity to get involved in novel applications of industrial robotics at that company. We were acquired into Google and that was around the time that a number of robotics companies were acquired, including Boston Dynamics we mentioned. Inside Google, I had the chance to see how all of our peers were thinking about these problems. We ultimately left Google about a year and a half ago because we were excited to ship products. The timeline for that is...

Mike: There's a very subtle danger there.

Ian: The timeline is long at Google for shipping products, but the experience was really invaluable. Personally, I was already interested in the tools and infrastructure side of robotics. Through building tools to support these teams inside Google and through seeing how people thought about problems like observability, software deployment, configuration management in the context of robotics; it became clear that there's actually a huge opportunity to bring some of the best practices that have been developed for decades in the backend distributed systems world to the robotics world.

That's where I find a lot of inspiration. The problem is similar enough that we have a lot to learn, but different enough that it does require some new thinking and some new technology.

Mike: That's a really great segue into a really good question of what is it look like to do observability in robots? You mentioned all these tools and all these techniques that infrastructure people rely on every day. I can think management and that sort of thing. How is that being applied in your work?

Ian: The fundamental requirement of observability and robotics is really no different than it is in monitoring backend systems. We want to maintain visibility into the state of the system. Use that information to allow humans to respond to changes in internal systems state and also automated systems to respond to those changes. But there's a few key differences. One is that the data types that are relevant to us in robotics are often different than they are in backend distributed systems. We have sensors generating a lot of data about the physical world. Those data types are often geometric or three-dimensional or media-based.

The infrastructure and tooling to ingest and index and visualize that type of data is different. The workflows that we used to debug issues are different. They often require making sense of a lot of that geometric and visual data. Another difference is that centralizing data is often ...

Aaron Sachs is a home brewer, banjo player, and also happens to like monitoring things. He helps make his customers look like monitoring badasses to their customers at Sensu, where he's a Customer Reliability Engineer.

Links Referenced
Sensu

Aaron.sachs.blog

Twitter: @asachs01

Transcript

Mike:  This is the Real World DevOps podcast and I'm your host Mike Gillian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools, to the organizers of amazing conferences, from the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

Mike:  This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database, Influx DB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, chronograph for visualization and capacitor for real time streaming. All of these are available as open source and as a hosted SAS solution. You can check all of it out at influxdata.com. My thanks to influx data for helping make this podcast possible.

Mike:  Hey Aaron, welcome to the show.

Aaron:  Hey Mike, thanks for having me on.

Mike:  So I want to start with a a little bit of an origin story because everyone loves a good origin story.

Aaron:  Oh yeah.

Mike:  You and I met over hot wings many, many, many years ago.

Aaron:  Hot wings, pizza and garlic knots. All good stuff.

Mike:  Right? Sadly the place is closed now, but we ended up going..

Aaron:  No.

Mike:  I know at the time I think you were working help desk support while studying for a communications degree?

Aaron:  Oh yeah, yeah. I was at a University of Tennessee's Office of Information Technology doing desktop support with a bunch of other students and yeah, working on my communication studies degree. Yep.

Mike:  Yeah. So the thing I find interesting about all this is that you never intended to go into IT at all. Like you weren't planning to go into tech in any way. You were really planning to go into something communication related.

Aaron:  Yeah. Yeah. So I was planning on being a communication professor and kind of aligned my career up to that point to do just that. And yeah... It was during that time, like so I was the whole reason I got out of that was I was writing my thesis on how one determines a blogger to be credible. Like what sort of behaviors do they exhibit when they were writing a blog uh and because that was nowhere near the specialty of the department I was in, there was just a lot of politics and I got so fed up with it and I was like, I'm done. And I'm already working at the Office of Information Technology to, do you know my grad assistantship, so why not just do IT?

Mike:  I remember you and I went out for beers, I want to say it was like a month after we met, and you you asked me a question of, "What if I stopped doing communications and went into tech? What would that look like?" And like the immediate follow up question was, "And can you help me?"

Aaron:  Yup. Yup. That's correct.

Mike:  Yeah. So, me not knowing what I was getting myself into of, "Yeah, let's just totally change the course of someone's career over beer."

Aaron:  By the way, for the listeners out there, if you ever want to sucker Mike into something, get him real drunk and then asking me if ask him for help. Totally works.

Mike:  Yeah apparently. So…

Aaron:  Haha

Mike:  This kind of started an interesting thing of you and I weren't really initially friends is more of a, you were looking to me for help on how to change your career and how to get better at a thing that you had not focused on at all. We actually became friends as a result of that like going through that whole thing, but it really developed into this informal mentorship situation.

Aaron:  Oh yeah. Yeah. I very fondly recall all of our a Tuesday nights at Old City Java in Knoxville, just gorging ourselves on Meg's croissants and downing gratuitous amounts of caffeine.

Mike:  Right. And wishing for a whiteboard to further explain concepts. Right?

Aaron:  Right.

Mike:  So through all this, clearly you've, you've done well you're working for Sensu now as a Customer Reliability Engineer, helping companies with improving their monitoring using Sensu. But one of your big focuses throughout your career, many years after us working together, has been to help other people improve their careers and improve their professional lives.

Aaron:  Absolutely. Yeah. It's so you're ready for that, by the way.

Mike:  Yeah. And so I know you've given several talks on this and you've had several other mentees over the years as well. I believe you were involved in a pretty formalized program at Rackspace on that topic as well.

Aaron:  Yeah, there was actually... I did a lot of mentorship with my team at Rackspace, so as I moved up in the admin ranks, I always took it under, rather took it upon myself to mentor the people that were coming in. And then that somehow turned into me mentoring folks that were in my wife, Ashley's department. We actually started like a meetup that would happen once a week at Rackspace and these are folks who came in as customer service technicians. They knew enough to spell DNS and grow them from just that to actually progressing into paths as like a one guy is a, I think he's a customer success manager for a startup in San Antonio and other guys as systems engineer now at Rackspace still, and then I've got another great friend of mine who's constantly blown away because she's actually taking this whole mentorship principle that we worked on at Rackspace and she's doing it with other people that she knows now. Her name's Elle, she works at I think Lennox Academy or Jupiter Broadcasting. So yeah, it's, it's been a crazy journey.

Mike:  Yeah, that sounds pretty awesome. So I want to dig into that. Let's back up and say what is mentorship?

Aaron:  So let's talk about what mentorship is not because I think that that's an even more useful way of discussing the concept of mentorship.

Aaron:  So mentorship, in my view is not simply... It's not transactional. It's not just the like... I mean I came to you initially and thought it was a transaction. I'm going to go to Mike and Mike's going to help me, which is fine, but that's not me...

Andrew leads technical strategy and architecture development for ACE IoT Solutions. Andrew also leads the development of technical and research strategy at The Enterprise Center, a non-profit focused on developing the innovation ecosystem in Chattanooga, TN. When not bringing his extensive professional experience in Industrial Control Systems, Critical Infrastructure Controls, and Network Engineering to his professional endeavors, he can most commonly be found with a camera in his hand. A deep passion for photography takes him off the beaten path the world over, and serves as a convenient excuse for a variety of other means for enjoying nature, including hiking, biking, and most board sports. Andrew loves sharing his travels and photography, and keeps an instagram account updated with his most recent adventures.

Links Referenced
LinkedIn

Twitter @acedrew

ACE IoT Solutions

Personal site

Transcript

Mike:  This is the Real World DevOps podcast, and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, from authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

Mike:  This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization, and Capacitor for real-time streaming. All of these are available as open-source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to InfluxData for helping make this podcast possible.

Mike:  Hi folks, I'm Mike Julian, your host for Real World DevOps podcast, and my guest this week is a friend of mine. Andrew Rogers, he's an expert in industrial control systems and co-founder of ACE IoT solutions where he helps companies with improving visibility in their operations and energy systems. Welcome to the show.

Andrew:  Thanks.

Mike:  Now you live in one of my favorite cities ever, which is Chattanooga, Tennessee. I can hear the Chattanooga fans in the background just, "Yeah, this is awesome." So one of the coolest things that I think there is in Chattanooga, aside from just the gorgeous weather and great food, is the low cost municipal internet.

Andrew:  Yeah, it's pretty fantastic. Um it's certainly part of the reason why I moved to Chattanooga in the first place. Uh and I think that it's had that effect on a lot of people over the years. So um we have, sort of, an out-sized technical community here based on the fact that it's easy to support a remote workforce when you have gigabit or 10 gigabit Internet available ubiquitously across the community.

Mike:  That was one of the things I never expected because there are actually some pretty significant companies based out of Chattanooga entirely as a result of this, highly available, municipal Internet like companies might know of Bellhops which is based there.

Andrew:   Yeah. So that project or that company started, based on another company in Chattanooga exiting and the founders starting a fund and looking for startups across the southeast that could benefit from the available broadband here. And you know it's a pretty big company now, well over 100 folks in their technical workforce, and they continue to you know grow and support the community. It's certainly been a really interesting success story for Chattanooga.

Mike:  You worked for a while as, I think, a technologist in residence for one of the startup incubators in the area.

Andrew:  So yeah, Chattanooga has a lot of really unique resources. You talked about one of those, the fiber broadband available over a 600 square mile area. And one of the other things that I think, is actually due to the fiber but also just due to the type of community Chattanooga is, and the effort and interest in working collaboratively, is a 501c3 nonprofit organization focused on entrepreneurship, and helping startups both at a scale of coming up and building a high growth startup, but also mom-and-pop shops, and helping them grow and build sustainable businesses or get to the next step in their business plan. They did, as part of when the fiber was first launched in 2009, a group of folks in the community came together to say, "Hey, how do we get the most out of this? This is an awesome asset. We know that the future of our economy is going to be based on growing businesses in the area. How can we use this new asset to support that, not just in luring a big company into town, and sort of the traditional economic development scenario, but growing companies locally."

Andrew:  And so one of the things they decided to try to do is launch a startup accelerator. It wasn't a venture funded accelerator, it was funded by this nonprofit. And it was focused on finding the companies around the country who had ideas that were only viable when ubiquitous broadband was available. Now, that brought a lot of really interesting folks to Chattanooga. And you know we did it about four years, I think. GIGTANK still exists today. It's changed a little bit, happens every summer. But yeah, one of the challenges with GIGTANK was that, you bring in a company, and you get a group of really great mentors from the business community here in Chattanooga. You surround them with professional services companies who are eager to support growing a business in Chattanooga. Then you tell them that you know you're trying to build a high growth startup, and your total addressable market is you know a million people in the US that actually have access to this high speed connectivity.

Andrew:  So what tended to happen, unfortunately, is the businesses were viable, but the reliance on the broadband wasn't. And now finally, in 2019, this was back... I was involved heavily in 2012, 2013, 2014. 2019, we're sitting here, and you see fiber deployments happening around the country. Talk of 5G is running thick right now, and so those businesses actually a lot more have a lot more viability. But what did happen was, the businesses came for the fiber and kind of stuck around for Chattanooga. So we ended up accruing and amazing tech talent pool, really interesting entrepreneurs uh who have gone on to focus on you know other things. We even took a little stint at additive manufacturing because we saw that, sort of, digital manufacturing gave you the same opportunity that, sort of, was the basis for deploying the fiber in the first place, which was making a smart grid that was truly smart, and making the trade of moving photons instead of electron to achieve the same sorts of quality of life improvements, et cetera, that rural electrification had in the early 20th century.

Jeremy Tangren is a Technical Program Manager specializing in infrastructure and vendor management. Throughout his 15+ years in IT program/project management, he has managed local- and global-scale multi-million dollar projects at companies like Facebook, Splunk, and Cisco. Jeremy is based in San Francisco, CA.

Transcript

Mike: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences or the authors of great books to fantastic public speakers I want to introduce you to the most interesting people I can find.

This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database influxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from Systems, Chronograf for visualization and Kapacitor for real-time streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to InfluxData for helping make this podcast possible.

Hi, folks, I'm Mike Julian, your hosts for the Real World DevOps podcast. My guest this week is Jeremy Tangren. He's an expert on everyone's favorite topic to completely ignore, vendor management. He's been a technical program manager for companies such as Cisco and Facebook, Splunk, and a whole bunch of other really interesting places. Most interesting to me is, Jeremy, you and I have known each other for God what? Nineteen years now?

Jeremy: Yeah, 19 going on 20 it's been a while.

Mike: It's insane. Yeah, which is weird. We met each other on a Final Fantasy Forum when we were like 12.

Jeremy: Thank you Final Fantasy Seven.

Mike: Yes. Thank you. So you've been working a lot with vendors throughout most of your career and I know you kind of fell into it accidentally.

Jeremy: Yeah, totally accidentally. It started when I was just a general IT guy and I was by myself. I needed more hands and more capabilities and I had to start bringing in vendors to get work done. It was a necessity.

Mike: Now, I imagine that most people listening to this thinking like, "Oh, my vendors, my vendors are awful," and I mean vendors are kind of annoying to work with aren't they?

Jeremy: They can be challenging. I definitely get the perception that most folks view vendors as a necessary evil and they would avoid interacting with them at all costs if they could.

Mike: That's been my experience as well. But I mean, there have definitely been a lot of vendors where I'm like, "I don't really want to work with them, but they get the job done that I need done and I don't want to do it myself." But at the same time, it doesn't really need to be that way.

Jeremy: No, not at all. You don't have to have an adversarial relationship with your vendors. In fact, you should be looking for more of a partnership than anything else with a vendor.

Mike: Tell me more about that. What do you mean by this partnership?

Jeremy: Well, take for example, when going back to my original IT guy explanation, I was by myself, but I had to have help. So when I brought in a vendor to install fiber or whatever it was, I had to work very closely with them to tell them where the work needed to be done, what my expectations were, all of the major details of the project, and then I had to again work closely with them during deployment and then finally on close of the project. At no point during that process was there an opportunity for me to step away from them and just let them run and expect things to go hunky-dory, and that's what most people tend to expect with vendors is that, "I pay them, I cut them a check, they get the job done. I don't think about it anymore," but there's just so much more than just cutting a check that goes into working with a vendor.

Mike: Okay. Let's dig into that. What else is there? Well, I mean when I have a problem I'm thinking, "I'm going to go find someone who is very good at this problem, some company, and I'm going to tell them what I need, cut a check and walk away." It's like hiring an electrician or a plumber. They're just going to get the job done. I don't have to think about it. So is there more to it than that? What else goes into it?

Jeremy: There's always more to it than that. When you're bringing in a vendor, they're specialists at whatever they do, the electrician, the plumber, but they still have to be told what it is that needs to be done. "Do you need to wire this entire house with three-phase power?" "Do you need to only fix this toilet?" They need to be given clear instructions and expectations or else they're just going to do what they think is right and that might not align with what your expectations are, so you have to work closely with them to get the right results.

Mike: I imagine that gets pretty complicated when we're talking about a software and IT. I can just imagine hiring a data comm person or data comm company to come in and wire a building for Ethernet and how complicated that would be.

Jeremy: Absolutely, you can't just bring in that cabling vendor and say, "Go." They don't know where all of the gotchas are in that building. They need a blueprint of the building. They need to be working with the facilities team. There're lots of cross-functional stuff that goes on just to wire a building with cables. So you have to have that partnership with the vendor and not just expect them to cowboy it up and take care of everything magically or again, you'll end up with results that you weren't expecting or didn't desire.

Mike: So on that note, the idea of hiring a vendor, just letting them loose. It seems like the thing that everyone does, and on that same note, people just don't think about their vendors, especially ones they're using long-term. I know for a long time I didn't until you told me how wrong that was, so why are we doing that? Why are we just kind of ignoring our vendors?

Jeremy: Because it's easy to do. I think that's what it really boils down to is, especially from an engineering standpoint, it's very easy to focus on what you're working on and what you need to get done versus the dependencies and interactions with these other teams, and vendors included. So it's very easy for an engineer or even a manager to forget that they even are working with a vendor. And when a vendor comes knocking, they go, "Hey, we're doing still doing x, y, z. Is it to your satisfaction?" And manager or engineer goes, "Oh, no, that's not right actually. You didn't do it the way I wanted," because they weren't partnering with and working with the vendor to guide them to a successful conclusion.

Mike: Something you said there made me think about this one particular thing that is often overlooked in vendor relationships, which is the role of the client. So as when me, the company, is hiring a vendor, I expect certain things from them, but to make a good relationship, they actuall...

Mary Thengvall is a connector of people at heart, both personally and professionally. She loves digging into the strategy of how to build and foster developer communities and has been doing so for over 10 years. After building community programs at O’Reilly Media, Chef Software, and SparkPost, she’s now consulting for companies looking to build out a Developer Relations strategy. In addition to her work, she's known for being "the one with the dog," thanks to her ever-present medical alert service dog Ember. She's the author of the first book on Developer Relations: The Business Value of Developer Relations (© 2018, Apress).

Mary is founder and co-host of Community Pulse, a podcast for Developer Relations professionals. She curates DevRel Weekly, a weekly newsletter that brings you a curated list of articles, job postings, and events every Thursday. She's also a founding member and "Benevolent Queen" of the DevRel Collective Slack team.

Mary is also a member of Prompt, a non-profit that encourages people to openly talk about mental illness in tech. She speaks at various conferences and events about building and fostering technical communities as well as how to prevent burnout in yourself and your team.

Links Referenced

• Twitter: @mary_grace
• Blog/Website: marygrace.community (or marythengvall.com)
• Biz Website: persea-consulting.com
• Newsletter: devrelweekly.com
• Podcast: communitypulse.io

Transcript

Mike: This is the Real World DevOps Podcast, and I'm your host, Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools, to the organizers of amazing conferences. From the authors of great books to fantastic public speakers, I want to introduce you to the most interesting people I can find.

This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools, Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of these are available as open-source, and as a hosted SaaS solution. You can check all of it out at Influxdata.com. My thanks to InfluxData for helping make this podcast possible.

Hi, folks, I'm Mike Julian your host of Real World DevOps Podcast. My guest this week is Mary Thengvall, she's a long-time builder of communities for companies like O'Reilly Media, and Chef. And author of the book on developer relations, The Business Value of Developer Relations, and runs a podcast, and newsletter of her own. Now she's helping companies create community-building strategy through a new consulting firm, Persea Consulting. Welcome to the show.

Mary: Thanks for having me, Mike, I appreciate it.

Mike: So, I want to start off with a really foundational topic.

Mary: Sure.

Mike: What is community in this context?

Mary: Absolutely. You start there, and I start there, they're most of my talks as well, just because I like to make sure that people have a shared definition. The definition that I always go back to with community is, and it relates to developer relations as well. But community for me is a group of people, who not only share common principles, but also develop and share practices that help individuals in the group thrive. So it's not just, "Hey, we're all on this platform." It's, "We're all on this platform, we have a specific goal in mind, and we're all helping each other."

Mike: Okay. A lot of us are pretty familiar with like the Python community, the Ruby community, or the Chef community, and then there's kind of the non-tech stuff. Like people that go to church, have a church community. Or we have the community of our neighborhoods. Are these the same community?

Mary: They have similarities.

Mike: Okay.

Mary: And it's that idea of, if you're in the Python community and someone reaches out on Twitter, or in a Slack team, or at PyCon for instance, which just happened. And they say, "Hey, I need help with these things." You'll likely find other people around you who go, "Oh, I know how to do that. Let me help. I'm more than willing to." In neighborhoods, you have things like next door, where someone will post, "Hey, I lost my cat." Or, "I need help with my garden." Or, "Help me out in these other areas." And people will jump in and say, "Oh, yeah, I'm more than willing to help out."

In those ways they can be very similar. There's obviously nuances with really just communities with technical communities, with neighborhood communities. And in some cases, they come together easier than others. Neighborhoods have the advantage of being in the same physical location, which is always really nice. Technical communities usually have forums around a particular product, or we use Twitter hashtags to kind of see what other people are talking about. And there's a big difference to me between communities of people, and community platforms.

And I know we have a lot to cover today, so I won't spend too much time on this, but just kind of a TLDR. The platforms are usually ways to bring people together to accomplish a specific purpose. A Python community platform for instance, you could have one for new Python developers, you could have one for experienced Python developers. You could have one for people who are willing to mentor folks, or people who are into Django, or whatever this specific piece of Python lore that you're interested in.

Mike: When you're talking about these platforms, are you talking about the medium on which they operate, or are you talking about kind of segmentations of that community?

Mary: A little bit of both, and those tend to go hand-in-hand. As you kind of see communities, larger communities, people who are interested in the same topics segmenting off, you'll often find software platforms that spring up that, you know, cool, the newbie Python people are hanging out on Twitter. The experienced Python people have a back channel slack team. The people, who are willing to mentor are on LinkedIn. So they're using already existing platforms to facilitate conversations around particular topics.

Mike: That seems like a bad thing.

Mary: It can be. The hard thing about that is, if there's not a specific person that's really taking the time to lead those groups or manage those groups, or build those groups of communities, that things can devolve fairly quickly with Twitter as we've seen. Or with Reddit or places like that where it's just not... It's hard to control when you don't own the platform, or when you don't own the conversations. But at the same time if you're not, owning the conversation as a company, then you aren't necessarily limiting people to what your viewpoint is. You're allowing people to talk and explore, a...

Kelly Shortridge is currently VP of Product Strategy at Capsule8. Kelly is known for research into the applications of behavioral economics to information security, which Kelly has presented conferences internationally, including Black Hat, AusCERT, Hacktivity, Troopers, and ZeroNights. Most recently, Kelly was the Product Manager for Analytics at SecurityScorecard. Previously, Kelly was the Product Manager for cross-platform detection capabilities at BAE Systems Applied Intelligence as well as co-founder and COO of IperLane, which was acquired. Prior to IperLane, Kelly was an investment banking analyst at Teneo Capital covering the data security and analytics sectors.

Kelly graduated from Vassar College with a B.A. in Economics and was awarded the Leo M. Prince Prize for Academic Achievement. In Kelly's spare time, she enjoys world-building, weight lifting, reading sci-fi novels, and playing open-world RPGs.

Links Referenced: 

• InfraGard
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
• RSA Conference
• Chaos Monkey

Transcript

Mike Julian: This is the Real World DevOps Podcast and I'm your host Mike Julian. I'm setting out to meet the world's most interesting people doing always work in the world of DevOps. From the creators of your favorite tools to the organizers or amazing conferences. From the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with our other tools. Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and as a hosted SaaS solution. You can check it out InfluxData.com.

My thanks for InfluxData for helping making this podcast possible.

Hi folks I'm Mike Julian, your host for the Real World DevOps Podcast. My guest this week is Kelly Shortridge, the VP of Product at Capsule8 and an internationally known speaker on InfoSec topics.

So Kelly, welcome to the show.

Kelly Shortridge: Thank you so much for having me Mike.

Mike Julian: You know I was looking at your LinkedIn and there was something that kind of stood out to me was your FINRA license. You started your career off in finance.

Kelly Shortridge: That's true.

Mike Julian: So what in the world happened there? How does that work?

Kelly Shortridge: Yeah. How does that even happen. So one, FINRA exams are very painful so I didn't want to have to re-up those, but mostly I started my career doing mergers and acquisitions covering information security companies. And while I quite liked the finance side I noticed that security had a ton of opportunity not just as far as vendors, but the problem space is huge and it's very unsolved and the incentive problems are enormous. So for someone like myself who had studied some Behavioral Economics, just all of the messy incentive problems were kind of like catnip for me. So I knew I had to go into the industry.

Mike Julian: Yeah. The Behavioral Economics side, the study of incentives. That's absolutely fascinating to me because, especially in security and systems got everything we do is incentives based.

Kelly Shortridge: Yes.

Mike Julian: And it's often incentive that we're not even paying attention to. Things we don't even think about. Like if you-

Kelly Shortridge: That's definitely true.

Mike Julian: Yeah. Like you make it hard to use two-factor and people aren't going to use two-factor.

Kelly Shortridge: Yup, so that's something that I feel like people outside of security understand immediately, but inside security they don't always understand the fact that if you don't make something that integrates into work flows, people are going to bypass it. But you're absolutely right, there's such a web of incentives and on the one hand you have things that are explicitly stated you know, security to your privacy is important, but then you have more of a tacit goals and priorities, which are that, well security's a cost center and really what matters is being able to deliver on time and you know releasing at a certain cadence.

So those tacit assumptions also create a bunch of incentive problems and InfoSec. But I always think that InfoSec because they wish they were more relevant try to be the culture of know and ram through really annoying technologies for people to use just to show that they're still relevant.

Mike Julian: Yeah, that hits home. I've seen that way too many times.

Kelly Shortridge: Yeah. I think most developers... It's not really a love hate relationship, it's mostly just a hate relationship for the most part. Somewhat bi-directional.

Mike Julian: What really about security drew you away from finance? Have you found there's good parallels that you've been seeing?

Kelly Shortridge: It's interesting. There's certainly parallels, particularly on the risk management side, particularly anything around risk centrality because that's a huge part of financial systems, so that's not really what I did day to day. I think what's interesting in security is there's a huge lack of effective communication. Even when you go to conferences you know, there will be some 0-day that's dropped or whatever, but it's often not communicated very well, and certainly when you look at enterprises, security priorities aren't really communicated well to the rest of the business. And a lot of investment banking is quite frankly effective communication.

It's about quickly researching something in general companies and understanding it very deeply to be able to talk about it and effectively persuade acquirers that's it's worth acquiring a company and so frankly even the excel and PowerPoint skills I learned along the way are really helpful in just being able to talk to people about security. You know I can speak to CEO's, I can speak to Board members, I can speak to DevOps people about security in a way that's still understandable and that's what I feel like we're still missing a lot of in information security.

Mike Julian: Yeah that makes a ton of sense. I was reading one of your articles, I can't remember which one it is. I'll have to go find it and throw it into the show notes, but you made mention of a tax from nation states. And I thought that was pretty interesting. It definitely stood out to me. And perhaps I have a bit more security exposure to that side of things than the average DevOps person might, just from, I worked for the government for a while so I've seen it. But for the vast majority of peo...

Christine delights in being a developer in a room full of ops folks. As a cofounder of Honeycomb.io, a tool for engineering teams to understand their production systems, she cares deeply about bridging the gap between devs and ops with technological and cultural improvements. Before Honeycomb, she built out an analytics product at Parse (bought by Facebook) and wrote software at a few now-defunct startups.

Links Referenced: 

• https://www.honeycomb.io
• https://www.heavybit.com/library/podcasts/o11ycast/

Transcript

Mike Julian: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, to the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

Mike Julian: This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast you're probably also interested in better monitoring tools and that's where Influx comes in. Personally I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with their other tools, Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to Influx Data for helping to make this podcast possible.

Mike Julian: Hi folks. Welcome to another episode of Real World DevOps podcast. I'm your host Mike Julian. My guest this week is a conversation I've been wanting to have for quite some time. I'm chatting with Christine Yen CEO and co-founder of Honeycomb, and previously an engineer at Parse. Welcome to the show.

Christine Yen: Hello. Thanks for having me.

Mike Julian: I want to start this conversation off in kind of what might sound like a really foundational question. What are we talking about when we're all talking about observability? What do we mean?

Christine Yen: When I think about observability, and I talk about observability I like to frame it in my head as the ability to ask questions of our systems. And the reason we've got that word rather than just say, "Okay well monitoring is asking questions about our system," is that we really feel like observability is about being a little bit more flexible and ad-hoc about asking those questions. Monitoring sort of brings to mind defining very constrict parameters within which to watch your systems, or thresholds, or putting your systems in a jail cell and monitoring that behavior, whereas, we're like, "Okay, our systems are going to do things, but they're not necessarily bad." But let's be able to understand what's happening and why. And let's observe and look at the data that your systems are putting out as well as thinking about how, asking more free form questions might impact how you even think about your systems, and how you even think about what to do with that data.

Mike Julian: When you say asking questions what do you mean?

Christine Yen: When I say asking questions of my system, I mean being able to proactively be able to investigate and dig deeper into data, rather than sort of passively sitting back and looking at the answers I've curated in the past. In order to illustrate this, to compare observability monitoring a little more directly with monitoring, especially traditional monitoring when we're curating these dashboards, what we're essentially doing is we are looking at sets of answers from questions that we posed when we pulled those dashboards together. All right, so if a dashboard has existed for six months the graphs that I'm looking at to answer a question like, what's going on in my system, are answers to the questions that I had in mind six months ago when I tried to figure out what information I would need to figure out whether my system was healthy or not. In contrast, an observability tool should let you say, "Oh is my system healthy?" What does healthy mean today? What do I care about today? And if is see some sort of anomaly in a graph, or I see something odd, I should be able to continue investigating that threat without losing track of where I am, or again relying on answers from past questions.

Mike Julian: So does that mean that curating these dashboards to begin with is just the wrong way to go? Like is it just a bad idea?

Christine Yen: I think dashboards can be useful, but I think that over use of them has led to a lot of really bad habits in our industry.

Mike Julian: Yeah, tell me more about the bad habits there.

Christine Yen: An analogy I like to use is, when you go to the doctor and you're not feeling well. A doctor looks at you and asks you, "What doesn't feel well. Oh, it's your head. What kind of pain are you feeling in your head? Is it acute? Is it just kind of a dull ache? Oh, it's acute. Where in your head?" They're asking progressively more detailed questions based on what they learned at each step along the way. Honestly this is kind of parallels a natural human problem solving concept. In contrast, I think the bad habits that dashboard lead us to build are things like it would be the equivalent of a doctor saying, "Oh well based on the charts from the last three times you visited, you broke your ankle and you skinned your knee." Pretend you go to the doctor to skin your knee. You know, "Oh okay, you broke your ankle last time, did you break your ankle again? No. Okay, did you ... How's your knee doing?"

With dashboards, we have built up this belief that these answers to past questions that we've asked are going to continue to be relevant today. And there's no guarantee that they are. Especially for our engineering teams that are staying on top of incidents and responding, and fixing things that they've found along the way. You're going to continue to run into new problems, new kind of strange interactions with routine components. And you're going to be able to ask new questions of what your systems are doing today.

Mike Julian: It seems like with that dashboard problem we have that same issue with alerting. I've started calling this kind of reflexive alerting strategy where it's like, "Oh God. We just had this problem. Well we better add a new alert so we catch it next time it happens." It's like well how many times is that new alert going to fire? Probably never. Like you're probably never going to see that issue again. With dashboards, dashboards are the same way. What you're describing, God I've seen this 100 billion times where someone curates a dashboard, is like, "Okay, well first thing now that we have this alert is let's go look at the dashboards and see what went wrong." I'm like, "Well no, graphs look fine." So no problem, but clearly the sites down.

Christine Yen: Yeah, there's a term that we've been playing with, dashboard blindness. Where if it doesn't exist in the dashboard it clearly hasn't happened, or you know it just can't exist, because people start to feel like, "Okay, we have so many dashboards...

James Turnbull is originally from Australia but now lives in Brooklyn, NY. He likes wine, food, and cooking (in that order) and tattoos, books, and cats (in no particular order).

He is a CTO in residence and lead startup advocacy at Microsoft. Prior to Microsoft, he was the founding CTO at Empatico. Before that, James was CTO at Kickstarter, VP of Engineering at Venmo, and in leadership roles at Docker and Puppet. He also had a long career in enterprise, working in banking, biotech, and e-commerce. James also chairs the O'Reilly Velocity conference series. In lieu of sleep, James has written eleven technical books, largely on infrastructure topics.

Links Referenced: 

• Twitter: @kartar
• Turnbull Press

Transcript

Mike Julian: This is the real world DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps from the creators of your favorite tools to the organizers of amazing conferences or the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization and capacitor for realtime streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to InfluxData for helping make this podcast possible.

Hi folks I'm Mike Julian, your host for the Real World DevOps podcast. My guest this week is James Turnbull. You probably know James from his seeming inability to stop writing technical books such as Monitoring with Prometheus, The Art of Monitoring, The Terraform book and like a bajillion others. He has also worked for some pretty neat companies too, like Puppet, Kickstarter and Venmo and now he works at Microsoft leading a team as CTO-in-residence. Welcome to the show, James.

James Turnbull: Hi Mike.

Mike Julian: I'm really curious like what is a CTO in residence?

James Turnbull: I guess my primary mission is to make Microsoft relevant to start ups much the same way that Microsoft is shaping its relevancy towards the open source community. We're also interested in looking at other audiences that we've traditionally not been involved with and so it's just one of those.

Mike Julian: Gotcha. And you're just leading a team of people that are focused on that sort of stuff?

James Turnbull: Yeah, so most of my team is people who've come from startups or and particularly from engineering management leadership roles in startups. One of my colleagues is ... was the CTO of SwiftKey and another one, fairly famous, Duncan Davidson who wrote Tomcat and Ant and has been around engineering management for a long time — and folks like that who really are here to help sort of startups understand a bit more about how to grow and scale. And I think some of the big challenges startups have are actually not technology related at all. They're really about, you know, how do I build a recruiting process? You know, I had 10 engineers last week, I have 100 this week. How do we structure the team? So we've sort of brought together a group of folks who have fairly deep experience in those sort of problems for startups and have sort of a deep empathy for the startup community.

Mike Julian: Yeah that's quite the task ahead of you.

James Turnbull: Yeah. Look, I think, I mean Microsoft traditionally been known as an enterprise software company. You know, a lot of startups are not sure of their relevance to us. I think increasingly we're seeing traction to cover is one is that obviously Azure is one of our focuses and the cloud platform in there. And that platform is looking more broadly at not just enterprise audiences but other groups. And secondly, a lot of startups ... Microsoft's deep in the middle of most of their customers. So particularly if you're a Beta based startups, something like that and you're, you know, you're trying to sell into enterprise business, Microsoft has been doing that for 30 years. They have all the connections and account managers and sales folks and you know, multimillion dollar relationships with some of the people you want to be customers with. We can provide you with A, some of those connections, but also a lot of advice and expertise about how to sell to those customers.

James Turnbull: And having worked at both Empatico and Docker, you know, a large part of my job was attempting to sell, you know, as a small startup, as a fairly early employee at both into big companies. You know, you can't walk in the door to Wall Street financial if you're a 30 person start up in Portland, Oregon without having a pretty credible story. So I'm happy to sort of help startups and I do some of that messaging and understand how to have some of those conversations.

Mike Julian: Yeah. It's one of the interesting things about my own company is my clients are all these large companies too and I'm a two person company, but selling into a very large company is not ... it's nothing like selling it to a small company. Everything works differently. People think about their jobs differently.

James Turnbull: Yeah.

Mike Julian: Yeah. I think maybe my most favorite thing of everything you just said is this isn't your daddy's Microsoft. The Microsoft we all grew to know and hate is not today's Microsoft at all. Not by a stretch and that's just absolutely incredible to see that turnaround.

James Turnbull: Yeah. I got a LinkedIn request from somebody yesterday and the message said, you know, I've read a bunch of your books and you know, I've used a bunch of different sorts things you've worked on. I was really surprised to see you at Microsoft. And I was like, okay, this could end badly the next couple of sentences, because I've certainly had a few people of my generation who remember the bad old days and “Linux is a cancer” and things like that. And he finished with, you know, it's really interesting to see companies grow and change. And I was like, wow, okay, that's a ... I thought that was going to go really badly but I think it's a fairly accurate reflection.

Microsoft is aware of the fact that this is not a position that pragmatically that was not a good business position to be in. The world is changing. It's moving towards the cloud, you know, the stacks in people's companies, the way they manage things, the infrastructure, the software, you know, things are changing. And I hesitate to say this conclusively, but I think open source won, you know this for certain values of one, given recent sort of events, discussions about large corporates and their contribution to open source, but as a technology choice, it's pretty clear to me that open source won. And I'm kind of a bit smug about that to be honest.

Mike Julian: Speaking of your books I would just straight up say you're th...

VM (aka Vicky) spent most of her twenty-plus years in the tech industry leading software development departments and teams, providing technical management and leadership consulting for small and medium businesses, and helping companies understand, use, release, and contribute to free and open source software in a way that's good for both their bottom line and for the community. Now, as the Director of Open Source Strategy for Juniper Networks, she leverages her nearly 30 years of free and open source software experience and a strong business background to help Juniper be successful through free and open source software.

She is the author of Forge Your Future with Open Source, the first and only book to detail how to contribute to free and open source software projects. The book is published by The Pragmatic Programmers and is now available at https://fossforge.com.

Vicky is a moderator and author for opensource.com, an author for Linux Journal, the former Vice President of the Open Source Initiative, and a frequent and popular speaker at free/open source conferences and events. She's the proud winner of the Perl White Camel Award (2014) and the O’Reilly Open Source Award (2016). She blogs about free/open source, business, and technical management at {anonymous => 'hash'};.

Links

• opensource.org
• Fossforge.com
• anonymoushash.vmbrasseur.com
• vmbrasseur.com
• marythengvall.com
• Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure

Transcript

Mike Julian: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools, to the organizers of amazing conferences. From the authors of great books, to fantastic public speakers, I want to introduce you to the most interesting people I can find.

This episode is sponsored by the lovely folks in InfluxData. If you're listening to this podcast you're probably also interested in better monitoring tools and this is where Influx comes in. Personally I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with our time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization and Kapacitor for real time streaming. All of this is available as open source, and they also have a hosted commercial version, too. You can check all of this out at influxdata.com.

Hi folks, I'm Mike Julian your host for the Real World DevOps podcast. My guest this week is VM Brasseur otherwise known as Vicky, an expert in open source strategy and the author of the book Forge Your Future with Open Source. She's previously the Vice President of the Open Source Initiative and currently Director of Open Source Strategy at Juniper Networks. Well Vicky thanks for coming on the show.

Vicky Brasseur: Well thanks for having me Mike, I'm very happy to be here.

Mike Julian: I want to start with a seemingly simple question, but I have recently learned in the past half hour that this is more complex than it seems. What is open source?

Vicky Brasseur: Yeah, can't imagine how you learned that. No, it's a question that a lot of folks in technology think they know the answer to, but unfortunately they're usually wrong. That's because they usually don't realize that there is a legitimate definition of what it means to be open source software. It is called the open source definition. It is maintained by the Open Source Initiative. If something does not adhere to each of those 10 points on the open source definition, it isn't really open source.

Unfortunately people just sort of assume, well if my source is out there, if my source code is out there, it's open, right? Well, not really, because if you restrict it in any way or if you don't put an appropriate license on it, then people don't know it's open source. If you just put your code out there without a license for instance, it's all rights reserved. You have the copyright over that code or your company if you developed it for your company. It's all rights reserved as far as copyright and no one else can use it, unless you put a license on and that's what the license does for you. Only an open source license, one that is approved by the Open Source Initiative, that's the only kind that you can be assured actually gives you all of the things that the open source definition guarantees.

Mike Julian: What's really interesting about that is, there's always people that go around GitHub onto like the main project and say, "Hey, I noticed that you don't have this license, you should really have a license file." I'd always thought that that was just kind of an oversight, like, "Oh yeah, it's fine, it's totally open source. There's just no license. There's no license file." What you're actually telling us is that, if you don't have that, if you haven't specified what license this is under, by default it's not open source. Like, it is “all rights reserved.”

Vicky Brasseur: It is, exactly. It is all rights reserved. The best you can call it is source available. You still retain all of the copyright over that, and therefore it is all rights reserved. You retain all rights to that code, no one can use that software at all unless you give them the rights to it. That means somebody could use your software and put themselves at legal risk by violating the copyright of your software and you. If you don't put a license on it, that's what they're doing. Therefore, they are at legal risk, they can get sued and if they are running a company and they're using your software, they can't really get acquired frankly if they are using software that is encumbered by somebody else's copyright. That's why it's so important for multiple reasons to make sure you have a license on there. It really takes care of all those legalities. It's a relatively short list of OSI approved licenses, you've got the Apache and the MIT and all your GPL flavors and LGPL and AGPL and yeah. There's a bunch of them and they cover a broad swath of things. If you just use one of them, you don't have to care about the legalities, somebody has already taken the time to figure that out for you.

Professional lawyers have written these things, gotten them approved by OSI. You know they give you everything from the open source definition and you know it's legal. Just use it. It's pretty easy.

Mike Julian: You just named off a whole bunch of different open source licensing. I'm always confused when I release a project, like what should I license this under? Screw it, I'll go with MIT or Apache and call it a day, and I never really put any thought i...

Ryn Daniels is a staff infrastructure software engineer who got their start in programming with TI-80 calculators back when GeoCities was still cool. Their work has focused on infrastructure operability, sustainable on-call practices, and the design of effective and empathetic engineering cultures. They are the co-author of O’Reilly’s Effective DevOps and have spoken at numerous industry conferences on devops engineering and culture topics. Ryn lives in Berlin, Germany with a perfectly reasonable number of cats and in their spare time can often be found powerlifting, playing cello, or handcrafting knitted server koozies for the data center.

Links

• ryn.works
• LinkedIn - Ryn Daniels
• @rynchantress
• Effective DevOps
• InfoQ Article: Crafting a Resilient Culture

Transcript

Mike Julian: This is the Real World DevOps Podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, or the authors of great books to fantastic public speakers, I want to introduce you to the most interesting people I can find. 

This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with our other tools. Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out InfluxData.com.

Mike Julian: Hi folks. I'm Mike Julian, your host with Real World DevOps. My guest this week is Ryn Daniels, co-author of O'Reilly's Effective DevOps, a public speaker and previously worked in engineering for both Etsy and Travis CI. Ryn, I hear you're working everyone's favorite infrastructure automation company now, HashiCorp is it?

Ryn Daniels: Yes, it is. I'm a working on the terraform ecosystem team. I'm going to be working on the AWS provider.

Mike Julian: You've been writing and talking a lot about this idea of resilient culture and you wrote a article for a InfoQ, which we'll link in the show notes, about crafting resilient culture, which talked about the Apache Snafu. You and I were just talking before the show about an earlier story about Postfix and Puppet and well, things exploding in your face.

Ryn Daniels: Yes, so it's a fun story with a little less of a happy ending than the Apache snafu. My first ops job I inherited two data centers that didn't even have a lonely bash script for company. I was doing everything by hand. There were a lot of dragons and nobody was really sure where are the dragons were lurking. One of the things that I was kind of put in charge of was the idea of, "What if we didn't do literally everything manually? What if we had some sort of automation?" So I got to do fun stuff like set up automated Linux installs instead of me going around carrying a USB DVD player and yeah.

Mike Julian: Definitely been there.

Ryn Daniels: Yeah, that that was ... Those were sad times. So I was starting to put together Puppet and it was mostly going pretty well. I was starting out with the what seemed like the safe stuff. And I asked the engineering team, I'm like, "So it seems like Postfix is configured a bit on these servers, but it's not running. Should it be running?" And people talked amongst themselves a little bit and they were like, "Yeah, it should definitely be running because the servers are set up to email us when something goes wrong." Okay.

Mike Julian: So clearly everything was fine because no emails were going out.

Ryn Daniels: Exactly. Exactly. So I clear this with everyone. I tell them, I'm like, "Okay, I'm going to roll out this change." And I turn on postfix everywhere. And this was my very first ops job, so we didn't have anything like a testing or a staging environment. I was really kind of playing everything by ear at that point and learning as I went. So I turn on Postfix and then a few minutes later somebody says the site's down. Like how did turning on Postfix take the site down?

Mike Julian: That's weird.

Ryn Daniels: And we kind of kind of poke a little bit on one of the servers that I was logged into and like the web server was still running. Everything looked like it should have been fine. What happened was there were eight years of emails queued up on every single server, and when Puppet turned on Postfix, those eight years of cued emails started sending all at once. And the way that networking was or wasn't configured back then, I think I just like saturated every single network link in our two data centers with all of these emails, and everyone's like, "Ryn, help, make it stop, get everything back on line." I'm like, "I don't know how to un-send eight years worth of email, folks. Like, we're just going to have to wait this out." Which is kind of what happened. And eventually, eventually all of the emails sent and shockingly, there were a lot of error emails as it turns out in this sort of environment.

Mike Julian: Surprise, surprise.

Ryn Daniels: Yeah. And after that everyone was a little twitchy anytime I mentioned making a Puppet change. So yeah, it was definitely an exciting afternoon slash couple of days trying to figure out what went wrong with automation and try and keep it from going that sideways in the future.

Mike Julian: How did your teammates react to all this? Like aside from like, "Ryn, what have you done?"

Ryn Daniels: It was, it was mostly just that kind of panic and then everyone trying to figure out what to do. People had differing amounts of visibility into what was going on. There was kind of a homegrown monitoring system that was set up that also lived in the data center, which may or may not have been very accessible during this time. Oh, I remember, I was stuck in the data center physically because nothing was configured to have a remote, out-of-band access. So most of my days were spe...

Baron is the CTO and founder of VividCortex, the best way to see what your production database servers are doing. Baron has written a lot of open source software, and several books including High Performance MySQL. He’s focused his career on learning and teaching about scalability, performance, and observability of systems generally (including the view that teams are systems and culture influences their performance), and databases specifically.

Links Referenced

• xaprb.com 
• High Performance MySQL: Optimization, Backups, and Replication
• eBook: DevOps for the Database
• @xaprb
• VividCortex

Transcript

Mike Julian: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps, from the creators of your favorite tools to the organizers of amazing conferences, from the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

Mike Julian: Ahh, crash reporting. An oft-forgotten piece of a solid monitoring strategy. If you struggled to replicate bugs or elusive performance issues you're hearing about it from your users, you should check out Raygun, whether you're responsible for web or mobile applications, Raygun makes it pretty easy to find and diagnose problems in minutes instead of what you usually do, which if you're anything like me, is ask the nearest person, "Hey, it's the app slow for you?" and getting a blank stare back because, hey, this is Starbucks and who's the weird guy asking questions about mobile app performance. Anyways, Raygun, my personal thanks to them for helping to make this podcast possible. You can check out their free trial today by going to raygun.com.

Mike Julian: Hi folks. I'm Mike Julian, your host of the Real World DevOps podcast. My guest this week is Baron Schwartz, the founder and CTO of Vivid Cortex and author of a book that I'm sure many of you have laying around, High Performance MySQL. Welcome to the show Baron.

Baron Schwartz: Thanks Mike. Great to be here.

Mike Julian: Baron some years ago, well, I also started writing a book in like 2015, I guess it was, 2016. And you've written a lot about your writing and I mean you are a very prolific writer on your blog, having also written the tome of High Performance MySQL.

Baron Schwartz: Maybe too much of a tome. Yeah.

Mike Julian: Right. It is a pretty sizable book. And I found this article that you had written about how you wrote, which I found absolutely fascinating. One of the pieces of advice I saw that I've used and have been regurgitating to everyone. How do you write the book while you outline? You start with an outline and then you keep outlining. And you keep and keep and keep on outlining until you're at the point where you should feel like you've stopped outlining long ago and then outline some more. Yeah, just a little bit more after that.

I thought that was incredibly insightful advice because most people, they do stop jotting down thoughts a bit too early.

Baron Schwartz: Yeah. I should use that advice myself. Every time I do it, I'm really happy with the results. The truth is I don't always, and then I'm like, "Why is this so hard?" It's just because I'm just slogging through not having outlined and now I've got ... So for folks who are not going to read this blog post, which I assume will be a lot of our listeners, the general idea that led me to try this approach was actually writing the second edition. So the blog post that you're talking about was written after the third edition of High Performance MySQL. And the second edition was an absolutely horrendous amount of work. I know because I used a timekeeping app and I basically spent like a half a year on it. So it was more than a thousand hours of really hard work.

And then afterwards I looked at what was most of that time spent doing and most of it was actually spent editing things that turned into pros too soon because pros has to have correct grammar. And if you don't feel right about the order of the ideas in a paragraph and you start fussing with it, then the grammar gets mucked up when you're sort of dragging things around or whatever. And so you try and fix that, but you're polishing something that is structurally bad and to fix it, you end up having to go back and restructure it and then re-edit it. But if you're re-editing for grammar and flow and everything the whole time, it's just an insane amount of work.

So the third edition, I did using mostly the outlining style that you're talking about, mostly after several years of having just collected notes, which is a part of the writing process that's largely invisible, just like filling files full of one bullet point URLs or quick little one liners or whatever. And I took that stuff, I sort of decomposed the second edition into its component parts, restructured it, put in the stuff that I wanted to update or new material, and then outlined and wrote. And the third edition was about 300 hours of work. Although it was a complete rewrite of the book and made it much larger. So it was a lot more material and a lot more accomplishment with a heck of a lot less work.

Mike Julian: Yeah. When I first started writing mine, I did actually start tracking time and then I stopped doing it after a while on the advice of one of the editors of the first edition of a friend of mine, Derek Balling, he gave me a piece of advice that do not ever under any circumstances, attempt to calculate your hourly rate of writing a book.

Baron Schwartz: Excellent advice. Yeah. If you do the math, what you end up earning from a book is almost always depressing in purely economic terms. However, I will say it's been a huge ... well, there's a couple of things. High Performance MySQL has been unusually best selling for a technical book and very few books ever strike gold that way, which is purely a function of just circumstances and luck and things like that. So I think if I calculated my hourly rate, it wouldn't be depressing, but that's-

Mike Julian: It would also just not be awesome.

Baron Schwartz: Yeah. It still wouldn't be awesome. But it's been good to get the royalty checks every now and then, which most authors never get. They never pay back their advances. And that's part of the publishing role. But yeah, even if you do, it's still ... But there's also a lot that you can't just reduce to an hourly rate about the benefits of writing a book, not only for example, the book lives on, and carries to many audiences, and reach as many people, and touches many lives that you never would have in other ways and that comes back and is very rewarding. People write me on a fairly regular basis and say something. I met with somebody a couple of months ago and he said, "I have to thank you for making me a millionaire." I said, "Well, can I get a finder's fee? Just 1%."

Mike Julian: Oh, wouldn't that be nice. It's always wild to me when I get the notifications that, "H...

Dr. Nicole Forsgren does research and strategy at Google Cloud following the acquisition of her startup DevOps Research and Assessment (DORA) by Google. She is co-author of the book Accelerate: The Science of Lean Software and DevOps, and is best known for her work measuring the technology process and as the lead investigator on the largest DevOps studies to date. She has been an entrepreneur, professor, sysadmin, and performance engineer. Nicole’s work has been published in several peer-reviewed journals. Nicole earned her PhD in Management Information Systems from the University of Arizona, and is a Research Affiliate at Clemson University and Florida International University.

Links Referenced: 

• 2019 State of DevOps Survey
• Previous State of DevOps Reports

Transcript

Mike Julian: This is The Real World DevOps Podcast, and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, and the authors of great books to fantastic public speakers, I want to introduce you to the most interesting people I can find.

Mike Julian: Ah, crash reporting. The oft-forgotten about piece of a solid monitoring strategy. Do you struggle to replicate bugs, or elusive performance issues you're hearing about from your users? You should check out Raygun. Whether you're responsible for web or mobile applications, Raygun makes it pretty easy to find and diagnose problems in minutes instead of what you usually do, which if you're anything like me, is ask the nearest person, "Hey, is the app slow for you?" And getting a blank stare back because hey, this is Starbucks, and who's the weird guy asking questions about mobile app performance? Anyways, Raygun, my personal thanks to them for helping to make this podcast possible. You can check out their free trial today by going to raygun.com.

Mike Julian: Hi folks. I'm Mike Julian, your host for the Real World DevOps Podcast. My guest this week is Dr. Nicole Forsgren. You may know her as the author of the book Accelerate: The Science of Lean Software and DevOps or perhaps as a researcher behind the annual State of DevOps report. Of course that's not all. She's also the founder of the DevOps Research and Assessment, recently acquired by Google, was a Professor of Management Information Systems and Accounting, and has also been a performance engineer and sysadmin. To say I'm excited to talk to you is probably an understatement here. So, welcome to the show.

Nicole Forsgren: Thank you. It's a pleasure to be here. I'm so glad we finally connected. How long have we been trying to do this?

Mike Julian: Months. I think I reached out to you, it's March now. I reached out in November, and you're like, "Well, you know, I have all this other stuff going on, and by the way, my company was acquired."

Nicole Forsgren: Well, back then, I had to be sly, right? I had to be like, "I've got this real big project. I'm sorry. Can we meet later?" And, God bless, you were very gracious and kind, and you said, "Sure-"

Mike Julian: Well thank you.

Nicole Forsgren: ... "we can chat later." And then I think you actually sent me a message after saying, "Oh, congrats on your 'big project'." I said, "Thank you."

Mike Julian: That sounds about right.

Nicole Forsgren: I appreciate it. Yeah. And then, you reached out again, and I said, "Oh, I'm actually working on another big project. But, this time ..."

Mike Julian: It's not an acquisition.

Nicole Forsgren: Yeah, it's not an acquisition. This time, it's a normal big project, and it's this year's State of DevOps report. And we just launched the survey, so I'm super excited we're collecting data again.

Mike Julian: So we can get that right out of the way, where can you find the State of DevOps report?

Nicole Forsgren: All of the State of DevOps reports are hosted at DORA's site. We still have the site up. And all of the reports that we've been involved in from, I want say we started in 2014, I'm so old I already forgot. All the reports that we've done are hosted. We'll post them in the show notes. If you can grab yourself a Diet Coke or coffee or a tea or a water, or if you want a bourbon. Get comfortable. Sit back, takes about 25 minutes. I know, right, everyone's like, "Girl, 25 minutes?"

Mike Julian: That's a big survey.

Nicole Forsgren: I know. It is. But it's because the State of DevOps report is scientific, right? We study prediction, and not just correlation. But sit back, get comfy and let me know what it's like to do your work. Because we're digging into some additional things this year; productivity, tool chains, additional things around burnout and happiness, and how we can get into flow, and really what that looks like. And some really great things are a bunch of people have already chimed in after taking the survey in really thoughtful ways. Also, by the way, I love you all for taking it if you have. Share it with your colleagues, share it with your peers.

But they've said that just by taking the survey, they've already come away, even before the report has come out, they've already walked away with really interesting ideas and tips and insights about how they can make their work better.

Mike Julian: Yeah, that's wild to think about, that the act of taking a survey actually improves my work. Because most surveys I take, I'm finished, I'm like, "Well, that was kind of a waste of time." It feels like I just gave away a bunch of stuff without getting anything.

Nicole Forsgren: Yeah, and I think the reason it works that way is because we're so careful about the way we write questions that sometimes just the act of taking the survey helps you think about the way you do your work. So just the act of kind of taking some of these questions helps people think about what they're doing. And then, of course, like I joked already, it's my circle of life, the survey will be open until May 3rd and then I will go into data analysis and report writing. And we expect the report itself to come out about mid-August.

Mike Julian: Well, why don't we take a few steps back and say ... Everyone loves a good origin story. I believe you and I met at a LISA many, many years ago. You were giving a joint workshop with Carolyn Rowland on-

Nicole Forsgren: Oh, I love Carolyn.

Mike Julian: Yes, she's also wonderful. I should have her on here.

Nicole Forsgren: My twin. Yes. Absolutely.

Mike Julian: So you were a professor then when I first met you. I'm like, you know that's kind of interesting that a professor's hanging out at a LISA and giving all this great advice on how to understand business value, which I thought was absolutely fascinating. Professor, hanging out in the DevOps world, how'd...

Thai helps teams build more resilient systems and improve their ability to effectively respond to incidents. A former EMT, he applies his experience managing emergency situations to the software industry. He writes about resilience engineering each week at ResilienceRoundup.com

Links Referenced: 

• Strongbad’s The System is Down 
• https://resilienceroundup.com/
• https://re-deploy.io/

Transcript

Mike Julian: This is the Real World DevOps podcast, and I'm your host, Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, from the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.

Ah, crash reporting, the oft forgotten about piece of solid monitoring strategy. You struggle to replicate bugs or elusive performance issues you're hearing about from your users, you should check out Raygun. Whether you're responsible for web or mobile applications, Raygun makes it pretty easy to find and diagnose problems in minutes instead of what you usually do, which, if you're anything like me, is ask the nearest person, “Hey, is the app slow for you?” and getting a blank stare back because hey, this is Starbucks, and who's the weird guy asking questions about mobile app performance? Anyways, Raygun, my personal thanks to them for helping to make this podcast possible. You can check out their free trial today by going to Raygun.com.

Mike Julian: Hi folks, I'm Mike Julian, your host for the Real World DevOps podcast. My guest this week is Thai Wood. He's an internal tools specialist at Fastly, the editor of Resilience Roundup, a fantastic newsletter that I'm a huge fan of. And perhaps really interesting is he's a former EMS professional. For those that don't know, EMS is what most of the Ops world keeps looking to figure out how can we improve our own on-call and incident management procedures. So, welcome to show Thai.

Thai Wood: Hey Mike, thanks for having me.

Mike Julian: So for those that really have no idea or frame of reference for what EMS is, what is it?

Thai Wood: EMS is sort of the broad umbrella of what happens anytime someone has a medical emergency, usually starts with someone calling 911. Someone gets sick or injured, they call 911 and the people who show up in whatever capacity are EMS workers and EMS professionals.

Mike Julian: So, would this also be like firefighters, police, doctors, nurses? Everyone's included in that or?

Thai Wood: Yeah, typically it's a big group of people just because who shows up actually, in the US strangely depends on where you live, what state you're in, what your county rules are, and a lot of things like that. So, here in Las Vegas, for example, we actually have three different fire departments, some of which run their own ambulance services, some of them do not. So it can also depend on what area of town you're in.

Mike Julian: Yes. San Francisco is the same way. The San Francisco Fire Department actually, I guess consumed the San Francisco Emergency Management into themselves so there's no such thing in San Francisco County is separate EMS. So that's why you hear so many fire trucks everywhere, it's because most of the incidents are not for fires, the city is not always burning down, it's just someone's always hurt. So they send a fire truck and police and more fire trucks. Where I'm from in Knoxville, Tennessee we actually had separate EMS, which was like they had the same vehicles as the ambulances except they were green instead of red, which was kind of cool. So yeah, that's fun.

So one of the things I really find interesting about EMS is when you look at people responding to incidents, and I'm using incident in a very broad, vague way here, someone gets hurt. And the people responding have just like the most cool, collected nature I've ever seen. And it's always like two people show up and everyone knows exactly what's going on. Meanwhile, the people who have called are freaking out, but the EMS personnel are calm and collected. How do you even get that way when someone's bleeding on the sidewalk, but EMS is perfectly cool about it.

Thai Wood: I think that's a really good question. It's actually something that I feel is missing in a lot of software, which is just a lot of it is experience. The hundredth time that you've been on a similar call, the thousandth time, of course like anything we habituate to it, and it gets easier. A lot of places with EMS, you have an opportunity to practice these things, and to get better at them. Oftentimes, that might be that you do ride-alongs even before you are certified. So, you're already becoming immersed in this world. You have an opportunity to do some clinical hours at hospitals, for example, sometimes, and you get to just be in these different situations. And of course, the first, maybe the fifth, maybe even the 10th, you feel the same way. There's also of course a culture where a lot of times what you're seeing is not in fact the truth.

Mike Julian: Okay. Tell me more about that.

Thai Wood: Well, depending on what it is that you're walking into, I or others might have an internal dialogue that say, “Oh, no, I have not seen this before.” But, we're not doing either of us a service by letting that dictate our outward response. If I'm letting that change my behavior or letting it allow me to be distracted or unfocused, I'm not helping you nor am I helping myself be more effective or be more effective in helping you.

Mike Julian: Yeah, that makes sense. I mean, I've definitely had that situation when I've been on call. You walk in has something like, well, everything's exploded, I have absolutely no idea what's going on. But as the senior engineer, everyone's looking to you like, expecting, you've seen this before. So you just kind of put on that veneer of, “Nope, I've got this,” while screaming internally.

Thai Wood: Absolutely. And I think that, depending on why it's done, I think it's actually a good thing. In software, I do tend to question a little bit, I think it should be okay to say, “I don't know, this thing. I haven't seen it before, but I'm going to figure it out,” whereas with … Because you're probably with personnel that you know at least somewhat if you're on call, right? Network operations, staff, people you've at least talked to before, which is not necessarily the case with emergency management, in the physical realm. These are strangers and you don't know how they're going to react. So your best service to them is to just keep that cool.

Mike Julian: Right. Yeah. It's just really hard to do sometimes or, most the time for me. Man, I do not miss being on call at all. I never liked the idea of having to present that coolness that everyone's looking for.

Thai Wood: I definitely understand that. It's a very visceral, physical experience. And I know a lot of us can habituate good and bad to some of this stuff. We know that noise, whatever our pagerduty alert noise is, we know that noise, right? Or even if you put it on vibrate, the sound of that phone vibrating against your nightstand, you know the ...

Cory G Watson is a Technical Director at SignalFx with 20 years of SWE, SRE and leadership experience at the likes of Twitter and Stripe. He's an observability wonk, optimist, and lover of sneakers. He hopes you're having a great day!

Links

• Twitter: @gphat
• Website: onemogin.com

Links Referenced

• Patrick McKenzie’s blog post, “I’m Joining Stripe to Work on Atlas”
• Book recommendation: Information Dashboard Design: The Effective Visual Communication of Data by Stephen Few

Transcript

Mike Julian: This is the Real World DevOps podcast. I'm your host, Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools, to the organizers of amazing conferences, from the authors of great books to fantastic public speakers, I want to introduce you to the most interesting people I can find.

Mike Julian: Ah, crash reporting, the oft forgotten about piece of a solid monitoring strategy. If you struggle to replicate bugs or elusive performance issues you're hearing about from your users, you should check out Raygun. Whether you're responsible for web or mobile applications, Raygun makes it pretty easy to find and diagnose problems in minutes instead of what you usually do which, if you're anything like me, is ask the nearest person, "Hey, is the app slow for you?" and getting a blank stare back because, "Hey, this is Starbucks, and who's the weird guy asking questions about mobile app performance?" Anyways, it's Raygun. My personal thanks to them for helping to make this podcast possible. You can check out their free trial today by going to raygun.com.

Mike Julian: Hi folks. I'm Mike Julian, your host for the Real World DevOps podcast. My guest this week is Cory Watson, Technical Director for the Office of the CTO at SignalFX. He's previously run observability at Stripe and Twitter. Cory, welcome to the show.

Cory Watson: Thanks for having me.

Mike Julian: I think it's interesting that you have gone from running observability teams at pretty interesting places like Stripe and Twitter.

Cory Watson: Thank you, thank you.

Mike Julian: To now, you're working for the enemy.

Cory Watson: That's a good way to put it.

Mike Julian: Yeah, you have suddenly gone to the vendor side.

Cory Watson: Yeah, the nicer way that I put it, is I say I switched sides of the table.

Mike Julian: Ah, yes. That is a much better way to put it. Apologies to my sponsors for insinuating that they're the enemy. You're completely right, it really is just the other side of the table.

Cory Watson: That implies that it's just a simple change in aspect, though. It's really not. It's actually a pretty fascinating difference, I think.

Mike Julian: Well, why don't you tell us more about that? What is that difference? What is it all about?

Cory Watson: I think, in the past, when you work at a place that uses these vendors, you're often trying to make sure you maintain, or at least it's always been my goal to maintain, a sort of neutrality or maybe not a shim layer is the right way to put it. But how do we retain our independence and make sure that we could switch, if necessary, and all these other things? It's both good, usually from a technical perspective, but also from a leverage perspective, right? Because you wanna be able to switch if you need to or go to the new, cool thing that might come out. Not that we jump and switch that frequently, but you wanna be able to retain that independence. Now, suddenly, I'm on the opposite side. I think there's two interesting bits about it. One is the change in perception or the change in the approach. The second is just the learning experience that I have from watching business being conducted from this side of the table. I guess we can start on the difference in perspective. To your earlier question, it's like, alright. Here I am previously sitting over here, going like, "Okay, don't tell them anything and pretend like you hate everything that they show you, and never show your true colors."

Mike Julian: Fantastic negotiating tactics.

Cory Watson: For lack of any better training, I think that it's the place that you try to go. But at the same time, it's actually somewhat different than I imagined, because I'm happy to work for a company that largely isn't trying to sell you something for the sake of selling it to you. I think this is true of pretty much all vendors, we only want you, as a customer, if you're ultimately going to be happy. Especially for the duration that many of these engagements, contracts, or purchase periods, or whatever last, you don't just want to ease in, make a buck, and then ease back out again. You've got to stay with it.

Cory Watson: In many ways, I think that my past experience of wanting to hold everything close to the vest, to use that idiom, don't really work that well because you need to give as much information as you can to the vendor so that the vendor can, hopefully, tailor the solution as well as possible, right? At the same time, I think it all comes down to price, though, at the end of the day. I think in that, luckily I don't have to have that conversation. I am only here to talk about the pros and cons and the approaches of how to do observability and how SignalFX can be helpful for you. I actually think that in switching sides, I've actually seen how it can actually hinder the process of adoption and understanding. Because, if you're like, "Well, we're not gonna tell you how many things we're monitoring, or how many metrics there are, or what our challenges are." It makes it extremely difficult to articulate a value proposition to the other people because suddenly I'm like, "If I can't help you size this, I don't know how to have the conversation." It's interesting to be on the side where you suddenly lack so much information because-

Mike Julian: I've run into that in the Amazon world, where people will say, "Oh, we're spending a ton of money with Amazon, but we don't want to tell them what our future plans are. We don't wanna tell them about our product roadmap because security reasons. What if they leak it, what if they try to do it themselves?" On some level, like it's AWS so maybe it will actually happen.

Cory Watson: Yeah, as we saw last week.

Mike Julian: Right. When you're spending that much money with a company, when a company is such a core aspect of how you do business, for example on monitoring product, it's no longer a vendor. They're a partner.

Cory Watson: Yeah, that's an excellent way to put it. I love that phrasing because I felt that way even as a customer. I've been a customer of many companies in this context, in the monitoring, observability, whatever context system stuff, and it's true. Once that spin gets to a certain amount or when it's such a critical part of your infrastructure, these systems...

Josh is a salary negotiation coach who helps experienced software developers negotiate their job offers and the author of Fearless Salary Negotiation: A step-by-step guide to getting paid what you're worth.

Links

• Twitter: @JoshDoody 
• Website: fearlesssalarynegotiation.com
• Read Josh’s book, Fearless Salary Negotiation
• Josh’s Salary Negotiation Coaching services
• A detailed article on how to answer the salary history and salary expectations questions: https://fearlesssalarynegotiation.com/the-dreaded-salary-question/
• Salary negotiation email (how to write a counter offer email): https://fearlesssalarynegotiation.com/salary-negotiation-email-sample/
• Salary increase letter (how to write an email asking for a raise): https://fearlesssalarynegotiation.com/salary-increase-letter-sample/

Transcript

Mike Julian: Running infrastructure at scale is hard. It's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: Alright folks, I've got a question. How do you know when your users are running into showstopping bugs? When they complain about you on Twitter? Maybe they're nice enough to open a support ticket? You know most people won't even bother telling your support about bugs. They'll just suffer through it all instead and God, don't even get me started about Twitter. Great teams are actually proactive about this. They have processes and tools in place to detect bugs in real time, well before they're frustrating all the customers. Teams from companies such as Twilio, Instacart and CircleCI rely on Rollbar for exactly this. Rollbar provides your entire team with a real-time feed of application errors and automatically collects all the relevant data presenting it to you in a nice and easy readable format. Just imagine — no more grappling logs and trying to piece together what happened. Rollbar even provides you with an exact stack trace, linked right into your code base. Any request parameters, browser operating system and affected users, so you can easily reproduce the issue all in one application. To sweeten the pot, Rollbar has a special offer for everyone. Visit rollbar.com/realworlddevops. Sign up and Rollbar will give you $100 to donate to an open source project of your choice through OpenCollective.com.

Mike Julian: Hi, folks. I'm Mike Julian, your host for the Real World DevOps podcast. My guest this week is Josh Doody, a good friend of mine and who is also a salary negotiation coach for engineers and author of the book, the Fearless Salary Negotiation. His articles have appeared in glassdoor.com and any number of other locations and he's amusingly also been interviewed live on the BBC, which was kind of cool when I found out about that. Josh, I'm just imagining BBC calling you and you're like, "Oh, shit. I should probably put some pants on."

Josh Doody: I already had pants on. I was at Starbucks and I started tweeting back and forth with whoever the producer was, and then within 45 minutes I was live on the air on international TV on the BBC being interviewed from my office. So that was interesting.

Mike Julian: Yeah, that's pretty cool. Yeah. I don't know too many people that say they've been interviewed by BBC live. So that's pretty awesome.

Josh Doody: That's a feather in my cap for sure.

Mike Julian: Absolutely.

Josh Doody: It was a pretty awesome, scary experience, and I'm really glad that it happened as quickly as it did because I think if I had time to think about the fact that that was going to happen I could've gotten nervous, but I didn't have time to get nervous. I literally was just scrambling to get some kind of lighting in my office and make sure my camera was working. So I spent 20 minutes on logistics and then I was on live international television being interviewed on BBC, and then it was over. Three minutes later it was done.

Mike Julian: That's a lot of work for such a short little time. So I want to talk to you today about salary negotiation, job negotiation, asking for raises, this whole gamut. But where I want to start is everyone's favorite topics. Everyone loves a good train wreck. So what's your favorite negotiation went totally sideways story?

Josh Doody: Well, I was thinking about this before we talked in case you asked about it, and I'm not sure I can tell you my favorite one because the only way I could tell it is if I censored so many pieces of it that it would be boring and meaningless to everyone. But the CliffsNotes for that one was it was a company that we have all heard of and use their services. They’re currently a private company. And just throughout the negotiation it was one red flag after another of my client trying to negotiate and then the company freaking out when he questions the value of their equity, which they're a private, so it's just monopoly money basically. And just one thing after another where the recruiter was just basically losing his mind, and eventually my client, even though this company had offered him more money, decided just to stay put. He was like, “You know what? I'm not going to go work there because if they're treating me like that right now I can't imagine what it's like to work there."

Josh Doody: But there's another one that can be a little bit less vague. It was a similar story, but this is a nice little way to kind of kill two birds with one stone because people like to ask, “Well, what happens if my job offer gets rescinded when I negotiate?” And my answer to that is I always tell my clients, I say ... They're like, "If we do what you're telling me to do are they going to rescind the offer? Are they're going to get mad at me?" I say, "Listen, I can't tell you that there's a 0% chance that will happen. It's greater than zero, but it's so small I can't even see it like on the graph." But it does happen occasionally. I think I've had I want to say two, maybe three clients in almost three and a half years that this has happened to.

Josh Doody: But I had a client who was going to work for an engineering firm. He was an experienced mechanical engineer. He got a pretty compelling offer from this company, but he knew that there was room to negotiate just based on market value research and some other kind of rudimentary stuff. And up to this point, there had been ...

Matty Stratton is a HumanOps Advocate at PagerDuty, where he helps dev and ops teams advance the practice of their craft and become more operationally mature. He collaborates with PagerDuty customers and industry thought leaders in the broader DevOps community, and back when he drove, his license plate actually said “DevOps”.

Matty has over 20 years experience in IT operations, ranging from large financial institutions such as JPMorganChase and internet firms, including Apartments.com. He is a sought-after speaker internationally, presenting at Agile, DevOps, and ITSM focused events, including ChefConf, DevOpsDays, Interop, PINK, and others worldwide. Matty is the founder and co-host of the popular Arrested DevOps podcast, as well as a global organizer of the DevOpsDays set of conferences.

He lives in San Francisco and has three awesome kids, who he loves just a little bit more than he loves Doctor Who. He is currently on a mission to discover the best pho in the world.

Links

• Twitter account: @mattstratton
• Website: mattstratton.com
• Arrested DevOps podcast
• Bryan Berry’s article: You Should Start a Technical Podcast

Transcript

Mike Julian: Running infrastructure at scale is hard. It's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: Alright folks, I've got a question. How do you know when your users are running into showstopping bugs? When they complain about you on Twitter? Maybe they're nice enough to open a support ticket? You know most people won't even bother telling your support about bugs. They'll just suffer through it all instead and God, don't even get me started about Twitter. Great teams are actually proactive about this. They have processes and tools in place to detect bugs in real time, well before they're frustrating all the customers. Teams from companies such as Twilio, Instacart and CircleCI rely on Rollbar for exactly this. Rollbar provides your entire team with a real-time feed of application errors and automatically collects all the relevant data presenting it to you in a nice and easy readable format. Just imagine — no more grappling logs and trying to piece together what happened. Rollbar even provides you with an exact stack trace, linked right into your code base. Any request parameters, browser operating system and affected users, so you can easily reproduce the issue all in one application. To sweeten the pot, Rollbar has a special offer for everyone. Visit rollbar.com/realworlddevops. Sign up and Rollbar will give you $100 to donate to an open source project of your choice through OpenCollective.com.

Mike Julian: Hi folks, I'm Mike Julian, your host for the Real World DevOps podcast. I'm here speaking to Matty Stratton today, DevOps advocate for PagerDuty and the host of my arch nemesis, the Arrested DevOps podcast. Welcome to the show.

Matty Stratton: Thanks Mike, I'm really excited to be here. It's always fun to be a guest on somebody else's show because you do a lot less work.

Mike Julian: Ain't that the truth? I've been watching or listening, I guess as it is, the Arrested DevOps podcast for, God since it began; it feels like a million years ago at this point.

Matty Stratton: It's over five years. December 2013 is when we started. I should know off the top of my head how many episodes we have but I don't. In the hundreds.

Mike Julian: Somewhere between five and 50,000.

Matty Stratton: Right. It's a non-zero number.

Mike Julian: Something that we were talking about before we started recording, was this idea of, there's a lot of people that shift from doing ops work into I guess you could call it an influencer role, where we're talking about the work and helping other people become better. It seems like that's what you've done five plus years ago with the Arrested DevOps as well as what you're doing now with developer advocacy or DevOps advocacy, whatever you want to call it. Is that about right?

Matty Stratton: I would say so. I have found that when I try to explain to people what I do for a living now, I say I worked in operations for 20 years and now they pay me to talk about it. But I didn't start out that way. The journey for Arrested DevOps was, it never started out to be a podcast. I didn't know what it was going to be, but when I was starting to learn about DevOps and started to do some DevOps transformation at an organization I was at, I was taking in a lot of the podcasts of the time, DevOps Café and the Ship Show and Food Fight. The one thing that I found was there was a lot of material out there that was related to DevOps but there wasn't a lot of stuff that seemed suited to people like me who were kind of dumb, for a better word. Just maybe a better way to say that is people who are very new to it. I actually originally intended to start a blog because I had done a lot of blogging, been blogging for years and years and years. I wanted to create a blog that was going to be very accessible. I don't remember if it was on Facebook or Twitter or somewhere when I was looking for ideas for what to call this blog and my friend Jessica, who is not in the tech industry, she's a writer. She's the one who coined the name,” Arrested DevOps.” I bought the domain but I didn't have anything to do with it for a while. I decided maybe I'll start a podcast. One thing I know about myself is I'm very good at starting things and very bad at doing them.

Mike Julian: Aren't we all?

Matty Stratton: Yeah. One of the things that helps with that is having a partner or having somebody else that keeps you accountable.

Mike Julian: Oh yeah.

Matty Stratton: I had met this cat named Trevor at an Azure meetup in Chicago and was like, he's a software engineer type and into this DevOps stuff and was like, "Hey Trevor, maybe we should do a podcast." That was back in November of 2013 — we did our first episode I think in December of ‘13. But what's interesting is how this, it never was intended to be in an influencer role. What I wanted to do was be able to take things that I was learning and make things accessible. One thing I learned very quickly with doing the show and Mike, I mentioned you've learned this lesson similarly with your newsletter and the show now, you can't control your audience.

Mike Julian: No, not at all.

Matty Stratton: Even in our very first episode, we had people who were tweeting at us because we used to live str...

Dan spent 12 years in the military as a fighter jet mechanic before transitioning to a career in technology as a Software/DevOps Engineer/Manager. He's now the Chief Architect at the National Association of Insurance Commissioners. He's leading the technical and cultural transformation for the NAIC, a non-profit focused on consumer protection in the insurance industry. Dan is also an organizer of the DevOps KC Meetup and the DevOpsDays KC conference.

Links Referenced: 

• Insure U: created by the NAIC for consumer insurance education 
• Dan’s talk at DevOps Enterprise Summit 2018 
• State Ahead Strategic Plan from NAIC
• Explore technology proposals, fiscal budget proposals, etc. on the NAIC site 
• Dan’s website

Transcript

Mike Julian: Running infrastructure at scale is hard. It's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: Alright folks, I've got a question. How do you know when your users are running into showstopping bugs? When they complain about you on Twitter? Maybe they're nice enough to open a support ticket? You know most people won't even bother telling your support about bugs. They'll just suffer through it all instead and God, don't even get me started about Twitter. Great teams are actually proactive about this. They have processes and tools in place to detect bugs in real time, well before they're frustrating all the customers. Teams from companies such as Twilio, Instacart and CircleCI rely on Rollbar for exactly this. Rollbar provides your entire team with a real-time feed of application errors and automatically collects all the relevant data presenting it to you in a nice and easy readable format. Just imagine — no more grappling logs and trying to piece together what happened. Rollbar even provides you with an exact stack trace, linked right into your code base. Any request parameters, browser operating system and affected users, so you can easily reproduce the issue all in one application. To sweeten the pot, Rollbar has a special offer for everyone. Visit rollbar.com/realworlddevops. Sign up and Rollbar will give you $100 to donate to an open source project of your choice through OpenCollective.com.

Mike Julian: Hi folks. I'm Mike Julian, your host for the Real World DevOps podcast. I'm here with Dan Barker and the chief architect for the National Association of Insurance Commissioners. Welcome to the show Dan.

Dan Barker: Hi Mike, it's great to be here.

Mike Julian: So National Association of Insurance Commissioners, it's like the four least interesting words in one title I've ever heard. What in the world do you do?

Dan Barker: We thought if we combine them that it would be more interesting, that may not have had the intended effect. So the NAIC is a nonprofit, about 150 years old. We kind of got our start organizing events for insurance regulators, so primarily the chief insurance regulators for each state and territory and we organize events to get them together. And we also created model law. Over time, as technology advanced, we started to host some centralized technologies within the NAIC and providing a lot of the back-end applications that regulators use. We also offer something called Insure U, where you can go and learn about insurance, all kinds of insurance, I'm sure that it will get flooded now.

Mike Julian: Because we're all chomping at the bit about insurance.

Dan Barker: Yeah. And so we still do a lot of the event planning, taking regulators and kind of informing them on technology. There's a big movement in Insure-Tech with tons of investment right now. And so we're trying to make sure everyone is staying up to date with things like blockchain, AI, a lot of the data-focused stuff particularly around unconscious bias and the data and how to clean that out. So we're a pretty diverse group and we have a lot of kind of different focuses but one of those is the technology side and I'm the chief architect and leading up a lot of the technological transformation as we move to Amazon Web Services, moving everything to the cloud and moving to some more open source tools and trying to move towards a DevOps culture.

Mike Julian: That sounds like a pretty fascinating situation.

Dan Barker: Yeah, it's really exciting. We have all kinds of new tools and new things. We're moving towards ... This company has done a pretty good job of standard blocking and tackling actually, what you might be surprised at in a nonprofit.

Mike Julian: Right.

Dan Barker: One of the only companies I've ever even heard of that had one version of Java and that has been-

Mike Julian: That's pretty impressive.

Dan Barker: Yes, I continue to ask in every meeting if that's true, and-

Mike Julian: Are you sure?

Dan Barker: Yeah. I'm sure there's one somewhere around here. But yeah, for all of our applications, they're all on the same version of Java and it's up to date to you know, like Java 4.

Mike Julian That's pretty impressive, congratulations.

Dan Barker: Yeah, so it's a great base to start on. And it's been a great journey so far. I've been here for a year and the CTO that kind of came in to start this off has been here for, I think about three years. So it's about the length of the transformation so far.

Mike Julian: Okay. So we're talking about transformation here. You actually gave a talk about this at DevOps Enterprise Summit in Vegas earlier this year, or 2018. What was the problem that started this entire transformation process? What were you trying to solve?

Dan Barker: So, we had several different opportunities that we were looking to kind of utilize moving forward. So we have this infrastructure that has been ... We're a nonprofit, so we don't have a ton of funding and it's been a bit challenging. So we haven't been able to move as efficiently because we haven't optimized a lot of our technological systems, much of what we have has been more by request, kind of an IT department within a company, non-technical company. And so we're trying to move towards being more of a technology company which requires a little bit more proactive planning. So one of those areas we're trying to gain some efficiencies, gain some efficiencies across all the different teams. So trying to standardize a lot of our [inaudible 00:06:30] me...

After many years of ghostwriting, Emily Freeman made the bold (insane?!) choice to switch careers into software engineering. Emily is the author of DevOps for Dummies (April 2019) and the curator of JavaScript January — a collection of JavaScript articles which attracts 30,000 visitors in the month of January. A former VP of Developer Relations, Emily is a CloudOps Advocate at Microsoft and lives in Denver, Colorado.

Links

• Website: emilyfreeman.io
• Twitter:  @editingemily

Links Referenced

• Book recommendations: 
  • Monitoring with Graphite by Jason Dixon
  • Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale by Jennifer Davis and Ryn Daniels
• Emily’s talks: 
  • Humpty Dumpty + DevOps
  • Scaling Sparta
  • Dr. Seuss Guide to Code Craftsmanship

Transcript

Mike Julian: Running infrastructure at scale is hard. It's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: All right folks, I've got a question. How do you know when your users are running into showstopping bugs? When they complain about you on Twitter? Maybe they're nice enough to open a support ticket? You know most people won't even bother telling your support about bugs. They'll just suffer through it all instead and God, don't even get me started about Twitter. Great teams are actually proactive about this. They have processes and tools in place to detect bugs in real time, well before they're frustrating all the customers. Teams from companies such as Twilio, Instacart and CircleCI rely on Rollbar for exactly this. Rollbar provides your entire team with a real-time feed of application errors and automatically collects all the relevant data presenting it to you in a nice and easy readable format. Just imagine — no more grappling logs and trying to piece together what happened. Rollbar even provides you with an exact stack trace, linked right into your code base. Any request parameters, browser operating system and affected users, so you can easily reproduce the issue all in one application. To sweeten the pot, Rollbar has a special offer for everyone. Visit rollbar.com/realworlddevops. Sign up and Rollbar will give you $100 to donate to an open source project of your choice through OpenCollective.com.

Mike Julian: Now folks, welcome to the Real World DevOps podcast. I'm here with Emily Freeman, Cloud Advocate at Microsoft. Welcome to the show, Emily.

Emily Freeman: Thank you, thank you so much for having me.

Mike Julian: Yeah, so for the folks who aren't as familiar with developer advocacy or developer relations or developer evangelism or the 50 bajillion other names that they go by, can you tell us what is that you do? What are you doing for Microsoft?

Emily Freeman: Yeah, absolutely. Along with the 50,000 names for developer relations, there are also 50,000 interpretations of the actual job. It will vary company to company and person to person. For me in Microsoft I like to say that, because I have a background in writing and I was a writer before I was a developer, I like to tell stories. My job is to tell the stories of the developers in the communities I am in and serve and represent. The most important role I think I play at Microsoft is bringing customer feedback back to the product teams. When I go to a conference, I am never one to just like helicopter in, give my talk and then peace out. I think that's just poor form and it doesn't benefit anyone at that point but my ego. My job really is to go and talk to people and to hear all their feedback, positive and negative. It's not personal, it's about making the product better for the community.

Mike Julian: That sounds like a much more reasonable job than you would hear from like reading shit shows on Twitter.

Emily Freeman: Yes, absolutely.

Mike Julian: That might be a good segue to the recent total shit storm that happened on Twitter about developer relations. I don't even know how these things started. It just suddenly appeared one day.

Emily Freeman: Yeah, and the criticisms from this last one aren't new. This is just one iteration of the sort of dev overall hate, which is like, on a very personal point, it's hurtful because it's like, “Okay this is my work, this is what I care about. I'm passionate about this and obviously I dedicate all of the time I spend at work to this role.” To hear that it just has no value or that we are simply low-key Twitter celebrities that our company shields and drink champagne on planes and party.

Mike Julian: That sounds like a fantastic life, can I get that job?

Emily Freeman: I know. I was telling someone I'm going to Milan next week for Microsoft Ignite | The Tour. It's very exciting, I've never been in Italy, I'm thrilled. I was just telling them like, "I sound a lot cooler than I am. I just got from Toronto and I'm going to Milan." No, I'm just on the plane sad. Yeah, the travel is part of it, I think it's the part that people cling on to. I also think that we as an industry have not done a great job recently at actually representing what the work is. Yeah, I think so much of the emphasis has been around Twitter and a lot of the developer relations folks especially the people Microsoft is picking up, have really — if not enormous Twitter following — substantial ones.

Mike Julian: Well they are already super well-known people to begin with in their own right.

Emily Freeman: Yeah, but there's a cult of personality thing happening. I think when we have that and when we focus so much on the persona and that sort of Twitter platform, then we lose the actual benefits of developer relations, both for the community and the company that they work for.

Mike Julian: Yeah, so some of the criticisms that I saw, you alluded to this one, that you are the representative of a company, but when you're speaking on behalf of the company, you're not saying that you are. Therefore, it's like influencer stuff, but you're not saying that you are. You're not divulging where you're getting your pay from. To me this sounds like a dubious argument, because your employer is right there on your profile.

Emily Freem...

Greg established and leads the Enterprise Monitoring Services team at Standard Chartered Bank, and together with his team wrote and implemented a strategy and approach to effectively monitor and leverage data from over 1,000 applications, 30,000 servers, 15,000 network devices, public and private cloud, mainframe, tandem, and multiple other technologies in a sustainable and scalable way. Applying Agile and DevOps techniques to the build, engineering, and support of the monitoring ecosystem at Standard Chartered, the team brought together tools across the technology stack and advocated techniques such as monitoring as code in order to improve monitoring quality and make it a mandatory part of the deployment pipeline.

Prior to that he worked at Barclays Capital in Singapore and Goldman Sachs in Tokyo, Japan in various infrastructure and engineering roles.

Links Referenced: 

• Connect with Greg on LinkedIn

Transcript

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools — and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, coronagraph for visualization and capacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike Julian: Hi folks. Welcome to the Real World DevOps podcast. I'm here with Greg Parker, head of enterprise monitoring services at Standard Chartered Bank, way out in Singapore. Welcome to the show Greg.

Greg Parker: Thanks,  Mike. I'm doing well. How are you doing?

Mike Julian: I'm doing fantastic. So Standard Chartered Bank, like what is this? It sounds like just a bank, but I've been talking to you about it and it sounds like it's a whole lot bigger than I imagined.

Greg Parker: Well, Standard Chartered operates across 70 countries. There's more than 1200 branches, there's 90,000 employees, and it's just a sprawling financial institution, but it's primarily operating in Africa, Middle East, a lot of emerging markets, and the headquarters for IT is in Singapore, though the bank is headquartered in London. And so out of Singapore we drive the technology strategy and across all of the markets over 70 countries. And we get a lot of diversity in our environment because of the different strategies that we have in each country. Coming from, I was working for Goldman Sachs for about ten years, where IT was very tightly controlled from the center from New York where, the word came down from the heavens around this is how you're going to do everything. And then I went to Barclays, which was a similar model except the word came down from London, and at Standard Chartered it was really Singapore saying, this is what we should be doing and this is how we operated for our group owned applications, but there were 70 other countries saying, this is how we have to do it in Nigeria, and this is how we have to do it Kenya, this is how you have to do it in Pakistan. And so you have all of those issues creep up when you're working across emerging markets and especially in a financial.

Mike Julian: What's your role in Standard Chartered?

Greg Parker: So my role at Standard Chartered is to run enterprise monitoring. And, it wasn't my original role. I came in to drive some infrastructure projects, large infrastructure projects, and when I got there, I saw that monitoring was essentially chaos. There was really no central strategy around how we're going to do it. And when I worked with some people there and we effectively established a central enterprise monitoring organization for Standard Chartered, the problem was there was no central strategy or tool set, or group of tools that we were using for monitoring, and there were multiple vendor deals, negotiated at different prices at different times with different countries. And so there's a lot of inefficiencies that were contributing to massive, MTTRs. Which meant, when an issue occurred, a thousand different teams got an alert, nobody know whose fault it was, and it took all this time to work out, what's the root cause and how are we going to resolve it? And I think a lot of that comes down to, the fact monitoring wasn't precise.

Mike Julian: And I'm sure in no small part do two countries not be able to talk to each other.

Greg Parker: Certain countries couldn't talk to each other and other countries just didn't know to talk to us. And so there was a lot of people working in silos.

Mike Julian: How does your strategy even look when you have all these different entities that are doing their own thing, and like culturally you're not able to say, “This is what we're going to do.” So what's your approach instead?

Greg Parker: Well, we do have the authority to dictate if we want to. And that's one of the things that came along with establishing this central organization which is backed by the CTO and the head of technology services, which is going to say, our mandate is to go out and fix monitoring for SCB. But at the same time it's not something that we want to do, is to just give people a mandate. And I've been saying this the whole time is that our strategy is not perfect and it's never going to be perfect. Our strategy is focused around corralling all the data that's out there, and translating it, and enriching it, and normalizing it, and then exposing it through APIs. And that's really the crux of it. But, it's never gonna be perfect, and our focus was to just implement a working framework, and help to improve monitoring so that we can reduce those mean time to resolve and mean time to detect times, and just give generally a better sense of observability for the company. So we have a sense like that we know what's going on. 

Mike Julian: Why don't we talk more about the strategy behind what you're doing. Like what does it actually look like? How did you come to this strategy? How are you implementing it? What is it even?

Greg Parker: So we started with multiple teams that had implemented their own monitoring with the tools that they wanted to work with. We have BMC, deployed across all of the infrastructure. A lot of the application teams had purchased ITRS. There were other tools like AppDynamics and Dynatrace and some open source tools out there with Grafana and Elastic and all of that. And so my first thought was, we're not going to standardize everybody to one tool, and there's not one tool that's going to, be this pan...

Yan is an experienced engineer who has run production workload at scale in AWS for nearly 10 years. He has been an architect and principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. He has worked extensively with AWS Lambda in production, and has been helping various UK clients adopt AWS and serverless as an independent consultant.

He is an AWS serverless Hero and a regular speaker at user groups and conferences internationally, and he is also the author of Production-Ready serverless.

Guest Links

• Yan’s blog
• Yan’s video course: Production Ready Serverless
• Find Yan on Twitter (@theburningmonk)
• Subscribe to Yan’s newsletter 
• Centralised logging for AWS Lambda

Transcript

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools — and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, coronagraph for visualization and capacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike Julian: Hi folks, I'm here with Yan Cui, an independent consultant who helps companies adopt serverless technologies. Welcome to the show, Yan.

Yan Cui: Hi Mike, it's good to be here.

Mike Julian: So tell me what do you do? You're and independent consultant helping companies with serverless. What does that mean?

Yan Cui: So I actually started using serverless quite a few years back, pretty much as soon as AWS announced it, I started playing around with it and the last couple of years I've done quite a lot of work building serverless applications and production. And I've also been really active in just writing about things I've learned along the way, so as part of that, a lot of people have been asking me questions because they saw my blog and talk about some problems that they've been struggling with, and asked me, "Hey can you come help me with this? I got some questions." So as part of the doing that, I like to help people, first of all and then just part of doing that is something that's been happening more and more often, so in the last couple months I have started to work as an independent consultant, helping companies who are looking at docking serverless or maybe moving to serverless for new projects and want to have some guidance in terms of things they should be thinking about and maybe have some architectural reviews on a regular basis. So for things like that, I've been helping with a number of companies, both in terms of workshops but also regular architectural reviews. And at the same time, I also work part-time at a company called The Zone, which is a sports streaming platform and we also use the serverless and is contained very heavily there as well.

Mike Julian: Okay, so why don't we back up like several steps. What the hell is serverless? Just to make sure that we're all talking about the same thing. What are we talking about?

Yan Cui: Yeah that's a good question, and I guess a lot of people has been asking the same question as well because now they say you see, pretty much everyone is throwing the serverless label at their product and services. And just going by popular definition out there based on what I see in the talks and blog posts, I guess in terms of my social media circle, I guess by the most popular definition, serverless is pretty much any technology where you don't pay for it when you are not using it because paying for OpTime is a very serverful way of thinking and planning, and two is, you don't have to worry about managing and patching servers because installing demons or Asians or any form of subsidiary or support software on it is again, definitely tied to having servers that you have to manage. And three, you don't have to worry about scaling and positioning because the systems just scale a number of underlying servers on demand. And by this definition, I think a lot of the traditional backend server's things out there like AWS S3 or Google BigQuery, they also qualify as the serverless as well.

Mike Julian: Okay, so Lambda is a good example of serverless, but there's also this thing of like a function as a service and they seem to be used interchangeably sometimes. What's going on there?

Yan Cui: So to me, functions as services, describes a change in terms of how we structure our applications and changing the unit of deployment and scaling to the function level that makes every application. A lot of the function and server solutions like a dual function or Lambda as you mentioned, they will also qualify as serverless, based on the definition we just talked about and generally I find that there are a lot overlap between the two concepts or paradigms between functions and service and the serverless. But I think there are some important subtleties in how they differ because you also have functions of service solutions like Kubeless or Knative that gives you the function oriented programming model and the reactive and event driven module for building applications, but then runs on your own Kubernetes cluster.

Yan Cui: So if you have to manage and run your own Kubernetes cluster, then you do have to worry about scaling, and you do have to worry about patching servers, and you do have to worry about paying for op time for those servers, even when no one is running stuff on them. So the line is blurred when you consider Kubernetes as service things like Amazon’s EKS or Google GKE where they offer Kubernetes as a service or Amazon's Fargate, which lets you run containers on Amazon's fleet of machines so you don't have to worry about positioning, and managing, and scaling servers yourself.

Yan Cui: At the end of the day, I think being serverless or having the right labels associated with your product is not important. It's all about delivering on business needs quickly, but having a well understood definition on those different ideas that we have, really helps us in terms of understanding the implicit assumptions we make when we talk about something. So now that everyone is talking about calling their services or prod...

Elliot Murphy is a senior executive and technologist with more than 20 years of success in critical software infrastructure, online services, and healthcare. Elliot is the founder of Kindly Ops, a cybersecurity firm bringing DevOps approaches to Governance Risk & Compliance, serving regulated industries such as biotech and fintech. His interest in Governance Risk & Compliance began when working as CTO of a healthcare startup and realizing the burden of regulatory compliance was slowing life-saving healthcare innovations from being brought to market.

Links Referenced

• Book recommendation: People-Centric Security by Lance Hayden
• Book recommendation: How to Measure Anything in Cybersecurity Risk by Douglas Hubbard and Richard Seiersen
• Book recommendation:How to Measure Anything by Douglas Hubbard

Transcript

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, Influx DB. But, you may not be as familiar with their other tools: Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all this out at influxdata.com.

Mike Julian: Hi, everyone, my name's Mike Julian. I'm here with Elliot Murphy, the CEO of KindlyOps. Welcome to the show, Elliot.

Elliot Murphy: Hey, nice to be here.

Mike Julian: Why don't you tell us a bit about what you and KindlyOps does?

Elliot Murphy: Yeah, so we're a cybersecurity consulting firm, and we mostly help regulated companies, biotechs, and fintechs with governance, risk, and compliance. We take a DevOps approach, so all of the good practices from that world, and apply them into what's traditionally been a pretty boring and bureaucratic set of activities.

Mike Julian: Okay, so what are these regulations we're talking about?

Elliot Murphy: Some people have heard a thousand times before, HIPAA, PCI, SOC 2. Others are a little newer, GDPR, or more specialized like FISMA. There's regulations from the Food and Drug Administration around medications and therapeutics. Pretty much every country in the world and every industry has their own set of regulations or laws. As software technology expands ever further into different parts of business, even very old businesses that didn't have much technology in the past, there's regulations that people have to comply with.

Mike Julian: Okay, so that makes a lot of sense, but every time I've been in a company with these regulatory requirements, it's really just been a huge pain in the ass. I don't like dealing with it. No one I worked with likes dealing with it. We meet the bare minimum and hope for the best. How do you view these regulations? Do you have that same take on them?

Elliot Murphy: It's funny, a few years ago, I was working as the CTO of a healthcare startup founded by a doctor. She had just some amazing innovations, and I got to see firsthand how expensive and burdensome regulatory compliance was. Then, even more upsetting, as I was meeting people, scientists and researchers at universities, I was hearing all these stories of how they were discouraged or prevented or didn't want to bother bringing actual life-saving innovations to market because of the burden of regulatory compliance. So I'm right there with you that by default they can have a pretty high cost. Our entire mission has been to try and get the value out of those regulations but without the human toil. It's a pretty existential question, why even have rules and laws?

Mike Julian: Right.

Elliot Murphy: I think some rules and laws are good, but they don't have to be as 

miserable as they have been. We really take a human-centric approach.

Mike Julian: Okay. What does that even mean, a human-centric approach to regulation? Regulations are, to me, they're handed down from on high. It's like, "You comply or face the consequences."

Elliot Murphy: Yeah, so it's funny, we've actually developed this four-step process, and it's very, very common when you're talking with people who are excited about building products and are excited about serving patients and customers and are just excited about doing their core job, they're usually pretty annoyed at externally imposed rules that aren't quite obvious how they're beneficial. Depending on people's life experiences, they have these different perspectives. Some people, particularly, if they've been doing risk and compliance work for their whole career, they see the value, and they get frustrated that other people push back on regulations. What does it actually mean to put people first? We start with building empathy for different worldviews. One definition of culture is that it's beliefs and assumptions that drive decisions and behavior. That makes sense. But, then, if you think about that again, what is another word for beliefs and assumptions that you use to make decisions, drive behavior? Those are mental models. That's a very freeing realization that, okay, what we're actually talking about when we say culture, security culture, for example, are mental models. You and I are going to have our favorite mental models, our default mental models. They're mental models we go back to immediately when we're in a high-stress situation. But, we're perfectly capable of learning other worldviews, other mental models and deciding when it's beneficial to apply them. We can try them on for size, and we can say, "Oh, yeah, I can see. Even though this is not my favorite way to think about the problem, I can see there's some utility in this case or that case."

Mike Julian: Yeah.

Elliot Murphy: Dr. Lance Hayden actually developed this fantastic Creative Commons licensed security culture diagnostic. It's called, in his book, People-Centric Security, and so that outlines four different cultures or mental models. There's a trust culture, an autonomy culture, a compliance culture,...

Dr. Sherry Walling is a clinical psychologist, entrepreneur, international speaker, yoga teacher, podcaster and best-selling author. She works with leaders and entrepreneurs around the world to help them tackle the many mental health and relationship challenges that go along with building a great business.

Her best-selling book, The Entrepreneur’s Guide to Keeping Your Sh*t Together, is a handbook for navigating life as an entrepreneur.

Married to a serial entrepreneur, Sherry combines her extensive experience helping people who have high-intensity jobs with her 18 years of personal experience in the trenches of the startup world. Sherry combines the insight and warmth of a therapist with the truth-telling mirth of someone who has been there.

When she’s not in the consulting room or hopping conferences; Sherry can be found on her paddleboard, in the yoga studio, or ushering her three kiddos through an art museum in some fabulous city.

She can also be found at ZenFounder.com, SherryWalling.com or on Twitter as @ZenFounder.

Links
Managing Founder Stress Guide

“How to Stay at the Top of Your Game” talk at BoS USA 2017 

Links Referenced: 

• Book recommendation: Play it Away by Charlie Hoehn
• Book recommendation: The Happiness Advantage by Shawn Achor 
• Sherry’s “How to Stay at the Top of Your Game” talk at BoS USA 2017 
• Managing Founder Stress Guide

Transcript:

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools — and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, coronagraph for visualization and capacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike Julian: Hi Folks, I'm Mike Julian, your hosts for the Real World DevOps podcast. I'm here with Dr. Sherry Walling, clinical psychologist, author and fellow podcaster at ZenFounder.com. Welcome to the show, Sherry.

Sherry Walling: Hey, it's my pleasure Mike. After a couple of reschedules, I'm glad we finally got this to come together.

Mike Julian: It was a bit of work to make that happen, but I'm really excited about this episode. So Sherry, why don't you start off by just telling us a bit about who you are and what you do.

Sherry Walling: Yeah. So like you said, I'm a clinical psychologist and I have spent my professional life working with people who have high intensity jobs. That looks like a couple of different things over the course of my career. Sometimes it's with folks in the military returning from military service. I've worked a lot with physicians, who have high intensity work either in the ER or in surgery. Then I actually work a lot with software entrepreneurs and software folks. My connection to the developer world is largely through my husband, Rob Walling, who maybe some of your audience know of MicroConf and things like that. So I work with really smart people who are trying to do really hard things, and often have a lot of pressure in their work lives.

Mike Julian: Yeah, that's some awesome stuff. The reason I'm especially interested in talking to you is as DevOps engineers, we lead extraordinarily stressful lives. A day in the life of a operations engineer… we’re project driven, and yet it's often interrupt heavy. So we're never really finishing anything, thanks to putting out fires constantly. Just so many fires is all the time, everything is always awful, everything's just a tire fire everywhere we look. So we're responsible for keeping systems of multimillion dollar, sometimes multi billion dollar companies running and available, and on call is a standard part of the job. And sometimes this is even this is really bad, like the on call rotations of some companies might be 15 pages a night for a week or two weeks at a time, and it just gets insane. This is not just a few years or one job, this is like an entire career. So to say that the role of an Ops engineer is stressed, is kind of an understatement. So I'm really excited to talk to you about how can we manage the stress? Is there something we can do about it? How can we improve our lives?

Sherry Walling: Well, I'm really empathizing with the way that you're describing this, and there's a couple of things that you said that would lead me to believe that folks who are doing this kind of work are really at a pretty high risk for burnout. Whenever there's high work demand, so lots of things to do without having appropriately accomplishable goals, goals that have a tight timeline and clear successes — those three things tend to be a combination that really causes burnout in a lot of folks. So I can empathize with the amount of stress that folks are feeling, and certainly guessing that some of your listeners are really struggling at this point with stress related difficulties, or things like burnout.

Mike Julian: Yeah, absolutely. So why don't we talk, why don't we just start with burnout? There's so many different ways that I want to tackle this conversation, but why don't we just start with burnout? What is burnout?

Sherry Walling: Yeah, so burnout is a newly recognized clinical syndrome. It has its own diagnostic code. It is a real thing.

Mike Julian: That's fantastic. It's about time.

Sherry Walling: It is about time. It exists in ICD 10, which is the International Classification of Disorders 10. It's called something like “burnout estate of vital exhaustion,” is the technical title which I think is really lovely language that paints the picture of what this feels like. Vital exhaustion. So burnout is something that was ...

Dave Mangot is the author of Mastering DevOps from Packt Publishing. He’s formerly the head of Site Reliability Engineering (SRE) for the SolarWinds Cloud companies and an accomplished systems engineer with over 20 years' experience. He has held positions in various organizations, from small startups to multinational corporations such as Cable & Wireless and Salesforce, from systems administrator to architect. He has led transformations at multiple companies in operational maturity and in a deeper adherence to DevOps thinking. He enjoys time spent as a mentor, speaker, and student to so many talented members of the community.

Guest Links

• Twitter: @davemangot
• Mastering DevOps course by Dave Mangot

Links Referenced: 

• Familiar Smells I’ve Detected in Your Ops Organization and How to Fix Them by Dave Mangot (LISA 2018)
• Knight Capital story
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation book by Jez Humble and David Farley
• Book recommendation: How to Measure Anything by Douglas W. Hubbard
• Calvin and Hobbes comic on Bridge Capacity
• Dan McKinley - Choose Boring Technology
• Nicole Forsgren’s DevOps research
• Book recommendation: The Lean Startup by Eric Ries

Transcript

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, coronagraph for visualization and capacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike Julian: Hey folks, this is Mike Julian, I'm here with Dave Mangot, former head of SRE for SolarWinds Cloud. Welcome to the show, Dave.

Dave Mangot: Thanks Mike. It's great to be here.

Mike Julian: So why don't you tell us a bit about yourself and what you've been up to lately.

Dave Mangot: Sure. Um, I've recently, like you said, left SolarWinds Cloud. There I was running the global SRE organization. We started that with about two people at Librato and wind up growing that into multiple teams in multiple locations with certainly lots of products you've heard of. Before that, I was an architect and technical operations for Salesforce, working on their internal infrastructure, monitoring, automation, configuration management, all kinds of fun stuff like that.

Dave Mangot: I've been able to take a bunch of the experience of those two companies plus lots of years of experience doing stuff before that into a pretty fun conference talk that I gave a few weeks ago in Nashville at the USENIX LISA ‘18 Conference. And I was talking about familiar smells I've detected in your systems engineering organization and how to fix them. And so it was really fun to be able to use a lot of the things that I've seen in my career and all the things I've seen as we try to mature engineering organizations, both operationally and all kinds of other fun DevOps ways, and be able to give a talk about that. And pretty, I guess you could say, controversial at times, but well accepted at other times sort of venue.

Mike Julian: It's always interesting when you start pointing out the smells and people's infrastructure, and their reactions are kind of all over the place. Like some people like, “Yes, you're absolutely right.” And other times it's like, “No, no, please don't say that. Like, I'm not cool with you pointing out my flaws.”

Dave Mangot: Yeah, I think the important thing is, you know, what I was trying to get across in the talk as a message is like nobody's perfect, we all have to get better. If I've seen those things in other organizations, there's a chance that I was involved in those things. It's not like I just kind of wandered in from Mars and I knew everything and everything was all good, like I've been growing throughout my career, and I try to continue to keep growing throughout my career. And if people feel a little bit uncomfortable, I think that's good, right? That's a sign of growth, if we're feeling uncomfortable. It's just, what do we do with that? If we get defensive, then we could double down on something that maybe we know in deep inside isn't really working for us as opposed to maybe we go home and sleep on it and wake up the next day and say like, "Well maybe there's a couple of things here that we could actually get some incremental improvement on." And that's ultimately how we make all this successful changes to our infrastructure. 

Mike Julian: I've had people come find me where they've taken my place in their previous job, and they'll be like, "Why did you do this really dumb thing?" And my answer is, "Yeah, it was really dumb. I'm sorry about that." Like that that was a technical smell that I created, and I'm really sorry.

Dave Mangot: Yeah.

Mike Julian: These are things that we did, like they're not necessarily ... we didn't do them because they're good ideas, we did them for all sorts of reasons.

Dave Mangot: Yeah. And also I think that's important with like John [Allspaw 00:04:53] things that it's easy to come back later and say, why did you do that? That was a bad idea. But we make the best decisions that we can with the information that we have at the time.

Mike Julian: Absolutely.

Dave Mangot: It turns out that was a bad idea.

Mike Julian: Yup. So this talk that you gave at LISA, sadly I couldn't make it to LISA to see it live, but you and I have been talking...

Corey is a Cloud Economist at the Quinn Advisory Group. He has a history as an engineering director, public speaker, and cloud architect. Corey specializes in helping companies address horrifying AWS bills, hosts the Screaming in the Cloud and curates LastWeekinAWS.com, a weekly newsletter summarizing the latest in AWS news, blogs, and tips, sprinkled with snark.

• Corey’s newsletter: Last Week in AWS
• Corey’s professional site: quinnadvisory.com
• Corey’s podcast, Screaming In the Cloud
• Corey’s Twitter

Links Referenced: 

• The Story of a Serverless Startup by Sam Kroonenburg
• Corey’s talk, Myth of Cloud Agnosticism
• Dan McKinley (@mcfunley)’s Choose Boring Technology

Transcript

Mike: Running infrastructure at scale is hard, it's messy, and it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're gonna talk about the rough edges. We're gonna talk about what it's really like running infrastructure at scale.

Mike: Welcome to the Real World DevOps podcast. I'm your host, Mike Jillian, editor and analyst for Monitoring Weekly, and author of O'Reilly’s Practical Monitoring.

Mike: This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their Time Series Database InfluxDB, but you may not be as familiar with their other tools. Telegraf for Metrics Collection from systems, Chronograf for visualization, and Kapacitor for Real-Time Streaming. All of this is available as open-source. They also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike: Hi folks, welcome to the Real World DevOps podcast. I'm here with Corey Quinn the editor of Last Week in AWS. Welcome to the show, Corey.

Corey: Thanks, Mike. It's always a pleasure to hear myself talking.

Mike: I'm sure it is. So for those who don't know, Corey is one of my closest friends. So, this might get a little off the wall and banter-y. But hopefully everyone will enjoy it. So for those who don't have the pleasure of having met Corey yet, Corey, what is it that you do?

Corey: A lot of things is probably the best and most honest response to that. But what I'm best know for is either shit-posting on Twitter and/or writing Last Week in AWS, which is a newsletter that gathers information from the AWS ecosystem every week, discards the crap that no one cares about, takes what's left, and then makes fun of it.

Mike: It is pretty fucking funny. So by day, you are also an AWS consultant?

Corey: Yes, but in a more directed sense. Specifically, I start and I stop professionally at fixing the horrifying AWS bill. It's one of those areas whereas a consultant, I find that I'm more effective when I'm direct. And so I'll aim at a very specific, very expensive problem.

Mike: Absolutely. So you and I were talking a while back and this is something I have repeatedly come into around isn't Amazon just a single point of failure in my infrastructure? Shouldn't I really be focused on trying to mitigate the failure of AWS going down? What happens if you us-east-1 explodes again? Like, my website's offline, and now I have huge problems. So shouldn't I at that point maybe start thinking about multi-region or maybe multi-provider or any number of other really dumb ideas?

Corey: The answer to all of that is generally it depends, which is accurate and completely useless. The fact of the matter is is depending on what your business is and what your constraints look like, you're the best person to wind up saying that this either an unacceptable risk or ultimately this is something that you absolutely need to address and focus on. The example here is if your application has people's' lives depending on it, then yeah, you need to be able to withstand everything up to and possibly including a nuclear event. Whereas if you're running my side project of “Twitter For Pets,” if your site is down for four hours because AWS region as an issue, maybe it's okay. Maybe the internet is better for it.

Mike: Yeah, that's a good point. It always seemed to me to be an incredible amount of over-engineering, of people trying to get their applications and infrastructure to be completely ... completely capable of surviving basically everything. It's like, "Hey, wait a second. You run a very small social media application that no one's going to care about."

Corey: We saw a fair bit of this a couple years back with the S3 Apocalypse wound up hitting. This is not me trying to bag on Amazon in any meaningful way. They run an incredible amount of complexity at a stupendous scale that boggles the mind. Things break. It's what they do. That's the nature of how working with computers plays out. And there was a knee jerk reaction that we saw from a lot of infrastructure types after this happened where they immediately want to turn on replication for every S3 Bucket, so it's now multi-region in one or more locations. I understand the reflex reaction to that. There's no one better than Ops people at fighting the last war. But there also needs to be a rationed, measured response to this.

Corey: One area that seems to get lost along the way somewhere is you're going to be doubling or tripling in some cases your infrastructure costs from a raw perspective. Plus, the additional complexity, plus people's time to set all of that up, plus data transfer to get things from one place to another. Is this a reasonable response for your business constraints or is it a knee jerk reaction to something that is very unlikely to ever occur again? If we look back, we don't see a litany of individual websites that were called out for things breaking. It was called out as Amazon's failure and things like Instagram were mentioned or American Airlines or Amazon's own status page as examples of things that were impacted. But that was the day the internet broke for a little bit. So, most of us just kind of shrugged and went outside. And eight hours later, things are working again and life went on.

Corey: If you're an ad tech company and you need to be able to sustain that type of outage, and maybe it makes sense to do that. People are not going to come back and look at an ad again. But conversely, I was buying, I think, a pair of socks I want to say five years ago on Amazon. It threw a 500 error and suddenly I'm staring at a dog, which is fascinating. That's amazing. To hear the way some infrastructure people talk about outages, that would ...

Steve Mushero is CEO of Ops Platform provider Siglos.io, and CEO of ChinaNetCloud, China's first Internet Managed Service Provider, AWS Partner, and manager of hundreds of large-scale (up to hundreds of millions of users each) systems. He's previously been CTO in a variety of organizations in Silicon Valley, Seattle, New York, and around the world. You can follow along with his work and insights on LinkedIn, Medium and Twitter.

Links Referenced

• ChinaNetCloud
• Siglos
• Taking Over & Managing Large Messy Systems (LISA 2018)
• Incidents as we Imagine Them Versus How They Actually Are by John Allspaw (PagerDuty Summit 2018)
• Brandon Gregg’s USE Model
• Steve’s tool, runqstat
• Observability vs. Monitoring, is it about Active vs. Passive or Dev vs. Ops? By Steve Mushero
• Dao of Troubleshooting
• How to Monitor the SRE Golden Signals

Transcript

Mike: Running infrastructure at scale is hard. It's messy. It's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps Podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly, and author of O'Reilly's Practical Monitoring.

Mike: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you might not be as familiar with their other tools, Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike: Hi, folks. My name is Mike Julian. I'm here with Steve Mushero, CEO of ChinaNetCloud, and his new startup, Siglos. Welcome to the show, Steve.

Steve: Thank you. Glad to be here.

Mike: You and I met through Monitoring Weekly some time ago, and we've just been chatting since then, and learning all about, or me learning, primarily, about the wild world of systems in China. I believe the company you run, ChinaNetCloud is one of the largest, if not the largest Chinese AWS partner?

Steve: Probably. We're the first one. We were their first sort of operations partner here in China, and also their first MSP, or Managed Service Provider. We've been AWS here and around the world almost 10 years. Yeah, it's been an interesting time.

Mike: Yeah, I'm sure. Why don't you tell us a bit more about what your company does? What are you doing in the Chinese market?

Steve: Here in China, we're basically a managed service provider, so we started 10 years ago now, really managing systems. I was CTO for Tudou, similar to YouTube in the US, so online video and so on. Actually, older and bigger than YouTube. We saw how that market really needed operations help so we really launched this to start helping people design, build, operate, manage, monitor, all that sort of stuff, internet systems. This was before the clouds and all that. Then, we were the first cloud company in China actually doing online VMs. You could buy actual virtual servers, now, of course you have AWS, and Aliyun, and others doing that. Today, our job is really just design, build, manage, monitor, troubleshoot, and do 7-by-24 support for large-scale internet systems, so we're way in the backend, you might say, but really sort of ditch-digging operations guys and girls, really dealing with the just 24-hour systems in clouds, and hybrids, and all the interesting problems you can have. Especially with Chinese scale that we do. That's what we do.

Mike: I think it's absolutely wild that there is ... To think about the size of YouTube, like, pretty much everyone everyone in the western world thinks about YouTube, it's like, oh, wow, that's absolutely huge. To think that there's a competitor in China that's bigger than that, the scale is staggering.

Steve: Yeah. I think now YouTube is larger, but back, this is 10 years ago, now-

Mike: Oh, okay.

Steve: ... It was different, because YouTube, even now in the US, think of it more like Netflix and YouTube combined, because YouTube historically has been short video clips, right? 10 years ago, it was, hey, my cat is cute, stuff like this, you know, 90-seconds, three-minutes, five-minutes. We were running full length movies and TV, and all that, not entirely legally, you might say. It was more like Netflix.

Steve: We had a 100 million viewers a day doing sort of full length stuff, so YouTube's average thing was whatever, you know, a couple minutes, ours was an hour. Completely different infrastructure needs. This is before, especially China had good infrastructure, and BGP routing, and all kinds of stuff, so you had sort of massive CDM problems, and lots of other stuff. I think we were the world's biggest bandwidth buyer for a while.

Mike: Oh, wow.

Steve: I know. Now, everybody else is bigger, but this is, again, a long time ago.

Mike: Oh, sure.

Steve: Because, DSL had just hit China, and everybody wanted to watch Western movies and TV, and a lot of people a lot of times had nothing else to do during the day, and so they're watching this everywhere. Before smartphones too. Yes, it was quite the interesting- Actually, that company's older than YouTube, so even though a lot of things come into China are sort of copies of Western things, those companies actually had much more features, and much richer than YouTube and even Netflix today ... 10 years ago. More advanced, innovative, actually, than the Americans at this point.

Mike: Right.

Steve: It was pretty cool. I learned a lot about infrastructure here, and the craziness of data centers, and large scale systems, you know, thousands of servers, and all kinds of locations, all this kind of stuff. It's all physical servers, no virtualization, no nothing, right? This is started in 2005, and I was here in 2007. It's hundreds of racks of equipment, and all that kind of stuff.

Mike: What's it like today? Thinking back to what I was doing for systems in 2007 ....

Silvia is a Principal Engineer at SendGrid, a cloud email provider with household name clients like eBay, Spotify, Pandora and Airbnb. She is an avid distributed systems and databases tester and spends a lot of her day trolling her Ops team. You can hear more from Silvia on her blog https://blog.dbsmasher.com and on Twitter at @dbsmasher

Key Takeaways

• Involve your DBAs in the engineering process early on to prevent problems in the future.
• Enabling engineering teams to manage their own schema is critical to reducing day-to-day toil of the DBA team and empowering Engineering.
• If you’re wary of automating your databases, set up guard rails to add some safety to it first.

Links Referenced

• SendGrid
• Andy Jassy’s 2018 Keynote at AWS re:Invent
• Stories From the DBA Trenches - Silvia Botros (Velocity London 2018)
• Database Reliability Engineering by Laine Campbell and Charity Majors (affiliate link)
• SendGrid blog
• State of DevOps Report by DORA
• Tool: slowquerydigest (pt-query-digest)
• Tool: Anemometer
• Tool: ProxySQL
• Tool: Vitess
• Auditing Databases at The Grid by Silvia Botros

Transcript

Mike: Running infrastructure at scale is hard. It's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we are going to talk about the rough edges. We are going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor of the Monitoring Weekly Newsletter, author of O’Reilly’s Practical Monitoring, and a DevOps consultant/analyst.

Mike: Hi everyone, I'm Mike Julian, I'm here with Silvia Botros, principle DBA at SendGrid. Welcome to the show, Silvia.

Silvia: Thank you for having me.

Mike: You work at SendGrid. There's a lot of people that have no idea what SendGrid is. Monitoring Weekly is actually a customer as well. I use it for all of my contact forms. It's wonderful. But for those who don't know, what does SendGrid do?

Silvia: First, thank you for being a customer.

Mike: Absolutely.

Silvia: I like to hear that. What we do is basically an email service in the cloud. A lot of companies realized years ago that they need as part of their engagement with their customers whether it's onboarding new customers or communication with existing users, that they need to send a lot of email. It turns out that sending email is a complicated thing. There's a lot of rules around it. SPF, whitelisting IPs, dedicated IP or not. Block lists that the inbox providers maintain. There's magics of times depending on the provider, whether email looks fine or it looks spam-y, or it looks fishy. Turns out there's a lot of logistics around that, that these companies all across the board don't really have the skills to deal with and don't want to spend resources on.

Silvia: Customers of ours include a large swath of companies large and small. Some of our big high profile customers include eBay, Spotify, Uber, Pandora, so you get the gist. You sign up for an account, you need a confirmation email. We have to click a link. They want that to come to the inbox, not to the spam folder and they want it to go to the inbox fast. That is basically the service we provide to them. We also now, as you mentioned have your contact list, we also have a marketing email, a marketing campaign's product that we provide to feed into this and provide the more advanced, the more geared towards marketing and that crowd. Not just the day-to-day transaction stuff.

Mike: Yep. Yeah, it is a surprising huge pain in the ass to actually manage all that stuff. I know several companies that just use their corporate email accounts to do it and surprise, surprise everything goes to spam.

Silvia: Yes. I will fully admit I did not know this was a thing, that this was a viable business model before I got my job at SendGrid. But once I started learning about all the details it dawned on me, how much it is ... Why would you build that yourself? If you are eBay, and you're focusing on bids or getting traffic or if you're Pandora and focusing on getting people to listen to playlists, why would you worry about spam lists and how every individual inbox provider decides how to judge your emails?

Mike: Yeah. That's a rabbit hole that seemingly never ends.

Silvia: True.

Mike: You're a DBA for them which is kind of weird. Those are still a thing? I thought when we all started rubbing DevOps on everything, that DBAs just kind of went away. We didn't really need them anymore because now the developers are handling it.

Silvia: I know, like we had thought DevOps were going to eliminate the sysadmin didn't we?

Mike: Right and it turns out we still need those, so I suppose we probably still need the DBA. What does a DBA actually do? What's your role now? What does it actually look like?

Silvia: A DBA in a company that is honestly trying to apply DevOps is practices, which is whole other rabbit hole we can go into. A DBA in this environment needs to be involved in setting standards of practice, that's helping engineers debug things early on. Helping design. A lot of what I do is early cycle, early life cycle of software where they're still building a new thing and I get to give the input on whether this thing has the potential of growing really fast and what does that provide as far as implications as to the data store. Is the data store they're using appropriate for the use case? A lot of guidance, a lot of things like that. There's also still a lot of day to day things in production. For example, we are SOC2 compliant and we are now public company so we also have to do cover Sarbanes-Oxley. There's a lot of compliance needs that don't necessarily fall into any feature work but still have to be covered because every year we have odd pairs, we have fiduciary requirements that we have to cover and so the database ops team, which is what we call my team right now, is responsible for doing the work that provides the evidence that we actually are compliant with all the things that we have to be compliant with.

Mike: That's kind of interesting. What compliance stuff actually surrounds the database? I know GDPR probably has a big component in here doesn't it?

Silvia: It does. That was definitely a big thing and it covers not just relational databases, it covers a lot of data sources so in practice it involves more than just the DB ops team. To provide context, currently my team is solely focused on relational databases so we have a pretty large MySQL footprint and that's where our focus is.

Mike: Yeah, every time I think about DBA's, I think about just relational databases but once I started thinking about that more and about you and I chatting, I realized...