109. The DevSecOps Playbook with Paul McCarty - Part 1
Paul McCarty is a DevSecOps evangelist, and his recent chat with Claire was so great, we had to split it into 2-parts. In part 1 they talk about his DevSecOps Playbook, the challenges of security and engineering teams working together harmoniously, and how to apply the Essential 8 to the software development lifecycle. You can hear Claire really enjoyed chatting to Paul about some of the more technical aspects of security and hearing his views on application security best practice.
Paul is the founder of SecureStack, the world's first DevSecOps Maturity Platform. Paul has been helping organisations build more secure applications for almost 30 years. He’s worked for large organisations like NASA, Boeing, Blue Cross/Blue Shield, John Deere, the US military, but he’s also worked with several startups going back to the mid nineties. Paul is a frequent contributor to open source and Linux projects and is a co-organiser of several community group meetups here in Australia.
Links:
The Security Collective podcast is proudly brought to you in partnership with LastPass, the leading password manager.
Transcript
CP: Hello, I'm Claire Pales, and welcome to The Security Collective podcast. This week's guest is Paul McCarty. Paul is a dev SEC ops evangelist and the founder of Secure Stack, the world's first DevSecOps maturity platform. Paul has worked for large organisations like NASA, Boeing, and the US military has also worked with several start-ups going back to the mid 90s. Paul is an active member of the security community here in Australia. He's also a pretty good snowboarder, and most importantly, a husband and father to three amazing kids. Paul and I talked a lot, so as is my usual course of action, we have split the conversation over two weeks. In the first part, Paul and I talked about his DevSecOps playbook, the challenges of security and engineering teams working together harmoniously, and how to apply the Essential 8 to the software development lifecycle. I really enjoyed chatting to Paul about some of the more technical aspects of security and hearing his views on application security best practice. So here's part one of my chat with Paul McCarty. So Paul, it is great to have you join me on The Security Collective podcast today.
PMc: Well, thanks for having me, Claire, I appreciate it.
CP: So I thought a good place to start might be talking about the playbook that you've authored the DevSecOps Playbook. Tell me a little bit about what in your mind defines DevSecOps, and what drove you to write the guidance in the first place?
PMc: DevSecOps is I have a passion for it. I call myself a DevSecOps evangelist. One of the things that kind of drives that evangelism is that I hear a lot of people talking about it, and it's a buzzword to them, right. And that's really unfortunate to me, because the way I see DevSecOps is it's a combination, an inclusivity of developers, security teams and operations teams, working together for a common purpose, which is to build and manage more secure application environments. And I think we forget that with all those like buzzwords, and all these vendors claiming to be DevSecOps this or that, ultimately, it's about teams working together with a common purpose. The second part of the question is what drove me to write the playbook? You know, I think that it might seem altruistic, but the reality is that there was a selfish component there, which was that I have a bunch of engineers that work for me. And when they went through uni, or TAFE, or whatever it is, they didn't get any real training on how to write and build secure code and applications in a secure way, which is complex. And so because weren't taught those tools, I thought, wouldn't it be good to have a descriptive, prescriptive document that says these are specific things that software engineers can do, that security teams can do, that operations teams can do to interact together to build more secure applications, like really like, these are very specific things, not high level guidance, very specific things. So ultimately, it was selfish, but then, you know, I wanted to open source it because I thought it could be a valuable document for other organisations.
CP: And I think the challenge that we have is, it's probably a podcast for another day, but people do go through uni, and they learn certain ways of doing things and certain crafts. And then they get into the corporate world, and it probably blows their mind a little bit. And for engineers, I suppose coming out of uni and learning their trade, and then going into an environment where there's all of a sudden, there's a security team, saying, actually, we need you to do things differently, or, actually, we need to put some gates into your process. I think there's sort of a challenge there between those two teams. And I'm wondering, do you think that's where it stems from is that they sort of come from different worlds? And how can we better manage this relationship between the two groups, the engineering side and the InfoSec side?
PMc: Yeah, I mean, this has been 25 plus years, I've been doing this and it was a problem then, and it's a problem now. The big elephant in the room is that software engineers create value in the form of features and revenue for organisations. So the business always sees them as their value being, how quickly they can deliver features. And because of that, that is intrinsically diametrically opposed to this idea of doing it in a secure, structured way. And I think that we've never really gotten past that we've never really kind of addressed why the InfoSec team and the software engineering teams can't work together more collaboratively. And that's another reason that I wrote the DevSecOps Playbook is to identify different tasks and specific things that the different teams can do. But ultimately, it comes down to the fact that the business incentivises the developers to deliver features as quickly as possible. And that is not what the InfoSec team is built to do. So those are diametrically opposed.
CP: We have the secure coding standards, often written by the security team, or the expectations put in place by the security team. And that's one thing having a standard there. But we often don't necessarily see that translate into secure coding action within the engineering teams. And do you think it's a lack of change management? And when I talk about change management, I mean, people change as opposed to technology change. Do you think that's where it's falling down? I guess where are we not influencing the engineers to see what's in it for them? And to see how it can be done better. Because we have to have the standards are we going to see vulnerabilities and exposures, but the two don't necessarily always marry up. And maybe it comes back to what you were talking about before in that pace and delivery side of things, not really factoring in that the need for these standards. I'm keen to hear your opinion.
PMc: Definitely, that's a significant part of it, for sure, right. And the fact that business is prescribing value to them that it doesn't work with the InfoSec mandate. But I'm going to take this opportunity to talk about DevOps, right, because I think that DevOps changed my career. And it changed the IT industry, because what it did is it took two teams that had been probably, if not openly hostile to each other, combative for a long time. And I know this sounds, I make this sound amazing, and it was in part, but the reality is that we actually did start working together in the late 2000s. I know, because I was there, I was on a platform team. You know, working collaboratively together, even in the nation state software engineers moved towards the middle. And they started to understand some of the operations constraints, right. And the operations team started to act like developers, right. And this is where infrastructures code and a lot of these basic things like Ansible, and CF engine that came out of this. And so those two teams started to learn from each other. Now the problem is, and this is getting back to your question is, InfoSec has never done that they've never tried to move towards the centre. Instead, their process is always like, you guys didn't do this. And they give this big list of all these vulnerabilities they found in source code, or they give them a security questionnaire, and God forbid, we need to get away from the security questionnaires, right. But if the security teams aren't interested in moving towards the centre and learning to actually collaborate with those teams, we're going to continue to have this problem, right. Ultimately, Secure Stack and what I'm trying to do is all about giving those teams the ability to use a common platform to do those things together. I think it's working.
CP: Following on from that, from an operating model perspective, do you see the application security capability or the security coding side of things, should that capability be something that sits in the engineering teams to enhance or encourage that collaboration? Or do you think that the AppSec, people should be in the Information Security team and wherever they sit, do you think it matters in order to have the capability in your business anyway?
PMc: The primary thing is to have the capability in your business. And the answer your question is, it depends, I know, that's such a cop out, right? We always do that. It just depends. Now, if you were to twist my arm, I'd say that it should probably be in the software engineering group, right. Just to take a stance on one side or the other. And there's really two reasons for that. The first is that getting back to my earlier observation, about InfoSec, not really moving towards the centre. And we can see this and a lot of the guidance that comes out of security teams around software development, it's just not topical. It's not actually viable. And so it talks a lot about things from the kind of classic network security stance, but it doesn't really understand the day to day operations of software engineers. And so if this team is instead of the software engineering group, then they understand more natively the kind of day to day processes and things they do. That's the first part of the answer. And the second part is that honestly, having it in the software engineering teams brings more visibility to it than probably if it was in the security side of things. And just experientially when I've seen application security be on the security side, it's typically orphaned, underfunded, and understaffed, right. And it eventually dies, because those people will go somewhere else. So if I were to take a stand, I'd say probably on the software engineering side.
CP: And is that sort of what Utopia looks like to you having a bunch of security people sitting with the engineers collaborating, you know, making sure that the operations kind of hum in that way? What does kind of well oiled application security function look like to you?
PMc: I think a lot of my friends think I'm a nihilist, but the reality is, I'm an optimist. Right and to your point, I genuinely think and I know because I saw it, you know, harkening back to my observation about DevOps, like, I can see teams evolve. Now, not everybody can, but the reality is, I've never been paid as a software engineer, right. I'm originally a Unix admin from the 90s that evolved. I took an opportunity in the early 2000s, to work with an InfoSec team, it changed the course of my career. So I think the answer is that we need to create groups that are cross functional, that are working together with a common goal. And that's not what we have right now, when the security team has this mandate that's very kind of static, and not fluid. And then you have a set of engineers whose whole job is fluid and agile, the two just don't come together. And we need to create a way where those teams can work together, and they can start working like each other and acting and talking to each other.
CP: Yeah, I definitely agree with your commentary around where you see AppSec people sit in the Information Security team, because their thinking is so different, often so different to their peers and information security team. But also, there are not many CISOs that come from that side of the business. A lot of CISOs have come up through network security, or GRC, or corporate security or that side of things. And often, I don't want to generalise, but you often see them come up through the more windows based side or the I guess that career path, you don't often see CISOs that come up through application security, or through the engineering side. Those that we have, we're very fortunate to have in the industry, but there are less of them. And so it's very difficult, I think, for a CISO to manage an application security operator or engineer and understand what they need, and to lead them.
PMc: The bigger issue underneath where you're talking about there is that we just really don't get application security, right, we just as an industry don't get it. And so that's why we have these kind of overly simplistic ideas of like, oh, let's just stick some static analysis in there. Right, and then that evolves into DevSecOps , this is the one that really gets me DevSecOps, we claim success when we've put a little bit of security tooling in the CIC process, we're like sweet, we got it, DevSecOps won. And ultimately, it always comes back to the fact that we really don't get application security. It's not something that the software engineering teams are built to understand natively, and it's certainly not something that InfoSec teams are built to understand. So you're right. And if you have a team that's one application security engineer, that is a box ticking exercise, right. Like if your AppSec team has one person that has a box being ticked, that's all that is right and that person is going to move on really quickly, because they're not going to have the power in the team behind them to actually affect the application at state, 100%.
CP: I completely agree with that. And they're either going to be overwhelmed with workload or underwhelmed because their under utilised, purely but because they're alone, as you've said, and you can affect change in that way. I liked what you said earlier, too, about how you know, InfoSec principles and policies and standards can be quite static. And yet, there's so much movement and fluidity that goes on in engineering. So that makes it even more strange that you would write a comprehensive guide around the Essential 8 standard for the software development lifecycle. But I love the way that you've kind of modernised the controls for these environments that are no longer simply Microsoft or on prem. For those that haven't seen the guide, and if we never had that sort of thinking from you, what's the impact if we if we stay with this outdated view of the Essential 8, in relation to software development?
PMc: Well, yeah, I mean, I think that we miss an opportunity to apply that set of concepts to other areas of IT. Like I mean, to take a step back, I moved here in 2016 and I have never come across the Essential 8 before, and I saw it everywhere. And it had ultimately, by I don't know, 2019 or whatever, it had pretty good starting to get pretty good penetration across the mainstream. And I initially, I mean, just being dead set honest, I initially ignored it, because I looked at it and I thought this was written by people that clearly have an enterprise Microsoft desktop focus. And so I just ignored it, because that's not what I do. But also too, my customers didn't fit that model. And I need to make the I need to point out during this kind of funny anecdote, I need to also point out the fact that the reality, is that description, that limited kind of perspective from the Essential 8, does not describe a vast number of Australian organisations. Startups or scaleups don't fit that model at all. They are not Microsoft shops, right? Their desktops are not managed in the same way that the enterprise desktops are. So I think that the Essential 8 with that simplistic lens, on doesn't really apply. And so it started me thinking Claire, I started thinking I was like, these concepts, right, these ideas of patching applications and restricting administrative access and MFA and backups, these are basic core concepts that we can apply across, you know, all kinds of areas and IT. And got me thinking about SDLC, the Software Development LifeCycle, and I thought there are real ways that we can implement these high level of security concepts in software development. MFA, right, you can turn MFA on both in your interaction with writing your code and pushing that code to source code management. You can enable MFA in your source code management platform like GitHub, or Bitbucket or Atlassian. Backups, like making sure that your applications are backed up. And that means more than just, you know, your source code repositories, understanding how the application fits together, the application environment being able to construct that again, using things like infrastructure of code. And patching applications, I mean, web applications we're building need to be patched and updated too as well. Struts and Log4J and everything else has taught us that. So these are high level concepts that I thought, hey, man, we can apply these to software development, let me do that, and kind of show people that, you know, there's a different lens that we can use for this.
CP: And the challenge, I guess, for the Essential 8, is that in government, they have, I guess, a desire to implement it, because it's something that's been driven by the government. And over the years, they have tried to make it less, not less rigid, but by building in the maturity levels, help people to understand entry level, and then maybe where they want to aspire to as well. Do you think that the change to a maturity level model has helped with the ability to implement it?
PMc: Adding the maturity levels has definitely helped evolve it, and it's made it easier for me to blend it into the software development lifecycle, right for my document. So for sure, I feel like the Essential 8 can be made even better. If ASD and the government were to reach out to industry and say, hey, how can you people who actually do this for a living, right, this is my job, I do it every day. How can you see this document evolving to better suit you and your teams? I'd be happy to respond, you know, and be involved. I think that's probably something that we need to really encourage. Let's get some industry involvement in these documents, the ISM can evolve in a good way if the industry is involved more.
CP: Yeah, certainly. And I think that in principle, what the Essential 8 stands for is, as you said earlier, it's that minimum viable security, it's the baseline that we want everybody to meet, but then the implementation of it and the maintenance of those Essential 8 can be challenging for some organisations, from a resource perspective, both financial and people, but also influencing the business to put things in place. So thinking this through it with a different mindset around the software development lifecycle is probably something that people haven't done in the past. And you know, when I read it, I thought this is really twisting the thinking a little bit and it's good to see because not everybody would take the time to take a standard like that and think, okay, well, in principle, I wouldn't usually use it, but how do I adapt it so that I can get the best from it?
That wraps up part one of my conversation with Paul McCarty, please tune in next week for part two when Paul and I chat about minimum viable security product, the software bill of materials and making governance material consumable for senior audiences, no matter how unsexy policies might be. So please tune in next week for part two with Paul McCarty.