97. The reality of cyber incident response with Ellis Brover
Claire chats with former Toyota Australia CIO Ellis Brover, as he shares his thoughts on incident response through the lens of the CIO. They discuss how security maturity can dictate reporting lines, how organisations should seek to test the reality of systems being shut down because of an incident, and really how moral support goes a long way during a cyber incident.
Ellis Brover is a recognised IT leader with a track record over three decades of building and leading world-class IT organisations, driving transformational change, and delivering tangible business value. His experience spans a range of roles and industries, across a range of organisational scales from startups to multi-nationals.
Most recently Ellis was CIO of Toyota Australia, where he led a transformation of the IT function from an internally-focussed service provider to a strategic enabler, driver of innovation, and role model for outstanding customer service. Ellis grew and led a team of 300+ that delivered an industry-leading digital business capability as well as a rapid transformation in cyber security maturity, whilst dramatically improving efficiency and contributing to business growth.
Ellis is now pursuing advisory and consulting opportunities, aiming to add value to the business success of organisations and the development of their people through his extensive experience.
Links:
The Security Collective podcast is proudly brought to you in partnership with LastPass, the leading password manager.
Transcript
CP: Hello, I'm Claire Pales and welcome to The Security Collective podcast. Today's guest is Ellis Brover. Ellis is a recognised it leader with a track record over three decades of building and leading world class IT organisations. Most recently, Ellis was the CIO of Toyota, Australia, where he led a transformation of the IT function from an internally focused service provider to a strategic enabler. Ellis is now pursuing advisory and consulting opportunities and aims to add value to the business success of organisations and the development of their people through his extensive experience. Ellis shared his thoughts on incident response today with me through the lens of the CIO, which is so valuable. We covered how security maturity can dictate reporting lines, how organisations should seek to test the reality of systems being shut down because of an incident, and really how moral support goes a long way during a cyber incident. There is a wealth of advice in this episode, I also think you'll be nodding along in violent agreement with some of the things that Ellis shares. So please enjoy my chat with Ellis Brover. Ellis, thanks so much for joining me on the security collective today.
EB: Yeah, great to be here. Thanks Claire.
CP: So I want to start with a topic that everybody loves, budgets. I was talking on a webinar this week to some board directors. And one of the questions we put to them was, what is the percentage of your IT budget that was spent on cyber this financial year, and I was probably not shocked to hear that of the few 100 we spoke to 30% had no idea what had been spent on cyber. What are your thoughts around this type of statistic around board's not really understanding how much money is being spent on cyber?
EB: Look, to be honest, I'm surprised that only 30% didn't know, I actually would have thought it would be higher than that. I think it's actually probably the simplest metric of all for a CISO or a CIO to use to agree overall settings for cyber investment with the board. It's easy to understand it's got extensive industry benchmarking. And you know, if your organisation cyber investment is substantially less than the industry benchmark, then obviously you need to ask, is there really a good reason to justify this, and it gives you a clear opening for a discussion with the board. But I think the security leader also needs to be able to enunciate to the board in non technical terms, what they will get for their money. It's not just the money question, obviously, you need to enunciate the value and hard with cyber. But for me, I found it's helpful to adopt something like you know, the NIST five tier scale. So you can communicate numerically or visually your current state, where you aspire to be, agree that future state, and have a mature conversation about the organisation's risk tolerance. And I think for many boards, it's an eye opener just to understand that we can never eliminate cyber risks entirely, that it's a matter of where we want to sit on a risk spectrum.
CP: And I think that's a really important point that you make about not just how much you're spending on cyber, but what you're getting for that money. And being able to articulate that in a way that the board can see, as you said visually, or through a risk measure, or a maturity measure, allows them to see current, state future state, and where you want to get to. Not all of those things are going to cost money either. And so that's a difficult balance to make, too.
EB: Yeah, absolutely. And look, I know for those of us who are more deeply involved in this, it always seems a little oversimplified doesn't it. To have to just come up and say, well, if you give me $5 million a year, I'll get you to a maturity level of three out of five. Of course, we all have all of these cases in our head where we think yeah, but, but it's more complex. but, but, but. But realistically, I think it needs to be able to simplify it for the board to have that overall high level conversation of where do we want to sit? And what does that mean, what is the three out of five actually mean? What kind of risks do we think we've got a good chance of defending ourselves against and what kind of risks realistically we just have to accept that we can't defend ourselves against those things. I think it's actually a very mature conversation to have with the board. The ones that I don't like hearing or having are the ones where the board is somehow under the misapprehension that if we give you this money, then that's it. We don't have to think about cyber anymore, and nothing bad will ever happen to us.
CP: And sometimes a really big chunk of that budget for some organisations can be their teams and, you know, having that thinking process around, how do you spend your resource investments? Do you have it inside your organisation, do you have it outside? Given the challenge we're having that you know, facing these difficult recruitment times at the moment, not just in cyber but across everything. What role do you think outsourcing plays because that can pay a significant investment. But for IT and cyber, if you can't hire the capability you need, what are your thoughts around using some of your investment dollars to bring in outside help?
EB: Absolutely. Look, I'd say, don't wait. You know, if you find yourself having that choice, I think that the simplest answer is don't wait, get capability in place immediately, even if it isn't via your preferred sourcing strategy, you can still continue to pursue in house expertise in parallel. But don't wait until you are able to get the right person. It is incredibly challenging at the moment, as we all know, to hire good cyber specialists. And many companies simply can't afford to pay the going salaries because of HR policies, insufficient scale, and so on. So many of us have to rely on outsourcing. That's the reality. And I think overall, that's okay, we have some strong cyber expertise amongst Australian service providers. But I'd say be clear on what you're outsourcing. If it's ongoing monitoring and so on, great, but don't rely on a vendor for your security strategy. Don't rely on a vendor for your internal executive engagement or user awareness. Because those things require influence and intimacy with the culture of the organisation. And be realistic about the level of expertise you're able to get, whether it's in house or from your service provider. It may well be fine for BAU, but maybe grossly insufficient for leading a breach incident response to a high level attack. So ensure you know who to go to for that specialised expertise when you do need it.
CP: So we talked in your bio, about the fact that you're sort of going into the next frontier of your career now and you're looking to consult. But if you were to build a cyber team today, where would you start? Who would you begin with that you know, apart from maybe a security leader, that might be your answer, but who is the most important person for you to hire as a first start, if you were bringing in a new cyber team today?
EB: I guess I'm probably talking to an audience of predominantly security leaders, I'd be foolish not to say a security leader, I definitely would realistically start with a strong CISO. But I think probably the interesting thing for me is what makes a strong CISO. So you know, we obviously want someone who knows their domain. But more importantly to me is able to communicate cyber risks with conviction, with clarity and with a pragmatic balance. It's no use having a highly knowledgeable security leader who can't engage and influence the company senior executives and the board. Because cyber investment will, as we know, inevitably be overlooked in favour of seemingly more urgent, more glamorous priorities. And for me the first test, to be honest, would be having that person having the courage to challenge me as the CIO, when I was in that role. I want to see them speaking up and advocating for cyber rather than waiting to be told or asked. But on the other hand, we can't just be one eyed you know, a CISO can't just be always be the boy who cried wolf, so to speak. Their style needs to be balanced and nuanced, so that they don't lose credibility by being seen as overzealous, always saying no, or lacking commercial judgement. So it's a tough role. It's one of those classic T shaped roles as we like to call them that are so hard to get for us in IT. Where we want someone with strong domain knowledge, but also excellent communication, influencing and collaboration skills. So that's where I'd start.
CP: So tell me, do you think that they should be reporting to you? I mean, this is a bit of a loaded question, because there's no right answer, but as a CIO, should the CISO or head of information security report to the head of technology?
EB: I'll probably say yes, and no. In a perfect world, I think I'd like to see the CISO reporting line at the most senior level in the organisation, because I think information security should be broader than just IT security. It should encompass threats that have nothing to do with IT. But that's a perfect world. And I think for many companies, that's simply a step too far, at least in the in an interim period. If the maturity is not high enough within the IT cybersecurity space, then I think it is more practical, at least initially, to have the CISO reporting into the CIO, and ensure that strong focus on uplifting the IT maturity. Maybe with a view in time, once that sort of stable and mature footing to shifting that reporting line upwards.
CP: I think that's probably the key for most organisations is having a level of maturity where the CISO would be able to have traction at the more senior levels. But also the CEO having a level of cyber literacy and maturity that if that role was to report to them, for example, that they could actually take on the strategic conversation and take on the cultural understanding that cyber needs. You know, their role as the CEO inside there is just as important as where that CISOs sits and who they report to. And if they haven't not mature enough, there's going to be a challenge as well.
EB: Yeah, absolutely agree with you. And I think that applies not just to that CISO role, but to lots of roles doesn't it. We often choose our reporting structure, maybe based not on the perfect functional logic, but based on the skills and the competencies of the people in the role. And if the person that you want someone to report to simply doesn't have the skills to manage and monitor their performance, then it's probably not going to work even if that's what the theory says. So yeah, I agree with you, I'd say be pragmatic about it.
CP: So I want to change directions a little bit and talk about some of the things that CISO needs to do. And in many cases, they are the leader of incident response when it comes to cybersecurity or certainly heavily involved in that sort of war room. I'd love your thoughts on a few things about cyber breaches. And one is, do you think organisations are thinking about cybersecurity incidents and what they would do during an incident, after an incident. You know, there's so much that can go on. What in your experience around your peers in the CIO roles, are people really thinking through the impact that a cyber incident is going to have? You know, maybe we have to shut down systems? Maybe we have to alert our customers. There's so much that goes on. How do you see CIOs feeling about this? And what's their planning been like?
EB: Look, I think, given the attention to this topic in the last few years, it's getting a lot better, I believe. But to be honest, when you kind of look past maybe the CISO and the CIO, I do think most organisations are not genuinely thinking about that. And most of us spend our days focusing on balancing the immediate burning issues with strategic initiatives for the future. And very few people love thinking about worst case scenarios, which I think always seem highly unlikely until they actually happen to you. Even if there's formally an incident response plan in the organisation, I think it's often quite generic and shallow. And it hasn't really thought through the implications of trying to operate the business during a long term systems outage. So I do think there's a gap there. Certainly not for all organisations but for many that I talked to, it's kind of a plan that sits in a drawer or on a shelf and isn't necessarily going to be useful in a real life cyber incident scenario.
CP: And the number of drills that I've been part of where the incident response plan is not even factored in, because, you know, thankfully, these are drills. But even then, you know, realising that you might have this 40 or 50 page document, that in the heat of a crisis, you're not going to sit there and page turn. I wonder about these documents, and I'd much prefer the checklist at the front half the time because it's a much more of a ready reckoner for some people, if you didn't have the CIO and the CISO at hand, or somebody who knows what they're doing from an outsource perspective. You know, those documents can be, as you said, popped in a drawer and gather dust really, and yet, so many compliance expectations require you to have one. I mean, that it's such a balance.
EB: Absolutely agree. And the thing I think in IT is we always end up kind of muddling through, don't we, even in a crisis, in an incident response situation, we generally get through it. But we wing it, we rely on a small number of talented individuals who've got a real depth of technical skill, but also a breadth of knowledge of the organisation's IT systems and business processes. And those people are fantastic to have. But it's a huge risk relying on this small number of people that are genuinely able to do that, and who have immense pressure put on them in that kind of environment. So we get through it with individual heroics, rather than with planning. But I think we've got to recognise that by doing that we're not only putting pressure on people, but we're exposing the organisation to unnecessary risk. And probably to a delay, if you look at it in the cold light of day, we probably take far longer to recover in a crisis scenario than we could have had we done our planning upfront.
CP: Yeah, I was going to say to you, you know, how exposed are companies who aren't prepared and who don't even do drills? I mean, drills are great, they're great learning experiences. But even if you're doing those, a lack of preparation, where does that leave an organisation?
EB: I think it leaves them with an unnecessary level of organisational risk, that often isn't really appreciated. I think the wonderful thing about the maturity of IT these days, is that most of our colleagues and business divisions rely implicitly on these systems. They're always going to be there, you can rely on them always being on in the morning when you come in until you can't. And the trouble with that is that I think organisations have lost their knack to actually work around the absence of IT systems to operate business processes manually. And I think that's a really important conversation that we need to have. Hopefully not during a crisis, but before. What would you actually do if all the systems are down for a few weeks? We don't normally think of that scenario, don't we? Generally, if we have an outage, it's, you know, it's for half an hour, it's for an hour, it's for a few hours, it's overnight. It's annoying, it's frustrating, but it doesn't genuinely disrupt the business and force people to operate manual processes. But what if it's for much longer than that? How would you operate your company for several weeks, if you imagine you have no IT systems at all. And I find most people just don't think through that scenario. Or if they do, it's very, very shallow. Oh we'll do it on paper, we'll do it with a spreadsheet. But if you haven't thought through how you will do it, you inevitably find it in the cold light of day that you've missed something. You've missed that one critical, tiny departmental system that sits in a corner that nobody thinks about, that doesn't have any backup, that doesn't have any disaster recovery plan, but that actually your key business process hinges upon.
CP: Yes. And I think the other thing is there's a big difference between being able to operate on pen and paper, and the reality of operating on pen and paper. And you know, if I think about a retail store, yes, you can operate potentially writing receipts on pen and paper and somebody's marking down the widgets that you're selling. But that could become very tiresome very quickly. And so yes, in theory, you can operate on pen and paper, but even for a few days or a few weeks, that could be pretty labour intensive for staff, and they're just going to get burnt out. It's a very difficult scenario to play out in your head until you're actually in it, even if in theory, it might work.
EB: Absolutely. And you've got to start thinking about things like how you're going to get paid for those things. It's all very well to ship out goods, provide services, how we're going to get paid, how we're going to keep track of what needs to be paid, how we're going to maintain our compliance obligations afterwards so we can actually update our backend systems to reflect what happens on paper. You've got to think about those things. And even Claire, to be honest, the simplest things that seems trite, but I've seen a number of times that situation where your plan for a manual workaround says I'm going to call my customers by phone, and I'm going to take their orders. Except of course, when all the IT systems go down, you don't know their phone numbers anymore, because your phone contacts have been wiped and your email systems are inaccessible because the IT people have turned them off. Then what do you do? I mean, it's incredibly simple, right. But if you don't think through that kind of thing, in advance, you get caught in real life.
CP: And that's definitely the value of running scenario exercises and drills and running them regularly, because the staff turnover in organisations can be significant enough that people around the table in an incident could change, even just between scenarios, let alone between incidents.
EB: Yeah, that's funny, I've definitely seen that the situation number of times where the organisation used to have the kind of corporate knowledge of how to run processes manually, simply because the people in those roles had been there for a long time, predated the modern IT systems that they now rely on. So they're still remembered how things used to be done. But as those people change, and new people come in, they've never had the experience of operating processes manually. And so if those things aren't written down, and if they aren't drilled, as you say, those people are never going to know how to actually react when the crisis comes.
CP: And for many cyber leaders, things like DR and BCP don't normally sit within their remit, it's really an IT operations, or even another part of the organisation that might run BCP. And IT might have DR. What are the role of those two functions do you think in incident response?
EB: Look, I think it all starts with the BCP. And I think it's critical. And I find it's often under done again, because it's not particularly glamorous and exciting for people to to worry about. But in a crisis scenario, the IT incident response team is going to be focused on flushing out the attackers, rebuilding their systems, testing, verifying, and so on. The rest of the company needs to know how to run their business processes, in the absence of a functioning IT landscape. If you don't have a BCP, you're not going to be able to realistically do that, or you're going to have to, as we said before wing it in reality. And I find that your most substantial organisation, certainly they have a BCP or DRP on the shelf, often they're not really fit for purpose. So for example, you know, we tend to focus on old fashioned physical threats like our data centre's on fire. Is that really the most likely scenario in this day and age? You know, we may have a failure scenario that says we're going to switch to our backup data centre. But in the event of a cyber breach, your backup systems may be just as compromised as your production systems. So you need to think through what you're actually going do, because your traditional disaster recovery approaches may just not work. So I find a common trap is that sort of failure to think deeply about manual process workarounds, and also relative priorities. If you start your BCP with the premise that everything's equally critical, I think it's kind of be largely worthless.
CP: And then if I think about the number of boards who asked me what their role might be in an incident, what are your thoughts about what they need to be across? And when should a board weigh in on an incident? And when should they just let the operations people try to resolve and investigate and contain and eradicate and get you all back up and running again? What should the board be doing as part of that, in your experience?
EB: I think, again, if we look at a perfect world, and humans are never perfect, but in a perfect world, I'd like to see the board firstly, providing some support to the IT team. That doesn't mean getting involved in micro detail. That definitely doesn't mean asking every hour, are we there yet? Are we there yet? Can you go any faster? That's not helpful. But being able to provide some, some cover, if I can put it that way. Some moral support for an IT team, whilst focusing the rest of the organization's attention on stuff that have nothing to do with IT. On what are the critical business processes we now need to focus on? And what are the risk trade offs that we're prepared to make in this environment, because there will be substantial decisions to make. You will need to decide how much are we prepared to put at risk operating with a pen and paper process. How much are we prepared to base on IOUs, on phone calls to customers saying you know you owe me this much money after our systems of backup, right, we can trust each other on this, and so on. So I think the board needs to be ready to rapidly make those kinds of priority calls when they're required. But not to get involved in the micro detail, that really does not help at all. One of my experiences, I remember, it seems maybe like a funny comment, but possibly the most useful thing that a board member did in terms of my IT incident response team was to walk into the room with, you know, dirty, sweaty, unwashed, IT guys who have been not sleeping for 72 hours or whatever, desperately trying to get things back up. And just saying, guys, tomorrow morning, the sun's still gonna come back up, and nobody's died. So don't worry, you're doing a great job, things are going be okay. And walking back out again. And that was probably the single biggest morale boost that those guys could have got in that moment, rather than the, right where are you up to? Are we there yet? What's next? When are we going to be up?
CP: That recognition, I think for people who are at the coalface just trying to get things working in, you know, they've been in problem solving mode for hours or days. Certainly that recognition that they're seen, and that, you know, they're not in a back office somewhere, and not recognised as the hard work they're putting in. I think moral support goes a long way.
EB: Absolutely. I think that and also being able to say in front of all of the company's executives, leave IT alone at the moment, they are doing what they can to get us out of this. Let's not bother them. Let's now focus on the business side, I think is a huge thing that the board or the CEO can do.
CP: I want to finish up by asking you about some of the risks that you might see coming over the horizon for types of organisations like wholesale distribution, retail sales, those types of industries. Do you think the threats are any different that are coming at us in the next 12 months? Or is it just what every business is facing now?
EB: I think all businesses obviously are facing a high level of risk now, even if it's just from the generic sort of ransomware type threats. I think for a distribution and sales business, specifically, obviously, they're usually highly dependent on IT to run their supply chain. And these days, many of those supply chains have very complex embedded business rules in their systems. So I think it really is just the focus on what I said earlier, which is to think in advance about how they would lead through a complete systems outage without stopping their entire business, and without impacting their customers. And I think for most businesses, that is absolutely possible. As I said before, they used to run things manually in many cases before thier modern IT systems automated their processes. But it takes a real conscious effort and a commitment to plan it and some creative thinking to find workarounds. I think the other thing that applies to many businesses, but certainly to a distribution and sales type business is to think about the reputational impact as well. How will you manage the communication? What will you say to your key customers? How much will you talk about what's actually happening, and why? How much do you have an obligation to, in fact, tell your customers about potential compromise that may that may infect their systems as well? I think that's worth thinking through in advance as well, because in the heat of a crisis is not a great time to be making those kinds of sensitive calls.
CP: Yeah, I mean, it comes back to what you said earlier about preparation and making some of those going in decisions or going in positions really clear. Even though during a crisis, you might change that. But having done some of the thinking before you're in a crisis about how might we respond? How might we feel about turning everything off? How might we feel about paying a ransom? At least you're not trying to start the thinking of that during a crisis?
EB: Absolutely. And I think Claire if the thinking as well as the consensus, the agreement on it, that takes time too of course, right. So again, in the heat of the crisis, the CIO, the CISO, and so on, they're pretty busy people. And if you're asking to also not only make these calls, but then go around the key senior executives to get support for those calls, that's not really something you want to be spending your time on when your systems are down all around you.
CP: Ellis thanks so much for joining me today. I really appreciate you providing my audience with the CIOs lens of a cyber incident. It's something that we don't probably talk about enough on the podcast and I really appreciate your time. It's been really valuable. Thank you.
EB: Oh, my pleasure. Thanks for having me Claire.