This morning when I was thinking about which of the dozens of ideas I have for a COSAC presentation I’d actually submit, I was re-reading some Gartner guidance on preparing for the future of security.
(A brief, totally voluntary PSA: if you’re thinking of speaking at one of the COSAC conferences, the CFP closes today. Even if you don’t attend, I’d highly recommend going.)
One of the things which struck me as particularly amusing in it, maybe because of lack of sleep, or maybe it’s because I got my morning Led Zeppelin fix, was an off-hand observation about some of the skills required in the modern security team:
“Some technologies, especially those using AI, require constant monitoring or investigation by a human security expert.”
I find this hilarious. And even more so given the amount of time I’ve spent playing with knowledge management technologies, expert systems and “big data” since before “big data” was even a thing.
What it says is that the automation that’s supposed to make security’s life easier and more effective…
…actually increases its overall workload. Which, of course, implies that it therefore also increases its stress, its chances of making a mistake because of said stress and fatigue, and the overall likelihood security analysts will run from the building screaming like asylum inmates.
And it means that we’re effectively turning security analysts into air traffic controllers—but with a key difference. While ATC staff work 5 shifts a week of 8 hours, during that shift, they rotate through two states. In the “on position” state, they’re staring at screens, keeping silver bullets from crashing into each other. However, they can only be effective doing this for 90-120 minutes at a time. Then they get a forced 30-minute break to clear their heads.
Oh, and there’s no offline analysis and investigation required when something blips in the wrong place on the screen.
I hear your skepticism, though. “But Andrew…SIEM management and response is nothing like ATC.”
And, in one key point, you’d probably be correct: it’s not that often that people will die if we miss a SIEM alert or get overwhelmed with false positives.
But the FOMO factor is alive and well in most SOCs I know, because there’s an awful lot of false-positives that need to get investigated. Last year’s stats say that almost half of security incident responders reported false-positive rates of 50% *or higher* in their daily work.
With any kind of control, you need to find the sweet spot, and until you do…even if you think it’s not impacting you, I guarantee it’s taking some of your attention away from more important things. In fact, this “lack of squelching” situation reminds me of when I was a kid driving to my music lessons every week with my mother in the ‘70s and ‘80s.
It was a 45 mile trip each way from home to where my music lessons were, and it was all on pretty busy, country highways. The Interstate didn’t go the shortest route, and would’ve added about an hour to trip, almost doubling it.
These were the days when CB radios were all the rage, and since the majority of the people on the road at that time were truckers, my parents also had a CB radio in the car. It was actually pretty interesting to listen to these guys, and there were a few “regulars” we’d encounter every few weeks making a standard trip of their own. However, the one thing that always used to annoy me as a kid – especially if I wanted to listen to one of the 8-Track tapes that lived in the car – was the constant static when nobody was talking.
ChhhhhhhhhhSSSSSSShhhhhhhhhhhccccccccchhhhhhhhSSSSSSSsssssssssBreaker, breaker. How ‘boutcha North-bound.Chhhhhsssshhhshshhhshhshhh
I mean, that was basically what it was. If someone was talking, you heard them, or if they keyed their mic and they were close enough, it’d stop. But otherwise, it was just that static.
For an hour. Each way.
When I asked my mother about it one time, she said, “Well, I know it’s annoying. But I don’t want to miss anyone saying anything about an accident or a Smokey up ahead.”
FOMO. Fear Of Missing Out.
It’s a human thing, and I admit, at the time, I accepted this, and it was just part of the trip. You got used to it, but it was distracting.
Until I started driving.
And then, I decided that I was more interested in rocking out to Guns N’ Roses, INXS, Van Halen, Whitesnake, The Animal House Soundtrack…and all that other good stuff—
But…I still didn’t really want any speeding tickets either. And, I must say that I’ve always agreed with Sammy Hagar’s “I Can’t Drive 55”. I don’t like pissing around when I’m on the road.
So…far too many years later…I decided that the incessant hissing needed to stop. Now the CB radio we had was an old Cobra 19, and it had about a million knobs on the front of it. Growing up, I remember a time when I’d messed with them, and the CB didn’t work for a week, and my parents had to take it to the shop to get it back working again.
They didn’t know what they did, and the message after that was clear: “DO NOT TOUCH THE OTHER SWITCHES AND KNOBS.”
But, finally, the time had come to touch the knobs and figure it out, so I started experimenting. There wasn’t any “just Google the manual” back then, and the instructions had been discarded long, long ago.
Eventually, I found the one labeled “Squelch.” I had no idea what it did, but what I finally figured out was there was a point at which the damn static stopped. But then, if you turned it too far…
…so did everything else.
The trick was finding the spot where you were happy you weren’t missing the ‘20s of the Smokies in the bushes, but you didn’t get the headache from the endless static.
Once I figured this out, there was a whole lot more “Paradise City” with the added confidence that I wouldn’t miss any important updates.
But our SOC friends don’t have a simple squelch control, because there’s potentially a bazillion different things that might “mean something” we need to know. And if we were to miss it, then people would wonder what good it did to spend all those millions of dollars on the tech that was supposed to keep up safe, when it was clearly not really up to the job.
We. Must. Let. Go.
You have to find our squelch, or you’re going to go insane, and the rotating door of security analysts will never stop spinning. And finding that squelch means knowing what matters and what doesn’t.
The big myth is that the answer to what matters and what doesn’t isn’t in the weeds of terabytes of log messages. The answer to what matters is understanding your world and how it actually supports the business.
Of course, that means you have to have the visibility of…your real security architecture.
Because without it, your security program isn’t just stuck listening to static…
…it’s broadcasting that static to the rest of the organization loudly enough to make them think they were in the audience in Rio for the 2006, record-setting Rolling Stones concert, A Bigger Bang.
How much luck do you think you’d have trying to set the strategic direction of a $100 billion company…
…or even smooth the ruffled feathers of an irate customer on the front lines…
…when you have SSSHhhhhhhhhsssssSCCCCCCHHHHHH blasting in your ears at 120+ dB?
Probably not much. And it probably isn’t making you very popular in your organization either.
What to do…what to do…what to do….
Well, one thing you could do if you wanted to find the squelch button, kill the alert fatigue and focus on actually keeping the organization safe instead of flailing against false-positives is figure out what you have, what really matters, and where you should spend your team’s time. And one possible way to do that might be to have a chat with me about it to see what we might be able to do together. I’ve no idea whether I can help you or not, but neither of us will know until we talk about it a bit.
If you’d like to get some different ideas beyond the typical “buy more tech”, “deploy more controls” and “outsource everything” advice, you can book a screening call to see if working together every week as part of Archistry’s Effective Security Leadership Coaching program might be a better way forward. To do that, just visit this link:
https://securityleadershipcoaching.com
If not, then might I recommend making sure your pain pills are the ones with the 65 mg of added caffeine. I know those are the ones that work best for me.
Stay safe,
ast
—
Andrew S. Townley
Archistry Chief Executive
