Crowdstrike wrap-up. Also some other stuff.  26th July 2024

Hi friends,

A merry Friday to you all.

This week I spent most of my writing time on providing a digest of the Crowdstrike update. It's a good event to properly absorb and understand just how interconnected (and fragile) many of our systems currently are. Good lessons are learned each time something this big happens.

Come to think of it, even the "Quick links" section contains mostly news on critical infrastructure and state actors, as is more often the case lately. I'm not sure if this is because of increased bias on my part (it's more relevant to my current day-job), or just more stuff happening on the digital <-> society playing field. Probably a bit of both :-)

Either way, hopefully it provides some interesting reading. Enjoy folks!

Cheers,

Crowdstrike wrap-up

Alright, that was quite the rollercoaster for Crowdstrike and their customers. And a wonderful lesson in just how fragile our digital society can be. One faulty update, not even a malicious one, and entire airports, hospitals, banks and other critical infrastructure fall on their asses. Much to be learned.

I'll try and give an overview of information that came out since last Friday:

• For starters, this is a good article that explains, in some technical detail, what went wrong: link.

In short: the faulty update was part of Crowdstrike's 'Rapid Response' updates, where config updates are pushed without the need for a full software update. They had rolled out several similar updates back in March and April which were more thoroughly tested and found to be working well, which gave them the misguided confidence that these would also be fine. Hence they rolled them out with less testing than was advisable.

They stopped rolling out the new update within the hour, but by then it was too late.

In their post-mortem they offer a list of improvements that they will make, including (in my mind the most important) canary deployments, where updates are rolled out in a staggered fashion to catch issues like this early, before the entire fleet is affected. One could (and should, in my opinion) argue that they should have been doing this for ages, considering they have direct kernel access to millions upon millions of devices. But there you go.

Don't be fooled though into thinking that all other software vendors already do this, or anything else on that list of improvements. Plenty of them will be sighing with relief that they weren't the ones in the spotlight this week, and will look closely at their own processes to see what they can learn from this. And maybe that list of improvements gives you a number of pointed questions to ask, next time you're in the market for new (security) software.

Other Crowdstrike related news and information:

• The landing page on the issue from Crowdstrike themselves, with a preliminary post-mortem, remediation info and a statement from the CEO: link.
• The Crowdstrike issue also impacted Windows 365 cloud vm's, which got stuck in a reboot loop: link.
• Unrelated, but with similar timing to make things worse and more confusing, Microsoft 365 had a large outage: link.
• Malware creators and phishers started to abuse the Crowdstrike news, acting as remediation information and software fixes: link.
• Crowdstrike sent $10 UberEats gift cards to affected parties as way of an apology, which wasn't exactly universally appreciated. To make matters worse, the gift cards didn't work, because Uber flagged them as fraud because of the high usage. Ouch. link.

That's it. If there's more by next week I'll continue the wrap-up service ;-)

Quick links

• Treasury sanctions Russian hackers that breached US water utilities: link.
• How Russia-linked malware cut heat to 600 Ukrainian buildings in deep winter: link.
• North Korean hacking group targeted weapons blueprints, nuclear facilities in cyber campaigns: link.
• Activists accuse proposed UN Cybercrime Treaty of empowering surveillance, repression: link.
• Meta nukes massive Instagram sextortion network of 63,000 accounts: link.
• Google rolls back decision to kill third-party cookies in Chrome: link.

Breaches and leaks

• Verizon to pay $16 million in TracFone data breach settlement: link.
• Major Russian banks hit with DDoS attacks as Ukraine claims responsibility: link
• Los Angeles Superior Court shuts down after ransomware attack: link.
• Greece’s Land Registry agency breached in wave of 400 cyberattacks: link.
• Middle East financial institution hit with six-day DDoS attack: link.
• DeFi exchange dYdX v3 website hacked in DNS hijack attack: link.
• BreachForums v1 hacking forum data leak exposes members’ info: link.
• Columbus reports cyber incident as multiple cities recover from ransomware attacks: link.

Issues and fixes

• Docker fixes critical 5-year old authentication bypass flaw: link.
• Telegram zero-day for Android allowed malicious files to masquerade as videos: link.
• PKfail Secure Boot bypass lets attackers install UEFI malware: link.
• Progress warns of critical RCE bug in Telerik Report Server: link.
• Critical ServiceNow RCE flaws actively exploited to steal credentials: link.