On July 19, 2024 I posted on X: <\/p>\n\n\n\n
\nMassive GLOBAl IT outage on Windows devices traced back to a BAD PATCH issued by Crowdstrike which caused a domino effect\u2026 NOT a cybersecurity event) Crowdstrike holds 24%of the market – that is why some businesses were affected and others were not\u2026<\/em><\/p>\n<\/blockquote>\n\n\n\n
The primary reason is that as it was hitting the United States news cycle that morning – there was lots of finger pointing to Microsoft for the outage, and after being up since 2:30am watching \/ listening to the outage as it unfold across Australia, Asia Europe for a few hours already, the cause was already known and frankly I was a bit annoyed at the initial US coverage. <\/p>\n\n\n\n
Sunday (yesterday ) 7\/21, a ran across GREAT video explaining why and how this happened was published by Dave Plummer on his Dave’s Garage <\/a>YouTube channel. In it he clearly, efficiently and in plain language (English) explains the situation… <\/p>\n\n\n\n
Key to remember, 1) the obvious: we need to make sure patches are well tested before releasing them, but more importantly 2) I think we will need some changes to the architecture of tools like CrowdStrike so this does not happen again. <\/p>\n\n\n\n
The key points of his video are: <\/p>\n\n\n\n
\n
- The CrowdStrike issue caused widespread blue screens (system crashes) due to a bad software update.<\/li>\n\n\n\n
- CrowdStrike’s Falcon sensor operates as a kernel-mode driver, giving it complete access to system data structures and services.<\/li>\n\n\n\n
- Kernel-mode code crashes cause system-wide crashes (blue screens), unlike user-mode application crashes.<\/li>\n\n\n\n
- CrowdStrike uses dynamic definition files to update their driver without going through the Windows Hardware Quality Labs (WHQL) certification process each time.<\/li>\n\n\n\n
- The issue stemmed from processing dynamic definition files containing executable code<\/strong>, not just data, which the driver executed without proper validation.<\/li>\n\n\n\n
- The crash was likely caused by the driver attempting to process an update file (CY file) that contained all zeros instead of the expected data.<\/li>\n\n\n\n
- The specific error involved dereferencing a null pointer, indicating inadequate error checking and parameter validation in the driver.<\/li>\n\n\n\n
- CrowdStrike’s driver is marked as a boot driver, making it difficult to boot into safe mode to fix the issue.<\/li>\n\n\n\n
- A temporary fix involves deleting the problematic update file (matching the pattern C0000291<\/em>.sys) from the Windows\\System32\\drivers folder.<\/li>\n\n\n\n
- The incident highlights the risks of running untrusted code in kernel mode and the importance of robust error handling in driver development.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
CrowdStrike blog posts with <\/strong>Analysis Report and remediation steps can be found at: <\/p>\n\n\n\n
\n
- https:\/\/www.crowdstrike.com\/falcon-content-update-remediation-and-guidance-hub\/<\/a><\/li>\n\n\n\n
- https:\/\/www.crowdstrike.com\/blog\/falcon-update-for-windows-hosts-technical-details\/<\/a><\/li>\n\n\n\n
- YouTube video with remediation steps: https:\/\/www.youtube.com\/watch?v=Bn5eRUaMZXk<\/a><\/li>\n<\/ul>\n\n\n\n
His Video, well worth the watch, follows: ( very well done Dave!) <\/p>\n\n\n\n
<\/p>\n\n\n