Cybersecurity News

Microsoft Releases USB Recovery Tool After CrowdStrike Update Causes BSOD on 8.5M Systems

23 July 2024

|

Zaker Adham

Summary

When reboots fail to resolve issues, bootable USB drives might offer a solution for some PCs.

By Monday morning, the significant disruptions caused by last week's faulty CrowdStrike security update had largely subsided. Major flight delays and cancellations were no longer making headlines, and nearby Starbucks locations had resumed app orders.

However, the cleanup is ongoing. Microsoft estimates that approximately 8.5 million Windows systems were impacted by the flawed update, which involved a problematic .sys file automatically pushed to PCs running CrowdStrike Falcon security software. This update led to the infamous Blue Screen of Death (BSOD) and initiated boot loops on affected systems.

David Weston, Microsoft VP of Enterprise and OS Security, noted in a blog post, "While software updates can sometimes cause issues, major incidents like the CrowdStrike event are rare. We estimate that the update affected 8.5 million Windows devices, representing less than one percent of all Windows machines. Despite the small percentage, the broad economic and societal impacts highlight the reliance on CrowdStrike by enterprises running critical services."

The primary fix recommended by both CrowdStrike and Microsoft involved repeatedly rebooting affected systems to download a new update file before crashing. For systems where this method was unsuccessful—Microsoft advised up to 15 reboots—manually deleting the problematic .sys file was suggested. This approach enables the system to boot and download a corrected file, resolving the crashes without compromising security.

To simplify this process, Microsoft released a recovery tool over the weekend. This tool automates repairs on some affected systems by creating bootable media with a 1GB to 32GB USB drive, booting from it, and offering two repair options. For devices unable to boot via USB—often due to corporate security settings—Microsoft provided a PXE boot option for network-based booting.

WinPE to the Rescue

The bootable drive utilizes the WinPE environment, a lightweight, command-line version of Windows typically used by IT administrators for applying Windows images and performing recovery operations.

One repair option boots directly into WinPE and removes the affected file without needing administrator privileges. However, if your drive uses BitLocker or another disk-encryption product, you must manually enter your recovery key so WinPE can access the drive and delete the file. According to Microsoft's documentation, the tool should automatically delete the faulty CrowdStrike update once it can read the disk.

If using BitLocker, the second recovery option attempts to boot Windows into Safe Mode using the recovery key stored in your device's TPM to unlock the disk automatically, as it does during a normal boot. Safe Mode loads the minimum drivers needed for Windows to boot, allowing you to locate and delete the CrowdStrike driver file without encountering the BSOD issue. The file is found at Windows/System32/Drivers/CrowdStrike/C-00000291*.sys on affected systems, or users can run "repair.cmd" from the USB drive to automate the fix.

CrowdStrike has established a "remediation and guidance hub" for affected customers. As of Sunday, the company reported testing a new technique to accelerate system remediation but has not provided further details. Other recommended fixes include multiple reboots, manual deletion of the affected file, or using Microsoft's boot media to automate the repair.

The CrowdStrike update's impact extended beyond flight delays and coffee orders, affecting doctor's offices, hospitals, 911 services, hotel check-in systems, and work-issued computers that were online during the update. In addition to providing fixes for client PCs and virtual machines in its Azure cloud, Microsoft has collaborated with

Google Cloud Platform, Amazon Web Services, and other cloud providers to address issues with Windows VMs running in their environments.