Free Trial
Search
Close this search box.

Fast CrowdStrike Recovery should be a reality for users of System Recovery software

CrowdStrike recovery is an activity and topic of focus for many organizations following the CrowdStrike bug that caused a global IT outage on July 18th and 19th, 2024. The number of users effected is unknown, however, we can glean some insights from news reports:

  • Widespread Disruption: News articles describe the outage as widespread, disrupting airlines, financial institutions, hospitals, and businesses.
  • Large Organizations: The affected entities suggest a significant number of users, potentially impacting thousands or even millions of devices.


It is clear the CrowdStrike update bug caused significant disruption for many organizations.

What caused the CrowdStrike Falon update outage?

The outage was caused by a faulty update for the Windows version of their Falcon sensor.
Here’s a breakdown of the issue:

  • Falcon Sensor: This is a core component of the CrowdStrike Falcon platform that runs locally on user devices and scans them for potential malware threats.
  • Faulty Update: A specific update for the Windows version of the Falcon sensor contained a defect.
  • System Crash: This defect triggered a “logic error” that caused the affected systems to crash and enter a continuous restart loop, often referred to as the “Blue Screen of Death” (BSOD) on Windows machines.


Here’s some additional information:

  • Limited Scope: The issue only affected the Windows version of the Falcon sensor, not Mac or Linux systems.
  • Recovery: CrowdStrike identified the issue and deployed a fix. However, restarting the affected systems multiple times might have been necessary to complete the recovery process.


It’s important to note that CrowdStrike has not released any official reports detailing the exact cause of the bug within the update. However, based on the available information, it appears to be a software error within the update itself that caused the system crashes.

How would Cristie Software customers recover faster from the CrowdStrike update crash?

CrowdStrike recovery presents a perfect use case for automated system recovery. In this explainer video, Sky News business correspondent Paul Kelso outlines the laborious manual process required to recover systems to a state that allows for deletion of the disruptive CrowdStrike driver file. Users with large server estates that do not utilize automated system recovery or boot management tools would face a significant amount of manual intervention and downtime in order to facilitate driver removal from all effected machines. Cristie Software bare machine recovery (BMR) provides system recovery from leading backup solutions such as Rubrik Security Cloud, Cohesity DataProtect, IBM Storage Protect and Dell Technologies backup solutions Avamar and Networker. Using Cristie recovery software automation, the following steps would be required to recover effected machines to a point before the disruptive CrowdStrike driver was applied:

  • Reboot in DR environment: Reboot systems into DR environment (this can be automated using boot management tools with our web-boot ISOs).
  • Recover systems to last known good point in time: Trigger recovery from backup server (Rubrik, Cohesity, IBM or Dell).
  • Reboot system: Machines would reboot to the last known good state prior to the application of the disruptive driver.

What is the manual CrowdStrike update recovery process?

The recovery process for the CrowdStrike Falcon update bug depended on the severity of the issue and your access to the affected system. Here are the two main approaches taken from online research. Users effected by the CrowdStrike update bug should conduct their own due diligence and refer to CrowdStrike support services to verify the procedure for their specific environment:

  1. Booting into Safe Mode or Windows Recovery Environment (WinRE):
    This method was recommended by CrowdStrike for situations where the system continuously rebooted into a loop (BSOD). Here’s how it worked:
    • Boot into Safe Mode: This can be achieved through various methods depending on your system configuration. One common approach is to repeatedly press the F8 key during system startup.
    • OR Boot into WinRE: If Safe Mode is inaccessible, you can try booting into the Windows Recovery Environment (WinRE). This may involve using a bootable USB drive or recovery media provided by your system manufacturer.
    • Navigate to the Target Directory: Once in Safe Mode or WinRE, locate the folder containing the problematic CrowdStrike files. The specific path might vary, but it’s typically something like C:\Windows\System32\drivers\CrowdStrike.
    • Delete the Faulty File: Look for a file named “C-00000291*.sys” (the asterisk represents any wildcard characters) and delete it.
    • Reboot Normally: After deleting the file, attempt to reboot your system normally. If the issue was resolved, the system should boot up successfully.
  2. Detaching the Disk from a Virtual Server (Advanced Users):
    This option was suitable for virtualized environments where the affected system was running on a virtual machine (VM). It’s important to note that this method requires technical expertise and should only be attempted by experienced users. Here’s a simplified overview:
    1. Detach Disk: Detach the virtual disk volume from the impacted virtual server. Create a backup or snapshot of the disk volume as a precaution.
    2. Mount Disk on Another Server: Attach or mount the detached disk volume to a separate virtual server with a working CrowdStrike installation.
    3. Access and Delete File: Follow steps similar to the Safe Mode method to access the C:\Windows\System32\drivers\CrowdStrike directory and delete the “C-00000291*.sys” file.
    4. Reattach Disk and Reboot: Detach the disk from the temporary server, reattach it to the original impacted virtual server, and attempt a normal reboot.


Additional Tips:

  • Consult CrowdStrike Support: If you are unsure about the recovery process or encounter difficulties, it’s advisable to reach out to CrowdStrike support for assistance.
  • Test Functionality: Once your system boots up successfully, verify that your CrowdStrike Falcon sensor is functioning correctly.


Remember: These are general guidelines taken from online resources. The specific steps may vary depending on your system configuration and the severity of the issue. It’s always best to consult with a qualified IT professional if you are unsure about any of the recovery procedures.

Conclusion

The CrowdStrike driver update failure has demonstrated how vulnerable enterprises are to system level driver changes that have the ability to disrupt the boot process of any operating system. Most companies invest in data backup solutions to safeguard application data but many fail to implement system recovery solutions that capture operating system configurations with the ability to restore complete systems to any available point in time. Furthermore, automated system recovery solutions such as the Cristie BMR suite which offer automation for physical machine recovery can eliminate manual intervention from the recovery process, potentially saving hours of administrative overhead when large scale server estate recovery is needed.

Contact the Cristie Software team if you have been effected by the CrowdStrike update failure and would like to learn more about system recovery and recovery automation.

Cristie Software Ltd

New Mill, Chestnut Lane
Stroud, GL5 3EW
United Kingdom

©2024 Cristie Software Ltd. Registered in England & Wales Number: 06395450
Youtube Linkedin Facebook Twitter Github
cristie software_logo-02

Contact Us

Thank you for contacting us. We have received your request.