Disaster preparedness is a crucial aspect of maintaining operational stability and resilience in today’s technology-driven world. Recent events have underscored the significance of having robust disaster recovery plans in place.
Continue reading to find out more about the largest IT outage in history, who was affected, how to fix the BSOD (blue screen of death) and what you can do to prepare for an IT outage in the future.
The Largest IT Outage in History – Crowdstrike
On a seemingly ordinary Friday, July 19th, a mass IT outage caused by an overnight content update by cybersecurity company CrowdStrike resulted in widespread disruption. The update crippled Microsoft’s PC operating systems, leading to significant consequences across various sectors. Problems with Microsoft’s 365 suite of apps and Azure cloud computing product were reported by users worldwide, with issues peaking around 1 a.m. Eastern time.
CrowdStrike confirmed that the update was responsible for the crashes, which left many PC users facing the infamous Blue Screen of Death (BSOD). The company’s CEO, George Kurtz, apologized for the global tech outage, stating that a fix had been deployed but acknowledging that solving the problem could take some time. This incident, described as potentially the largest IT outage in history, had far-reaching effects.
Fixing the BSOD: A Complex Challenge
While CrowdStrike has implemented a fix for the update that originally caused widespread problems, many systems remain offline. Banks, airlines, supermarkets, and TV broadcasters are struggling to operate without their critical machines. The process to resolve the Blue Screen of Death (BSOD) issue, however, is not straightforward.
Immediate Workaround Steps
IT administrators have been employing an initial workaround provided by CrowdStrike. This method involves booting Windows systems into Safe Mode and deleting a problematic system file. The steps are as follows:
- Boot Windows into Safe Mode or the Windows Recovery Environment.
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
- Locate the file matching “C-00000291.sys” and delete it.*
- Reboot the host.
These steps force Windows to boot into a Safe Mode environment where third-party drivers, such as CrowdStrike’s kernel-level driver, do not load. IT admins must then locate and delete the faulty driver on the disk. This workaround often requires physical access to the machine, and complications can arise due to disk encryption (e.g., BitLocker) or insufficient admin rights.
Challenges in Applying the Fix
The workaround method is not without its challenges. Physical access to machines is often necessary, and environments with disk encryption or restricted admin rights add layers of complexity. Additionally, IT admins in some cases are simply rebooting machines repeatedly, hoping the CrowdStrike update will push through before the system crashes again. Surprisingly, turning machines off and on again has worked for some, with reports of machines coming back online after multiple reboots.
CrowdStrike’s update servers and content delivery networks are under immense pressure from millions of machines seeking updates. This demand means the reboot method may take some time to be effective.
Alternative Solutions for Different Environments
Businesses running virtual desktops may recover more quickly by restoring affected hosts to a state before the faulty update. In cases where repeated reboots do not work, booting into Safe Mode remains the best option.
Long-term Recovery Efforts
Resolving this issue will not happen in a matter of hours, unlike typical internet outages from cloud providers. CrowdStrike CEO George Kurtz has emphasized that it may take some time for all systems to recover fully. The company’s mission is to ensure every customer is fully operational as soon as possible.
In his interview with NBC News, Kurtz apologized for the disruption caused by the faulty update. However, questions remain about how such an update could impact thousands or even millions of machines globally.
Impact on Various Sectors
The outage grounded major U.S. airlines, including American, Delta, and United, causing widespread travel disruptions. Banks in Australia, South Africa, and the U.K. were also affected, leading to significant financial sector interruptions. Media outlets, including British TV channel Sky News, faced broadcasting challenges, while the London Stock Exchange experienced delays in opening and news services.
Lessons in Disaster Preparedness
The CrowdStrike incident highlights several critical lessons in disaster preparedness:
- Cybersecurity Vigilance: The incident emphasizes the need for continuous monitoring and proactive management of cybersecurity measures. Despite being a non-security-related issue, the disruption caused by CrowdStrike’s update demonstrates how crucial it is to ensure that all updates and changes are thoroughly tested before implementation.
- Redundancy and Backup Systems: Businesses must have robust redundancy and backup systems in place to mitigate the impact of IT outages. This includes maintaining on premise backup servers, data centers, and alternative communication channels to ensure continuity of operations during a crisis.
- Flexible and Scalable Solutions: Leveraging technology rentals is a smart move for disaster recovery. Renting equipment such as laptops, servers, routers, and office equipment (copiers, scanners, printers) provides businesses with the flexibility and scalability needed to maintain operations during emergencies. For example, during the hurricane season in 2021, a company in Texas rented hundreds of laptops to ensure its operations kept running smoothly.
- Effective Communication Plans: Clear and effective communication is essential during a disaster. Businesses must have communication plans in place to inform employees, customers, and stakeholders about the situation and the steps being taken to address it.
- Comprehensive Disaster Recovery Plans: A well-structured disaster recovery plan should include detailed procedures for responding to various types of disruptions, including IT outages. This plan should be regularly reviewed and updated to account for new risks and vulnerabilities.
Let us be your Disaster Recovery Plan
The largest IT outage in history serves as a stark reminder of the importance of disaster preparedness. By investing in robust cybersecurity measures, implementing redundancy and backup systems, leveraging technology rentals, and maintaining effective communication and recovery plans, businesses can better navigate and mitigate the impact of unexpected disruptions. Disaster preparedness is not just a precaution; it is a vital component of ensuring business continuity and resilience in an increasingly digital world.
In the wake of unforeseen IT disasters like the recent CrowdStrike-induced outage, the importance of a solid disaster recovery plan cannot be overstated. At Rentacomputer.com, we understand that maintaining business continuity during crises is paramount. Our technology rental solutions provide the flexibility, scalability, and reliability that businesses need to navigate disruptions smoothly.
From laptops and servers to routers and printers, we offer a wide range of rental equipment tailored to your specific needs. Whether you’re facing a temporary office shutdown, IT outages, or any other unexpected event, our rentals ensure that your operations remain uninterrupted.
Our team of experts is ready to support you with quick deployment and seamless integration, minimizing downtime and maximizing productivity.
Don’t let an IT disaster catch you off guard. Partner with Rentacomputer.com and let us be your disaster recovery plan. With our technology rental solutions, you can face any challenge with confidence, knowing that your business will continue to thrive no matter what comes your way.
When you choose to rent from us, you not only receive the finest equipment but also receive exceptional support from expert tech travel agents with extensive experience in technology rentals and event production. Our agents manage all delivery and installation logistics, allowing you to focus on your top priorities.
Our account manager Randy Moore specializes in exhibitor rentals like displays, kiosks, event engagement rentals, laptops, tablets, smartphones, internet hotspots, accessory rentals, and much more. Randy will gladly handle any questions you may have regarding rentals to make your event exhibition easier. Contact Randy today at 1-800-736-8772 ext. 225 or fill out a fast, easy online quote form.
Visit our PC & AV Rental Pricing Factors page to find out about our pricing estimates.
Related reading