Last Sunday marked the two-year anniversary of a huge lesson in risk and complexity: a CrowdStrike software bug took millions of Windows machines offline and ground businesses to a halt.

▶️ How did it happen? ◀️

Software bugs are nothing new. Most are small and inconsequential.

In the case of CrowdStrike, a software bug existed deep in a privileged space of the operating system. It caused the affected machine to lock up.

Even worse, customers didn't request the update on their own terms – CrowdStrike pushed it to them, en masse. One mistake went worldwide in an instant.

▶️ What can we learn from it? ◀️

At a high level, CrowdStrike is a lesson in technology change management. Since software bugs are inevitable, we need stronger risk controls to protect us against updates gone awry. One big risk control is to let customers choose when to update – this way they can roll out the change in phases and on their own schedule.

There's a second lesson that is specific to genAI: even though genAI reliability has been uneven, companies insist on pushing AI functionality into products and business-critical processes. Some are even connecting genAI agents to commerce and automated purchasing decisions. Without risk controls in place, we can expect more frequent incidents and perhaps more damaging than what happened with CrowdStrike.

▶️ The take-away ◀️

It's never too soon to think about technology risks and risk management.

▶️ Read more ◀️

Want a more in-depth look at what happened, and what to learn from it?

The CrowdStrike incident gave me so much to think about that it took two issues of my newsletter to cover it all. See "Blue Screen for Armageddon" and "Embers and Ashes."