There’s always a lot of attention focused on updating operations technology (OT) software as part of a manufacturing facility’s good cybersecurity practices. But firmware updates are no less important.
As Eric Byres, chief technology officer at aDolus (a supplier of secure supply chain software) noted in a recent blog post, this common IT practice of firmware updates is not so simple for OT professionals. He said: “PLCs cannot just be turned off and patched when tons of logs are racing through a sawmill, dependent on those PLCs safely guiding their journey. Nor can the PLCs controlling the painting process be stopped while an automobile assembly line is still running. They’re not like executive laptops that can be rebooted at midnight when they need the latest Windows update.”
As a result, Byres said, “the need to ensure that an update will have no accidental impact on safety or production inevitably delays patching. Most guidelines for OT patch management suggest that companies push down patches to machines on a priority basis. This takes time. For example, AstraZeneca estimated…that safe patching of OT systems requires 34 working days from when the patch is first released to when the most mission-critical OT device is patched.
Another consideration that can limit timely updates, according to Byres, are certifications. “For example, a SIL Safety Certification is often required in situations where a PLC is controlling a hazardous process. Obtaining certifications is expensive, so equipment manufacturers often don’t pursue certifications for every firmware version. Asset owners may thus need to skip some patches until the next certified version is available,” he said. This means that the patch cycle for OT devices can be years long for processes where high availability is critical. The costs involved in stopping are significant, so typically these patches are scheduled on an annual or biannual basis as part of a normal maintenance cycle. “Or at least they should be,” he said.
Byres said he was shocked to see, while reviewing the distribution of firmware versions across Microsoft customers’ PLCs, that 60% were running “ancient” firmware versions with eight or more exploitable CVEs (common vulnerabilities and exposures). Even more surprising was that the updates that should have been installed had been available for more than 10 years.
As Byres noted: “There are more patching constraints in OT than in IT. But that excuse eventually expires.” He added that the issue of keeping OT systems up to date is “more of a people problem than a technical problem given the availability of the patches.”
Some of the reasons behind this “people problem” are:
- An “if it ain't broke, don’t fix it” philosophy.
- Lack of awareness.
- Procrastination.
The critical upside to keeping your OT firmware updated is a dramatic reduction in risk to cyberattacks—an increasingly important factor as more industrial companies are targeted by hackers.
Byres said that, based on research aDolus conducted with Microsoft, “we determined that if operators updated their firmware to the latest version, the number of devices free of exploitable CVEs would increase from 4% to 40%. Another way to look at the impact of updating those PLCs to the latest version: Asset owners could reduce the percentage of the devices with more than eight exploitable CVEs down to only 18%. Again, that’s still a lot of vulnerable PLCs on the OT network, but the goal should be improvement rather than perfection.”
For those operations that still find it difficult to update their controllers’ firmware, Byres recommends they “at least look at hardening configuration settings or using compensating controls.” However, Byres still advises performing the regular patch updates, because, “in most cases, deploying and maintaining compensating controls takes far more effort than installing patches.”
