Category: Best Practices

We all know it: patching is hard. Even well-organized IT departments will at some time fall behind on patching – often without even knowing it. With new vulnerabilities disclosed every day and vendors releasing patch after patch, this is hardly a surprise.

For attackers this makes things all too easy: 60 percent of data breaches involved known vulnerabilities for which patches are available, according to a meta-research conducted by CSO Online.

But there are ways to get on top of your patching. Dedicated patch management solutions are designed to make patching easy and keep you in control and in the know. Here is how a patch management software will help you keep systems up-to-date and protect your business:

Reason 1: You can measure your patching health

How can you protect what you don’t see? Often the main reason why IT departments forget to apply an important patch is that they don’t know that a system is missing a patch in the first place. Vendors are releasing new patches every day. Our patch trackers routinely count more than 200 new patches each month (see our Patch Recap for February 2021 as an example). Keeping up with all the latest patches quickly becomes a full-time job. A patch management solution helps you sort through the noise and understand which patches apply to your systems. You can see at one glance all systems that are currently unpatched and take action immediately. This reduces the time a system remains in a vulnerable state and makes it harder for attackers to take advantage of new vulnerabilities.

Reason 2: You can patch with one click

If you are applying patches manually you are investing a lot of resources into an error-prone process. You need to access every endpoint and start the update process by hand. A better solution is to use scripting languages and automation tools to automate and standardize this process. For Windows-based environments PowerShell is the natural choice. For Linux the Ansible automation tool comes with lots of handy modules that can make patching easier.

If you want to go a step further and streamline the patching process even more have a look at patch management solutions. With these solutions you can patch individual systems with one click from a dashboard or apply patches to group of systems. The result: You save time and reduce the risk of missing systems which then remain vulnerable to attacks.

Reason 3: You can patch Windows, Linux and Mac with one tool

Many organizations use not just Windows but also Linux and Mac as operating systems. Your employee laptops and desktops may all run Windows but what about your Linux-based web server infrastructure and all the Macbooks in the marketing and development departments? 

The diversity of operating systems nowadays makes patching even harder. Many traditional patching tools like Windows Server Update Service (WSUS) have not kept up with this trend (see also our post on WSUS alternatives if you are thinking about switching from WSUS). If you are in this category a cross-platform patch management solution is the right choice for you. These solutions have agents for all major operating systems and let you manage all your endpoints from a single dashboard.

Reason 4: You can test patches

Are you afraid that patches will break a perfectly fine setup? Unfortunately you are right. Although vendors have become much better at releasing stable patches and the majority of patches do not cause any issues there is the occasional hiccup. Such as freshly patched Windows systems descending into a blue-screen-of-death when trying to print a document. 

But not applying patches is also not a solution since this will keep your systems vulnerable. What to do? Test your patches!

Most vendors test new patches extensively before release but it is hard for them to cover all possible environments and configurations. So you better do your own testing on top of that. One way to do this is to set up a group of test devices and connect this group to your patch management solution. You can then apply patches to this group or configure a policy to always patch these systems automatically. After the latest round of patches was applied you check if everything still works. If no errors pop up you can roll out the patches to the rest of your environment.

You can extend this concept and build multiple groups of systems based on how critical errors are for them. For example, you can start by testing patches on dedicated test systems, then roll them out to non-critical workstations, and only after everything looks good continue with more critical systems like servers.

Reason 5: You can automate your patching

Automation is key to having a well-organized and maintainable IT environment. Patching should be no exception to this. Automated patch management lets you define rules about which patches should be applied when. However, it is not the same as putting all your systems on the “Auto Update” setting. A good patch management solution gives you control over the automation process. For example, you can set maintenance windows so that patches are only applied outside of normal business hours.

Not every patching process should be automated though. Some systems need extra care and should only receive well-tested patches (see Reason 4 above). This applies mainly to server patching. Workstations on the other hand, which are a major attack surface for ransomware these days, greatly benefit from an automated patching process since this shortens the time window for attackers to take advantage of new vulnerabilities.

Conclusion

Patching is just a fact of life for IT professionals but it does not need to be the painful experience it still is in many organizations. A good patch management solution makes patching easy, provides smart automations and still keeps you in full control.

If you ask around which patch management software to use for Windows endpoints the first answer usually is: The Windows Server Update Services or short WSUS. WSUS has been around for more than a decade and is made by Microsoft, so it seems an obvious choice. But WSUS is not for everybody. Let’s have a look for which situations Microsoft’s traditional patch management solution is the right tool, where it falls short and what alternatives to WSUS exist.

What is WSUS?

WSUS was released in 2005 by Microsoft to give system administrators a way to centrally manage updates to Windows clients and servers. To set up WSUS in your environment, you need to create a dedicated WSUS server by installing Microsoft Windows Server and then adding the WSUS Server role. You can have one or multiple WSUS servers in your organization, depending on how many endpoints you need to manage and whether you want to distribute the load of downloading patches.

After setting up the server you configure your endpoints to use the WSUS server as the source for updates instead of going to Microsoft’s update servers directly. And that’s it – at least on a very high level. There are several additional steps required to get a stable and well-configured WSUS setup. For a detailed configuration guide see the Microsoft Docs page on WSUS.

What WSUS does well

After you have set up WSUS successfully you can use it to apply updates to your endpoints, either to individual systems or to groups. You can also choose to only apply specific updates, configure automatic patching and many more options. These are the things that WSUS does well:

• Fine-grained control over which patches get applied to which systems
• Good integration into other parts of a Windows-based architecture like Group Policies, Active Directory, etc.
• WSUS now also supports several PowerShell cmdlets which make automation of update tasks easier

In addition to these points there is another advantage of WSUS that makes it an attractive choice for many businesses: it is included in your normal Windows Server licensing fee and does not cost anything extra beyond that. However, although WSUS does not incur additional fees many system administrators have made the experience that it requires a lot of work to maintain. Just search for WSUS questions on Stack Overflow or Reddit to read some of the issues people have experienced. The maintenance effort of WSUS needs to be factored into the total cost of ownership. This leads us to the next section.

Where WSUS falls short

WSUS is more than 15 years old. And although Microsoft is still officially supporting it the core structure of the product has certainly not aged well. This leads to multiple problems, among them:

• It needs a lot of maintenance: WSUS contains many moving parts and after some time problems do pop up in nearly every WSUS setup – often accompanied by hard to decipher error messages. As a consequence, more and more endpoints will miss patches. If this happens to your setup it is time for the infamous WSUS clean-up. Many system administrators use custom-built scripts for these clean-up tasks and the time to write and maintain these scripts need to be factored in when assessing the costs of using WSUS.
• WSUS is Windows-only: Microsoft did build WSUS with a clear focus on pure Windows environments but this is not how many modern environments look like. Nowadays most businesses use a combination of Windows, Linux and Mac systems but everything that is not Windows is left out when it comes to patching through WSUS. If you are looking for patch management software for Linux or Mac WSUS will not help you.
• Not on the network? Not gonna be patched. The WSUS server lives on your internal network so only endpoints that are on-premise or connected via VPN can reach it. This is a challenge for companies that make extensive use of remote work and it can mean remote endpoints miss important patches.

Who should use WSUS and who shouldn’t?

Now that we have looked at the things that WSUS does well and where it falls short we can answer the question: For which type of IT environment is WSUS the right choice? 

If you are a pure Windows shop where all of your devices are on the same network all the time and you have the IT resources to maintain and care for the WSUS server, then Microsoft’s solution to patch management is a good fit.

But what if you have a diverse IT environment with some endpoints running Windows, others Linux and even have some Macs around? What if your employees do not always work from the office but are sometimes traveling or working remotely? And what if your system administrators are already busy enough and don’t have the time to maintain yet another picky server? In this case you should be looking for WSUS alternatives.

What are good WSUS alternatives?

If WSUS does not meet your requirements there are several good alternatives around. Microsoft itself seems to be moving away from WSUS and towards their new service Azure Update Management. It supports patching of Windows and Linux endpoints and different forms of automation. This service is completely based in Microsoft Azure so it makes the most sense for organizations that already have a large Azure footprint. It can be used to manage patching for virtual machines inside Azure as well as on-premise systems. In the latter case, you will need to set up the Log Analytics service for these endpoints. Detailed explanations and documentation can be found on the Microsoft Docs page on Update Management.

For organizations that need advanced features and cross-platform support, the best WSUS alternative is to use a dedicated patch management solution. Most modern patch management software does support patching Windows, Linux and Mac endpoints from a single dashboard and therefore makes it easy to bring heterogeneous IT environments up-to-date quickly. These solutions often come with advanced options to configure patch policies and generate reports so you can always see the current patch status of your environment. Plus, solutions that are cloud-based let you patch every endpoint no matter if it is on-premise or moving around.

Since you are on our blog we of course recommend Patchdeck 😉 But there are many good solutions around.

Can you combine WSUS and patch management solutions?

If your organization already has a working WSUS setup but you are looking for features that WSUS cannot provide on its own, there is no need to completely burn down what is already working. WSUS can work well together with other patch management solutions. For example, you can use your WSUS server simply as a central repository for Windows patches while the process of patching, automating and reporting is offloaded to a dedicated patch management solution. This has the advantage of saving bandwidth as the Windows updates are only downloaded once to your WSUS server and from there distributed to all Windows endpoints. At the same time, you can use your dedicated patch management solution to get an overview of the patching status of your endpoints and have full control over how you organize groups and automations. You can read in our docs how Patchdeck integrates with WSUS.

Conclusion

WSUS has been the natural choice for Window patch management for over a decade but its age is definitely showing. It remains a good tool for organizations that are Windows-only and have all endpoints on-premise all the time. But if you don’t fall into this category Microsoft’s new cloud offering or a dedicated patch management solution are better choices. And even if you already have a WSUS setup working you can combine it with a patch management solution to get the best of both worlds.

To shed some light on this question we simulated the tech stack of a fictional company and used our data set of patch data to analyze how often they would need to apply a security patch or an update to the software they use. Here is what our fictional company uses:

Clients:

• Windows 10 workstations with Microsoft Office

Servers:

• Windows Server 2019
• Windows Server 2012
• Ubuntu Server
• CentOS Server

Virtualization:

• VMware

Development and Operations:

• Docker
• Python
• Jenkins

A real-life company would probably use several additional technologies but we want to keep it simple and cover some of the main components of a typical tech stack.

Four patches a day

For our analysis we took data points from our patch data set for the four-week period between May 14 and June 11. The data shows that all in all the system administrators at our example company would need to apply 83 patches in four weeks which translates to roughly 4 patches per day. This already gives an indication about the effort involved in keeping everything up-to-date and secure. Let’s break the data down a little further.

Microsoft software is probably top of the list when it comes to patching in many companies. During the analyzed period Microsofts patch Tuesday happened on June 9th and comprised of 129 vulnerabilities, 11 of which had a rating of critical. Since our example company uses only four Microsoft products most of these patches did not apply and the sysadmins could happily ignore them and only apply the updates to Windows 10, Windows Server 2019, Windows Server 2012 and Microsoft Office. We did count these updates as single items although they each fixed many vulnerabilities: 81 for Windows Server 2019, 36 for Windows Server 2012, 51 for Windows 10 and 5 for Microsoft Office.

The two Linux servers in our setup received 64 patches: 47 for Ubuntu and 18 for CentOS. Here it is important to note that we did not make further assumptions about the installed packages on both Linux distributions and some of the patches only applied to specific packages so the real patching needs could be lower.

But the server team at our example company was not only kept busy by their Linux systems: VMware released 6 patches in the four-week timeframe, among them the fix for a serious bug that would allow command injection in the VMware Cloud Director.

The development and operations department had a quieter month with only 8 patches. Amongst them were 3 new versions of Docker, 1 new Python release and 4 releases for Jenkins.

Patching takes up a lot of sysadmin time

Our analysis shows that every piece of technology used by our example company needed an update during our analysis timeframe. Estimating the effort needed to apply all these patches is difficult since the mechanics of patching and updating differ widely between technologies. Microsoft updates are (or at least should be) automated in most companies, however, this in many cases only applies to clients and client software (e.g. Windows, Office). For Microsoft server operating systems many teams use at least a partially manual process, for example by using a testing environment and a gradual rollout, to ensure that patches don’t bring down critical servers. The same process applies to Linux servers. Virtualization software often needs special caution while updating because it forms the backbone of a company IT architecture. Updating programming languages like Python does need to be planned carefully as well since it can break application features and make code rewrites necessary. Tools like Docker or Jenkins on the other hand can often be upgraded in a quick fashion.

If we assume a rather optimistic scenario in which it takes a system administrator on average 45 minutes to research, install and test a patch our example company would have spent around 62 man hours in the four weeks timeframe on patching and updating.

Looking at these numbers it is not surprising to see so many companies falling behind on patching their tech stack. Patching takes up around a third of a system administrators time, even in our optimistic scenario which assumes a fair degree of automation and patch management. This back-of-the-envelope calculation already shows how expensive it is to keep all systems patched. However, all this money is still a very good investment when considering how quickly unpatched systems are compromised by attackers. A new report by FireEye shows that 12 percent of vulnerabilities were exploited within one week after a patch was released and 15 percent were exploited within a month (most of the remaining ones were zero days which were exploited even before a patch was available). And a recent advisory by the Cybersecurity and Infrastructure Security Agency (CISA) of the Department of Homeland Security warns that cyber criminals are still actively exploiting several bugs in Pulse Secure VPN servers, nearly a year after patches for these bugs have been released.

Missing patches is dangerous

This also demonstrates the importance of not missing critical patches for your tech stack. You can’t patch what you don’t know. It’s a safe guess that many of the companies that are currently being compromised via the Pulse Secure vulnerabilities have missed this patch all together. It is easy to blame them but our analysis shows: Keeping up with patch publications is a lot of work. While some companies, most notably Microsoft, have well-known dates for releasing patch information, most security patches are released freely throughout a month. During the four-week timeframe there were only 6 days where no new patches were released.

A robust patch alerting system should therefore be part of every patch management program.

Conclusion

Patching your systems is one of the most important things you can do to protect against attacks but with todays diverse tech stacks it has become a high burden on IT teams. Patch management processes that are based on automation, testing and event-driven workflows can help lower this burden.