Categories
How To

How to do server patching right

Servers are the backbone of modern business. They run applications, handle email and manage backups. Server downtime can cost real money so many administrators get sweaty hands when they think about patching their servers.

But patch they must: Precisely because they are so critical for many organizations, servers are a prime target for ransomware gangs. An encrypted laptop is a nuisance for a single employee and can be quickly rebuilt. But when a critical server is affected by ransomware, chances are that nobody can do any work anymore. This is why patch management for servers is so important.

And the good news is: patching your servers does not need to be a panic-inducing process. With a bit of planning and the right tools, you can tame the beast of server patching.

Here is our 6-step-checklist how to create a server patching process that follows best practices.

Server Patching Checklist

Step 1: Inventory

Before you think about how to patch you need to know what to patch. Compiling an accurate list of all your servers is the first step to a successful server patching process. An asset management tool is the best way to get a detailed and up-to-date overview of your server inventory. Two good tools are (no affiliation here, just based on good experiences we have made ourselves):

The big advantage of using an asset management tool compared to a simpler solution like an Excel spreadsheet is that with the first one you always get a live view of your environment. When you decommission systems or add new ones an asset management tool will pick up these changes automatically.

Armed with the output of your asset management tool you can now start to organize your inventory so that you can make sensible patching decisions.

Step 2: Group

Not all servers are created equal. Some are mission-critical such as the primary web servers for a Software-as-a-Service company or the servers running the customer database for a professional service firm. Others are important but can survive a short downtime such as a time tracking system. And others fall into the “nice-to-have” category, like the server running your intranet homepage.

To prepare your server patching process it makes sense to create groups according to these criticality ratings. You can create these groups in your asset management tool or your patch management solution.

Step 3: Observe

Having visibility into your server landscape is one thing, but you also need visibility into the never-ending flow of patch releases and identify which patches are relevant to your systems. 

For some vendors, this is easier than for others. Microsoft releases most new patches on the second Tuesday of every month, aka Patch Tuesday. But there are exceptions to this rule: When Microsoft discovers an especially critical vulnerability that needs to be patched right away they sometimes release out-of-band patches.

And if your servers run Linux chances are you have no equivalent to Patch Tuesday since most distributions do not follow a fixed schedule for patch releases.

To keep up with this you need a process to track relevant patches. One way to do this is to subscribe to the relevant mailing lists for your software. Most Linux distributions have a mailing list dedicated to patches and new releases. Microsoft provides a web portal and an API for querying patch release information for their products: https://msrc.microsoft.com/update-guide/

You can also use the Patch Alert functionality in Patchdeck to receive alerts when there are new patches for the software you use.

Another good source of patch information is the Cybersecurity & Infrastructure Security Agency (CISA) who publishes several mailing lists. However, note that these mailing lists mainly focus on high-risk vulnerabilities and do not include all patches that are relevant to your systems.

Step 4: Plan and Prioritize

Now that you know which patches you need to apply to which systems, you can draw up a battle plan. Start with the groups you created in step 2: For each group decide which patches you want to apply. For the non-critical systems you can normally apply all outstanding patches without much thinking. But handling the mission-critical servers you may need to be a bit more picky. You can for example decide to only apply security patches. 

In any case: It makes sense to write down your patching plan and share it with your team members so that everybody is on the same page. One way to do this is via the note-taking functionality of your asset management tool.

Step 5: Test

There is no better way to reduce server patching anxiety than to run tests. Especially if you manage critical servers you should invest the time to build a test environment that closely mimics the production environment. You can then apply all patches you selected in Step 4 to the test systems and check if everything is still working as expected. 

Virtualization, containers and automation tools like Ansible or Terraform make it easy to stand up complex environments in a few minutes and to automatically tear everything down after your testing is done.

If you want to play advanced mode you can combine this with a patch management solution to build an integrated workflow where an automation tool stands up a testing environment, the patch management solution applies all new patches and testing scripts ensure that everything still works. Finally, the automation tool tears down the test environment and generates a report.

Not every environment needs a complex patch testing workflow though. If your servers can be quickly rebuilt and downtime does not matter much you can skip the testing and just apply all necessary patches right away.

In this case you should definitely use automation. Most patch management tools have patch policy features that allow you to configure automated patching based on maintenance windows or patch features.

Step 6: Apply

After you have run all your tests and everything looks good you can finally proceed to the most important step: Applying the patches. Some servers can be patched in bulk based on the groups you created in step 2. For others you will probably apply patches one-by-one.

In general, even if you have good tests, it makes sense to roll out patches to production systems slowly and in waves to catch errors early and to always have some systems in a known good configuration.

What is a sensible server patching schedule?

If you have applied all patches it’s time to celebrate! You have mastered the challenge of server patching and have successfully secured your most valuable assets against attacks.

But, the thing with patching is: Soon you will have to start the whole process all over again because new patches will be released. This leads to the question: How often should you patch? What is a sensible server patching schedule?

Most patches are not so critical that you need to apply them right away. A good rule of thumb is to run the process described above at least once every two weeks. This gives you enough time for thorough testing and still ensures that you plug security holes quickly enough before attackers can target them at a large scale.

However, you should use the methods described in step 3 to always be on the lookout for particularly critical vulnerabilities that need to be patched immediately. If Microsoft releases an out-of-band patch that applies to your servers you should drop what you are doing and start patching right away. These patches fix vulnerabilities that are so critical that exploitation will start very soon. Or, even worse: Attacks have already begun.

This does not only apply to Microsoft. In January a patch was released for all major Linux distributions fixing a privilege escalation vulnerability in the Sudo command (Ubuntu advisoryCentOS/RedHat advisory). A patch like this should also be applied immediately because of its big potential impact.

Conclusion

Server patching does not need to be a nightmare. If you follow best practices, build a sound server patching process and use helpful tools like asset management and patch management solutions you will reduce the stress that comes with applying changes to your most critical pieces of infrastructure.

Try out a modern patch management solution

Patchdeck helps you stay on top of your patching with advanced automations, customizable alerts, detailed reporting and more. Check out our cool features and start your free 14-day trial!

Categories
Best Practices

What is a good WSUS alternative?

If you ask around which patch management software to use for Windows endpoints the first answer usually is: The Windows Server Update Services or short WSUS. WSUS has been around for more than a decade and is made by Microsoft, so it seems an obvious choice. But WSUS is not for everybody. Let’s have a look for which situations Microsoft’s traditional patch management solution is the right tool, where it falls short and what alternatives to WSUS exist.

What is WSUS?

WSUS was released in 2005 by Microsoft to give system administrators a way to centrally manage updates to Windows clients and servers. To set up WSUS in your environment, you need to create a dedicated WSUS server by installing Microsoft Windows Server and then adding the WSUS Server role. You can have one or multiple WSUS servers in your organization, depending on how many endpoints you need to manage and whether you want to distribute the load of downloading patches.

After setting up the server you configure your endpoints to use the WSUS server as the source for updates instead of going to Microsoft’s update servers directly. And that’s it – at least on a very high level. There are several additional steps required to get a stable and well-configured WSUS setup. For a detailed configuration guide see the Microsoft Docs page on WSUS.

What WSUS does well

After you have set up WSUS successfully you can use it to apply updates to your endpoints, either to individual systems or to groups. You can also choose to only apply specific updates, configure automatic patching and many more options. These are the things that WSUS does well:

  • Fine-grained control over which patches get applied to which systems
  • Good integration into other parts of a Windows-based architecture like Group Policies, Active Directory, etc.
  • WSUS now also supports several PowerShell cmdlets which make automation of update tasks easier

In addition to these points there is another advantage of WSUS that makes it an attractive choice for many businesses: it is included in your normal Windows Server licensing fee and does not cost anything extra beyond that. However, although WSUS does not incur additional fees many system administrators have made the experience that it requires a lot of work to maintain. Just search for WSUS questions on Stack Overflow or Reddit to read some of the issues people have experienced. The maintenance effort of WSUS needs to be factored into the total cost of ownership. This leads us to the next section.

Where WSUS falls short

WSUS is more than 15 years old. And although Microsoft is still officially supporting it the core structure of the product has certainly not aged well. This leads to multiple problems, among them:

  • It needs a lot of maintenance: WSUS contains many moving parts and after some time problems do pop up in nearly every WSUS setup – often accompanied by hard to decipher error messages. As a consequence, more and more endpoints will miss patches. If this happens to your setup it is time for the infamous WSUS clean-up. Many system administrators use custom-built scripts for these clean-up tasks and the time to write and maintain these scripts need to be factored in when assessing the costs of using WSUS.
  • WSUS is Windows-only: Microsoft did build WSUS with a clear focus on pure Windows environments but this is not how many modern environments look like. Nowadays most businesses use a combination of Windows, Linux and Mac systems but everything that is not Windows is left out when it comes to patching through WSUS. If you are looking for patch management software for Linux or Mac WSUS will not help you.
  • Not on the network? Not gonna be patched. The WSUS server lives on your internal network so only endpoints that are on-premise or connected via VPN can reach it. This is a challenge for companies that make extensive use of remote work and it can mean remote endpoints miss important patches.

Who should use WSUS and who shouldn’t?

Now that we have looked at the things that WSUS does well and where it falls short we can answer the question: For which type of IT environment is WSUS the right choice? 

If you are a pure Windows shop where all of your devices are on the same network all the time and you have the IT resources to maintain and care for the WSUS server, then Microsoft’s solution to patch management is a good fit.

But what if you have a diverse IT environment with some endpoints running Windows, others Linux and even have some Macs around? What if your employees do not always work from the office but are sometimes traveling or working remotely? And what if your system administrators are already busy enough and don’t have the time to maintain yet another picky server? In this case you should be looking for WSUS alternatives.

What are good WSUS alternatives?

If WSUS does not meet your requirements there are several good alternatives around. Microsoft itself seems to be moving away from WSUS and towards their new service Azure Update Management. It supports patching of Windows and Linux endpoints and different forms of automation. This service is completely based in Microsoft Azure so it makes the most sense for organizations that already have a large Azure footprint. It can be used to manage patching for virtual machines inside Azure as well as on-premise systems. In the latter case, you will need to set up the Log Analytics service for these endpoints. Detailed explanations and documentation can be found on the Microsoft Docs page on Update Management.

For organizations that need advanced features and cross-platform support, the best WSUS alternative is to use a dedicated patch management solution. Most modern patch management software does support patching Windows, Linux and Mac endpoints from a single dashboard and therefore makes it easy to bring heterogeneous IT environments up-to-date quickly. These solutions often come with advanced options to configure patch policies and generate reports so you can always see the current patch status of your environment. Plus, solutions that are cloud-based let you patch every endpoint no matter if it is on-premise or moving around.

Since you are on our blog we of course recommend Patchdeck 😉 But there are many good solutions around.

Can you combine WSUS and patch management solutions?

If your organization already has a working WSUS setup but you are looking for features that WSUS cannot provide on its own, there is no need to completely burn down what is already working. WSUS can work well together with other patch management solutions. For example, you can use your WSUS server simply as a central repository for Windows patches while the process of patching, automating and reporting is offloaded to a dedicated patch management solution. This has the advantage of saving bandwidth as the Windows updates are only downloaded once to your WSUS server and from there distributed to all Windows endpoints. At the same time, you can use your dedicated patch management solution to get an overview of the patching status of your endpoints and have full control over how you organize groups and automations. You can read in our docs how Patchdeck integrates with WSUS.

Conclusion

WSUS has been the natural choice for Window patch management for over a decade but its age is definitely showing. It remains a good tool for organizations that are Windows-only and have all endpoints on-premise all the time. But if you don’t fall into this category Microsoft’s new cloud offering or a dedicated patch management solution are better choices. And even if you already have a WSUS setup working you can combine it with a patch management solution to get the best of both worlds.

Try out a modern patch management solution

Patchdeck helps you stay on top of your patching with advanced automations, customizable alerts, detailed reporting and more. Check out our cool features and start your free 14-day trial!

Categories
Best Practices

5 reasons why you need a patch management software

We all know it: patching is hard. Even well-organized IT departments will at some time fall behind on patching – often without even knowing it. With new vulnerabilities disclosed every day and vendors releasing patch after patch, this is hardly a surprise.

For attackers this makes things all too easy: 60 percent of data breaches involved known vulnerabilities for which patches are available, according to a meta-research conducted by CSO Online.

But there are ways to get on top of your patching. Dedicated patch management solutions are designed to make patching easy and keep you in control and in the know. Here is how a patch management software will help you keep systems up-to-date and protect your business:

Reason 1: You can measure your patching health

How can you protect what you don’t see? Often the main reason why IT departments forget to apply an important patch is that they don’t know that a system is missing a patch in the first place. Vendors are releasing new patches every day. Our patch trackers routinely count more than 200 new patches each month (see our Patch Recap for February 2021 as an example). Keeping up with all the latest patches quickly becomes a full-time job. A patch management solution helps you sort through the noise and understand which patches apply to your systems. You can see at one glance all systems that are currently unpatched and take action immediately. This reduces the time a system remains in a vulnerable state and makes it harder for attackers to take advantage of new vulnerabilities.

Reason 2: You can patch with one click

If you are applying patches manually you are investing a lot of resources into an error-prone process. You need to access every endpoint and start the update process by hand. A better solution is to use scripting languages and automation tools to automate and standardize this process. For Windows-based environments PowerShell is the natural choice. For Linux the Ansible automation tool comes with lots of handy modules that can make patching easier.

If you want to go a step further and streamline the patching process even more have a look at patch management solutions. With these solutions you can patch individual systems with one click from a dashboard or apply patches to group of systems. The result: You save time and reduce the risk of missing systems which then remain vulnerable to attacks.

Reason 3: You can patch Windows, Linux and Mac with one tool

Many organizations use not just Windows but also Linux and Mac as operating systems. Your employee laptops and desktops may all run Windows but what about your Linux-based web server infrastructure and all the Macbooks in the marketing and development departments? 

The diversity of operating systems nowadays makes patching even harder. Many traditional patching tools like Windows Server Update Service (WSUS) have not kept up with this trend (see also our post on WSUS alternatives if you are thinking about switching from WSUS). If you are in this category a cross-platform patch management solution is the right choice for you. These solutions have agents for all major operating systems and let you manage all your endpoints from a single dashboard.

Reason 4: You can test patches

Are you afraid that patches will break a perfectly fine setup? Unfortunately you are right. Although vendors have become much better at releasing stable patches and the majority of patches do not cause any issues there is the occasional hiccup. Such as freshly patched Windows systems descending into a blue-screen-of-death when trying to print a document

But not applying patches is also not a solution since this will keep your systems vulnerable. What to do? Test your patches!

Most vendors test new patches extensively before release but it is hard for them to cover all possible environments and configurations. So you better do your own testing on top of that. One way to do this is to set up a group of test devices and connect this group to your patch management solution. You can then apply patches to this group or configure a policy to always patch these systems automatically. After the latest round of patches was applied you check if everything still works. If no errors pop up you can roll out the patches to the rest of your environment.

You can extend this concept and build multiple groups of systems based on how critical errors are for them. For example, you can start by testing patches on dedicated test systems, then roll them out to non-critical workstations, and only after everything looks good continue with more critical systems like servers.

Reason 5: You can automate your patching

Automation is key to having a well-organized and maintainable IT environment. Patching should be no exception to this. Automated patch management lets you define rules about which patches should be applied when. However, it is not the same as putting all your systems on the “Auto Update” setting. A good patch management solution gives you control over the automation process. For example, you can set maintenance windows so that patches are only applied outside of normal business hours.

Not every patching process should be automated though. Some systems need extra care and should only receive well-tested patches (see Reason 4 above). This applies mainly to server patching. Workstations on the other hand, which are a major attack surface for ransomware these days, greatly benefit from an automated patching process since this shortens the time window for attackers to take advantage of new vulnerabilities.

Conclusion

Patching is just a fact of life for IT professionals but it does not need to be the painful experience it still is in many organizations. A good patch management solution makes patching easy, provides smart automations and still keeps you in full control.

Try out a modern patch management solution

Patchdeck helps you stay on top of your patching with advanced automations, customizable alerts, detailed reporting and more. Check out our cool features and start your free 14-day trial!