How to do server patching right

Servers are the backbone of modern business. They run applications, handle email and manage backups. Server downtime can cost real money so many administrators get sweaty hands when they think about patching their servers.

But patch they must: Precisely because they are so critical for many organizations, servers are a prime target for ransomware gangs. An encrypted laptop is a nuisance for a single employee and can be quickly rebuilt. But when a critical server is affected by ransomware, chances are that nobody can do any work anymore. This is why patch management for servers is so important.

And the good news is: patching your servers does not need to be a panic-inducing process. With a bit of planning and the right tools, you can tame the beast of server patching.

Here is our 6-step-checklist how to create a server patching process that follows best practices.

Server Patching Checklist

Step 1: Inventory

Before you think about how to patch you need to know what to patch. Compiling an accurate list of all your servers is the first step to a successful server patching process. An asset management tool is the best way to get a detailed and up-to-date overview of your server inventory. Two good tools are (no affiliation here, just based on good experiences we have made ourselves):

• Rumble
• Snipe-IT

The big advantage of using an asset management tool compared to a simpler solution like an Excel spreadsheet is that with the first one you always get a live view of your environment. When you decommission systems or add new ones an asset management tool will pick up these changes automatically.

Armed with the output of your asset management tool you can now start to organize your inventory so that you can make sensible patching decisions.

Step 2: Group

Not all servers are created equal. Some are mission-critical such as the primary web servers for a Software-as-a-Service company or the servers running the customer database for a professional service firm. Others are important but can survive a short downtime such as a time tracking system. And others fall into the “nice-to-have” category, like the server running your intranet homepage.

To prepare your server patching process it makes sense to create groups according to these criticality ratings. You can create these groups in your asset management tool or your patch management solution.

Step 3: Observe

Having visibility into your server landscape is one thing, but you also need visibility into the never-ending flow of patch releases and identify which patches are relevant to your systems. 

For some vendors, this is easier than for others. Microsoft releases most new patches on the second Tuesday of every month, aka Patch Tuesday. But there are exceptions to this rule: When Microsoft discovers an especially critical vulnerability that needs to be patched right away they sometimes release out-of-band patches.

And if your servers run Linux chances are you have no equivalent to Patch Tuesday since most distributions do not follow a fixed schedule for patch releases.

To keep up with this you need a process to track relevant patches. One way to do this is to subscribe to the relevant mailing lists for your software. Most Linux distributions have a mailing list dedicated to patches and new releases. Microsoft provides a web portal and an API for querying patch release information for their products: https://msrc.microsoft.com/update-guide/

You can also use the Patch Alert functionality in Patchdeck to receive alerts when there are new patches for the software you use.

Another good source of patch information is the Cybersecurity & Infrastructure Security Agency (CISA) who publishes several mailing lists. However, note that these mailing lists mainly focus on high-risk vulnerabilities and do not include all patches that are relevant to your systems.

Step 4: Plan and Prioritize

Now that you know which patches you need to apply to which systems, you can draw up a battle plan. Start with the groups you created in step 2: For each group decide which patches you want to apply. For the non-critical systems you can normally apply all outstanding patches without much thinking. But handling the mission-critical servers you may need to be a bit more picky. You can for example decide to only apply security patches. 

In any case: It makes sense to write down your patching plan and share it with your team members so that everybody is on the same page. One way to do this is via the note-taking functionality of your asset management tool.

Step 5: Test

There is no better way to reduce server patching anxiety than to run tests. Especially if you manage critical servers you should invest the time to build a test environment that closely mimics the production environment. You can then apply all patches you selected in Step 4 to the test systems and check if everything is still working as expected. 

Virtualization, containers and automation tools like Ansible or Terraform make it easy to stand up complex environments in a few minutes and to automatically tear everything down after your testing is done.

If you want to play advanced mode you can combine this with a patch management solution to build an integrated workflow where an automation tool stands up a testing environment, the patch management solution applies all new patches and testing scripts ensure that everything still works. Finally, the automation tool tears down the test environment and generates a report.

Not every environment needs a complex patch testing workflow though. If your servers can be quickly rebuilt and downtime does not matter much you can skip the testing and just apply all necessary patches right away.

In this case you should definitely use automation. Most patch management tools have patch policy features that allow you to configure automated patching based on maintenance windows or patch features.

Step 6: Apply

After you have run all your tests and everything looks good you can finally proceed to the most important step: Applying the patches. Some servers can be patched in bulk based on the groups you created in step 2. For others you will probably apply patches one-by-one.

In general, even if you have good tests, it makes sense to roll out patches to production systems slowly and in waves to catch errors early and to always have some systems in a known good configuration.

What is a sensible server patching schedule?

If you have applied all patches it’s time to celebrate! You have mastered the challenge of server patching and have successfully secured your most valuable assets against attacks.

But, the thing with patching is: Soon you will have to start the whole process all over again because new patches will be released. This leads to the question: How often should you patch? What is a sensible server patching schedule?

Most patches are not so critical that you need to apply them right away. A good rule of thumb is to run the process described above at least once every two weeks. This gives you enough time for thorough testing and still ensures that you plug security holes quickly enough before attackers can target them at a large scale.

However, you should use the methods described in step 3 to always be on the lookout for particularly critical vulnerabilities that need to be patched immediately. If Microsoft releases an out-of-band patch that applies to your servers you should drop what you are doing and start patching right away. These patches fix vulnerabilities that are so critical that exploitation will start very soon. Or, even worse: Attacks have already begun.

This does not only apply to Microsoft. In January a patch was released for all major Linux distributions fixing a privilege escalation vulnerability in the Sudo command (Ubuntu advisory, CentOS/RedHat advisory). A patch like this should also be applied immediately because of its big potential impact.

Conclusion

Server patching does not need to be a nightmare. If you follow best practices, build a sound server patching process and use helpful tools like asset management and patch management solutions you will reduce the stress that comes with applying changes to your most critical pieces of infrastructure.