Categories
How To

How does patching work in Windows 11

Windows 11 is the future of Windows. Microsoft released its new operating system on October 5 and has already made clear that it will stop supporting Windows 10 in 2025. So why not get ahead of the curve and learn what changes with Windows 11 in regard to one of the most important tasks for Windows admins: Keeping the system up-to-date and patched.

Let’s take a look at how Windows 11 is different from its predecessor when it comes to patching.

Smaller updates

Microsoft has invested a lot of work in improving the inner workings of the Windows update mechanism. One result: The cumulative update size in Windows 11 is about 40 percent smaller than in Windows 10.

This is thanks to a more efficient packaging of the individual update files and the removal of special files called “reverse differentials” that took up a lot of space in the older update packages. In Windows 11 the information contained in reverse differentials is computed on the fly during an update. If you want to read more about how this works Microsoft has a good blog post about it.

Quicker downloads

Smaller update sizes mean faster update downloads but there is more. Microsoft has brought back the Express technology in Windows 11, a feature that was initially present in Windows 10 but got replaced in Windows 10 1809. The revamped Express technology helps the Windows update mechanism to request only the update package that it really needs. Again Microsoft’s blog post explains this in more detail.

A smoother update experience

Smaller package sizes and quicker downloads thanks to a more selective download approach should lead to a smoother overall update experience. This will reduce the interruption that a Windows update means for end-users and system administrators. That is a big deal since many updates get postponed because they simply take so long. Speeding up the update process will therefore directly improve the security of devices and networks.

Patch Tuesday is here to stay

Some things will probably never change and one of them is Microsoft’s famous Patch Tuesday. With Windows 11, Microsoft will keep with its traditions and publish new patches on the second Tuesday of each month.

One change to the update release cycle however seems to be coming: In the official release information for Windows 11, Microsoft writes that the new operating system will receive only one feature update per year. Windows 10 so far received two new feature updates per year. An annual update cycle for feature updates makes more sense and will simplify things by reducing the number of current Windows versions in circulation.

You can keep your old workflow

So what does this all mean for the actual process of bringing your Windows 11 system up-to-date? From a user point of view nothings changes. The Windows update mechanism still works the same as in Windows 10. So no matter whether you use a patch management solution like Patchdeck, distribute patches via WSUS or are still doing manual updates, you can keep your old workflow and still benefit from the speed improvements.

Conclusion

Windows 11 brings some exciting improvements to the patching process. Reduced package sizes and better downloading logic should make Windows updates faster than ever. Let’s hope that Microsoft continues to invest engineering resources in optimizing the Windows update mechanism.

Try out a modern patch management solution

Patchdeck helps you stay on top of your patching with advanced automations, customizable alerts, detailed reporting and more. Check out our cool features!

Features and PLANS
Categories
How To

How to do server patching right

Servers are the backbone of modern business. They run applications, handle email and manage backups. Server downtime can cost real money so many administrators get sweaty hands when they think about patching their servers.

But patch they must: Precisely because they are so critical for many organizations, servers are a prime target for ransomware gangs. An encrypted laptop is a nuisance for a single employee and can be quickly rebuilt. But when a critical server is affected by ransomware, chances are that nobody can do any work anymore. This is why patch management for servers is so important.

And the good news is: patching your servers does not need to be a panic-inducing process. With a bit of planning and the right tools, you can tame the beast of server patching.

Here is our 6-step-checklist how to create a server patching process that follows best practices.

Server Patching Checklist

Step 1: Inventory

Before you think about how to patch you need to know what to patch. Compiling an accurate list of all your servers is the first step to a successful server patching process. An asset management tool is the best way to get a detailed and up-to-date overview of your server inventory. Two good tools are (no affiliation here, just based on good experiences we have made ourselves):

The big advantage of using an asset management tool compared to a simpler solution like an Excel spreadsheet is that with the first one you always get a live view of your environment. When you decommission systems or add new ones an asset management tool will pick up these changes automatically.

Armed with the output of your asset management tool you can now start to organize your inventory so that you can make sensible patching decisions.

Step 2: Group

Not all servers are created equal. Some are mission-critical such as the primary web servers for a Software-as-a-Service company or the servers running the customer database for a professional service firm. Others are important but can survive a short downtime such as a time tracking system. And others fall into the “nice-to-have” category, like the server running your intranet homepage.

To prepare your server patching process it makes sense to create groups according to these criticality ratings. You can create these groups in your asset management tool or your patch management solution.

Step 3: Observe

Having visibility into your server landscape is one thing, but you also need visibility into the never-ending flow of patch releases and identify which patches are relevant to your systems. 

For some vendors, this is easier than for others. Microsoft releases most new patches on the second Tuesday of every month, aka Patch Tuesday. But there are exceptions to this rule: When Microsoft discovers an especially critical vulnerability that needs to be patched right away they sometimes release out-of-band patches.

And if your servers run Linux chances are you have no equivalent to Patch Tuesday since most distributions do not follow a fixed schedule for patch releases.

To keep up with this you need a process to track relevant patches. One way to do this is to subscribe to the relevant mailing lists for your software. Most Linux distributions have a mailing list dedicated to patches and new releases. Microsoft provides a web portal and an API for querying patch release information for their products: https://msrc.microsoft.com/update-guide/

You can also use the Patch Alert functionality in Patchdeck to receive alerts when there are new patches for the software you use.

Another good source of patch information is the Cybersecurity & Infrastructure Security Agency (CISA) who publishes several mailing lists. However, note that these mailing lists mainly focus on high-risk vulnerabilities and do not include all patches that are relevant to your systems.

Step 4: Plan and Prioritize

Now that you know which patches you need to apply to which systems, you can draw up a battle plan. Start with the groups you created in step 2: For each group decide which patches you want to apply. For the non-critical systems you can normally apply all outstanding patches without much thinking. But handling the mission-critical servers you may need to be a bit more picky. You can for example decide to only apply security patches. 

In any case: It makes sense to write down your patching plan and share it with your team members so that everybody is on the same page. One way to do this is via the note-taking functionality of your asset management tool.

Step 5: Test

There is no better way to reduce server patching anxiety than to run tests. Especially if you manage critical servers you should invest the time to build a test environment that closely mimics the production environment. You can then apply all patches you selected in Step 4 to the test systems and check if everything is still working as expected. 

Virtualization, containers and automation tools like Ansible or Terraform make it easy to stand up complex environments in a few minutes and to automatically tear everything down after your testing is done.

If you want to play advanced mode you can combine this with a patch management solution to build an integrated workflow where an automation tool stands up a testing environment, the patch management solution applies all new patches and testing scripts ensure that everything still works. Finally, the automation tool tears down the test environment and generates a report.

Not every environment needs a complex patch testing workflow though. If your servers can be quickly rebuilt and downtime does not matter much you can skip the testing and just apply all necessary patches right away.

In this case you should definitely use automation. Most patch management tools have patch policy features that allow you to configure automated patching based on maintenance windows or patch features.

Step 6: Apply

After you have run all your tests and everything looks good you can finally proceed to the most important step: Applying the patches. Some servers can be patched in bulk based on the groups you created in step 2. For others you will probably apply patches one-by-one.

In general, even if you have good tests, it makes sense to roll out patches to production systems slowly and in waves to catch errors early and to always have some systems in a known good configuration.

What is a sensible server patching schedule?

If you have applied all patches it’s time to celebrate! You have mastered the challenge of server patching and have successfully secured your most valuable assets against attacks.

But, the thing with patching is: Soon you will have to start the whole process all over again because new patches will be released. This leads to the question: How often should you patch? What is a sensible server patching schedule?

Most patches are not so critical that you need to apply them right away. A good rule of thumb is to run the process described above at least once every two weeks. This gives you enough time for thorough testing and still ensures that you plug security holes quickly enough before attackers can target them at a large scale.

However, you should use the methods described in step 3 to always be on the lookout for particularly critical vulnerabilities that need to be patched immediately. If Microsoft releases an out-of-band patch that applies to your servers you should drop what you are doing and start patching right away. These patches fix vulnerabilities that are so critical that exploitation will start very soon. Or, even worse: Attacks have already begun.

This does not only apply to Microsoft. In January a patch was released for all major Linux distributions fixing a privilege escalation vulnerability in the Sudo command (Ubuntu advisory, CentOS/RedHat advisory). A patch like this should also be applied immediately because of its big potential impact.

Conclusion

Server patching does not need to be a nightmare. If you follow best practices, build a sound server patching process and use helpful tools like asset management and patch management solutions you will reduce the stress that comes with applying changes to your most critical pieces of infrastructure.

Try out a modern patch management solution

Patchdeck helps you stay on top of your patching with advanced automations, instant patch alerts, customizable notifications, API and more. Check out our cool features!

Features and PLANS
Categories
Best Practices

5 reasons why you need a patch management software

We all know it: patching is hard. Even well-organized IT departments will at some time fall behind on patching – often without even knowing it. With new vulnerabilities disclosed every day and vendors releasing patch after patch, this is hardly a surprise.

For attackers this makes things all too easy: 60 percent of data breaches involved known vulnerabilities for which patches are available, according to a meta-research conducted by CSO Online.

But there are ways to get on top of your patching. Dedicated patch management solutions are designed to make patching easy and keep you in control and in the know. Here is how a patch management software will help you keep systems up-to-date and protect your business:

Reason 1: You can measure your patching health

How can you protect what you don’t see? Often the main reason why IT departments forget to apply an important patch is that they don’t know that a system is missing a patch in the first place. Vendors are releasing new patches every day. Our patch trackers routinely count more than 200 new patches each month (see our Patch Recap for February 2021 as an example). Keeping up with all the latest patches quickly becomes a full-time job. A patch management solution helps you sort through the noise and understand which patches apply to your systems. You can see at one glance all systems that are currently unpatched and take action immediately. This reduces the time a system remains in a vulnerable state and makes it harder for attackers to take advantage of new vulnerabilities.

Reason 2: You can patch with one click

If you are applying patches manually you are investing a lot of resources into an error-prone process. You need to access every endpoint and start the update process by hand. A better solution is to use scripting languages and automation tools to automate and standardize this process. For Windows-based environments PowerShell is the natural choice. For Linux the Ansible automation tool comes with lots of handy modules that can make patching easier.

If you want to go a step further and streamline the patching process even more have a look at patch management solutions. With these solutions you can patch individual systems with one click from a dashboard or apply patches to group of systems. The result: You save time and reduce the risk of missing systems which then remain vulnerable to attacks.

Reason 3: You can patch Windows, Linux and Mac with one tool

Many organizations use not just Windows but also Linux and Mac as operating systems. Your employee laptops and desktops may all run Windows but what about your Linux-based web server infrastructure and all the Macbooks in the marketing and development departments? 

The diversity of operating systems nowadays makes patching even harder. Many traditional patching tools like Windows Server Update Service (WSUS) have not kept up with this trend (see also our post on WSUS alternatives if you are thinking about switching from WSUS). If you are in this category a cross-platform patch management solution is the right choice for you. These solutions have agents for all major operating systems and let you manage all your endpoints from a single dashboard.

Reason 4: You can test patches

Are you afraid that patches will break a perfectly fine setup? Unfortunately you are right. Although vendors have become much better at releasing stable patches and the majority of patches do not cause any issues there is the occasional hiccup. Such as freshly patched Windows systems descending into a blue-screen-of-death when trying to print a document. 

But not applying patches is also not a solution since this will keep your systems vulnerable. What to do? Test your patches!

Most vendors test new patches extensively before release but it is hard for them to cover all possible environments and configurations. So you better do your own testing on top of that. One way to do this is to set up a group of test devices and connect this group to your patch management solution. You can then apply patches to this group or configure a policy to always patch these systems automatically. After the latest round of patches was applied you check if everything still works. If no errors pop up you can roll out the patches to the rest of your environment.

You can extend this concept and build multiple groups of systems based on how critical errors are for them. For example, you can start by testing patches on dedicated test systems, then roll them out to non-critical workstations, and only after everything looks good continue with more critical systems like servers.

Reason 5: You can automate your patching

Automation is key to having a well-organized and maintainable IT environment. Patching should be no exception to this. Automated patch management lets you define rules about which patches should be applied when. However, it is not the same as putting all your systems on the “Auto Update” setting. A good patch management solution gives you control over the automation process. For example, you can set maintenance windows so that patches are only applied outside of normal business hours.

Not every patching process should be automated though. Some systems need extra care and should only receive well-tested patches (see Reason 4 above). This applies mainly to server patching. Workstations on the other hand, which are a major attack surface for ransomware these days, greatly benefit from an automated patching process since this shortens the time window for attackers to take advantage of new vulnerabilities.

Conclusion

Patching is just a fact of life for IT professionals but it does not need to be the painful experience it still is in many organizations. A good patch management solution makes patching easy, provides smart automations and still keeps you in full control.

Try out a modern patch management solution

Patchdeck helps you stay on top of your patching with advanced automations, instant patch alerts, customizable notifications, API and more. Check out our cool features!

Features and PLANS
Categories
Best Practices

What is a good WSUS alternative?

If you ask around which patch management software to use for Windows endpoints the first answer usually is: The Windows Server Update Services or short WSUS. WSUS has been around for more than a decade and is made by Microsoft, so it seems an obvious choice. But WSUS is not for everybody. Let’s have a look for which situations Microsoft’s traditional patch management solution is the right tool, where it falls short and what alternatives to WSUS exist.

What is WSUS?

WSUS was released in 2005 by Microsoft to give system administrators a way to centrally manage updates to Windows clients and servers. To set up WSUS in your environment, you need to create a dedicated WSUS server by installing Microsoft Windows Server and then adding the WSUS Server role. You can have one or multiple WSUS servers in your organization, depending on how many endpoints you need to manage and whether you want to distribute the load of downloading patches.

After setting up the server you configure your endpoints to use the WSUS server as the source for updates instead of going to Microsoft’s update servers directly. And that’s it – at least on a very high level. There are several additional steps required to get a stable and well-configured WSUS setup. For a detailed configuration guide see the Microsoft Docs page on WSUS.

What WSUS does well

After you have set up WSUS successfully you can use it to apply updates to your endpoints, either to individual systems or to groups. You can also choose to only apply specific updates, configure automatic patching and many more options. These are the things that WSUS does well:

  • Fine-grained control over which patches get applied to which systems
  • Good integration into other parts of a Windows-based architecture like Group Policies, Active Directory, etc.
  • WSUS now also supports several PowerShell cmdlets which make automation of update tasks easier

In addition to these points there is another advantage of WSUS that makes it an attractive choice for many businesses: it is included in your normal Windows Server licensing fee and does not cost anything extra beyond that. However, although WSUS does not incur additional fees many system administrators have made the experience that it requires a lot of work to maintain. Just search for WSUS questions on Stack Overflow or Reddit to read some of the issues people have experienced. The maintenance effort of WSUS needs to be factored into the total cost of ownership. This leads us to the next section.

Where WSUS falls short

WSUS is more than 15 years old. And although Microsoft is still officially supporting it the core structure of the product has certainly not aged well. This leads to multiple problems, among them:

  • It needs a lot of maintenance: WSUS contains many moving parts and after some time problems do pop up in nearly every WSUS setup – often accompanied by hard to decipher error messages. As a consequence, more and more endpoints will miss patches. If this happens to your setup it is time for the infamous WSUS clean-up. Many system administrators use custom-built scripts for these clean-up tasks and the time to write and maintain these scripts need to be factored in when assessing the costs of using WSUS.
  • WSUS is Windows-only: Microsoft did build WSUS with a clear focus on pure Windows environments but this is not how many modern environments look like. Nowadays most businesses use a combination of Windows, Linux and Mac systems but everything that is not Windows is left out when it comes to patching through WSUS. If you are looking for patch management software for Linux or Mac WSUS will not help you.
  • Not on the network? Not gonna be patched. The WSUS server lives on your internal network so only endpoints that are on-premise or connected via VPN can reach it. This is a challenge for companies that make extensive use of remote work and it can mean remote endpoints miss important patches.

Who should use WSUS and who shouldn’t?

Now that we have looked at the things that WSUS does well and where it falls short we can answer the question: For which type of IT environment is WSUS the right choice? 

If you are a pure Windows shop where all of your devices are on the same network all the time and you have the IT resources to maintain and care for the WSUS server, then Microsoft’s solution to patch management is a good fit.

But what if you have a diverse IT environment with some endpoints running Windows, others Linux and even have some Macs around? What if your employees do not always work from the office but are sometimes traveling or working remotely? And what if your system administrators are already busy enough and don’t have the time to maintain yet another picky server? In this case you should be looking for WSUS alternatives.

What are good WSUS alternatives?

If WSUS does not meet your requirements there are several good alternatives around. Microsoft itself seems to be moving away from WSUS and towards their new service Azure Update Management. It supports patching of Windows and Linux endpoints and different forms of automation. This service is completely based in Microsoft Azure so it makes the most sense for organizations that already have a large Azure footprint. It can be used to manage patching for virtual machines inside Azure as well as on-premise systems. In the latter case, you will need to set up the Log Analytics service for these endpoints. Detailed explanations and documentation can be found on the Microsoft Docs page on Update Management.

For organizations that need advanced features and cross-platform support, the best WSUS alternative is to use a dedicated patch management solution. Most modern patch management software does support patching Windows, Linux and Mac endpoints from a single dashboard and therefore makes it easy to bring heterogeneous IT environments up-to-date quickly. These solutions often come with advanced options to configure patch policies and generate reports so you can always see the current patch status of your environment. Plus, solutions that are cloud-based let you patch every endpoint no matter if it is on-premise or moving around.

Since you are on our blog we of course recommend Patchdeck 😉 But there are many good solutions around.

Can you combine WSUS and patch management solutions?

If your organization already has a working WSUS setup but you are looking for features that WSUS cannot provide on its own, there is no need to completely burn down what is already working. WSUS can work well together with other patch management solutions. For example, you can use your WSUS server simply as a central repository for Windows patches while the process of patching, automating and reporting is offloaded to a dedicated patch management solution. This has the advantage of saving bandwidth as the Windows updates are only downloaded once to your WSUS server and from there distributed to all Windows endpoints. At the same time, you can use your dedicated patch management solution to get an overview of the patching status of your endpoints and have full control over how you organize groups and automations. You can read in our docs how Patchdeck integrates with WSUS.

Conclusion

WSUS has been the natural choice for Window patch management for over a decade but its age is definitely showing. It remains a good tool for organizations that are Windows-only and have all endpoints on-premise all the time. But if you don’t fall into this category Microsoft’s new cloud offering or a dedicated patch management solution are better choices. And even if you already have a WSUS setup working you can combine it with a patch management solution to get the best of both worlds.

Try out a modern patch management solution

Patchdeck helps you stay on top of your patching with advanced automations, customizable alerts, detailed reporting and more. Check out our cool features and start your free 14-day trial!

Features and PLANS
Categories
Best Practices

Four patches a day keep the sysadmin busy

To shed some light on this question we simulated the tech stack of a fictional company and used our data set of patch data to analyze how often they would need to apply a security patch or an update to the software they use. Here is what our fictional company uses:

Clients:

  • Windows 10 workstations with Microsoft Office

Servers:

  • Windows Server 2019
  • Windows Server 2012
  • Ubuntu Server
  • CentOS Server

Virtualization:

  • VMware

Development and Operations:

  • Docker
  • Python
  • Jenkins

A real-life company would probably use several additional technologies but we want to keep it simple and cover some of the main components of a typical tech stack.

Four patches a day

For our analysis we took data points from our patch data set for the four-week period between May 14 and June 11. The data shows that all in all the system administrators at our example company would need to apply 83 patches in four weeks which translates to roughly 4 patches per day. This already gives an indication about the effort involved in keeping everything up-to-date and secure. Let’s break the data down a little further.

Microsoft software is probably top of the list when it comes to patching in many companies. During the analyzed period Microsofts patch Tuesday happened on June 9th and comprised of 129 vulnerabilities, 11 of which had a rating of critical. Since our example company uses only four Microsoft products most of these patches did not apply and the sysadmins could happily ignore them and only apply the updates to Windows 10, Windows Server 2019, Windows Server 2012 and Microsoft Office. We did count these updates as single items although they each fixed many vulnerabilities: 81 for Windows Server 2019, 36 for Windows Server 2012, 51 for Windows 10 and 5 for Microsoft Office.

The two Linux servers in our setup received 64 patches: 47 for Ubuntu and 18 for CentOS. Here it is important to note that we did not make further assumptions about the installed packages on both Linux distributions and some of the patches only applied to specific packages so the real patching needs could be lower.

But the server team at our example company was not only kept busy by their Linux systems: VMware released 6 patches in the four-week timeframe, among them the fix for a serious bug that would allow command injection in the VMware Cloud Director.

The development and operations department had a quieter month with only 8 patches. Amongst them were 3 new versions of Docker, 1 new Python release and 4 releases for Jenkins.

Patching takes up a lot of sysadmin time

Our analysis shows that every piece of technology used by our example company needed an update during our analysis timeframe. Estimating the effort needed to apply all these patches is difficult since the mechanics of patching and updating differ widely between technologies. Microsoft updates are (or at least should be) automated in most companies, however, this in many cases only applies to clients and client software (e.g. Windows, Office). For Microsoft server operating systems many teams use at least a partially manual process, for example by using a testing environment and a gradual rollout, to ensure that patches don’t bring down critical servers. The same process applies to Linux servers. Virtualization software often needs special caution while updating because it forms the backbone of a company IT architecture. Updating programming languages like Python does need to be planned carefully as well since it can break application features and make code rewrites necessary. Tools like Docker or Jenkins on the other hand can often be upgraded in a quick fashion.

If we assume a rather optimistic scenario in which it takes a system administrator on average 45 minutes to research, install and test a patch our example company would have spent around 62 man hours in the four weeks timeframe on patching and updating.

Looking at these numbers it is not surprising to see so many companies falling behind on patching their tech stack. Patching takes up around a third of a system administrators time, even in our optimistic scenario which assumes a fair degree of automation and patch management. This back-of-the-envelope calculation already shows how expensive it is to keep all systems patched. However, all this money is still a very good investment when considering how quickly unpatched systems are compromised by attackers. A new report by FireEye shows that 12 percent of vulnerabilities were exploited within one week after a patch was released and 15 percent were exploited within a month (most of the remaining ones were zero days which were exploited even before a patch was available). And a recent advisory by the Cybersecurity and Infrastructure Security Agency (CISA) of the Department of Homeland Security warns that cyber criminals are still actively exploiting several bugs in Pulse Secure VPN servers, nearly a year after patches for these bugs have been released.

Missing patches is dangerous

This also demonstrates the importance of not missing critical patches for your tech stack. You can’t patch what you don’t know. It’s a safe guess that many of the companies that are currently being compromised via the Pulse Secure vulnerabilities have missed this patch all together. It is easy to blame them but our analysis shows: Keeping up with patch publications is a lot of work. While some companies, most notably Microsoft, have well-known dates for releasing patch information, most security patches are released freely throughout a month. During the four-week timeframe there were only 6 days where no new patches were released.

A robust patch alerting system should therefore be part of every patch management program.

Conclusion

Patching your systems is one of the most important things you can do to protect against attacks but with todays diverse tech stacks it has become a high burden on IT teams. Patch management processes that are based on automation, testing and event-driven workflows can help lower this burden.