Introduction To Patching

A well-known pain point in the hosting industry is managing the security of websites owned and managed by end-users. While the hosting provider has a complete say over their own systems, they must ultimately grant management of the hosting space to those who use it. An unfortunate reality remains that a sizable portion of end-users does not have the time, resources, or inclination to properly maintain their code or applications. This is particularly noticeable with CMS applications, which still tend to lag behind their latest release.

Patchman offers a solution to this with automatic patch testing or automating the process of finding and correcting vulnerabilities in outdated CMS applications across entire hosting infrastructures. When an outdated application contains a vulnerability, Patchman will detect and patch this vulnerability within the code, rendering an outdated application for which it provides patches as secure as the latest release.

However, accomplishing this is not trivial. Patchman scans thousands upon thousands of files for its customers, and Patches can have a significant reach and impact; as many as half a million websites per single vulnerability for larger applications such as WordPress — and that is still only a fraction of the total number of websites protected by the Patchman solution.

It follows that proper testing and QA is absolutely essential and hold number one priority for our Research team when a new patch is created. This article will give a brief glimpse into Patchman’s QA practices, specifically the patch validation process, and talk about a key improvement we’ve made in this area recently, automated patch testing.

Creating and validating patches

When a new application release comes to our attention, through (automated monitoring of) official release channels, direct involvement with application developers, or via another route, our work begins. We first evaluate such a new release through code review and examination of the changelog, to establish which changes, if any, address security issues in the application. When identified, these security-relevant changes are candidates to become patches.

When creating new patches we always strive to stay as close as possible to the security fixes implemented by the official developer and backport as far as the presence of the vulnerability and technical viability allow, with a final limitation being that we don’t backport patches to versions of an application that require PHP versions of 5.3 or before. These practices allow the patches we create and distribute to address security issues in affected versions, but without negatively impacting application and website functionality. The latter— the certainty that patches don’t break things— is verified through automated and manual testing, which we collectively refer to as the patch validation process.

To better explain, it helps to look at an example WordPress release, say WordPress 5.5.2 (dated 30th of October, 2020. This added escaping to a variety of admin section elements in two application files to resolve a cross-site scripting vulnerability:

• wordpress/wp-admin/includes/media.php
• wordpress/wp-admin/includes/template.php

As part of the patch validation process we apply a newly created patch to all versions of the application which it affects, on all versions of PHP that version of the application runs on, and then perform extensive testing to ensure that the Patched application remains fully functional, and that the security vulnerability is no longer present or exploitable.

This can be a rigorous endeavor, given that some security vulnerabilities affect dozens of different application versions, with patches spanning multiple files, and each unique patch would have to be set-up and tested against locally. This makes the process very labor-intensive.

Enter automated patch testing

Earlier this year, we built a new internal testing system to replace the previous patch validation process workflows. Internally, we refer to this as automated patch testing, and it is a platform for us to apply and test patches in parallel in a central environment.

Built into our own internal tooling, automated patch testing enables us to concurrently spin up hundreds of containers, each with a fully configured CMS application. Across these containers we employ a unit testing framework to unit test every relevant combination of application version, PHP version, and patch. This lets us move away from the previous manual functional testing in a local environment, and not only makes the testing itself more comprehensive, but also makes the entire process far more scalable.

This leads to faster cycle times because of centralized test automation and parallelization, and improved quality because the former enables us to test more rigorously. These improvements benefit Patchman customers by enabling us to deliver Patches more quickly, and with exceedingly thorough QA.

For more information on Patchman, email us at [email protected] or visit Patchman.co for a free trial.