Application Compatibility Testing with Windows 10 and Windows-as-a-Service

Summary: Microsoft’s Windows 10 new delivery and update cycles will drive significant change in the current enterprise application compatibility testing and verification operational processes. These processes will need to be done on an ongoing basis as opposed to being done prior to large OS migration projects. Enterprises will need to quickly adjust to this new operational model, where the verification of most applications shifts to early pilot end-users rather than IT, as the Windows 10 mix grows within their organizations. To this point definition of deployment ring and inclusion of a representative set of users across the organization becomes extremely important.

The ways of the past

Microsoft developed new versions of Window OS every few years, and provided servicing to those releases via service packs, and individual patch updates.

This traditional deployment schedule meant that organizations needed to:

Plan for large fairly-large deployments cycles
Develop heavy robust application compatibility testing processes
Wait long periods for new features
Impose a training burden on users because the feature revisions were often significant.

The resulting consequence of this traditional way, was a significant refresh project every few years which tended to be large, lengthy, and expensive.

Application compatibility testing was part of the burden when approaching a Windows OS deployment or upgrade. Organizations have developed and fined-tuned application release management and compatibility testing processes around these release cycles. This work effort which has been relied on for years, is labor intensive and very taxing on large organizations.

In today’s rapidly changing world, in which new security threads are released daily and updated capabilities are needed by end-users and administrators, this traditional slow OS release model does not work.

Moving Forward

With the release of Windows 10 and Windows-as-a-Service, Microsoft has introduced a modern approach of perpetual updates rather than monolithic ones moving forward.

With Windows 10, there are two release types: feature updates that add new functionality two to three times per year, and quality updates that provide security and reliability fixes at least once a month. These releases spread evenly over time versus spiking every few years providing incremental on-going functionality and security improvements.

Once deployed, Windows 10’s new approach makes the process simpler and therefore the planning effort much more straightforward. This new cadence, has significant impact on existing organization’s application management and compatibility testing processes and practice.

Existing processes and established practice call for the thorough testing of applications before the deployment of a new version, service pack or significant security update to the Windows OS, causing significant cost and effort to insure end-users can use applications without disruption.

In the Windows as a service model, Microsoft is making a commitment to maintaining the compatibility of the underlying OS. This means Microsoft will make a concerted effort to ensure that there are no breaking changes that impact the app ecosystem negatively. In this scenario, when there is a release of a Windows build, most apps (those with no kernel dependencies) will continue to work.

With Microsoft’s new Windows 10 architecture, and the commitment to a high-level of compatibility, organizations are expected to adopt a more streamlined testing process, which will include:

For the most important business-critical applications, organizations should still perform testing on a regular basis to validate compatibility with new builds.

Identify mission-critical applications and focus testing efforts on this subset., early in the Windows release cycle (Windows Insider) to identify potential issues. Report any issues you encounter with the Windows Feedback tool, so that these issues can be addressed prior to the next Windows release.

For remaining applications, consider validating them as part of a pilot deployment process to reduce the time spent on compatibility testing.

For less critical applications, leverage early adopter pilot-based approach, by deploying new Windows upgrades and updates to groups of machines (rings), growing gradually in size and potential impact, to verify compatibility. Reactively address issues before you expand the pilot to more machines.

Build deployment rings that target groups of computers in your selected update-management product.

To reduce risk as much as possible, construct your deployment rings in a way that splits individual departments into multiple rings. This way, if you were to encounter an issue, you don’t prevent any critical business from continuing. By using this method, each deployment ring reduces risk as more and more people have been updated in any department.

Within the next 18 months, based on Microsoft’s update cadence, three new upgrades will go into end-of-life. Sooner rather than later organizations must get on board with the new and fast servicing schedule Microsoft sets. Part of that onboarding will mean a change in their application compatibility and verification operational processes to a more streamlined and ongoing-cycle.