What is configuration testing?

Configuration testing is a software testing methodology that aims to find the optimal configuration for the software system in regard to the functional requirements.

Software is tested through multiple combinations of software and hardware to assess whether configuration requirements are met without any defects. Configurations can range from different operating systems versions, different databases, different browsers, different CPUs and GPUs, different memory sizes, and so on. Below is a sample list of multiple configurations:

• Operating systems: Ubuntu 18.04, Ubuntu 20.10, Fedora 34, Mac OS X, Windows 10
• Databases: MySQL, PostgreSQL, Oracle, SQLite
• Browsers: Google Chrome, Firefox, Safari, TOR browser, Microsoft Edge

Objectives

• Determine optimal configurations that meet performance and functional requirements.
• Find any defects and flaws when testing the system on multiple versions and report them.
• Check how backward compatible the system is to different OS versions.
• Assess the degree of reproducibility of bugs with configuration changes.
• Analyze the performance of the system with configuration changes.

How to perform configuration testing

There are two types of configuration testing:

• Hardware configuration testing: This type of testing is performed in a lab with multiple physical machines to check different combinations of memory storage, CPUs, and GPUs. The test suite is performed on all physical machines and results are compiled.
• Software configuration testing: This type of testing involves assessing different OS versions, browser versions, and different software updates. One of the most adopted practices is to use different virtual machines to test on different OS versions. This saves the time of installing/re-installing OS on physical machines and provides the same feel as a physical machine. Software configuration testing is performed after the system has passed the unit and integration testing phases. Test environments are built, configuration requirements are specified, and then the test suite is executed.