FAQAS - FAQAS2
APPLICABILITY OF MUTATION TESTING METHOD FOR FLIGHT SOFTWARE.
Activity No. 100028866 in the esa-star system
IMPROVE MUTATION TESTING IN SPACE SOFTWARE SYSTEMS.
Activity No. 1000033730 in the esa-star system
Project Overview
From spacecrafts to ground stations, software has a prominent role in space systems; for this reason, the success of space projects depends on the quality of the system hardware as much on the quality of its software. International standards for the development of space software emphasize the importance of software quality assurance and regulate the procedures to ensure the quality of the developed software. Software testing is the primary software quality assurance activity, and standards enforce the adoption of software testing best practices.
Unfortunately, the software testing procedures enforced by space software standards heavily rely on the capability of software engineers to define test suites that extensively exercise the software. Automated and effective methods to evaluate the quality of test suites are thus necessary. Also, methods to automatically generate test cases will speed-up the improvement of these test suites.
The FAQAS and FAQAS2 projects (Fault-based, Automated Quality Assurance Assessment for Space Software) aim to overcome these limitations by developing a framework, dedicated to space software, to evaluate and improve the quality of test suites. The project consortium includes the SVV lab of the Interdisciplinary Centre for Security Reliability and Trust of the University of Luxembourg, GomSpace Luxembourg, LuxSpace, Huld Finland, and the European Space Agency, which is the promoter of the project. The project is supported by the ESA activities AO/1-9873/19/NL/AS (completed) and RFQ/3-17554/21/NL/AS/kk (ongoing till 2024).
Since the primary objective of software testing is to identify the presence of software faults (e.g., coding errors), an effective way to assess the quality of a test suite is by artificially injecting faults in the software (i.e., creating software mutants) and by verifying if the test suite can detect the injected faults. Mutation testing refers to methods that assess test suites using metrics (i.e., the mutation score) that capture the portion of injected faults discovered.
Despite its potential usefulness, mutation testing is not adopted by the space industry. The main reason is the limited scalability of mutation testing, which is exacerbated in the case of space systems because their testing process relies on simulators and dedicated hardware. Also, the literature lacks of clear guidelines for the definition of acceptance criteria based on mutation testing. Finally, the feasibility of automated test generation in this context has been partially studied.
FAQAS and FAQAS2 are leading to a framework to evaluate and improve the quality of test suites for space software by (1) relying on mutation testing to automatically assess the quality of test suites and by (2) leveraging automated test generation techniques to generate test cases that detect the presence of mutants. FAQAS and FAQAS2 aim to identify solutions to make mutation testing scale and enable the automated generation of test cases. The developed framework is evaluated with the case study systems provided by the industry partners of the consortium.