Real Time Linux Workshops
1999 - 2000 - 2001 - 2002 - 2003 - 2004 - 2005 - 2006 - 2007 - 2008 - 2009 - 2010 - 2011 - 2012 - 2013 - 2014 - 2015 - 2017
17th Real Time Linux Workshop, October 21 to 22, 2015 at the Virtual Vehicle Research Center, Graz, Austria
Announcement - Call for participation (ASCII) - Hotels - Directions - Agenda - Paper Abstracts - Presentations - Registration - Abstract Submission - Exhibitors and Sponsors - Gallery
Test Suite Coverage Measurement and Reporting for Testing an Operating System without Instrumentation
Hermann Felbinger, Virtual Vehicle Research Center
Joel Sherrill, OAR Corporation
Gedare Bloom, George Washington University
Franz Wotawa, Graz University of Technology
Measuring the coverage of a test suite provides common metrics to assess the quality of a test suite. In safety-critical applications as in the domains of avionics and automotive complete coverage is required for certification. Usual approaches to measure the coverage require instrumentation of the source code or the object code of the system under test to obtain processable execution traces. However, instrumentation might change the behavior of the system under test. In this paper we show an approach to measure the coverage of a test suite and to generate human-readable reports without instrumentation of the system under test. As a system under test we use an operating system. Our approach is based on the execution traces obtained from an instrumented QEMU CPU emulator. We use this emulator to execute the operating system and the test cases. From the execution of the test cases we obtain execution traces. We provide a framework to map these execution traces back to the source code and to generate a detailed report exposing execution and branching (taken/not taken) information at the assembly language level and high level language level.
To evaluate our approach we generate coverage reports for the RTEMS real time operating system. We provide detailed coverage results for RTEMS running on different CPUs in this paper. Coverage of a test suite can be used by operating system developers to assess test suite quality and guide test case creation. Our approach is due to the lack of instrumentation of source code and object code broadly applicable for development of embedded systems applications.