Advances in Computers (eBook)

Atif Memon (Herausgeber)

eBook Download: PDF | EPUB

2015 | 1. Auflage
260 Seiten
Elsevier Science (Verlag)
978-0-12-802339-6 (ISBN)

Infos

Advances in Computers carries on a tradition of excellence, presenting detailed coverage of innovations in computer hardware, software, theory, design, and applications. The book provides contributors with a medium in which they can explore their subjects in greater depth and breadth than journal articles typically allow. The articles included in this book will become standard references, with lasting value in this rapidly expanding field. - Presents detailed coverage of recent innovations in computer hardware, software, theory, design, and applications - Includes in-depth surveys and tutorials on new computer technology pertaining to computing: combinatorial testing, constraint-based testing, and black-box testing - Written by well-known authors and researchers in the field - Includes extensive bibliographies with most chapters - Presents volumes devoted to single themes or subfields of computer science

Textauszug

Chapter Two

Constraint-Based Testing

An Emerging Trend in Software Testing

Arnaud Gotlieb    Simula Research Laboratory, Norway

Abstract

Constraint-based testing is an emerging trend in software testing research and innovation. The general idea beneath this testing paradigm is to exploit constraint solving and optimization to generate test cases for testing a software system, so that the fault-revealing capabilities of the testing process can be improved. The incredible progresses achieved these last years on the development of powerful constraint solvers have fostered the emergence of effective and efficient constraint-based testing techniques. This chapter reviews in detail the main achievements reached in the field by illustrating them on simple examples and mentioning the main bibliographic references.

Keywords

Constraint-based testing

Code-based testing

Automatic test input generation

Constraint solving

Constraint programming

SAT and SMT solving

1 Introduction

In business-critical or safety-critical systems, software is often considered as the weakest link of the chain. As a matter of fact, the development of these systems relies more on highly skilled and experienced software engineers, than processes and techniques which guarantee the production of reliable, dependable, and secure software. As a consequence, even if handcrafted software systems are usually of high quality, they are also vulnerable to software defects because their development processes is not immune to human error. In addition, the current complexity and size growth of software, especially in the field of embedded systems, requires the development of automated testing techniques because hand-made test cases are no longer sufficient to validate these systems.

1.1 Software Testing

Among verification and validation approaches which have solid basis, software testing is the most widely used. Software testing requires the actual execution of the software system under test, which brings an immense advantage over other techniques such as software model checking, static analysis, or program proving. Indeed, software testing enables to verify the binary code actually executed, instead of checking only the source code of the system. In particular, software testing does not require to make strong hypothesis on the semantics of programming languages such as binary encoding formats of intermediate data which are almost impossible to verify when optimizing compilers are used and systems are executed on specific machines. Additionally, software testing enables to verify the system under test within its limited-resources execution environment, using the actual linked libraries in a process managed by the operating system.

These elements have placed software testing on top of validation and verification techniques for the industrial world. However, it can be pointed out that software testing suffers from an essential drawback, which relies to its fundamental incompleteness. Quoting the famous saying by Edsger W. Dijkstra, we know that:

Program testing can be used to show the presence of bugs, but never to show their absence!

Indeed, executing the program under test in all possible behaviors is almost always impossible because their number is by far too high. Software testing cannot formally guarantee the correctness of the program with respect to its specification and it cannot guarantee either the total absence of fault. Although often mentioned as the main drawback of software testing, this limitation is well accepted in practice as residual faults in the program are usually well tolerated as soon as they remain infrequent. Actually, reliability models of software systems, even for the most safety-critical ones, do not require the total absence of faults1.

In fact, software testing suffers more from the cost of its systematic application than from this too theoretical limitation. Each test requires to be formalized in a document with a precise definition of its objective, that is, the explanation of its selection. Test scripts, which have to be treated as any other artifact within the software development chain, must also include a prediction of the expected outcome of the program under test. This latter element is crucial to establish a test verdict, which can be either pass if the expected outcome is similar to the actual outcome, or fail if they are different. Predicting the expected outcome of a program is not easy without a deep understanding of the tested application and sufficient knowledge of the execution platform. Test scripts are usually costly to develop, validate, maintain, and document. This explains why a particularly important challenge in software testing is concerned with the automation of test production, selection, and scheduling. In industrial development of large-scale software systems, the test process represents approximatively half of the overall development process. In order to address this challenge, researchers have dealt with the following problems in software testing, namely:

• The test data selection problem. Software testing requires to pick-up input test data for evaluating the correctness of a program under test. Sometimes, test inputs have to be selected within a large input search space. Ideally, one would like to select the inputs which are the most likely to reveal faults in the program, but in practice, as the faults are unknown, one resorts to select test inputs from imperfect testing criteria. The selection is performed by using either a model of the program specifications (model-based testing or black-box testing) or from the program under test itself (code-based testing or white-box testing), or from models of possible faults (mutation testing), or from usage profiles (statistical testing). Testing criteria are used to guide the test data selection process, limit the size of test suites, and streamline the testing process. The stumbling block of these techniques involves reaching a sufficient level of confidence in the covering of the behaviors of the program under test. Consequently, validation engineers often have to compromise between the cost of the test and the expected level of reliability of the testing process. This is called the test data selection problem. When existing test suites, generated for testing a previous version of a program under test, are used to test a new version of the program, one wants to find pertinent subsets of test cases to re-execute to check the absence of previously detected faults (non-regression testing). In this case, identifying minimal subsets of test cases to re-execute while maintaining the coverage of certain test requirements is a must (test suite reduction). Equally important is to find an ideal order in which to re-execute the test cases for detecting regression faults (test suite prioritization).

• The test oracle problem. Running the program with input test data produces outputs that must be controlled with a procedure called a test oracle. Obtaining a complete error-free test oracle, i.e., an oracle which correctly respond to any given input data, is illusory due to the growing complexity of modern software systems. In fact, only a deep knowledge of the system under test enables the writing of useful test oracles. Automating the production of test oracles is thus an issue. Actually, the test oracles used in practice are often restricted to trivial properties that can easily be observed by the developers (for instance to check whether the program crashes or not). This oversimplified treatment of the problem is highly prejudicial to the quality of programs, which may contain a lot of residual bugs. For critical software systems, manual oracles are still in use, meaning that the outcome of the programs are carefully examined by validation engineers. A challenge in this area consists in finding ways to automatically produce these oracles from models, templates or other executable programs.

1.2 Constraint-Based Testing

Unexpectedly, the above-mentioned challenges require to solve hard combinatorial problems. Firstly, the search space originating from an unbounded input space or a large number of different program behaviors can be incredibly huge, sometimes even up to the limit of all available computer resources. Secondly, the test objectives that are required to be covered to ensure a certain level of quality of the program under test can lead to the characterization of very small areas of the input search space. So, selecting test data within those areas becomes like finding a needle in a haystack. For instance, finding an input data such that the execution flow of the program can reach an deeply imbricated line of code or finding a program behavior which provokes a memory leak or exhaust the available memory of the computer are typical concrete examples of hard-to-find test inputs. Despite theoretical limitations of automatic test case generation2, dealing with these hard combinatorial problems for real-world programs is challenging.

Constraint-based testing was introduced to tackle these problems for various applications, including automatic white-box test inputs generation for imperative programs [3–12] and object-oriented programs [13–18], test pattern...

Produkt-Details

• Erscheint lt. Verlag: 18.8.2015
• Sprache: englisch
• Themenwelt: Mathematik / Informatik ► Informatik ► Software Entwicklung
• Themenwelt: Mathematik / Informatik ► Informatik ► Theorie / Studium
• Themenwelt: Informatik ► Weitere Themen ► Hardware
• Themenwelt: Mathematik / Informatik ► Mathematik
• Themenwelt: Technik
• ISBN-10: 0-12-802339-2 / 0128023392
• ISBN-13: 978-0-12-802339-6 / 9780128023396