Advanced Software Testing - What does that mean?

The term “software quality” can be interpreted in many ways, often depending on one’s perspective. For example, a developer may consider the quality of software to be dependent on the design and maintainability of the code. On the other hand, an end-user may consider speed and reliability to be indicators of high-quality software. In general, though, the perceptions of quality are rooted in how well software conforms to its requirements. Software requirements, when adequately defined, describe how software should behave functionally (how to respond to particular input) and non-functionally (maintainability, security, performance, etc.).

Well tested software decreases the possibility of failure at a greater extent and improves the quality and dependability at a greater extent. Software Testing may eventually reduce Errors, Failures, Faults or Incidents. There are different types of testing that can be initiated to cover different areas of a software system. The testing path for the Software include (Unit Tests, Integration Tests, Functionality Tests, UI Tests, Usability Tests, and User Acceptance Tests etc.).

The requirements themselves were derived from the accompanying documentation and were functional in nature. Thus, we are measuring the quality of the units--methods, in this case--by how well they adhere to the specified requirements. Here, we tried to cover basic Unit testing scenarios.

To do that we have created a Test Suite which has numerous test cases that exercises different units being revised by correction and repairing faults caused by errors. The test cases also find the failures that was caused by faults and eventually caused by specific errors. Test cases may have many test stubs and test drivers. Software testing needs rigorous test planning at the first place. Which may include: Analysis Software Model/Requirements and estimate expected results. Then deriving effective test cases followed by executing tests and compare results with expected results. At the end of test suite generation there is a need to verify the coverage of the test suite.

Our strategy, at its core, is driven by the requirements specified in the documentation. We do not want to make assumptions, either based on how we think the methods should behave, or how we were observe them behaving. In essence, we’re focused on testing how well the methods conform to their specifications.

Basic Testing Definitions: 

• Errors: Mistakes that are being initiated by people (There were no Database Cluster environment) 
• Fault: Fault is the result of an Error (Database is offline or down) 
• Failure: Failure occurs when fault executes (Web system cannot communicate with the database) 
• Incident: Incidents are basically consequences of failure (I cannot log into my bank account) 

Goals of Software Testing:

According to Dijkstra, 1972 - “Program testing can be used to show the presence of bugs, but never to show their absence”. The main goal of software testing is to have dependable software on which we can gain only sufficient confidence. Which eventually means, the software may be faulty even we try to cover all bits and pieces of the SUT (System Under Test) in out testing procedure.

Testing Process Overview:

• Derive Test cases from Software Representations 
• Estimate the expected result for each test case (Oracle) 
• Execute the test cases 
• Get test results 
• Compare Test Result with Expected Result

Testing should be effective, repeatable, systematic, automated and should help locate faults for debugging the program. Dependability of software systems may have several aspects (Correctness, Reliability, Availability, Safety, Robustness or may be combination of those). We need to determine when to stop testing by having sufficient confidence.

Black and White Box Testing:

Black-box testing is a testing process through which the specifications of the software system are being tested where as White-box testing is for the Implementation of the software solution. Missing functionality cannot be revealed by white-box testing whereas unexpected functionality cannot be revealed by black box testing.

There are different techniques for black box testing:

• Equivalence Class Partitioning: partitions of the input set in which input data have the same effect on the program. By completeness we mean entire input set is covered. By WECT we mean, choosing one variable value form each equivalence class (one ai, bi, and ci) such that all classes are covered. By SECT we mean the Cartesian product of the partition subsets. 
• Boundary-Value Analysis: Testing the boundary values of input space. Min, Min+, Max, Max- are some boundary values for testing 
• Category-Partition: The system is divided into several “functions” that can be tested. Categories are the major properties or characteristics for each parameter. 
• Decision tables: It helps to express test requirements in a usable form. 
• Cause-Effect Graphs: It is a graphical technique that helps derive decision tables 
• Logic Functions: It maps from n Boolean input variables to 1 Boolean output variable 
• Black-box Test Criteria: In Black-box testing criteria there exist number of test coverage criteria:

                   Predicate coverage
                   Clause coverage
                   Combinatorial coverage
                   (in)active clause coverage

According to Marick, the following things need to be addressed to have a large scale testing: 

• To test every functionality, generate Black-box tests from requirements 
• Check the code coverage after the functional tests are all verified. 
• In case of imperfect structural coverage, generate functional tests for additional coverage. 

There are different techniques for white box testing as well:

• Control flow coverage 

Statement/Node Coverage (Covering all nodes in CFG)

Edge Coverage (Select a test set T such that, by executing P for each test case t in T, each edge of P’s control flow graph is traversed at least once)

Condition Coverage (Select a test set T such that, by executing P for each element in T, each edge of P’s control flow graph is traversed, and all possible values of the constituents of compound conditions (defined below) are exercised at least once)

Path Coverage (Select a test set T such that, by executing P for each test case t in T, all paths leading from the initial to the final node of P’s control flow graph are traversed)

• Data flow coverage: Assignment of values to variables and their uses 
• Analyzing coverage data
• Mutation Testing: It is typically Fault based testing. Faults are being injected to monitor different behavior (these faults versions are called mutants). Then kill the mutants one by one. By killing the mutants, we mean if the test cases detect the differences then the mutants are killed and test is considered adequate if all the mutants get killed by the test cases. 
• Integration testing 
• Generating test data 
• Tools

Ref: Dr. Vahid Garousi