Testing is a major part of the verification process at most embedded software development organizations. Studies estimate that around 25 – 30% of development time is spent on testing, and in some cases, this can be as high as 50% [1].
There are two key goals of verification:
- Ensuring functional correctness and verifying the software meets requirements
- Ensuring run-time correctness and verifying that the implementation is correct and robust
These verification goals are orthogonal to each other and have traditionally been addressed through testing, which include functional tests, performance tests, boundary values tests, robustness tests, random tests, and tests for statement coverage and decision coverage. All of these strategies can be grouped into two brackets:
- Requirement-based testing strategies for functional correctness
- Coverage-based testing strategies for run-time correctness
However, is testing embedded software in this way the best approach to embedded software verification? Here is an explanation from an article on embedded software testing [2]:
“In the ideal case, you want to test every possible behavior in your program. This implies testing every possible combination of inputs or every possible decision path at least once.
This is a noble, but utterly impractical, goal. For example, in The Art of Software Testing, Glen Ford Myers describes a small program with only five decisions that has 10^14 unique execution paths. He points out that if you could write, execute, and verify one test case every five minutes, it would take one billion years to test exhaustively this program.
Obviously, the ideal situation is beyond reach, so you must use approximations to this ideal. As you’ll see, a combination of functional testing and coverage testing provides a reasonable second-best alternative. The basic approach is to select the tests some functional, some coverage that have the highest probability of exposing an error.”
The most glaring example of the weaknesses inherent in only using functional and coverage-based testing is the recent software issue reported by a major airline manufacturer. Despite the DO 178 guidelines that enforce rigorous verification and safety requirements that call for extensive amounts of testing and manual reviews, it failed to prevent an overflow error until after the product was delivered [3].
Clearly, verification that relies solely on testing is incomplete. Fortunately, the advent of tools like Polyspace products can alleviate this problem and verify every possible combination of inputs with every possible decision path.
The Polyspace static analysis solution uses a formal methods technique known as abstract interpretation. This technique bridges the gap between conventional static analysis techniques and dynamic testing by verifying the dynamic properties of software applications at compilation time.
Without executing the program itself, abstract interpretation investigates all possible behaviors of a program—that is, all possible combinations of inputs and all possible execution sequences—in a single pass to determine how and under which conditions the program can fail. You can find more details in this white paper on the Polyspace static analysis solution.
Not only can you analyze every possible behavior of your program to completely verify your software, you can also significantly reduce the effort and development time spent in testing your software. You can leverage the dynamic run time behavior provided by Polyspace tools to:
- Replace or reduce robustness tests
- Perform dead code coverage
- Reduce the number of test cycles
- Identify relevant test vectors
In future posts, we will discuss how Polyspace helps with each of these tasks. The benefit of using this combination of Polyspace static analysis and testing is in the amount of effort you can save and reduce the overall development time. One major automotive supplier was able to reduce its testing cycle by as much as 50%, mostly by replacing robustness tests and reducing the number of test cycles. The artifacts produced from Polyspace demonstrate how to achieve the required level of quality. You can leverage Polyspace tools at various stages of your testing process such as:
- Unit testing
- Integration testing
- Acceptance testing
