SuperTest compiler test and validation suite

SuperTest is the most comprehensive test and validation suite for compilers. Based upon 35+ years of our experience in compiler construction and validation, SuperTest provides a unique level of compiler test coverage and will find difficult problem cases in any compiler.
SuperTest starts where other suites end.

It's all about confidence

You, as a compiler vendor and/or user are responsible to produce or use quality compilers that are able to face the outside world. As a software engineer, you know how much you trust on the correctness of the compiler. Each and every line of code you write needs to be translated correctly, or you will never get your own software to work.

Any application engineer takes the robustness of his compiler tools for granted, yet expects ever increasing performance from it. Compilers are complex products, translating applications into machine code and require extensive testing for language conformance, correctness and quality. With its millions of tests, SuperTest assists both compiler developers to validate their compiler and compiler users to check the provided compilers before use.

The content

The core of SuperTest is a large collection of self-testing programs, which can roughly be partitioned into a number of classes:

• Conformance test. These are tests that check the basic constructs of the language.
• Tests that are derived from earlier compiler problems. Experience learns that these same problems tend to surface in other compilers as well.
• Negative tests. Compilers used during development will see more incorrect programs than correct ones. Not handling errors gracefully will erode the programmer's trust in the compiler.
• Stress tests. Seeking the limits of standard conformance in terms of sizes, number of basic blocks, expression depth, number of live variables, etc.
• Generated tests. Systematically exploring combinations of operators, types, storage classes and constant values.
• Tests for the C++ Standard Library that verifies both conformance to the standard and correctness of the implementation.

The suite is organized in a directory tree of test files, each file contains a couple of runtime tests. The directory structure follows the chapters and paragraphs of the appropriate language standard. Subsets of the full suite can be hierarchically defined using single test files, suite subdirectories and explicitly listed sets as primitives.

The tools

The main tool is valid, a driver program which will interpret the test set definition and other test run parameters. Its feeds the test through the compiler ad optionally running the resulting programs. At essential points user-configurable scripts are used, e.g. to define the command line interface to the compiler and to run the resulting test program. These script plug-ins allow for essential flexibility like running cross validations, remote execution, simulated execution and statistics gathering. The driver program allows for multi-threaded and remote execution to optimally employ multi processor machines.

Additional tools are:

• A report generator, giving an excellent overview of the results of a validation run, either in plain text or in html. It will generate reports of multiple runs side-by-side, providing extensive regression reports against a reference run.
• Tempest, a template based test generator, that can be used to generate random variations of any particular test.
• Cvalsplit, a tool to handle multi file test programs
• Leash, a utility that can limit runtime and output of the compiler and test program invocations.
• Parallel testing, automatically splitting validation runs into preconfigured numbers of threads.
• ABI Tester, expose typical errors as found in calling conventions and binary interfaces between software libraries

Contact ACE

If you are interested in more details about using SuperTest for testing the quality and robustness of the compilers that you use or develop, then do not hesitate to contact us or fill out the web-request form.