SuperTest compiler test and validation suite

SuperTest is the most comprehensive test and validation suite for compilers. Based upon 35+ years of our experience in compiler construction and validation, SuperTest provides a unique level of compiler test coverage and will find difficult problem cases in any compiler.
SuperTest
starts where other suites end.

It's all about confidence

You, as a compiler vendor and/or user are responsible to produce or use quality compilers that are able to face the outside world. As a software engineer, you know how much you trust on the correctness of the compiler. Each and every line of code you write needs to be translated correctly, or you will never get your own software to work.

Any application engineer takes the robustness of his compiler tools for granted, yet expects ever increasing performance from it. Compilers are complex products, translating applications into machine code and require extensive testing for language conformance, correctness and quality. With its millions of tests, SuperTest assists both compiler developers to validate their compiler and compiler users to check the provided compilers before use.

The content

The core of SuperTest is a large collection of self-testing programs, which can roughly be partitioned into a number of classes:

  • Conformance test. These are tests that check the basic constructs of the language.
  • Tests that are derived from earlier compiler problems. Experience learns that these same problems tend to surface in other compilers as well.
  • Negative tests. Compilers used during development will see more incorrect programs than correct ones. Not handling errors gracefully will erode the programmer's trust in the compiler.
  • Stress tests. Seeking the limits of standard conformance in terms of sizes, number of basic blocks, expression depth, number of live variables, etc.
  • Generated tests. Systematically exploring combinations of operators, types, storage classes and constant values.
  • Tests for the C++ Standard Library that verifies both conformance to the standard and correctness of the implementation.

The suite is organized in a directory tree of test files, each file contains a couple of runtime tests. The directory structure follows the chapters and paragraphs of the appropriate language standard. Subsets of the full suite can be hierarchically defined using single test files, suite subdirectories and explicitly listed sets as primitives.

The tools

Check here how to install SuperTest.

The main tool is valid, a driver program which will interpret the test set definition and other test run parameters. Its feeds the test through the compiler ad optionally running the resulting programs. At essential points user-configurable scripts are used, e.g. to define the command line interface to the compiler and to run the resulting test program. These script plug-ins allow for essential flexibility like running cross validations, remote execution, simulated execution and statistics gathering. The driver program allows for multi-threaded and remote execution to optimally employ multi processor machines.

Additional tools are:

  • A report generator, giving an excellent overview of the results of a validation run, either in plain text or in html. It will generate reports of multiple runs side-by-side, providing extensive regression reports against a reference run.
  • Tempest, a template based test generator, that can be used to generate random variations of any particular test.
  • Cvalsplit, a tool to handle multi file test programs
  • Leash, a utility that can limit runtime and output of the compiler and test program invocations.
  • Parallel testing, automatically splitting validation runs into preconfigured numbers of threads.
  • ABI Tester, expose typical errors as found in calling conventions and binary interfaces between software libraries

 

Features

  • >3 million C and C++ language conformance, correctness
    and compiler quality checks
  • Powerful and flexible test generator
  • Test for C++ Standard Library
  • Tempest test generator
  • Regression tests for known bugs
  • Positive and negative testing
  • Seamless integration in compiler development environment
  • Easy addition of tests
  • Selective subset testing, including previously failed tests
  • Clear reporting in HTML
  • Parallel and remote testing

Conformance testing

  • ISO 9899:1990 C standard
  • ISO 9899:1999 C99 standard
  • TR18037 Embedded C extension
  • ISO 14882 C++ standard

Full Expert support

  • Maintenance support
  • Development support

Licensing

  • Source Site
  • Corporate

Contact ACE

If you are interested in more details about using SuperTest for testing the quality and robustness of the compilers that you use or develop, then do not hesitate to contact us or fill out the web-request form.

CoSy is a registered trademark of ACE Associated Computer Experts bv. CoSy Express is a trademark of ACE Associated Computer Experts bv. SuperTest is a trademark of ACE Associated Computer Experts bv.