LDC Lit-based testsuite

From D Wiki
Revision as of 10:02, 25 May 2016 by JohanEngelen (talk | contribs) (LDC-specific features)
Jump to: navigation, search

This page describes the Lit-based test suite used for testing LDC.


Lit is a tool created by the LLVM developers to run compiler tests, written in Python.
Read about it from the horse's mouth: LLVM Blog: 'lit' it.
Lit is an important piece of LLVM's Testing Infrastructure.

LDC uses the Python package as available from https://pypi.python.org/pypi/lit , which can be best installed using Python's pip.

The rest of this page will describe important functionality of the testsuite. It is very instructive to look at LLVM/Clang/LDC tests to see how different kind of tests can be constructed!


FileCheck is an LLVM tool that reads a file passed as argument and checks whether the input received from stdin corresponds to what that file says it should look like. Confusing? An example:

   FileCheck test.d < test.ll

That command loads FileCheck with file test.d. That file contains directives for FileCheck (starting with // CHECK, telling it what to look for in stdin. The directive // CHECK: LDC is awesome will check that "LDC is awesome" appears in stdin. FileCheck will succeed when all checks are satisfied.

A typical test case does:

  • Run LDC with some commandline flags and a D input file, and have it output LLVM IR to a temporary file.
  • Run FileCheck with the D input file as parameter, passing the temporary LLVM IR file through stdin

A simple test case example

To add a test case to the testsuite, all you have to do is create a file inside a subdirectory of the tests directory (not the d2 subdirectory!).

Let's say we created a file tests/codegen/attr_fastmath.d:

 1 // RUN: %ldc -c -output-ll -of=%t.ll %s && FileCheck %s < %t.ll
 3 import ldc.attributes;
 5 @fastmath
 6 double foo(double a, double b)
 7 {
 8 // CHECK: fmul fast
 9     auto c += a * b;
10 }

Special directives for Lit and FileCheck are given in commented out parts of a file (e.g. with "//" in C/C++/D source and with ";" in LLVM IR source).
The example contains a Lit directive on line 1 and a FileCheck directive on line 8.

On line 1, "// RUN:" signals Lit that this file is a testcase (indeed!) and also instructs Lit what to do for this test. The string following "RUN:" is just the shell command line (with some substitutions and other special tricks).

  • %s is substituted with the current file
  • %t is substituted with a temporary file, so %t.ll becomes a temp file with ".ll" extension.

In this example, LDC will be run with the current file as input and flags -c -output-ll where the output is directed to a temporary file that is sent to FileCheck.

On line 8, "// CHECK:" instructs FileCheck to look for the string "fmul fast" anywhere in the input received from stdin. Remember that the LLVM IR generated by LDC is passed through stdin (via file %t.d), and so in this test we check that the generated LLVM IR contains the string "fmul fast".Often you will have to be more precise about where "fmul fast" should occur. What if there are more multiplications in the source file? What if the D source contains a line string str = "fmul fast is great";?!
FileCheck's documentation contains a tutorial displaying FileCheck's functionality for narrowing tests. Make sure you understand the directives CHECK, CHECK-NOT, CHECK-LABEL, CHECK-SAME, CHECK-NEXT, CHECK-DAG.

FileCheck directives can use regular expressions, that are put in between double curly braces {{ ... }} or double rectangular braces [[VARNAME: ... ]] for capturing a matched string in VARNAME.


Lit will substitute a few special strings inside "// RUN:" directives. Substitutions start with a %.

Lit pre-defined substitutions

Replaced by a single %. This allows escaping other substitutions.
File path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF/foo_test.s"
Directory path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF"
File path to a temporary file name that could be used for this test case. The file name won’t conflict with other test cases. You can append to it if you need multiple temporaries. Example: "/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp"
Directory of %t. Example: "/home/user/llvm.build/test/MC/ELF/Output"

(This is not an exhaustive list, see Lit's reference for all)

LDC-specific substitutions

Replaced by the LDC binary built by CMake. You will probably always use this, because you don't want to test the LDC that is found on the system path! Example: "/Users/johan/ldc/pgo/build38release/bin/ldc2"
The file extension used for executables on the current OS. On Windows: ".exe". Empty string on other OSes.
The file extension used for object files on the current OS. On Windows: ".obj". Other OSes: ".o"

To add more substitutions, look at tests/lit.site.cfg.in.


Some tests can/should only be run on certain setups. You can specify which "features" a test requires, for example:

   // REQUIRES: atleast_llvm307

which means that the test requires at least LLVM 3.7 to be run. The test will show up as "unsupported" when the requirements are not met.

LDC-specific features

llvm305, llvm306, llvm307, etc.
The feature is set when LDC's LLVM version is equal to <number>. (LLVM 3.7 == llvm307)
atleast_llvm305, atleast_llvm306, atleast_llvm307, etc.
The feature is set when LDC's LLVM version is equal or higher than <number>.
Darwin, Linux, Windows, ...
The string returned by Python's platform.system() is added as a feature.
target_X86, target_ARM, target_PowerPC, target_AArch64, ...
The feature is set when LDC's LLVM can target that architecture.
The feature is set when LDC targets Windows 32bit x86 by default.

To add more substitutions, look at tests/lit.site.cfg.in.

Fail tests

If the test command is supposed to fail, you can use the not tool:

// RUN: not %ldc -commandflagdoesnotexist

Use case: test compiler diagnostics!

If a test is known to fail on a certain platform, you can mark it as XFAIL:

// XFAIL: Windows

This means that the test is expected to fail on Windows (i.e. bug still to be fixed!).

More complicated example

Consider the file tests/codegen/attr_target_x86.d:

 1 // REQUIRES: atleast_llvm307
 2 // REQUIRES: target_X86
 4 // RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-ll -of=%t.ll %s && FileCheck %s --check-prefix LLVM < %t.ll
 5 // RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-s -of=%t.s %s && FileCheck %s  --check-prefix ASM < %t.s
 7 import ldc.attributes;
 9 // LLVM-LABEL: define{{.*}} void @{{.*}}foo
10 // ASM-LABEL: _D15attr_target_x863fooFPfPffZv:
11 void foo(float *A, float* B, float K) {
12     for (int i = 0; i < 128; ++i)
13         A[i] *= B[i] + K;
14 // ASM-NOT: addps
15 }
17 // LLVM-LABEL: define{{.*}} void @{{.*}}foo_sse
18 // LLVM-SAME: #[[SSE:[0-9]+]]
19 // ASM-LABEL: _D15attr_target_x867foo_sseFPfPffZv:
20 @(target("sse"))
21 void foo_sse(float *A, float* B, float K) {
22     for (int i = 0; i < 128; ++i)
23         A[i] *= B[i] + K;
24 // ASM: addps
25 }
27 // LLVM-DAG: attributes #[[SSE]] = {{.*}} "target-features"="+sse"

Multiple RUN directives

There can be multiple RUN directives! Important: they can (and will) be run in parallel! Their execution should not depend on each other in any way (make sure they use different intermediate files).