LDC Lit-based testsuite

This page describes the Lit-based test suite used for testing LDC.

How to run the tests

You can run the tests by going into the tests directory of your build directory (important: don't go into the source's tests directory checked out from github!), and then you run the runlit.py script:

   .../ldc/build/tests>  python runlit.py -v .

With -v the output becomes more verbose such that you can see what exactly failed if a test fails.
The "." means: all tests in this path. Important: the path is from the tests directory in the source directory! To run just one test, you do:

   .../ldc/build/tests>  python runlit.py -v codegen/align.d

To run all tests of a test subdirectory, you can use the "." as wildcard:

   .../ldc/build/tests>  python runlit.py -v codegen/.
   .../ldc/build/tests>  python runlit.py -v debuginfo/.

Lit

Lit is a tool created by the LLVM developers to run compiler tests, written in Python.
Read about it from the horse's mouth: LLVM Blog: 'lit' it.
Lit is an important piece of LLVM's Testing Infrastructure.

LDC uses the Python package as available from https://pypi.python.org/pypi/lit , which can be best installed using Python's pip.

The rest of this page will describe important functionality of the testsuite. It is very instructive to look at LLVM/Clang/LDC tests to see how different kind of tests can be constructed!

FileCheck

FileCheck is an LLVM tool that reads a file passed as argument and checks whether the input received from stdin corresponds to what that file says it should look like. Confusing? An example:

   FileCheck test.d < test.ll

That command loads FileCheck with file test.d. That file contains directives for FileCheck (starting with // CHECK, telling it what to look for in stdin. The directive // CHECK: LDC is awesome will check that "LDC is awesome" appears in stdin. FileCheck will succeed when all checks are satisfied.

A typical test case does:

• Run LDC with some commandline flags and a D input file, and have it output LLVM IR to a temporary file.
• Run FileCheck with the D input file as parameter, passing the temporary LLVM IR file through stdin

A simple test case example

To add a test case to the testsuite, all you have to do is create a file inside a subdirectory of the tests directory (not the d2 subdirectory!).

Let's say we created a file tests/codegen/attr_fastmath.d:

 1 // RUN: %ldc -c -output-ll -of=%t.ll %s && FileCheck %s < %t.ll
 2 
 3 import ldc.attributes;
 4 
 5 @fastmath
 6 double foo(double a, double b)
 7 {
 8 // CHECK: fmul fast
 9     auto c += a * b;
10 }

Special directives for Lit and FileCheck are given in commented out parts of a file (e.g. with "//" in C/C++/D source and with ";" in LLVM IR source).
The example contains a Lit directive on line 1 and a FileCheck directive on line 8.

On line 1, "// RUN:" signals Lit that this file is a testcase (indeed!) and also instructs Lit what to do for this test. The string following "RUN:" is just the shell command line (with some substitutions and other special tricks).

• %s is substituted with the current file
• %t is substituted with a temporary file, so %t.ll becomes a temp file with ".ll" extension.

In this example, LDC will be run with the current file as input and flags -c -output-ll where the output is directed to a temporary file that is sent to FileCheck.

On line 8, "// CHECK:" instructs FileCheck to look for the string "fmul fast" anywhere in the input received from stdin. Remember that the LLVM IR generated by LDC is passed through stdin (via file %t.ll), and so in this test we check that the generated LLVM IR contains the string "fmul fast".Often you will have to be more precise about where "fmul fast" should occur. What if there are more multiplications in the source file? What if the D source contains a line string str = "fmul fast is great";?!
FileCheck's documentation contains a tutorial displaying FileCheck's functionality for narrowing tests. Make sure you understand the directives CHECK, CHECK-NOT, CHECK-LABEL, CHECK-SAME, CHECK-NEXT, CHECK-DAG.

FileCheck directives can use regular expressions, that are put in between double curly braces {{ ... }} or double rectangular braces [[VARNAME: ... ]] for capturing a matched string in VARNAME.

Substitutions

Lit will substitute a few special strings inside "// RUN:" directives. Substitutions start with a %.

Lit pre-defined substitutions

 %%

Replaced by a single %. This allows escaping other substitutions.

 %s

File path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF/foo_test.s"

 %S

Directory path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF"
This is useful for accessing additional files (put in the inputs subfolder), for example %S/inputs/attr_weak_input.d.

 %t

File path to a temporary file name that could be used for this test case. The file name won’t conflict with other test cases. You can append to it if you need multiple temporaries. Example: "/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp"

(This is not an exhaustive list, see Lit's reference for all)

LDC-specific substitutions

 %ldc

Replaced by the LDC binary built by CMake. You will probably always use this, because you don't want to test the LDC that is found on the system path! Example: "/Users/johan/ldc/pgo/build38release/bin/ldc2"

 %arch

Windows only. Replaced by "x86" or "x64" depending on the default target (32bit or 64bit, respectively).

 %cdb

Windows only, and only when feature "cdb" is set. Replaced by the system cdb binary (the path is different for the 32bit and 64bit environments).

 %exe

The file extension used for executables on the current OS. On Windows: ".exe". Empty string on other OSes.

 %lib

The file extension used for libraries on the current OS. On Windows: ".lib". Other OSes: ".a"

 %obj

The file extension used for object files on the current OS. On Windows: ".obj". Other OSes: ".o"

 %profdata

Replaced by the ldc-profdata binary built by CMake. (similar to %ldc, only available when LDC's PGO is available)

 %profdata

Replaced by the ldc-prune-cache binary built by CMake. (similar to %ldc)

 %_gdb_dflags

Only available when feature "gdb" is set. Replaced with compile flags for LDC (!) needed when testing with gdb.

 %gnu_make

Replaced by the path to the gnu make binary (gmake, gnumake or make depending on your operating system).

To add more substitutions, look at tests/lit.site.cfg.in.

"Features"

Some tests can/should only be run on certain setups. You can specify which "features" a test requires, for example:

   // REQUIRES: atleast_llvm307

which means that the test requires at least LLVM 3.7 to be run. The test will show up as "unsupported" when the requirements are not met.

LDC-specific features

llvm<number>

llvm305, llvm306, llvm307, etc.

The feature is set when LDC's LLVM version is equal to <number>. (LLVM 3.7 == llvm307)

atleast_llvm<number>

atleast_llvm305, atleast_llvm306, atleast_llvm307, etc.

The feature is set when LDC's LLVM version is equal or higher than <number>.

atmost_llvm<number>

atmost_llvm305, atmost_llvm306, atmost_llvm307, etc.

The feature is set when LDC's LLVM version is equal or lower than <number>.

cdb

The feature is set when the cdb debugger is available (Windows).

Darwin, Linux, Windows, ...

The string returned by Python's platform.system() is added as a feature.

gdb

The feature is set when the gdb debugger is available.

host_X86, host_ARM, host_PowerPC, host_AArch64, ...

Compiler host architecture

LTO

The feature is set when LTO is supported (by LDC and linker).

target_X86, target_ARM, target_PowerPC, target_AArch64, ...

The feature is set when LDC's LLVM can target that architecture.

Windows_x86

The feature is set when LDC targets Windows 32bit x86 by default.

Windows_x64

The feature is set when LDC targets Windows 64bit x86_64 by default.

To add more features, look at tests/lit.site.cfg.in.

Environment variables

You can set environment variables using the env command:

// RUN: env SOME_ENV_VAR=hello %ldc %s

Fail tests

If the test command is supposed to fail, you can use the not tool:

// RUN: not %ldc -commandflagdoesnotexist

Use case: test compiler diagnostics!

If a test is known to fail on a certain platform, you can mark it as XFAIL:

// XFAIL: Windows

This means that the test is expected to fail on Windows (i.e. bug still to be fixed!).

More complicated example

Consider the file tests/codegen/attr_target_x86.d:

 1 // REQUIRES: atleast_llvm307
 2 // REQUIRES: target_X86
 3 
 4 // RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-ll -of=%t.ll %s && FileCheck %s --check-prefix LLVM < %t.ll
 5 // RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-s -of=%t.s %s && FileCheck %s  --check-prefix ASM < %t.s
 6 
 7 import ldc.attributes;
 8 
 9 // LLVM-LABEL: define{{.*}} void @{{.*}}foo
10 // ASM-LABEL: _D15attr_target_x863fooFPfPffZv:
11 void foo(float *A, float* B, float K) {
12     for (int i = 0; i < 128; ++i)
13         A[i] *= B[i] + K;
14 // ASM-NOT: addps
15 }
16 
17 // LLVM-LABEL: define{{.*}} void @{{.*}}foo_sse
18 // LLVM-SAME: #[[SSE:[0-9]+]]
19 // ASM-LABEL: _D15attr_target_x867foo_sseFPfPffZv:
20 @(target("sse"))
21 void foo_sse(float *A, float* B, float K) {
22     for (int i = 0; i < 128; ++i)
23         A[i] *= B[i] + K;
24 // ASM: addps
25 }
26 
27 // LLVM-DAG: attributes #[[SSE]] = {{.*}} "target-features"="+sse"

Multiple RUN directives

There can be multiple RUN directives! Important: they can (and will) be run in parallel! Their execution should not depend on each other in any way (make sure they use different intermediate files).

FileCheck prefix

When --check-prefix is specified for FileCheck, the specified string is recognized instead of the default "CHECK" string. Useful when running multiple tests with different compiler settings on the same file.

My workflow when working on LDC (Johan)

I find it very convenient to work with the lit-based testsuite during development and bug fixing.
I have two terminal windows open: one where I build LDC, and another where I run the test(s). So in one terminal I (pretty much) only run the command:

   ...../ldc/build>  ninja

And in the other terminal I (pretty much) only run the command (assuming you are working on something related to the codegen/align.d test:

   ...../ldc/build/tests>  ./runlit.py -v codegen/align.d

(note the directories that I am in)

The workflow then is:

1. Change a bit of LDC code, or add/change a test
2. run ninja in terminal 1
3. run runlit.py in terminal 2
4. goto 1.

Note: $ ninja builds the whole LDC suite. Most of the time, you'll want to only rebuild the ldc2 binary. You can do that with $ ninja ldc2.

---