LDC Lit-based testsuite

This page describes the Lit-based test suite used for testing LDC.

Lit

Lit is a tool created by the LLVM developers to run compiler tests, written in Python.
Read about it from the horse's mouth: LLVM Blog: 'lit' it.
Lit is an important piece of LLVM's Testing Infrastructure.

LDC uses the Python package as available from https://pypi.python.org/pypi/lit , which can be best installed using Python's pip.

The rest of this page will describe important functionality of the testsuite. It is very instructive to look at LLVM/Clang/LDC tests to see how different kind of tests can be constructed!

FileCheck

FileCheck is an LLVM tool that reads a file passed as argument and checks whether the input received from stdin corresponds to what that file says it should look like. Confusing? An example:

   FileCheck test.d < test.ll

That command loads FileCheck with file test.d. That file contains directives for FileCheck (starting with // CHECK, telling it what to look for in stdin. The directive // CHECK: LDC is awesome will check that "LDC is awesome" appears in stdin. FileCheck will succeed when all checks are satisfied.

A typical test case does:

• Run LDC with some commandline flags and a D input file, and have it output LLVM IR to a temporary file.
• Run FileCheck with the D input file as parameter, passing the temporary LLVM IR file through stdin

A simple test case example

To add a test case to the testsuite, all you have to do is create a file inside a subdirectory of the tests directory (not the d2 subdirectory!).

Let's say we created a file tests/codegen/attr_fastmath.d:

 1 // RUN: %ldc -c -output-ll -of=%t.ll %s && FileCheck %s < %t.ll
 2 
 3 import ldc.attributes;
 4 
 5 @fastmath
 6 double foo(double a, double b)
 7 {
 8 // CHECK: fmul fast
 9     auto c += a * b;
10 }

Special directives for Lit and FileCheck are given in commented out parts of a file (e.g. with "//" in C/C++/D source and with ";" in LLVM IR source).
The example contains a Lit directive on line 1 and a FileCheck directive on line 8.

On line 1, "// RUN:" signals Lit that this file is a testcase (indeed!) and also instructs Lit what to do for this test. The string following "RUN:" is just the shell command line (with some substitutions and other special tricks).

• %s is substituted with the current file
• %t is substituted with a temporary file, so %t.ll becomes a temp file with ".ll" extension.

In this example, LDC will be run with the current file as input and flags -c -output-ll where the output is directed to a temporary file that is sent to FileCheck.

On line 8, "// CHECK:" instructs FileCheck to look for the string "fmul fast" anywhere in the input received from stdin. Remember that the LLVM IR generated by LDC is passed through stdin (via file %t.d), and so in this test we check that the generated LLVM IR contains the string "fmul fast".Often you will have to be more precise about where "fmul fast" should occur. What if there are more multiplications in the source file? What if the D source contains a line string str = "fmul fast is great";?!
FileCheck's documentation contains a tutorial displaying FileCheck's functionality for narrowing tests. Make sure you understand the directives CHECK, CHECK-NOT, CHECK-LABEL, CHECK-SAME, CHECK-NEXT, CHECK-DAG.

FileCheck directives can use regular expressions, that are put in between double curly braces {{ ... }} or double rectangular braces [[VARNAME: ... ]] for capturing a matched string in VARNAME.

More complicated example

// Tests @target attribute for x86

// REQUIRES: atleast_llvm307
// REQUIRES: target_X86

// RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-ll -of=%t.ll %s && FileCheck %s --check-prefix LLVM < %t.ll
// RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-s -of=%t.s %s && FileCheck %s  --check-prefix ASM < %t.s

import ldc.attributes;

// LLVM-LABEL: define{{.*}} void @{{.*}}foo
// ASM-LABEL: _D15attr_target_x863fooFPfPffZv:
void foo(float *A, float* B, float K) {
    for (int i = 0; i < 128; ++i)
        A[i] *= B[i] + K;
// ASM-NOT: addps
}

// LLVM-LABEL: define{{.*}} void @{{.*}}foo_sse
// LLVM-SAME: #[[SSE:[0-9]+]]
// ASM-LABEL: _D15attr_target_x867foo_sseFPfPffZv:
@(target("sse"))
void foo_sse(float *A, float* B, float K) {
    for (int i = 0; i < 128; ++i)
        A[i] *= B[i] + K;
// ASM: addps
}

---