LDC Lit-based testsuite

From D Wiki
Revision as of 09:43, 25 May 2016 by JohanEngelen (talk | contribs) (More complicated example)
Jump to: navigation, search

This page describes the Lit-based test suite used for testing LDC.

Lit

Lit is a tool created by the LLVM developers to run compiler tests, written in Python.
Read about it from the horse's mouth: LLVM Blog: 'lit' it.
Lit is an important piece of LLVM's Testing Infrastructure.

LDC uses the Python package as available from https://pypi.python.org/pypi/lit , which can be best installed using Python's pip.

The rest of this page will describe important functionality of the testsuite. It is very instructive to look at LLVM/Clang/LDC tests to see how different kind of tests can be constructed!

FileCheck

FileCheck is an LLVM tool that reads a file passed as argument and checks whether the input received from stdin corresponds to what that file says it should look like. Confusing? An example:

   FileCheck test.d < test.ll

That command loads FileCheck with file test.d. That file contains directives for FileCheck (starting with // CHECK, telling it what to look for in stdin. The directive // CHECK: LDC is awesome will check that "LDC is awesome" appears in stdin. FileCheck will succeed when all checks are satisfied.

A typical test case does:

  • Run LDC with some commandline flags and a D input file, and have it output LLVM IR to a temporary file.
  • Run FileCheck with the D input file as parameter, passing the temporary LLVM IR file through stdin

A simple test case example

To add a test case to the testsuite, all you have to do is create a file inside a subdirectory of the tests directory (not the d2 subdirectory!).

Let's say we created a file tests/codegen/attr_fastmath.d:

 1 // RUN: %ldc -c -output-ll -of=%t.ll %s && FileCheck %s < %t.ll
 2 
 3 import ldc.attributes;
 4 
 5 @fastmath
 6 double foo(double a, double b)
 7 {
 8 // CHECK: fmul fast
 9     auto c += a * b;
10 }

Special directives for Lit and FileCheck are given in commented out parts of a file (e.g. with "//" in C/C++/D source and with ";" in LLVM IR source).
The example contains a Lit directive on line 1 and a FileCheck directive on line 8.

On line 1, "// RUN:" signals Lit that this file is a testcase (indeed!) and also instructs Lit what to do for this test. The string following "RUN:" is just the shell command line (with some substitutions and other special tricks).

  • %s is substituted with the current file
  • %t is substituted with a temporary file, so %t.ll becomes a temp file with ".ll" extension.

In this example, LDC will be run with the current file as input and flags -c -output-ll where the output is directed to a temporary file that is sent to FileCheck.

On line 8, "// CHECK:" instructs FileCheck to look for the string "fmul fast" anywhere in the input received from stdin. Remember that the LLVM IR generated by LDC is passed through stdin (via file %t.d), and so in this test we check that the generated LLVM IR contains the string "fmul fast".Often you will have to be more precise about where "fmul fast" should occur. What if there are more multiplications in the source file? What if the D source contains a line string str = "fmul fast is great";?!
FileCheck's documentation contains a tutorial displaying FileCheck's functionality for narrowing tests. Make sure you understand the directives CHECK, CHECK-NOT, CHECK-LABEL, CHECK-SAME, CHECK-NEXT, CHECK-DAG.

FileCheck directives can use regular expressions, that are put in between double curly braces {{ ... }} or double rectangular braces [[VARNAME: ... ]] for capturing a matched string in VARNAME.

Substitutions

Lit will substitute a few special strings inside "// RUN:" directives. Substitutions start with a %.

Lit pre-defined substitutions

 %%
Replaced by a single %. This allows escaping other substitutions.
 %s
File path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF/foo_test.s"
 %S
Directory path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF"
 %t
File path to a temporary file name that could be used for this test case. The file name won’t conflict with other test cases. You can append to it if you need multiple temporaries. Example: "/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp"
 %T
Directory of %t. Example: "/home/user/llvm.build/test/MC/ELF/Output"

(This is not an exhaustive list, see Lit's reference for all)

LDC-specific substitutions

 %ldc
Replaced by the LDC binary built by CMake. You will probably always use this, because you don't want to test the LDC that is found on the system path! Example: "/Users/johan/ldc/pgo/build38release/bin/ldc2"
 %exe
The file extension used for executables on the current OS. On Windows: ".exe". Empty string on other OSes.
 %obj
The file extension used for object files on the current OS. On Windows: ".obj". Other OSes: ".o"


To add more substitutions, look at tests/lit.site.cfg.in.

"Features"

More complicated example

// Tests @target attribute for x86

// REQUIRES: atleast_llvm307
// REQUIRES: target_X86

// RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-ll -of=%t.ll %s && FileCheck %s --check-prefix LLVM < %t.ll
// RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-s -of=%t.s %s && FileCheck %s  --check-prefix ASM < %t.s

import ldc.attributes;

// LLVM-LABEL: define{{.*}} void @{{.*}}foo
// ASM-LABEL: _D15attr_target_x863fooFPfPffZv:
void foo(float *A, float* B, float K) {
    for (int i = 0; i < 128; ++i)
        A[i] *= B[i] + K;
// ASM-NOT: addps
}

// LLVM-LABEL: define{{.*}} void @{{.*}}foo_sse
// LLVM-SAME: #[[SSE:[0-9]+]]
// ASM-LABEL: _D15attr_target_x867foo_sseFPfPffZv:
@(target("sse"))
void foo_sse(float *A, float* B, float K) {
    for (int i = 0; i < 128; ++i)
        A[i] *= B[i] + K;
// ASM: addps
}

// LLVM-DAG: attributes #[[SSE]] = {{.*}} "target-features"="+sse"

Multiple RUN directives

There can be multiple RUN directives! Important: they can (and will) be run in parallel! Their execution should not depend on each other in any way (make sure they use different intermediate files).