View source for LDC Lit-based testsuite

This page describes the Lit-based test suite used for testing LDC.

== How to run the tests ==

You can run the tests by going into the <tt>tests</tt> directory of your <i>build</i> directory (important: don't go into the source's tests directory checked out from github!), and then you run the <tt>runlit.py</tt> script:

    .../ldc/build/tests>  python runlit.py -v .

With <b>-v</b> the output becomes more verbose such that you can see what exactly failed if a test fails.<br>
The "." means: all tests in this path. Important: the path is from the tests directory in the source directory! To run just one test, you do:

    .../ldc/build/tests>  python runlit.py -v codegen/align.d

== Lit ==

[http://llvm.org/docs/CommandGuide/lit.html Lit] is a tool created by the LLVM developers to run compiler tests, written in Python.<br>
Read about it from the horse's mouth: [http://blog.llvm.org/2009/12/lit-it.html LLVM Blog: 'lit' it].<br>
Lit is an important piece of [http://llvm.org/docs/TestingGuide.html LLVM's Testing Infrastructure].

LDC uses the Python package as available from https://pypi.python.org/pypi/lit , which can be best installed using Python's <tt>pip</tt>.

The rest of this page will describe important functionality of the testsuite. It is very instructive to look at LLVM/Clang/LDC tests to see how different kind of tests can be constructed!

== FileCheck ==

[http://llvm.org/docs/CommandGuide/FileCheck.html FileCheck] is an LLVM tool that reads a file passed as argument and checks whether the input received from <tt>stdin</tt> corresponds to what that file says it should look like. Confusing? An example:

    FileCheck test.d < test.ll

That command loads FileCheck with file <tt>test.d</tt>. That file contains directives for FileCheck (starting with <tt>// CHECK</tt>, telling it what to look for in <tt>stdin</tt>. The directive <tt>// CHECK: LDC is awesome</tt> will check that "LDC is awesome" appears in <tt>stdin</tt>. FileCheck will succeed when all checks are satisfied.

A typical test case does: 
* Run LDC with some commandline flags and a D input file, and have it output LLVM IR to a temporary file.
* Run FileCheck with the D input file as parameter, passing the temporary LLVM IR file through <tt>stdin</tt>

== A simple test case example ==

To add a test case to the testsuite, all you have to do is create a file inside a subdirectory of the <tt>tests</tt> directory (not the <tt>d2</tt> subdirectory!).

Let's say we created a file <tt>tests/codegen/attr_fastmath.d</tt>:
<source lang="d" line>
// RUN: %ldc -c -output-ll -of=%t.ll %s && FileCheck %s < %t.ll

import ldc.attributes;

@fastmath
double foo(double a, double b)
{
// CHECK: fmul fast
    auto c += a * b;
}
</source>

Special directives for Lit and FileCheck are given in commented out parts of a file (e.g. with "//" in C/C++/D source and with ";" in LLVM IR source).<br>
The example contains a Lit directive on line 1 and a FileCheck directive on line 8.

On line 1, "// RUN:" signals Lit that this file is a testcase (indeed!) and also instructs Lit what to do for this test. The string following "RUN:" is just the shell command line (with some substitutions and other special tricks).
* <tt>%s</tt> is substituted with the current file
* <tt>%t</tt> is substituted with a temporary file, so <tt>%t.ll</tt> becomes a temp file with ".ll" extension.
In this example, LDC will be run with the current file as input and flags <tt>-c -output-ll</tt> where the output is directed to a temporary file that is sent to FileCheck.

On line 8, "// CHECK:" instructs FileCheck to look for the string "fmul fast" <i>anywhere</i> in the input received from <tt>stdin</tt>. Remember that the LLVM IR generated by LDC is passed through <tt>stdin</tt> (via file<tt> %t.d</tt>), and so in this test we check that the generated LLVM IR contains the string "fmul fast".Often you will have to be more precise about where "fmul fast" should occur. What if there are more multiplications in the source file? What if the D source contains a line <code>string str = "fmul fast is great";</code>?!<br>
[http://llvm.org/docs/CommandGuide/FileCheck.html FileCheck's documentation] contains a tutorial displaying FileCheck's functionality for narrowing tests. Make sure you understand the directives
<tt>CHECK</tt>, <tt>CHECK-NOT</tt>, <tt>CHECK-LABEL</tt>, <tt>CHECK-SAME</tt>, <tt>CHECK-NEXT</tt>, <tt>CHECK-DAG</tt>.

FileCheck directives can use regular expressions, that are put in between double curly braces <nowiki>{{ ... }}</nowiki> or double rectangular braces <nowiki>[[VARNAME: ... ]]</nowiki> for capturing a matched string in VARNAME.

== Substitutions ==

Lit will substitute a few special strings inside "// RUN:" directives. Substitutions start with a <tt>%</tt>.

=== Lit pre-defined substitutions ===
; %%
: Replaced by a single %. This allows escaping other substitutions.

; %s
: File path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF/foo_test.s"

; %S
:Directory path to the test case’s source. Example: "/home/user/llvm/test/MC/ELF"<br> This is useful for accessing additional files (put in the <tt>input</tt> subfolder).

; %t
:File path to a temporary file name that could be used for this test case. The file name won’t conflict with other test cases. You can append to it if you need multiple temporaries. Example: "/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp"

; %T
: Directory of %t. Example: "/home/user/llvm.build/test/MC/ELF/Output"

(This is not an exhaustive list, see [http://llvm.org/docs/CommandGuide/lit.html#pre-defined-substitutions Lit's reference for all])

=== LDC-specific substitutions ===

; %ldc
: Replaced by the LDC binary built by CMake. You will probably always use this, because you don't want to test the LDC that is found on the system path! Example: "/Users/johan/ldc/pgo/build38release/bin/ldc2"

; %exe
: The file extension used for executables on the current OS. On Windows: ".exe". Empty string on other OSes.

; %obj
: The file extension used for object files on the current OS. On Windows: ".obj". Other OSes: ".o"

; %profdata
: Replaced by the ldc-profdata binary built by CMake. (similar to <tt>%ldc</tt>, only available when LDC's PGO is available)

To add more substitutions, look at <tt>tests/lit.site.cfg.in</tt>.

== "Features" ==

Some tests can/should only be run on certain setups. You can specify which "features" a test requires, for example:

    // REQUIRES: atleast_llvm307

which means that the test requires at least LLVM 3.7 to be run. The test will show up as "unsupported" when the requirements are not met.

=== LDC-specific features ===

; <tt>llvm<number></tt>
: <tt>llvm305</tt>, <tt>llvm306</tt>, <tt>llvm307</tt>, etc.
: The feature is set when LDC's LLVM version is equal to <number>. (LLVM 3.7 == <tt>llvm307</tt>)

; <tt>atleast_llvm<number></tt>
: <tt>atleast_llvm305</tt>, <tt>atleast_llvm306</tt>, <tt>atleast_llvm307</tt>, etc.
: The feature is set when LDC's LLVM version is equal or higher than <number>.

; <tt>Darwin</tt>, <tt>Linux</tt>, <tt>Windows</tt>, ...
: The string returned by Python's <tt>platform.system()</tt> is added as a feature.

; <tt>target_X86</tt>, <tt>target_ARM</tt>, <tt>target_PowerPC</tt>, <tt>target_AArch64</tt>, ...
: The feature is set when LDC's LLVM can target that architecture.

;<tt>Windows_x86</tt>
: The feature is set when LDC targets Windows 32bit x86 by default.

To add more substitutions, look at tests/lit.site.cfg.in.

== Fail tests ==

If the test command is supposed to fail, you can use the <tt>not</tt> tool:

<source lang="d">
// RUN: not %ldc -commandflagdoesnotexist
</source>

Use case: test compiler diagnostics!


If a test is known to fail on a certain platform, you can mark it as XFAIL:

<source lang="d">
// XFAIL: Windows
</source>

This means that the test is expected to fail on Windows (i.e. bug still to be fixed!).

== More complicated example ==
Consider the file <tt>tests/codegen/attr_target_x86.d</tt>:
<source lang="d" line>
// REQUIRES: atleast_llvm307
// REQUIRES: target_X86

// RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-ll -of=%t.ll %s && FileCheck %s --check-prefix LLVM < %t.ll
// RUN: %ldc -O -c -mcpu=i386 -mtriple i386-linux-gnu -output-s -of=%t.s %s && FileCheck %s  --check-prefix ASM < %t.s

import ldc.attributes;

// LLVM-LABEL: define{{.*}} void @{{.*}}foo
// ASM-LABEL: _D15attr_target_x863fooFPfPffZv:
void foo(float *A, float* B, float K) {
    for (int i = 0; i < 128; ++i)
        A[i] *= B[i] + K;
// ASM-NOT: addps
}

// LLVM-LABEL: define{{.*}} void @{{.*}}foo_sse
// LLVM-SAME: #[[SSE:[0-9]+]]
// ASM-LABEL: _D15attr_target_x867foo_sseFPfPffZv:
@(target("sse"))
void foo_sse(float *A, float* B, float K) {
    for (int i = 0; i < 128; ++i)
        A[i] *= B[i] + K;
// ASM: addps
}

// LLVM-DAG: attributes #[[SSE]] = {{.*}} "target-features"="+sse"
</source>

=== Multiple RUN directives ===

There can be multiple RUN directives! Important: they can (and will) be run in parallel! Their execution should not depend on each other in any way (make sure they use different intermediate files).

=== FileCheck prefix ===

When <tt>--check-prefix</tt> is specified for FileCheck, the specified string is recognized instead of the default "CHECK" string. Useful when running multiple tests with different compiler settings on the same file.

== My workflow when working on LDC (Johan) ==

I find it very convenient to work with the lit-based testsuite during development and bug fixing.<br>
I have two terminal windows open: one where I build LDC, and another where I run the test(s).
So in one terminal I (pretty much) only run the command:

    ...../ldc/build>  ninja

And in the other terminal I (pretty much) only run the command (assuming you are working on something related to the <tt>codegen/align.d</tt> test:

    ...../ldc/build/tests>  ./runlit.py -v codegen/align.d

(note the directories that I am in)

The workflow then is:
# Change a bit of LDC code, or add/change a test
# run <tt>ninja</tt> in terminal 1
# run <tt>runlit.py</tt> in terminal 2
# goto 1.    

----
[[Category:LDC]]