View source for Contributing to dlang.org

This document is aimed at people who want to contribute to the [http://dlang.org dlang.org] website. It includes a brief tutorial in the technologies used and step-by-step instructions for typical tasks.

== Installing ==

We assume you already have a dmd rig up and running. If not, follow the steps at [[Starting as a Contributor]] to get it running. The directory structure we'll assume in the following is:

 ~/
   d/
     dmd/
     druntime/
     phobos/

With this setup, let's proceed to downloading the source of dlang.org as follows:

<syntaxhighlight lang=bash>
cd ~/d
git clone https://github.com/D-Programming-Language/dlang.org
</syntaxhighlight>

After this, the <tt>dlang.org</tt> directory will end up parallel to <tt>dmd</tt>, <tt>druntime</tt>, and <tt>phobos</tt>. Let's build the site (for now without the standard library documentation) by using the following command:

<syntaxhighlight lang=bash>
cd ~/d/dlang.org
make -j32 -f posix.mak html
</syntaxhighlight>

The <tt>html</tt> target instructs <tt>make</tt> to only build the site. By default (if you specify no target), <tt>make</tt> also builds the core runtime and standard library documentation, both for the latest release and for the current code residing on your machine. That may get pretty involved, so let's leave it for later. For now, let's inspect the result of the build, all of which goes in the <tt>dlang.org/web</tt> directory. On Linux, for example, the command <tt>sensible-browser</tt> opens your default browser with a given file or address so we can use it as such:

<syntaxhighlight lang=bash>
sensible-browser ~/d/dlang.org/web/index.html
</syntaxhighlight>

At this point if all went well a nicely-formatted HTML file pops up featuring a local replica of the dlang.org homepage. Congratulations!

== Editing Content ==

Browsing through <tt>~/d/dlang.org/</tt> reveals that most files have the <tt>.dd</tt> extension. Those files are in Ddoc format; in order to work on dlang.org a basic understanding of the Ddoc format is needed.

At its core, Ddoc is a pure macro expansion system. In this context "pure" means the macro language has no relationship to other file format; all the expansion engine does is take in macro definitions and then munch through text and expand macros as they come along. A few macros are predefined to values that make HTML generation easy, so in that sense a slight affinity with HTML does exist; however, those macros can be trivially redefined to any other purpose. Also, Ddoc recognizes sections marked in a particular way as D source code. There are a few subtleties inherent to macro processors (e.g. how recursion works or in what order nested macros are expanded), but aside from those Ddoc is deceptively simple and very flexible. Exploiting the favorable relationship between Ddoc's simplicity and power is key to using it effectively.

Ddoc source files have the following structure:

<syntaxhighlight lang=text>
Ddoc
Text with embedded macros such as $(MACRO1) and $(MACRO2) goes here.
Macros:
  MACRO1=definition1
  MACRO2=definition2
</syntaxhighlight>

That is, a Ddoc file consists of the actual word "Ddoc" followed by a newline, then followed by the actual text of the document, followed by a line containing "Macros:", followed by macro definitions of the form <tt>NAME=value</tt>. The "Macros:" section is optional (as is the indentation of the macro definitions; it's present here for aesthetic reasons only). You might have guessed already that the syntax <tt>$(MACRONAME)</tt> expands the macro called <tt>MACRONAME</tt> into whatever text was ascribed to it in the "Macros:" section. Let's actually test that by saving the following text into a file called e.g. <tt>test.dd</tt>:

<syntaxhighlight lang=text>
Ddoc
Text with embedded macros such as $(MACRO1) and $(MACRO2) goes here.
Macros:
  MACRO1=definition1
  MACRO2=definition2
  DDOC_COMMENT=
  DDOC=$(BODY)
</syntaxhighlight>

The last two macro definitions seem to come out of nowhere and deserve some explanation, which this document will provide soon. For now let's  "build" this file like this:

<syntaxhighlight lang=bash>
~/d/dmd/src/dmd test.dd
cat test.html
</syntaxhighlight>

(You may of course just type <tt>dmd</tt> if it's in your <tt>$PATH</tt>) The produced file, by default carrying the <tt>.html</tt> extension, contains:

<syntaxhighlight lang=text>
Text with embedded macros such as definition1 and definition2 goes here.
</syntaxhighlight>

So the macro names got expanded to the text in their respective definitions. Sweet!

=== The Ddoc Expansion Process ===

Time to clear the air about the mysterious <tt>DDOC_COMMENT=</tt> and <tt>DDOC=$(BODY)</tt> definitions. When <tt>dmd</tt> processes a <tt>.ddoc</tt> file, it doesn't immediately expand the text; the process goes as follows:

* accumulate the text (sans the opening <tt>Ddoc\n</tt>) in memory and put it in a variable called <tt>BODY</tt>;
* when processing reaches the <tt>\nMacros:\n</tt> section, read and memorize the macro definitions underneath;
* expand and output <tt>$(DDOC_COMMENT Generated by Ddoc from filename.dd)</tt>;
* expand and output the macro <tt>DDOC</tt>.

The default values of <tt>DDOC_COMMENT</tt> and <tt>DDOC</tt> are geared toward building simple HTML files. Indeed, if we remove the macro definitions from the <tt>test.dd</tt> file and rebuild, the resulting <tt>test.html</tt> contains:

<syntaxhighlight lang=text>
<html><head>
        <META http-equiv="content-type" content="text/html; charset=utf-8">
        <title>test</title>
        </head><body>
        <h1>test</h1>
        <!-- Generated by Ddoc from test.dd -->

Text with embedded macros such as definition1 and definition2 goes here.

        <hr><small>Page generated by <a href="http://dlang.org/ddoc.html">Ddoc</a>. </small>
        </body></html>
</syntaxhighlight>

which is a serviceable albeit bland HTML document.

=== Macros with Parameters ===

Macros are key to the power of Ddoc for at least two reasons. First, they shorten and simplify the document by allowing you to replace clumsy typesetting directives such as <tt>&lt;span class="important">Attention!&lt;/span></tt> with <tt>$(ATTENTION)</tt>. Second, they elevate the level of the document you're writing by leveraging the traditional "extra level of indirection". Depending on how you define <tt>ATTENTION</tt>, you get to format <tt>$(ATTENTION)</tt> in various formats. For example, for HTML you'd use:

<syntaxhighlight lang=text>
ATTENTION=<span class="important">Attention!</span>
</syntaxhighlight>

If you don't care to define css classes etc. you may want to just go sloppy:

<syntaxhighlight lang=text>
ATTENTION=<font color="red">Attention!</font>
</syntaxhighlight>

To format the document as plain text, just write:

<syntaxhighlight lang=text>
ATTENTION=Attention!
</syntaxhighlight>

And if you want to output LaTeX, write something like:

<syntaxhighlight lang=text>
ATTENTION={\color{red}Attention!}
</syntaxhighlight>

In short, you get to expand Ddoc documents into many other formats by using macros and combining the documents with appropriately-defined macro batteries. (This document will discuss soon how to effect such combinations.)

For now, let's note that <tt>ATTENTION</tt> is not quite a sterling example of flexibility; it allows us to render "Attention!" in a special way, but often the need is to render various other words and phrases as attention-attracting text. So what's needed is a macro taking the text to render as a parameter. Here's how to do so in Ddoc.

In a macro definition, certain constructs access arguments as follows:

* <tt>$1</tt>, <tt>$2</tt>, <tt>$3</tt>, ..., <tt>$9</tt>  expand to the first, second, third, ..., ninth argument;
* <tt>$+</tt> expands to all arguments except for the first;
* <tt>$0</tt> simply expands to all macro arguments.

To pass arguments to a macro, insert a space after the macro name, then pass the arguments separated them by commas, then close the paren: for example, <tt>$(MYMACRO how are you doing, Jeff?)</tt> passes the arguments "how are you doing" and "Jeff?" to a macro called <tt>MYMACRO</tt>.

There are a couple of subtleties related to passing and expanding parameters. The first whitespace after the macro name is "munched", i.e. it just disappears from the expansion. But if there's any whitespace character following the first one, it will be considered part of the first argument. An example will make this clear. Consider the macro definition <tt>TEST=|$1|</tt>. Then, the invocation

<syntaxhighlight lang=text>$(TEST abc)</syntaxhighlight>

produces

<syntaxhighlight lang=text>|abc|</syntaxhighlight>

whereas the invocation  (note the extra space)

<syntaxhighlight lang=text>$(TEST  abc)</syntaxhighlight>

produces (note the extra space before the letters):

<syntaxhighlight lang=text>| abc|</syntaxhighlight>

The whitespace after comma (if present) in multiple argument lists is handled with a similar logic. The first whitespace character immediately after the comma is considered "aesthetic" and not present in the expanded text. However, if the source inserts more than one whitespace character after comma, the extra ones are considered part of the argument. By means of example, consider the macro definition <tt>TEST=|$1|$2|</tt>. Then we have the following expansions:

<syntaxhighlight lang=text>$(TEST abc,xyz)</syntaxhighlight>

produces

<syntaxhighlight lang=text>|abc|xyz|</syntaxhighlight>

then the invocation (note the extra space)

<syntaxhighlight lang=text>$(TEST abc, xyz)</syntaxhighlight>

produces no change in output:

<syntaxhighlight lang=text>|abc|xyz|</syntaxhighlight>

However, the expansion (there are THREE spaces after the comma):

<syntaxhighlight lang=text>$(TEST abc,   xyz)</syntaxhighlight>

produces the output:

<syntaxhighlight lang=text>|abc| xyz|</syntaxhighlight>

which inserts TWO spaces before "xyz". One has been munched, the other two copied.

Ddoc "understands" parentheses and likes them paired. If a comma occurs within parenthesized text, it won't be considered a macro argument separator. For example, in the invocation <tt>$(MYMACRO abc, (xyz, tuv))</tt>, <tt>MYMACRO</tt> receives two arguments, not three. Nested parentheses work as expected, too; only top-level commas within the macro are considered argument separators.

=== Whitespace in macro definition ===

In the "Macros:" section, any number of whitespace characters preceding or following the <tt>=</tt> sign are ignored. That makes it problematic to define macros that start with a space. To remedy that, define this macro:

<syntaxhighlight lang=text>
SPACE = $(SPACE) $(SPACE)
</syntaxhighlight>

The recursive expansion (which will be explained in detail below) expands to nothing while the macro itself is being expanded; so the final expansion will be a space flanked by two empty strings, which is exactly what we needed. Now it's easy to use space to e.g. indent some text:

<syntaxhighlight lang=text>
INDENT = $(SPACE)$(SPACE)$(SPACE)$(SPACE)$0
</syntaxhighlight>

The <tt>INDENT</tt> macro inserts four spaces before its arguments. Note that the space between "<tt>=</tt>" and the first "<tt>$(SPACE)</tt>" is not part of the output.

Similarly, to define a macro that inserts exactly one newline:

<syntaxhighlight lang=text>
NEWLINE = $(NEWLINE)
$(NEWLINE)
</syntaxhighlight>

=== Special characters: "<tt>(</tt>", "<tt>)</tt>", "<tt>$</tt>", "<tt>,</tt>" ===

As shown above, the characters "<tt>(</tt>", "<tt>)</tt>", "<tt>$</tt>", and "<tt>,</tt>" have special meaning to Ddoc. Sometimes it's necessary to "escape" them, i.e. make them part of the produced output without them being interpreted by Ddoc. To achieve that, the following macros are useful:

<syntaxhighlight lang=text>
ARGS = $0
COMMA = ,
COMMENT = 
DOLLAR = $
LPAREN = (
RPAREN = )
TAIL = $+
</syntaxhighlight>

The <tt>ARGS</tt> macro is useful when we want to pass a large text (possibly including comma) as a single argument to another macro. Consider:

<syntaxhighlight lang=text>
SECTION=<h2>$1</h2>$+
</syntaxhighlight>

Then if the first argument to <tt>SECTION</tt> includes one ore more commas, use it like this:

<syntaxhighlight lang=text>
$(SECTION $(ARGS Trinkets, Treasures, and Other Tchotchkes),
Here goes the text of the section...
)
</syntaxhighlight>

Alternatively (and this brings us to the second useful macro), you could use <tt>COMMA</tt> for the same effect:

<syntaxhighlight lang=text>
$(SECTION Trinkets$(COMMA) Treasures$(COMMA) and Other Tchotchkes,
Here goes the text of the section...
)
</syntaxhighlight>

The macro <tt>COMMENT</tt> expands to absolutely nothing whatsoever, which makes it a great macro for inserting comments in Ddoc files (including commenting out portions of the document). Pay attention, however: comments must still pair parentheses properly.

The <tt>DOLLAR</tt> macro expands to the dollar sign in a way that makes it impossible for later use for initiating a macro expansion. Same goes about <tt>LPAREN</tt> and <tt>RPAREN</tt>: they expand to parentheses, but only in a blind textual way; they don't carry the usual meaning of parentheses. So you get to use these macros if you want e.g. to define a macro that includes the dollar sign and/or includes unbalanced parentheses.

Last but (somewhat ironically) not least, the <tt>TAIL</tt> macro expands to all of its arguments except the last one. Why is not <tt>$+</tt> just enough? Because you may apply <tt>TAIL</tt> to <tt>$+</tt> itself to access all arguments except the first two ones. Consider:

<syntaxhighlight lang=text>
SAYS=$1 $2 says: ==$(TAIL $+)==
</syntaxhighlight>

Then, <tt>$(SAYS Mad, Hatter, A land full of wonder, mystery, and danger!)</tt> expands to "Mad Hatter says: ==A land full of wonder, mystery, and danger!==" as we'd need it to.

Unfortunately, <tt>$(TAIL $(TAIL $+))</tt> does not expand to all arguments except the first three ones, as we'd wish. Instead, it expands to nothing at all. This is to keep the overall expansion mechanics simple: once the first macro expansion occurs, the commas found it it are "painted" to be inert so they can't be reused as commas to separate arguments to the outer macro.

All is not lost, however, with a little handwritten drudgery. These macros do work properly:

<syntaxhighlight lang=text>
TAIL=$+
TAIL2=$(TAIL $+)
TAIL3=$(TAIL2 $+)
TAIL4=$(TAIL3 $+)
</syntaxhighlight>

You may continue this for as long as needed. A natural limit is 9 because numbered arguments go up to <tt>$9</tt>.

=== Recursive Macros ===

Ddoc strives to avoid being a Turing-complete language, thus keeping the complexity/power balance in a safe area. That means recursion is theoretically off the table. However, there are two crude mechanisms that make recursive macros usable in Ddoc:

* If a macro is expanded within its own expansion more than 1000 levels, ddoc compilation fails with an error message.
* If a macro is expanded without any argument within its own definition, it "disappears" i.e. it expands to the null string.

The first rule is a hamfisted means to enforce that any ddoc expansion will finish and is rather uninteresting. The only thing to note is the limit on expansion depth, which is generous enough to not make it a nuisance. The second rule is the interesting one because it allows us to write recursive macros as long as the recursive invocation is always smaller than the input.

Consider, for example, generating an HTML list which needs to look like this:

<syntaxhighlight lang=text>
<ul>
<li>Apples</li>
<li>Oranges</li>
<li>Bananas</li>
<li>Pineapples</li>
</ul>
</syntaxhighlight>

One direct but awkward approach is:

<syntaxhighlight lang=text>
UL = <ul>$0</ul>
LI = <li>$0</li>
</syntaxhighlight>

which is then used like this:

<syntaxhighlight lang=text>
$(UL
$(LI Apples)
$(LI Oranges)
$(LI Bananas)
$(LI Pineapples)
)
</syntaxhighlight>

This works, but there's a more compact way of creating the list by using recursive macros:

<syntaxhighlight lang=text>
UL = <ul>$(LIs $0)</ul>
LIs = <li>$1</li>$(LIs $+)
</syntaxhighlight>

Now all you need to type to create the list is <tt>$(UL Apples, Oranges, Bananas, Pineapples)</tt>. Sweet! (Literally.) Got an argument with commas inside? No problem, <tt>ARGS</tt> is ready to help: <tt>$(UL Apples, Oranges, $(ARGS Mandarins, Tangerines, Clementines, etc.), Bananas, Pineapples)</tt> expands as desired.

=== Other Special Characters and Patterns ===

There are two more patterns you should know about. Snippets of D code should go in between lines consisting of three or more equal signs, as follows:

<syntaxhighlight lang=text>
===
void main() {
    import std.stdio;
    writeln("Hello, world");
}
===
</syntaxhighlight>

Then the Ddoc processor understands and syntax-colorizes the code appropriately. The coloring is done by means of predefined macros, all of which may be redefined by the user. Only D is currently supported; no other languages are "understood" by the Ddoc engine.

Another pattern worth noting is words or short phrases enclosed in backticks (those small back apostrophes <tt>`</tt> that look like specs of dust on the screen to those in need of glasses). Such <tt>`text`</tt> enclosed in backticks, when both the opening and the closing backtick occur on the same line, is rewritten as <tt>$(DDOC_BACKQUOTED text)</tt>. By default the macro expands to an HTML span tag, but you may change it as needed.

=== Macro Definition Batteries: <tt>.ddoc</tt> files ===

TODO

=== Predefined Macros ===

TODO