Starting as a Contributor
This page describes how to build D, put together a correct patch, and contribute it as a GitHub pull request.
Contents
Copyright assignment
Please note that all contributions to DMD frontend, DMD backend, Druntime, Phobos, or the official tools require that the copyright to that code be assigned to the D Language Foundation.
Existing tools
There exist tools which can do some of the below steps automatically:
- tools/update.sh is a simple script that either installs a new or updates an existing D development tree. Just download the script and run.
- Digger - can download and build D from any point in its recent history.
- DVM - can build and locally install D from source code.
Prerequisites
POSIX
To build D on POSIX, you will need to have make, g++, libcurl4-openssl-dev, and git installed on your system, as well as a working GitHub account. To install the appropriate dependencies on e.g. Ubuntu:
sudo apt-get install curl git make g++ libcurl4-openssl-dev
On OS X with Homebrew:
$ xcode-select --install $ brew install git openssl
To build the 32-bit phobos on a 64-bit machine, some 32-bit packages are also needed:
sudo apt-get install gcc-multilib libc6-dev-i386 libcurl4-gnutls-dev:i386
Other versions and variations of libcurl may work as well.
Windows
On Windows, you will need Git for Windows, the DigitalMars C++ compiler, and, for 64-bit or 32-bit COFF builds, the Microsoft Windows SDK.
Building D
Posix
Fetch dmd from GitHub
Let's start by getting the current development (master) branch of dmd from GitHub. Assume the root directory for everything D-related is ~/code (replace appropriately). This is easily done by running at a command prompt:
cd ~/code
git clone https://github.com/dlang/dmd
After this step completes successfully, the directory ~/code/dmd should be up and filled with good stuff.
Bootstrap dmd
This step is interesting because in order to build dmd, dmd is necessary. Fortunately, the steps of downloading and using a preexisting dmd compiler are automated. All you need to do is run this command:
cd ~/code/dmd
make -f posix.mak AUTO_BOOTSTRAP=1
That's going to take a while. To make it faster, passing -j8 accelerates things by running eight processes in parallel. The build produces the compiler binary ~/code/dmd/src/dmd.
To make dmd builds faster in the future, you need to obviate the need for bootstrapping. Install dmd from the download page or simply put the freshly built dmd binary in a place accessible through $PATH (a popular choice is ~/bin).
On Windows, you will need
Fetch and build druntime
druntime is the core runtime library for D, needed for building most every D program, including the standard library itself. So it's the next step in the progression (note that it requires a properly built dmd, so make sure the above steps have completed successfully). To fetch and build druntime, issue these commands:
cd ~/code
git clone https://github.com/dlang/druntime
cd druntime
make -f posix.mak
All that should go pretty fast. The somewhat anticlimactic result of the build is a library called libdruntime.a situated in an OS-dependent directory such as ~/code/druntime/generated/linux/release/64/. Make sure it's there. This library is made available to your applications through Phobos, so you must follow the next step to reflect any changes to druntime in your application.
Fetch and build phobos
Most D programs use D's standard library phobos. To get and build it, make sure you first fetch and build the latest dmd and druntime. Then:
cd ~/code
git clone https://github.com/dlang/phobos
cd phobos
make -f posix.mak
The build produces (with similar anticlimacticity) static and shared libraries such as ~/code/phobos/generated/linux/release/64/libphobos2.a and ~/code/phobos/generated/linux/release/64/libphobos2.so.
If you want to contribute to Phobos, continue with this guide
Windows
The following instructions work for win32. May or may not work with win64. This scheme is a suggestion. These instructions should work when building from a clean repository, however, this repository contains autogenerated code that may be left behind after switching branches so running a git clean after switching branches is a good idea:
git clean -xfd
Assuming your sources are checked out C:\D, and that make from digital mars is in your path, you can do the following to build them:
set DM_HOME=C:\D
cd %DM_HOME%\dmd2\src\dmd\src
make -fwin32.mak release
From there, it is suggested to move the built binaries into your %DM_HOME%\windows\bin directory, and add that to your path:
copy *.exe %DM_HOME%\dmd2\windows\bin
set path=%path%;%DM_HOME%\dmd2\windows\bin
From there, you have to create a sc.ini in your DMD.exe directory. It is suggested to just copy paste the one provided in the packaged 2.110.0, instead of writing your own.
Now build druntime:
cd %DM_HOME%\dmd2\src\druntime
make -fwin32.mak
And phobos:
cd %DM_HOME%\dmd2\src\phobos
make -fwin32.mak
You should copy the phobos lib into your windows\lib folder:
copy phobos.lib %DM_HOME%\dmd2\windows\lib
Optionally, you can build rdmd from source if you have checked out tools in your sources:
cd %DM_HOME%\dmd2\src\tools
make -fwin32.mak rdmd.exe
copy *.exe %DM_HOME%\dmd2\windows\bin
The last step is getting the additional libs. curl for D2 can be found at the bottom of the download section of dlang.org: [download].
Additional libs that are necessary can simply be copy pasted from the 2.110.0 package (without overwriting your phobos.lib)
The very last step is to verify that everything works by unittesting phobos:
cd %DM_HOME%\dmd2\src\phobos
make -fwin32.mak unittest
Common Windows issues
Missing MASM386
If when building druntime you get errors about missing MASM386, it's due to a required assembling of a file called minit.asm. However the druntime repository includes a prebuilt minit.obj file so you shouldn't need to assemble it again. As a workaround for the make error create an empty masm386.bat file and put it in a directory that's in your PATH.
It's also recommended that you use the cmd.exe terminal. Others, like PowerShell, are known to experience issues with legacy tools.
Intermediate files lead to several errors
The three main components (dmd, druntime, phobos) should always be built together with matching versions. The intermediate files generated by a previous build can lead to a failure so it's advisable to run
make -fwin32.mak clean
on each component before starting the process.
Unittest phobos
If you want to work on phobos itself, you need to run unittests—either for the full library, a package, or a module. To unittest the entire library:
make -j16 -f posix.mak unittest
Adjust the parameter passed to -j depending on your machine (beefier machines support larger parameters). This command unittests phobos in both debug and release mode. To only test one of them, add BUILD=debug or BUILD=release to the command line, for example:
make -j16 -f posix.mak BUILD=debug unittest
Specifying BUILD makes unittesting faster so it is recommended in iterative development. Just make sure both debug and release builds are tested before e.g. submitting a pull request.
While changing one specific package or module, it's useful to be able to only unittest that particular entity. The following two commands only unittest (in debug mode) the std.algorithm package and the std.conv module, respectively:
make -j16 -f posix.mak BUILD=debug std/algorithm.test
make -j16 -f posix.mak BUILD=debug std/conv.test
Several modules, packages, or mix thereof may be specified for testing in the same command line. For example, this command also tests (and also in debug mode) the std.algorithm package and the std.conv module, with better parallelism:
make -j16 -f posix.mak BUILD=debug std/algorithm.test std/conv.test
The auto-tester will fail your PR if your changes contain trailing whitespace or incorrect line endings. You can run this test locally with the command:
make -j16 -f posix.mak checkwhitespace
Running Independent Programs
The rig created so far allows making changes and testing dmd, druntime, and phobos by using their respective makefiles. However, running independent programs (such as by using ~/code/dmd/src/dmd ~/mytest) will not know where to pick up the libraries from, even though the compiler path is correctly specified. As a consequence such programs will not compile, or (worse) will pick up libraries of a preexisting dmd installation on the target system and will fail in mysterious ways.
The remedy is to specify include and library paths properly. To build mytest.d using ~/code/dmd, ~/code/dmd, and ~/code/dmd, use this command line:
~/code/dmd/src/dmd -I~/code/druntime/import -I~/code/phobos -L-L$HOME/code/phobos/generated/*/release/64 mytest.d
The two -I uses bring the import files for druntime and phobos into visibility. The -L directive forwards the rest of the argument to the linker, i.e. the linker will get $HOME/code/phobos/generated/*/release/64/ with $HOME expanded properly. Sadly we cannot use the tilde (~) for -L because it won't get expanded properly. (If you need a 32-bit build, replace /64 with /32 and of course use the -m32 flag for the compiler.)
Of course if you need to use this line often, you may want to put it into a batch file or a dmd.conf file (see more info about dmd.conf on Linux, Windows, OSX, and FreeBSD).
(What's with the * in the path provided to the linker? If you browse to /code/phobos/generated/, you'll see a directory with an OS name, such as linux or osx. Virtually always that directory is only for one operating system, so using * will simply navigate appropriately.)
Fetch and build dlang.org
This step is optional. Significant changes to phobos' documentation require that the site (which includes automatically generated phobos documentation) builds successfully. The following will build all documentation in all forms (see the note below about building just the html documentation):
cd ~/code
git clone https://github.com/dlang/dlang.org
cd dlang.org
make -f posix.mak
All of dmd, druntime, and phobos are needed for the site to build. Note that one of the first lines output during the make run looks like this:
LATEST=2.110.0 <-- place in the command line to skip network traffic.
That's advice worth heeding because fetching LATEST automatically involves network traffic, which adds time to the build. So for future builds use this:
make -f posix.mak LATEST=2.110.0
Of course, parallelizing with -j improves speed as well.
The build produces the entire site at ~/code/dlang.org/web. To informally test it, open the appropriate HTML documents in that directory. Note that the currently released phobos has documentation in ~/code/dlang.org/web/phobos, whereas the current (fresh) build of phobos has documentation in ~/code/dlang.org/web/phobos-prerelease. So, for example, if you change the embedded documentation in ~/code/phobos/std/conv.d, the changes are visible in ~/code/dlang.org/web/phobos-prerelease/std_conv.html. (The build process replaces the slashes in submodules with underscores.)
Note that the above steps will build all of the documentation in all forms, including Kindle builds and various other things that may require installing additional tools, and may download/build old versions of DMD. To prevent that, you can add the html option:
make -f posix.mak LATEST=2.110.0 html
Alternatively to building the documentation locally, you can use the documentation autotester service, which will build documentation, generate a diff of the results, and add a link to your GitHub pull request.
Ancillary stuff
dconf.org, dub, dub-registry, installer, tools, and visuald
These ancillary repositories are of somewhat specific interest. Their installation mimics that of the repositories described above. If you get to the point where you need to work on one of these, chances are you're already versed in what needs doing. If not, ask away.
Additional Tools
If you cloned dlang/tools.git, you also have a tools folder where small helping programs live. There is no need to build them, you can just compile them using DMD:
dmd rdmd.d;
dmd ddemangle.d;
dmd dtab;
dmd tolf;
rdmd builds your D modules automatically, from the one containing main. It'll deduce dependencies and compile/link them for you. ddemangle will demangle its input, replacing all mangled D symbols with their unmangled form. dtab transforms tabs into spaces in source code. tolf replaces line endings with LF.
Using dtab and tolf is a good idea if you want to contribute to the dlang repos.
Typical Contributor Workflow
There are many ways to use git and GitHub to contribute. Here's a typical one.
First, fork the github repository or repositories you'd like to contribute to (dmd, druntime, phobos etc) by navigating to their respective pages on github.com and clicking "Fork". Then, set up your local git repository to reflect that. For example, consider you want to contribute to phobos and have forked it. Then run these commands:
cd ~/code/phobos
git remote add myfork https://github.com/username/phobos.git
git remote update
(Replace username with your actual github user name.) This adds the "myfork" repository and makes sure everything is synchronized between your local copy and the remote repositories. Then, it's best to work in branches as shown below:
git checkout -b awesome-new-feature
# ... get some good work done here ...
git commit -am "Awesome new feature ..."
git push -f myfork
With this, your work is in your github fork of the phobos (or whichever) repository. After that, visit your fork on github.com, which looks like https://github.com/username/phobos/tree/awesome-new-feature.
Create a pull request
Once you have tested all your changes and pushed them to your fork on GitHub, you are ready to submit a pull request.
- Navigate to your fork of the project on GitHub.
- Important: Select the branch that you made your changes in, say issue_1234.
- Click on the "Pull Request" button.
This will submit your changes for review by the D maintainers. If your changes are approved, they will be merged into the master branch. Otherwise, if the maintainers have some comments or feedback, you can refine your changes by editing and testing in your local workspace, and pushing the new changes to the same git branch. The new changes will be automatically included in your pull request.
Choose a title for your pull request that clearly states what it does. When fixing a bug, the usual thing to do is to use the summary from the bugzilla report. Eg a title like "Fix 3797" or "Issue 3797" contains much less information than "Fix Issue 3797 - Regression(2.038): Implicit conversion between incompatible function pointers" and requires a lot more effort for the reviewers to determine if it is something they are interested in.
Pull request descriptions should contain a hyperlink to the Bugzilla issue that is being fixed. This is usually added at the end of the description.
After the pull request is created, add the 'pull' keyword to the corresponding Bugzilla issue and a link to the pull request posted in a comment.
Characteristics of a Good Pull Request
- Addresses one topic only.
- Refactorings should not be mixed in with bug fixes or enhancements.
- Refactorings should be marked as such in the subject, and must not contain any behavior changes.
- Larger refactorings need to broken up into smaller PRs.
The problems with large PRs are:
- They are hard to review.
- github is very slow at serving pages with large diffs on them.
- If the PR introduces a regression, a large diff is much harder to debug than a small one.
- They imply an all or nothing view of it, when it actually may have good parts and bad parts.
Autotester
Pull requests are automatically picked up by the autotester, which compiles the code in the pull request and runs it through the dmd, druntime, and phobos unittests on all supported platforms. Generally, pull requests must pass all tests before they will be merged. The status of the tests can be monitored through the pull request page.
Every user must be manually approved before the autotester will start testing their pull requests. Users can be approved by anyone with commit access.
Rebasing
Sometimes, if a particular change you are working on is taking a long time, or if you encounter a problem that is fixed by a new commit upstream, you may need to sync your local branch with master in order to keep the code up-to-date. In this case, it is recommended that you use git rebase to apply your changes on top of the latest git master, so that when you submit a pull request, the change history will be easier for the reviewers to follow. Using git merge is not recommended, as it may produce a lot of merge commits that may not be relevant to your changes.
For example, you may be working on your changes:
cd /usr/src/d/phobos
git checkout mybranch
vim std/algorithm.d # apply lots of cool changes here
First, before you resync with master, make sure all your changes are checked in (or stashed):
git commit -a
If you forked from the official D programming language repositories you may need to add an upstream remote to pull in the latest official changes. If this is the case you can add an upstream remote like this:
git remote add upstream git@github.com:dlang/phobos
This adds another remote to your repository called upstream and only needs to be done once. Once the upstream remote is added, you can update your repository's master branch by running the following:
git checkout master
git pull --ff-only upstream master
The --ff-only option is to ensure that your master branch is identical to the official D sources' master branch, since otherwise you will end up with a very messy history that will be hard to clean up (and the reviewers will probably reject your pull request due to having unrelated merge commits).
Now go back to your branch and rebase it:
git checkout mybranch
git rebase master
Now your sources should be up-to-date. Recompile and test everything to make sure it all works.
Note that after rebasing, you will need to force an update to your fork on GitHub with the -f flag, otherwise it will fail because the histories don't match anymore:
git push -f origin mybranch
You may wish to read up on how git rebase works if you're not familiar with the concept.
If, during the 'git rebase' command, you encounter conflicts, you may want to learn how to resolve a conflict during git rebase.
Squashing
After receiving feedback on your PR, it's common for it to have lots of commits that don't add much by being separate. For example, consider the following git history on a PR:
commit [ffffff] Added new function: foobar commit [aaaaaa] Spelling error fix in foobar docs commit [cccccc] Clarified Docs for foobar
Nothing is gained from having these as three separate commits as they are all focused on one feature. Instead, they should be one commit so the history looks like this
commit [333333] Added new function: foobar
while still retaining all of your changes. In order to perform this, please consult this guide
You can also directly append to your last commit and force an update of your PR:
git commit --amend
git push -f
Stable Branch
If you are working on a fix for a regression, chances are it should go into the next point release, and not the next major version (e.g. 2.067.1 instead of 2.068). In this case, you should check out the stable branch of each subproject BEFORE you create your topic branch:
cd dmd
git checkout stable
cd ../druntime
git checkout stable
cd ../phobos
git checkout stable
Then follow the instructions for making a branch.
If you forget to do this, or didn't realize it, It's not possible to simply re-target your branch for pulling into the stable branch. GitHub will let you do this, but your branch will include many of the changes from the unstable branch!
In order to fix such a problem, you can rebase your changes from master on top of the stable branch. First you need to pull in the stable branch from your fork on github:
git checkout stable
Then, you go back to your branch, and replay the changes from master using rebase:
git checkout mybranch
git fetch upstream
git rebase --onto upstream/stable upstream/master mybranch
You may have to follow the instructions in the Rebasing section on adding the upstream branch, substituting stable for master, if you need to update to the latest stable changes.
This sometimes may not work, as the changes between the stable and master are too drastic. In this case, you may have to re create your changes after a clean checkout of the stable branch.
When creating a pull request, you need to tell github to target the stable branch instead of master on the upstream repository. This is done via a drop-down at the top of the page, make sure to do this before submitting your pull request as this cannot be changed after the PR is created (you will have to close the PR and create a new one).
If you notice in your PR a whole slew of changes that seem to have nothing to do with your changes, it's likely because you forgot one of these steps.
Reviews
Any pull requests that make language changes must be approved by Walter and Andrei. This includes druntime changes that implement the specification.
Any pull requests that make significant changes to code should be reviewed by more than one person. This means that at least two people need to approve the pull request before it is merged. One person must be a person with commit rights, but the other need not be, as long as that person is trusted within the developer community.
Pull requests that are trivial (typos, obvious minor bug fixes, etc.) may be pulled without a second review.
Please note that any updates pushed to the candidate branch do not automatically notify a subscribed person. If you update your branch to correct an issue, please also put in a comment indicating it.
Contributing FAQ
A file that I made a change on was modified by a different merged PR, and now my PR can't be merged, now what?
What you need to do is rebase your git branch on the master branch. What this does is rewrite the history of your git branch to make it seem like it was merged off of the head of master rather than the older commit where you actually branched. This will include the new commits in your PR so your PR no longer conflicts. See this tutorial for more details.
