Development and Release Process
If you're an experienced git user, skip to the summary.
Contents
Definitions
Version numbers
We follow the traditional dmd versioning: 2.061. Tags have a v prepended, so the tag for version 2.061.1 is v2.061.1. Versions where the third part is bigger than zero are called Minor Releases. Versions where the third part is equal zero are Major Releases. Minor Releases will only contain regression fixes, Major Releases will contain bug fixes and new features.
The third digit can be omitted for Major Releases: 2.061 is the same as 2.061.0.
Meta: The dmd contributors should choose one style for all offical statements: Tags, packages, Notes on the download page. Either always write out the .0 part or always omit it.
Regression fix vs. Bug fix
A regression fix for release X is a fix for a bug which was newly introduced in release X. We distinguish between regression fixes and bug fixes to avoid introducing more regressions in minor releases. A regression fix especially should not make code invalid which was valid before the regression occurred. As an example: If we have releases 2.061 and 2.061.1 then 2.061.1 should always be "better" than 2.061. It should be tried heavily not to introduce new regressions in minor releases. Code which compiled with 2.061 should still compile in 2.061.1 (there's one rare exception: If a regression allowed code to work in 2.061 which did not work in 2.060 and which is not supposed to work, it can be disabled again in 2.061.1).
So a regression fix is a bug fix which should not have side-effects, doesn't introduce new features and fixes a bug which only recently appeared.
Long standing bugs are especially not regressions! (Or technically probably a regression in a very old release, but as we support only 1 major release we can treat those like bugs. We only care about new regressions)
Small enhancement vs. Big feature
We later suggest different workflows for those two enhancements. Those are easily distinguishable: If you implement the feature by yourself and don't need help / someone to participate / extended review and the code changes are reasonably small, it's a small enhancement. Otherwise it's a big feature.
The workflow for big features allows other contributors to participate in that feature, make special test releases for that feature, etc. If you want / need those, use the Big Feature workflow.
Stable
When is a release stable and therefore ready to be released?
The workflow already guarantees that when a release can be made there has been a feature freeze for a long time. So only bug fixes and regression fixes where allowed for some time which should already stabilize the release. But we should add some more criteria:
- All bugs marked as critical in bugzilla have been fixed
- For practical reasons we allow exceptions from this rule for the first releases after this workflow has been established
- There are no newly introduced (known) regressions
- This is a hard rule, there are no exceptions to it. Have a regression? Don't release!
- Later a yet to be determined percentage of non-critical bug will need to be fixed (or new features introduced)
- This is to make sure a new release actually provides something new and prevent us from releasing too often. Currently this is not an issue.
Apart from these criteria, the final decision is made by the release manager.
Don'ts
- do not cherry pick from one official branch into another
- do not rebase official branches
- do not rewrite git history on official branches
- do not merge master into a version branch
- do not merge staging into a version branch (The only time when commits go from staging to a version branch is when the version branch is being created)
- do not merge feature branches into staging or version branch. Feature branches are always merged into master.
- do not merge staging into a feature branch. Only merge staging into master, the master into the feature branch
Release Schedule
<x> = time in weeks
vxxx = Tag for version xxx (Marks the release)
rest = branches
\ = merge (merges from version to staging/master not shown)
------------------------------------------------------------------------------------ master branch. development is never stopped
\ <8> \
------------------------------------------------------------------------------------ staging branch
| | | |
v2.062.0-b1 | 2.063.0-b1 |
| |
| <8> ---------------- 2.063 release branch
| | <2> |
| v2.063.0 v2.063.1
|
-------------------------------------------------------- 2.062 release branch
| <2> | <2> |
v2.062.0 v2.062.1 v2.062.2
A new release is made approximately every 2 months. Releases are stabilized in the staging branch and beta releases are tagged as part of staging. When a release is ready a new version branch is created from staging. The first commit of this version branch is tagged as the release (v2.0xx). Immediately after creating the version branch master is merged into staging. Note: master is only ever merged into staging immediately after a major release. Then the staging branch is stabilizing again until the next release is made.
Once the initial release of the version branch is done, only regression fixes will be pushed to the version branch, then a regression fix update is published every 2 weeks from that version branch as long as it's supported (see #Preparing a new minor release).
Feature Previews: A feature branch may need it's own beta release before being merged into master. These are unplanned, announced on the NG, and highly volatile.
As an example we would see these releases for things like 64bit support and User Defined Attributes.
Support times
Currently only the most recent major release is supported. (This might change in the future)
As an example: version 2.061 is released. 2.061 will get updates until 2.062 is released. Then 2.062 will get updates until 2.063 is released etc.
Only supported releases receive minor update releases.
Branching model
Official branches
The following branches will be available on the official (upstream) dmd, druntime and phobos repositories:
- feature branches: New features, in general, are not developed directly on the master branch, but in a dedicated feature branch. An example for such a branch would be an UDA branch for user defined attributes. Contributors can collaborate on the feature by opening pull request targeting that feature branch. When the feature's implementation is more-or-less complete, the feature branch will be merged into the master branch. The master branch is also sporadically merged into the feature branches to get regression fixes, bug fixes and to make sure the feature branch can be merged into master later on without problems.
--------------------------------------------------------\ feature64BitWindows
/ / / \
/ / / \
/ / / \
------------------------------------------------------------------------------------ master branch. development is never stopped
/
/
------------------------------------------------------------------------------------ staging branch
/
fix123
- master: The feature branches are merged into this branch if they are considered complete. So the master branch will contain new features and may have breaking changes. Regression fixes and bug fixes will be merged from staging into master.
--------------------------------------------------------\ feature64BitWindows
\
-------------------\ featureUDA \
\ \
------------------------------------------------------------------------------------ master branch. development is never stopped
\ / <8> \
\ / \
------------------------------------------------------------------------------------ staging branch
/
fix123
- staging: Bug fixes will be pushed directly into staging if they are not regression fixes. To determine where a regression fix should go, see #Regression fix. staging usually only gets bugfix updates, but after every major release, new features are merged from master. Regression fixes will be merged from version branches into staging.
------------------------------------------------------------------------------------ master branch. development is never stopped
\ <8> \
------------------------------------------------------------------------------------ staging branch
/ / / / /
fix123 fix234 fix987 / fix999
/
------------------------------------------------------ 2.0.62 version branch
/
regression1
- version branches: Once we determine it's time to do a new release, a version branch is made by branching from the staging branch. A version branch will not receive any new features, only regression fixes. To determine to which version branch regression fixes should go, see #Regression fix. Version branches are used to support the version for an extended period of time to enable updated minor releases which fix regressions.
Suggested local branches
How user repositories are managed is up to the end user, but we propose this branches:
- feature branches: Tracking upstream feature branches. You only need a feature branch if you want to work on that feature. To work on a feature, make changes to the feature branch, push to your repository and file a pull request on github targeting the feature branch
- master: Tracking the upstream master branch. New feature branches should be branched from this branch.
- version branches: Tracking the upstream version branches. If you want to push a regression fix to an older release. (See #Regression fix)
- bugfix branches: based on staging. Used to develop bugfixes (see Bug fix)
Git workflow
Local repository setup
All instructions on this page assume the following setup for the local repository. Experienced git users of course may use different setups.
First, create a fork of the official repository (if not already done) on github. Then follow these instructions:
#We use your own fork repository, not the official one!!!!
git clone git://github.com/yourname/dmd.git
#Now add the official repository as a remote
git remote add upstream git://github.com/D-Programming-Language/dmd.git
git remote update
#sync / create branches on origin
git push origin upstream/master:refs/heads/master
git push origin upstream/staging:refs/heads/staging
#Repeat this for every release & feature branch you want to use
git push origin upstream/2.061:refs/heads/2.061
#Now checkout / create local branches
git checkout -b staging origin/staging
#Repeat for every release & feature branch you want to use
git checkout -b 2.061 origin/2.061
The above instructions have to be executed only once. But if a new feature/release branch is available and you want to use it, you have to repeat this part:
git remote update
#Repeat this for every release & feature branch you want to use
git push origin upstream/2.061:refs/heads/2.061
#Repeat for every release & feature branch you want to use
git checkout -b 2.061 origin/2.061
Note: Before creating new branches, etc you should always do this:
git remote update
The instructions on this page always explicitly mention when you have to use this command.
Working on code updates
What type of update?
To determine to which branch an update should be made, we have to consider the type of the change:
- It's a new, big feature: This will need to get a feature branch. See #New feature / Big enhancement
- It's a small enhancement: This will go to master. See #Small enhancement
- It's a bug fix: This will go straight to staging. See Bug fix
- It's a regression fix: This will need to go to the oldest supported version branch which is affected by the regression. See #Regression fix
New feature / Big enhancement
We need to create a new branch based on master. As a normal user you can't create feature branches on the official repository though.
Step-By-Step: The one implementing the feature has to do this:
#Make sure you've read "Local repository setup"
git remote update
git checkout master #This is the branch we base feature on
git pull origin master #Fetch changes from origin
git pull upstream master #Fetch changes from upstream
git branch coolFeature
#Make your changes
git commit -m "Implement cool feature"
#Push to your fork on github
git push origin coolFeature
#Now request in the digitalmars.D newsgroup that someone creates a coolFeature feature branch on the official repository.
#Now go to github and file a pull request for coolFeature. The pull request must target the new official feature branch (here also called coolFeature)!
If someone requests a new feature branch the dmd staff has to do this:
#Make sure you've read "Local repository setup"
git remote update
git checkout master #This is the branch we base features on
git pull upstream master #Fetch changes from upstream
git branch coolFeature #Use a useful name
git push upstream coolFeature
Small enhancement
We merge directly into master.
Step-By-Step: The one implementing the enhancement has to do this:
#We use your own fork repository, not the official one!!!!
#Make sure you've read "Local repository setup"
git remote update
git checkout master #This is the branch we base feature on
git branch coolEnhancement
#Make your changes
git commit -m "Implement cool enhancement"
#Push to your fork on github
git push origin coolEnhancement
#Now go to github and file a pull request for coolEnhancement. The pull request must target the master branch!
Bug fix (not a regression!)
We need to create a local branch based on staging. Then file a pull request on github targeting staging.
Step-By-Step:
#Make sure you've read "Local repository setup"
git remote update
git checkout staging #This is the branch we base bugfix on
git pull origin staging
git pull upstream staging
git branch bugFixNNN
#Make your changes
git commit -m "Implement bugfix #NNN"
#Push to your fork on github
git push origin bugFixNNN
#Now go to github and file a pull request for bugFixNNN. The pull request must target the staging branch!
To make sure that bug fixes are also available in the master development version and in the feature test branches, the dmd staff should merge staging into master. This is not urgent and it's not necessary to do this after every single bug fix, it can be done sporadically.
DMD staff:
#Make sure you've read "Local repository setup"
git remote update
git checkout staging
git pull upstream staging
git checkout master
git pull upstream master
git merge staging
git push upstream master
Regression fix
The regression fix must be pushed to the oldest, still supported version branch affected by the regression. To determine if a version is still supported, see #Support times.
As an example (this doesn't match with #Support times as we need a more complicated setup to describe the concept): 2.063 is the most recent release and supported. 2.062 is the previous release and supported. 2.061 is no longer supported. An regression is found that affects 2.061, 2.062, 2.063, staging and master.
2.061 is no longer supported, so the oldest supported and affected release is 2.062. Therefore the regression fix has to be made on the 2.062 release branch.
Note: Regression fixes are always based on an version branch, never on staging / master!
Step-By-Step: The one fixing the regression has to do this:
#Make sure you've read "Local repository setup"
git remote update
git checkout 2.062 #This is the branch we base our fix on
git pull origin 2.062
git pull upstream 2.062
git branch fixRegressionNNN
#Edit and fix the regression
git commit -m "Fix regression #nnn"
#Push to your fork on github
git push origin fixRegressionNNN
#Now go to github and file a pull request for fixRegressionNNN. The pull request must target our base branch, e.g. 2.062!
The dmd staff now has to merge that pull request. The regression fix will be applied to the 2.062 branch. The dmd staff now also has to make sure this fix is also applied to all newer release branches (2.063 in our example) and to staging and master.
#Make sure you've read "Local repository setup"
git remote update
git checkout 2.062 #This is the branch the fix was based on
git pull upstream 2.062 #Fetch changes from upstream
#Repeat for all newer release branches
git checkout 2.063
git pull upstream 2.063 #Fetch remote changes from upstream first
git merge 2.062 #Now merge 2.062 branch into newer branch
git push upstream 2.063 #Now publish those changes
#We have to repeat that for staging
git checkout staging
git pull upstream staging #Fetch remote changes from upstream first
git merge 2.062 #Now merge 2.062 branch into staging branch
git push upstream staging #Now publish those changes
#We have to repeat that for master
git checkout master
git pull upstream master #Fetch remote changes from upstream first
git merge staging #Now merge staging branch into master branch
git push upstream master #Now publish those changes
Preparing a new major release
As described in #Release Schedule every 2 months a new major release will be made based on staging.
To start the release process (This is always done after a major release has been made) merge master into staging:
#Make sure you've read "Local repository setup"
git remote update
git checkout master
git pull upstream master
git checkout staging
git pull upstream staging
git merge master
git push upstream staging
Once a beta release is made, the dmd staff adds a tag on staging!
#Make sure you've read "Local repository setup"
git remote update
git checkout staging
git pull upstream staging
git tag v2.063-b1 #b1=>first beta
git push upstream v2.063-b1
For the final release, a new version branch is created:
#Make sure you've read "Local repository setup"
git remote update
git checkout staging
git pull upstream staging
git branch 2.063 #replace 2.063 with the new release number
git push upstream 2.063
#And make a tag
git tag v2.063
git push upstream v2.063
Preparing a new minor release
As described in #Release Schedule every 2 weeks a new minor release will be prepared for all supported releases.
To put further regression fixes into this release version follow the descriptions in #Regression fix
Once the release is made, the dmd staff adds a tag!
#Make sure you've read "Local repository setup"
git remote update
git checkout 2.063 #checkout the base branch where a new release is made
git pull upstream 2.063
git tag v2.063.1 #.1=>first minor update
git push upstream v2.063.1
Merging a feature branch
If a features implementation is sufficiently complete and has been tested well enough, the feature branch can be merged into master.
DMD staff:
#Make sure you've read "Local repository setup"
git remote update
git checkout featureXXX #checkout the feature branch
git pull upstream featureXXX
git checkout master
git pull upstream master
git merge featureXXX #if merge fails, do not fix it yourself. contact the maintainer of the feature branch
git push upstream master
Short Summary
Contributors
- Regression fixes go into version branches
- Bug fixes go into the staging branch
- Big new features are developed in feature branches and merged into master
- Small enhancements can go directly into master
DMD staff
- Merge regression fixes into version branch asap.
- Merge regressions fixes from version to other version and to staging branch immediately.
- Merge staging branch into master regularly
- Merge bug fixes into staging regularly.
- Merge feature branches into master on demand.
- Merge master into staging after every release
Rationale
Staging
This setup has the advantage that staging stabilizes over a long time. staging get's a feature updates only after every major release is made. After that staging only recieves bug fixes. It also has the advance that bug fixes also have a clear target branch: With staging we can always target staging. Without staging we'd have to target the version branch of the currently stabilizing version branch. But that branch changes over time, so we'd have to update github pull requests, etc.
Regression vs bug fix
we need to differentiate between bug fixes that do not introduce new bugs vs bug fixes that may introduce new bugs *or* break existing code for the current release version. We also *have* to prevent new features and structural changes (destabilizers) from leaking into a stable release.
http://forum.dlang.org/post/dticrdtkkqdatdnhtzrc@forum.dlang.org
- The regression/bug differentiation is therefore a very strict model. It's needed for dmd as every bug fix has a probability of breaking user code and therefore we should keep the changes as few as possible. In dmd we also have the issue that we have long-standing bugs: Bugs that were not introduced, but are differences between dmd and the specification. We don't want to get any of these fixed in minor releases. For druntime and phobos there are few such bugs: Every bug has been introduced in some version. The risk of a bug fix causing regressions also seems to be lower for phobos and druntime, so we could allow bug fixes in release branches for phobos/druntime after the process has been well established.
Regression fixes in release branches
Now why we fix regressions in the release branches: Let's say we have dmd 2.061, 2.062, 2.062.1. A regression was introduced in 2.062 and my code that compiled fine in 2.061 doesn't work in 2.062 anymore. But then the regression fix is pushed to the 2.062 release branch, a minor release is made and 2.062.1 successfully compiles that code again.
If the regression already occured in 2.061 should it be fixed in 2.062? This is a different question, but that situation can't happen as long as we only support 1 release and only fix newly introduced regressions.
Stable
- http://forum.dlang.org/post/fbtxyjvpqqrxskptcvgk@forum.dlang.org
- http://forum.dlang.org/post/noycjddhpobnnxkqsytx@forum.dlang.org
Possible optimization
Time spans
- If it turns out that staging does not get reasonably stable to create the release immediately from staging we could introduce a time span (e.g 3 weeks) between branching the new version branch and releasing the version / setting the tag. In this timespan only regression fixes would be merged into the version branch. After that time span the release will be tagged on the version branch
- additional betas can be released from the version branch during that time span
<x> = time in weeks
vxxx = Tag for version xxx (Marks the release)
rest = branches
\ = merge (merges from version to staging/master not shown)
---------------------------------------------------------------------------------------------- master branch. development is never stopped
\ <8> \
---------------------------------------------------------------------------------------------- staging branch
| | | |
v2.062.0-b1 | v2.063.0-b1 |
| | <3> .
| <8> -------------------------- 2.063 release branch
| | | <2> |
| rc1 v2.063.0 v2.063.1
| <3> .
------------------------------------------------------------------ 2.062 release branch
| | <2> | <2> |
v2.062.0-rc1 v2.062.0 v2.062.1 v2.062.2
Release version numbering
- When we start shipping shared libraries of druntime and phobos we might have to adjust our versioning scheme as those dictate special rules. (Think of ABI compatibility between minor releases, max 3 numbers for a version, etc)
rebasing
I guess we should have a hard stance on not rebasing any of the official branches. (this especially includes feature branches where this might seem to be useful, but probably is still a bad idea)
