https://wiki.dlang.org/api.php?action=feedcontributions&user=Schuetzm&feedformat=atomD Wiki - User contributions [en]2024-03-19T09:46:36ZUser contributionsMediaWiki 1.31.2https://wiki.dlang.org/?title=DIP89&diff=7531DIP892016-05-22T12:24:56Z<p>Schuetzm: /* Usage */</p>
<hr />
<div>{| class="wikitable"<br />
!Title:<br />
!'''@mutable members in immutable data structures'''<br />
|-<br />
|DIP:<br />
|85<br />
|-<br />
|Version:<br />
|1<br />
|-<br />
|Status:<br />
|Draft<br />
|-<br />
|Created:<br />
|2016-02-21<br />
|-<br />
|Last Modified:<br />
|{{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY}}<br />
|-<br />
|Author:<br />
|Marc Schütz<br />
|-<br />
|Links:<br />
|[https://forum.dlang.org/post/uilqvjlvkuvayygjnyen@forum.dlang.org Forum thread]<br />
|}<br />
<br />
== Abstract ==<br />
This DIP proposes an officially sanctioned way to mutate members in immutable data structures.<br />
<br />
== Rationale ==<br />
D's <code>immutable</code> signifies - in constrast to C++'s <code>const</code> - physical immutability, i.e. a guarantee that memory occupied by immutable variables will never change during their lifetime. This contrasts with logical immutability, which means that the underlying memory can change as long as the object represented by it remains semantically unchanged. Applications of logical immutability include lazy initialization, mutexes, reference counters or other embedded house-keeping data, as well as members changed for debugging purposes.<br />
<br />
Because D's <code>immutable</code> is as strict as it is (casting it away results in undefined behaviour), and <code>const</code> data may actually be <code>immutable</code>, in order to use the above-mentioned techniques, variables must either be mutable (which, because of <code>const</code> and <code>immutable</code>'s transitivity implies that many other variables and parameters cannot be marked as <code>const</code> either), or storing the mutable parts outside of the structures, which has considerable complexity, runtime and memory overhead, and can even be unsafe in combination with implicit sharing of <code>immutable</code> data (see below).<br />
<br />
With the proposed change, logical immutability (i.e. no changes are observable from the outside) can be achieved without provoking undefined behaviour while still having some basic statically enforced safety.<br />
<br />
== Description ==<br />
A new annotation <code>@mutable</code> for member variables and aggregate types is proposed. It is neither a type constructor, nor a storage class; it can be implemented as a compiler-recognized UDA. A member annotated as <code>@mutable</code> triggers the following behaviours:<br />
<br />
# It is required to be <code>private</code><br />
# Access to it is <code>@system</code><br />
# No static immutable objects with a <code>@mutable</code> member may be created<br />
# Dynamically created immutable objects with <code>@mutable</code> members are allowed if all <code>@mutable</code> members are marked as <code>shared</code> (analogously for implicit conversion of unique objects to immutable)<br />
<br />
These rules are enforced statically. Rationale for the rules:<br />
<br />
* The first rule (<code>private</code>) enforces encapsulation. This is the basic property of logical const-ness: an observer must never observe a change to an immutable object.<br />
* The second rule (<code>@system</code>) prevents accidental accesses that violate the above guarantee. This includes not just actual mutation of <code>@mutable</code> members, but even reads from them, because these can leak changed data to the outside. (If desired, this rule can be relaxed: reads in non-pure methods can be @safe.)<br />
* The third rule (no static immutables) is necessary because static immutable objects could be placed in physically read-only memory by the linker and therefore cannot be modified. Even though existing memory can be made read-only after initialization (using system calls like <code>mmap(2)</code>), doing this is not supposed to be prevented by the type system, because the <code>mmap</code>ed region can just as well contain normal mutable data.<br />
* The fourth rule (<code>shared</code>) prevents race conditions for implicitly shared immutable objects. Access to shared <code>@mutable</code> members must be atomic or otherwise synchronized.<br />
<br />
The compiler needs to make sure not to apply optimizations based on the assumption that a <code>@mutable</code> member never changes. Because D supports opaque data structures (<code>struct S;</code>), the <code>@mutable</code> annotation can also be attached to struct declarations: <code>@mutable struct S;</code>.<br />
<br />
To enable introspection, two traits are added: <code>isMutable</code>, and <code>hasMutableMembers</code>. The latter determines whether a types contains any mutable members, either directly, or embedded through another member.<br />
<br />
== Usage ==<br />
<source lang="D"><br />
struct S {<br />
@safe pure int expensiveComputation() const;<br />
private @mutable int bar_;<br />
@trusted @property bar() const {<br />
if(!bar_)<br />
bar_ = expensiveComputation();<br />
return bar_;<br />
}<br />
}<br />
</source><br />
<br />
==About the AA solution==<br />
It has been proposed to place the mutable members into an external associate array, with the object as a key. This approach is surprisingly complex: not only does it have a considerable computational and memory cost (including caching effects), it also requires lifetime management of the AA's values.<br />
<br />
Additionally, it can have unexpected effects with shared objects (including immutable ones, which are implicitly shareable): while strictly speaking, it doesn't really violate safety by itself, it can have surprising consequences that the compiler is unable to guard against, because the associative array and the objects themselves can have non-matching shared-ness, as there is no formal relationship between the two. Take a reference counted immutable object as an example:<br />
<br />
<source lang="D"><br />
int[const(RCObject)] refcounts;<br />
struct RCObject {<br />
@disable this();<br />
static make() {<br />
immutable(RCObject) result;<br />
refcounts[result] = 1;<br />
return result;<br />
}<br />
this(this) immutable {<br />
refcounts[this]++;<br />
}<br />
~this() immutable {<br />
if(--refcounts[this] == 0)<br />
releaseResources();<br />
}<br />
}<br />
void foo() {<br />
immutable(RCObject) o = RCObject.make();<br />
send(otherTid, o);<br />
}<br />
</source><br />
<br />
Because <code>refcounts</code> in the example above is not marked as shared, it will be a thread-local instance. An immutable object sent to another thread will not have an entry in that thread's AA. The correct solution in this case would be to make the AA <code>shared</code> and to use atomic operations on its values. On the other hand, if it's guaranteed that the objects never cross a thread-boundary, the code is sufficient as-is. Unfortunately, the compiler cannot enforce the correct solution here.<br />
<br />
Now, using the changes proposed in this DIP, the code can be made safe by providing a shareable and a thread-local implementation of <code>RCObject</code>. Should the user choose the wrong one, the compiler will reject it because of rule 4.<br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=DIP89&diff=7530DIP892016-05-22T12:24:19Z<p>Schuetzm: /* Usage */</p>
<hr />
<div>{| class="wikitable"<br />
!Title:<br />
!'''@mutable members in immutable data structures'''<br />
|-<br />
|DIP:<br />
|85<br />
|-<br />
|Version:<br />
|1<br />
|-<br />
|Status:<br />
|Draft<br />
|-<br />
|Created:<br />
|2016-02-21<br />
|-<br />
|Last Modified:<br />
|{{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY}}<br />
|-<br />
|Author:<br />
|Marc Schütz<br />
|-<br />
|Links:<br />
|[https://forum.dlang.org/post/uilqvjlvkuvayygjnyen@forum.dlang.org Forum thread]<br />
|}<br />
<br />
== Abstract ==<br />
This DIP proposes an officially sanctioned way to mutate members in immutable data structures.<br />
<br />
== Rationale ==<br />
D's <code>immutable</code> signifies - in constrast to C++'s <code>const</code> - physical immutability, i.e. a guarantee that memory occupied by immutable variables will never change during their lifetime. This contrasts with logical immutability, which means that the underlying memory can change as long as the object represented by it remains semantically unchanged. Applications of logical immutability include lazy initialization, mutexes, reference counters or other embedded house-keeping data, as well as members changed for debugging purposes.<br />
<br />
Because D's <code>immutable</code> is as strict as it is (casting it away results in undefined behaviour), and <code>const</code> data may actually be <code>immutable</code>, in order to use the above-mentioned techniques, variables must either be mutable (which, because of <code>const</code> and <code>immutable</code>'s transitivity implies that many other variables and parameters cannot be marked as <code>const</code> either), or storing the mutable parts outside of the structures, which has considerable complexity, runtime and memory overhead, and can even be unsafe in combination with implicit sharing of <code>immutable</code> data (see below).<br />
<br />
With the proposed change, logical immutability (i.e. no changes are observable from the outside) can be achieved without provoking undefined behaviour while still having some basic statically enforced safety.<br />
<br />
== Description ==<br />
A new annotation <code>@mutable</code> for member variables and aggregate types is proposed. It is neither a type constructor, nor a storage class; it can be implemented as a compiler-recognized UDA. A member annotated as <code>@mutable</code> triggers the following behaviours:<br />
<br />
# It is required to be <code>private</code><br />
# Access to it is <code>@system</code><br />
# No static immutable objects with a <code>@mutable</code> member may be created<br />
# Dynamically created immutable objects with <code>@mutable</code> members are allowed if all <code>@mutable</code> members are marked as <code>shared</code> (analogously for implicit conversion of unique objects to immutable)<br />
<br />
These rules are enforced statically. Rationale for the rules:<br />
<br />
* The first rule (<code>private</code>) enforces encapsulation. This is the basic property of logical const-ness: an observer must never observe a change to an immutable object.<br />
* The second rule (<code>@system</code>) prevents accidental accesses that violate the above guarantee. This includes not just actual mutation of <code>@mutable</code> members, but even reads from them, because these can leak changed data to the outside. (If desired, this rule can be relaxed: reads in non-pure methods can be @safe.)<br />
* The third rule (no static immutables) is necessary because static immutable objects could be placed in physically read-only memory by the linker and therefore cannot be modified. Even though existing memory can be made read-only after initialization (using system calls like <code>mmap(2)</code>), doing this is not supposed to be prevented by the type system, because the <code>mmap</code>ed region can just as well contain normal mutable data.<br />
* The fourth rule (<code>shared</code>) prevents race conditions for implicitly shared immutable objects. Access to shared <code>@mutable</code> members must be atomic or otherwise synchronized.<br />
<br />
The compiler needs to make sure not to apply optimizations based on the assumption that a <code>@mutable</code> member never changes. Because D supports opaque data structures (<code>struct S;</code>), the <code>@mutable</code> annotation can also be attached to struct declarations: <code>@mutable struct S;</code>.<br />
<br />
To enable introspection, two traits are added: <code>isMutable</code>, and <code>hasMutableMembers</code>. The latter determines whether a types contains any mutable members, either directly, or embedded through another member.<br />
<br />
== Usage ==<br />
<source lang="D"><br />
struct S {<br />
@safe pure int expensiveComputation();<br />
private @mutable int bar_;<br />
@trusted @property bar() const {<br />
if(!bar_)<br />
bar_ = expensiveComputation();<br />
return bar_;<br />
}<br />
}<br />
</source><br />
<br />
==About the AA solution==<br />
It has been proposed to place the mutable members into an external associate array, with the object as a key. This approach is surprisingly complex: not only does it have a considerable computational and memory cost (including caching effects), it also requires lifetime management of the AA's values.<br />
<br />
Additionally, it can have unexpected effects with shared objects (including immutable ones, which are implicitly shareable): while strictly speaking, it doesn't really violate safety by itself, it can have surprising consequences that the compiler is unable to guard against, because the associative array and the objects themselves can have non-matching shared-ness, as there is no formal relationship between the two. Take a reference counted immutable object as an example:<br />
<br />
<source lang="D"><br />
int[const(RCObject)] refcounts;<br />
struct RCObject {<br />
@disable this();<br />
static make() {<br />
immutable(RCObject) result;<br />
refcounts[result] = 1;<br />
return result;<br />
}<br />
this(this) immutable {<br />
refcounts[this]++;<br />
}<br />
~this() immutable {<br />
if(--refcounts[this] == 0)<br />
releaseResources();<br />
}<br />
}<br />
void foo() {<br />
immutable(RCObject) o = RCObject.make();<br />
send(otherTid, o);<br />
}<br />
</source><br />
<br />
Because <code>refcounts</code> in the example above is not marked as shared, it will be a thread-local instance. An immutable object sent to another thread will not have an entry in that thread's AA. The correct solution in this case would be to make the AA <code>shared</code> and to use atomic operations on its values. On the other hand, if it's guaranteed that the objects never cross a thread-boundary, the code is sufficient as-is. Unfortunately, the compiler cannot enforce the correct solution here.<br />
<br />
Now, using the changes proposed in this DIP, the code can be made safe by providing a shareable and a thread-local implementation of <code>RCObject</code>. Should the user choose the wrong one, the compiler will reject it because of rule 4.<br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=DIP89&diff=7273DIP892016-02-21T18:49:07Z<p>Schuetzm: fixes</p>
<hr />
<div>{| class="wikitable"<br />
!Title:<br />
!'''@mutable members in immutable data structures'''<br />
|-<br />
|DIP:<br />
|85<br />
|-<br />
|Version:<br />
|1<br />
|-<br />
|Status:<br />
|Draft<br />
|-<br />
|Created:<br />
|2016-02-21<br />
|-<br />
|Last Modified:<br />
|{{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY}}<br />
|-<br />
|Author:<br />
|Marc Schütz<br />
|-<br />
|Links:<br />
|[https://forum.dlang.org/post/uilqvjlvkuvayygjnyen@forum.dlang.org Forum thread]<br />
|}<br />
<br />
== Abstract ==<br />
This DIP proposes an officially sanctioned way to mutate members in immutable data structures.<br />
<br />
== Rationale ==<br />
D's <code>immutable</code> signifies - in constrast to C++'s <code>const</code> - physical immutability, i.e. a guarantee that memory occupied by immutable variables will never change during their lifetime. This contrasts with logical immutability, which means that the underlying memory can change as long as the object represented by it remains semantically unchanged. Applications of logical immutability include lazy initialization, mutexes, reference counters or other embedded house-keeping data, as well as members changed for debugging purposes.<br />
<br />
Because D's <code>immutable</code> is as strict as it is (casting it away results in undefined behaviour), and <code>const</code> data may actually be <code>immutable</code>, in order to use the above-mentioned techniques, variables must either be mutable (which, because of <code>const</code> and <code>immutable</code>'s transitivity implies that many other variables and parameters cannot be marked as <code>const</code> either), or storing the mutable parts outside of the structures, which has considerable complexity, runtime and memory overhead, and can even be unsafe in combination with implicit sharing of <code>immutable</code> data (see below).<br />
<br />
With the proposed change, logical immutability (i.e. no changes are observable from the outside) can be achieved without provoking undefined behaviour while still having some basic statically enforced safety.<br />
<br />
== Description ==<br />
A new annotation <code>@mutable</code> for member variables and aggregate types is proposed. It is neither a type constructor, nor a storage class; it can be implemented as a compiler-recognized UDA. A member annotated as <code>@mutable</code> triggers the following behaviours:<br />
<br />
# It is required to be <code>private</code><br />
# Access to it is <code>@system</code><br />
# No static immutable objects with a <code>@mutable</code> member may be created<br />
# Dynamically created immutable objects with <code>@mutable</code> members are allowed if all <code>@mutable</code> members are marked as <code>shared</code> (analogously for implicit conversion of unique objects to immutable)<br />
<br />
These rules are enforced statically. Rationale for the rules:<br />
<br />
* The first rule (<code>private</code>) enforces encapsulation. This is the basic property of logical const-ness: an observer must never observe a change to an immutable object.<br />
* The second rule (<code>@system</code>) prevents accidental accesses that violate the above guarantee. This includes not just actual mutation of <code>@mutable</code> members, but even reads from them, because these can leak changed data to the outside. (If desired, this rule can be relaxed: reads in non-pure methods can be @safe.)<br />
* The third rule (no static immutables) is necessary because static immutable objects could be placed in physically read-only memory by the linker and therefore cannot be modified. Even though existing memory can be made read-only after initialization (using system calls like <code>mmap(2)</code>), doing this is not supposed to be prevented by the type system, because the <code>mmap</code>ed region can just as well contain normal mutable data.<br />
* The fourth rule (<code>shared</code>) prevents race conditions for implicitly shared immutable objects. Access to shared <code>@mutable</code> members must be atomic or otherwise synchronized.<br />
<br />
The compiler needs to make sure not to apply optimizations based on the assumption that a <code>@mutable</code> member never changes. Because D supports opaque data structures (<code>struct S;</code>), the <code>@mutable</code> annotation can also be attached to struct declarations: <code>@mutable struct S;</code>.<br />
<br />
To enable introspection, two traits are added: <code>isMutable</code>, and <code>hasMutableMembers</code>. The latter determines whether a types contains any mutable members, either directly, or embedded through another member.<br />
<br />
== Usage ==<br />
<source lang="D"><br />
struct S {<br />
@safe int expensiveComputation();<br />
private @mutable int bar_;<br />
@trusted @property bar() const {<br />
if(!bar_)<br />
bar_ = expensiveComputation();<br />
return bar_;<br />
}<br />
}<br />
</source><br />
<br />
==About the AA solution==<br />
It has been proposed to place the mutable members into an external associate array, with the object as a key. This approach is surprisingly complex: not only does it have a considerable computational and memory cost (including caching effects), it also requires lifetime management of the AA's values.<br />
<br />
Additionally, it can have unexpected effects with shared objects (including immutable ones, which are implicitly shareable): while strictly speaking, it doesn't really violate safety by itself, it can have surprising consequences that the compiler is unable to guard against, because the associative array and the objects themselves can have non-matching shared-ness, as there is no formal relationship between the two. Take a reference counted immutable object as an example:<br />
<br />
<source lang="D"><br />
int[const(RCObject)] refcounts;<br />
struct RCObject {<br />
@disable this();<br />
static make() {<br />
immutable(RCObject) result;<br />
refcounts[result] = 1;<br />
return result;<br />
}<br />
this(this) immutable {<br />
refcounts[this]++;<br />
}<br />
~this() immutable {<br />
if(--refcounts[this] == 0)<br />
releaseResources();<br />
}<br />
}<br />
void foo() {<br />
immutable(RCObject) o = RCObject.make();<br />
send(otherTid, o);<br />
}<br />
</source><br />
<br />
Because <code>refcounts</code> in the example above is not marked as shared, it will be a thread-local instance. An immutable object sent to another thread will not have an entry in that thread's AA. The correct solution in this case would be to make the AA <code>shared</code> and to use atomic operations on its values. On the other hand, if it's guaranteed that the objects never cross a thread-boundary, the code is sufficient as-is. Unfortunately, the compiler cannot enforce the correct solution here.<br />
<br />
Now, using the changes proposed in this DIP, the code can be made safe by providing a shareable and a thread-local implementation of <code>RCObject</code>. Should the user choose the wrong one, the compiler will reject it because of rule 4.<br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=DIP89&diff=7271DIP892016-02-21T15:04:23Z<p>Schuetzm: Created page with "{| class="wikitable" !Title: !'''@mutable members in immutable data structures''' |- |DIP: |85 |- |Version: |1 |- |Status: |Draft |- |Created: |2016-02-21 |- |Last Modified: |..."</p>
<hr />
<div>{| class="wikitable"<br />
!Title:<br />
!'''@mutable members in immutable data structures'''<br />
|-<br />
|DIP:<br />
|85<br />
|-<br />
|Version:<br />
|1<br />
|-<br />
|Status:<br />
|Draft<br />
|-<br />
|Created:<br />
|2016-02-21<br />
|-<br />
|Last Modified:<br />
|{{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY}}<br />
|-<br />
|Author:<br />
|Marc Schütz<br />
|-<br />
|Links:<br />
|[https://forum.dlang.org/post/uilqvjlvkuvayygjnyen@forum.dlang.org Forum thread]<br />
|}<br />
<br />
== Abstract ==<br />
This DIP proposes an officially sanctioned way to mutate members in immutable data structures.<br />
<br />
== Rationale ==<br />
D's <code>immutable</code> signifies - in constrast to C++'s <code>const</code> - physical immutability, i.e. a guarantee that memory occupied by immutable variables will never change during their lifetime. This contrasts with logical immutability, which means that the underlying memory can change as long as the object represented by it remains semantically unchanged. Applications of logical immutability include lazy initialization, mutexes, reference counters or other embedded house-keeping data, as well as members changed for debugging purposes.<br />
<br />
Because D's <code>immutable</code> is as strict as it is (casting it away results in undefined behaviour), and <code>const</code> data may actually be <code>immutable</code>, in order to use the above-mentioned techniques, variables must either be mutable (which, because of <code>const</code> and <code>immutable</code>'s transitivity implies that many other variables and parameters cannot be marked as <code>const</code> either), or storing the mutable parts outside of the structures, which has considerable complexity, runtime and memory overhead, and can even be unsafe in combination with implicit sharing of <code>immutable</code> data (see below).<br />
<br />
With the proposed change, logical immutability (i.e. no changes are observable from the outside) can be achieved without provoking undefined behaviour while still having some basic statically enforced safety.<br />
<br />
== Description ==<br />
A new annotation <code>@mutable</code> for member variables and aggregate types is proposed. It is neither a type constructor, not a storage class; it can be implemented as a compiler-recognized UDA. A member annotated as <code>@mutable</code> triggers the following behaviours:<br />
<br />
* it is required to be <code>private</code><br />
* access to it is <code>@system</code><br />
* no static immutable objects with a <code>@mutable</code> member may be created<br />
* dynamically created immutable objects with <code>@mutable</code> members are allowed if all <code>@mutable</code> members are marked as <code>shared</code> (analogously for implicit conversion of unique objects to immutable)<br />
<br />
These rules are enforced statically. Rationale for the rules:<br />
<br />
* The first rule (<code>private</code>) enforces encapsulation. This is the basic property of logical const-ness: an observer must never observe a change to an immutable object.<br />
* The second rule (<code>@system</code>) prevents accidental accesses that violate the above guarantee. This includes not just actual mutation of <code>@mutable</code> members, but even reads from them, because these can leak changed data to the outside. (If desired, this rule can be relaxed: reads in non-pure methods can be @safe.)<br />
* The third rule (no static immutables) is necessary because static immutable objects could be placed in physically read-only memory by the linker and therefore cannot be modified. Even though existing memory can be made read-only after initialization (using system calls like <code>mmap(2)</code>), doing this is not supposed to be prevented by the type system, because the <code>mmap</code>ed region can just as well contain normal mutable data.<br />
* The fourth rule (<code>shared</code>) prevents race conditions for implicitly shared immutable objects. Access to shared <code>@mutable</code> members must be atomic or otherwise synchronized.<br />
<br />
The compiler needs to make sure not to apply optimizations based on the assumption that a <code>@mutable</code> member never changes. Because D supports opaque data structures (<code>struct S;</code>), the <code>@mutable</code> annotation can also be attached to struct declarations: <code>@mutable struct S;</code>.<br />
<br />
To enable introspection, two traits are added: <code>isMutable</code>, and <code>hasMutableMembers</code>. The latter determines whether a types contains any mutable members, either directly, or embedded through another member.<br />
<br />
== Usage ==<br />
<source lang="D"><br />
struct S {<br />
@safe int expensiveComputation();<br />
lazy private int bar_;<br />
@trusted @property bar() const {<br />
if(!bar_)<br />
bar_ = expensiveComputation();<br />
return bar_;<br />
}<br />
}<br />
</source><br />
<br />
==About the AA solution==<br />
It has been proposed to place the mutable members into an external associate array, with the object as a key. This approach is surprisingly complex: not only does it have a considerable computational and memory cost (including caching effects), it also requires lifetime management of the AA's values.<br />
<br />
Additionally, it can have unexpected effects with shared objects (including immutable ones, which are implicitly shareable): while strictly speaking, it doesn't really violate safety by itself, it can have surprising consequences that the compiler is unable to guard against, because the associative array and the objects themselves can have non-matching shared-ness, as there is no formal relationship between the two. Take a reference counted immutable object as an example:<br />
<br />
<source lang="D"><br />
int[const(RCObject)] refcounts;<br />
struct RCObject {<br />
this() {<br />
refcounts[this] = 1;<br />
}<br />
this(this) {<br />
refcounts[this]++;<br />
}<br />
~this() {<br />
if(--refcounts[this] == 0)<br />
releaseResources();<br />
}<br />
}<br />
void foo() {<br />
auto o = new immutable(RCObject);<br />
send(otherTid, o);<br />
}<br />
</source><br />
<br />
Because <code>refcounts</code> in the example above is not marked as shared, it will be a thread-local instance. An immutable object sent to another thread will not have an entry in that thread's AA. The correct solution in this case would be to make the AA <code>shared</code> and to use atomic operations on its values. On the other hand, if it's guaranteed that the objects never cross a thread-boundary, the code is sufficient as-is. Unfortunately, the compiler cannot enforce the correct solution here.<br />
<br />
Now, using the changes proposed in this DIP, the code can be made safe by providing a shareable and a thread-local implementation of <code>RCObject</code>. Should the user choose the wrong one, the compiler will reject it because of rule 4.<br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=GSOC_2016_Ideas&diff=7186GSOC 2016 Ideas2016-01-15T10:09:57Z<p>Schuetzm: /* Phobos: D Standard Library */</p>
<hr />
<div>This is the D Google Summer of Code page for 2016 - it is currently under heavy construction, and may remain so until February, when it will be finalized. To learn more about this year's event, see the [https://www.google-melange.com/gsoc/homepage/google/gsoc2015 Google Summer of Code 2015 page]. If you are interested in participating in the 2016 GSOC as either a student or mentor, and want to do something related to D, please feel free to contact our GSOC administrator Craig Dillabaugh (firstname dot lastname at gmail dot com).<br />
<br />
<br />
== Timeline ==<br />
The timeline for GSoC for 2016 has now been posted [https://developers.google.com/open-source/gsoc/ here].<br />
<br />
== Ideas ==<br />
<br />
Plenty of challenging and important projects exist in the D world. They range from writing new or improving existing modules of D's standard library ([http://dlang.org/phobos/index.html Phobos]), working on its compilers ([[Compilers]]), shaping GUI libraries for D, integrating D with other languages and more.<br />
<br />
=== SDC Project - D Compiler as a Library ===<br />
----<br />
<br />
==== Project Desription: ==== <br />
The SDC project (https://github.com/deadalnix/SDC) is an effort to provide a D compiler as a library.<br />
Any ideas to further the development of this project are welcome,<br />
but for a student who would like a specific project we <br />
propose the following <br />
* Start by implementing with @property feature of D. This feature will allow a D programmer to create functions that are called using the same syntax as variable access.<br />
* Using the @property feature the student will be able to implement the runtime support for slices and associative arrays. The operations to implement are as follows:<br />
** Implement arrray operations like concatenation and appending, and implement a sound memory management strategy for the underlying data.<br />
** Implement a generic and efficient hash table. The data structure and algorithms used must be flexibile enough to be adapted to any type of data that might be stored in the table. A concurrent version of the table is need for shared data.<br />
* Finally, the student will implement masquerading of D syntax into calls for the runtime.<br />
<br />
==== Its Good To Know ====<br />
* Please watch Amaury's DConf [http://dconf.org/2014/talks/sechet.html talk] on SDC.<br />
* SDC is developed in D (of course) so you will need to be proficient in D by the time you start coding.<br />
* You should have taken at least one course on compilers, or at the least be willing to educate yourself in this regard. There is a decent course availabe through Coursera https://www.coursera.org/course/compilers<br />
* You should familiarize yourself with classicial data structures for arrays and have knowledge of various schemes for table implementations, (it is worthwhile to read up on hopscotch and robin hood hashing).<br />
* SDC uses LLVM for code generation, so some familiarity with LLVM will be required (see http://llvm.org/docs/tutorial/index.html).<br />
<br />
==== Proposed Project Mentor: [http://wiki.dlang.org/GSOC_mentors#Amaury_Sechet Amaury Sechet] ====<br />
<br />
=== GDC Project - The GNU D Compiler ===<br />
---- <br />
GDC is a D compiler based on the GCC backend. There are a few outstanding issues in GDC that have ended up in the backlog of known issues to fix, as well as changes that could be beneficial to upstream and other compilers.<br />
<br />
====Compiler Improvements====<br />
* Implement support for compiling with LTO (<code>-flto</code>) turned on.<br />
* There are various XBUG markers in GDC's testsuite, some haven't been looked at in a long time. There should be fewer!<br />
* Internal Documentation. Ideally every function should have a comment describing its use and input values. As the bulk of what happens in the glue is just providing for backend members declared in the front-end. Most should be pretty trivial to work through.<br />
* Debugging Support. Enumerals of non-scalar types currently emit no debugging information. A solution should be proposed and implemented for this.<br />
* Improving codegen for better optimisations. Currently all shared types are marked volatile, which is a workaround for not being able to prevent shared decls from being cached into registers by other means. Improvements could also be done around parameter passing, taking advantage of D storage attributes.<br />
<br />
====Library Improvements====<br />
* Get shared library support working. The current implementation upstream in DM druntime has not been tested or merged in, and it is not clear what changes are required to support it, or whether is even going to be feasibly possible with GDC.<br />
* GDC EH support is lacking the ability to chain multiple thrown exceptions together in its deh personality routines.<br />
* More targets are expected to land in GDC pretty soon. In preparation for this, we should investigate an alternate layout of druntime before the long lists of version conditions becomes a problem for port maintainers (See [https://d.puremagic.com/issues/show_bug.cgi?id=11666 #11666]).<br />
<br />
==== Proposed Project Mentor: [http://wiki.dlang.org/GSOC_mentors#Iain_Buclaw Iain Buclaw] ====<br />
<br />
<br />
=== DDT ===<br />
----<br />
[http://code.google.com/p/ddt/ DDT] is an Eclipse-based IDE for the D programming lanaguage, aiming to support rich functionality, particularly semantic functionality (code completion, find definition, find references, etc.). There are two areas which prospective students could target for improvement:<br />
* DDT Core Engine<br />
** Make the DDT semantic engine available as a command-line daemon tool (similar to [https://github.com/Hackerpilot/DCD DCD]).<br />
** Add support for source formatting (with formatting options).<br />
** Add support for semantic search in the semantic engine (search symbols by anem and type). For example, "where in my code is the function std.stdio.writeln" called?", or "which classes in my code subclass the given class?".<br />
** Improve semantic engine / code completion capabilities. For example, understand template instantiation, function overloads, etc.<br />
* Eclipse Specific Enhancements<br />
** Improve/add UI support for [http://code.dlang.org/about DUB] multiple build configurations and launch.<br />
** Reduce usage of [https://eclipse.org/dltk/ DLTK] code, possibly refactoring or rewriting DLTK functionality into an IDE-neutral layer (LangEclipseIDE).<br />
** Add support for continous build mode (build and report errors on the fly).<br />
<br />
Some of these ideas are a bit small for a full GGoC project, so it might make sense to combine a couple of tasks into a single project.<br />
<br />
<br />
<br />
==== Proposed Project Mentor: [http://wiki.dlang.org/GSOC_mentors#Bruno_Medeiros Bruno Medeiros] ====<br />
<br />
=== QML Bindings ===<br />
---- <br />
The [https://github.com/filcuc/DOtherSide DOtherSide] project is an effort to develop D bindings to QML. There are a number of possible contributions that a student could contribute to. These include:<br />
* (Easy) (Examples) Mut be put on par with those of Nim<br />
* (Difficult) (D) Find a way for simplifiing the syntax by injecting code generated at compile time. This task implies a deep/rich knowledge of D metaprogramming features like template mixins and mixin.<br />
* (Medium) (D) Enable the creation QObject subclasses. Right now this is limited to one level of hierarchy in the D side. We can use the same solution implemented in Nim.<br />
<br />
Given these points there're other tasks regarding the plain C++ backend:<br />
- (Medium) Find a way to create instances of types created in D/Nim and make them instantiable in Qml.<br />
<br />
Some projects for which there is not already a solution:<br />
* (Medium) Right now there's no support for connecting signals and slots between object defined in the D/Nim side.<br />
* (Difficult) Find a way to bundle the QML as resources from both D and Nim and make them available to Qml<br />
* (Medium) Expose an API for adding a custom image provider<br />
<br />
<br />
==== Proposed Project Mentors: Russel Winder, Filippo Cucchetto ====<br />
<br />
=== Bare Metal ARM Support ===<br />
----<br />
This is a project focuses on bare metal cross-compilers and an appealing eco-system to surround it. The goal is to raise the awareness of D in the open hardware community at large and prove D's systems programming lanugage title along the way. It's hard not to notice the growing momentum of the open hardware movement. Yesterday's newbies awkwardly messing with theirs first [http://arduino.cc/ Arduinos] are today's mature embedded systems hackers. The hanger for ever more powerful systems takes them en mass towards ARM development boards. These systems often lack a good open source toolchain, e.g. linker scripts, HAL, so that D certainly can be a compelling offering in this setting (compared to C/C++ in usability). The project is two-fold and can be geared towards any of the 2 major parts:<br />
* Getting a bare-metal D runtime support on Cortex-M3/M4 with as many language features as possible. A minor victory would be having a minimal thread-less and GC-less runtime. A more ambitious plan here is creating a small-ish RTOS in D, to provide full runtime with threads, mutexes etc. <br />
* Create a [http://wiring.org.co/reference/ Wiring] or [http://developer.mbed.org/ mbed] look-alike library in D for a selected family of popular chips (and consequently a range of boards). Instead of cloning a popular API you might also design your own hardware abstraction library using D to it's full extend. The challenge here is to tackle things like GPIOs, USARTs, I2C, SPIs, PWM, ADC and other peripherals commonly found in MCUs in a future-proof way.<br />
<br />
==== Its Good To Know ====<br />
Cool demos (including public videos) are an obligatory part of the project ;) For a starting point see this [https://bitbucket.org/timosi/minlibd project] (some D working on STM32 boards). Michael V. Franklin has done some work bare-metal ARM which is worthwhile to review:<br />
* [http://dconf.org/2014/talks/franklin.html D Conf Talk]<br />
* [https://github.com/JinShil/D_Runtime_ARM_Cortex-M_study/wiki/1.0-Introduction ARM Cortex-M study]<br />
<br />
==== Proposed Project Mentor: [http://wiki.dlang.org/GSOC_mentors#Martin_Nowak Martin Nowak] ====<br />
<br />
=== Phobos: D Standard Library ===<br />
----<br />
There is still signficant work to be done in order to get the D Standard<br />
Libraries into a proper state. The following libraries are good candidates<br />
for GSoC projects:<br />
* '''std.benchmark''': (Russell) <br />
* '''std.i18n''': (Russell) Design and implement a basic internationalization framework. It may be possible to implement this with '''pragma(msg)'''. For proof of concept see http://arsdnet.net/dcode/i18n.d . It should provide at least the following functionality:<br />
** A locale part, std.i18n.locale which should detect a user's default lanaguage, select a global application lanaguage, and provide types to describe and work with locales.<br />
** A text translation part, std.i18n.translation, which should be [https://www.gnu.org/software/gettext/ gettext] compatible, preferably using the gettext serialized hashtables in .mo files, and provide low level (gettext like) and high level (boost::locale like) APIs.<br />
** A tool to extract strings which need to be translated from D code. This should preferably be based on [https://github.com/Hackerpilot/Dscanner DScanner] but alternately could use regular expressions. Optionally support for D code could be added to xgettext.<br />
* '''std.parallelism''' (Russell) std.parallelism needs a review and some benchmarking - prior to making improvements. As part of this is would be good to have a standard benchmarking framework, hence the idea of std.benchmark. However there is no need for it to be in std (and hence Phobos) in the first instance. So the project(s) would be to create a comparative benchmarking framework that can then be used to analyse std.parallelism on a more scientific basis than has been done to date.<br />
* '''std.units''': unit conversion library<br />
* '''std.xml ''' The std.xml module badly needs a rewrite. For inspiration you can look at an abandoned attempt to update it [http://bazaar.launchpad.net/~michael-rynn-500/d2-xml/d2-xml-dev/files/head:/std/xmlp/ here]. Another bit of code you may want to check out for ideas is Adam D. Ruppe's [https://github.com/adamdruppe/arsd dom.d].<br />
* '''std.serialization''': A flexible (de)serialization framework that can be used as a standardized building block for various serialization related things: std.json, std.csv, Protocol Buffers, Cap'n Proto, vibe.d... One important goal would be to define how a user-specified type has to work in order to make it serializable, as well as to allow the end-user (e.g. a user of std.json) to change the serialization of third-party types that cannot themselves be modified.<br />
==== Proposed Project Mentors: [http://wiki.dlang.org/GSOC_mentors#Andrei_Alexandrescu Andrei Alexandrescu], Jacob Ovrum, Russel Winder ====<br />
<br />
== Ideas From Previous Years ==<br />
GSoC idea pages from past years:<br />
* [http://wiki.dlang.org/GSOC_2015_Ideas GSoC 2015 Ideas] (not accepted)<br />
* [http://wiki.dlang.org/GSOC_2014_Ideas GSoC 2014 Ideas] (not accepted)<br />
* [http://wiki.dlang.org/GSOC_2013_Ideas GSoC 2013 Ideas] (not accepted)<br />
* [http://www.prowiki.org/wiki4d/wiki.cgi?GSOC_2012_Ideas GSoC 2012 Ideas]<br />
* [http://www.prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas GSoC 2011 Ideas]<br />
<br />
== Tips for students ==<br />
Daniel Pocock has [http://danielpocock.com/getting-selected-for-google-summer-of-code-2015 written a detailed blog about initiatives students can take if they want to have a serious chance of being selected in GSoC] without a focus on one specific organization.<br />
<br />
To learn more about potential mentors check our mentor page [http://wiki.dlang.org/GSOC_mentors here].<br />
<br />
== Tips for Mentors ==<br />
<br />
If you are interested in mentoring, please check out the [http://en.flossmanuals.net/GSoCMentoring/ organization administrator and mentor] manual for more information.<br />
<br />
<br />
[[Category:GSOC]]</div>Schuetzmhttps://wiki.dlang.org/?title=The_D_Programming_Language&diff=6997The D Programming Language2015-11-27T12:28:45Z<p>Schuetzm: dconf 2015 is in the past</p>
<hr />
<div>__NOTOC__<br />
<table cellspacing="10" style="width:100%; margin-top:-3em;"><br />
<tr><br />
<td><br />
[http://dlang.org/index.html The D Programming Language] has been said to be "what C++ wanted to be," which is a better C. D is developed with system level programming in mind, but brings to the table modern language design with a simple C-like syntax. For these reasons D makes for a good language choice for both performance code and application development.<br />
<br />
D is rapidly reaching a stable [http://dlang.org/spec.html specification] and [http://dlang.org/download.html implementation].<br />
<br />
[http://www.amazon.com/exec/obidos/ASIN/0321635361/classicempire "The D Programming Language"] by Andrei Alexandrescu is available on Amazon and other locations. <br />
<br />
</td><br />
<td><br />
<table cellspacing="10" style="width:100%;"><br />
<tr><br />
<td style="vertical-align:top;"><br />
== Hello World ==<br />
<br />
<syntaxhighlight lang="D">import std.stdio;<br />
<br />
void main()<br />
{<br />
writeln("Hello, world!");<br />
}</syntaxhighlight><br />
[http://dpaste.dzfl.pl/97ec8a2c &#9654; Run code]<br />
<br />
</td><br />
</tr><br />
<tr><br />
<td><br />
[[File:dconf_logo_2016.png|left|link=http://dconf.org/2016/index.html]]<br />
</td><br />
</tr><br />
</table><br />
</td><br />
</tr><br />
<br />
</table><br />
<br />
<table style="width:100%;"><br />
<tr><br />
<td style="vertical-align:top; width:50%;"><br />
<table cellspacing="10" style="width:100%;"><br />
<br />
<tr><br />
<td style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #e9e7d4; border-bottom: 2px solid #c9c7b4; border-top: 1px solid #fff; padding:0 1em 1em 1em; background-color: #f9f7e4; align:right;vertical-align:top;"><br />
== [[File:Icon intro 32.png|link=Why program in D]] Introduction ==<br />
* [[Why program in D|Why program in D?]]<br />
* [[Getting Started | Getting started]]<br />
* [[First Language | Learning D as a first language]] &mdash; [[Coming From | As a second language]]<br />
* [[Development With D | Development with D]]<br />
* [[Current D Use | Current uses of D]] by organizations and notable projects <br />
</td><br />
</tr><br />
<br />
<tr><br />
<td style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #c0d9e2; border-bottom: 2px solid #a0b9c2; border-top: 1px solid #fff; padding:0 1em 1em 1em; background-color: #e4f4f9; align:right;vertical-align:top;"><br />
== [[File:Icon community 32.png|link=http://forum.dlang.org]] Community ==<br />
* [http://forum.dlang.org Forums] / [news://news.digitalmars.com Newsgroup] / [http://lists.puremagic.com/mailman/listinfo Mailing lists] (Same content)<br />
* [[Jobs]]<br />
* [irc://irc.freenode.net/d IRC: Join <tt>#d</tt>] on [http://freenode.net Freenode]<br />
* [http://planet.dsource.org/ Planet D]<br />
* [http://arsdnet.net/this-week-in-d/ This Week in D]<br />
* [http://www.linkedin.com/groups/D-Developer-Network-3923820 LinkedIn: D Developer Network (DDN)]<br />
* [https://plus.google.com/communities/100033468228217743303 Google+: D Programming Enthusiasts]<br />
* [https://www.facebook.com/dlang.org Facebook: D Programming Language]<br />
* [https://www.xing.com/net/dlang XING - D Programming Language]<br />
* [[D User Groups|D User Groups (DUGs)]] — [[Events]]<br />
* [http://stackoverflow.com/questions/tagged/d D on StackOverflow]<br />
* [http://rosettacode.org/wiki/Category:D D on RosettaCode] - See solutions of common programming tasks in the D programming language.<br />
* [http://www.reddit.com/r/d_language/ reddit: /r/d_language]<br />
* [[GSOC_2015_Ideas | Google Summer of Code 2015 (GSoC)]]<br />
* [[How_You_Can_Help | How you can help with D]]<br />
</td><br />
</tr><br />
<br />
<tr><br />
<td style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #dddddd; border-bottom: 2px solid #bbb; border-top: 1px solid #fff; padding:0 1em 1em 1em; background-color:#f2f2f2; align:right;vertical-align:top;"><br />
== [[File:Icon dev 32.png|link=Get involved]] Core Development ==<br />
* [[Get involved]]<br />
* [https://trello.com/dlang Trello Board]<br />
* [[DIPs|D Improvement Proposals]] (DIPs)<br />
* [[Review Queue]]<br />
* [[Runtime internals]]<br />
* [[Vision/2015H2|High-level vision for 2015 2nd half]]<br />
* [[Beta Testing]]<br />
* [[Language_issues | Language Issues]]<br />
* [[Language_design_discussions | Language Design Discussions]]<br />
* [https://www.bountysource.com/teams/d/issues Bounties]<br />
</tr><br />
<br />
</table><br />
</td><br />
<td style="vertical-align:top; width:50%;"><br />
<table cellspacing="10" style="width:100%;"><br />
<br />
<tr><br />
<td style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #d0e2c0; border-bottom: 2px solid #b0c2a0; border-top: 1px solid #fff; padding:0 1em 1em 1em; background-color: #eaf7df; align:right;vertical-align:top;"><br />
== [[File:Icon docs 32.png|link=http://dlang.org/spec.html]] Documentation ==<br />
* [http://dlang.org/spec.html D Language Specification]<br />
* [http://dlang.org/phobos/index.html Standard Library Reference (Phobos)]<br />
* [http://ddocs.org Third-party library documentation]<br />
* [[Commonly-Used Acronyms]]<br />
* [[Books|Books (Online and printed)]]<br />
* [[Articles]] — [[Tutorials]] — [[Cookbook]]<br />
* [[ResearchPapers| Research papers]]<br />
* [[Language Designs Explained]]<br />
</td><br />
</tr><br />
<br />
<tr><br />
<td style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #e9e7d4; border-bottom: 2px solid #c9c7b4; border-top: 1px solid #fff; padding:0 1em 1em 1em; background-color: #f1f7df; align:right;vertical-align:top;"><br />
== [[File:Icon tools 32.png|link=Compilers]] Compilers &amp; Tools ==<br />
* [[Compilers | D Compilers]] ([[DMD]], [[GDC]], [[LDC]])<br />
* [[Experimental compilers]] ([[SDC]], [[DIL]])<br />
* [[IDEs]] &mdash; [[Editors | Text Editors ]]<br />
* [[Development tools]]<br />
* [[Lexers Parsers | Lexers, Parsers]]<br />
* [[Build Tools|Build tools]]<br />
* [[Debuggers]] <br />
* [[Online compilers|Online compilers and disassemblers]]<br />
</td><br />
</tr><br />
<br />
<tr><br />
<td style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #dde0e9; border-bottom: 2px solid #bdc0c9; border-top: 1px solid #fff; padding:0 1em 1em 1em; background-color: #edf0f9; align:right;vertical-align:top;"><br />
== [[File:Icon resources 32.png|link=Libraries and Frameworks]] Resources & Directory ==<br />
* [[Libraries and Frameworks]]<br />
* [[Open Source Projects]]<br />
* [[Bindings]]<br />
* [[Videos]]<br />
* [[Websites]] ([[Websites/Blogs|Blogs]])<br />
* [[People]] ([[People/Github|On Github]])<br />
* [[BenchMarks]]<br />
* [[Art]] (Logos, etc...)<br />
</td><br />
</tr><br />
<br />
<tr><br />
<td style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #eee; border-top: 2px solid #ccc; border-bottom: 1px solid #f6f6f6; padding:0 1em 1em 1em; background-color:#fff; align:right;vertical-align:top;"><br />
== [[File:Icon meta 32.png|link=Meta]] <span style="color:#999;">Meta</span> ==<br />
* [[DWiki:General discussion|Discuss this wiki]] <span style="color:#999;">(Criticise, suggest improvements etc.)</span><br />
* [[DWiki:Community portal|DWiki Community Portal]]<br />
</td><br />
</tr><br />
<br />
</table><br />
</td><br />
</tr><br />
</table><br />
<br />
{{:DMD Widget}}</div>Schuetzmhttps://wiki.dlang.org/?title=File:Dconf_logo_2016.png&diff=6996File:Dconf logo 2016.png2015-11-27T12:27:25Z<p>Schuetzm: </p>
<hr />
<div></div>Schuetzmhttps://wiki.dlang.org/?title=DIP85&diff=6958DIP852015-11-14T15:03:15Z<p>Schuetzm: </p>
<hr />
<div>{| class="wikitable"<br />
!Title:<br />
!'''Lazy Initialization of const Members'''<br />
|-<br />
|DIP:<br />
|85<br />
|-<br />
|Version:<br />
|2<br />
|-<br />
|Status:<br />
|Draft<br />
|-<br />
|Created:<br />
|2015-11-14<br />
|-<br />
|Last Modified:<br />
|{{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY}}<br />
|-<br />
|Author:<br />
|Marc Schütz<br />
|-<br />
|Links:<br />
|[http://forum.dlang.org/post/jverkpodicspfopunhkw@forum.dlang.org Discussion in the forum]<br />
|}<br />
<br />
== Abstract ==<br />
This DIP proposes an officially sanctioned way to initialize const members (or mutable members of const structs) lazily by allowing limited mutation of const objects.<br />
<br />
== Rationale ==<br />
Lazy initialization is a widely applied technique to 1) defer initialization of a struct or class member until it is actually needed, and 2) cache and reuse the result of an expensive computation. With current const semantics in D, this require objects and structs containing such variables to be mutable. As a consequence, any method that wants to make use of a lazily calculated member cannot be annotated as const, and all functions calling these methods require non-const references to such objects in turn.<br />
<br />
== Description ==<br />
A new annotation of member variables is proposed, reusing the existing keyword <code>lazy</code>. A member annotated as <code>lazy</code> triggers the following behaviours:<br />
<br />
* it is required to be private<br />
* no static immutable objects with a lazy member may be created<br />
* dynamically created immutable objects with lazy members are allowed if all lazy members are marked as <code>shared</code><br />
* it can be assigned to even if it is const, but only if it has been read at most once<br />
<br />
The first three rules are enforced statically. The last one is checked by an <code>assert()</code> at runtime using a hidden flag member; in <code>-release</code> mode, this check (including the hidden member) is removed.<br />
<br />
The second rule is necessary because static immutable objects could be placed in physically read-only memory by the linker and therefore cannot be modified.<br />
<br />
The third rule prevents race conditions for implicitly shared immutable objects. Access to shared lazy members must be atomic or otherwise synchronized.<br />
<br />
The last rule ensures that external code can never observe a change to the field's value.<br />
<br />
The compiler needs to make sure not to apply optimizations based on the assumption that a <code>lazy</code> member never changes. Use of <code>lazy</code> in this way is therefore <code>@safe</code>. It is, however, limited to values that can be written to the member directly. More complex applications, e.g. memoization depending on parameters using an associative array, or reference counting (which requires more than one mutation) can be implemented by casting the const-ness away. The compiler will still make sure that no breaking optimization are applied, but it can no longer enforce correctness, which makes casting away const un-@safe.<br />
<br />
== Usage ==<br />
<source lang="D"><br />
import std.stdio;<br />
class MyClass {<br />
void hello() { writeln("Hello, world!"); }<br />
}<br />
struct S {<br />
// reference types<br />
lazy private MyClass foo_;<br />
@safe @property foo() const {<br />
if(!foo_)<br />
foo_ = new MyClass;<br />
return foo_;<br />
}<br />
// value types<br />
lazy private int bar_;<br />
@safe @property bar() const {<br />
if(!bar_)<br />
bar_ = expensiveComputation();<br />
return bar_;<br />
}<br />
// complex<br />
lazy uint[uint] factorial_;<br />
@trusted factorial(uint x) const pure {<br />
if(factorial_ is null)<br />
factorial_ = createEmptyAA!(uint[uint]); // initialize once<br />
auto tmp = cast(uint[uint]) factorial_; // cast to mutable<br />
if(auto found = x in tmp)<br />
return *found;<br />
auto result = computeFactorial(x);<br />
tmp[x] = result;<br />
return result;<br />
}<br />
}<br />
<br />
void main() {<br />
const S s;<br />
s.foo.hello();<br />
writeln("bar = ", s.bar);<br />
writeln("5! = " s.factorial(5));<br />
writeln("6! = " s.factorial(6));<br />
}<br />
</source><br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=DIP85&diff=6957DIP852015-11-14T14:31:54Z<p>Schuetzm: </p>
<hr />
<div>{| class="wikitable"<br />
!Title:<br />
!'''Lazy Initialization of const Members'''<br />
|-<br />
|DIP:<br />
|85<br />
|-<br />
|Version:<br />
|1<br />
|-<br />
|Status:<br />
|Draft<br />
|-<br />
|Created:<br />
|2015-11-14<br />
|-<br />
|Last Modified:<br />
|{{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY}}<br />
|-<br />
|Author:<br />
|Marc Schütz<br />
|-<br />
|Links:<br />
|[http://forum.dlang.org/post/jverkpodicspfopunhkw@forum.dlang.org Discussion in the forum]<br />
|}<br />
<br />
== Abstract ==<br />
This DIP proposes an officially sanctioned way to initialize const members (or mutable members of const structs) lazily by allowing limited mutation of const objects.<br />
<br />
== Rationale ==<br />
Lazy initialization is a widely applied technique to 1) defer initialization of a struct or class member until it is actually needed, and 2) cache and reuse the result of an expensive computation. With current const semantics in D, this require objects and structs containing such variables to be mutable. As a consequence, any method that wants to make use of a lazily calculated member cannot be annotated as const, and all functions calling these methods require non-const references to such objects in turn.<br />
<br />
== Description ==<br />
A new annotation of member variables is proposed, reusing the existing keyword <code>lazy</code>. A member annotated as <code>lazy</code> triggers the following behaviours:<br />
<br />
* it is required to be private<br />
* no immutable objects with a lazy member may be created<br />
* it can be assigned to even if it is const, but only if it has been read at most once<br />
<br />
The first two rules are enforced statically. The last one is checked by an <code>assert()</code> at runtime using a hidden flag member; in <code>-release</code> mode, this check (including the hidden member) is removed.<br />
<br />
The second rule is necessary because static immutable objects could be placed in physically read-only memory by the linker and therefore cannot be modified, while dynamically created immutable objects can be shared across threads and modifications would be prone to race conditions.<br />
<br />
The last rule ensures that external code can never observe a change to the field's value.<br />
<br />
The compiler needs to make sure not to apply optimizations based on the assumption that a <code>lazy</code> member never changes. Use of <code>lazy</code> in this way is therefore <code>@safe</code>. It is, however, limited to values that can be written to the member directly. More complex applications, e.g. memoization depending on parameters using an associative array, can be implemented by casting the const-ness away. The compiler will still make sure that no breaking optimization are applied, but it can no longer enforce correctness, which makes casting away const un-@safe.<br />
<br />
== Usage ==<br />
<source lang="D"><br />
import std.stdio;<br />
class MyClass {<br />
void hello() { writeln("Hello, world!"); }<br />
}<br />
struct S {<br />
// reference types<br />
lazy private MyClass foo_;<br />
@safe @property foo() const {<br />
if(!foo_)<br />
foo_ = new MyClass;<br />
return foo_;<br />
}<br />
// value types<br />
lazy private int bar_;<br />
@safe @property bar() const {<br />
if(!bar_)<br />
bar_ = expensiveComputation();<br />
return bar_;<br />
}<br />
// complex<br />
lazy uint[uint] factorial_;<br />
@trusted factorial(uint x) const pure {<br />
if(factorial_ is null)<br />
factorial_ = createEmptyAA!(uint[uint]); // initialize once<br />
auto tmp = cast(uint[uint]) factorial_; // cast to mutable<br />
if(auto found = x in tmp)<br />
return *found;<br />
auto result = computeFactorial(x);<br />
tmp[x] = result;<br />
return result;<br />
}<br />
}<br />
<br />
void main() {<br />
const S s;<br />
s.foo.hello();<br />
writeln("bar = ", s.bar);<br />
writeln("5! = " s.factorial(5));<br />
writeln("6! = " s.factorial(6));<br />
}<br />
</source><br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=DIPs&diff=6956DIPs2015-11-14T14:22:20Z<p>Schuetzm: </p>
<hr />
<div>{| class="wikitable sortable"<br />
! ID<br />
! Title<br />
! Status<br />
! Description<br />
|-<br />
| [[DIP1]]<br />
| Template DIP<br />
| Draft<br />
| This is a DIP template that can be used as a model to start a new proposal.<br />
|-<br />
| [[DIP2]]<br />
| Const code bloat<br />
| '''Approved'''<br />
| proposed solution for [http://d.puremagic.com/issues/show_bug.cgi?id=1961 bug 1961].<br />
|-<br />
| [[DIP3]]<br />
| Remove inheritance protection<br />
| '''Approved'''<br />
| non-public inheritance in a single inheritance language that has a single root object hierarchy makes little sense.<br />
|-<br />
| [[DIP4]]<br />
| Properties<br />
| [[DIP6|Superseded by DIP6]]<br />
| an alternate usage/definition syntax for properties.<br />
|-<br />
| [[DIP5]]<br />
| Properties 2<br />
| [[DIP6|Superseded by DIP6]]<br />
| a variant of DIP4.<br />
|-<br />
| [[DIP6]]<br />
| Annotations<br />
| '''Approved'''<br />
| extend the D programming language with annotations.<br />
|-<br />
| [[DIP7]]<br />
| Operator overloading<br />
| '''Approved'''<br />
| Revamped operator overloading.<br />
|-<br />
| [[DIP8]]<br />
| Improving Runtime Type Info<br />
| Draft<br />
| Templating RTTI.<br />
|-<br />
| [[DIP9]]<br />
| Redo toString API<br />
| Draft<br />
| Replace inefficient aggregate toString API with efficient delegate-based version that supports formatting.<br />
|-<br />
| [[DIP10]]<br />
| Qualified constructors and destructors for structs<br />
| Draft<br />
|<br />
|-<br />
| [[DIP11]]<br />
| Automatic downloading of imports<br />
| Draft<br />
| Support automatically downloading imported files via new import path specifications or pragmas.<br />
|-<br />
| [[DIP12]]<br />
| C API Headers<br />
| '''Approved'''<br />
| Collection of D headers for common C libraries.<br />
|-<br />
| [[DIP13]]<br />
| Import path binding<br />
| Draft<br />
| Allow to bind an import path to a package.<br />
|-<br />
| [[DIP14]]<br />
| Import path binding in source code<br />
| Draft<br />
| Allow to bind an import path to a package from within sources.<br />
|-<br />
| [[DIP15]]<br />
| Import of packages<br />
| [[DIP37|Superseded by DIP37]]<br />
| Improve imports of packages with submodules.<br />
|-<br />
| [[DIP16]]<br />
| Transparently substitute module with package<br />
| [[DIP37|Superseded by DIP37]]<br />
|<br />
|-<br />
| [[DIP17]]<br />
| Sane volatile statement<br />
| [[DIP20|Superseded by DIP20]]<br />
|<br />
|-<br />
| [[DIP18]]<br />
| Non-GC Threads<br />
| Draft<br />
|<br />
|-<br />
| [[DIP20]]<br />
| Volatile read/write intrinsics<br />
| '''Implemented'''<br />
| Intrinsics to perform C-style volatile loads/stores from/to memory.<br />
|-<br />
| [[DIP21]]<br />
| Fixing property<br />
| Draft<br />
|<br />
|-<br />
| [[DIP22]]<br />
| Private symbol visibility<br />
| Draft<br />
| Modification of private & Co symbol rules to reduce name clashes. Add internal linkage storage class.<br />
|-<br />
| [[DIP23]]<br />
| Fixing property redux<br />
| Draft<br />
| Initial<br />
|-<br />
| [[DIP24]]<br />
| Fixing properties and optional parens.<br />
| Draft<br />
| Fixing properties and optional parens. Only those. In a general and straightforward way. (counter proposal to DIP23)<br />
|-<br />
| [[DIP25]]<br />
| Sealed references<br />
| '''Implemented 2.067'''<br />
|<br />
|-<br />
| [[DIP26]]<br />
| Properties the other way round<br />
| Draft<br />
|<br />
|-<br />
| [[DIP27]]<br />
| Function definition<br />
| Draft<br />
|<br />
|-<br />
| [[DIP28]]<br />
| Property definition<br />
| Draft<br />
|<br />
|-<br />
| [[DIP29]]<br />
| Unique pointers<br />
| Draft<br />
| Unique pointers can be implicitly cast to/from immutable and shared.<br />
|-<br />
| [[DIP30]]<br />
| Delegate definition<br />
| Draft<br />
|<br />
|-<br />
| [[DIP31]]<br />
| Defined compile time paradox resolution<br />
| Draft<br />
|<br />
|-<br />
| [[DIP32]]<br />
| Uniform tuple syntax<br />
| Draft<br />
|<br />
|-<br />
| [[DIP33]]<br />
| A standard exception hierarchy<br />
| Draft<br />
|<br />
|-<br />
| [[DIP34]]<br />
| Static array literals<br />
| Draft<br />
|<br />
|-<br />
| [[DIP35]]<br />
| [[DIP25|Sealed References]] Amendment<br />
| Draft<br />
|<br />
|-<br />
| [[DIP36]]<br />
| Rvalue References<br />
| Rejected<br />
|<br />
|-<br />
| [[DIP37]]<br />
| Importing Packages as if They Were Modules<br />
| '''Implemented'''<br />
| Allow for packages to be imported as if they were modules by providing a ''package.d'' file.<br />
|<br />
|-<br />
| [[DIP38]]<br />
| Safe references without runtime checks<br />
| Draft<br />
|<br />
|-<br />
| [[DIP39]]<br />
| Safe rvalue references: backwards compatible, safe against ref/nonref code evolution, compatible with UFCS and DIP38.<br />
| Draft<br />
|<br />
|-<br />
| [[DIP40]]<br />
| Template parameter deduction for constructors.<br />
| Draft<br />
|<br />
|-<br />
| [[DIP41]]<br />
| dmd/rdmd command line overhaul.<br />
| Draft<br />
|<br />
|-<br />
| [[DIP42]]<br />
| Support enum E(T) = expression; for manifest constants<br />
| '''Implemented'''<br />
|<br />
|-<br />
| [[DIP43]]<br />
| D/Objective-C<br />
| Draft<br />
| Adds language extensions to make D ABI compatible with Objective-C.<br />
|-<br />
| [[DIP44]]<br />
| scope(this)<br />
| Draft<br />
| Extends scope guards to class / struct lifetime.<br />
|-<br />
| [[DIP45]]<br />
| making export an attribute<br />
| Draft<br />
| Turn export into an attribute and fix various shortcomings of its current implementation.<br />
|-<br />
| [[DIP46]]<br />
| region based storage allocation<br />
| Draft<br />
| Library addition to GC to allow for region based allocation strategies<br />
|-<br />
| [[DIP47]]<br />
| Outline Member Functions of Aggregates<br />
| Draft<br />
| Allow class/struct member function definitions to be placed outside of the class/struct declaration.<br />
|-<br />
| [[DIP48]]<br />
| Interface specifications for aggregate types<br />
| Draft<br />
|<br />
|-<br />
| [[DIP49]]<br />
| Define qualified postblit<br />
| Draft<br />
| Fix D2 type-system hole.<br />
|-<br />
| [[DIP50]]<br />
| AST Macros<br />
| Draft<br />
| Abstract Syntax Tree macros. A general solution for language extensions.<br />
|-<br />
| [[DIP51]]<br />
| not-virtual-by-default<br />
| Draft<br />
| Change the behaviour of class methods so they are not virtual by default.<br />
|-<br />
| [[DIP52]]<br />
| Implicit conversions<br />
| Draft<br />
| Add more implicit conversion options for user-defined types.<br />
|-<br />
| [[DIP53]]<br />
| Qualified constructor revisited<br />
| Draft<br />
| Make qualified constructor definition more simple and understandable.<br />
|-<br />
| [[DIP54]]<br />
| Revamp of Phobos tuple types<br />
| Draft<br />
| Address frequent informational confusion around std.typetuple & friends, initiate transition to std.meta.* package<br />
|-<br />
| [[DIP55]]<br />
| Access Data Items In Ancestor Stack Frames<br />
| Draft<br />
| Support for *caller pointer in functions and adding class features to functions<br />
|-<br />
| [[DIP56]]<br />
| Provide Pragma to control function inlining<br />
| Draft<br />
| Add pragma(inline,true); and pragma(inline,false);<br />
|-<br />
| [[DIP57]]<br />
| static foreach<br />
| Draft<br />
|<br />
|-<br />
| [[DIP58]]<br />
| Make ".." a binary operator<br />
| Draft<br />
|<br />
|-<br />
| [[DIP59]]<br />
| ??<br />
| Draft<br />
|<br />
|-<br />
| [[DIP60]]<br />
| Add @nogc function attribute<br />
| '''Implemented'''<br />
|<br />
|-<br />
| [[DIP61]]<br />
| Add namespace scopes to support calling external C++ functions in C++ namespaces<br />
| '''Implemented'''<br />
|<br />
|-<br />
| [[DIP62]]<br />
| Volatile type qualifier for unoptimizable variables in embedded programming<br />
| '''Rejected'''<br />
|<br />
|-<br />
| [[DIP63]]<br />
| Operator overloading for raw templates<br />
| Draft<br />
|<br />
|-<br />
| [[DIP64]]<br />
| Attribute Cleanup<br />
| Draft<br />
|<br />
|-<br />
| [[DIP65]]<br />
| Exception handling syntax cleanup<br />
| '''Rejected'''<br />
|<br />
|-<br />
| [[DIP66]]<br />
| (Multiple) alias this<br />
| Conditionally Approved<br />
|<br />
|-<br />
| [[DIP67]]<br />
| Associative Ranges<br />
| '''Rejected'''<br />
|<br />
|-<br />
| [[DIP68]]<br />
| Add @nogc attribute on types<br />
| Draft<br />
|<br />
|-<br />
| [[DIP69]]<br />
| Implement scope for escape proof references<br />
| Draft<br />
|<br />
|-<br />
| [[DIP70]]<br />
| @api/extern(noinfer) attribute<br />
| Draft<br />
|<br />
|-<br />
| [[DIP71]]<br />
| 'noscope' and 'out!param' attributes<br />
| Draft<br />
|<br />
|-<br />
| [[DIP72]]<br />
| Provide a Uniform Compiler Info Syntax<br />
| Draft<br />
|<br />
|-<br />
| [[DIP73]]<br />
| D Drafting Library<br />
| Draft<br />
|<br />
|-<br />
| [[DIP74]]<br />
| Reference Counted Class Objects<br />
| Draft<br />
|<br />
|-<br />
| [[DIP75]]<br />
| Release Process<br />
| Approved experimentally for 2.068<br />
|<br />
|-<br />
| [[DIP76]]<br />
| Autodecode should not throw<br />
| Draft<br />
| Like std.utf.byDchar, it should emit Replacement Character instead<br />
|<br />
|-<br />
| [[DIP77]]<br />
| Fix Unsafe RC Pass By Ref<br />
| Draft<br />
| Fix memory safety issue in DIP25<br />
|-<br />
| [[DIP78]]<br />
| AST Macros Lite<br />
| Draft<br />
| Abstract Syntax Tree macros without syntax extensions in the language.<br />
|-<br />
| [[DIP79]]<br />
| Negation of attributes<br />
| Draft<br />
| Negation of attributes without the need for new keywords.<br />
|-<br />
| [[DIP80]]<br />
| Phobos additions<br />
| Draft<br />
| Additions that would make phobos even more useful.<br />
|-<br />
| [[DIP81]]<br />
| Writing files at compile time<br />
| Draft<br />
|-<br />
| [[DIP82]]<br />
| static unittest blocks<br />
| Draft<br />
| Allow for unittest blocks to be declared inside of templates which are compiled and run exactly once rather than once per instantiation.<br />
|-<br />
| [[DIP83]]<br />
| Configurable Assert Diagnostics<br />
| Draft<br />
| Allow for assert's to do pretty printing of its failing expression. Printing is configurable via specific sets of (template) function overloads.<br />
|-<br />
| [[DIP84]]<br />
| Static Inheritance<br />
| Draft<br />
| Express intent to the compiler that a user-defined type conforms to a static interface.<br />
|-<br />
| [[DIP85]]<br />
| Lazy initialization of const members<br />
| Draft<br />
| Allow limited mutation of const objects, ensuring the mutation is not observable by external code.<br />
|}<br />
<br />
----<br />
[[Category:DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=DIP85&diff=6955DIP852015-11-14T14:20:41Z<p>Schuetzm: Created page with "{| class="wikitable" !Title: !'''Layz Initialization of const Members''' |- |DIP: |85 |- |Version: |1 |- |Status: |Draft |- |Created: |2015-11-14 |- |Last Modified: |{{REVISIO..."</p>
<hr />
<div>{| class="wikitable"<br />
!Title:<br />
!'''Layz Initialization of const Members'''<br />
|-<br />
|DIP:<br />
|85<br />
|-<br />
|Version:<br />
|1<br />
|-<br />
|Status:<br />
|Draft<br />
|-<br />
|Created:<br />
|2015-11-14<br />
|-<br />
|Last Modified:<br />
|{{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY}}<br />
|-<br />
|Author:<br />
|Marc Schütz<br />
|-<br />
|Links:<br />
|<br />
|}<br />
<br />
== Abstract ==<br />
This DIP proposes an officially sanctioned way to initialize const members (or mutable members of const structs) lazily.<br />
<br />
== Rationale ==<br />
Lazy initialization is a widely applied technique to 1) defer initialization of a struct or class member until it is actually needed, and 2) cache and reuse the result of an expensive computation. With current const semantics in D, this require objects and structs containing such variables to be mutable. As a consequence, any method that wants to make use of a lazily calculated member cannot be annotated as const, and all functions calling these methods require non-const references to such objects in turn.<br />
<br />
== Description ==<br />
A new annotation of member variables is proposed, reusing the existing keyword <code>lazy</code>. A member annotated as <code>lazy</code> triggers the following behaviours:<br />
<br />
* it is required to be private<br />
* no immutable objects with a lazy member may be created<br />
* it can be assigned to even if it is const, but only if it has been read at most once<br />
<br />
The first two rules are enforced statically. The last one is checked by an <code>assert()</code> at runtime using a hidden flag member; in <code>-release</code> mode, this check (including the hidden member) is removed.<br />
<br />
The second rule is necessary because static immutable objects could be placed in physically read-only memory by the linker and therefore cannot be modified, while dynamically created immutable objects can be shared across threads and modifications would be prone to race conditions.<br />
<br />
The last rule ensures that external code can never observe a change to the field's value.<br />
<br />
The compiler needs to make sure not to apply optimizations based on the assumption that a <code>lazy</code> member never changes. Use of <code>lazy</code> in this way is therefore <code>@safe</code>. It is, however, limited to values that can be written to the member directly. More complex applications, e.g. memoization depending on parameters using an associative array, can be implemented by casting the const-ness away. The compiler will still make sure that no breaking optimization are applied, but it can no longer enforce correctness, which makes casting away const un-@safe.<br />
<br />
== Usage ==<br />
<source lang="D"><br />
import std.stdio;<br />
class MyClass {<br />
void hello() { writeln("Hello, world!"); }<br />
}<br />
struct S {<br />
// reference types<br />
lazy private MyClass foo_;<br />
@safe @property foo() const {<br />
if(!foo_)<br />
foo_ = new MyClass;<br />
return foo_;<br />
}<br />
// value types<br />
lazy private int bar_;<br />
@safe @property bar() const {<br />
if(!bar_)<br />
bar_ = expensiveComputation();<br />
return bar_;<br />
}<br />
// complex<br />
lazy uint[uint] factorial_;<br />
@trusted factorial(uint x) const pure {<br />
if(factorial_ is null)<br />
factorial_ = createEmptyAA!(uint[uint]); // initialize once<br />
auto tmp = cast(uint[uint]) factorial_; // cast to mutable<br />
if(auto found = x in tmp)<br />
return *found;<br />
auto result = computeFactorial(x);<br />
tmp[x] = result;<br />
return result;<br />
}<br />
}<br />
<br />
void main() {<br />
const S s;<br />
s.foo.hello();<br />
writeln("bar = ", s.bar);<br />
writeln("5! = " s.factorial(5));<br />
writeln("6! = " s.factorial(6));<br />
}<br />
</source><br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=Compilers&diff=6685Compilers2015-09-27T10:48:25Z<p>Schuetzm: /* Package and/or binary availability, by platform and compiler */</p>
<hr />
<div>__NOTOC__<br />
{{:Compilers_Widget}}<br />
<table style="width:100%;" cellspacing="10"><br />
<tr><br />
<td style="width:33%; vertical-align: top; padding: 10px 10px; background: #f9f9f9; border: 1px solid #ccc; border-top: 1px solid #eee; border-bottom: 2px solid #999;"><br />
= [[DMD|DMD &raquo;]] =<br />
'''Digital Mars D compiler''' <br/>The formal D compiler.<br />
<div style="text-align: right;">[[DMD|DMD Portal &raquo;]]</div><br />
</td><br />
<br />
<td style="width:33%; vertical-align: top; padding: 10px 10px; background: #f9f9f9; border: 1px solid #ccc; border-top: 1px solid #eee; border-bottom: 2px solid #999;"><br />
<br />
<br />
<br />
[[Category:Compiler]]<br />
<br />
= [[GDC|GDC &raquo;]] =<br />
'''GCC D compiler'''<br/><br />
Fast and open source.<br />
<div style="text-align: right;">[[GDC|GDC Portal &raquo;]]</div><br />
</td><br />
<br />
<td style="width:33%; vertical-align: top; padding: 10px 10px; background: #f9f9f9; border: 1px solid #ccc; border-top: 1px solid #eee; border-bottom: 2px solid #999;"><br />
= [[LDC|LDC &raquo;]] =<br />
'''LLVM D compiler'''<br/><br />
Fast and open source.<br />
<div style="text-align: right;">[[LDC|LDC Portal &raquo;]]</div><br />
</td><br />
<br />
</tr><br />
</table><br />
<br />
<br />
Also see: [[Experimental compilers]]<br />
<br />
<br />
== Which compiler should I use? ==<br />
If you're a beginner DMD is the recommended choice, as it is the implementation closest to the D Language Specification. Otherwise it depends on what you need, what platforms you intend to develop for, and your personal preferences. GDC and LDC both generate substantially faster binaries than DMD.<br />
<br />
== Comparison ==<br />
<br />
{| class="wikitable" style="width: 65em"<br />
!<br />
!DMD<br />
!GDC<br />
!LDC<br />
|---<br />
|'''Platforms'''<br />
|<br />
*Windows<br />
*Linux<br />
*OS X<br />
*FreeBSD<br />
|<br />
*Windows (alpha)<br />
*Linux<br />
*OS X (untested)<br />
*FreeBSD (untested)<br />
|<br />
*Windows<br />
*Linux<br />
*OS X<br />
*FreeBSD<br />
*OpenSolaris<br />
*iOS (experimental)<br />
|---<br />
<br />
|'''Architectures'''<br />
|<br />
* i386<br />
* amd64<br />
|<br />
Complete (runtime / standard library) support:<br />
* i386<br />
* amd64<br />
* x32<br />
* armel<br />
* armhf<br />
<br />
Partial or bare-metal only support (packages for gdc in [https://packages.debian.org/sid/gdc-5 debian]):<br />
* alpha<br />
* arm64 (aarch64)<br />
* hppa<br />
* hurd-i386<br />
* kfreebsd-amd64<br />
* kfreebsd-i386<br />
* m68k<br />
* mips<br />
* mipsel<br />
* ppc<br />
* pcc64<br />
* ppc64el<br />
* s390x<br />
* sparc64<br />
|<br />
Complete (runtime / standard library) support:<br />
* i386<br />
* amd64<br />
<br />
Near-complete support:<br />
* arm64 (aarch64)<br />
* ppc<br />
* ppc64<br />
* ppc64el<br />
* mips64<br />
<br />
Partial or bare-metal only support:<br />
* armel<br />
* armhf<br />
* mips<br />
* s390x<br />
|---<br />
|'''Distribution'''<br />
|<br />
*Source<br />
*Multi-platform source/binary archive<br />
*Multi-platform installer ([https://github.com/jacob-carlborg/dvm DVM])<br />
*Windows installer<br />
*OS X package (.dmg)<br />
*Debian/Ubuntu package (.deb)<br />
*Fedora package (.rpm)<br />
*OpenSUSE (.rpm) package<br />
*Debian/Ubuntu repository via http://d-apt.sourceforge.net<br />
*OS-X homebrew and macports repositories<br />
|<br />
*Source<br />
*Windows / Linux binary archive<br />
*Debian/Ubuntu repository<br />
*Gentoo repository<br />
*Archlinux repository<br />
|<br />
*Source<br />
*Linux / OS X binary archive<br />
*Debian/Ubuntu repository<br />
*Fedora repository<br />
*Gentoo repository<br />
|---<br />
|'''Backend'''<br />
| [https://github.com/D-Programming-Language/dmd/tree/master/src/backend DMD] ([http://digitalmars.com/download/freecompiler.html DMC] fork)<br />
| [http://gcc.gnu.org/ GCC]<br />
| [http://llvm.org/ LLVM]<br />
|---<br />
|'''License'''<br />
| Frontend: GPL 1 or later / Artistic License<br />
Backend: [https://github.com/D-Programming-Language/dmd/blob/master/src/backendlicense.txt custom]<br />
| GPL 3 or later<br />
| LDC-specific code: 3-clause BSD<br />
|---<br />
|'''Inline assembler'''<br />
|<br />
* DMD Intel-like syntax (i386/amd64)<br />
|<br />
* GCC syntax (all targets)<br />
|<br />
* DMD Intel-like syntax (i386/amd64)<br />
* GCC syntax (all targets)<br />
* [[LDC inline IR|LLVM inline IR]]<br />
|---<br />
|'''SIMD'''<br />
| Partial (?)<br />
| Partial (?)<br />
| Partial (?)<br />
|---<br />
|'''Phobos as a shared library'''<br />
|<br />
* Linux<br />
|<br />
|<br />
* Linux<br />
|---<br />
|'''Building D code as shared library'''<br />
|<br />
* Linux<br />
* Windows ?<br />
|<br />
|<br />
* Linux<br />
|---<br />
|'''Dynamic loading of D shared libraries'''<br />
|<br />
* Linux<br />
|<br />
|<br />
* Linux<br />
|---<br />
!colspan="4" align="center"|<br/>'''Linux specific'''<br />
|---<br />
|'''Object file format'''<br />
| ELF<br />
| ELF<br />
| ELF<br />
|---<br />
!colspan="4" align="center"|<br/>'''Mac specific'''<br />
|---<br />
|'''Object file format'''<br />
| Mach-O<br />
| Mach-O<br />
| Mach-O<br />
|---<br />
!colspan="4" align="center"|<br/>'''Windows specific'''<br />
|---<br />
|'''Object file format'''<br />
| OMF (32) / COFF (64)<br />
| COFF<br />
| COFF<br />
|}<br />
<br />
==Package and/or binary availability, by platform and compiler==<br />
<br />
Version number is given in brackets. Many of the manual download links have 10s of versions available, only the most recent few are listed here.<br />
<br />
Some unofficial repositories and downloads are listed here, but of course many more do exist. With a little searching, you may be able to find something more up to date for your chosen OS.<br />
<br />
Very old compilers are (mostly) omitted, as they are unlikely to be of interest to users.<br />
<br />
{| class="wikitable" style="width: 65em"<br />
!rowspan="2"|<br/>Platform<br />
!colspan="3" align="center"|Compiler<br />
|---<br />
!DMD<br />
!GDC<br />
!LDC<br />
|---<br />
!Windows<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
*[https://chocolatey.org Chocolatey] (2.067.0, 2.068.1 available awaiting moderation)<br />
|<br />
*[http://gdcproject.org/downloads Manual download] (5.2.0 w/ dmd frontend 2.066.1)<br />
|<br />
*[https://github.com/ldc-developers/ldc/releases Manual download] (0.15.1, 0.15.2-beta2, 0.16.0-beta1)<br />
|---<br />
!OS X<br />
|<br />
*[http://brew.sh Homebrew] (2.068.2)<br />
*[https://www.macports.org MacPorts] (2.068.0)<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
|<br />
|<br />
*[http://brew.sh Homebrew] (0.15.2-beta2, 0.16.0-alpha4)<br />
*[https://github.com/ldc-developers/ldc/releases Manual download] (0.15.1, 0.15.2-beta2, 0.16.0-beta1)<br />
|---<br />
!Linux (generic)<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
|<br />
*[http://gdcproject.org/downloads Manual download] (5.2.0 w/ dmd frontend 2.066.1)<br />
|<br />
*[https://github.com/ldc-developers/ldc/releases Manual download] (0.15.1, 0.15.2-beta2, 0.16.0-beta1)<br />
|---<br />
!Cross-platform<br />
|<br />
* [https://github.com/jacob-carlborg/dvm DVM] (Any version)<br />
|<br />
|<br />
|---<br />
!colspan="4" | Distribution-specific packages<br />
|---<br />
!Debian<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
*[http://d-apt.sourceforge.net APT repository] (2.068.0)<br />
|<br />
*Stable ([https://packages.debian.org/jessie/gdc-4.8 4.8.4], [https://packages.debian.org/jessie/gdc-4.9 4.9.2])<br />
*Testing (([https://packages.debian.org/stretch/gdc-4.8 4.8.5], [https://packages.debian.org/stretch/gdc-4.9 4.9.3], [https://packages.debian.org/stretch/gdc-5 5.2.1]))<br />
*Unstable ([https://packages.debian.org/sid/gdc-4.8 4.8.5], [https://packages.debian.org/sid/gdc-4.9 4.9.3], [https://packages.debian.org/sid/gdc-5 5.2.1])<br />
|<br />
*Stable/Testing/Unstable (0.14.0)<br />
*Experimental (0.15.1)<br />
|---<br />
!Ubuntu<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
*[http://d-apt.sourceforge.net APT repository] (2.068.0)<br />
|<br />
*14.04 ([http://packages.ubuntu.com/trusty-updates/gdc-4.8 4.8.4])<br />
*15.04 ([http://packages.ubuntu.com/vivid/gdc-4.8 4.8.4], [http://packages.ubuntu.com/vivid/gdc-4.9 4.9.2])<br />
*15.10 ([http://packages.ubuntu.com/wily/gdc-4.8 4.8.5], [http://packages.ubuntu.com/wily/gdc-4.9 4.9.3], [http://packages.ubuntu.com/wily/gdc-5 5.2.1])<br />
|<br />
*15.04 ([http://packages.ubuntu.com/vivid/ldc 0.14.0])<br />
*15.10 ([http://packages.ubuntu.com/wily/ldc 0.14.0])<br />
|---<br />
!Fedora<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
|<br />
|<br />
See https://apps.fedoraproject.org/packages/ldc<br />
*21 (0.13.0)<br />
*22 (0.15.1)<br />
*23 (0.15.2.beta1)<br />
*Rawhide (0.16.0.alpha3)<br />
|---<br />
!OpenSuse<br />
|<br />
*[https://build.opensuse.org/package/show/devel:languages:D/dmd devel:languages:D] (2.068.2)<br />
|<br />
|<br />
*[https://build.opensuse.org/project/show/devel:languages:D devel:languages:D] (0.15.1)<br />
*[https://build.opensuse.org/project/show/home:marc_schuetz:branches:devel:languages:D home:marc_schuetz] (0.15.2-beta2)<br />
|---<br />
!CentOS<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
|<br />
|<br />
|---<br />
!Arch Linux<br />
|<br />
*Community (2.068.2)<br />
|<br />
*Community (5.2.0)<br />
*AUR ([https://aur.archlinux.org/packages/gdc-git/ git HEAD])<br />
|<br />
*Community (0.15.1)<br />
|---<br />
!Gentoo<br />
|colspan="3" align=center | see https://wiki.gentoo.org/wiki/Dlang<br />
|---<br />
!FreeBSD<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.2)<br />
*Ports (2.067.0)<br />
|<br />
|<br />
|}<br />
<br />
<br />
[[Category:Compiler]]</div>Schuetzmhttps://wiki.dlang.org/?title=Compilers&diff=6421Compilers2015-08-11T09:39:45Z<p>Schuetzm: 2.068.0 on openSUSE</p>
<hr />
<div>__NOTOC__<br />
{{:Compilers_Widget}}<br />
<table style="width:100%;" cellspacing="10"><br />
<tr><br />
<td style="width:33%; vertical-align: top; padding: 10px 10px; background: #f9f9f9; border: 1px solid #ccc; border-top: 1px solid #eee; border-bottom: 2px solid #999;"><br />
= [[DMD|DMD &raquo;]] =<br />
'''Digital Mars D compiler''' <br/>The formal D compiler.<br />
<div style="text-align: right;">[[DMD|DMD Portal &raquo;]]</div><br />
</td><br />
<br />
<td style="width:33%; vertical-align: top; padding: 10px 10px; background: #f9f9f9; border: 1px solid #ccc; border-top: 1px solid #eee; border-bottom: 2px solid #999;"><br />
<br />
<br />
<br />
[[Category:Compiler]]<br />
<br />
= [[GDC|GDC &raquo;]] =<br />
'''GCC D compiler'''<br/><br />
Fast and open source.<br />
<div style="text-align: right;">[[GDC|GDC Portal &raquo;]]</div><br />
</td><br />
<br />
<td style="width:33%; vertical-align: top; padding: 10px 10px; background: #f9f9f9; border: 1px solid #ccc; border-top: 1px solid #eee; border-bottom: 2px solid #999;"><br />
= [[LDC|LDC &raquo;]] =<br />
'''LLVM D compiler'''<br/><br />
Fast and open source.<br />
<div style="text-align: right;">[[LDC|LDC Portal &raquo;]]</div><br />
</td><br />
<br />
</tr><br />
</table><br />
<br />
<br />
Also see: [[Experimental compilers]]<br />
<br />
<br />
== Which compiler should I use? ==<br />
If you're a beginner DMD is the recommended choice, as it is the implementation closest to the D Language Specification. Otherwise it depends on what you need, what platforms you intend to develop for, and your personal preferences. GDC and LDC both generate substantially faster binaries than DMD.<br />
<br />
== Comparison ==<br />
<br />
{| class="wikitable" style="width: 65em"<br />
!<br />
!DMD<br />
!GDC<br />
!LDC<br />
|---<br />
|'''Platforms'''<br />
|<br />
*Linux<br />
*Mac<br />
*FreeBSD<br />
*Win<br />
|<br />
*Linux<br />
*Mac<br />
*FreeBSD<br />
*Win<br />
|<br />
*Linux<br />
*Mac<br />
*FreeBSD<br />
*Win<br />
|---<br />
|'''Architectures'''<br />
|<br />
* i386<br />
* amd64<br />
|<br />
* i386<br />
* amd64<br />
|<br />
* i386<br />
* amd64<br />
* ppc<br />
* ppc64<br />
|---<br />
|'''Distribution'''<br />
| Source, multi-platform source/binary archive, multi-platform installer ([https://github.com/jacob-carlborg/dvm DVM]),Windows installer, packages for OS X (.dmg), Debian/Ubuntu (.deb), Fedora (.rpm), OpenSUSE (.rpm), Debian/Ubuntu repository via http://d-apt.sourceforge.net, OS-X homebrew and macports repositories.<br />
| Source, Windows binary archive, Debian/Ubuntu repository<br />
| Source, Linux / OS X binary archive, Debian/Ubuntu repository, Fedora repository, Gentoo repository<br />
|---<br />
|'''Backend'''<br />
| [https://github.com/D-Programming-Language/dmd/tree/master/src/backend DMD] ([http://digitalmars.com/download/freecompiler.html DMC] fork)<br />
| [http://gcc.gnu.org/ GCC]<br />
| [http://llvm.org/ LLVM]<br />
|---<br />
|'''License'''<br />
| Frontend: GPL 1 or later / Artistic License<br />
Backend: [https://github.com/D-Programming-Language/dmd/blob/master/src/backendlicense.txt custom]<br />
| GPL 3 or later<br />
| LDC-specific code: 3-clause BSD<br />
|---<br />
|'''Inline assembler'''<br />
| Yes, x86<br />
| Yes, GCC-style only (all targets)<br />
| Yes, D-style (x86) and GCC-style (all targets)<br />
|---<br />
|'''SIMD'''<br />
| Partial (?)<br />
| Partial (?)<br />
| Partial (?)<br />
|---<br />
!colspan="4" align="center"|<br/>'''Linux specific'''<br />
|---<br />
|'''Object file format'''<br />
| ELF<br />
| ELF<br />
| ELF<br />
|---<br />
!colspan="4" align="center"|<br/>'''Mac specific'''<br />
|---<br />
|'''Object file format'''<br />
| Mach-O<br />
| Mach-O<br />
| Mach-O<br />
|---<br />
!colspan="4" align="center"|<br/>'''Windows specific'''<br />
|---<br />
|'''Object file format'''<br />
| OMF (32) / COFF (64)<br />
| COFF<br />
| COFF<br />
|}<br />
<br />
==Package and/or binary availability, by platform and compiler==<br />
<br />
Version number is given in brackets. Many of the manual download links have 10s of versions available, only the most recent few are listed here.<br />
<br />
Some unofficial repositories and downloads are listed here, but of course many more do exist. With a little searching, you may be able to find something more up to date for your chosen OS.<br />
<br />
Very old compilers are (mostly) omitted, as they are unlikely to be of interest to users.<br />
<br />
{| class="wikitable" style="width: 65em"<br />
!rowspan="2"|<br/>Platform<br />
!colspan="3" align="center"|Compiler<br />
|---<br />
!DMD<br />
!GDC<br />
!LDC<br />
|---<br />
!Windows<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.0)<br />
*[https://chocolatey.org Chocolatey] (2.067.0, 2.067.1 available awaiting moderation)<br />
|<br />
*[http://gdcproject.org/downloads Manual download] (4.9.2)<br />
|<br />
*[https://github.com/ldc-developers/ldc/releases Manual download] (0.15.1, 0.15.2-beta2)<br />
|---<br />
!OS X<br />
|<br />
*[http://brew.sh Homebrew] (2.068.0)<br />
*[https://www.macports.org MacPorts] (2.067.0)<br />
*[http://dlang.org/download.html Manual download] (2.068.0)<br />
|<br />
|<br />
*[http://brew.sh Homebrew] (0.15.1, 0.15.2-beta2)<br />
*[https://github.com/ldc-developers/ldc/releases Manual download] (0.15.1, 0.15.2-beta2)<br />
|---<br />
!Linux (generic)<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.068.0)<br />
|<br />
*[http://gdcproject.org/downloads Manual download] (4.9.2)<br />
|<br />
*[https://github.com/ldc-developers/ldc/releases Manual download] (0.15.1, 0.15.2-beta2)<br />
|---<br />
!Cross-platform<br />
|<br />
* [https://github.com/jacob-carlborg/dvm DVM] (Any version)<br />
|<br />
|<br />
|---<br />
!colspan="4" | Distribution-specific packages<br />
|---<br />
!Debian<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.067.1)<br />
*[http://d-apt.sourceforge.net APT repository] (2.067.1)<br />
|<br />
*Stable/Testing (4.8.4, 4.9.2)<br />
*Unstable (4.8.4, 4.9.2, 5.1.1)<br />
*Experimental (5.x.x)<br />
|<br />
*Stable/Testing/Unstable (0.14.0)<br />
*Experimental (0.15.1)<br />
|---<br />
!Ubuntu<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.067.1)<br />
*[http://d-apt.sourceforge.net APT repository] (2.067.1)<br />
|<br />
*14.04 (4.8.2)<br />
*14.10 (4.8.3, 4.9.1)<br />
*15.04 (4.8.4, 4.9.2)<br />
|<br />
*14.10 (0.14.0)<br />
|---<br />
!Fedora<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.067.1)<br />
*20/21 [https://build.opensuse.org/package/show?project=home%3Azhonghuaren&package=dmd Unofficial repository] (2.066.1)<br />
|<br />
|<br />
*20 (between 0.13.0-alpha1 and 0.13.0-alpha2)<br />
*21 (0.13.0)<br />
*22 (0.15.1)<br />
*Rawhide (0.15.2-beta1)<br />
|---<br />
!OpenSuse<br />
|<br />
*[https://build.opensuse.org/project/show/devel:languages:D devel:languages:D] (2.068.0)<br />
<!--*[https://build.opensuse.org/project/show/home:marc_schuetz:branches:devel:languages:D home:marc_schuetz] (2.067.0)--><br />
|<br />
|<br />
*[https://build.opensuse.org/project/show/devel:languages:D devel:languages:D] (0.15.1)<br />
*[https://build.opensuse.org/project/show/home:marc_schuetz:branches:devel:languages:D home:marc_schuetz] (0.15.2-beta1)<br />
|---<br />
!CentOS<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.067.1)<br />
|<br />
|<br />
|---<br />
!Arch Linux<br />
|<br />
*Community (2.068.0)<br />
|<br />
*Community (5.2.0)<br />
|<br />
*Community (0.15.1)<br />
|---<br />
!Gentoo<br />
|colspan="3" align=center | see https://wiki.gentoo.org/wiki/Dlang<br />
|---<br />
!FreeBSD<br />
|<br />
*[http://dlang.org/download.html Manual download] (2.067.1)<br />
*Ports (2.067.0)<br />
|<br />
|<br />
|}<br />
<br />
<br />
[[Category:Compiler]]</div>Schuetzmhttps://wiki.dlang.org/?title=DIP36&diff=6320DIP362015-06-25T08:06:21Z<p>Schuetzm: /* Discussions on topic */</p>
<hr />
<div>{|class="wikitable"<br />
!Title: <br />
!'''rvalue references'''<br />
|-<br />
|DIP: <br />
|36<br />
|-<br />
|Version:<br />
|2<br />
|-<br />
|Status:<br />
|Rejected<br />
|-<br />
|Created:<br />
|2013-04-08<br />
|-<br />
|Last Modified:<br />
|2013-04-08<br />
|-<br />
|Author:<br />
|Randy Schütt, Михаил Страшун<br />
|-}<br />
<br />
== Abstract ==<br />
This DIP describes issues with current usage of various rvalues as a function parameters and proproses possible improvement - formalization of scope references. As a positive side effect, intentions of <code>scope</code> qualifier are better defined into formal restrictions. Minor redefinition of <code>ref</code> in regards to <code>@safe</code> is also proposed.<br />
<br />
== Rationale ==<br />
There is a quite common necessity to pass some data as a function argument without both copying and caring about its referrability. Currently D provides no means to do it properly. <code>ref T</code> can't be used with rvalue literals. <code>T</code> results in (possibly) costly value copying for aggregate rvalues. <code>auto ref T</code> is template-based and thus bloats at least the symbol table. A solution is needed that will be simple, provide some guarantees and avoid extra bloat.<br />
<br />
<source lang="D"><br />
void main()<br />
{<br />
// Only signature for func1 which accepts both is func1(int), which is acceptable for trivial types<br />
int x;<br />
func1(x);<br />
func1(42);<br />
<br />
// For aggregate types passing by reference may be desired for both performance and avoiding side-effects of non-trivial constructors.<br />
// func2(ref Aggr) will accept only lvalues<br />
struct Aggr { }<br />
Aggr s;<br />
func2(s);<br />
func2(Aggr()); // fail<br />
<br />
// Now the question is, how can I say "I want to process this data with no side-effects, won't mutate it and don't care if is adressable"?<br />
// func3(T)(auto ref T) is current solution but it works by adding extra template instantiation for each case. This is both not needed<br />
// and does not scale with argument count.<br />
}<br />
</source><br />
<br />
Discussion threads:<br />
* http://forum.dlang.org/post/ntsyfhesnywfxvzbemwc@forum.dlang.org<br />
* http://forum.dlang.org/post/uswucstsooghescofycp@forum.dlang.org<br />
<br />
== Description ==<br />
=== Core proposal ===<br />
# <code>scope ref</code> is similar to <code>ref</code> but may be allowed in <code>@safe</code> code as it is prohibited to escape or store scoped reference, as well as taking its address. It is allowed to accept rvalues, temporary lvalues are created for them automatically.<br />
# <code>const scope ref</code> (or <code>in ref</code>) is like <code>scope ref</code> but prohibits mutation. It imposes usage restrictions (can't modify, can't store reference, can't take address) than make working with them indistinguishable from working with value types. Compiler can abuse it to create temporary variables for trivial type literals and pass references to them instead - it can't possibly change function semantics.<br />
<br />
==== Example ====<br />
<source lang="D"><br />
import std.stdio;<br />
<br />
struct A {<br />
public:<br />
int id;<br />
<br />
this(int id) {<br />
this.id = id;<br />
}<br />
<br />
this(this) {<br />
writeln("A Postblit for ", this.id);<br />
}<br />
}<br />
<br />
// Like a normal ref A, more restrictive because of scope, more permissive because accepts rvalue temporaries<br />
void test1(scope ref A a) {<br />
// A* addr = &a; // prohibited, see explanations later<br />
}<br />
<br />
// Almost nothing can be done with parameter, only value-style read access. Accepts rvalues.<br />
void test12(in ref A a) {<br />
}<br />
<br />
// Does not pretend to be @safe any more<br />
void test2(ref A a) {<br />
}<br />
<br />
// Similar to "auto ref" but no extra template instantations. Compiler creates temporaries for rvalues.<br />
void test3(T)(in ref T id) {<br />
}<br />
<br />
// Consistent with test1. Also no extra template instances.<br />
void test32(T)(scope ref T id) {<br />
}<br />
<br />
void main() {<br />
test1(A(42)); // @safe, this temporary value is valid for mutation and "scope" ensures it does not leak scope<br />
A a = A(23); // no difference<br />
test1(a);<br />
test2(A(1337)); // Prohibited, plain "ref" can't accept rvalues<br />
test2(a); // fine, but not @safe, unless it can be verified that a is allocated on heap in GC memory<br />
test3(1337); // Fine and @safe. Temporary int variable with value 1337 is created.<br />
test3(a.id); // Same but no temporary is needed.<br />
test32(1337); // Fine, temporary mutable int variable is created with value 1337<br />
test32(a.id); // fine and @safe<br />
}<br />
</source><br />
<br />
=== Definition of "scope" qualifier for ref types ===<br />
<br />
Following limitation apply to <code>scope ref</code> function parameters (including <code>in ref</code>):<br />
# Address of parameter can't be taken (and thus saved)<br />
# Parameter can't be returned from function<br />
# Parameter can only be used as an argument for other function if it also accepts <code>scope ref</code>, no implicit casting away.<br />
<br />
=== @safe concerns ===<br />
<br />
One of issues related to reference parameters that is raised in [[DIP25]] is that they currently allow subverting <code>@safe</code> limitations, despite being considered <code>@safe</code>. One of beneficial effects of this proposal is that it somewhat mitigates this issue in a simple manner. <code>scope ref</code> parameter limitations make them perfectly legal to use in safe code with no additional analysis. At the same time, such references cover a considerable number of use cases that may be required by safe code while not harming the power of unsafe ones. This potentially allows restricting the uses of plain <code>ref</code> in <code>@safe</code> code and leaving only <code>scope ref</code> allowed there. (Please see [[DIP35]] for a further examination of this issue.)<br />
<br />
This approach is not required for core proposal and is bonus opportunity that will become available as a side-effect.<br />
<br />
=== Backwards compatibility ===<br />
<br />
* <code>in ref</code> has been allowed from 2.060 : http://d.puremagic.com/issues/show_bug.cgi?id=8105<br />
* <code>scope ref</code> is still disallowed. ("Error: scope cannot be ref or out")<br />
* <code>in</code> is equivalent to <code>const scope</code> : http://dlang.org/function.html<br />
* Currently <code>scope</code> affects only delegate parameters. In other cases, <code>scope</code> has no meaning.<br />
<br />
It could possibly break code that already uses <code>in ref</code> (wrongly assuming it as abbreviation for const ref) and expects only an lvalue is accepted. Such code is both incorrect and very unlikely to exist. So no real code breaking change. Furthermore, many users (especially Jonathan) always warned not to use <code>in ref</code> as an abbreviation for <code>const ref</code> because it is wrong by language specification.<br />
<br />
Other code breakage should not be possible because:<br />
* <code>in ref</code> became more permissive, no restrictions added<br />
* <code>scope ref</code> is currently not allowed at all<br />
<br />
=== Overloading rules ===<br />
If you have several overloads of a function, that take their arguments by value, by ref and by scope ref, scope ref has always minor priority:<br />
<br />
<source lang="D"><br />
void foo(A a) {<br />
<br />
}<br />
<br />
void foo(ref A a) {<br />
<br />
}<br />
<br />
void foo(scope ref A a) {<br />
<br />
}<br />
<br />
void main() {<br />
foo(A()); // call: foo(A a)<br />
A a;<br />
foo(a); // call: foo(ref A a)<br />
}<br />
</source><br />
<br />
The same applies for 'const A', 'ref const A' and 'in ref A'.<br />
This behaviour is already implemented by Kenji's Pull Request.<br />
<br />
=== Interconnecton with DIP25 ===<br />
<br />
This DIP shares an area of domain with [[DIP25]] but suits different goals. DIP36 is about defining single reference type that can accept both rvalues and lvalues. DIP25 is about using reference types in <code>@safe</code>. They don't conflict and don't depend on each other.<br />
<br />
However, adopting this DIP provides some new options to solve DIP25 problem, providing reference type that is restricted enough to be safe on its own. This may be take advantage of or may not and is not really relevant. Code examples in this DIP have comments in regards of safety though to provide an overview for someone interested in this topic.<br />
<br />
See also [[DIP35]], which contains an amendment to DIP25 and finds another use for 'scope'. Fortunately, it is not likely to cause conflict with the use proposed here.<br />
<br />
=== Other proposed syntaxes ===<br />
<br />
{|class="wikitable"<br />
!Syntax <br />
!Issues<br />
|-<br />
|@ref<br />
|looks like a one-character hack. Not clear what it means.<br />
|-<br />
|ref&<br />
|looks like a one-character hack. Not clear what it means. Looks more like double ref (T&& from C++)<br />
|-<br />
|auto ref<br />
|Template bloat, [http://forum.dlang.org/post/mailman.293.1364249651.4724.digitalmars-d-learn@puremagic.com nicely explained by Jonathan]<br />
|-<br />
|const ref<br />
|Rejected by Andrei because const is far more restrictive in D than in C++. Does not allow mutable rvalues.<br />
|-<br />
|immutable ref<br />
|Same as const ref<br />
|-<br />
|@temp ref<br />
|Is somewhat clearer, but adds an entirely new attribute to the language<br />
|-<br />
|final ref<br />
|Not clear what it means, introducing new meaning for 'final'.<br />
|-<br />
|A& (like C++'s ref)<br />
|Need a 'hack' in mtype.h / mtype.c. Type struct need a new boolean property: isRvRef<br />
|}<br />
<br />
=== Discussions on topic ===<br />
* http://forum.dlang.org/post/ylebrhjnrrcajnvtthtt@forum.dlang.org<br />
* http://forum.dlang.org/post/zteryxwxyngvyqvukqkm@forum.dlang.org<br />
* http://forum.dlang.org/post/yhnbcocwxnbutylfeoxi@forum.dlang.org<br />
* http://forum.dlang.org/post/qbirgbuvjndkviymuypr@forum.dlang.org<br />
* http://forum.dlang.org/post/tkzyjhshbqjqxwzppdin@forum.dlang.org<br />
* http://forum.dlang.org/post/kcksvf$314v$1@digitalmars.com<br />
* http://forum.dlang.org/post/nirfuenixutsbgyrcsla@forum.dlang.org<br />
<br />
=== Experimental implementations ===<br />
* https://github.com/9rnsr/dmd/commits/new_inref (by Kenji)<br />
* https://github.com/Dgame/dmd/commits/scope_in_ref (by me)<br />
<br />
== Copyright ==<br />
This document has been placed in the Public Domain.<br />
<br />
[[Category: DIP]]</div>Schuetzmhttps://wiki.dlang.org/?title=Bind_D_to_C&diff=6283Bind D to C2015-06-07T10:14:16Z<p>Schuetzm: /* Global variables */</p>
<hr />
<div><br />
== Introduction ==<br />
[http://www.gamedev.net/page/resources/_/technical/game-programming/binding-d-to-c-r3122 Article] and [http://www.gamedev.net/blog/1140/entry-2254003-binding-d-to-c/ Series] on creating bindings to C libraries for the D programming language.<br />
<br />
<br />
== Response to the article / more information ==<br />
=== Global variables ===<br />
<br />
Global variables need to have an extra <code>extern</code> and the <code>__gshared</code> storage.<br />
<br />
''Example in C:''<br />
<br />
int a;<br />
<br />
''Translated to D:''<br />
<br />
extern (C) extern __gshared int a;<br />
<br />
For TLS variables __gshared is not used.<br />
<br />
=== Typedefs ===<br />
<br />
When I do a binding I'm trying to figure out why they used a typedef in the first place.<br />
<br />
There's a couple of reasons:<br />
<br />
* To get a fixed type on all platforms. In the C language there are no fixed types, it's just relations between the sizes of the types. I.e. long => int => short => char, or something like that. In this case, use the native D type. In D a given type has a fixed size on all platforms (except for real). Examples of these can be: "int8_t", "uint16_t", "uint32_t" and so on.<br />
<br />
* To get different sizes on different platforms. For example, a 32bit value of 32bit platforms and a 64bit value of 64bit platforms. In this case you're basically forced to use an alias.<br />
<br />
* Opaque types. Perhaps the API is hiding the actual type behind a void* or for other reasons uses a void*. Then it uses a tyepdef on top of that to give it an idea of what type we're dealing with. In this case use the typedef.<br />
<br />
* The typedef is well known in the API. Examples of this would be GLint. I don't remember why they use typedefs but if I recall correctly GLint is used in all examples, tutorials, books and so on. In this case it's best to the typedef.<br />
<br />
* Hiding a complex type. An example of this could be function pointers. In this case use the typedef.<br />
<br />
=== Function pointers ===<br />
<br />
With function pointers there are (at least) two cases where an alias have to be used, instead of a function pointer. <br />
<br />
* When declaring function parameters with a specific linkage.<br />
<br />
The following is syntactically invalid in D:<br />
<br />
void foo (extern(C) void function () callback);<br />
<br />
Use an alias:<br />
<br />
alias extern (C) void function () Callback; <br />
void foo (Callback callback);<br />
<br />
* When using a cast with a specific linkage. You won't see this in a binding, if you're not converting inline functions.<br />
<br />
This is invalid in D as well:<br />
<br />
void* foo;<br />
...<br />
auto bar = cast(extern (C) void function ()) foo;<br />
<br />
Use the same approach as above.<br />
<br />
=== Structs/Unions ===<br />
<br />
You actually didn't mention unions at all but basically all the same rules that apply when translating structs apply to unions as well. Two other things that you didn't mention was nested structs and anonymous structs.<br />
<br />
=== Structs ===<br />
<br />
* For named structs in typedefs I feel a bit undecided of how I want to have it. You can either just use the name of the typedef in D when declaring the struct. Or you can use the real name of the struct and then use an alias as well. In some cases the actual name of the struct could be used, i.e. "struct foo".<br />
<br />
=== Anonymous Structs ===<br />
<br />
* If an anonymous struct is used directly to declare a variable you're forced to invent a name for the struct in D, since D doesn't support anonymous structs. Example:<br />
<br />
struct<br />
{<br />
int a;<br />
int b;<br />
} c; <br />
<br />
Translate to:<br />
<br />
struct _AnonymousStruct1<br />
{<br />
int a;<br />
int b;<br />
}<br />
<br />
_AnonymousStruct1 c;<br />
<br />
Any name can be used in this case. In my tool, DStep, for automatically generating bindings I'm using names similar to above.<br />
<br />
* If an anonymous struct is used in a typedef just use the name of the typedef. No alias is required in the D code in this case. I actual noticed now that you have this example.<br />
<br />
=== Nested structs ===<br />
<br />
I always fail to remember how these should be translated to D.<br />
<br />
* For nested structs that are named. Example:<br />
<br />
struct Foo<br />
{<br />
int a;<br />
struct Bar<br />
{<br />
int b;<br />
} bar;<br />
}<br />
<br />
In this case translate it to a named struct in D as well:<br />
<br />
struct Foo<br />
{<br />
int a;<br />
struct Bar<br />
{<br />
int b;<br />
}<br />
Bar bar;<br />
}<br />
<br />
* For nested struct that are anonymous. Example:<br />
<br />
struct Foo<br />
{<br />
int a;<br />
struct<br />
{<br />
int b;<br />
} bar;<br />
}<br />
<br />
In this case you could translate it to an anonymous struct in D as well:<br />
<br />
struct Foo<br />
{<br />
int a;<br />
struct<br />
{<br />
int b;<br />
}<br />
}<br />
<br />
The problem with this is that the API changes. Instead of accessing "b" like this:<br />
<br />
struct Foo foo;<br />
foo.bar.b = 1;<br />
<br />
You would do like this:<br />
<br />
Foo foo;<br />
foo.b = 1;<br />
<br />
I actually looked at the documentation now and see that this example is not used anymore:<br />
<br />
[http://dlang.org/interfaceToC.html http://dlang.org/interfaceToC.html]<br><br />
[http://dlang.org/htod.html http://dlang.org/htod.html]<br />
<br />
It's still on the D1 page:<br />
<br />
[http://digitalmars.com/d/1.0/htomodule.html http://digitalmars.com/d/1.0/htomodule.html]<br />
<br />
== See also ==<br />
<br />
* [[D binding for C]]<br />
* [[Binding generators]]<br />
* [[Bindings]]<br />
* [http://p0nce.github.io/d-idioms/#Porting-from-C-gotchas Porting from C gotchas]<br />
<br />
<br />
<br />
[[Category:Binding]]<br />
[[Category:HowTo]]</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=6259User:Schuetzm/scope32015-06-03T10:05:29Z<p>Schuetzm: /* @safe-ty violations with borrowing */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
#: Copying a ''borrowed'' value however increases the ''owner'''s loan count.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared (goes out of scope), the ''owner'' no longer counts as ''loaned''.<br />
#:Alternatively: after the last access to any of the ''borrowed'' values. (But destructors need to count as an access, too.)<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
# Loops, '''goto'''s, and branches are treated conservatively; when a ''borrowing'' (or copy of a ''borrowed'' value) can potentially take place, it is assumed it does. To ease implementation, special restrictive rules can be added for '''goto'''.<br />
<br />
==== Examples ====<br />
<br />
No mutation while ''borrowed'' values exist:<br />
<br />
<source lang="D"><br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // opAssign takes mutable `this` (= loaned `arr`),<br />
// therefore the assignment is @system<br />
writeln(*p);<br />
}<br />
}<br />
</source><br />
<br />
Available strategies when it happens (user decides which one is suited best):<br />
<br />
<source lang="D"><br />
// defer mutation until it's ok<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0];<br />
writeln(*p);<br />
}<br />
RCArray!int other;<br />
arr = other; // now ok, `arr` is not loaned<br />
}<br />
<br />
// operate an a copy<br />
// (like DIP77, but explicit and obvious)<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
auto tmp = arr;<br />
{<br />
int* p = &tmp[0]; // `tmp` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // now ok, `arr` is not loaned<br />
writeln(*p);<br />
}<br />
}<br />
<br />
// verify manually that it's @safe<br />
void foo() @trusted { ... }<br />
</source><br />
<br />
@safe ''borrowing'' of multiple values:<br />
<br />
<source lang="D"><br />
void bar(scope int* p);<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned once<br />
int* q = &arr[1]; // `arr` is now loaned twice<br />
int* r = q; // copy: `arr` is loaned three times<br />
int* s;<br />
if(halting_problem())<br />
s = q; // conservatively assume branch is taken<br />
// => `arr` is now loaned four times<br />
bar(p); // @safe, because `arr` cannot possibly<br />
// be mutated through `p`<br />
// `p`, `q` go out of scope<br />
}<br />
// `arr`'s loan count is now 0<br />
RCArray!int other;<br />
arr = other; // now @safe, not loaned anymore<br />
}<br />
</source><br />
<br />
==== Alternative ====<br />
<br />
As an alternative to making certain operations @system, values with outstanding loans can instead be treated as const. The user can then use casts to mutate such values on their own risk.<br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=Conventional_module_name_for_importing_all_modules_in_a_package&diff=6172Conventional module name for importing all modules in a package2015-05-20T08:17:08Z<p>Schuetzm: </p>
<hr />
<div>In Java, if you want to import all of the modules in a package, you'd do this:<br />
<syntaxhighlight lang="java"><br />
import javax.swing.*;<br />
</syntaxhighlight><br />
<br />
Since D doesn't have a "*" (and it's unlikely that D would get such a shortcut), this capability could be added to a class by adding a module, that imports the other modules. So what should such a module be called. There have been a few suggestions.<br />
<br />
These are common conventions:<br />
<syntaxhighlight lang="D"><br />
import mylibrary.mypackage.all;<br />
import mylibrary.mypackage._;<br />
</syntaxhighlight><br />
<br />
== package.d ==<br />
<br />
Since several releases ago, D supports importing an entire package using the <code>package.d</code> module. This file is used as a fallback by the compiler if the imported module turns out to be a directory. Example:<br />
<br />
<code>source/example/package.d</code><br />
<source lang="D"><br />
module example;<br />
<br />
public import example.types;<br />
public import example.api;<br />
public import example.helpers;<br />
</source><br />
<br />
<code>source/app.d</code><br />
<source lang="D"><br />
import example; // imports example/package.d<br />
<br />
void main() {<br />
// ...<br />
}<br />
</source><br />
<br />
[[Category:CommonIdiom]]</div>Schuetzmhttps://wiki.dlang.org/?title=Useful_D_snippets&diff=6119Useful D snippets2015-05-16T10:11:59Z<p>Schuetzm: formatting</p>
<hr />
<div>==Converting a time_t unix timestamp into std.datetime types==<br />
http://dlang.org/intro-to-datetime.html<br />
<br />
<source lang="D"><br />
import std.datetime;<br />
import core.stdc.time;<br />
void main() {<br />
time_t unixTime = core.stdc.time.time(null);<br />
auto stdTime = unixTimeToStdTime(unixTime);<br />
auto st = SysTime(stdTime);<br />
}<br />
</source></div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/isolated&diff=5864User:Schuetzm/isolated2015-04-12T15:56:22Z<p>Schuetzm: </p>
<hr />
<div>(Warning: not well thought through yet)<br />
<br />
== Isolated islands ==<br />
<br />
Currently, this works:<br />
<br />
<source lang="D"><br />
void main() {<br />
immutable int* p = new int;<br />
}<br />
</source><br />
<br />
This doesn't:<br />
<br />
<source lang="D"><br />
struct S {<br />
int* p;<br />
}<br />
<br />
void main() {<br />
auto tmp = new S;<br />
tmp.p = new int;<br />
immutable S* s = tmp; // Error: cannot implicitly convert expression (tmp) of type S* to immutable(S*)<br />
}<br />
</source><br />
<br />
But if we wrap the construction in a pure function, it does work:<br />
<br />
<source lang="D"><br />
struct S {<br />
int* p;<br />
}<br />
<br />
S* makeS() pure {<br />
auto tmp = new S;<br />
tmp.p = new int;<br />
return tmp;<br />
}<br />
<br />
void main() {<br />
immutable S* s = makeS();<br />
}<br />
</source><br />
<br />
By utilizing deadalnix' "isolated" idea, we can make the second example work, too. This will be useful for incremental construction of complex immutable data structures, as well as data structures that should be transferred to another thread.<br />
<br />
TODO: specify the rules (islands concept)<br />
<br />
== Isolated across function boundaries ==<br />
<br />
Currently, values are inferred to be unique when they originate from certain types of expressions, most importantly `new` and calls to strongly pure functions (or weakly pure functions with parameters having incompatible types with the return value). But there are some functions that are known to return unique values by the programmer, but the compiler cannot know this. [http://dlang.org/phobos/std_typecons.html#.Unique.release std.typecons.Unique.release] is a good example. On the other hand, functions might require a unique argument, without casting it to immutable. Again, std.typecons.Unique's constructor comes to mind.<br />
<br />
An @isolated annotation is proposed to express this intent. Applied to a parameter (or after the function for `this`), the function can only be called with unique arguments for that parameter. Applied on the left side of a function declaration, the function will be treated as returning a unique value. Additionally, aggregate members can be designated with it.<br />
<br />
The compiler statically verifies that values assigned to @isolated variables are unique, and treats values read from @isolated variables as such. Anything written to an @isolated variable becomes inaccessible; when an @isolated variable is read from, it also becomes inaccessible (exception: '''const scope'''). @isolated members can only be read from an @isolated aggregate; this ensures that the entire aggregate will become inaccessible in the calling function.<br />
<br />
With this tool, not only can std.typecons.Unique be implemented more safely, but it's now also possible to safely transfer ownership of mutable data structures across threads.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/isolated&diff=5863User:Schuetzm/isolated2015-04-12T15:27:13Z<p>Schuetzm: Created page with "== Isolated == Currently, this works: <source lang="D"> void main() { immutable int* p = new int; } </source> This doesn't: <source lang="D"> struct S { int* p; }..."</p>
<hr />
<div>== Isolated ==<br />
<br />
Currently, this works:<br />
<br />
<source lang="D"><br />
void main() {<br />
immutable int* p = new int;<br />
}<br />
</source><br />
<br />
This doesn't:<br />
<br />
<source lang="D"><br />
struct S {<br />
int* p;<br />
}<br />
<br />
void main() {<br />
auto tmp = new S;<br />
tmp.p = new int;<br />
immutable S* s = tmp; // Error: cannot implicitly convert expression (tmp) of type S* to immutable(S*)<br />
}<br />
</source><br />
<br />
But if we wrap the construction in a pure function, it does work:<br />
<br />
<source lang="D"><br />
struct S {<br />
int* p;<br />
}<br />
<br />
S* makeS() pure {<br />
auto tmp = new S;<br />
tmp.p = new int;<br />
return tmp;<br />
}<br />
<br />
void main() {<br />
immutable S* s = makeS();<br />
}<br />
</source><br />
<br />
By utilizing deadalnix' "isolated" idea, we can make the second example work, too. This will be useful for incremental construction of complex immutable data structures, as well as data structures that should be transferred to another thread.<br />
<br />
TODO: specify the rules (islands concept)</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5862User:Schuetzm/scope32015-04-12T11:06:45Z<p>Schuetzm: /* @safe-ty violations with borrowing */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
#: Copying a ''borrowed'' value however increases the ''owner'''s loan count.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared (goes out of scope), the ''owner'' no longer counts as ''loaned''.<br />
#:Alternatively: after the last access to any of the ''borrowed'' values. (But destructors need to count as an access, too.)<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
# Loops, '''goto'''s, and branches are treated conservatively; when a ''borrowing'' (or copy of a ''borrowed'' value) can potentially take place, it is assumed it does. To ease implementation, special restrictive rules can be added for '''goto'''.<br />
<br />
==== Examples ====<br />
<br />
No mutation while ''borrowed'' values exist:<br />
<br />
<source lang="D"><br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // opAssign takes mutable `this` (= loaned `arr`),<br />
// therefore the assignment is @system<br />
writeln(*p);<br />
}<br />
}<br />
</source><br />
<br />
Available strategies when it happens (user decides which one is suited best):<br />
<br />
<source lang="D"><br />
// defer mutation until it's ok<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0];<br />
writeln(*p);<br />
}<br />
RCArray!int other;<br />
arr = other; // now ok, `arr` is not loaned<br />
}<br />
<br />
// operate an a copy<br />
// (like DIP77, but explicit and obvious)<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
auto tmp = arr;<br />
{<br />
int* p = &tmp[0]; // `tmp` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // now ok, `arr` is not loaned<br />
writeln(*p);<br />
}<br />
}<br />
<br />
// verify manually that it's @safe<br />
void foo() @trusted { ... }<br />
</source><br />
<br />
@safe ''borrowing'' of multiple values:<br />
<br />
<source lang="D"><br />
void bar(scope int* p);<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned once<br />
int* q = &arr[1]; // `arr` is now loaned twice<br />
int* r = q; // copy: `arr` is loaned three times<br />
int* s;<br />
if(halting_problem())<br />
s = q; // conservatively assume branch is taken<br />
// => `arr` is now loaned four times<br />
bar(p); // @safe, because `arr` cannot possibly<br />
// be mutated through `p`<br />
// `p`, `q` go out of scope<br />
}<br />
// `arr`'s loan count is now 0<br />
RCArray!int other;<br />
arr = other; // now @safe, not loaned anymore<br />
}<br />
</source><br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5861User:Schuetzm/scope32015-04-12T10:35:36Z<p>Schuetzm: /* Examples */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
#: Copying a ''borrowed'' value however increases the ''owner'''s loan count.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared (goes out of scope), the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
# Loops, '''goto'''s, and branches are treated conservatively; when a ''borrowing'' (or copy of a ''borrowed'' value) can potentially take place, it is assumed it does. To ease implementation, special restrictive rules can be added for '''goto'''.<br />
<br />
==== Examples ====<br />
<br />
No mutation while ''borrowed'' values exist:<br />
<br />
<source lang="D"><br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // opAssign takes mutable `this` (= loaned `arr`),<br />
// therefore the assignment is @system<br />
writeln(*p);<br />
}<br />
}<br />
</source><br />
<br />
Available strategies when it happens (user decides which one is suited best):<br />
<br />
<source lang="D"><br />
// defer mutation until it's ok<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0];<br />
writeln(*p);<br />
}<br />
RCArray!int other;<br />
arr = other; // now ok, `arr` is not loaned<br />
}<br />
<br />
// operate an a copy<br />
// (like DIP77, but explicit and obvious)<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
auto tmp = arr;<br />
{<br />
int* p = &tmp[0]; // `tmp` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // now ok, `arr` is not loaned<br />
writeln(*p);<br />
}<br />
}<br />
<br />
// verify manually that it's @safe<br />
void foo() @trusted { ... }<br />
</source><br />
<br />
@safe ''borrowing'' of multiple values:<br />
<br />
<source lang="D"><br />
void bar(scope int* p);<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned once<br />
int* q = &arr[1]; // `arr` is now loaned twice<br />
int* r = q; // copy: `arr` is loaned three times<br />
int* s;<br />
if(halting_problem())<br />
s = q; // conservatively assume branch is taken<br />
// => `arr` is now loaned four times<br />
bar(p); // @safe, because `arr` cannot possibly<br />
// be mutated through `p`<br />
// `p`, `q` go out of scope<br />
}<br />
// `arr`'s loan count is now 0<br />
RCArray!int other;<br />
arr = other; // now @safe, not loaned anymore<br />
}<br />
</source><br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5854User:Schuetzm/scope32015-04-11T11:32:53Z<p>Schuetzm: /* @safe-ty violations with borrowing */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
#: Copying a ''borrowed'' value however increases the ''owner'''s loan count.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared (goes out of scope), the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
# Loops, '''goto'''s, and branches are treated conservatively; when a ''borrowing'' (or copy of a ''borrowed'' value) can potentially take place, it is assumed it does. To ease implementation, special restrictive rules can be added for '''goto'''.<br />
<br />
==== Examples ====<br />
<br />
No mutation while ''borrowed'' values exist:<br />
<br />
<source lang="D"><br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // opAssign takes mutable `this` (= loaned `arr`),<br />
// therefore the assignment is @system<br />
writeln(*p);<br />
}<br />
}<br />
</source><br />
<br />
@safe ''borrowing'' of multiple values:<br />
<br />
<source lang="D"><br />
void bar(scope int* p);<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned once<br />
int* q = &arr[1]; // `arr` is now loaned twice<br />
int* r = q; // copy: `arr` is loaned three times<br />
int* s;<br />
if(halting_problem())<br />
s = q; // conservatively assume branch is taken<br />
// => `arr` is now loaned four times<br />
bar(p); // @safe, because `arr` cannot possibly<br />
// be mutated through `p`<br />
// `p`, `q` go out of scope<br />
}<br />
// `arr`'s loan count is now 0<br />
RCArray!int other;<br />
arr = other; // now @safe, not loaned anymore<br />
}<br />
</source><br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5853User:Schuetzm/scope32015-04-11T11:27:11Z<p>Schuetzm: /* @safe-ty violations with borrowing */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Copying a ''reference'' or taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared, the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
<br />
==== Examples ====<br />
<br />
No mutation while ''borrowed'' values exist:<br />
<br />
<source lang="D"><br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // opAssign takes mutable `this` (= loaned `arr`),<br />
// therefore the assignment is @system<br />
writeln(*p);<br />
}<br />
}<br />
</source><br />
<br />
@safe ''borrowing'' of multiple values:<br />
<br />
<source lang="D"><br />
void bar(scope int* p);<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned once<br />
int* q = &arr[1]; // `arr` is now loaned twice<br />
int* r = q; // copy: `arr` is loaned three times<br />
int* s;<br />
if(halting_problem())<br />
s = q; // conservatively assume branch is taken<br />
// => `arr` is now loaned four times<br />
bar(p); // @safe, because `arr` cannot possibly<br />
// be mutated through `p`<br />
// `p`, `q` go out of scope<br />
}<br />
// `arr`'s loan count is now 0<br />
RCArray!int other;<br />
arr = other; // now @safe, not loaned anymore<br />
}<br />
</source><br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5852User:Schuetzm/scope32015-04-11T11:21:47Z<p>Schuetzm: /* Examples */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Copying a ''reference'' or taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared, the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
<br />
==== Examples ====<br />
<br />
No mutation while ''borrowed'' values exist:<br />
<br />
<source lang="D"><br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // opAssign takes mutable `this` (= loaned `arr`),<br />
// therefore the assignment is @system<br />
writeln(*p);<br />
}<br />
}<br />
</source><br />
<br />
@safe ''borrowing'' of multiple values:<br />
<br />
<source lang="D"><br />
void bar(scope int* p);<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned once<br />
int* q = &arr[1]; // `arr` is now loaned twice<br />
int *r = q; // copy: `arr` is loaned three times<br />
bar(p); // @safe, because `arr` cannot possibly<br />
// be mutated through `p`<br />
// `p`, `q` go out of scope<br />
}<br />
// `arr`'s loan count is now 0<br />
RCArray!int other;<br />
arr = other; // now @safe, not loaned anymore<br />
}<br />
</source><br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5851User:Schuetzm/scope32015-04-11T11:06:03Z<p>Schuetzm: /* @safe-ty violations with borrowing */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Copying a ''reference'' or taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared, the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
<br />
==== Examples ====<br />
<br />
No mutation while ''borrowed'' values exist:<br />
<br />
<source lang="D"><br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned<br />
// scope inferrence takes care of `p`<br />
RCArray!int other;<br />
arr = other; // opAssign takes mutable `this` (= loaned `arr`),<br />
// therefore the assignment is @system<br />
writeln(*p);<br />
}<br />
}<br />
</source><br />
<br />
@safe ''borrowing'' of multiple values:<br />
<br />
<source lang="D"><br />
void bar(scope int* p);<br />
void foo() @safe {<br />
RCArray!int arr = [0,1,2];<br />
{<br />
int* p = &arr[0]; // `arr` is now loaned once<br />
int* q = &arr[1]; // `arr` is now loaned twice<br />
bar(p); // @safe, because `arr` cannot possibly<br />
// be mutated through `p`<br />
// `p`, `q` go out of scope<br />
}<br />
// `arr`'s loan count is now 0<br />
RCArray!int other;<br />
arr = other; // now @safe, not loaned anymore<br />
}<br />
</source><br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5850User:Schuetzm/scope32015-04-11T10:54:03Z<p>Schuetzm: /* Terminology */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Copying a ''reference'' or taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared, the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' anywhere. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5849User:Schuetzm/scope32015-04-11T10:53:08Z<p>Schuetzm: /* Terminology */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Copying a ''reference'' or taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared, the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is ''scope''d to an ''owner'' into a variable in a location with finite ''scope''. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5848User:Schuetzm/scope32015-04-11T10:52:57Z<p>Schuetzm: /* Terminology */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Copying a ''reference'' or taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared, the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a value that is '''scope''d to an ''owner'' into a variable in a location with finite ''scope''. This includes temporaries, and passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5847User:Schuetzm/scope32015-04-11T10:51:40Z<p>Schuetzm: /* @safe-ty violations with borrowing */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
The rules are therefore:<br />
# ''Borrowing'' from a global variable is always @system.<br />
# ''Borrowing'' from a local variable marks that variable (the ''owner'') as ''loaned''.<br />
#: Copying a ''reference'' or taking the address of a '''ref''' parameter does ''not'' count as ''borrowing'', because no new ''reference'' is created.<br />
# When the last ''borrowed'' value to an ''owner'' has disappeared, the ''owner'' no longer counts as ''loaned''.<br />
# Any potential mutation of a loaned owner is @system.<br />
#: Passing a ''reference'' to a ''loaned'' ''owner'' to a function as a mutable parameter must be treated as potential mutation.<br />
# A ''borrowed'' value can contain a ''reference'' to the ''owner''. Therefore, if a ''borrowed'' value contains a mutable ''reference'' compatible with the ''owner'''s type, it must be treated as if it were the ''owner'' in the context of the above rules.<br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a ''reference'' to a variable in a location with finite ''scope''. This includes passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5846User:Schuetzm/scope32015-04-11T10:31:20Z<p>Schuetzm: </p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite scope can be assigned through an indirection, while references read through an indirection get the same scope as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.<br />
<br />
== Terminology ==<br />
<br />
;reference<br />
:An object reference, a '''ref''' parameter, a pointer, a slice, or a delegate. Also, any aggregate containing at least one of those.<br />
;lifetime<br />
:The entire duration during which a variable exists. A variable comes into existence through constructions, and stops existing when it is destroyed. A variable's ''lifetime'' can be determined by the compiler (locals, temporaries), by the programmer through explicit manual memory management, or by some other mechanism (reference counting, tracing GC).<br />
;scope<br />
:The minimum guaranteed ''lifetime'' of a variable. The variable will exist as least as long as its ''scope'' indicates. ''Scope''s are arranged in a hierarchy based on the ''lexical scope'' and order of declaration of their ''references''. For any two ''scope''s, one is either completely contained in the other, or they are disjoint.<br />
;borrowing<br />
:Storing a ''reference'' to a variable in a location with finite ''scope''. This includes passing a ''reference'' to a function.<br />
;owner<br />
:The original variable a ''borrowed'' ''reference'' comes from.</div>Schuetzmhttps://wiki.dlang.org/?title=User_talk:Schuetzm/scope2&diff=5844User talk:Schuetzm/scope22015-04-11T10:11:25Z<p>Schuetzm: /* Dereferencing yields static scope */ slicing doesn't dereference</p>
<hr />
<div>== Example for the inference algorithm ==<br />
<br />
=== Dereferencing yields <code>static</code> scope ===<br />
<br />
Note that this is depending on whether we're on the LHS or RHS of an assignment. The inference algorithm only assigns scopes from LHS to RHS; therefore we always use <code>static</code>. On the RHS, a dereference gets the scope of the reference it's been reached through, because we at least known that it can't have a shorter lifetime than that.<br />
<br />
Dereferencing includes the dereference operator, implicit dereferencing of struct/union pointers, of class references, as well as indexing.<br />
<br />
Applied to the function deadalnix used to demonstrate the rvalue/lvalue problem:<br />
<br />
(1) <code>foo()</code> is <code>@safe</code>, making its param scoped:<br />
<br />
<source lang="D"><br />
void foo(scope int** a) {<br />
// no inference for `a` => self-owned<br />
// ("mystery scope" in your terms)<br />
// => SCOPE(a) := [a] (final)<br />
int** b;<br />
// => SCOPE(b) := [b] (final)<br />
b = a;<br />
// scope of `a` is fixed => do nothing<br />
int d;<br />
int* c;<br />
// => SCOPE(c) := [c] (incomplete)<br />
c = &d;<br />
*b = c;<br />
// assignment from `c`:<br />
// => SCOPE(c) |= SCOPE(*b) = [c] | [static] (final)<br />
// (dereferencing loses information => static)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [a]<br />
// SCOPE(b) = [b]<br />
// SCOPE(c) = [static]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
int** b = a; // [b] := [a] => OK<br />
int d;<br />
int* c = &d; // [static] := [d] => BAD<br />
*b = c; // [static] := [static] => OK<br />
// => invalid assignment<br />
// note how it even traces where the bad value comes from<br />
}<br />
</source><br />
<br />
(2) <code>foo()</code> is in a template, param scope gets inferred<br />
<br />
<source lang="D"><br />
void foo(T)(T** a) {<br />
// start with empty scope for params<br />
// => SCOPE(a) := [a] (incomplete)<br />
T** b;<br />
// start with empty scope for locals<br />
// => SCOPE(b) := [b] (final)<br />
b = a;<br />
// only access to `a`:<br />
// => SCOPE(a) |= SCOPE(b) = [a] | [b] = [a]<br />
T d;<br />
T* c;<br />
// => SCOPE(c) = [c] (incomplete)<br />
c = &d;<br />
*b = c;<br />
// assignment from `c`:<br />
// => SCOPE(c) |= SCOPE(*b) = [c] | [static] = [static]<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [a]<br />
// SCOPE(b) = [b]<br />
// SCOPE(c) = [static]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
int** b = a; // [b] := [a] => OK<br />
int d;<br />
int* c = &d; // [static] := [d] => BAD<br />
*b = c; // [static] := [static] => OK<br />
// => invalid assignment<br />
// again, the culprit can be detected<br />
}<br />
</source><br />
<br />
(3) <code>foo()</code> takes unscoped pointer<br />
<br />
<source lang="D"><br />
void foo(int** a) {<br />
// no inference for `a` => static<br />
// => SCOPE(a) := [static] (final)<br />
int** b;<br />
// => SCOPE(b) := [b] (final)<br />
b = a;<br />
// scope of `a` is fixed => do nothing<br />
int d;<br />
int* c;<br />
// => SCOPE(c) := [c] (incomplete)<br />
c = &d;<br />
*b = c;<br />
// assignment from `c`:<br />
// => SCOPE(c) |= SCOPE(*b) = [c] | [static] (final)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [static]<br />
// SCOPE(b) = [b]<br />
// SCOPE(c) = [static]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
int** b = a; // [b] := [static] => OK<br />
int d;<br />
int* c = &d; // [static] := [d] => BAD<br />
*b = c; // [static] := [static] => OK<br />
// => invalid assignment<br />
// escape detection for locals works even without scope params<br />
// and in @system code<br />
}<br />
</source><br />
<br />
=== <code>return</code> inference ===<br />
<br />
<source lang="D"><br />
T[] findSubstring(T)(T[] haystack, T[] needle) {<br />
// find first occurrence of substring<br />
// naive implementation<br />
<br />
// => SCOPE(return) := [return] (fixed)<br />
// [return] is a scope higher than all params<br />
// => SCOPE(haystack) := [haystack] (incomplete)<br />
// => SCOPE(needle) := [needle] (final)<br />
<br />
for(int i = 0; i < haystack.length - needle.length; i++) {<br />
T[] sub;<br />
// => SCOPE(sub) := [sub] (incomplete)<br />
sub = haystack[i .. i + needle.length];<br />
// haystack and needle are read again; ignoring from now on<br />
// => SCOPE(haystack) |= SCOPE(sub)<br />
// => DEFER because SCOPE(sub) is incomplete<br />
if(sub == needle) {<br />
return haystack[i .. $];<br />
// => SCOPE(haystack) |= SCOPE(return) = [haystack] | [return] = [return]<br />
}<br />
}<br />
return null;<br />
// nothing to do here, `null` has fixed static scope<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(haystack) = [return]<br />
// SCOPE(needle) = [needle]<br />
// SCOPE(sub) = [sub]<br />
// unresolved:<br />
// SCOPE(haystack) |= SCOPE(sub)<br />
// resolve:<br />
// SCOPE(haystack) := [return] | [sub] = [return]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
for(int i = 0; i < haystack.length - needle.length; i++) {<br />
T[] sub;<br />
sub = haystack[i .. i + needle.length];<br />
// [sub] := [return] => OK<br />
if(sub == needle) {<br />
// equivalent to: if(opEquals(sub, needle))<br />
// let's assume `opEquals` takes both params by scope<br />
// sub: [] := [sub] => OK<br />
// needle: [] := [needle] => OK<br />
return haystack[i .. $];<br />
// [return] := [return] => OK<br />
}<br />
}<br />
return null;<br />
// [return] := [static] => OK<br />
}<br />
<br />
// the inferred signature for T == immutable(char) is then<br />
string findSubstring(scope string haystack return, scope string needle);<br />
</source><br />
<br />
Multiple <code>return</code> params:<br />
<br />
<source lang="D"><br />
T chooseStringAtRandom(T)(T a, T b) {<br />
// => SCOPE(a) = [a] (incomplete)<br />
// => SCOPE(b) = [a] (incomplete)<br />
return random() % 2 == 0 ? a : b;<br />
// => SCOPE(a) |= [a] = [a]<br />
// => SCOPE(b) |= [b] = [b]<br />
// ?: operator affects both `a` and `b`<br />
// => SCOPE(a) |= SCOPE(return) = [a] | [return] = [return]<br />
// => SCOPE(b) |= SCOPE(return) = [b] | [return] = [return]<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [return]<br />
// SCOPE(b) = [return]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
return random() % 2 == 0 ? a : b;<br />
// RHS SCOPE(cond ? a : b) = MIN(SCOPE(a), SCOPE(b)) =<br />
// = MIN([return], [return]) = [return]<br />
// [return] := [return] => OK<br />
}<br />
<br />
// the inferred signature for T == string is then<br />
string chooseStringAtRandom(scope string a return, scope string b return);<br />
</source><br />
<br />
=== Calling functions ===<br />
<br />
<source lang="D"><br />
string global_string;<br />
void foo() {<br />
string[13] text = "Hello, world!";<br />
// => nothing, string[13] has no indirections<br />
string a;<br />
// => SCOPE(a) := [a] (incomplete)<br />
a = findSubstring(text, "world");<br />
// equivalent to: ... findSubstring(text[], ...)<br />
// text[] is returned, but SCOPE(text[]) is fixed<br />
// => no inference<br />
global_string = findSubstring(text, "world");<br />
// ditto<br />
string[5] s1 = "Hello";<br />
string b;<br />
// => SCOPE(b) := [b] (final)<br />
b = chooseStringAtRandom(s1, a);<br />
// SCOPE(s1[]) = [s1] (final)<br />
// `a` is returned and stored in `b`<br />
// => SCOPE(a) |= SCOPE(b) = [a] | [b] = [a] (final)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [a]<br />
// SCOPE(b) = [b]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
string[13] text = "Hello, world!";<br />
string a;<br />
a = findSubstring(text, "world");<br />
// equivalent to: ... findSubstring(text[], ...)<br />
// findSubstring takes both params as scope<br />
// text[]: [] := [text] => OK<br />
// "world": [] := [static] => OK<br />
// findSubstring returns haystack<br />
// => equivalent to `a = text[]`<br />
// return: [a] := [text] => OK<br />
global_string = findSubstring(text, "world");<br />
// [static] := [text] => BAD<br />
string[5] s1 = "Hello";<br />
string b;<br />
b = chooseStringAtRandom(s1, a);<br />
// [b] := MIN([s1], [a]) = [s1] => // OK<br />
// ---------------------------------------------------<br />
// => invalid assignment to global_string<br />
}<br />
</source><br />
<br />
=== Wrappers ===<br />
<br />
<source lang="D"><br />
string findSubstring(scope string haystack return, scope string needle);<br />
<br />
auto trace(alias func, Args...)(Args args) {<br />
import std.conv : to;<br />
writeln("Calling " ~ func.stringof ~ "(" ~ args.to!string ~ ")");<br />
return func(args);<br />
}<br />
<br />
auto s = trace!findSubstring(haystack, needle);<br />
<br />
// expands to (scope/return not yet in the signature):<br />
<br />
string trace_findSubstring(string haystack, string needle) {<br />
// => SCOPE(haystack) = [haystack] (incomplete)<br />
// => SCOPE(needle) = [needle] (final)<br />
import std.conv : to;<br />
// ignoring the writeln(), works as usual<br />
writeln("Calling findSubstring(...)");<br />
return findSubstring(haystack, needle);<br />
// `haystack` is assigned to the return value<br />
// => SCOPE(haystack) |= [return] = [haystack] | [return] = [return] (final)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(haystack) = [return]<br />
// SCOPE(needle) = [needle]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
return findSubstring(haystack, needle);<br />
// haystack: [] := [return] => OK<br />
// needle: [] := [needle] => OK<br />
// return: [return] := SCOPE(haystack) = [return] => OK<br />
}<br />
<br />
// inferred annotations:<br />
string trace_findSubstring(scope string haystack return, scope string needle);<br />
</source><br />
<br />
<source lang="D"><br />
auto filter(alias pred, T)(T input) {<br />
static struct FilterImpl {<br />
private scope T _input;<br />
private void skip() {<br />
while(!_input.empty && !pred(_input.front))<br />
_input.popFront();<br />
}<br />
@property ElementType!T front() {<br />
skip();<br />
return _input.front;<br />
}<br />
@property bool empty() {<br />
skip();<br />
return _input.empty;<br />
}<br />
void popFront() {<br />
_input.popFront();<br />
}<br />
}<br />
return FilterImpl(input);<br />
}<br />
<br />
alias odd = filter!(n => n % 2, int[]);<br />
<br />
// becomes (without annotations):<br />
<br />
auto odd_filter(int[] input) {<br />
static struct odd_FilterImpl {<br />
private scope int[] _input;<br />
// scope member is equivalent to:<br />
private @property ref prop_input() return { return _input; }<br />
private void skip() {<br />
// `empty`, `front` and `popFront` are template functions<br />
// => their scope/return annotations are inferred as needed<br />
// => they take _input as scope<br />
while(!_input.empty && !(_input.front % 2))<br />
_input.popFront();<br />
}<br />
@property int front() {<br />
// can be inferred to take `this` as scope<br />
skip();<br />
return _input.front;<br />
}<br />
@property bool empty() {<br />
// ditto<br />
skip();<br />
return _input.empty;<br />
}<br />
void popFront() {<br />
// ditto<br />
_input.popFront();<br />
}<br />
}<br />
return odd_FilterImpl(input);<br />
// equivalent to:<br />
odd_FilterImpl tmp;<br />
tmp._input = input;<br />
return tmp;<br />
// equivalent to:<br />
// => SCOPE(input) = [input]<br />
odd_FilterImpl tmp;<br />
// => SCOPE(tmp) = [tmp]<br />
ref tmp2 = tmp.prop_input;<br />
// => SCOPE(tmp2) = [tmp2]<br />
// call `tmp.prop_input`:<br />
// (`this` is returned and assigned to `tmp2`)<br />
// => SCOPE(tmp) |= SCOPE(tmp2)<br />
// => DEFER because tmp2 is incomplete<br />
// `tmp2` is a ref => assume it will be written to<br />
// => SCOPE(tmp2) |= SCOPE(tmp)<br />
// => DEFER because tmp is incomplete<br />
tmp2 = input;<br />
// `input` is assigned to `tmp2`<br />
// => SCOPE(input) |= SCOPE(tmp2)<br />
// => DEFER because tmp2 is incomplete<br />
return tmp;<br />
// => SCOPE(tmp) |= [return] = [return] (incomplete)<br />
// doesn't need to be deferred, because `|` is commutative<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(input) = [input]<br />
// SCOPE(tmp) = [return]<br />
// SCOPE(tmp2) = [tmp2]<br />
// unresolved:<br />
// SCOPE(tmp) |= SCOPE(tmp2)<br />
// SCOPE(tmp2) |= SCOPE(tmp)<br />
// SCOPE(input) |= SCOPE(tmp2)<br />
// resolve loop using union:<br />
// SCOPE(tmp) := [return] | [tmp2] = [return]<br />
// SCOPE(tmp2) := [return] | [tmp2] = [return]<br />
// unresolved:<br />
// SCOPE(input) |= SCOPE(tmp2)<br />
// resolve:<br />
// SCOPE(input) := [input] | [return] = [return]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
odd_FilterImpl tmp;<br />
ref tmp2 = tmp.prop_input;<br />
// call (`tmp` is the implicit `this` argument):<br />
// [] := [return] => OK<br />
// `tmp` is returned and assigned to `tmp2`<br />
// [return] := [return] => OK<br />
tmp2 = input;<br />
// [return] := [return] => OK<br />
return tmp;<br />
// [return] := [return] => OK<br />
}<br />
<br />
// inferred signature:<br />
// - SCOPE(input) is not [static]<br />
// => `input` can be `scope`<br />
// - SCOPE(input) refers to [return]<br />
// => `input` is annotated with `return`<br />
auto odd_filter(scope int[] input return);<br />
</source><br />
<br />
== Details ==<br />
<br />
<code>scope</code> is a storage class. It applies to function parameters (including <code>this</code>), local variables, the return value (treated as if it were an <code>out</code> parameter), and member variables of aggregates. It participates in overloading.<br />
<br />
With every variable, a particular lifetime (called '''scope''') is associated.<br />
It usually refers to a local variable or parameter; additionally, an infinite scope is defined that corresponds to global/static variables or GC managed memory.<br />
Scopes and lifetimes are defined purely based on lexical scope and order of declaration of their corresponding variables. Therefore, for any two lifetimes, one is either completely contained in the other, or they are disjoint.<br />
By annotating a variable with <code>scope</code>, it's scope is defined to be equal to the variable's lifetime, instead of the default (infinity).<br />
<br />
For any expression involving at least one scope value, two lifetimes (''LHS lifetime'' and ''RHS lifetime'') are computed in a way that ensures that the resulting RHS lifetime will not be greater than that of any of the expression's parts, and the LHS lifetime will not be smaller.<br />
The exact rules will be defined in one of the following sections.<br />
An '''assignment''' (i.e., <code>=</code> operator, passing to a function, returning from a function, capturing in a closure, throwing, etc.) involving scope values is only permissible, if the destination's LHS lifetime is fully contained in the source's RHS lifetime.<br />
Throwing is considered assignment to a variable with static lifetime.<br />
<br />
The following invariant is enforced to always be true on every assignment: A location with scope ''a'' will never contain references pointing to values with a lifetime shorter than ''a''.<br />
<br />
To allow a function to return a value it received as a parameter, the parameter can be annotated with the <code>return</code> keyword,<br />
as in [[DIP25]].<br />
It's also possible to express that a parameter escapes through another parameter (including <code>this</code>) by using the <code>return!identifier</code> syntax.<br />
Multiple such annotations can appear for each parameter.<br />
When a function is called, the compiler checks (for each argument and the return value) that only expressions with a lifetime longer than those of all the corresponding <code>return</code> annotations are passed in, and that the return value is used in a conforming way.<br />
<br />
Because all relevant information about lifetimes is contained in the function signature, no explicit <code>scope</code> annotations for local variables<br />
are necessary; the compiler can figure them out by itself.<br />
Additionally, inference of annotations is done in the usual situations, i.e. nested functions and templates.<br />
In a subsequent section, an algorithm is presented that can be used for inference of scopes of local variables as well as annotations of parameters.<br />
<br />
In parameters of <code>@safe</code> functions, all reference types (class references, pointers, slices, <code>ref</code> parameters) or aggregates containing references are implicitly treated as <code>scope</code> unless the parameter is annotated with the keyword <code>static</code>. This does however not apply to the return value.<br />
<br />
<code>ref</code> and <code>out</code> parameters can also be treated as implicitly scoped, but this has the potential to break lots of code and needs to be considered carefully.<br />
<br />
A <code>scope</code> annotation on a member variable shall be equivalent to a <code>@property</code> function returning a reference to that member, scoped to <code>this</code>.<br />
<br />
'''Borrowing''' is the term used for assignment of a value to a variable with narrower scope. This typically happens on function calls, when a value is passed as an argument to a parameter annotated with (or inferred as) <code>scope</code>.<br />
<br />
== Examples ==<br />
<br />
=== RCArray ===<br />
<br />
Walter's <code>RCArray</code>, adjusted to this proposal:<br />
<br />
<source lang="D"><br />
@safe:<br />
<br />
struct RCArray(E) {<br />
@disable this();<br />
<br />
this(E[] a)<br />
{<br />
array = a.dup;<br />
count = new int;<br />
*count = 1;<br />
}<br />
<br />
~this() @trusted<br />
{<br />
if (--*count == 0)<br />
{<br />
delete count;<br />
// either `delete` in `@system` code will accept `scope`:<br />
delete array;<br />
// or a helper needs to be used to remove `scope`:<br />
delete assumeStatic(array);<br />
}<br />
}<br />
<br />
this(this)<br />
{<br />
++*count;<br />
}<br />
<br />
@property size_t length()<br />
{<br />
return array.length;<br />
}<br />
<br />
ref E opIndex(size_t i)<br />
{<br />
return array[i];<br />
}<br />
<br />
E[] opSlice(size_t lwr, size_t upr)<br />
{<br />
return array[lwr .. upr];<br />
}<br />
<br />
E[] opSlice()<br />
{<br />
return array[];<br />
}<br />
<br />
private:<br />
scope E[] array; // this is the only explicit annotation<br />
// enforces treatment as `scope`, to avoid<br />
// accidental access as unscoped<br />
int* count;<br />
}<br />
</source><br />
<br />
== Implementation ==<br />
<br />
=== Scope inference ===<br />
<br />
This algorithm works at the function level. Scope inference for variables and parameters in one function is independent from all other functions.<br />
<br />
It takes as input a list of variables whose scope is already fixed (by explicit annotations) and another list whose scopes are to be inferred. It will choose the narrowest possible scopes for them for which the function will still compile. This is based on the observation that variables that are read from need to have a scope at least as large as their destination. Therefore, we can start with the smallest possible scope, the lifetime of the variable itself, and extend it to the scope of the destination, if it isn't already larger.<br />
<br />
----<br />
<br />
1. Let <code>Q</code> be a list of all variables whose scopes are to be inferred. This includes template function parameters not otherwise annotated and all local variables.<br />
<br />
2. Assign all elements of <code>Q</code> an initial scope equivalent to their own lifetime:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
var.scope := [var];<br />
}<br />
</source><br />
<br />
3. For each <code>ASSIGNMENT</code> whose <code>RHS_SCOPE</code> depends on a variable in <code>Q</code>, expand that variable's scope to at least the <code>LHS_SCOPE</code>. For all variables the <code>LHS_SCOPE</code> depends on and that are in <code>Q</code>, record a dependency:<br />
<source lang="D"><br />
foreach(ass; ASSIGNMENTS) {<br />
if(ass.rhs_scope.depends_on(Q)) {<br />
foreach(rhs_var; ass.rhs_scope.vars) {<br />
if(not rhs_var in Q)<br />
continue;<br />
foreach(lhs_var; ass.lhs_scope.vars) {<br />
rhs_var.scope |= ass.lhs_scope;<br />
if(lhs_var in Q)<br />
rhs_var.deps ~= lhs_var;<br />
}<br />
}<br />
}<br />
}<br />
</source><br />
<br />
4. Remove all variables from <code>Q</code> that have no dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
if(var.deps.empty)<br />
Q.remove(var);<br />
}<br />
</source><br />
<br />
5. If <code>Q</code> is empty, terminate, else remember length of <code>Q</code>:<br />
<source lang="D"><br />
if(Q.empty)<br />
return;<br />
old_Q_len := Q.length;<br />
</source><br />
<br />
6. Expand all variables' scopes to at least that of their dependencies; if a dependency has no dependencies itself, remove it from the variable's dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
foreach(dep; var.deps) {<br />
var.scope |= dep.scope;<br />
if(dep.deps.empty)<br />
var.deps.remove(dep);<br />
}<br />
}<br />
</source><br />
<br />
7. If the length changed, we made progress. We can repeat from step 4. Otherwise we have a dependency loop. Find a cycle (for example using [//en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm Tarjan's algorithm]). Collect all elements in the cycle, remove their dependencies from <code>DEPENDENCIES</code>, and assign them all the union of their scopes:<br />
<source lang="D"><br />
if(Q.length != old_Q_len)<br />
goto step4;<br />
cycle := tarjan(DEPENDENCIES);<br />
new_scope := []<br />
foreach(var; cycle) {<br />
new_scope |= var.scope;<br />
var.deps.remove_each(cycle);<br />
}<br />
foreach(var; cycle) {<br />
var.scope := new_scope;<br />
}<br />
</source><br />
<br />
8. Go to step 4.<br />
<br />
----<br />
<br />
At this point, each variable will have a scope assigned. Now, all assignment can be checked to verify that they never place a reference in a location that outlives the reference's target.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope3&diff=5843User:Schuetzm/scope32015-04-11T10:04:22Z<p>Schuetzm: Created page with "== Introduction == The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the functio..."</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. '''scope''' has no influence on overloading. However, a value passed as '''scope''' will not have its postblit or destructor called, it will only be bitblitted on copy and abandoned after use. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
// magic, to be explained later<br />
T borrow() return {<br />
return payload;<br />
}<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, postblit called<br />
// no destructor for `object` called<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no postblit called:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types can define an appropriate '''alias this''' to convert implicitly to a member (e.g. payload).<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates, nested functions and lambdas, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, int* b);<br />
</source><br />
<br />
Also, to keep the semantics of [[DIP25]], '''return''' and '''ref''' imply '''scope'''.<br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the source will vanish as soon as the reference through which it is accessed goes out of scope. This implies that only references with infinite lifetimes can be assigned through an indirection, while references read through an indirection get the same lifetime as the indirection itself. Violating these restrictions will result in @system code.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.</div>Schuetzmhttps://wiki.dlang.org/?title=Programming_in_D_for_Ruby_Programmers&diff=5833Programming in D for Ruby Programmers2015-04-08T08:54:32Z<p>Schuetzm: </p>
<hr />
<div>This page is under community development.<br />
<br />
In the meantime, you may find the following links of interest as a Ruby programmer:<br />
<br />
*[http://hawkins.io/2015/04/excited-about-d/ Ruby Programmer shares his excitement about D]<br />
*[http://bit.ly/1FcMtMm forum discussion of post]<br />
*[http://bit.ly/1aHqBMg Jacob Carlborg writes on embedding Ruby in a D application]<br />
*[https://rounin.livejournal.com/24639.html David Oftedal writes on Project Euler problem 61: from Ruby to D]<br />
*[https://www.google.de/search?q=ruby&domains=dlang.org&sourceid=google-search&sitesearch=forum.dlang.org&gws_rd=cr&ei=fT0kVdbAD8H6sAHFrYDwBQ#q=ruby+site:forum.dlang.org&domains=dlang.org&start=10 forum posts on Ruby]<br />
<br />
== Adding methods to existing classes / UFCS ==<br />
<br />
In Ruby, it's possible to add methods to existing classes, even builtin ones:<br />
<br />
<source lang="ruby"><br />
class String<br />
def underline<br />
puts self<br />
puts "=" * self.length<br />
end<br />
end<br />
<br />
"Hello, world!".underline<br />
<br />
# prints:<br />
# Hello, world!<br />
# =============<br />
</source><br />
<br />
Because it uses a static compilation model, it's not possible to do exactly the same in D. However, D allows calling a free function as if it were a method, which mostly has the same effect from a caller's point of view, and has the additional advantage of not polluting the type's namespace:<br />
<br />
<source lang="D"><br />
void underline(string s) {<br />
import std.stdio : writeln;<br />
import std.array : replicate;<br />
writeln(s);<br />
writeln("=".replicate(s.length));<br />
}<br />
<br />
void main() {<br />
"Hello, world!".underline();<br />
}<br />
</source></div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5720User:Schuetzm/scope22015-03-25T13:19:59Z<p>Schuetzm: /* scope inference */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) static {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() static {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
The rules for overloading are:<br />
* If only an overload accepting '''scope''' is defined, it is selected.<br />
* If only an overload accepting '''static''' (the default) is defined, it can only be called if the argument also has static scope.<br />
* If both overloads are defined, the static one is called for arguments with static scope, and the scope one for all others.<br />
<br />
Because scope is inferred for templates, we must explicitly specify '''static''' and '''scope''' if we want to overload on them.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler will infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5717User:Schuetzm/scope22015-03-22T17:33:42Z<p>Schuetzm: /* @safe-ty violations with borrowing */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) static {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() static {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
The rules for overloading are:<br />
* If only an overload accepting '''scope''' is defined, it is selected.<br />
* If only an overload accepting '''static''' (the default) is defined, it can only be called if the argument also has static scope.<br />
* If both overloads are defined, the static one is called for arguments with static scope, and the scope one for all others.<br />
<br />
Because scope is inferred for templates, we must explicitly specify '''static''' and '''scope''' if we want to overload on them.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them: they can make their code @trusted if they verify that it is indeed safe, but the compiler just can't know it, or they can rewrite it in a way that allows the compiler to proof the safety.<br />
<br />
The operations that are potentially unsafe are:<br />
* borrowing from a mutable global variable<br />
: Global variables can be accessed and therefore be mutated from anywhere.<br />
* re-borrowing from a mutable variable to which another borrowed reference is currently accessible<br />
: A @safe function can then assume that its parameters don't alias in a dangerous way.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5707User:Schuetzm/scope22015-03-16T17:19:11Z<p>Schuetzm: /* scope for value types & overloading */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) static {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() static {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
The rules for overloading are:<br />
* If only an overload accepting '''scope''' is defined, it is selected.<br />
* If only an overload accepting '''static''' (the default) is defined, it can only be called if the argument also has static scope.<br />
* If both overloads are defined, the static one is called for arguments with static scope, and the scope one for all others.<br />
<br />
Because scope is inferred for templates, we must explicitly specify '''static''' and '''scope''' if we want to overload on them.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5706User:Schuetzm/scope22015-03-16T17:16:06Z<p>Schuetzm: /* scope for value types & overloading */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
The rules for overloading are:<br />
* If only an overload accepting '''scope''' is defined, it is selected.<br />
* If only an overload accepting '''static''' (the default) is defined, it can only be called if the argument also has static scope.<br />
* If both overloads are defined, the static one is called for arguments with static scope, and the scope one for all others.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5705User:Schuetzm/scope22015-03-16T17:07:41Z<p>Schuetzm: /* Implicit scope and opt-out */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Postblits and destructors can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5703User:Schuetzm/scope22015-03-16T13:49:58Z<p>Schuetzm: /* Returning scoped parameters */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions, and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Postblits and destructors can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. If necessary, these annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
int* foo(<br />
scope int* input return return!output return!output2,<br />
int** output,<br />
out int* output2<br />
);<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5702User:Schuetzm/scope22015-03-16T13:47:14Z<p>Schuetzm: /* scope for value types & overloading */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions, and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Postblits and destructors can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5701User:Schuetzm/scope22015-03-15T20:22:35Z<p>Schuetzm: /* Returning scoped parameters */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions, and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental non-scoped access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5700User:Schuetzm/scope22015-03-15T20:22:00Z<p>Schuetzm: /* Implicit scope and opt-out */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' implies '''scope''' even in @system functions, and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5699User:Schuetzm/scope22015-03-15T13:59:19Z<p>Schuetzm: </p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' also implies '''scope''', and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.</div>Schuetzmhttps://wiki.dlang.org/?title=User_talk:Schuetzm/scope2&diff=5698User talk:Schuetzm/scope22015-03-15T13:59:01Z<p>Schuetzm: </p>
<hr />
<div>== Example for the inference algorithm ==<br />
<br />
=== Dereferencing yields <code>static</code> scope ===<br />
<br />
Note that this is depending on whether we're on the LHS or RHS of an assignment. The inference algorithm only assigns scopes from LHS to RHS; therefore we always use <code>static</code>. On the RHS, a dereference gets the scope of the reference it's been reached through, because we at least known that it can't have a shorter lifetime than that.<br />
<br />
Dereferencing includes the dereference operator, implicit dereferencing of struct/union pointers, of class references, as well as indexing and slicing.<br />
<br />
Applied to the function deadalnix used to demonstrate the rvalue/lvalue problem:<br />
<br />
(1) <code>foo()</code> is <code>@safe</code>, making its param scoped:<br />
<br />
<source lang="D"><br />
void foo(scope int** a) {<br />
// no inference for `a` => self-owned<br />
// ("mystery scope" in your terms)<br />
// => SCOPE(a) := [a] (final)<br />
int** b;<br />
// => SCOPE(b) := [b] (final)<br />
b = a;<br />
// scope of `a` is fixed => do nothing<br />
int d;<br />
int* c;<br />
// => SCOPE(c) := [c] (incomplete)<br />
c = &d;<br />
*b = c;<br />
// assignment from `c`:<br />
// => SCOPE(c) |= SCOPE(*b) = [c] | [static] (final)<br />
// (dereferencing loses information => static)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [a]<br />
// SCOPE(b) = [b]<br />
// SCOPE(c) = [static]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
int** b = a; // [b] := [a] => OK<br />
int d;<br />
int* c = &d; // [static] := [d] => BAD<br />
*b = c; // [static] := [static] => OK<br />
// => invalid assignment<br />
// note how it even traces where the bad value comes from<br />
}<br />
</source><br />
<br />
(2) <code>foo()</code> is in a template, param scope gets inferred<br />
<br />
<source lang="D"><br />
void foo(T)(T** a) {<br />
// start with empty scope for params<br />
// => SCOPE(a) := [a] (incomplete)<br />
T** b;<br />
// start with empty scope for locals<br />
// => SCOPE(b) := [b] (final)<br />
b = a;<br />
// only access to `a`:<br />
// => SCOPE(a) |= SCOPE(b) = [a] | [b] = [a]<br />
T d;<br />
T* c;<br />
// => SCOPE(c) = [c] (incomplete)<br />
c = &d;<br />
*b = c;<br />
// assignment from `c`:<br />
// => SCOPE(c) |= SCOPE(*b) = [c] | [static] = [static]<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [a]<br />
// SCOPE(b) = [b]<br />
// SCOPE(c) = [static]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
int** b = a; // [b] := [a] => OK<br />
int d;<br />
int* c = &d; // [static] := [d] => BAD<br />
*b = c; // [static] := [static] => OK<br />
// => invalid assignment<br />
// again, the culprit can be detected<br />
}<br />
</source><br />
<br />
(3) <code>foo()</code> takes unscoped pointer<br />
<br />
<source lang="D"><br />
void foo(int** a) {<br />
// no inference for `a` => static<br />
// => SCOPE(a) := [static] (final)<br />
int** b;<br />
// => SCOPE(b) := [b] (final)<br />
b = a;<br />
// scope of `a` is fixed => do nothing<br />
int d;<br />
int* c;<br />
// => SCOPE(c) := [c] (incomplete)<br />
c = &d;<br />
*b = c;<br />
// assignment from `c`:<br />
// => SCOPE(c) |= SCOPE(*b) = [c] | [static] (final)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [static]<br />
// SCOPE(b) = [b]<br />
// SCOPE(c) = [static]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
int** b = a; // [b] := [static] => OK<br />
int d;<br />
int* c = &d; // [static] := [d] => BAD<br />
*b = c; // [static] := [static] => OK<br />
// => invalid assignment<br />
// escape detection for locals works even without scope params<br />
// and in @system code<br />
}<br />
</source><br />
<br />
=== <code>return</code> inference ===<br />
<br />
<source lang="D"><br />
T[] findSubstring(T)(T[] haystack, T[] needle) {<br />
// find first occurrence of substring<br />
// naive implementation<br />
<br />
// => SCOPE(return) := [return] (fixed)<br />
// [return] is a scope higher than all params<br />
// => SCOPE(haystack) := [haystack] (incomplete)<br />
// => SCOPE(needle) := [needle] (final)<br />
<br />
for(int i = 0; i < haystack.length - needle.length; i++) {<br />
T[] sub;<br />
// => SCOPE(sub) := [sub] (incomplete)<br />
sub = haystack[i .. i + needle.length];<br />
// haystack and needle are read again; ignoring from now on<br />
// => SCOPE(haystack) |= SCOPE(sub)<br />
// => DEFER because SCOPE(sub) is incomplete<br />
if(sub == needle) {<br />
return haystack[i .. $];<br />
// => SCOPE(haystack) |= SCOPE(return) = [haystack] | [return] = [return]<br />
}<br />
}<br />
return null;<br />
// nothing to do here, `null` has fixed static scope<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(haystack) = [return]<br />
// SCOPE(needle) = [needle]<br />
// SCOPE(sub) = [sub]<br />
// unresolved:<br />
// SCOPE(haystack) |= SCOPE(sub)<br />
// resolve:<br />
// SCOPE(haystack) := [return] | [sub] = [return]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
for(int i = 0; i < haystack.length - needle.length; i++) {<br />
T[] sub;<br />
sub = haystack[i .. i + needle.length];<br />
// [sub] := [return] => OK<br />
if(sub == needle) {<br />
// equivalent to: if(opEquals(sub, needle))<br />
// let's assume `opEquals` takes both params by scope<br />
// sub: [] := [sub] => OK<br />
// needle: [] := [needle] => OK<br />
return haystack[i .. $];<br />
// [return] := [return] => OK<br />
}<br />
}<br />
return null;<br />
// [return] := [static] => OK<br />
}<br />
<br />
// the inferred signature for T == immutable(char) is then<br />
string findSubstring(scope string haystack return, scope string needle);<br />
</source><br />
<br />
Multiple <code>return</code> params:<br />
<br />
<source lang="D"><br />
T chooseStringAtRandom(T)(T a, T b) {<br />
// => SCOPE(a) = [a] (incomplete)<br />
// => SCOPE(b) = [a] (incomplete)<br />
return random() % 2 == 0 ? a : b;<br />
// => SCOPE(a) |= [a] = [a]<br />
// => SCOPE(b) |= [b] = [b]<br />
// ?: operator affects both `a` and `b`<br />
// => SCOPE(a) |= SCOPE(return) = [a] | [return] = [return]<br />
// => SCOPE(b) |= SCOPE(return) = [b] | [return] = [return]<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [return]<br />
// SCOPE(b) = [return]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
return random() % 2 == 0 ? a : b;<br />
// RHS SCOPE(cond ? a : b) = MIN(SCOPE(a), SCOPE(b)) =<br />
// = MIN([return], [return]) = [return]<br />
// [return] := [return] => OK<br />
}<br />
<br />
// the inferred signature for T == string is then<br />
string chooseStringAtRandom(scope string a return, scope string b return);<br />
</source><br />
<br />
=== Calling functions ===<br />
<br />
<source lang="D"><br />
string global_string;<br />
void foo() {<br />
string[13] text = "Hello, world!";<br />
// => nothing, string[13] has no indirections<br />
string a;<br />
// => SCOPE(a) := [a] (incomplete)<br />
a = findSubstring(text, "world");<br />
// equivalent to: ... findSubstring(text[], ...)<br />
// text[] is returned, but SCOPE(text[]) is fixed<br />
// => no inference<br />
global_string = findSubstring(text, "world");<br />
// ditto<br />
string[5] s1 = "Hello";<br />
string b;<br />
// => SCOPE(b) := [b] (final)<br />
b = chooseStringAtRandom(s1, a);<br />
// SCOPE(s1[]) = [s1] (final)<br />
// `a` is returned and stored in `b`<br />
// => SCOPE(a) |= SCOPE(b) = [a] | [b] = [a] (final)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(a) = [a]<br />
// SCOPE(b) = [b]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
string[13] text = "Hello, world!";<br />
string a;<br />
a = findSubstring(text, "world");<br />
// equivalent to: ... findSubstring(text[], ...)<br />
// findSubstring takes both params as scope<br />
// text[]: [] := [text] => OK<br />
// "world": [] := [static] => OK<br />
// findSubstring returns haystack<br />
// => equivalent to `a = text[]`<br />
// return: [a] := [text] => OK<br />
global_string = findSubstring(text, "world");<br />
// [static] := [text] => BAD<br />
string[5] s1 = "Hello";<br />
string b;<br />
b = chooseStringAtRandom(s1, a);<br />
// [b] := MIN([s1], [a]) = [s1] => // OK<br />
// ---------------------------------------------------<br />
// => invalid assignment to global_string<br />
}<br />
</source><br />
<br />
=== Wrappers ===<br />
<br />
<source lang="D"><br />
string findSubstring(scope string haystack return, scope string needle);<br />
<br />
auto trace(alias func, Args...)(Args args) {<br />
import std.conv : to;<br />
writeln("Calling " ~ func.stringof ~ "(" ~ args.to!string ~ ")");<br />
return func(args);<br />
}<br />
<br />
auto s = trace!findSubstring(haystack, needle);<br />
<br />
// expands to (scope/return not yet in the signature):<br />
<br />
string trace_findSubstring(string haystack, string needle) {<br />
// => SCOPE(haystack) = [haystack] (incomplete)<br />
// => SCOPE(needle) = [needle] (final)<br />
import std.conv : to;<br />
// ignoring the writeln(), works as usual<br />
writeln("Calling findSubstring(...)");<br />
return findSubstring(haystack, needle);<br />
// `haystack` is assigned to the return value<br />
// => SCOPE(haystack) |= [return] = [haystack] | [return] = [return] (final)<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(haystack) = [return]<br />
// SCOPE(needle) = [needle]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
return findSubstring(haystack, needle);<br />
// haystack: [] := [return] => OK<br />
// needle: [] := [needle] => OK<br />
// return: [return] := SCOPE(haystack) = [return] => OK<br />
}<br />
<br />
// inferred annotations:<br />
string trace_findSubstring(scope string haystack return, scope string needle);<br />
</source><br />
<br />
<source lang="D"><br />
auto filter(alias pred, T)(T input) {<br />
static struct FilterImpl {<br />
private scope T _input;<br />
private void skip() {<br />
while(!_input.empty && !pred(_input.front))<br />
_input.popFront();<br />
}<br />
@property ElementType!T front() {<br />
skip();<br />
return _input.front;<br />
}<br />
@property bool empty() {<br />
skip();<br />
return _input.empty;<br />
}<br />
void popFront() {<br />
_input.popFront();<br />
}<br />
}<br />
return FilterImpl(input);<br />
}<br />
<br />
alias odd = filter!(n => n % 2, int[]);<br />
<br />
// becomes (without annotations):<br />
<br />
auto odd_filter(int[] input) {<br />
static struct odd_FilterImpl {<br />
private scope int[] _input;<br />
// scope member is equivalent to:<br />
private @property ref prop_input() return { return _input; }<br />
private void skip() {<br />
// `empty`, `front` and `popFront` are template functions<br />
// => their scope/return annotations are inferred as needed<br />
// => they take _input as scope<br />
while(!_input.empty && !(_input.front % 2))<br />
_input.popFront();<br />
}<br />
@property int front() {<br />
// can be inferred to take `this` as scope<br />
skip();<br />
return _input.front;<br />
}<br />
@property bool empty() {<br />
// ditto<br />
skip();<br />
return _input.empty;<br />
}<br />
void popFront() {<br />
// ditto<br />
_input.popFront();<br />
}<br />
}<br />
return odd_FilterImpl(input);<br />
// equivalent to:<br />
odd_FilterImpl tmp;<br />
tmp._input = input;<br />
return tmp;<br />
// equivalent to:<br />
// => SCOPE(input) = [input]<br />
odd_FilterImpl tmp;<br />
// => SCOPE(tmp) = [tmp]<br />
ref tmp2 = tmp.prop_input;<br />
// => SCOPE(tmp2) = [tmp2]<br />
// call `tmp.prop_input`:<br />
// (`this` is returned and assigned to `tmp2`)<br />
// => SCOPE(tmp) |= SCOPE(tmp2)<br />
// => DEFER because tmp2 is incomplete<br />
// `tmp2` is a ref => assume it will be written to<br />
// => SCOPE(tmp2) |= SCOPE(tmp)<br />
// => DEFER because tmp is incomplete<br />
tmp2 = input;<br />
// `input` is assigned to `tmp2`<br />
// => SCOPE(input) |= SCOPE(tmp2)<br />
// => DEFER because tmp2 is incomplete<br />
return tmp;<br />
// => SCOPE(tmp) |= [return] = [return] (incomplete)<br />
// doesn't need to be deferred, because `|` is commutative<br />
// ---------------------------------------------------<br />
// we now have:<br />
// SCOPE(input) = [input]<br />
// SCOPE(tmp) = [return]<br />
// SCOPE(tmp2) = [tmp2]<br />
// unresolved:<br />
// SCOPE(tmp) |= SCOPE(tmp2)<br />
// SCOPE(tmp2) |= SCOPE(tmp)<br />
// SCOPE(input) |= SCOPE(tmp2)<br />
// resolve loop using union:<br />
// SCOPE(tmp) := [return] | [tmp2] = [return]<br />
// SCOPE(tmp2) := [return] | [tmp2] = [return]<br />
// unresolved:<br />
// SCOPE(input) |= SCOPE(tmp2)<br />
// resolve:<br />
// SCOPE(input) := [input] | [return] = [return]<br />
// => all resolved<br />
// ---------------------------------------------------<br />
// => satisfiable, now let's check the assignments:<br />
<br />
odd_FilterImpl tmp;<br />
ref tmp2 = tmp.prop_input;<br />
// call (`tmp` is the implicit `this` argument):<br />
// [] := [return] => OK<br />
// `tmp` is returned and assigned to `tmp2`<br />
// [return] := [return] => OK<br />
tmp2 = input;<br />
// [return] := [return] => OK<br />
return tmp;<br />
// [return] := [return] => OK<br />
}<br />
<br />
// inferred signature:<br />
// - SCOPE(input) is not [static]<br />
// => `input` can be `scope`<br />
// - SCOPE(input) refers to [return]<br />
// => `input` is annotated with `return`<br />
auto odd_filter(scope int[] input return);<br />
</source><br />
<br />
== Details ==<br />
<br />
<code>scope</code> is a storage class. It applies to function parameters (including <code>this</code>), local variables, the return value (treated as if it were an <code>out</code> parameter), and member variables of aggregates. It participates in overloading.<br />
<br />
With every variable, a particular lifetime (called '''scope''') is associated.<br />
It usually refers to a local variable or parameter; additionally, an infinite scope is defined that corresponds to global/static variables or GC managed memory.<br />
Scopes and lifetimes are defined purely based on lexical scope and order of declaration of their corresponding variables. Therefore, for any two lifetimes, one is either completely contained in the other, or they are disjoint.<br />
By annotating a variable with <code>scope</code>, it's scope is defined to be equal to the variable's lifetime, instead of the default (infinity).<br />
<br />
For any expression involving at least one scope value, two lifetimes (''LHS lifetime'' and ''RHS lifetime'') are computed in a way that ensures that the resulting RHS lifetime will not be greater than that of any of the expression's parts, and the LHS lifetime will not be smaller.<br />
The exact rules will be defined in one of the following sections.<br />
An '''assignment''' (i.e., <code>=</code> operator, passing to a function, returning from a function, capturing in a closure, throwing, etc.) involving scope values is only permissible, if the destination's LHS lifetime is fully contained in the source's RHS lifetime.<br />
Throwing is considered assignment to a variable with static lifetime.<br />
<br />
The following invariant is enforced to always be true on every assignment: A location with scope ''a'' will never contain references pointing to values with a lifetime shorter than ''a''.<br />
<br />
To allow a function to return a value it received as a parameter, the parameter can be annotated with the <code>return</code> keyword,<br />
as in [[DIP25]].<br />
It's also possible to express that a parameter escapes through another parameter (including <code>this</code>) by using the <code>return!identifier</code> syntax.<br />
Multiple such annotations can appear for each parameter.<br />
When a function is called, the compiler checks (for each argument and the return value) that only expressions with a lifetime longer than those of all the corresponding <code>return</code> annotations are passed in, and that the return value is used in a conforming way.<br />
<br />
Because all relevant information about lifetimes is contained in the function signature, no explicit <code>scope</code> annotations for local variables<br />
are necessary; the compiler can figure them out by itself.<br />
Additionally, inference of annotations is done in the usual situations, i.e. nested functions and templates.<br />
In a subsequent section, an algorithm is presented that can be used for inference of scopes of local variables as well as annotations of parameters.<br />
<br />
In parameters of <code>@safe</code> functions, all reference types (class references, pointers, slices, <code>ref</code> parameters) or aggregates containing references are implicitly treated as <code>scope</code> unless the parameter is annotated with the keyword <code>static</code>. This does however not apply to the return value.<br />
<br />
<code>ref</code> and <code>out</code> parameters can also be treated as implicitly scoped, but this has the potential to break lots of code and needs to be considered carefully.<br />
<br />
A <code>scope</code> annotation on a member variable shall be equivalent to a <code>@property</code> function returning a reference to that member, scoped to <code>this</code>.<br />
<br />
'''Borrowing''' is the term used for assignment of a value to a variable with narrower scope. This typically happens on function calls, when a value is passed as an argument to a parameter annotated with (or inferred as) <code>scope</code>.<br />
<br />
== Examples ==<br />
<br />
=== RCArray ===<br />
<br />
Walter's <code>RCArray</code>, adjusted to this proposal:<br />
<br />
<source lang="D"><br />
@safe:<br />
<br />
struct RCArray(E) {<br />
@disable this();<br />
<br />
this(E[] a)<br />
{<br />
array = a.dup;<br />
count = new int;<br />
*count = 1;<br />
}<br />
<br />
~this() @trusted<br />
{<br />
if (--*count == 0)<br />
{<br />
delete count;<br />
// either `delete` in `@system` code will accept `scope`:<br />
delete array;<br />
// or a helper needs to be used to remove `scope`:<br />
delete assumeStatic(array);<br />
}<br />
}<br />
<br />
this(this)<br />
{<br />
++*count;<br />
}<br />
<br />
@property size_t length()<br />
{<br />
return array.length;<br />
}<br />
<br />
ref E opIndex(size_t i)<br />
{<br />
return array[i];<br />
}<br />
<br />
E[] opSlice(size_t lwr, size_t upr)<br />
{<br />
return array[lwr .. upr];<br />
}<br />
<br />
E[] opSlice()<br />
{<br />
return array[];<br />
}<br />
<br />
private:<br />
scope E[] array; // this is the only explicit annotation<br />
// enforces treatment as `scope`, to avoid<br />
// accidental access as unscoped<br />
int* count;<br />
}<br />
</source><br />
<br />
== Implementation ==<br />
<br />
=== Scope inference ===<br />
<br />
This algorithm works at the function level. Scope inference for variables and parameters in one function is independent from all other functions.<br />
<br />
It takes as input a list of variables whose scope is already fixed (by explicit annotations) and another list whose scopes are to be inferred. It will choose the narrowest possible scopes for them for which the function will still compile. This is based on the observation that variables that are read from need to have a scope at least as large as their destination. Therefore, we can start with the smallest possible scope, the lifetime of the variable itself, and extend it to the scope of the destination, if it isn't already larger.<br />
<br />
----<br />
<br />
1. Let <code>Q</code> be a list of all variables whose scopes are to be inferred. This includes template function parameters not otherwise annotated and all local variables.<br />
<br />
2. Assign all elements of <code>Q</code> an initial scope equivalent to their own lifetime:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
var.scope := [var];<br />
}<br />
</source><br />
<br />
3. For each <code>ASSIGNMENT</code> whose <code>RHS_SCOPE</code> depends on a variable in <code>Q</code>, expand that variable's scope to at least the <code>LHS_SCOPE</code>. For all variables the <code>LHS_SCOPE</code> depends on and that are in <code>Q</code>, record a dependency:<br />
<source lang="D"><br />
foreach(ass; ASSIGNMENTS) {<br />
if(ass.rhs_scope.depends_on(Q)) {<br />
foreach(rhs_var; ass.rhs_scope.vars) {<br />
if(not rhs_var in Q)<br />
continue;<br />
foreach(lhs_var; ass.lhs_scope.vars) {<br />
rhs_var.scope |= ass.lhs_scope;<br />
if(lhs_var in Q)<br />
rhs_var.deps ~= lhs_var;<br />
}<br />
}<br />
}<br />
}<br />
</source><br />
<br />
4. Remove all variables from <code>Q</code> that have no dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
if(var.deps.empty)<br />
Q.remove(var);<br />
}<br />
</source><br />
<br />
5. If <code>Q</code> is empty, terminate, else remember length of <code>Q</code>:<br />
<source lang="D"><br />
if(Q.empty)<br />
return;<br />
old_Q_len := Q.length;<br />
</source><br />
<br />
6. Expand all variables' scopes to at least that of their dependencies; if a dependency has no dependencies itself, remove it from the variable's dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
foreach(dep; var.deps) {<br />
var.scope |= dep.scope;<br />
if(dep.deps.empty)<br />
var.deps.remove(dep);<br />
}<br />
}<br />
</source><br />
<br />
7. If the length changed, we made progress. We can repeat from step 4. Otherwise we have a dependency loop. Find a cycle (for example using [//en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm Tarjan's algorithm]). Collect all elements in the cycle, remove their dependencies from <code>DEPENDENCIES</code>, and assign them all the union of their scopes:<br />
<source lang="D"><br />
if(Q.length != old_Q_len)<br />
goto step4;<br />
cycle := tarjan(DEPENDENCIES);<br />
new_scope := []<br />
foreach(var; cycle) {<br />
new_scope |= var.scope;<br />
var.deps.remove_each(cycle);<br />
}<br />
foreach(var; cycle) {<br />
var.scope := new_scope;<br />
}<br />
</source><br />
<br />
8. Go to step 4.<br />
<br />
----<br />
<br />
At this point, each variable will have a scope assigned. Now, all assignment can be checked to verify that they never place a reference in a location that outlives the reference's target.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5697User:Schuetzm/scope22015-03-15T13:48:45Z<p>Schuetzm: /* Overview */</p>
<hr />
<div>== Introduction ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations. Credits are due to the authors of that DIP, ''Andrei'' and ''Walter'', then to ''Zach the Mystic'' who had the idea to generalize DIP25 as well as provided inspiration for the inference algorithm, ''deadalnix'' for his many valuable arguments, for example pointing out the intricacies of handling multiple indirections safely, and various other members of the community who provided useful contributions in past discussions in the news groups.<br />
<br />
== Overview ==<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' also implies '''scope''', and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.<br />
<br />
== Details ==<br />
<br />
<code>scope</code> is a storage class. It applies to function parameters (including <code>this</code>), local variables, the return value (treated as if it were an <code>out</code> parameter), and member variables of aggregates. It participates in overloading.<br />
<br />
With every variable, a particular lifetime (called '''scope''') is associated.<br />
It usually refers to a local variable or parameter; additionally, an infinite scope is defined that corresponds to global/static variables or GC managed memory.<br />
Scopes and lifetimes are defined purely based on lexical scope and order of declaration of their corresponding variables. Therefore, for any two lifetimes, one is either completely contained in the other, or they are disjoint.<br />
By annotating a variable with <code>scope</code>, it's scope is defined to be equal to the variable's lifetime, instead of the default (infinity).<br />
<br />
For any expression involving at least one scope value, two lifetimes (''LHS lifetime'' and ''RHS lifetime'') are computed in a way that ensures that the resulting RHS lifetime will not be greater than that of any of the expression's parts, and the LHS lifetime will not be smaller.<br />
The exact rules will be defined in one of the following sections.<br />
An '''assignment''' (i.e., <code>=</code> operator, passing to a function, returning from a function, capturing in a closure, throwing, etc.) involving scope values is only permissible, if the destination's LHS lifetime is fully contained in the source's RHS lifetime.<br />
Throwing is considered assignment to a variable with static lifetime.<br />
<br />
The following invariant is enforced to always be true on every assignment: A location with scope ''a'' will never contain references pointing to values with a lifetime shorter than ''a''.<br />
<br />
To allow a function to return a value it received as a parameter, the parameter can be annotated with the <code>return</code> keyword,<br />
as in [[DIP25]].<br />
It's also possible to express that a parameter escapes through another parameter (including <code>this</code>) by using the <code>return!identifier</code> syntax.<br />
Multiple such annotations can appear for each parameter.<br />
When a function is called, the compiler checks (for each argument and the return value) that only expressions with a lifetime longer than those of all the corresponding <code>return</code> annotations are passed in, and that the return value is used in a conforming way.<br />
<br />
Because all relevant information about lifetimes is contained in the function signature, no explicit <code>scope</code> annotations for local variables<br />
are necessary; the compiler can figure them out by itself.<br />
Additionally, inference of annotations is done in the usual situations, i.e. nested functions and templates.<br />
In a subsequent section, an algorithm is presented that can be used for inference of scopes of local variables as well as annotations of parameters.<br />
<br />
In parameters of <code>@safe</code> functions, all reference types (class references, pointers, slices, <code>ref</code> parameters) or aggregates containing references are implicitly treated as <code>scope</code> unless the parameter is annotated with the keyword <code>static</code>. This does however not apply to the return value.<br />
<br />
<code>ref</code> and <code>out</code> parameters can also be treated as implicitly scoped, but this has the potential to break lots of code and needs to be considered carefully.<br />
<br />
A <code>scope</code> annotation on a member variable shall be equivalent to a <code>@property</code> function returning a reference to that member, scoped to <code>this</code>.<br />
<br />
'''Borrowing''' is the term used for assignment of a value to a variable with narrower scope. This typically happens on function calls, when a value is passed as an argument to a parameter annotated with (or inferred as) <code>scope</code>.<br />
<br />
== Examples ==<br />
<br />
=== RCArray ===<br />
<br />
Walter's <code>RCArray</code>, adjusted to this proposal:<br />
<br />
<source lang="D"><br />
@safe:<br />
<br />
struct RCArray(E) {<br />
@disable this();<br />
<br />
this(E[] a)<br />
{<br />
array = a.dup;<br />
count = new int;<br />
*count = 1;<br />
}<br />
<br />
~this() @trusted<br />
{<br />
if (--*count == 0)<br />
{<br />
delete count;<br />
// either `delete` in `@system` code will accept `scope`:<br />
delete array;<br />
// or a helper needs to be used to remove `scope`:<br />
delete assumeStatic(array);<br />
}<br />
}<br />
<br />
this(this)<br />
{<br />
++*count;<br />
}<br />
<br />
@property size_t length()<br />
{<br />
return array.length;<br />
}<br />
<br />
ref E opIndex(size_t i)<br />
{<br />
return array[i];<br />
}<br />
<br />
E[] opSlice(size_t lwr, size_t upr)<br />
{<br />
return array[lwr .. upr];<br />
}<br />
<br />
E[] opSlice()<br />
{<br />
return array[];<br />
}<br />
<br />
private:<br />
scope E[] array; // this is the only explicit annotation<br />
// enforces treatment as `scope`, to avoid<br />
// accidental access as unscoped<br />
int* count;<br />
}<br />
</source><br />
<br />
== Implementation ==<br />
<br />
=== Scope inference ===<br />
<br />
This algorithm works at the function level. Scope inference for variables and parameters in one function is independent from all other functions.<br />
<br />
It takes as input a list of variables whose scope is already fixed (by explicit annotations) and another list whose scopes are to be inferred. It will choose the narrowest possible scopes for them for which the function will still compile. This is based on the observation that variables that are read from need to have a scope at least as large as their destination. Therefore, we can start with the smallest possible scope, the lifetime of the variable itself, and extend it to the scope of the destination, if it isn't already larger.<br />
<br />
----<br />
<br />
1. Let <code>Q</code> be a list of all variables whose scopes are to be inferred. This includes template function parameters not otherwise annotated and all local variables.<br />
<br />
2. Assign all elements of <code>Q</code> an initial scope equivalent to their own lifetime:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
var.scope := [var];<br />
}<br />
</source><br />
<br />
3. For each <code>ASSIGNMENT</code> whose <code>RHS_SCOPE</code> depends on a variable in <code>Q</code>, expand that variable's scope to at least the <code>LHS_SCOPE</code>. For all variables the <code>LHS_SCOPE</code> depends on and that are in <code>Q</code>, record a dependency:<br />
<source lang="D"><br />
foreach(ass; ASSIGNMENTS) {<br />
if(ass.rhs_scope.depends_on(Q)) {<br />
foreach(rhs_var; ass.rhs_scope.vars) {<br />
if(not rhs_var in Q)<br />
continue;<br />
foreach(lhs_var; ass.lhs_scope.vars) {<br />
rhs_var.scope |= ass.lhs_scope;<br />
if(lhs_var in Q)<br />
rhs_var.deps ~= lhs_var;<br />
}<br />
}<br />
}<br />
}<br />
</source><br />
<br />
4. Remove all variables from <code>Q</code> that have no dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
if(var.deps.empty)<br />
Q.remove(var);<br />
}<br />
</source><br />
<br />
5. If <code>Q</code> is empty, terminate, else remember length of <code>Q</code>:<br />
<source lang="D"><br />
if(Q.empty)<br />
return;<br />
old_Q_len := Q.length;<br />
</source><br />
<br />
6. Expand all variables' scopes to at least that of their dependencies; if a dependency has no dependencies itself, remove it from the variable's dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
foreach(dep; var.deps) {<br />
var.scope |= dep.scope;<br />
if(dep.deps.empty)<br />
var.deps.remove(dep);<br />
}<br />
}<br />
</source><br />
<br />
7. If the length changed, we made progress. We can repeat from step 4. Otherwise we have a dependency loop. Find a cycle (for example using [//en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm Tarjan's algorithm]). Collect all elements in the cycle, remove their dependencies from <code>DEPENDENCIES</code>, and assign them all the union of their scopes:<br />
<source lang="D"><br />
if(Q.length != old_Q_len)<br />
goto step4;<br />
cycle := tarjan(DEPENDENCIES);<br />
new_scope := []<br />
foreach(var; cycle) {<br />
new_scope |= var.scope;<br />
var.deps.remove_each(cycle);<br />
}<br />
foreach(var; cycle) {<br />
var.scope := new_scope;<br />
}<br />
</source><br />
<br />
8. Go to step 4.<br />
<br />
----<br />
<br />
At this point, each variable will have a scope assigned. Now, all assignment can be checked to verify that they never place a reference in a location that outlives the reference's target.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5696User:Schuetzm/scope22015-03-15T13:41:25Z<p>Schuetzm: /* Multiple indirections */</p>
<hr />
<div>== Overview ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations.<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' also implies '''scope''', and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behind more than one indirection. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.<br />
<br />
== Details ==<br />
<br />
<code>scope</code> is a storage class. It applies to function parameters (including <code>this</code>), local variables, the return value (treated as if it were an <code>out</code> parameter), and member variables of aggregates. It participates in overloading.<br />
<br />
With every variable, a particular lifetime (called '''scope''') is associated.<br />
It usually refers to a local variable or parameter; additionally, an infinite scope is defined that corresponds to global/static variables or GC managed memory.<br />
Scopes and lifetimes are defined purely based on lexical scope and order of declaration of their corresponding variables. Therefore, for any two lifetimes, one is either completely contained in the other, or they are disjoint.<br />
By annotating a variable with <code>scope</code>, it's scope is defined to be equal to the variable's lifetime, instead of the default (infinity).<br />
<br />
For any expression involving at least one scope value, two lifetimes (''LHS lifetime'' and ''RHS lifetime'') are computed in a way that ensures that the resulting RHS lifetime will not be greater than that of any of the expression's parts, and the LHS lifetime will not be smaller.<br />
The exact rules will be defined in one of the following sections.<br />
An '''assignment''' (i.e., <code>=</code> operator, passing to a function, returning from a function, capturing in a closure, throwing, etc.) involving scope values is only permissible, if the destination's LHS lifetime is fully contained in the source's RHS lifetime.<br />
Throwing is considered assignment to a variable with static lifetime.<br />
<br />
The following invariant is enforced to always be true on every assignment: A location with scope ''a'' will never contain references pointing to values with a lifetime shorter than ''a''.<br />
<br />
To allow a function to return a value it received as a parameter, the parameter can be annotated with the <code>return</code> keyword,<br />
as in [[DIP25]].<br />
It's also possible to express that a parameter escapes through another parameter (including <code>this</code>) by using the <code>return!identifier</code> syntax.<br />
Multiple such annotations can appear for each parameter.<br />
When a function is called, the compiler checks (for each argument and the return value) that only expressions with a lifetime longer than those of all the corresponding <code>return</code> annotations are passed in, and that the return value is used in a conforming way.<br />
<br />
Because all relevant information about lifetimes is contained in the function signature, no explicit <code>scope</code> annotations for local variables<br />
are necessary; the compiler can figure them out by itself.<br />
Additionally, inference of annotations is done in the usual situations, i.e. nested functions and templates.<br />
In a subsequent section, an algorithm is presented that can be used for inference of scopes of local variables as well as annotations of parameters.<br />
<br />
In parameters of <code>@safe</code> functions, all reference types (class references, pointers, slices, <code>ref</code> parameters) or aggregates containing references are implicitly treated as <code>scope</code> unless the parameter is annotated with the keyword <code>static</code>. This does however not apply to the return value.<br />
<br />
<code>ref</code> and <code>out</code> parameters can also be treated as implicitly scoped, but this has the potential to break lots of code and needs to be considered carefully.<br />
<br />
A <code>scope</code> annotation on a member variable shall be equivalent to a <code>@property</code> function returning a reference to that member, scoped to <code>this</code>.<br />
<br />
'''Borrowing''' is the term used for assignment of a value to a variable with narrower scope. This typically happens on function calls, when a value is passed as an argument to a parameter annotated with (or inferred as) <code>scope</code>.<br />
<br />
== Examples ==<br />
<br />
=== RCArray ===<br />
<br />
Walter's <code>RCArray</code>, adjusted to this proposal:<br />
<br />
<source lang="D"><br />
@safe:<br />
<br />
struct RCArray(E) {<br />
@disable this();<br />
<br />
this(E[] a)<br />
{<br />
array = a.dup;<br />
count = new int;<br />
*count = 1;<br />
}<br />
<br />
~this() @trusted<br />
{<br />
if (--*count == 0)<br />
{<br />
delete count;<br />
// either `delete` in `@system` code will accept `scope`:<br />
delete array;<br />
// or a helper needs to be used to remove `scope`:<br />
delete assumeStatic(array);<br />
}<br />
}<br />
<br />
this(this)<br />
{<br />
++*count;<br />
}<br />
<br />
@property size_t length()<br />
{<br />
return array.length;<br />
}<br />
<br />
ref E opIndex(size_t i)<br />
{<br />
return array[i];<br />
}<br />
<br />
E[] opSlice(size_t lwr, size_t upr)<br />
{<br />
return array[lwr .. upr];<br />
}<br />
<br />
E[] opSlice()<br />
{<br />
return array[];<br />
}<br />
<br />
private:<br />
scope E[] array; // this is the only explicit annotation<br />
// enforces treatment as `scope`, to avoid<br />
// accidental access as unscoped<br />
int* count;<br />
}<br />
</source><br />
<br />
== Implementation ==<br />
<br />
=== Scope inference ===<br />
<br />
This algorithm works at the function level. Scope inference for variables and parameters in one function is independent from all other functions.<br />
<br />
It takes as input a list of variables whose scope is already fixed (by explicit annotations) and another list whose scopes are to be inferred. It will choose the narrowest possible scopes for them for which the function will still compile. This is based on the observation that variables that are read from need to have a scope at least as large as their destination. Therefore, we can start with the smallest possible scope, the lifetime of the variable itself, and extend it to the scope of the destination, if it isn't already larger.<br />
<br />
----<br />
<br />
1. Let <code>Q</code> be a list of all variables whose scopes are to be inferred. This includes template function parameters not otherwise annotated and all local variables.<br />
<br />
2. Assign all elements of <code>Q</code> an initial scope equivalent to their own lifetime:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
var.scope := [var];<br />
}<br />
</source><br />
<br />
3. For each <code>ASSIGNMENT</code> whose <code>RHS_SCOPE</code> depends on a variable in <code>Q</code>, expand that variable's scope to at least the <code>LHS_SCOPE</code>. For all variables the <code>LHS_SCOPE</code> depends on and that are in <code>Q</code>, record a dependency:<br />
<source lang="D"><br />
foreach(ass; ASSIGNMENTS) {<br />
if(ass.rhs_scope.depends_on(Q)) {<br />
foreach(rhs_var; ass.rhs_scope.vars) {<br />
if(not rhs_var in Q)<br />
continue;<br />
foreach(lhs_var; ass.lhs_scope.vars) {<br />
rhs_var.scope |= ass.lhs_scope;<br />
if(lhs_var in Q)<br />
rhs_var.deps ~= lhs_var;<br />
}<br />
}<br />
}<br />
}<br />
</source><br />
<br />
4. Remove all variables from <code>Q</code> that have no dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
if(var.deps.empty)<br />
Q.remove(var);<br />
}<br />
</source><br />
<br />
5. If <code>Q</code> is empty, terminate, else remember length of <code>Q</code>:<br />
<source lang="D"><br />
if(Q.empty)<br />
return;<br />
old_Q_len := Q.length;<br />
</source><br />
<br />
6. Expand all variables' scopes to at least that of their dependencies; if a dependency has no dependencies itself, remove it from the variable's dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
foreach(dep; var.deps) {<br />
var.scope |= dep.scope;<br />
if(dep.deps.empty)<br />
var.deps.remove(dep);<br />
}<br />
}<br />
</source><br />
<br />
7. If the length changed, we made progress. We can repeat from step 4. Otherwise we have a dependency loop. Find a cycle (for example using [//en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm Tarjan's algorithm]). Collect all elements in the cycle, remove their dependencies from <code>DEPENDENCIES</code>, and assign them all the union of their scopes:<br />
<source lang="D"><br />
if(Q.length != old_Q_len)<br />
goto step4;<br />
cycle := tarjan(DEPENDENCIES);<br />
new_scope := []<br />
foreach(var; cycle) {<br />
new_scope |= var.scope;<br />
var.deps.remove_each(cycle);<br />
}<br />
foreach(var; cycle) {<br />
var.scope := new_scope;<br />
}<br />
</source><br />
<br />
8. Go to step 4.<br />
<br />
----<br />
<br />
At this point, each variable will have a scope assigned. Now, all assignment can be checked to verify that they never place a reference in a location that outlives the reference's target.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5695User:Schuetzm/scope22015-03-15T13:40:36Z<p>Schuetzm: /* Returning scoped parameters */</p>
<hr />
<div>== Overview ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations.<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' also implies '''scope''', and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it were always accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behinds multiple indirections. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.<br />
<br />
== Details ==<br />
<br />
<code>scope</code> is a storage class. It applies to function parameters (including <code>this</code>), local variables, the return value (treated as if it were an <code>out</code> parameter), and member variables of aggregates. It participates in overloading.<br />
<br />
With every variable, a particular lifetime (called '''scope''') is associated.<br />
It usually refers to a local variable or parameter; additionally, an infinite scope is defined that corresponds to global/static variables or GC managed memory.<br />
Scopes and lifetimes are defined purely based on lexical scope and order of declaration of their corresponding variables. Therefore, for any two lifetimes, one is either completely contained in the other, or they are disjoint.<br />
By annotating a variable with <code>scope</code>, it's scope is defined to be equal to the variable's lifetime, instead of the default (infinity).<br />
<br />
For any expression involving at least one scope value, two lifetimes (''LHS lifetime'' and ''RHS lifetime'') are computed in a way that ensures that the resulting RHS lifetime will not be greater than that of any of the expression's parts, and the LHS lifetime will not be smaller.<br />
The exact rules will be defined in one of the following sections.<br />
An '''assignment''' (i.e., <code>=</code> operator, passing to a function, returning from a function, capturing in a closure, throwing, etc.) involving scope values is only permissible, if the destination's LHS lifetime is fully contained in the source's RHS lifetime.<br />
Throwing is considered assignment to a variable with static lifetime.<br />
<br />
The following invariant is enforced to always be true on every assignment: A location with scope ''a'' will never contain references pointing to values with a lifetime shorter than ''a''.<br />
<br />
To allow a function to return a value it received as a parameter, the parameter can be annotated with the <code>return</code> keyword,<br />
as in [[DIP25]].<br />
It's also possible to express that a parameter escapes through another parameter (including <code>this</code>) by using the <code>return!identifier</code> syntax.<br />
Multiple such annotations can appear for each parameter.<br />
When a function is called, the compiler checks (for each argument and the return value) that only expressions with a lifetime longer than those of all the corresponding <code>return</code> annotations are passed in, and that the return value is used in a conforming way.<br />
<br />
Because all relevant information about lifetimes is contained in the function signature, no explicit <code>scope</code> annotations for local variables<br />
are necessary; the compiler can figure them out by itself.<br />
Additionally, inference of annotations is done in the usual situations, i.e. nested functions and templates.<br />
In a subsequent section, an algorithm is presented that can be used for inference of scopes of local variables as well as annotations of parameters.<br />
<br />
In parameters of <code>@safe</code> functions, all reference types (class references, pointers, slices, <code>ref</code> parameters) or aggregates containing references are implicitly treated as <code>scope</code> unless the parameter is annotated with the keyword <code>static</code>. This does however not apply to the return value.<br />
<br />
<code>ref</code> and <code>out</code> parameters can also be treated as implicitly scoped, but this has the potential to break lots of code and needs to be considered carefully.<br />
<br />
A <code>scope</code> annotation on a member variable shall be equivalent to a <code>@property</code> function returning a reference to that member, scoped to <code>this</code>.<br />
<br />
'''Borrowing''' is the term used for assignment of a value to a variable with narrower scope. This typically happens on function calls, when a value is passed as an argument to a parameter annotated with (or inferred as) <code>scope</code>.<br />
<br />
== Examples ==<br />
<br />
=== RCArray ===<br />
<br />
Walter's <code>RCArray</code>, adjusted to this proposal:<br />
<br />
<source lang="D"><br />
@safe:<br />
<br />
struct RCArray(E) {<br />
@disable this();<br />
<br />
this(E[] a)<br />
{<br />
array = a.dup;<br />
count = new int;<br />
*count = 1;<br />
}<br />
<br />
~this() @trusted<br />
{<br />
if (--*count == 0)<br />
{<br />
delete count;<br />
// either `delete` in `@system` code will accept `scope`:<br />
delete array;<br />
// or a helper needs to be used to remove `scope`:<br />
delete assumeStatic(array);<br />
}<br />
}<br />
<br />
this(this)<br />
{<br />
++*count;<br />
}<br />
<br />
@property size_t length()<br />
{<br />
return array.length;<br />
}<br />
<br />
ref E opIndex(size_t i)<br />
{<br />
return array[i];<br />
}<br />
<br />
E[] opSlice(size_t lwr, size_t upr)<br />
{<br />
return array[lwr .. upr];<br />
}<br />
<br />
E[] opSlice()<br />
{<br />
return array[];<br />
}<br />
<br />
private:<br />
scope E[] array; // this is the only explicit annotation<br />
// enforces treatment as `scope`, to avoid<br />
// accidental access as unscoped<br />
int* count;<br />
}<br />
</source><br />
<br />
== Implementation ==<br />
<br />
=== Scope inference ===<br />
<br />
This algorithm works at the function level. Scope inference for variables and parameters in one function is independent from all other functions.<br />
<br />
It takes as input a list of variables whose scope is already fixed (by explicit annotations) and another list whose scopes are to be inferred. It will choose the narrowest possible scopes for them for which the function will still compile. This is based on the observation that variables that are read from need to have a scope at least as large as their destination. Therefore, we can start with the smallest possible scope, the lifetime of the variable itself, and extend it to the scope of the destination, if it isn't already larger.<br />
<br />
----<br />
<br />
1. Let <code>Q</code> be a list of all variables whose scopes are to be inferred. This includes template function parameters not otherwise annotated and all local variables.<br />
<br />
2. Assign all elements of <code>Q</code> an initial scope equivalent to their own lifetime:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
var.scope := [var];<br />
}<br />
</source><br />
<br />
3. For each <code>ASSIGNMENT</code> whose <code>RHS_SCOPE</code> depends on a variable in <code>Q</code>, expand that variable's scope to at least the <code>LHS_SCOPE</code>. For all variables the <code>LHS_SCOPE</code> depends on and that are in <code>Q</code>, record a dependency:<br />
<source lang="D"><br />
foreach(ass; ASSIGNMENTS) {<br />
if(ass.rhs_scope.depends_on(Q)) {<br />
foreach(rhs_var; ass.rhs_scope.vars) {<br />
if(not rhs_var in Q)<br />
continue;<br />
foreach(lhs_var; ass.lhs_scope.vars) {<br />
rhs_var.scope |= ass.lhs_scope;<br />
if(lhs_var in Q)<br />
rhs_var.deps ~= lhs_var;<br />
}<br />
}<br />
}<br />
}<br />
</source><br />
<br />
4. Remove all variables from <code>Q</code> that have no dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
if(var.deps.empty)<br />
Q.remove(var);<br />
}<br />
</source><br />
<br />
5. If <code>Q</code> is empty, terminate, else remember length of <code>Q</code>:<br />
<source lang="D"><br />
if(Q.empty)<br />
return;<br />
old_Q_len := Q.length;<br />
</source><br />
<br />
6. Expand all variables' scopes to at least that of their dependencies; if a dependency has no dependencies itself, remove it from the variable's dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
foreach(dep; var.deps) {<br />
var.scope |= dep.scope;<br />
if(dep.deps.empty)<br />
var.deps.remove(dep);<br />
}<br />
}<br />
</source><br />
<br />
7. If the length changed, we made progress. We can repeat from step 4. Otherwise we have a dependency loop. Find a cycle (for example using [//en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm Tarjan's algorithm]). Collect all elements in the cycle, remove their dependencies from <code>DEPENDENCIES</code>, and assign them all the union of their scopes:<br />
<source lang="D"><br />
if(Q.length != old_Q_len)<br />
goto step4;<br />
cycle := tarjan(DEPENDENCIES);<br />
new_scope := []<br />
foreach(var; cycle) {<br />
new_scope |= var.scope;<br />
var.deps.remove_each(cycle);<br />
}<br />
foreach(var; cycle) {<br />
var.scope := new_scope;<br />
}<br />
</source><br />
<br />
8. Go to step 4.<br />
<br />
----<br />
<br />
At this point, each variable will have a scope assigned. Now, all assignment can be checked to verify that they never place a reference in a location that outlives the reference's target.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5694User:Schuetzm/scope22015-03-15T13:33:43Z<p>Schuetzm: /* Implicit scope and opt-out */</p>
<hr />
<div>== Overview ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations.<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' also implies '''scope''', and the '''this''' parameter is always passed as '''scope''' (though not treated as such for overloading!) unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it we're alway accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behinds multiple indirections. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.<br />
<br />
== Details ==<br />
<br />
<code>scope</code> is a storage class. It applies to function parameters (including <code>this</code>), local variables, the return value (treated as if it were an <code>out</code> parameter), and member variables of aggregates. It participates in overloading.<br />
<br />
With every variable, a particular lifetime (called '''scope''') is associated.<br />
It usually refers to a local variable or parameter; additionally, an infinite scope is defined that corresponds to global/static variables or GC managed memory.<br />
Scopes and lifetimes are defined purely based on lexical scope and order of declaration of their corresponding variables. Therefore, for any two lifetimes, one is either completely contained in the other, or they are disjoint.<br />
By annotating a variable with <code>scope</code>, it's scope is defined to be equal to the variable's lifetime, instead of the default (infinity).<br />
<br />
For any expression involving at least one scope value, two lifetimes (''LHS lifetime'' and ''RHS lifetime'') are computed in a way that ensures that the resulting RHS lifetime will not be greater than that of any of the expression's parts, and the LHS lifetime will not be smaller.<br />
The exact rules will be defined in one of the following sections.<br />
An '''assignment''' (i.e., <code>=</code> operator, passing to a function, returning from a function, capturing in a closure, throwing, etc.) involving scope values is only permissible, if the destination's LHS lifetime is fully contained in the source's RHS lifetime.<br />
Throwing is considered assignment to a variable with static lifetime.<br />
<br />
The following invariant is enforced to always be true on every assignment: A location with scope ''a'' will never contain references pointing to values with a lifetime shorter than ''a''.<br />
<br />
To allow a function to return a value it received as a parameter, the parameter can be annotated with the <code>return</code> keyword,<br />
as in [[DIP25]].<br />
It's also possible to express that a parameter escapes through another parameter (including <code>this</code>) by using the <code>return!identifier</code> syntax.<br />
Multiple such annotations can appear for each parameter.<br />
When a function is called, the compiler checks (for each argument and the return value) that only expressions with a lifetime longer than those of all the corresponding <code>return</code> annotations are passed in, and that the return value is used in a conforming way.<br />
<br />
Because all relevant information about lifetimes is contained in the function signature, no explicit <code>scope</code> annotations for local variables<br />
are necessary; the compiler can figure them out by itself.<br />
Additionally, inference of annotations is done in the usual situations, i.e. nested functions and templates.<br />
In a subsequent section, an algorithm is presented that can be used for inference of scopes of local variables as well as annotations of parameters.<br />
<br />
In parameters of <code>@safe</code> functions, all reference types (class references, pointers, slices, <code>ref</code> parameters) or aggregates containing references are implicitly treated as <code>scope</code> unless the parameter is annotated with the keyword <code>static</code>. This does however not apply to the return value.<br />
<br />
<code>ref</code> and <code>out</code> parameters can also be treated as implicitly scoped, but this has the potential to break lots of code and needs to be considered carefully.<br />
<br />
A <code>scope</code> annotation on a member variable shall be equivalent to a <code>@property</code> function returning a reference to that member, scoped to <code>this</code>.<br />
<br />
'''Borrowing''' is the term used for assignment of a value to a variable with narrower scope. This typically happens on function calls, when a value is passed as an argument to a parameter annotated with (or inferred as) <code>scope</code>.<br />
<br />
== Examples ==<br />
<br />
=== RCArray ===<br />
<br />
Walter's <code>RCArray</code>, adjusted to this proposal:<br />
<br />
<source lang="D"><br />
@safe:<br />
<br />
struct RCArray(E) {<br />
@disable this();<br />
<br />
this(E[] a)<br />
{<br />
array = a.dup;<br />
count = new int;<br />
*count = 1;<br />
}<br />
<br />
~this() @trusted<br />
{<br />
if (--*count == 0)<br />
{<br />
delete count;<br />
// either `delete` in `@system` code will accept `scope`:<br />
delete array;<br />
// or a helper needs to be used to remove `scope`:<br />
delete assumeStatic(array);<br />
}<br />
}<br />
<br />
this(this)<br />
{<br />
++*count;<br />
}<br />
<br />
@property size_t length()<br />
{<br />
return array.length;<br />
}<br />
<br />
ref E opIndex(size_t i)<br />
{<br />
return array[i];<br />
}<br />
<br />
E[] opSlice(size_t lwr, size_t upr)<br />
{<br />
return array[lwr .. upr];<br />
}<br />
<br />
E[] opSlice()<br />
{<br />
return array[];<br />
}<br />
<br />
private:<br />
scope E[] array; // this is the only explicit annotation<br />
// enforces treatment as `scope`, to avoid<br />
// accidental access as unscoped<br />
int* count;<br />
}<br />
</source><br />
<br />
== Implementation ==<br />
<br />
=== Scope inference ===<br />
<br />
This algorithm works at the function level. Scope inference for variables and parameters in one function is independent from all other functions.<br />
<br />
It takes as input a list of variables whose scope is already fixed (by explicit annotations) and another list whose scopes are to be inferred. It will choose the narrowest possible scopes for them for which the function will still compile. This is based on the observation that variables that are read from need to have a scope at least as large as their destination. Therefore, we can start with the smallest possible scope, the lifetime of the variable itself, and extend it to the scope of the destination, if it isn't already larger.<br />
<br />
----<br />
<br />
1. Let <code>Q</code> be a list of all variables whose scopes are to be inferred. This includes template function parameters not otherwise annotated and all local variables.<br />
<br />
2. Assign all elements of <code>Q</code> an initial scope equivalent to their own lifetime:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
var.scope := [var];<br />
}<br />
</source><br />
<br />
3. For each <code>ASSIGNMENT</code> whose <code>RHS_SCOPE</code> depends on a variable in <code>Q</code>, expand that variable's scope to at least the <code>LHS_SCOPE</code>. For all variables the <code>LHS_SCOPE</code> depends on and that are in <code>Q</code>, record a dependency:<br />
<source lang="D"><br />
foreach(ass; ASSIGNMENTS) {<br />
if(ass.rhs_scope.depends_on(Q)) {<br />
foreach(rhs_var; ass.rhs_scope.vars) {<br />
if(not rhs_var in Q)<br />
continue;<br />
foreach(lhs_var; ass.lhs_scope.vars) {<br />
rhs_var.scope |= ass.lhs_scope;<br />
if(lhs_var in Q)<br />
rhs_var.deps ~= lhs_var;<br />
}<br />
}<br />
}<br />
}<br />
</source><br />
<br />
4. Remove all variables from <code>Q</code> that have no dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
if(var.deps.empty)<br />
Q.remove(var);<br />
}<br />
</source><br />
<br />
5. If <code>Q</code> is empty, terminate, else remember length of <code>Q</code>:<br />
<source lang="D"><br />
if(Q.empty)<br />
return;<br />
old_Q_len := Q.length;<br />
</source><br />
<br />
6. Expand all variables' scopes to at least that of their dependencies; if a dependency has no dependencies itself, remove it from the variable's dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
foreach(dep; var.deps) {<br />
var.scope |= dep.scope;<br />
if(dep.deps.empty)<br />
var.deps.remove(dep);<br />
}<br />
}<br />
</source><br />
<br />
7. If the length changed, we made progress. We can repeat from step 4. Otherwise we have a dependency loop. Find a cycle (for example using [//en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm Tarjan's algorithm]). Collect all elements in the cycle, remove their dependencies from <code>DEPENDENCIES</code>, and assign them all the union of their scopes:<br />
<source lang="D"><br />
if(Q.length != old_Q_len)<br />
goto step4;<br />
cycle := tarjan(DEPENDENCIES);<br />
new_scope := []<br />
foreach(var; cycle) {<br />
new_scope |= var.scope;<br />
var.deps.remove_each(cycle);<br />
}<br />
foreach(var; cycle) {<br />
var.scope := new_scope;<br />
}<br />
</source><br />
<br />
8. Go to step 4.<br />
<br />
----<br />
<br />
At this point, each variable will have a scope assigned. Now, all assignment can be checked to verify that they never place a reference in a location that outlives the reference's target.</div>Schuetzmhttps://wiki.dlang.org/?title=User:Schuetzm/scope2&diff=5693User:Schuetzm/scope22015-03-15T13:28:09Z<p>Schuetzm: /* Basics */</p>
<hr />
<div>== Overview ==<br />
<br />
The current D language specification reserves the '''scope''' keyword in function signatures to specify that a parameter will not be escaped by the function, making it @safe to pass references to local variables or manually managed memory to it, among other things. This feature is currently unimplemented, apart from its use with lambdas where it guarantees the closure will be allocated on the stack instead of the GC. This proposal intends to change that. It will allow the safe and efficient implementation of various memory management strategies (including reference counting), as well as unified handling of references to GC, reference counted data, local variables, containers, and others.<br />
<br />
The proposal is mostly a superset of [[DIP25]], but is generalized to all types of references and adds inference to alleviate the need for explicit annotations.<br />
<br />
=== Basics ===<br />
<br />
'''scope''' is a storage class; it will only be applicable to parameters in function signatures (which include the implicit '''this''' parameter for methods, as well as the context pointer for delegates). It will have the semantics one expects: when a function with a scope parameter returns, the corresponding argument will not have been stored in a global variable or on the heap, etc:<br />
<br />
<source lang="D"><br />
void sendData(scope ubyte[] data);<br />
void someOtherFunction(ubyte[] data);<br />
<br />
void main() {<br />
ubyte[1024] chunkOfData = ...;<br />
sendData(chunkOfData);<br />
// this is @safe: no reference to the local has escaped<br />
someOtherFunction(chunkOfData);<br />
// this is @system: the callee gives no guarantees about the param<br />
}<br />
</source><br />
<br />
As we can see, certain operations, like taking the address of a local (or slicing of a fixed-size array, which is equivalent), no longer need to be @system per se. Instead, it's what is done with the resulting reference that decides whether it's @system or @safe.<br />
<br />
=== Implicit '''scope''' and opt-out ===<br />
<br />
To reduce the need for manual annotations, @safe functions take all their reference parameters as '''scope'''. '''ref''' also implies '''scope''', and the '''this''' parameter is always passed as '''scope''' unless opted out. Because sometimes a @safe function may actually want to accept non-scope params, there is an opt-out in the form of '''static'''. Coupled with scope inference for templates, and an optional change like "@safe by default" (currently being discussed), this will get rid of most explicit scope annotations:<br />
<br />
<source lang="D"><br />
void doSomething(int[] data) @safe;<br />
// equivalent to:<br />
void doSomething(scope int[] data) @safe;<br />
<br />
void foo(int[] input, static int* output) @safe;<br />
// `input` is scope, `output` isn't<br />
<br />
void bar(ref MyStruct s) @safe;<br />
// equivalent to:<br />
void bar(scope ref MyStruct s) @safe;<br />
</source><br />
<br />
=== '''scope''' for value types & overloading ===<br />
<br />
'''scope''' applies to all types with indirections: pointers, slices, class references, '''ref''' parameters, delegates, and aggregates containing such. Functions and methods can be overloaded on '''scope'''. This allows efficient passing of RC wrappers for instance:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
// ...<br />
this(scope this) {<br />
// increment refcount<br />
count++;<br />
}<br />
~this() {<br />
// decrement refcount<br />
if(--count == 0)<br />
destroy(payload);<br />
}<br />
this(scope this) scope {<br />
// DON'T increment refcount<br />
}<br />
~this() scope {<br />
// DON'T decrement refcount<br />
}<br />
// magic, to be explained later<br />
alias borrow this;<br />
}<br />
<br />
void foo(scope MyClass object);<br />
<br />
RC!MyClass global;<br />
void bar(scope RC!MyClass object) {<br />
if(some_condition)<br />
global = object; // make a copy, adjust refcount<br />
}<br />
<br />
void main() {<br />
RC!MyClass x = ...;<br />
// auto conversion to MyClass, no refcount update:<br />
foo(x);<br />
// no refcount update at call site,<br />
// no needless double indirection with `ref`:<br />
bar(x);<br />
}<br />
</source><br />
<br />
All of this can be implemented in user code or in the standard library. The compiler doesn't need to be aware of reference counting.<br />
<br />
=== Implicit conversions ===<br />
<br />
A '''scope''' parameter doesn't care how the data it refers to has been allocated. All it requires is that the reference stays valid for the duration of the function call. Therefore, it's a perfect fit for library functions. They don't need to be templated to support different resource management strategies of the library's user. It acts as a bridge between different types of strategies, just like '''const''' acts as a bridge between mutable and immutable data.<br />
<br />
<source lang="D"><br />
// no template bloat, no knowledge about RC etc.:<br />
double computeAverage(scope int[] data);<br />
<br />
void main() {<br />
int[20] local = [1,2,3,...];<br />
writeln(computeAverage(local)); // OK<br />
int[] heap = ...;<br />
writeln(computeAverage(heap)); // OK<br />
RC!(int[]) rc = ...;<br />
writeln(computeAverage(rc)); // OK<br />
}<br />
</source><br />
<br />
This is achieved by allowing non-scoped types to convert to '''scope''' implicitly. For builtin references, the language does this automatically. User-defined types must opt in by defining an appropriate '''alias this'''.<br />
<br />
=== Returning scoped parameters ===<br />
<br />
Some functions want to return a parameter that is passed in, or something reachable through one, e.g. a member of '''this'''. They can express this by annotating the parameter with the keyword '''return''', just as in [[DIP25]]. To specify that the value is returned through another parameter, the '''return!ident''' syntax can be used. These annotations can be used multiple times per parameter, when the reference can be returned through several other parameters:<br />
<br />
<source lang="D"><br />
struct RC(T) if(is(T == class)) {<br />
scope T payload;<br />
T borrow() return { // `return` applies to `this`<br />
return payload;<br />
}<br />
}<br />
</source><br />
<br />
To prevent accidental access to a member (e.g. ''payload'' in the above example), the member can be annotated with '''scope'''. The compiler will then treat it as if it we're alway accessed through an appropriately annotated property that returns a (scoped) reference to it.<br />
<br />
The compiler will make sure the returned value is not used in any way that is un-@safe. In particular, it will verify that the returned references' lifetimes won't exceed the lifetimes of the arguments they're coming from.<br />
<br />
=== '''scope''' inference ===<br />
<br />
For templates and nested functions, the compiler can infer the scope annotations, just as it infers purity and @safe-ty. Generic code therefore rarely needs any explicit annotations:<br />
<br />
<source lang="D"><br />
T* foo(T)(T* a, T* b) {<br />
static T* cache;<br />
cache = b;<br />
return a;<br />
}<br />
<br />
// `foo!int` will be inferred as:<br />
int* foo_int(scope int* a return, static int* b);<br />
// (`static` is the default anyway, only here for clarity)<br />
</source><br />
<br />
=== Multiple indirections ===<br />
<br />
Multiple indirections are also handled in a way that preserves the guarantees about lifetimes. Because '''scope''' is not a type modifier, it cannot encode information about the lifetimes of objects behinds multiple indirections. Therefore, the compiler must be conservative. For the left-hand side of assignments, it must assume that the destination has infinite lifetime, while for the right-hand side, it must assume that the destination will vanish as soon as the reference through which it is accessed goes out of scope.<br />
<br />
=== @safe-ty violations with borrowing ===<br />
<br />
When borrowing is combined with explicit, non lexical-scope based memory management (of which reference counting is one form), there will inevitably be problems as the one discussed in [http://forum.dlang.org/post/huspgmeupgobjubtsmfe@forum.dlang.org this forum thread]. To deal with them in a safe way requires some kind of data flow and aliasing analysis. Rust is an example of a language that uses very sophisticated analysis algorithms for that. This proposal will include a simplified algorithm to detect potentially unsafe uses at compile time, at the cost of detecting some false positives, for which there will however be workarounds. Instead of disallowing these operations, they will be treated as @system. It is therefore up to the end user to decide how to deal with them.<br />
<br />
== Details ==<br />
<br />
<code>scope</code> is a storage class. It applies to function parameters (including <code>this</code>), local variables, the return value (treated as if it were an <code>out</code> parameter), and member variables of aggregates. It participates in overloading.<br />
<br />
With every variable, a particular lifetime (called '''scope''') is associated.<br />
It usually refers to a local variable or parameter; additionally, an infinite scope is defined that corresponds to global/static variables or GC managed memory.<br />
Scopes and lifetimes are defined purely based on lexical scope and order of declaration of their corresponding variables. Therefore, for any two lifetimes, one is either completely contained in the other, or they are disjoint.<br />
By annotating a variable with <code>scope</code>, it's scope is defined to be equal to the variable's lifetime, instead of the default (infinity).<br />
<br />
For any expression involving at least one scope value, two lifetimes (''LHS lifetime'' and ''RHS lifetime'') are computed in a way that ensures that the resulting RHS lifetime will not be greater than that of any of the expression's parts, and the LHS lifetime will not be smaller.<br />
The exact rules will be defined in one of the following sections.<br />
An '''assignment''' (i.e., <code>=</code> operator, passing to a function, returning from a function, capturing in a closure, throwing, etc.) involving scope values is only permissible, if the destination's LHS lifetime is fully contained in the source's RHS lifetime.<br />
Throwing is considered assignment to a variable with static lifetime.<br />
<br />
The following invariant is enforced to always be true on every assignment: A location with scope ''a'' will never contain references pointing to values with a lifetime shorter than ''a''.<br />
<br />
To allow a function to return a value it received as a parameter, the parameter can be annotated with the <code>return</code> keyword,<br />
as in [[DIP25]].<br />
It's also possible to express that a parameter escapes through another parameter (including <code>this</code>) by using the <code>return!identifier</code> syntax.<br />
Multiple such annotations can appear for each parameter.<br />
When a function is called, the compiler checks (for each argument and the return value) that only expressions with a lifetime longer than those of all the corresponding <code>return</code> annotations are passed in, and that the return value is used in a conforming way.<br />
<br />
Because all relevant information about lifetimes is contained in the function signature, no explicit <code>scope</code> annotations for local variables<br />
are necessary; the compiler can figure them out by itself.<br />
Additionally, inference of annotations is done in the usual situations, i.e. nested functions and templates.<br />
In a subsequent section, an algorithm is presented that can be used for inference of scopes of local variables as well as annotations of parameters.<br />
<br />
In parameters of <code>@safe</code> functions, all reference types (class references, pointers, slices, <code>ref</code> parameters) or aggregates containing references are implicitly treated as <code>scope</code> unless the parameter is annotated with the keyword <code>static</code>. This does however not apply to the return value.<br />
<br />
<code>ref</code> and <code>out</code> parameters can also be treated as implicitly scoped, but this has the potential to break lots of code and needs to be considered carefully.<br />
<br />
A <code>scope</code> annotation on a member variable shall be equivalent to a <code>@property</code> function returning a reference to that member, scoped to <code>this</code>.<br />
<br />
'''Borrowing''' is the term used for assignment of a value to a variable with narrower scope. This typically happens on function calls, when a value is passed as an argument to a parameter annotated with (or inferred as) <code>scope</code>.<br />
<br />
== Examples ==<br />
<br />
=== RCArray ===<br />
<br />
Walter's <code>RCArray</code>, adjusted to this proposal:<br />
<br />
<source lang="D"><br />
@safe:<br />
<br />
struct RCArray(E) {<br />
@disable this();<br />
<br />
this(E[] a)<br />
{<br />
array = a.dup;<br />
count = new int;<br />
*count = 1;<br />
}<br />
<br />
~this() @trusted<br />
{<br />
if (--*count == 0)<br />
{<br />
delete count;<br />
// either `delete` in `@system` code will accept `scope`:<br />
delete array;<br />
// or a helper needs to be used to remove `scope`:<br />
delete assumeStatic(array);<br />
}<br />
}<br />
<br />
this(this)<br />
{<br />
++*count;<br />
}<br />
<br />
@property size_t length()<br />
{<br />
return array.length;<br />
}<br />
<br />
ref E opIndex(size_t i)<br />
{<br />
return array[i];<br />
}<br />
<br />
E[] opSlice(size_t lwr, size_t upr)<br />
{<br />
return array[lwr .. upr];<br />
}<br />
<br />
E[] opSlice()<br />
{<br />
return array[];<br />
}<br />
<br />
private:<br />
scope E[] array; // this is the only explicit annotation<br />
// enforces treatment as `scope`, to avoid<br />
// accidental access as unscoped<br />
int* count;<br />
}<br />
</source><br />
<br />
== Implementation ==<br />
<br />
=== Scope inference ===<br />
<br />
This algorithm works at the function level. Scope inference for variables and parameters in one function is independent from all other functions.<br />
<br />
It takes as input a list of variables whose scope is already fixed (by explicit annotations) and another list whose scopes are to be inferred. It will choose the narrowest possible scopes for them for which the function will still compile. This is based on the observation that variables that are read from need to have a scope at least as large as their destination. Therefore, we can start with the smallest possible scope, the lifetime of the variable itself, and extend it to the scope of the destination, if it isn't already larger.<br />
<br />
----<br />
<br />
1. Let <code>Q</code> be a list of all variables whose scopes are to be inferred. This includes template function parameters not otherwise annotated and all local variables.<br />
<br />
2. Assign all elements of <code>Q</code> an initial scope equivalent to their own lifetime:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
var.scope := [var];<br />
}<br />
</source><br />
<br />
3. For each <code>ASSIGNMENT</code> whose <code>RHS_SCOPE</code> depends on a variable in <code>Q</code>, expand that variable's scope to at least the <code>LHS_SCOPE</code>. For all variables the <code>LHS_SCOPE</code> depends on and that are in <code>Q</code>, record a dependency:<br />
<source lang="D"><br />
foreach(ass; ASSIGNMENTS) {<br />
if(ass.rhs_scope.depends_on(Q)) {<br />
foreach(rhs_var; ass.rhs_scope.vars) {<br />
if(not rhs_var in Q)<br />
continue;<br />
foreach(lhs_var; ass.lhs_scope.vars) {<br />
rhs_var.scope |= ass.lhs_scope;<br />
if(lhs_var in Q)<br />
rhs_var.deps ~= lhs_var;<br />
}<br />
}<br />
}<br />
}<br />
</source><br />
<br />
4. Remove all variables from <code>Q</code> that have no dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
if(var.deps.empty)<br />
Q.remove(var);<br />
}<br />
</source><br />
<br />
5. If <code>Q</code> is empty, terminate, else remember length of <code>Q</code>:<br />
<source lang="D"><br />
if(Q.empty)<br />
return;<br />
old_Q_len := Q.length;<br />
</source><br />
<br />
6. Expand all variables' scopes to at least that of their dependencies; if a dependency has no dependencies itself, remove it from the variable's dependencies:<br />
<source lang="D"><br />
foreach(var; Q) {<br />
foreach(dep; var.deps) {<br />
var.scope |= dep.scope;<br />
if(dep.deps.empty)<br />
var.deps.remove(dep);<br />
}<br />
}<br />
</source><br />
<br />
7. If the length changed, we made progress. We can repeat from step 4. Otherwise we have a dependency loop. Find a cycle (for example using [//en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm Tarjan's algorithm]). Collect all elements in the cycle, remove their dependencies from <code>DEPENDENCIES</code>, and assign them all the union of their scopes:<br />
<source lang="D"><br />
if(Q.length != old_Q_len)<br />
goto step4;<br />
cycle := tarjan(DEPENDENCIES);<br />
new_scope := []<br />
foreach(var; cycle) {<br />
new_scope |= var.scope;<br />
var.deps.remove_each(cycle);<br />
}<br />
foreach(var; cycle) {<br />
var.scope := new_scope;<br />
}<br />
</source><br />
<br />
8. Go to step 4.<br />
<br />
----<br />
<br />
At this point, each variable will have a scope assigned. Now, all assignment can be checked to verify that they never place a reference in a location that outlives the reference's target.</div>Schuetzm