November 04, 2009
On Tue, Nov 3, 2009 at 3:54 PM, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> Leandro Lucarella wrote:
>>
>> Andrei Alexandrescu, el  3 de noviembre a las 16:33 me escribiste:
>>>
>>> SafeD is, unfortunately, not finished at the moment. I want to leave in place a stub that won't lock our options. Here's what we currently have:
>>>
>>> module(system) calvin;
>>>
>>> This means calvin can do unsafe things.
>>>
>>> module(safe) susie;
>>>
>>> This means susie commits to extra checks and therefore only a subset of D.
>>>
>>> module hobbes;
>>>
>>> This means hobbes abides to whatever the default safety setting is.
>>>
>>> The default safety setting is up to the compiler. In dmd by default it is "system", and can be overridden with "-safe".
>>
>> What's the rationale for letting the compiler decide? I can't see nothing but trouble about this. A module will tipically be writen to be safe or system, I think the default should be defined (I'm not sure what the default should be though).
>
> The parenthesis pretty much destroys your point :o).
>
> I don't think letting the implementation decide is a faulty model. If you know what you want, you say it. Otherwise it means you don't care.

How can you not care?  Either your module uses unsafe features or it doesn't.  So it seems if you don't specify, then your module must pass the strictest checks, because otherwise it's not a "don't care" situation -- it's a "system"-only situation.

--bb
November 04, 2009
Andrei Alexandrescu wrote:
> Sketch of the safe rules:
> 
> \begin{itemize*}
> \item No @cast@ from a pointer type to an integral type and vice versa

replace integral type with non-pointer type.

> \item No @cast@ between unrelated pointer types
> \item Bounds checks on all array accesses
> \item  No  unions  that  include  a reference  type  (array,  @class@,
>   pointer, or @struct@ including such a type)

pointers are not a reference type. Replace "reference type" with "pointers or reference types".

> \item No pointer arithmetic

> \item No escape of a pointer  or reference to a local variable outside
>   its scope

revise: cannot take the address of a local or a reference.

> \item Cross-module function calls must only go to other @safe@ modules
> \end{itemize*}

add:
. no inline assembler
. no casting away of const, immutable, or shared
November 04, 2009
Bill Baxter wrote:
> On Tue, Nov 3, 2009 at 2:33 PM, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org> wrote:
>> SafeD is, unfortunately, not finished at the moment. I want to leave in
>> place a stub that won't lock our options. Here's what we currently have:
>>
>> module(system) calvin;
>>
>> This means calvin can do unsafe things.
>>
>> module(safe) susie;
>>
>> This means susie commits to extra checks and therefore only a subset of D.
>>
>> module hobbes;
>>
>> This means hobbes abides to whatever the default safety setting is.
>>
>> The default safety setting is up to the compiler. In dmd by default it is
>> "system", and can be overridden with "-safe".
>>
>> Sketch of the safe rules:
>>
>> \begin{itemize*}
>> \item No @cast@ from a pointer type to an integral type and vice versa
>> \item No @cast@ between unrelated pointer types
>> \item Bounds checks on all array accesses
>> \item  No  unions  that  include  a reference  type  (array,  @class@,
>>  pointer, or @struct@ including such a type)
>> \item No pointer arithmetic
>> \item No escape of a pointer  or reference to a local variable outside
>>  its scope
>> \item Cross-module function calls must only go to other @safe@ modules
>> \end{itemize*}
>>
>> So these are my thoughts so far. There is one problem though related to the
>> last \item - there's no way for a module to specify "trusted", meaning:
>> "Yeah, I do unsafe stuff inside, but safe modules can call me no problem".
>> Many modules in std fit that mold.
>>
>> How can we address that? Again, I'm looking for a simple, robust, extensible
>> design that doesn't lock our options.
> 
> I have to say that I would be seriously annoyed to see repeated
> references to a feature that turns out to be vaporware.  (I'm guessing
> there will be repeated references to SafeD based on the Chapter 4
> sample, and I'm guessing it will be vaporware based on the question
> you're asking above).  I'd say leave SafeD for the 2nd edition, and
> just comment that work is underway in a "Future of D" chapter near the
> end of the book.  And of course add a "Look to <the publishers website
> || digitalmars.com> for the latest!"
> 
> Even if not vaporware, it looks like whatever you write is going to be
> about something completely untested in the wild, and so has a high
> chance of turning out to need re-designing in the face of actual use.
> 
> --bb

Ok, I won't use the term SafeD as if it were a product. But -safe is there, some checks are there, and Walter is apparently willing to complete them. It's not difficult to go with an initially conservative approach - e.g., "no taking the address of a local" as he wrote in a recent post - although a more refined approach would still allow to take addresses of locals, as long as they don't escape.


Andrei
November 04, 2009
On Tue, 03 Nov 2009 17:55:15 -0600, Andrei Alexandrescu wrote:

> There's a lot more, but there are a few useful subspaces. One is, if an entire application only uses module(safe) that means there is no memory error in that application, ever.
> 
> Andrei

Does that mean that a module that uses a "trusted" module must also be marked as "trusted?" I would see this as pointless since system modules are likely to be used in safe code a lot.

I think the only real option is to have the importer decide if it is trusted. I don't see a reasonable way to have third party certification. It is between the library writer and application developer. Since the library writer's goal should be to have a system module that is safe, he would likely want to mark it as trusted. This would leave "system" unused because everyone wants to be safe.

In conclusion, here is a chunk of possible import options. I vote for the top two.

import(system) std.stdio;
system import std.stdio;
trusted import std.stdio;
import(trusted) std.stdio;
import("This is a system module and I know that it is potentially unsafe,
but I still want to use it in my safe code") std.stdio;
November 04, 2009
Walter Bright Wrote:

> Andrei Alexandrescu wrote:
> > Sketch of the safe rules:
> > 
> > \begin{itemize*}
> > \item No @cast@ from a pointer type to an integral type and vice versa
> 
> replace integral type with non-pointer type.
> 
> > \item No @cast@ between unrelated pointer types
> > \item Bounds checks on all array accesses
> > \item  No  unions  that  include  a reference  type  (array,  @class@,
> >   pointer, or @struct@ including such a type)
> 
> pointers are not a reference type. Replace "reference type" with "pointers or reference types".
> 
> > \item No pointer arithmetic
> 
> > \item No escape of a pointer  or reference to a local variable outside
> >   its scope
> 
> revise: cannot take the address of a local or a reference.
> 
> > \item Cross-module function calls must only go to other @safe@ modules \end{itemize*}
> 
> add:
> . no inline assembler
> . no casting away of const, immutable, or shared

How does casting away const, immutable, or shared cause memory corruption? If I understand SafeD correctly, that's its only goal. If it does more, I'd also argue casting to shared or immutable is, in general, unsafe. I'm also unsure if safeD has really fleshed out what would make use of (lockfree) shared variables safe. For example, array concatenation in one thread while reading in another thread could allow reading of garbage memory (e.g. if the length was incremented before writing the cell contents)
November 04, 2009
Jesse Phillips wrote:
> On Tue, 03 Nov 2009 17:55:15 -0600, Andrei Alexandrescu wrote:
> 
>> There's a lot more, but there are a few useful subspaces. One is, if an
>> entire application only uses module(safe) that means there is no memory
>> error in that application, ever.
>>
>> Andrei
> 
> Does that mean that a module that uses a "trusted" module must also be marked as "trusted?" I would see this as pointless since system modules are likely to be used in safe code a lot.

Same here.

> I think the only real option is to have the importer decide if it is trusted.

That can't work. I can't say that stdc.stdlib is trusted no matter how hard I try. I mean free is there!

> I don't see a reasonable way to have third party certification. It is between the library writer and application developer. Since the library writer's goal should be to have a system module that is safe, he would likely want to mark it as trusted. This would leave "system" unused because everyone wants to be safe.

Certain modules definitely can't aspire to be trusted. But for example std.stdio can claim to be trusted because, in spite of using untrusted stuff like FILE* and fclose, they are encapsulated in a way that makes it impossible for a safe client to engender memory errors.

> In conclusion, here is a chunk of possible import options. I vote for the top two.
> 
> import(system) std.stdio;
> system import std.stdio;
> trusted import std.stdio;
> import(trusted) std.stdio;
> import("This is a system module and I know that it is potentially unsafe, but I still want to use it in my safe code") std.stdio;

Specifying a clause with import crossed my mind too, it's definitely something to keep in mind.


Andrei
November 04, 2009
Jason House wrote:
> Walter Bright Wrote:
> 
>> Andrei Alexandrescu wrote:
>>> Sketch of the safe rules:
>>> 
>>> \begin{itemize*} \item No @cast@ from a pointer type to an
>>> integral type and vice versa
>> replace integral type with non-pointer type.
>> 
>>> \item No @cast@ between unrelated pointer types \item Bounds
>>> checks on all array accesses \item  No  unions  that  include  a
>>> reference  type  (array,  @class@, pointer, or @struct@ including
>>> such a type)
>> pointers are not a reference type. Replace "reference type" with "pointers or reference types".
>> 
>>> \item No pointer arithmetic \item No escape of a pointer  or
>>> reference to a local variable outside its scope
>> revise: cannot take the address of a local or a reference.
>> 
>>> \item Cross-module function calls must only go to other @safe@
>>> modules \end{itemize*}
>> add: . no inline assembler . no casting away of const, immutable,
>> or shared
> 
> How does casting away const, immutable, or shared cause memory
> corruption?

If you have an immutable string, the compiler may cache or enregister
the length and do anything (such as hoisting checks out of loops) in confidence the length will never change. If you do change it -> memory error.

> If I understand SafeD correctly, that's its only goal. If it does
> more, I'd also argue casting to shared or immutable is, in general,
> unsafe. I'm also unsure if safeD has really fleshed out what would
> make use of (lockfree) shared variables safe. For example, array
> concatenation in one thread while reading in another thread could
> allow reading of garbage memory (e.g. if the length was incremented
> before writing the cell contents)

Shared arrays can't be modified.


Andrei
November 04, 2009
Jason House wrote:
> How does casting away const, immutable, or shared cause memory
> corruption? If I understand SafeD correctly, that's its only goal. If
> it does more, I'd also argue casting to shared or immutable is, in
> general, unsafe.

They can cause memory corruption because inadvertent "tearing" can occur when two parts to a memory reference are updated, half from one and half from another alias.

> I'm also unsure if safeD has really fleshed out what
> would make use of (lockfree) shared variables safe. For example,
> array concatenation in one thread while reading in another thread
> could allow reading of garbage memory (e.g. if the length was
> incremented before writing the cell contents)

That kind of out-of-order reading is just what shared is meant to prevent.

November 04, 2009
On Tue, 03 Nov 2009 17:33:39 -0500, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> SafeD is, unfortunately, not finished at the moment. I want to leave in place a stub that won't lock our options. Here's what we currently have:
>
> module(system) calvin;
>
> This means calvin can do unsafe things.
>
> module(safe) susie;
>
> This means susie commits to extra checks and therefore only a subset of D.
>
> module hobbes;
>
> This means hobbes abides to whatever the default safety setting is.
> ...
> \item Cross-module function calls must only go to other @safe@ modules
> \end{itemize*}
>
> So these are my thoughts so far. There is one problem though related to the last \item - there's no way for a module to specify "trusted", meaning: "Yeah, I do unsafe stuff inside, but safe modules can call me no problem". Many modules in std fit that mold.

My interpretation of the module decorations was:

module(system) calvin;

This means calvin uses unsafe things, but is considered safe for other modules (it overrides the setting of the compiler, so can be compiled even in safe mode).

module(safe) susie;

This means susie commits to extra checks, and will be compiled in safe mode even if the compiler is in unsafe mode.  Susie can only import module(safe) or module(system) modules, or if the compiler is in safe mode, any module.

module hobbes;

This means hobbes doesn't care whether he's safe or not.  (note the important difference from your description)

My rationale for interpreting module(system) is: why declare a module as system unless you *wanted* it to be compilable in safe mode?

I would expect that very few modules are marked as module(system).

And as for the default setting, I think that unsafe is a reasonable default.  You can always create a shortcut/script/symlink to the compiler that adds the -safe flag if you wanted a safe-by-default version.

-Steve
November 04, 2009
"Andrei Alexandrescu" <SeeWebsiteForEmail@erdani.org> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ: news:hcr2hb$dvm$1@digitalmars.com...
> Jesse Phillips wrote:
>> On Tue, 03 Nov 2009 17:55:15 -0600, Andrei Alexandrescu wrote:
>>
>>> There's a lot more, but there are a few useful subspaces. One is, if an entire application only uses module(safe) that means there is no memory error in that application, ever.
>>>
>>> Andrei
>>
>> Does that mean that a module that uses a "trusted" module must also be marked as "trusted?" I would see this as pointless since system modules are likely to be used in safe code a lot.
>
> Same here.
>
>> I think the only real option is to have the importer decide if it is trusted.
>
> That can't work. I can't say that stdc.stdlib is trusted no matter how hard I try. I mean free is there!
>
>> I don't see a reasonable way to have third party certification. It is between the library writer and application developer. Since the library writer's goal should be to have a system module that is safe, he would likely want to mark it as trusted. This would leave "system" unused because everyone wants to be safe.
>
> Certain modules definitely can't aspire to be trusted. But for example std.stdio can claim to be trusted because, in spite of using untrusted stuff like FILE* and fclose, they are encapsulated in a way that makes it impossible for a safe client to engender memory errors.
>
>> In conclusion, here is a chunk of possible import options. I vote for the top two.
>>
>> import(system) std.stdio;
>> system import std.stdio;
>> trusted import std.stdio;
>> import(trusted) std.stdio;
>> import("This is a system module and I know that it is potentially unsafe,
>> but I still want to use it in my safe code") std.stdio;
>
> Specifying a clause with import crossed my mind too, it's definitely something to keep in mind.
>
>
> Andrei
>
How about this:
system module foo ;
... (code)
trusted module foo2 ;
... (code)
safe module bar ;
... (code)

import foo, foo2, bar ; // status defined automatically from module
declaration.
//  error: used system module 'foo' in safe application.