On 18.02.2016 20:17, Era Scarecrow wrote:
> On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer wrote:
>> On 2/18/16 1:30 PM, Timon Gehr wrote:
>>> No problem here. There is no way to assign to a void[] without doing 2.
>>
>> foo(void[] arr)
>> {
>>    void[] arr2 = [1234, 5678, 91011];
>>    arr[] = arr2[0 .. arr.length];
>> }
>
>   Since void throws away type information (and all the safety related to
> it), would it be easier to simply require @safe code can't cast
> implicitly to void? It seems like explicit casting would take care of
> most of this, or disallowing to/from void converting period while in
> @safe code.

The conversion is fine. It just throws away type information. There's no way to soundly type-check the block assignment after that, so that's the operation that should be disallowed. This also prevents @safe code from mutating untyped memory that was passed through from @system code without preventing it to pass the block back to @trusted code.

On Thursday, 18 February 2016 at 16:37:10 UTC, H. S. Teoh wrote:
> (*ahem*std.socket*cough*) liberally sprinkle @trusted on every function without regard to whether it's truly justified (e.g., see https://issues.dlang.org/show_bug.cgi?id=15672)

How is bug 15672 related to std.socket? From quick glance at first thousand lines of std.socket I don't see functions incorrectly marked as trusted.

Well, ok, getprotobyname indeed looks thread unsafe for some reason, but this can happen for any function.

On Thursday, 18 February 2016 at 19:28:16 UTC, Kagamin wrote:
> On Thursday, 18 February 2016 at 16:37:10 UTC, H. S. Teoh wrote:
>> (*ahem*std.socket*cough*) liberally sprinkle @trusted on every function without regard to whether it's truly justified (e.g., see https://issues.dlang.org/show_bug.cgi?id=15672)
>
> How is bug 15672 related to std.socket? From quick glance at first thousand lines of std.socket I don't see functions incorrectly marked as trusted.

The problem with std.socket is that calls to stuff like recv involve processing the data from void[], which is inherently unsafe, but the D functions calling them (e.g. receive) are marked as @trusted. It's conceptually similar to 15672 in that recv is taking the void* that it's given (from the void[]) and essentially converting it to ubyte* (or char*, since it's C), which is more or less equivalent to casting the void[] to ubyte[], which is what 15672 is about. However, even if 15672 is fixed, it wouldn't affect std.socket any, because it's the fact that a pointer to void[] is passed to a C function that converts it to ubyte*/char* that's the problem, not that void[] is converted to ubyte[] (since that never actually happens).

- Jonathan M Davis

On Thursday, 18 February 2016 at 18:58:24 UTC, Steven Schveighoffer wrote:
> And implicit casts to void[] should be disallowed.


We could strike a sane medium: in a @safe function or when calling a @safe function, implicit conversions of any array WITH REFERENCES is not allowed, while other arrays may be reinterpreted.

So you can still pass ubyte[], char[], long[], etc. to a @safe void[] thing, but not Object[] or ubyte*[] or struct Foo {int *a }[].

This should cover the vast majority of the sane use-cases for Socket.receive, etc., anyway without being a problem.

On Thu, Feb 18, 2016 at 07:25:16PM +0000, Jonathan M Davis via Digitalmars-d wrote:
> On Thursday, 18 February 2016 at 18:58:56 UTC, H. S. Teoh wrote:
> >On Thu, Feb 18, 2016 at 06:50:34PM +0000, Jonathan M Davis via Digitalmars-d wrote:
> >>On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer wrote:
> >[...]
> >>>foo(void[] arr)
> >>>{
> >>>   void[] arr2 = [1234, 5678, 91011];
> >>>   arr[] = arr2[0 .. arr.length];
> >>>}
> >>
> >>Well, I'm not sure that that's actually not @safe.
> >
> >How can it possibly be @safe??? Consider:
> >
> >	void main() @safe {
> >		Object[] objs = [ new Object() ];
> >		foo(objs);
> >	}
> >
> >Now the pointer to the Object instance has been corrupted.
> 
> Does it matter what state the void[] is in until you actually attempt to cast it to something else? If you have
> 
> Object[] oArr = [ new Object ];
> void[] vArr = oArr;
> vArr = vArr[0 .. $ - 1];
> 
> it's not like that's going to cause any problems - not by itself.

I think you misread Steven's code.  This has nothing to do with reducing the length of an array. Read the code again.  It has to do with *overwriting* the contents of one void[] with another void[], which is a legal and @safe operation under the current rules. Unfortunately, the first void[] is an array of Object references, and the second void[] is an array of ints, so this is overwriting pointers with arbitrary int values.

Furthermore, as the caller of foo(), main() has no idea what has happened to its Object[], because it continues to see the same area of memory as Object[], even though it has been overwritten by an array of ints.  So when it next tries to dereference the Object[], which is perfectly legal and does not involve any casting, it ends up dereferencing an arbitrary int as a pointer instead. Ergo, @safe has been compromised.


[...]
> >I think you missed the point of his example. :-) The point is that
> >it's perfectly legal to (1) cast an array of int to void[], and (2)
> >it's also perfectly legal to cast an array of anything to void[], and
> >(3) under current rules, it's perfectly legal to copy one void[] to
> >another void[].
> >
> >Arguably, (3) should not be allowed in @safe code. Which again brings us back to the point, that if any function takes void[] as an argument, is there *anything* it can do with the void[] other than reading it, that *won't* break @safe?
> >
> >If there's *nothing* it can legally do with a void[] (other than reading it) without violating @safe, then shouldn't it be a requirement that all functions marked @safe must take const(void)[] rather than void[]?
> 
> It could pass it to something else without actually doing anything to it, in which case, I don't see why it couldn't be @safe. It's interpreting a void[] that's the problem. _That_ needs to be @system. Merely converting to void[] or passing it around is harmless.

No. Please look at Steven's code again. What it is doing, is (1) implicitly casting Object[] to void[], which is perfectly legal according to what you said. Then (2) it casts an int[] literal to void[], again, another perfectly legal operation. Finally, (3) it copies the contents of the second void[] to the first void[]. Since both arrays are typed void[] at this point, the copy is perfectly legal (or so the compiler thinks), and here is where the Object[] got overwritten by an int[] and the compiler did not realize it.

Then back in main(), the code does not see void[] at all; all it sees is the Object[] it has created all along.  There is no interpretation of void[] going on here at all.  All that happened was (1) we passed Object[] to void[], which is legal according to what you said, and (2) we dereferenced the Object[], which is not suspect at all because, well, it's an Object[], what could possibly go wrong?  At this point, however, we are dereferencing an illegal pointer, because foo() has corrupted our Object[]. Note that *nowhere* in main() is there any reinterpretation of void[] as Object[].  The only thing that led to this mess, as far as main() is concerned, is that it passed an Object[] to a function that takes void[].

In other words, from the caller's POV, as soon as you pass an array to a function that takes void[], all bets are off, anything could happen to your precious data, and @safety is no longer guaranteed.  However, foo() is marked @safe and the compiler does not complain.  Also, inside foo() we did nothing unseemly, according to current rules.  All we did was to convert two arrays to void[], which is perfectly legal and harmless, as you claim, and copy data from one void[] to another void[], which, under the current rules, is perfectly legal since we are just copying data between two arrays *of the same type*. What could possibly go wrong with copying data between two arrays of the same type?

The only problem is, they *aren't* of the same type after all. Just because is(void[] == void[]) is true does NOT mean it is @safe to copy one array to another, because the void here is a wildcard placeholder representing arbitrary types.

So the bottom line is that the array copy cannot be @safe.

The larger question, is what, if anything, *can* you do with void[] besides read-only operations, that doesn't break @safe?  Once something is (implicitly or otherwise) cast to void[], all type information is forgotten, and there is no way, that I can tell, to write to a void[] without causing @safe breakage.  If so, wouldn't it make sense to require that the type should be const(void)[] rather than void[]?


[...]
> The problem with std.socket is not that it accepts void[] and considers that @safe; it's that it passes that void[] to something else, treating that as @safe, when it's not. And since it's a C function that's being called, and it accesses the ptr member of the void[], it's already @system, forcing @trusted to be used to make the compiler happy. So, the problem is that @trusted was used when it shouldn't have been, not that the type system allowed void[] to be passed in without marking it as @system. And since the function was marked @trusted, it's not like having the compiler treat a function that accepted void[] as @system would have helped anyway.
[...]

Certainly, the @trusted was wrongly applied to that particular function. That's easy to fix: remove @trusted and mark it as @system, as it should be.  Or, to be a bit nicer to our current users, qualify it with a sig constraint so that it only accepts arrays that have no further indirections.

But I'm thinking about the larger question here.  If modifying the data in a void[] is *always* @system, because of the type erasure that basically eliminates all possibility of the lower-level function recovering the type and adapting its operation to be @safe, then why allow passing void[] in @safe code in the first place?  If *any* modification to the data in the void[] carries the risk of breaking @safe, then shouldn't we require that we only allow const(void)[] in @safe code?

It's not as if we don't have @trusted as a fallback in those cases where we *know*, outside the type system, that a particular function call is actually safe even though it's marked @system.  But the point is, by disabling this dangerous operation *by default* (albeit allowing it via the @trusted escape hatch), we eliminate the possibility of screwups like Steven's code demonstrated.


T

-- 
Never step over a puddle, always step around it. Chances are that whatever made it is still dripping.

On Thu, Feb 18, 2016 at 07:28:16PM +0000, Kagamin via Digitalmars-d wrote:
> On Thursday, 18 February 2016 at 16:37:10 UTC, H. S. Teoh wrote:
> >(*ahem*std.socket*cough*) liberally sprinkle @trusted on every function
> >without regard to whether it's truly justified (e.g., see
> >https://issues.dlang.org/show_bug.cgi?id=15672)
> 
> How is bug 15672 related to std.socket? From quick glance at first thousand lines of std.socket I don't see functions incorrectly marked as trusted.

Sorry, I pasted the wrong link. It should be:

	https://issues.dlang.org/show_bug.cgi?id=14137


T

-- 
Some days you win; most days you lose.

On Thursday, 18 February 2016 at 20:24:18 UTC, H. S. Teoh wrote:
> On Thu, Feb 18, 2016 at 07:25:16PM +0000, Jonathan M Davis via Digitalmars-d wrote:
>> On Thursday, 18 February 2016 at 18:58:56 UTC, H. S. Teoh wrote:
>> >On Thu, Feb 18, 2016 at 06:50:34PM +0000, Jonathan M Davis via Digitalmars-d wrote:
>> >>On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer wrote:
>> >[...]
>> >>>foo(void[] arr)
>> >>>{
>> >>>   void[] arr2 = [1234, 5678, 91011];
>> >>>   arr[] = arr2[0 .. arr.length];
>> >>>}
>> >>
>> >>Well, I'm not sure that that's actually not @safe.
>> >
>> >How can it possibly be @safe??? Consider:
>> >
>> >	void main() @safe {
>> >		Object[] objs = [ new Object() ];
>> >		foo(objs);
>> >	}
>> >
>> >Now the pointer to the Object instance has been corrupted.
>> 
>> Does it matter what state the void[] is in until you actually attempt to cast it to something else? If you have
>> 
>> Object[] oArr = [ new Object ];
>> void[] vArr = oArr;
>> vArr = vArr[0 .. $ - 1];
>> 
>> it's not like that's going to cause any problems - not by itself.
>
> I think you misread Steven's code.

Yes. It looks like I did.

> So the bottom line is that the array copy cannot be @safe.

Exactly. Passing it around is fine, but mutating it definitely is not.

> The larger question, is what, if anything, *can* you do with void[] besides read-only operations, that doesn't break @safe?  Once something is (implicitly or otherwise) cast to void[], all type information is forgotten, and there is no way, that I can tell, to write to a void[] without causing @safe breakage.  If so, wouldn't it make sense to require that the type should be const(void)[] rather than void[]?

Why? The problem isn't that void[] was passed it. It's that what was done to it after it was passed in was not @safe. We need to fix it so that the compiler doesn't consider mutating void[] or casting away from it or doing anything with it that could corrupt memory @safe, but passing it around is perfectly @safe, even if it's not very useful by itself. So, I see no reason to make any requirements about const. As long as dmd correctly catches the operations that aren't @safe, the function which is passed the void[] and does more than pass it around is going to be forced to be @system anyway. So making any requirements about const(void[]) buys us nothing.

- Jonathan M Davis

On Thu, 18 Feb 2016 19:17:27 +0000, Era Scarecrow wrote:

> On Thursday, 18 February 2016 at 18:41:25 UTC, Steven Schveighoffer wrote:
>> On 2/18/16 1:30 PM, Timon Gehr wrote:
>>> No problem here. There is no way to assign to a void[] without doing 2.
>>
>> foo(void[] arr)
>> {
>>    void[] arr2 = [1234, 5678, 91011];
>>    arr[] = arr2[0 .. arr.length];
>> }
> 
>   Since void throws away type information (and all the safety
> related to it), would it be easier to simply require @safe code can't
> cast implicitly to void? It seems like explicit casting would take care
> of most of this, or disallowing to/from void converting period while in
> @safe code.

Casting *from* void[] is also a big issue. Disallow all implicit and explicit casts between void[] and anything else to start, and we can look at the rest case-by-caste.

We can probably cast to const(void)[] safely, and we can probably cast arrays that contain no pointers to void[] safely. Casting from void[] to const(T)[] where T contains no pointers (or arrays or functions or delegates) should also be safe.

>   To be honest, I think there's only 1 time I actually used a
> void[] in my code, and that was while writing a section in the BitArray
> replacement code years back in the case you wanted to use/read another
> block of data as the source for the BitArray. Beyond that I never
> touched it.

A lot of the IO stuff in Phobos uses void[]. std.socket is lousy with it. I think the intention is that you can send arbitrary data over the wire without having to explicitly marshal it into a ubyte[].