On Monday, 15 June 2015 at 12:08:31 UTC, Dmitry Olshansky wrote:
> On 13-Jun-2015 14:32, rsw0x wrote:
>> http://dlang.org/garbage.html
>>
>> Do not take advantage of alignment of pointers to store bit flags in the
>> low order bits:
>> p = cast(void*)(cast(int)p | 1);  // error: undefined behavior
>>
>> if this restriction is actually imposed - why does
>> std.bitmanip.tagged{ClassRef,Pointer} even exist?
>
> AFAIK the restriction was that pointers _themselves_ have to be stored at word-aligned addresses. This allows GC to scan memory cheaper w/o considering if some misaligned address may contain a pointer.

this doesn't make any sense, it's referring to an object p (of void*), not the location of p. It's setting the lsb of p to 1 and claiming it's undefined behavior, when it's clearly not.

Unless I misunderstand it.

On Monday, 15 June 2015 at 15:33:41 UTC, rsw0x wrote:
>
> this doesn't make any sense, it's referring to an object p (of void*), not the location of p. It's setting the lsb of p to 1 and claiming it's undefined behavior, when it's clearly not.
>
> Unless I misunderstand it.

`p` is assumed to be a pointer. `cast(int) p` gets the integer representation of the pointer (though it should be `uintptr_t` but whatever). It then does an integer bitwise or, then converts the result back to a pointer.

It's undefined behavior in D because the spec just said so (regardless of its defined behavior in C).

On 6/15/15 11:18 AM, Dmitry Olshansky wrote:
> On 15-Jun-2015 15:49, Steven Schveighoffer wrote:
>> On 6/15/15 8:08 AM, Dmitry Olshansky wrote:
>>> On 13-Jun-2015 14:32, rsw0x wrote:
>>>> http://dlang.org/garbage.html
>>>>
>>>> Do not take advantage of alignment of pointers to store bit flags in
>>>> the
>>>> low order bits:
>>>> p = cast(void*)(cast(int)p | 1);  // error: undefined behavior
>>>>
>>>> if this restriction is actually imposed - why does
>>>> std.bitmanip.tagged{ClassRef,Pointer} even exist?
>>>
>>> AFAIK the restriction was that pointers _themselves_ have to be stored
>>> at word-aligned addresses. This allows GC to scan memory cheaper w/o
>>> considering if some misaligned address may contain a pointer.
>>
>> That doesn't make sense. Why would you want to do this?
>>
>
> What exactly? Storing pointer at misaligned memory location - no good
> reason whatsoever. As in how it may happen - explicitly tightly packed
> structs with align(1) might end up doing this.
>
>  From http://dlang.org/phobos/core_memory.html which seems to be more
> recent:
>
> Implementations are free to assume that GC pointers are only stored on
> word boundaries. Unaligned pointers may be ignored entirely.

Right, and that's already assumed in (and would be ignored by) the current GC. What's NOT assumed (and I have no idea how we could make this assumption) is that interior pointers that point at parts of a larger struct/type/etc aren't valid GC pointers, and so the GC can ignore those too.

For example:

void *p = &someGCInt;
p++; // equivalent to p = cast(void *)(cast(size_t)p | 1);

The documentation above implies that if p is the only pointer to someGCInt, the GC may collect it. I think this is not possible for us to claim, even if we wanted to, given D's type system.

>> The only rational thing I can think of is that you wouldn't want to
>> store the result in an *actual* int pointer (lest it be used thinking it
>> was valid without masking out the lower bits). But the example is
>> storing it in a void *...
>>
>
> The example doesn't show what memory location that p refers to thus it's
> not possible to say if it's valid.

The whole premise (as stated in the doc right before the example) is that you know the low order bits are 0 :) this is the complete basis of std.bitmanip.taggedPtr. It's quite genius I think, and the fact that it works with GC (and IMO, any possible GC you could create for D) without issue is awesome. We should not be attaching a scarlet comment to the docs to warn people away from it.

-Steve

On 6/15/15 11:49 AM, Alex Parrill wrote:
> On Monday, 15 June 2015 at 15:33:41 UTC, rsw0x wrote:
>>
>> this doesn't make any sense, it's referring to an object p (of void*),
>> not the location of p. It's setting the lsb of p to 1 and claiming
>> it's undefined behavior, when it's clearly not.
>>
>> Unless I misunderstand it.
>
> `p` is assumed to be a pointer. `cast(int) p` gets the integer
> representation of the pointer (though it should be `uintptr_t` but
> whatever). It then does an integer bitwise or, then converts the result
> back to a pointer.
>
> It's undefined behavior in D because the spec just said so (regardless
> of its defined behavior in C).

Where? In the GC document? I don't think that document is correct in this regard. Undefined behavior is something different than implementation defined behavior, which is what this is. The spec says (http://dlang.org/type.html):

"Casting pointers to non-pointers and vice versa is allowed in D, however, do not do this for any pointers that point to data allocated by the garbage collector."

How can the compiler POSSIBLY know whether a pointer points at GC data in order to make it undefined behavior?

I don't see any justification for any of these restrictions. The "to allow maximum flexibility" justification needs more explanation, what possible feature could you implement given this restriction?

It even allows using unions to do exactly the same thing. There is just no way this cannot be allowed.

-Steve

On Monday, 15 June 2015 at 16:16:28 UTC, Steven Schveighoffer wrote:
> How can the compiler POSSIBLY know whether a pointer points at GC data in order to make it undefined behavior?

It doesn't need to, the undefined behaviour comes from the unfulfilled expectations of the GC. For example, a "sufficiently advanced GC" with enough knowledge of types, including of local vars, could by chance run just at the moment when the only instance of the address is stored in an int variable, and disregard that variable because it's not a pointer. It is of course unlikely that GCs will ever get as much knowledge about local variables, or even unnamed temporaries...

>
> I don't see any justification for any of these restrictions. The "to allow maximum flexibility" justification needs more explanation, what possible feature could you implement given this restriction?
>
> It even allows using unions to do exactly the same thing. There is just no way this cannot be allowed.

For unions, an advanced GC would know the type and would make sure to treat them conservatively. The restrictions are for the other cases where the types are references. The GC needs to rely on them being pointer if it wants to move them, for example.

But I think you are right that the current restrictions are too strong. After some thoughts, I can't really see any realistic problem with tagged pointers.

On 6/15/15 12:37 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
> On Monday, 15 June 2015 at 16:16:28 UTC, Steven Schveighoffer wrote:
>> How can the compiler POSSIBLY know whether a pointer points at GC data
>> in order to make it undefined behavior?
>
> It doesn't need to, the undefined behaviour comes from the unfulfilled
> expectations of the GC. For example, a "sufficiently advanced GC" with
> enough knowledge of types, including of local vars, could by chance run
> just at the moment when the only instance of the address is stored in an
> int variable, and disregard that variable because it's not a pointer. It
> is of course unlikely that GCs will ever get as much knowledge about
> local variables, or even unnamed temporaries...

That is clearly not the case in the documentation. 'p' maintains a pointer to the data the entire time, before and after it's augmented.

I'm perfectly OK changing the documentation to say "do not use integral types as the sole pointer to GC data." I can see how that may cause issues with the GC that has more advanced type information.

> But I think you are right that the current restrictions are too strong.
> After some thoughts, I can't really see any realistic problem with
> tagged pointers.

Thanks for the consideration.

-Steve

On Monday, 15 June 2015 at 12:49:25 UTC, Steven Schveighoffer wrote:
> On 6/15/15 8:08 AM, Dmitry Olshansky wrote:
>> On 13-Jun-2015 14:32, rsw0x wrote:
>>> http://dlang.org/garbage.html
>>>
>>> Do not take advantage of alignment of pointers to store bit flags in the
>>> low order bits:
>>> p = cast(void*)(cast(int)p | 1);  // error: undefined behavior
>>>
>>> if this restriction is actually imposed - why does
>>> std.bitmanip.tagged{ClassRef,Pointer} even exist?
>>
>> AFAIK the restriction was that pointers _themselves_ have to be stored
>> at word-aligned addresses. This allows GC to scan memory cheaper w/o
>> considering if some misaligned address may contain a pointer.
>
> That doesn't make sense. Why would you want to do this?
>
> The only rational thing I can think of is that you wouldn't want to store the result in an *actual* int pointer (lest it be used thinking it was valid without masking out the lower bits). But the example is storing it in a void *...
>
> -Steve

I think you miread it.

The memory the pointer points to can be misaligned.
The memory the pointer is in must be aligned.

Note that the second is mandatory is various hardware architecture to begin with, so it is not like it is a big constraint. It allow the GC to scan faster. It is all good.

The first one do not make any sense as GC need to support interior pointers anyway.