On Friday, April 20, 2012 23:45:11 Manu wrote:
> I need to clarify some things that have confused me a few times. I've reconsidered these basic attributes a few times, and I thought I understood them, but I obviously don't.
> 
> The case that has confused me is here: http://d.puremagic.com/issues/show_bug.cgi?id=7897
> 
> In the global scope:
> 
> int x; <- x is TLS
> 
> but:
> 
> static int x; <- this is... a 'static' global instance, whatever that means? Is this TLS or not? If so, how is it distinct from 'int x;' above? I presume it must still be TLS, and effectively meaningless at the global scope; dlang.org states "static is ignored when applied to other declarations". It is just used for members of struct/classes? Why not produce a syntax error rather than ignore it?

Attributes get ignored all over the place in D. There's at least one bug report, on it:

http://d.puremagic.com/issues/show_bug.cgi?id=3934

Whether that'll change or not, I don't know, since it's not clear that Walter considers it to be a problem from what I recall about his comments on discussions on it.

> immutable int x; <- this can't possibly change, so why would it be TLS? it must be a single static instance... right?

immutable is implicitly shared.

> __gshared int x; <- this should behave exactly like a C global right? ie, no TLS + addressable at compile time.

I believe so, yes.

> static immutable x; <- i've seen this a few times, what does it mean?

Assuming that static is valid in that context, it means that you have a static variable which is immutable. If it's not valid (e.g. module-level variable), then it's just an immutable variable. In either case, because it's immutable, it's implicitly shared.

> There's another thing that that's had me scratching my head for a while, and gave me the contradictory feeling that the use of TLS data is fairly relaxed:
> 
> struct S
> {
> int x;
> static int y;
> }
> 
> In this case y is static, ie, globally shared by all instances, but I
> expect it SHOULD be TLS, since it is mutable, and therefore not thread
> safe...
> That said, I write code that passes static members of structs into
> templates as alias parameters all the time, and generate code that writes
> to them.
> How is this possible? And given that it is, why is it not possible for at
> least __gshared at global scope?

I don't understand the problem. y is static, so there's a single instance for it for the class per thread, whereas each instance of S will have its own x. I think that I'd have to see actual code to understand exactly what your issue is with regard to templates.

> struct S
> {
> static x;
> __gshared y;
> }
> 
> What is the difference between x and y here?

x is in TLS, so there's one instance per thread. I'm not quite sure whether __gshared does anything here though, since it's a member variable rather than a class variable or module-level variable. I expect that __gshared is ignored here, but I don't know. If you had

struct S
{
 static x;
 static __gshared y;
}

then there would be one instance of y for the entire program, but it wouldn't be protected in the same way that it would be if it were shared.

> I've obviously missed something rather fundamental, but the dlang descriptions of each of these attributes are a little light, and leaves me quite unclear on the details here.

Yeah. The documentation in general tends to be a bit light, and this is definitely one area where it could use some fleshing out.

- Jonathan M Davis

On 21 April 2012 00:05, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Friday, April 20, 2012 23:45:11 Manu wrote:
> > I need to clarify some things that have confused me a few times. I've reconsidered these basic attributes a few times, and I thought I
> understood
> > them, but I obviously don't.
> >
> > The case that has confused me is here: http://d.puremagic.com/issues/show_bug.cgi?id=7897
> >
> > In the global scope:
> >
> > int x; <- x is TLS
> >
> > but:
> >
> > static int x; <- this is... a 'static' global instance, whatever that means? Is this TLS or not? If so, how is it distinct from 'int x;'
> above? I
> > presume it must still be TLS, and effectively meaningless at the global scope; dlang.org states "static is ignored when applied to other declarations". It is just used for members of struct/classes? Why not produce a syntax error rather than ignore it?
>
> Attributes get ignored all over the place in D. There's at least one bug report, on it:
>
> http://d.puremagic.com/issues/show_bug.cgi?id=3934
>
> Whether that'll change or not, I don't know, since it's not clear that
> Walter
> considers it to be a problem from what I recall about his comments on
> discussions on it.
>

Hmm, I tend to think it would be valuable to have a warning or error in the case of redundant attribution, that way I know I'm not polluting my code with spurious junk, but I can see there might be some implication for meta-programming.


> static immutable x; <- i've seen this a few times, what does it mean?
>
> Assuming that static is valid in that context, it means that you have a
> static
> variable which is immutable. If it's not valid (e.g. module-level
> variable),
> then it's just an immutable variable. In either case, because it's
> immutable,
> it's implicitly shared.
>

Implicitly 'shared', or implicitly '__gshared'? .. In this case, there should be no practical difference, but 'shared' comes loaded with some other bonuses that claim it may perform auto-locking when it is accessed, which is obviously not necessary for anything immutable.


> There's another thing that that's had me scratching my head for a while,
> > and gave me the contradictory feeling that the use of TLS data is fairly relaxed:
> >
> > struct S
> > {
> > int x;
> > static int y;
> > }
> >
> > In this case y is static, ie, globally shared by all instances, but I
> > expect it SHOULD be TLS, since it is mutable, and therefore not thread
> > safe...
> > That said, I write code that passes static members of structs into
> > templates as alias parameters all the time, and generate code that writes
> > to them.
> > How is this possible? And given that it is, why is it not possible for at
> > least __gshared at global scope?
>
> I don't understand the problem. y is static, so there's a single instance
> for
> it for the class *per thread*, whereas each instance of S will have its
> own x. I
> think that I'd have to see actual code to understand exactly what your
> issue
> is with regard to templates.
>

I think you've missed my point. If y is TLS, then it's location can't be
known at compile time, in which case, how is it possible to supply it an an
alias template parameter?
And given that it IS possible, I don't understand why it is not possible to
alias something that is __gshared.

This is the main thing I'm trying to understand.
What mechanisms affect whether the compiler knows the address of
static/global/shared variables at compile time or not? TLS being the
obvious one, but that seems to contradict observation in both cases here.


> struct S
> > {
> > static x;
> > __gshared y;
> > }
> >
> > What is the difference between x and y here?
>
> x is in TLS, so there's one instance per thread. I'm not quite sure whether
> __gshared does anything here though, since it's a member variable rather
> than
> a class variable or module-level variable. I expect that __gshared is
> ignored
> here, but I don't know. If you had
>
> struct S
> {
>  static x;
>  static __gshared y;
> }
>
> then there would be one instance of y for the entire program, but it
> wouldn't
> be protected in the same way that it would be if it were shared.
>

But the documentation for __gshared states that this is not necessary, it says quite clearly that __gshared is the same as 'static' in this context, but that doesn't make sense, since static should be TLS, but __gshared is not...

On 21 April 2012 00:50, Manu <turkeyman@gmail.com> wrote:

> On 21 April 2012 00:05, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
>>
>> Whether that'll change or not, I don't know, since it's not clear that
>> Walter
>> considers it to be a problem from what I recall about his comments on
>> discussions on it.
>>
>
> Hmm, I tend to think it would be valuable to have a warning or error in the case of redundant attribution, that way I know I'm not polluting my code with spurious junk, but I can see there might be some implication for meta-programming.
>

Or perhaps more significantly, I would know when I'm erroneously applying
attributes that don't have any effect, or don't make sense, I may have
actually meant something else, but it silently works, and I'm not given any
reason to think I made a mistake.
It may alleviate misunderstandings like this somewhat, and naturally give a
clearer understanding of what does what exactly.

Jonathan M Davis:

> http://d.puremagic.com/issues/show_bug.cgi?id=3934
>
> Whether that'll change or not, I don't know, since it's not clear that Walter considers it to be a problem from what I
> recall about his comments on discussions on it.

It's a significant problem because:
1) It's a _ridiculous_ situation. Allowing some of those things I've reported in bug 3934 is bad PR for D language. The DMD compiler looks incredibly sloppy compared to every other language compiler/interpreter, including old C compilers. It's about at 7.5 Mega KermitFrog sloppiness, that's almost PHP-level sloppy.
2) It makes much harder for D newbies to learn what's correct and what's not correct. When you are learning something new, you really want a strict teacher, otherwise you will have a much harder time learning what's correct (I've experimentally seen that even the total frienshipping inside D modules makes less easy to learn the usage of private/protected tags for new OOP D programmers).
3) It's confusing even for not newbie programmers, as Manu shows. And confusion is bad, error prone, makes people waste time, etc.
4) Different D compilers will be sloppy in slightly different ways, this means D code will not be portable across different compilers.


>> struct S
>> {
>> static x;
>> __gshared y;
>> }
>> 
>> What is the difference between x and y here?
>
> x is in TLS, so there's one instance per thread. I'm not quite sure whether
> __gshared does anything here though, since it's a member variable rather than
> a class variable or module-level variable. I expect that __gshared is ignored
> here, but I don't know.

I expect __gshared to not be ignored here. I think "__gshared" there is the same thing as "__gshared static".

Bye,
bearophile

On 4/20/12, Manu <turkeyman@gmail.com> wrote:
> I need to clarify some things that have confused me a few times.

When in doubt you can either do a few write calls:

import std.stdio;
import std.concurrency;

struct FooStruct
{
    static int FooX;
    __gshared int FooY;
}

void test()
{
    writeln(FooStruct.FooX);
    writeln(FooStruct.FooY);
}

void main()
{
    writeln(FooStruct.FooX);
    writeln(FooStruct.FooY);
    FooStruct.FooX = 1;
    FooStruct.FooY = 1;

    spawn(&test);
}

Or better yet look at the disassembly:

Here's FooX:
SECTION .tls$   align=16 noexecute                      ; section number 8, data
_D4test9FooStruct4FooXi:                                ; dword
        dd 00000000H                                    ; 0000 _ 0

Here's how it's loaded:
mov     eax, dword [__tls_index]                ; 0003 _ A1, 00000000(segrel)
mov     ecx, dword [fs:__tls_array]             ; 0008 _ 64: 8B. 0D,
00000000(segrel)
mov     edx, dword [ecx+eax*4]                  ; 000F _ 8B. 14 81
mov     eax, dword [_D4test9FooStruct4FooXi+edx]; 0012 _ 8B. 82, 00000000

And here's FooY:
SECTION .BSS    align=16 noexecute                      ; section number 4, bss
_D4test9FooStruct4FooYi:                                ; dword
        resd    1                                       ; 0004

And here's how it's loaded (notice the lack of having to calculate the
address based on TLS:
mov     eax, dword [_D4test9FooStruct4FooYi]    ; 001D _ A1, 00000004(segrel)

Adding static before __gshared doesn't seem to have any effect. DMD ignores a lot of nonsense keyword combinations. Maybe this will improve one day..

On Saturday, April 21, 2012 00:50:39 Manu wrote:
> On 21 April 2012 00:05, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > On Friday, April 20, 2012 23:45:11 Manu wrote:
> > static immutable x; <- i've seen this a few times, what does it mean?
> > 
> > Assuming that static is valid in that context, it means that you have a
> > static
> > variable which is immutable. If it's not valid (e.g. module-level
> > variable),
> > then it's just an immutable variable. In either case, because it's
> > immutable,
> > it's implicitly shared.
> 
> Implicitly 'shared', or implicitly '__gshared'? .. In this case, there should be no practical difference, but 'shared' comes loaded with some other bonuses that claim it may perform auto-locking when it is accessed, which is obviously not necessary for anything immutable.

It's implicitly shared across threads. I'm not sure that it's really either shared or __gshared. Any protections that shared has which would be applicable to immutable would certainly be on it though. However, many of them aren't properly implemented on shared yet anyway. It's just that because you can't ever assign to an immutable variable after initializing it, it's safe for the compiler to not put in TLS. Certainly, whenever it's discussed, it's stated that immutable is implicitly shared and _not_ that it's implicitly __gshared. I don't think that I've ever seen __gshared come up in a discussion on immutable before.

One place where the sharing of immutable becomes evident is in

http://d.puremagic.com/issues/show_bug.cgi?id=6114

It should be illegal to initialize immutable module-level and immutable static variables in non-static constructors, but it isn't at present. So if you do, you can actually initialize immutable variables multiple times (which obviously good). Presumably, it'll be fixed at some point though. Regardless, it shows how immutable variables aren't put in TLS.

> > There's another thing that that's had me scratching my head for a while,
> > 
> > > and gave me the contradictory feeling that the use of TLS data is fairly relaxed:
> > > 
> > > struct S
> > > {
> > > int x;
> > > static int y;
> > > }
> > > 
> > > In this case y is static, ie, globally shared by all instances, but I
> > > expect it SHOULD be TLS, since it is mutable, and therefore not thread
> > > safe...
> > > That said, I write code that passes static members of structs into
> > > templates as alias parameters all the time, and generate code that
> > > writes
> > > to them.
> > > How is this possible? And given that it is, why is it not possible for
> > > at
> > > least __gshared at global scope?
> > 
> > I don't understand the problem. y is static, so there's a single instance
> > for
> > it for the class *per thread*, whereas each instance of S will have its
> > own x. I
> > think that I'd have to see actual code to understand exactly what your
> > issue
> > is with regard to templates.
> 
> I think you've missed my point. If y is TLS, then it's location can't be
> known at compile time, in which case, how is it possible to supply it an an
> alias template parameter?
> And given that it IS possible, I don't understand why it is not possible to
> alias something that is __gshared.
> 
> This is the main thing I'm trying to understand.
> What mechanisms affect whether the compiler knows the address of
> static/global/shared variables at compile time or not? TLS being the
> obvious one, but that seems to contradict observation in both cases here.

Please give code examples. I'm much more likely to understand what you mean that way.

> > struct S
> > 
> > > {
> > > static x;
> > > __gshared y;
> > > }
> > > 
> > > What is the difference between x and y here?
> > 
> > x is in TLS, so there's one instance per thread. I'm not quite sure
> > whether
> > __gshared does anything here though, since it's a member variable rather
> > than
> > a class variable or module-level variable. I expect that __gshared is
> > ignored
> > here, but I don't know. If you had
> > 
> > struct S
> > {
> > 
> > static x;
> > static __gshared y;
> > 
> > }
> > 
> > then there would be one instance of y for the entire program, but it
> > wouldn't
> > be protected in the same way that it would be if it were shared.
> 
> But the documentation for __gshared states that this is not necessary, it says quite clearly that __gshared is the same as 'static' in this context, but that doesn't make sense, since static should be TLS, but __gshared is not...

static shouldn't have anything to do with TLS beyond the fact that it doesn't make any sense for a non-static member variable to be marked as shared or __gshared, since whether it's in TLS or not is determined whether the object that it's in is TLS or not.

In the case of a struct or class, static merely states that there's only one instance per class/struct rather than it being a member variable, not anything about TLS. And if the documentation states that the static is implied for __gshared (I haven't read it recently), then that that just means that __gshared is always static so that you don't get the weird situation where you have a __gshared member variable (which would be meaningless as far as I can tell).

- Jonathan M Davis

On Saturday, April 21, 2012 00:05:01 bearophile wrote:
> Jonathan M Davis:
> > http://d.puremagic.com/issues/show_bug.cgi?id=3934
> > 
> > Whether that'll change or not, I don't know, since it's not clear that Walter considers it to be a problem from what I recall about his comments on discussions on it.
> 
> It's a significant problem because:
> 1) It's a _ridiculous_ situation. Allowing some of those things
> I've reported in bug 3934 is bad PR for D language. The DMD
> compiler looks incredibly sloppy compared to every other language
> compiler/interpreter, including old C compilers. It's about at
> 7.5 Mega KermitFrog sloppiness, that's almost PHP-level sloppy.
> 2) It makes much harder for D newbies to learn what's correct and
> what's not correct. When you are learning something new, you
> really want a strict teacher, otherwise you will have a much
> harder time learning what's correct (I've experimentally seen
> that even the total frienshipping inside D modules makes less
> easy to learn the usage of private/protected tags for new OOP D
> programmers).
> 3) It's confusing even for not newbie programmers, as Manu shows.
> And confusion is bad, error prone, makes people waste time, etc.
> 4) Different D compilers will be sloppy in slightly different
> ways, this means D code will not be portable across different
> compilers.

I tend to agree. There might be situations (particularly with regards to generic programming) where some attributes need to be ignored, but in general, it's definitely bad. The problem is getting someone to implement the changes and get Walter to accept them, since from what I recall, he doesn't really view it as a problem.

- Jonathan M Davis

On 21 April 2012 01:17, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> On 4/20/12, Manu <turkeyman@gmail.com> wrote:
> > I need to clarify some things that have confused me a few times.
>
> When in doubt you can either do a few write calls:
>

Sure, but as I'm trying to say, I'm already suspicious the compiler isn't doing the 'right' thing in the first place... I want to rather know what the word is, ie, the conceptual definition.


import std.stdio;
> import std.concurrency;
>
> struct FooStruct
> {
>    static int FooX;
>    __gshared int FooY;
> }
>
> void test()
> {
>    writeln(FooStruct.FooX);
>    writeln(FooStruct.FooY);
> }
>
> void main()
> {
>    writeln(FooStruct.FooX);
>    writeln(FooStruct.FooY);
>    FooStruct.FooX = 1;
>    FooStruct.FooY = 1;
>
>    spawn(&test);
> }
>
> Or better yet look at the disassembly:
>
> Here's FooX:
> SECTION .tls$   align=16 noexecute                      ; section number
> 8, data
> _D4test9FooStruct4FooXi:                                ; dword
>        dd 00000000H                                    ; 0000 _ 0
>
> Here's how it's loaded:
> mov     eax, dword [__tls_index]                ; 0003 _ A1,
> 00000000(segrel)
> mov     ecx, dword [fs:__tls_array]             ; 0008 _ 64: 8B. 0D,
> 00000000(segrel)
> mov     edx, dword [ecx+eax*4]                  ; 000F _ 8B. 14 81
> mov     eax, dword [_D4test9FooStruct4FooXi+edx]; 0012 _ 8B. 82, 00000000
>
> And here's FooY:
> SECTION .BSS    align=16 noexecute                      ; section number
> 4, bss
> _D4test9FooStruct4FooYi:                                ; dword
>        resd    1                                       ; 0004
>
> And here's how it's loaded (notice the lack of having to calculate the
> address based on TLS:
> mov     eax, dword [_D4test9FooStruct4FooYi]    ; 001D _ A1,
> 00000004(segrel)
>
> Adding static before __gshared doesn't seem to have any effect. DMD ignores a lot of nonsense keyword combinations. Maybe this will improve one day..
>

Right, this is precisely what I expect, and apparently the compiler knows
enough about that variable that even when addressing an alias at compile
time, it can still generate the appropriate TLS code. So that shows that
TLS is not a problem for compile time addressing, it is only important that
it is statically addressable.
So... back to my bug that Walter rejected, I don't understand the problem?

On 21 April 2012 01:48, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> Please give code examples. I'm much more likely to understand what you mean that way.
>

The code in question was in my OP, but that said...


static shouldn't have anything to do with TLS beyond the fact that it
> doesn't
> make any sense for a non-static member variable to be marked as shared or
> __gshared, since whether it's in TLS or not is determined whether the
> object
> that it's in is TLS or not.
>
> In the case of a struct or class, static merely states that there's only
> one
> instance per class/struct rather than it being a member variable, not
> anything
> about TLS. And if the documentation states that the static is implied for
> __gshared (I haven't read it recently), then that that just means that
> __gshared is always static so that you don't get the weird situation where
> you
> have a __gshared member variable (which would be meaningless as far as I
> can
> tell).
>

Okay, I've just done some more tests taking what has been said here.
I realise I wasn't confused to begin with, I've been introducing confusion
into my brain in order to attempt to understand why my test case was
rejected.

I've simplified it, like so:


// member contained in a struct
struct Thing
{
    int x;
}

Thing thing; // this is effectively identical to declaring 'int x;' globally, it's just wrapped in a thin struct

int x; // i'll also do it directly, to prove it works.

void main()
{
    // these 3 statements should be effectively identical
    thing.x = 0; // this works, obviously
    AliasTheStruct!( thing )(); // this works
    AliasTheInt!( thing.x )();  // this doesn't

    AliasTheInt!( x )();        // of course, this works fine
}

void AliasTheStruct( alias x )()
{
    x.x = 10;
}

void AliasTheInt( alias x )()
{
    x = 10;
}


As far as I can tell, this should work. It doesn't. This was my bug report.
Is this a legitimate bug?
This has hindered a lot of my code for days, cost me a lot of time, and I
see no technical reason why it shouldn't work.

All the __gshared/static business was confusion that was seeded while trying to understand what Walter meant while rejecting my bug request. I do clearly understand all that stuff after all ;) .. lets forget I brought that up, although I think the observation that it is rather confusing in the first place, and the fact the compiler silently ignores invalid attributes might be significant. There's probably some room for improvement there.