On Wednesday, May 24, 2017 16:56:49 Timothee Cour via Digitalmars-d wrote:
> I could look at source to figure it out but others might wonder and I couldn't find it in the docs in https://dlang.org/library/std/functional/memoize.html whether memoize works per thread (thread local) or globally (__gshared)

Definitely thread-local, and there would be problems if it were shared, since for that to work properly, it would really need to be given either shared data or data that implicitly converted to shared. And while __gshared might skirt the compiler yelling at you, it would have the same issues; it's just that the compiler wouldn't be yelling at you about them, so it would be harder to catch them and more likely that you'd get weird, hard-to-track-down bugs.

- Jonathan M Davis

thanks; i think docs for this should still make that clear.

How about adding memoizeShared for shared variables?
There definitely are use cases for this.


On Wed, May 24, 2017 at 5:19 PM, Jonathan M Davis via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Wednesday, May 24, 2017 16:56:49 Timothee Cour via Digitalmars-d wrote:
>> I could look at source to figure it out but others might wonder and I couldn't find it in the docs in https://dlang.org/library/std/functional/memoize.html whether memoize works per thread (thread local) or globally (__gshared)
>
> Definitely thread-local, and there would be problems if it were shared, since for that to work properly, it would really need to be given either shared data or data that implicitly converted to shared. And while __gshared might skirt the compiler yelling at you, it would have the same issues; it's just that the compiler wouldn't be yelling at you about them, so it would be harder to catch them and more likely that you'd get weird, hard-to-track-down bugs.
>
> - Jonathan M Davis
>

On Thursday, 25 May 2017 at 01:17:41 UTC, Timothee Cour wrote:
> thanks; i think docs for this should still make that clear.
>
> How about adding memoizeShared for shared variables?
> There definitely are use cases for this.
>

Perhaps we should first actually properly document and implement what shared *is*.

On Thursday, 25 May 2017 at 03:51:05 UTC, Stanislav Blinov wrote:
> On Thursday, 25 May 2017 at 01:17:41 UTC, Timothee Cour wrote:
>> thanks; i think docs for this should still make that clear.
>>
>> How about adding memoizeShared for shared variables?
>> There definitely are use cases for this.
>>
>
> Perhaps we should first actually properly document and implement what shared *is*.

I'm looking into creating a small interest group composed of experts to lead this effort. We'd put a DIP together that nails down shared. Who is interested? -- Andrei

On Thursday, 25 May 2017 at 10:41:23 UTC, Andrei Alexandrescu wrote:
> On Thursday, 25 May 2017 at 03:51:05 UTC, Stanislav Blinov wrote:
>> On Thursday, 25 May 2017 at 01:17:41 UTC, Timothee Cour wrote:
>>> thanks; i think docs for this should still make that clear.
>>>
>>> How about adding memoizeShared for shared variables?
>>> There definitely are use cases for this.
>>>
>>
>> Perhaps we should first actually properly document and implement what shared *is*.
>
> I'm looking into creating a small interest group composed of experts to lead this effort. We'd put a DIP together that nails down shared. Who is interested? -- Andrei

Count me const scope.

On Thursday, May 25, 2017 10:41:23 Andrei Alexandrescu via Digitalmars-d wrote:
> On Thursday, 25 May 2017 at 03:51:05 UTC, Stanislav Blinov wrote:
> > On Thursday, 25 May 2017 at 01:17:41 UTC, Timothee Cour wrote:
> >> thanks; i think docs for this should still make that clear.
> >>
> >> How about adding memoizeShared for shared variables?
> >> There definitely are use cases for this.
> >
> > Perhaps we should first actually properly document and implement what shared *is*.
>
> I'm looking into creating a small interest group composed of experts to lead this effort. We'd put a DIP together that nails down shared. Who is interested? -- Andrei

I would be, though it is one of those topics where you think that you know what you're doing, and then you find out about some compiler optimization or other low level item that you don't know about, and some of what you thought was true wasn't. So, this is one area where I'm not sure that I'd ever be willing to call myself an expert no much how much I knew.

But we do need to get this ironed out well enough that we can definitely tell folks that shared is what it's going to be so that they'll stop using __gshared all over the place to try and work around shared - or at least if they do so, they won't be able to do so with the excuse that shared is not complete. But the idea that you do almost nothing with an object that is shared without casting it first (after protecting it appropriately with a mutex of course) except for using atomics seems to be too much for many folks, even though aside from the cast, the way you actually use a shared object is basically what you'd do in C/C++. I'd say that ultimately, shared is more for storing the object and protecting it against operations that could operate on it incorrectly than it is for actually operating on the object. Unfortunately, synchronized classes are the only construct that we've come up with that allows for the locking and casting to to be done safely and automatically rather than requiring an explicit cast, and not only do we not have synchronized classes yet, but even if we did they would only work on the outer layer of shared, so in many cases, you'd be forced to cast anyway.

I think that ultimately the big problems that we need to solve are making it clear what shared is and how you use it and ensuring that we have the memory model stuff ironed out well enough that the required casting and other operations are actually guaranteed to do what they're supposed to do. Then it's an issue of educating folks about shared rather than needing to fix anything about it. I don't expect that shared will never be entirely pretty, but ultimately, it's a lot like @safe/@system/@trusted in that it allows you to segregate the code that you need to worry about for a particular class of problem - and ultimately, the biggest benefit of shared is that everything else is thread-local.

- Jonathan M Davis

On Thursday, 25 May 2017 at 14:56:25 UTC, Jonathan M Davis wrote:
> able to do so with the excuse that shared is not complete. But the idea that you do almost nothing with an object that is shared without casting it first (after protecting it appropriately with a mutex of course) except for using atomics seems to be too much for many folks, even though aside from the cast, the way you actually use a shared object is basically what you'd do in C/C++. I'd say that ultimately, shared is more for storing the object and protecting it against operations that could operate on it incorrectly than it is for actually operating on the object.

You should look at Pony which has a sound type system for transitioning objects read/write local/shared:

https://tutorial.ponylang.org/capabilities/reference-capabilities.html

It is kinda like Rust, but more advanced.

On Thursday, May 25, 2017 19:57:17 Ola Fosheim Grøstad via Digitalmars-d wrote:
> On Thursday, 25 May 2017 at 14:56:25 UTC, Jonathan M Davis wrote:
> > able to do so with the excuse that shared is not complete. But the idea that you do almost nothing with an object that is shared without casting it first (after protecting it appropriately with a mutex of course) except for using atomics seems to be too much for many folks, even though aside from the cast, the way you actually use a shared object is basically what you'd do in C/C++. I'd say that ultimately, shared is more for storing the object and protecting it against operations that could operate on it incorrectly than it is for actually operating on the object.
>
> You should look at Pony which has a sound type system for transitioning objects read/write local/shared:
>
> https://tutorial.ponylang.org/capabilities/reference-capabilities.html
>
> It is kinda like Rust, but more advanced.

Oh, having some sort of ownership model or something similar in the type system which made it possible to know when something was only referenced once on a thread and thus could be passed safely to another and things like that would be very cool. For instance, it would make std.concurrency far easier to use safely with mutable data. The problem is that it seems that in order to for that to be possible, it adds quite a bit of complication to the language. Certainly, from what I know of Rust, it's far more complicated because of that sort of thing, and glancing over that link on Pony, it looks like it's getting a fair bit of complication as well in order to deal with the problem.

With D, having shared (and thus having everything else be thread-local) simplifies a lot of stuff, leaving the nasty stuff with regards to threads to shared, which is great, but dealing with the stuff that involves sharing across threads then requires that the programmer be much more careful than would be the case if the type system were actually helping you beyond preventing you from doing stuff that isn't thread safe to a shared object without casting it to thread-local first. So, we're stuck with an annoying area that would be great to clean up, but cleaning it up seems like it requires complicating everything else...

I don't know what the right answer is. Rust seems to get both praise and complaints about its approach - as do we for ours. But both approaches are safer than what you get with C/C++.

- Jonathan M Davis

On Thursday, 25 May 2017 at 11:15:19 UTC, Stanislav Blinov wrote:
> Count me const scope.

https://www.youtube.com/watch?v=YIp-0V6YKfQ

On Thursday, 25 May 2017 at 20:43:36 UTC, Jonathan M Davis wrote:
> complication to the language. Certainly, from what I know of Rust, it's far more complicated because of that sort of thing, and glancing over that link on Pony, it looks like it's getting a fair bit of complication as well in order to deal with the problem.

I think Pony uses a GC (also to collect dead threads/actors).

But I have found that trying to understand their model to be a good exercise for thinking about where problems can arise and what it takes to resolve it through a typesystem.

> stuff with regards to threads to shared, which is great, but dealing with the stuff that involves sharing across threads then requires that the programmer be much more careful than would be the case if the type system were actually helping you beyond preventing you from doing stuff that isn't thread safe to a shared object without casting it to thread-local first.

Yes, that transition to/from shared is problematic. There are ways to deal with it, proving concurrency patterns to be correct, but it takes even more machinery than Pony/Rust.

> I don't know what the right answer is. Rust seems to get both praise and complaints about its approach - as do we for ours. But both approaches are safer than what you get with C/C++.

Depends on what you do in C++. If you only share though a ready-made framework, then you probably can do quite well in terms of safety. With a small performance cost.

If you want max performance in the general case then I think all these languages will have safety troubles. Because you need proper full-blown verification then...