On Tuesday, 18 June 2019 at 22:30:28 UTC, Jonathan M Davis wrote:
> Storing a reference to a scope object in @trusted code would violate what scope is supposed to guarantee.

Ok, so you are only considering stack allocated objects?

My question is more like this. If I use GC throughout, then I might write a @trusted framework with that in mind, and take references wherever it makes sense. No problem, right?

However, when someone pass a shared (still GC managed) object that has been temporarily "unshared", and pass it into that to the framework... then it will break badly.

So, from a memory-management perspective the framework is sound, but not from a thread safety point of view.

Is that right?


> but if your @trusted code does anything which violates the guarantees that @safe is supposed to make, then the programmer who verified that it was okay to mark it as @trusted screwed up.

But only if it was vetted with "shared" in mind, because if it was vetted for safe release of memory, and you use GC or reference counting, then it was perfectly OK, but still breaks for "unshared" shared?

Maybe this will demand too much from the person doing the vetting of generic code, to both get it right for memory management and thread safety?

> @safe really only deals with memory safety, not thread-safety. Converting between shared and thread-local does require a cast, which is @trusted, because you're basically stepping outside of the type system.

Ok, but what if you didn't step outside the type system? What would D need to cover this within the type system?

> What Manu is proposing is a scenario where the type system is able to guarantee that no references to the variable escape and that based on that assumption, temporarily converting to shared wouldn't violate the guarantees that come with the object actually being thread-local.

That sounds reasonable.

> scope object actually escaped. If the programmer screws that up, then the implicit conversion to shared will have violated the guarantees that are supposed to go with the object being thread-local, and unlike now, the point of the conversion

Yes, but this is where a more elaborate type system would help. I haven't mentioned Pony
( https://www.ponylang.io/ ) in a while, so to recap:

Pony tracks whether an object-reference is unique among other things.

https://tutorial.ponylang.io/reference-capabilities/reference-capabilities.html

So, that might be worth considering.

I don't think it has to be tedious if you use auto/type deduction/flow typing.

> On the surface, what Manu is proposing _seems_ sound (assuming that any @trusted code involved is vetted properly), but as Walter points out, it's really easy to screw up threading stuff.

Yes, that is true, and "shared" isn't even the most challenging problem. The real challenge is to prove that starvation/deadlock cannot happen in a complex system.

(which of course is a good reason to avoid complex systems in the first place)

> Also, normally, shared objects either deal with their thread-safety stuff internally (e.g. by having an internal mutex that locks appropriately when accessing the object's members),

Only works in very simple scenarios, you often need to grab multiple resources at once. Also, fine granularity locks tend to come with high performance penalties.

So, I think one should exclude that solution and focus on the general case. Single object sync isn't really a solution that will convince anyone that a language has nailed concurrency…

> or they require that you deal with the thread synchronization stuff explicitly (e.g. by directly dealing with the mutex whenever the shared object needs to be accessed).

Yes, I agree if you change "object" to "objects".

> Having a function that's expecting a shared variable be given a thread-local one seems off to me.

Well, if you use reference-capabilites in the vein of Pony then I think it follows naturally.

And that appears to be what D is trying to do with "shared", but without the precision needed to solve issues that seems reasonable to people working in the trenches (like Manu).

> shared without letting it escape, but I'm still inclined to think that the conversion should be vetted by the programmer rather than being considered okay and done implicitly just because scope is involved.

But, to play the devil's advocate:

If the compiler doesn't provide strong semantic passes for "shared" why do you then need "shared" to be part of the type system?  Why couldn't you then just let "shared" be implemented as a template within a templated pointer framework?

For "shared" to be justified as a language feature it has to provide something that cannot be done within the meta-programming capabilities of the language.

For "const" it is obvious, the transitive const cannot be done within the meta-programming capabilities of the language (or maybe it can, I am making an assumption).

But how does "shared" justify itself?  Is it all about @safe/@trusted/@system?

In that case, maybe those capabilities could be available as meta-programming mechanisms so that "shared" could be done as a library feature.

Or rather, what prevents "shared" from being a library feature (assuming all pointers are templates).

> Regardless, as discussed at dconf this year, D's memory model and the exact semantics of shared really need to be properly locked down before we start making changes like this.

Indeed.

> but not all of the details have been properly ironed out yet, and the devil is in the details.

Certainly.

I would also argue that whatever you land on for managing a resource like memory, also should be considered to be extended to manage a "resource" like read/write access to shared objects.

Ola.

On Tuesday, 18 June 2019 at 23:30:07 UTC, Manu wrote:
> `scope` has a lot to say in this whole space; it must prevent escaping references that are only intended to have a finite life. We may reach limitations with `scope` today and need to make improvements, but that's fine. Some limitations in this space are fine to work through... but even in lieu of watertight solutions, we can make working solutions which are very helpful and hard to break unless you deliberately go out of your way to do so.

Right, but a type system should not be evolved, it should be designed as a whole where everything fits together in a wholesome manner.

So, what you are asking for isn't unreasonable, but to me it suggests that the type system as a whole should be reconsidered.

> The worst thing that happens is that we stop making progress because a limitation of this sort inhibits development on other axiis. We're often too timid, and it hurts D's velocity immensely.

Maybe, I think the special cases hurts. The documentation for functions is suitable for scaring away most programmers with all the various ways to specify parameters and return types:

https://dlang.org/spec/function.html

Function prototypes ought to be simple to write…

I think D would benefit from taking a good hard look on what has to be in the language, what could be done as a library feature, and how the metaprogramming capabilites could be extended to move more out of the language and into libraries.

Moving more to the std library would not mean that compilation would slow down, as compilers could optimize std library features (hardcore the mechanism in a compatible manner).

But that is only my opinion, although I think it is the ideal in language design that most people would agree on (in theory at least).

Ola.

On Wednesday, June 19, 2019 1:29:21 AM MDT Ola Fosheim Grøstad via Digitalmars-d wrote:
> On Tuesday, 18 June 2019 at 22:30:28 UTC, Jonathan M Davis wrote:
> > Storing a reference to a scope object in @trusted code would violate what scope is supposed to guarantee.
>
> Ok, so you are only considering stack allocated objects?
>
> My question is more like this. If I use GC throughout, then I might write a @trusted framework with that in mind, and take references wherever it makes sense. No problem, right?
>
> However, when someone pass a shared (still GC managed) object that has been temporarily "unshared", and pass it into that to the framework... then it will break badly.
>
> So, from a memory-management perspective the framework is sound, but not from a thread safety point of view.
>
> Is that right?

scope doesn't care where the object is stored. If something is scope, then you can't take any references to it in @safe code. The whole point is to ensure that no references escape. Whether the objects are GC-allocated are not is irrelevant.

> > but if your @trusted code does anything which violates the guarantees that @safe is supposed to make, then the programmer who verified that it was okay to mark it as @trusted screwed up.
>
> But only if it was vetted with "shared" in mind, because if it was vetted for safe release of memory, and you use GC or reference counting, then it was perfectly OK, but still breaks for "unshared" shared?
>
> Maybe this will demand too much from the person doing the vetting of generic code, to both get it right for memory management and thread safety?

The only thing that @safe has to do with shared is that casting to or from shared is @system. And _any_ code which is @trusted needs to maintain all of the guarantees that go with @safe, or the programmer screwed up.

For Manu's suggestion to work, it would have to be guaranteed that @trusted code vetted purely on the basis of memory safety (which is what @safe/@trusted/@system is all about) would not run afoul of threading issues, because the compiler implicitly cast a thread-local object to shared. If that can't be guaranteed, then Manu's suggestion isn't tenable.

> > @safe really only deals with memory safety, not thread-safety. Converting between shared and thread-local does require a cast, which is @trusted, because you're basically stepping outside of the type system.
>
> Ok, but what if you didn't step outside the type system? What would D need to cover this within the type system?

D's type system would actually have to understand ownership and concurrency so that it would know when it was safe to implicitly cast to or from shared. As it is, in general, D can't even know if there are multiple references to the same object. Without that kind of information, it's not even possible to know whether it's safe to pass an object from one thread to another. And without some understanding of which concurrency primitives are being used to protect an object or group of objects, it can't do something as simple as know that locking a mutex has actually protected those objects such that they can safely be manipulated from the current thread.

I don't know exactly what it would look like for the type system to have the kind of information it would need to be able to implicitly cast to or from shared, but I think that it's pretty clear that we're not going to actually do whatever would be necessary to make that happen, because it would require significant changes to how D works. Even the basics of how pointers currently work wouldn't work with a system that had to keep track of ownership or how many references to an object existed, and even if we were willing to make such changes to D (and I'm quite sure that Walter is not), how pointers currently work in D is pretty critical for how D code interacts with C/C++ code. I'm pretty sure that most anything that requires significant changes to D's type system is never going to happen. Even something like what DIP 1000 has done with scope has been a royal pain to get to where it is.

I think that it's quite clear at this point that most anything involving shared is going to require that the programmer handle it all properly (including converting to or from shared) and that there will be little to no help from the compiler outside of preventing certain things in @safe code and/or without casting. _Maybe_, there will be a few, small places like with what Manu is suggesting here where we will be able to leverage what the compiler knows to implicitly convert something, but in general, that's really not going to work.

> > What Manu is proposing is a scenario where the type system is able to guarantee that no references to the variable escape and that based on that assumption, temporarily converting to shared wouldn't violate the guarantees that come with the object actually being thread-local.
>
> That sounds reasonable.
>
> > scope object actually escaped. If the programmer screws that up, then the implicit conversion to shared will have violated the guarantees that are supposed to go with the object being thread-local, and unlike now, the point of the conversion
>
> Yes, but this is where a more elaborate type system would help. I
> haven't mentioned Pony
> ( https://www.ponylang.io/ ) in a while, so to recap:
>
> Pony tracks whether an object-reference is unique among other things.
>
> https://tutorial.ponylang.io/reference-capabilities/reference-capabilities .html
>
> So, that might be worth considering.
>
> I don't think it has to be tedious if you use auto/type deduction/flow typing.

I'd be very surprised if anything like that were considered acceptable for D. When stuff like Rust's borrowing has been brought up before, Walter and Andrei have made it pretty clear that we're not going to do something like that with D's type system. Even getting something like DIP 1000 has been a major ordeal, and I don't think that it would have ever happened unless Walter had been convinced that it was absolutely needed for @safe code to be able to do stuff like reference counting.

> > Having a function that's expecting a shared variable be given a thread-local one seems off to me.
>
> Well, if you use reference-capabilites in the vein of Pony then I think it follows naturally.
>
> And that appears to be what D is trying to do with "shared", but without the precision needed to solve issues that seems reasonable to people working in the trenches (like Manu).

All that D is trying to do with shared is separate thread-local data from shared data. That's why shared exists. That way, the vast majority of code doesn't have to care about threads, and the code that does have to care about it is clearly segregated. Not much thought beyond that went into the initial creation of shared. It was simply the natural side effect of making everything thread-local by default.

And shared works fine as-is if you're dealing with concurrency in the same way that you would in C++. It's just that it requires some additional casting, because C++ doesn't have thread-local vs shared as part of its type system. The primary things that shared lacks at this point are:

1. Making read/write operations illegal on shared objects, since they're not
thread-safe.
2. Updating the library primitives for concurrency to use shared properly
(in particular, the stuff in core.sync was written prior to shared really
being a thing, and it hasn't been properly updated).
3. The memory model needs to be ironed out so that it's clear what exactly
happens with shared. Presumably, we're going to end up with something that's
basically C++'s memory model with any tweaks that need to be made for D, but
the details still need to be figured out.

There are likely some other things that would come up with ironing out the details, but with those three main things ironed out, shared should largely be where it needs to be. It does mean that casting to/from shared will be required, and such code will have to be @trusted, but everything necessary to use shared is there. It just doesn't involve the compiler doing much for you except protecting you from doing stuff that's clearly wrong unless you cast, at which point, the code is @system/@trusted and easily tracked down so that it can be properly vetted to make sure that it does the concurrency stuff correctly.

Additional improvements on top of that would be nice, but since there's pretty much no way that significant changes are going to be made to D's type system at this point, it's pretty questionable that we're going to be able to do much in the way of things like implicitly casting away shared. The only viable feature along those lines that's been proposed thus far is TDPL's synchronized classes, and even those could only strip away the outer layer of shared (i.e. what's directly in the object), making them pretty useless - not to mention, they require classes, which doesn't fit in well with how D code is normally written.

> > shared without letting it escape, but I'm still inclined to think that the conversion should be vetted by the programmer rather than being considered okay and done implicitly just because scope is involved.
>
> But, to play the devil's advocate:
>
> If the compiler doesn't provide strong semantic passes for "shared" why do you then need "shared" to be part of the type system?  Why couldn't you then just let "shared" be implemented as a template within a templated pointer framework?
>
> For "shared" to be justified as a language feature it has to provide something that cannot be done within the meta-programming capabilities of the language.
>
> For "const" it is obvious, the transitive const cannot be done within the meta-programming capabilities of the language (or maybe it can, I am making an assumption).
>
> But how does "shared" justify itself?  Is it all about @safe/@trusted/@system?
>
> In that case, maybe those capabilities could be available as meta-programming mechanisms so that "shared" could be done as a library feature.
>
> Or rather, what prevents "shared" from being a library feature
> (assuming all pointers are templates).

shared exists so that almost everything in D can be treated as thread-local, and the type system prevents you from converting between thread-local and shared without casting. Even immutable ends up as part of it, because it's implicitly shared. The compiler knows which objects are thread-local and which are shared and can use that information in code generation. It also means that it's possible to prevent operations on shared objects that are not clearly thread-safe, thereby making it so that if the programmer is going to screw up with objects that are shared across threads, they're going to have to cast to force the matter. @system/@trusted then gets involved in the sense that the cast is @system, but beyond that, @safety doesn't have anything to do with it. I don't see how any of that could be done via a library rather than the type system.

- Jonathan M Davis

On Monday, June 17, 2019 5:46:44 PM MDT Manu via Digitalmars-d wrote:
> Is this valid?
>
> int x;
> void fun(scope ref shared(int) x) { ... }
> fun(x); // implicit promotion to shared in this case
>
> This appears to promote a thread-local to shared. The problem with such promotion is that it's not valid that a thread-local AND a shared reference to the same thing can exist at the same time.
>
> With scope, we can guarantee that the reference doesn't escape the callee. Since the argument is local to the calling thread, and since the calling thread can not be running other code at the same time as the call is executing, there is no way for any code to execute with a thread-local assumption while the callee makes shared assumptions.
>
> I think this might be safe?

Another thing to consider here is something that's come up before when you've made suggestions like this - which is that unlike shared, scope is not transitive. So, even if what you're suggesting here would work with int, it would not work with types in general.

- Jonathan M Davis

On Wednesday, 19 June 2019 at 08:45:10 UTC, Jonathan M Davis wrote:
> scope doesn't care where the object is stored. If something is scope, then you can't take any references to it in @safe code. The whole point is to ensure that no references escape. Whether the objects are GC-allocated are not is irrelevant.

So, the same as this:
https://dlang.org/spec/function.html#return-scope-parameters

Does this mean then that when you obtain write access to an object that "write access reference" will be scope qualified?

And then you cannot use that reference with any function that does not have scope qualified parameters?

So basically all library code has to add "scope" to all its parameters whether it is written with shared in mind or not.

So, "scope" becomes like "pure". Something that ought to be the default, but has to be added manually to all function prototypes in order to make safe multi-threaded programming less annoying.

Or will the compiler automatically deduce that a function parameter is fulfilling the "scope" requirements even when it has not been specified?

I guess it should, otherwise you'll end up with the "const" transition in C++, where you had to do ugly cast-hacks when calling functions that did treat parameters as const but the function signature had not specified it.

> implicitly cast to or from shared. As it is, in general, D can't even know if there are multiple references to the same object. Without that kind of information, it's not even possible to know whether it's safe to pass an object from one thread to another.

You can do this with dataflow in many cases, maybe in most useful cases. Although something like unique_ptr in C++ helps (by convention).

> And without some understanding of which concurrency primitives are being used to protect an object or group of objects, it can't do something as simple as know that locking a mutex has actually protected those objects such that they can safely be manipulated from the current thread.

Right. But you could do something dynamic with reference counting and "framework conventions".

e.g..:

auto dblock = database.allocate_lock();
auto obj_ref_lockcounting = dblock.require_write_access(obj_id);

Then maybe you just need some metaprogramming mechanisms to inform the type system what the framework "reference_counting_lock_object" does, so that the type system knows what objects have been locked and that when dblock goes out of scope then all locks have been released.

Seems plausible, but I haven't given it much thought.

> be necessary to make that happen, because it would require significant changes to how D works.

Maybe, although I wonder how much you could do as library code. So I don't know.

If metaprogramming is going to be the main focus of D then it makes sense to focus on allowing library authors to inform the type system of what their library types are capable of. Basically the opposite of a traditional type system where the typesystem checks constraints.

So, the basic idea would be that the library tells the type system "my code provide these guarantees", rather than the the library asking the type system to verify that a set of guarantees hold.

> casting. _Maybe_, there will be a few, small places like with what Manu is suggesting here where we will be able to leverage what the compiler knows to implicitly convert something, but in general, that's really not going to work.

I think that make such special cases is a bit dangerous. It is really going down the path of C++ that leads to a situation where you only appeal to the same audience as C++.

It also leaves the field open to Rust. I agree that borrowing might be tedious for memory safety, but might be generally suitable for making safer multi-threading available to more programmers. At least on the surface level, but I haven't spent a lot of time thinking about the limitations of borrow-checking in relation to multi-threading. So, it is just a hunch.

I think it would be a mistake to outright dismiss tracking references.  There might be libraries available that can do it for you even, e.g. provide a compiler switch that enables the compiler to generate a graph that is passed onto an external library that does the checking.

> I'd be very surprised if anything like that were considered acceptable for D. When stuff like Rust's borrowing has been brought up before, Walter and Andrei have made it pretty clear that we're not going to do something like that with D's type system.

Ok, but there are two issues here:

1. What you can query the type system for.

This has to be available at all times and has to be tightly integrated into the type system. There is a high cost to adding such features.

2. Type annotations that enable verification.

This does not have to be tightly integrated into the type system. You can provide it as a separate module with clear boundaries. And you might even be able to turn it off for faster recompilation.
There is a low cost to adding such features.


> Even getting something like DIP 1000 has been a major ordeal, and I don't think that it would have ever happened unless Walter had been convinced that it was absolutely needed for @safe code to be able to do stuff like reference counting.

DIP1000 looks too limiting, but I don't know what it has enabled and where it falls short in practice.

I guess strength and weaknesses will show up as people build new frameworks around it.

> shared exists so that almost everything in D can be treated as thread-local, and the type system prevents you from converting between thread-local and shared without casting. Even immutable ends up as part of it, because it's implicitly shared. The compiler knows which objects are thread-local and which are shared and can use that information in code generation.

Ok, so the basic idea is to allow D to shuffle instructions/code more around because the compiler can assume that nothing held in registers is ever stale.

And "shared" was just added to push "everything multi-threaded" into the future. So, now you have to figure out what that means in order to make it more useful to people who do more advanced programming in D (like Manu).

> force the matter. @system/@trusted then gets involved in the sense that the cast is @system, but beyond that, @safety doesn't have anything to do with it. I don't see how any of that could be done via a library rather than the type system.

Well, it is currently just a question. And in my view the answer is not known.

I think there might be options to do it if we change your view on what a type system is from:

1. The usage of the type system specifies what the compiler should verify.

to

2. Vetted libraries can inform the type system of what it is capable of.


Ola.

On Wednesday, June 19, 2019 3:56:18 AM MDT Ola Fosheim Grøstad via Digitalmars-d wrote:
> On Wednesday, 19 June 2019 at 08:45:10 UTC, Jonathan M Davis
>
> wrote:
> > scope doesn't care where the object is stored. If something is scope, then you can't take any references to it in @safe code. The whole point is to ensure that no references escape. Whether the objects are GC-allocated are not is irrelevant.
>
> So, the same as this: https://dlang.org/spec/function.html#return-scope-parameters
>
> Does this mean then that when you obtain write access to an object that "write access reference" will be scope qualified?
>
> And then you cannot use that reference with any function that does not have scope qualified parameters?

I don't follow. What do you mean by obtain write access? If you're talking about shared, scope doesn't care. It just makes sure that references can't escape in @safe code. Currently, scope does nothing special with shared at all. All scope does is prevent escaping in @safe code. You use the variables pretty much the same as normal otherwise, though the fact that you can't escape any references can potentially get very restrictive.

> So basically all library code has to add "scope" to all its parameters whether it is written with shared in mind or not.
>
> So, "scope" becomes like "pure". Something that ought to be the default, but has to be added manually to all function prototypes in order to make safe multi-threaded programming less annoying.

Actually, using scope when you don't need to would be a serious problem. For instance, scope does _not_ interact well with range-based code. The typical thing for a lot of range-based functions to do is to take the argument, wrap it in a struct that's a new range, and return it, whereas if the parameter were scope, then you couldn't put the argument or any reference to it in the struct.

> Or will the compiler automatically deduce that a function parameter is fulfilling the "scope" requirements even when it has not been specified?
>
> I guess it should, otherwise you'll end up with the "const" transition in C++, where you had to do ugly cast-hacks when calling functions that did treat parameters as const but the function signature had not specified it.

AFAIK, the only time that scope is inferred is when auto is used for a variable declaration, and the value assigned to it is already scope. And being unable to escape references is a big enough hinderance in many cases that I don't know how useful it's really going to be in practice outside of code that simply operates on pointers. How useful it's going to be in practice is really an open question at this point.

> > implicitly cast to or from shared. As it is, in general, D can't even know if there are multiple references to the same object. Without that kind of information, it's not even possible to know whether it's safe to pass an object from one thread to another.
>
> You can do this with dataflow in many cases, maybe in most useful cases. Although something like unique_ptr in C++ helps (by convention).

Walter is against data flow analysis in most cases largely because when language semantics depend on it, you have to specify exactly how the data flow analysis works, and it's never clear exactly how far you should go with it. He routinely shoots down features that would require data flow analysis.

> > be necessary to make that happen, because it would require significant changes to how D works.
>
> Maybe, although I wonder how much you could do as library code. So I don't know.
>
> If metaprogramming is going to be the main focus of D then it makes sense to focus on allowing library authors to inform the type system of what their library types are capable of. Basically the opposite of a traditional type system where the typesystem checks constraints.
>
> So, the basic idea would be that the library tells the type system "my code provide these guarantees", rather than the the library asking the type system to verify that a set of guarantees hold.

I have no clue how feasible that would be. I'm also not sure what benefit there would be in telling the compiler that a piece of code guarantees something that the compiler doesn't already check for. The closest to that that we currently have is @trusted, which allows code that the compiler couldn't guarantee was @safe to be treated as @safe on the basis that the programmer vetted it. But if @safe wasn't something that the compiler checked already, then indicating to the compiler that a piece of code was vetted by the programmer for memory safety doesn't seem very useful.

It has been proposed in the past that folks wanted to use UDAs to indicate stuff that was somehow then checked by the compiler, but I don't know exactly what the expectation there was. Certainly, there are bound to be things that we could do along those lines, but a concrete proposal would be necessary.

> > casting. _Maybe_, there will be a few, small places like with what Manu is suggesting here where we will be able to leverage what the compiler knows to implicitly convert something, but in general, that's really not going to work.
>
> I think that make such special cases is a bit dangerous. It is really going down the path of C++ that leads to a situation where you only appeal to the same audience as C++.
>
> It also leaves the field open to Rust. I agree that borrowing might be tedious for memory safety, but might be generally suitable for making safer multi-threading available to more programmers. At least on the surface level, but I haven't spent a lot of time thinking about the limitations of borrow-checking in relation to multi-threading. So, it is just a hunch.
>
> I think it would be a mistake to outright dismiss tracking references.  There might be libraries available that can do it for you even, e.g. provide a compiler switch that enables the compiler to generate a graph that is passed onto an external library that does the checking.

Honestly, given how D's type system works, I don't think that it's really possible to do anything substantially different from C++ as far the built-in concurrency stuff goes. That being said, that doesn't mean that libraries can't be built on top of that which make things nicer. How far we can go with that, I don't know, but I think that it's pretty clear at this point that shared itself is going to be pretty low level in terms of what it does. It's mostly about segregating the data that's shared across threads and preventing operations that are clearly not thread-safe - and that in of itself is already a considerable improvement over C++. Regardless, before we can be sure of nicer features that we might be able to build on top of shared, we first need to get the low level aspects sorted out.

> > Even getting something like DIP 1000 has been a major ordeal, and I don't think that it would have ever happened unless Walter had been convinced that it was absolutely needed for @safe code to be able to do stuff like reference counting.
>
> DIP1000 looks too limiting, but I don't know what it has enabled and where it falls short in practice.
>
> I guess strength and weaknesses will show up as people build new frameworks around it.

Yeah. We'll have to see. It's clearly of some use with pointers in particular, but from what I've seen thus far, I suspect that it's simply too restrictive to be very useful in general. We'll just have to wait and see though.

- Jonathan M Davis

On Wednesday, 19 June 2019 at 10:50:45 UTC, Jonathan M Davis wrote:
> I don't follow. What do you mean by obtain write access? If you're talking about shared, scope doesn't care.

Yes, my formulation wasn't clear.  What I meant was "what would be the best way to write an OO-database interface where you can request read/write access to an object or a set of objects?".

From what has been said in this thread I imagine that the database would hand out a "scope"-qualified reference of some sort that prevents the lock from lingering after the scope has been exited.

Is "shared" the basic mechanism that one ought to use to write robust thread safe APIs in D?

So, maybe you would get something like this when requesting objects from an in-memory database:

- Obtaining a write-locked-object: returned as a "scoped unshared" reference-type.

- Obtaining a read-locked-object: returned as a "scoped immutable" reference-type.


I am just trying to figure out what the provided mechanisms could be useful for, and what the missing bits might be.  What does it enable?  What does it not enable?  In terms of API design, static type checking and "dynamic reference counting of locks".

How does it affect the ability to call into libraries that was never written with multi-threading in mind?  What can go wrong if you do so? Etc.


> Actually, using scope when you don't need to would be a serious problem. For instance, scope does _not_ interact well with range-based code. The typical thing for a lot of range-based functions to do is to take the argument, wrap it in a struct that's a new range, and return it, whereas if the parameter were scope, then you couldn't put the argument or any reference to it in the struct.

Ah. But this could be resolved if "scope" kept count of the block level you are at?

You would just need an implicit scope-level count. So the level where a reference were turned into "scoped" would be level 0, whenever it is transferred to a new scope it would have +1 added to its level.


> Walter is against data flow analysis in most cases largely because when language semantics depend on it, you have to specify exactly how the data flow analysis works, and it's never clear exactly how far you should go with it. He routinely shoots down features that would require data flow analysis.

I think it might be possible to reuse existing code though, if you just generate a suitable graf from the internal representation in the compiler.

There is a generic theoretical conception for data flow analysis called Monotone Frameworks.

https://www.cs.cmu.edu/~aplatzer/course/Compilers/27-monframework.pdf


> I have no clue how feasible that would be. I'm also not sure what benefit there would be in telling the compiler that a piece of code guarantees something that the compiler doesn't already check for.

The code gen has to know where the sequencing points are, but maybe it could do better with more information. I guess sequencing can be fixed by casting, but there might be other things you want to verify. E.g.: does references escape, are functions idempotent…


I believe the generic idea is the right one for a language like D:

There are many concurrency paradigms, it will be difficult to pick one over another since hardware keeps changing and use contexts are different.

So if you can move concurrency to libraries, and also provide crude verification of correctness (or at least reduce the number of common mistakes), then people can write different concurrency frameworks and experiment.

One problem with concurrent languages like Pony and Chapel, that assume a particular paradigm, is that they make assumptions about the hardware architecture.

It is not at all obvious that the unified-memory-with-manycores-architecture will win out over the next decade.  Event today that seems to be primarily a server architecture.

The current client/embedded architecture appears to be a small handfull of cores with many specialised hardware units with library support.

So I spot a diverging trend there. But it is hard to predict where the market will head in the next decade. Will the architecture be unified again by CPUs absorbing what co-processors do, or will it move more towards a heterogeneous distributed architecture (many independent units with local memory)?

D's focus on single threaded programming is fairly reasonable in terms of being able to adapt to many types of hardware in a way that fits how most programmers approach programming today.

So, it might be reasonable to figure out how to enable concurrency libraries that provide verification mechanisms (like escape analysis/guarantees) rather than special casing language features.


> It has been proposed in the past that folks wanted to use UDAs to indicate stuff that was somehow then checked by the compiler, but I don't know exactly what the expectation there was. Certainly, there are bound to be things that we could do along those lines, but a concrete proposal would be necessary.

Yes. So one should think about what kind of basic mechanisms one would want to be able to implement as std-library features. And what prevents that from being possible and effective/robust.


> Honestly, given how D's type system works, I don't think that it's really possible to do anything substantially different from C++ as far the built-in concurrency stuff goes.

Doing better than C++ is important though.  C++ was conceived before concurrency became pressing (C++ was designed at a time when multiple cores was prohibitively expensive), so concurrency is C++'s big weakness.  The concurrency shortcomings of C++ is also why people create languages like Chapel.  So, I'd have to agree with Manu that it is important to get this right, but it is also difficult to get right, so one should not rush it.


> considerable improvement over C++. Regardless, before we can be sure of nicer features that we might be able to build on top of shared, we first need to get the low level aspects sorted out.

One has to do both.  It is absolutely necessary to specify all the use cases that should be present in a solid concurrency standard library before deciding on the shared semantics.

D has to break out of the evolutionary type system design cycle so that the type system design does not get amended with special casing and counter-intuitive exceptions after release.


Ola.

On 19.06.19 10:47, Jonathan M Davis wrote:
> On Monday, June 17, 2019 5:46:44 PM MDT Manu via Digitalmars-d wrote:
>> Is this valid?
>>
>> int x;
>> void fun(scope ref shared(int) x) { ... }
>> fun(x); // implicit promotion to shared in this case
>>
>> This appears to promote a thread-local to shared. The problem with
>> such promotion is that it's not valid that a thread-local AND a shared
>> reference to the same thing can exist at the same time.
>>
>> With scope, we can guarantee that the reference doesn't escape the callee.
>> Since the argument is local to the calling thread, and since the calling
>> thread can not be running other code at the same time as the call is
>> executing, there is no way for any code to execute with a thread-local
>> assumption while the callee makes shared assumptions.
>>
>> I think this might be safe?
> 
> Another thing to consider here is something that's come up before when
> you've made suggestions like this - which is that unlike shared, scope is
> not transitive. So, even if what you're suggesting here would work with int,
> it would not work with types in general.
> 
> - Jonathan M Davis
> 
 Delegate contexts are opaque, so perhaps if the delegate is `pure` and doesn't return any `shared` references we can derive implicit promotion rules that work with more complex data types.