Who knows about DIP 1014? (struct move hook)
Is it well received? Is it likely to be accepted soon?

I'm working on the std::string binding, it's almost finished... but
then I hit a brick wall.
GNU's std::string implementation stores an interior pointer! >_<

No other implementation does this. It's a really bad implementation actually, quite inefficient. It could make better use of its space for small-strings if it wasn't wasting 8-bytes for an interior pointer to a small string buffer...

Anyway, I'm blocked until this DIP is accepted; so, is it looking promising?

On Sunday, 30 September 2018 at 04:34:20 UTC, Manu wrote:
> Who knows about DIP 1014? (struct move hook)
> Is it well received? Is it likely to be accepted soon?
>
> I'm working on the std::string binding, it's almost finished... but
> then I hit a brick wall.
> GNU's std::string implementation stores an interior pointer! >_<
>
> No other implementation does this. It's a really bad implementation actually, quite inefficient. It could make better use of its space for small-strings if it wasn't wasting 8-bytes for an interior pointer to a small string buffer...
>
> Anyway, I'm blocked until this DIP is accepted; so, is it looking promising?

It's pending a decision from Walter & Andrei, which I hope to hear soon.

On 9/29/2018 9:34 PM, Manu wrote:
> GNU's std::string implementation stores an interior pointer! >_<
> 
> No other implementation does this. It's a really bad implementation
> actually, quite inefficient. It could make better use of its space for
> small-strings if it wasn't wasting 8-bytes for an interior pointer to
> a small string buffer...

Could you post a synopsis of the layout of std::string?

On Sat, Sep 29, 2018 at 11:50 PM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 9/29/2018 9:34 PM, Manu wrote:
> > GNU's std::string implementation stores an interior pointer! >_<
> >
> > No other implementation does this. It's a really bad implementation actually, quite inefficient. It could make better use of its space for small-strings if it wasn't wasting 8-bytes for an interior pointer to a small string buffer...
>
> Could you post a synopsis of the layout of std::string?

The code's all in the PR if you wanna dig into it.

The synopsis is:
struct string
{
  char* ptr;
  size_t len;
  union
  {
    char[16] localBuffer;
    size_type allocatedCapacity;
  }

  bool isAllocated() { return ptr != &localBuffer[0]; }
  bool capacity() { return isAllocated() ? allocatedCapacity :
localBuffer.length; }

  this(DefaultCtor) { ptr = &localBuffer[0]; }  // <- and here it is.
interior pointer that breaks move semantics
}

Other implementations make much better use of that built-in space by not wasting 8 bytes on an interior pointer for small-strings.

On 30/09/18 10:26, Manu wrote:
> 
> Other implementations make much better use of that built-in space by
> not wasting 8 bytes on an interior pointer for small-strings.
> 

I will point out that a pointer that *sometimes* points to an internal member was one of the use cases I documented when I submitted the DIP.

Starting a long discussion about the merits of the design is a bit off-topic. I will point out that branch prediction considerations *might* make this a wise choice, despite the loss of 8 bytes of potential storage.

Either way, this is a design that is highly sensitive to precise use pattern, which, admitably, GNU's std::string probably can't know.

Shachar

On Sunday, September 30, 2018 1:35:28 AM MDT Shachar Shemesh via Digitalmars-d wrote:
> On 30/09/18 10:26, Manu wrote:
> > Other implementations make much better use of that built-in space by not wasting 8 bytes on an interior pointer for small-strings.
>
> I will point out that a pointer that *sometimes* points to an internal member was one of the use cases I documented when I submitted the DIP.
>
> Starting a long discussion about the merits of the design is a bit off-topic. I will point out that branch prediction considerations *might* make this a wise choice, despite the loss of 8 bytes of potential storage.
>
> Either way, this is a design that is highly sensitive to precise use pattern, which, admitably, GNU's std::string probably can't know.

I think that the key thing here is that if GNU's std::string is using a design like this, it's that much more critical that we have a way to hook into moves to do stuff like adjust pointers. It's one more example of a real world use case where the DIP (or something like it) is needed, or there are things that we simply won't be able to do in D - and given the push to interface with C++, it's that much more important. And while a discussion could certainly be had as to whether GNU's design decision was a good one or not, it's secondary to what really matters here, which is what the state of the DIP is how we're going to deal with interfacing with this C++ code. We need to worry about how to interface with it whether it's the best design ever or whether it's the worst design ever - and we have to take into account the fact the its design could actually change in future versions if they decide that a different way is better (e.g. GNU could change to match other implementations, or other implementations could change to match GNU, depending on what actually turned out to be best in practice when all of the various factors were taken into account - including developers making future decisions that aren't necessarily good ones; we have to interface with the code whether it's good or bad).

All in all though, if anything, I have to think that this issue increases the chances of the DIP being accepted given the importance that Walter and Andrei have been placing on interfacing with C++. And having it come up while they're in the middle of discussing it probably doesn't hurt - though maybe they were already going to accept it. I don't know. Personally, while I tend to think that it's generally better to avoid designs where opPostMove is necessary if possible, I think that the case was well made that we need a solution like it in certain cases, and if we want to interface with C++, which can do more or less arbitrary stuff in its move constructors, I don't see how we can avoid having an analogue unless we want to give up on interfacing with that code without an extra compatibility layer.

- Jonathan M Davis

On 9/29/2018 9:34 PM, Manu wrote:
> Who knows about DIP 1014? (struct move hook)

When discussing DIP 1014, a link is helpful:

https://github.com/dlang/DIPs/blob/38cec74a7471735559e3b8a7553f55102d289d28/DIPs/DIP1014.md

On 9/29/2018 9:34 PM, Manu wrote:
> Who knows about DIP 1014? (struct move hook)
> Is it well received? Is it likely to be accepted soon?
> 
> I'm working on the std::string binding, it's almost finished... but
> then I hit a brick wall.
> GNU's std::string implementation stores an interior pointer! >_<


The rationale behind D allowing structs to be moveable is to enable a copying garbage collector.

Some solutions to this problem:

1. Don't allow moving of C++ structs
2. Add a struct attribute that means "not moveable"
3. DIP 1014, which is add a __move_post_blit() function (most complex solution)
4. Use copy/destruct for C++ structs that have copy constructors (this is the old C++ solution, and is less efficient than the move constructor)

A discussion of the rationale for the C++ move constructor is:

https://akrzemi1.wordpress.com/2011/08/11/move-constructor/


> Anyway, I'm blocked until this DIP is accepted; so, is it looking promising?

Pragmatically, I suggest for the moment just ignore the problem, file a bug report for std::string, and move on.

On 10/2/2018 2:17 AM, Walter Bright wrote:
> 1. Don't allow moving of C++ structs
> 2. Add a struct attribute that means "not moveable"
> 3. DIP 1014, which is add a __move_post_blit() function (most complex solution)
> 4. Use copy/destruct for C++ structs that have copy constructors (this is the old C++ solution, and is less efficient than the move constructor)

The postblit solution can also work today,

https://issues.dlang.org/show_bug.cgi?id=17448#c37

as the DMD compiler doesn't actually move structs. So you're OK for the time being until DMD does, or a copying garbage collector is implemented.

On Tue, Oct 2, 2018 at 2:20 AM Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 9/29/2018 9:34 PM, Manu wrote:
> > Who knows about DIP 1014? (struct move hook)
> > Is it well received? Is it likely to be accepted soon?
> >
> > I'm working on the std::string binding, it's almost finished... but
> > then I hit a brick wall.
> > GNU's std::string implementation stores an interior pointer! >_<
>
>
> The rationale behind D allowing structs to be moveable is to enable a copying garbage collector.

Do we do that? What's the use of that?
Does the opPostMove() concept not work here? If the GC wanted to move
something, it can call it like any other code?

> Some solutions to this problem:
>
> 1. Don't allow moving of C++ structs

std::string, std::vector, etc are quite impotent without move semantics. I'm pretty sure the first time I ever started using those containers was around 2015 when we became secure we could use C++11 in our code (supported by all compiler vendors). Before that, they were banned and we had alternative solutions.

> 2. Add a struct attribute that means "not moveable"

They must be movable though.

> 3. DIP 1014, which is add a __move_post_blit() function (most complex solution)

It's alleged you're working through the DIP... what's the story there? Are you unhappy with it?

> 4. Use copy/destruct for C++ structs that have copy constructors (this is the old C++ solution, and is less efficient than the move constructor)

If that's where we land on this, I'll struggle to find value in my work. Containers become as impotent as back before 2011 :/

> Pragmatically, I suggest for the moment just ignore the problem, file a bug report for std::string, and move on.

But dangling pointer is an instant crash/memory corruption... it's a pretty bad 'bug'.