Sorry if this is a stupid question, but is there something special required to call Appender.clear?  When I attempt even just a simple use I am getting compile errors relating to `template object.clear`.

When I try:

    import std.array;

    void main(string[] args){
      auto foo = appender!string;
      foo.clear;
    }

I receive the following output:

    $ ldc2 source/tmp.d
    source/tmp.d(5): Error: template object.clear cannot deduce function from argument types !()(Appender!string), candidates are:
    /usr/local/include/d/ldc/object.d(2041):        object.clear(T : Value[Key], Value, Key)(T aa)
    /usr/local/include/d/ldc/object.d(2046):        object.clear(T : Value[Key], Value, Key)(T* aa)

    $ dmd source/tmp.d
    source/tmp.d(5): Error: template object.clear cannot deduce function from argument types !()(Appender!string), candidates are:
    /usr/include/dmd/druntime/import/object.d(1983):        object.clear(T : Value[Key], Value, Key)(T aa)
    /usr/include/dmd/druntime/import/object.d(1988):        object.clear(T : Value[Key], Value, Key)(T* aa)


versions:

    $ dmd --version
    DMD64 D Compiler v2.078.0
    Copyright (c) 1999-2017 by The D Language Foundation written by Walter Bright

    $ ldc2 --version
    LDC - the LLVM D compiler (1.8.0git-921bb7f):
      based on DMD v2.078.3 and LLVM 4.0.1
      built with DMD64 D Compiler v2.078.0


Is there something I am missing here?

Thanks

On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
> Sorry if this is a stupid question, but is there something special required to call Appender.clear?  When I attempt even just a simple use I am getting compile errors relating to `template object.clear`.
>
> When I try:
>
>     import std.array;
>
>     void main(string[] args){
>       auto foo = appender!string;
>       foo.clear;
>     }
>
> I receive the following output:
>
>     $ ldc2 source/tmp.d
>     source/tmp.d(5): Error: template object.clear cannot deduce function from argument types !()(Appender!string), candidates are:
>     /usr/local/include/d/ldc/object.d(2041):        object.clear(T : Value[Key], Value, Key)(T aa)
>     /usr/local/include/d/ldc/object.d(2046):        object.clear(T : Value[Key], Value, Key)(T* aa)
>
>     $ dmd source/tmp.d
>     source/tmp.d(5): Error: template object.clear cannot deduce function from argument types !()(Appender!string), candidates are:
>     /usr/include/dmd/druntime/import/object.d(1983):        object.clear(T : Value[Key], Value, Key)(T aa)
>     /usr/include/dmd/druntime/import/object.d(1988):        object.clear(T : Value[Key], Value, Key)(T* aa)
>
>
> versions:
>
>     $ dmd --version
>     DMD64 D Compiler v2.078.0
>     Copyright (c) 1999-2017 by The D Language Foundation written by Walter Bright
>
>     $ ldc2 --version
>     LDC - the LLVM D compiler (1.8.0git-921bb7f):
>       based on DMD v2.078.3 and LLVM 4.0.1
>       built with DMD64 D Compiler v2.078.0
>
>
> Is there something I am missing here?
>
> Thanks

I would say that's a bug, because it seems like it's trying to get the clear() function used for associative arrays.

On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
> Sorry if this is a stupid question, but is there something special required to call Appender.clear?  When I attempt even just a simple use I am getting compile errors relating to `template object.clear`.
>
> When I try:
>
>     import std.array;
>
>     void main(string[] args){
>       auto foo = appender!string;
>       foo.clear;
>     }

Appender only implements clear for mutable types, and string is an array of immutables. I'm not sure but it looks like you're hitting the associative array clear by UFCS instead?

Make it Appender!(char[]) and it will work.

> // only allow overwriting data on non-immutable and non-const data
> static if (isMutable!T)
> {
>     /**
>       * Clears the managed array.  This allows the elements of the array to be reused
>       * for appending.
>       *
>       * Note: clear is disabled for immutable or const element types, due to the
>       * possibility that $(D Appender) might overwrite immutable data.
>       */
>      void clear() @trusted pure nothrow
>      {
>          if (_data)
>          {
>              _data.arr = _data.arr.ptr[0 .. 0];
>          }
>      }
> [...]

https://github.com/dlang/phobos/blob/master/std/array.d#L3140

On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
> Sorry if this is a stupid question, but is there something special required to call Appender.clear?  When I attempt even just a simple use I am getting compile errors relating to `template object.clear`.

From the documentation for Appender:

> Note
> clear is disabled for immutable or const element types, due to the possibility that Appender might overwrite immutable data.

Since string is immutable(char)[], clear() is simply not available for appender!string.

--
  Simen

On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
> On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
>> Sorry if this is a stupid question, but is there something special required to call Appender.clear?  When I attempt even just a simple use I am getting compile errors relating to `template object.clear`.
>
> From the documentation for Appender:
>
>> Note
>> clear is disabled for immutable or const element types, due to the possibility that Appender might overwrite immutable data.
>
> Since string is immutable(char)[], clear() is simply not available for appender!string.
>
> --
>   Simen

Isn't this wrong, though? Appender controls the memory it references. It could just choose to allocate non-immutable memory internally. As long as any const data put into the appender is *returned* as const, there is no chance of immutable memory being overwritten.

On 01/24/2019 04:35 AM, FeepingCreature wrote:
> On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
>> On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
>>> Sorry if this is a stupid question, but is there something special
>>> required to call Appender.clear?  When I attempt even just a simple
>>> use I am getting compile errors relating to `template object.clear`.
>>
>> From the documentation for Appender:
>>
>>> Note
>>> clear is disabled for immutable or const element types, due to the
>>> possibility that Appender might overwrite immutable data.
>>
>> Since string is immutable(char)[], clear() is simply not available for
>> appender!string.
>>
>> --
>>   Simen
>
> Isn't this wrong, though? Appender controls the memory it references. It
> could just choose to allocate non-immutable memory internally. As long
> as any const data put into the appender is *returned* as const, there is
> no chance of immutable memory being overwritten.

I think Appender is trying to protect previous data from Appender's later use. If it handed out immutable data, the user is expecting it to not change. So, Appender cannot clear it for later use.

Ali

On 1/24/19 7:35 AM, FeepingCreature wrote:
> On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
>> On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
>>> Sorry if this is a stupid question, but is there something special required to call Appender.clear?  When I attempt even just a simple use I am getting compile errors relating to `template object.clear`.
>>
>> From the documentation for Appender:
>>
>>> Note
>>> clear is disabled for immutable or const element types, due to the possibility that Appender might overwrite immutable data.
>>
>> Since string is immutable(char)[], clear() is simply not available for appender!string.
>>
> 
> Isn't this wrong, though? Appender controls the memory it references. It could just choose to allocate non-immutable memory internally. As long as any const data put into the appender is *returned* as const, there is no chance of immutable memory being overwritten.

Appender is sometimes given a starting array. clear isn't callable in that case, and we don't distinguish the difference in the type or at runtime.

I did a lot of work on std.array.Appender when I redid the array runtime to fix array appending back in 2010 to make it safe. e.g.: https://github.com/schveiguy/phobos/commit/7e89201cdae96e71e550afe72b59ef3db145916f

Disabling clear from appender for non-mutable element types was one of the changes: https://github.com/schveiguy/phobos/commit/6636569318d26545bf7c36ddac029830a6d52531

This is before we were on github, so I don't know if there's any discussion of the issues, perhaps on the mailing list, but looking at my emails around that time, there are some emails that aren't in the mailing list.

A funny thing I wrote to Walter/Sean around that time, when I was just getting druntime commit access:

"I don't mind helping out in druntime, but I'm not really any kind of expert on the runtime, and I doubt I'll make many contributions besides this one without assignment."

;)

FYI, this thread is almost a year old.

-Steve

On Thursday, 24 January 2019 at 17:49:34 UTC, Ali Çehreli wrote:
> On 01/24/2019 04:35 AM, FeepingCreature wrote:
> > On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
> >> On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper
> wrote:
> >>> Sorry if this is a stupid question, but is there something
> special
> >>> required to call Appender.clear?  When I attempt even just
> a simple
> >>> use I am getting compile errors relating to `template
> object.clear`.
> >>
> >> From the documentation for Appender:
> >>
> >>> Note
> >>> clear is disabled for immutable or const element types, due
> to the
> >>> possibility that Appender might overwrite immutable data.
> >>
> >> Since string is immutable(char)[], clear() is simply not
> available for
> >> appender!string.
> >>
> >> --
> >>   Simen
> >
> > Isn't this wrong, though? Appender controls the memory it
> references. It
> > could just choose to allocate non-immutable memory
> internally. As long
> > as any const data put into the appender is *returned* as
> const, there is
> > no chance of immutable memory being overwritten.
>
> I think Appender is trying to protect previous data from Appender's later use. If it handed out immutable data, the user is expecting it to not change. So, Appender cannot clear it for later use.
>
> Ali

Aren't the semantics of .clear that it's invalid to access references to .data after calling .clear, period? And if not, then shouldn't they be? Consider if Appender managed its own memory and held on to previously-allocated arrays while you were appending, only to free them on .clear. That seems like something Appender should be allowed to do. If you don't want it, just reinitialize Appender instead of calling .clear.

> Appender is sometimes given a starting array. clear isn't callable in that case, and we don't distinguish the difference in the type or at runtime.

Any reason that the semantics of .clear should be different for a starting array? Anyway if so, I'd prefer to just make that a runtime error.

I'm mostly fishing around if anyone has an objection to a PR to change this.

On 1/25/19 3:20 AM, FeepingCreature wrote:
> On Thursday, 24 January 2019 at 17:49:34 UTC, Ali Çehreli wrote:
>> On 01/24/2019 04:35 AM, FeepingCreature wrote:
>> > On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
>> >> On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper
>> wrote:
>> >>> Sorry if this is a stupid question, but is there something
>> special
>> >>> required to call Appender.clear?  When I attempt even just
>> a simple
>> >>> use I am getting compile errors relating to `template
>> object.clear`.
>> >>
>> >> From the documentation for Appender:
>> >>
>> >>> Note
>> >>> clear is disabled for immutable or const element types, due
>> to the
>> >>> possibility that Appender might overwrite immutable data.
>> >>
>> >> Since string is immutable(char)[], clear() is simply not
>> available for
>> >> appender!string.
>> >>
>> >> --
>> >>   Simen
>> >
>> > Isn't this wrong, though? Appender controls the memory it
>> references. It
>> > could just choose to allocate non-immutable memory
>> internally. As long
>> > as any const data put into the appender is *returned* as
>> const, there is
>> > no chance of immutable memory being overwritten.
>>
>> I think Appender is trying to protect previous data from Appender's later use. If it handed out immutable data, the user is expecting it to not change. So, Appender cannot clear it for later use.
>>
>> Ali
> 
> Aren't the semantics of .clear that it's invalid to access references to .data after calling .clear, period? And if not, then shouldn't they be? Consider if Appender managed its own memory and held on to previously-allocated arrays while you were appending, only to free them on .clear. That seems like something Appender should be allowed to do. If you don't want it, just reinitialize Appender instead of calling .clear.

You are advised not to. But it's not unsafe, as the memory is still there.

> 
>> Appender is sometimes given a starting array. clear isn't callable in that case, and we don't distinguish the difference in the type or at runtime.
> 
> Any reason that the semantics of .clear should be different for a starting array? Anyway if so, I'd prefer to just make that a runtime error.

Yes, because Appender doesn't know where that memory comes from. A string could be, for instance, passed to another thread and being used, you wouldn't want to overwrite it.

Generally speaking, overwriting immutable data is UB in D anyway.

Making it a runtime error would be possible, but there has to be a good reason to make it that way.

> 
> I'm mostly fishing around if anyone has an objection to a PR to change this.

Without good reasons to change, I don't see why it would be accepted.

Maybe you can describe your use case?

-Steve

On Friday, 25 January 2019 at 14:33:16 UTC, Steven Schveighoffer wrote:
> On 1/25/19 3:20 AM, FeepingCreature wrote:
>> On Thursday, 24 January 2019 at 17:49:34 UTC, Ali Çehreli wrote:
>> Aren't the semantics of .clear that it's invalid to access references to .data after calling .clear, period? And if not, then shouldn't they be? Consider if Appender managed its own memory and held on to previously-allocated arrays while you were appending, only to free them on .clear. That seems like something Appender should be allowed to do. If you don't want it, just reinitialize Appender instead of calling .clear.
>
> You are advised not to. But it's not unsafe, as the memory is still there.
>

That's stupid. Why would you advise me not to, if there's no problem with it? Either it should be accepted or it should be forbidden, just like warnings.

>> Any reason that the semantics of .clear should be different for a starting array? Anyway if so, I'd prefer to just make that a runtime error.
>
> Yes, because Appender doesn't know where that memory comes from. A string could be, for instance, passed to another thread and being used, you wouldn't want to overwrite it.
>

In which case, you can't reuse the Appender and should not call `clear`! It seems like we're gimping functionality of a very basic tool for memory efficiency for the sake of not confusing users. This rarely pays off.

> Generally speaking, overwriting immutable data is UB in D anyway.
>
> Making it a runtime error would be possible, but there has to be a good reason to make it that way.
>
>> 
>> I'm mostly fishing around if anyone has an objection to a PR to change this.
>
> Without good reasons to change, I don't see why it would be accepted.
>
> Maybe you can describe your use case?
>

My use case is simply the standard usecase of Appender: I want to build up an array in a way that reduces GC churn. Maybe it's an array of structs that contain const members that I'll serialize to JSON and send on a socket. In that case, I know for a fact that no references will hang around past the serialization. That's what clear _is for._ I don't see why this would be different with const or immutable data; if you hold references past .clear being called you're in trouble *anyways.*

I consider initializing Appender with an array referencing immutable data a borderline error anyways. The entire point of Appender is that it caches capacity data of GC managed memory, which is never immutable. On the first append to an immutable-memory array, it has to reallocate *anyways*. There is no benefit to initializing an Appender with immutable memory over just appending it first thing, unless you never plan to append to it ever.