February 22, 2012
On Wed, 22 Feb 2012 11:33:57 -0600, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> On Wednesday, February 22, 2012 08:19:38 Robert Jacques wrote:
>> To Variant? Yes, definitely. To Appender? I don't think so. There is an
>> slight change in API behavior necessitated by performance considerations,
>> but I don't think it warrants a review by the community at large.
>> Specifically, appender's internal data structure is now sealed, which means
>> that .data must always make a copy. My preference would be to deprecate
>> .data in favor of a .dup/.idup pair. It'll break a bunch of code (which I
>> don't like), but it will make sure no one is calling .data twice in a row,
>> resulting in a silent performance problem.
>
> I've definitely written code that needed to get at data while appending. I
> would consider it to be a huge downside to not be able to efficiently get at the
> current state of the array within the appender. Why on earth would you seal
> it?
>
> - Jonathan M Davis

There's a big difference between sealed and not accessible. .data's API requires exposing an array, and there's no way to do this without leaking memory like a sieve in one way or another. However, if all you need is to iterate the contents, that's easy to do. I'm currently adding backwards iteration. Even indexing is fairly efficient (not yet implemented), particularly relative indexing (i.e. n from back or front).

I haven't seen too many use cases yet where accessing the underlying array is important, nor has it come up on bugzilla. I've found one case in Phobos where appender was used as a stack. What's your example? What features does it have to support and how efficient does it have to be?
February 22, 2012
On Wednesday, February 22, 2012 12:16:43 Robert Jacques wrote:
> There's a big difference between sealed and not accessible. .data's API requires exposing an array, and there's no way to do this without leaking memory like a sieve in one way or another. However, if all you need is to iterate the contents, that's easy to do. I'm currently adding backwards iteration. Even indexing is fairly efficient (not yet implemented), particularly relative indexing (i.e. n from back or front).
> 
> I haven't seen too many use cases yet where accessing the underlying array is important, nor has it come up on bugzilla. I've found one case in Phobos where appender was used as a stack. What's your example? What features does it have to support and how efficient does it have to be?

It's can be useful to just get at the underlying array and pass it to functions which are going to use it but not alter it (or at least not append to it). Iterating over it doesn't give you an array. And since appender's entire purpose is simply to make appending to an array more efficient, making it impossible to treat it as one until you're done appending is overly restrictive IMHO. Yes, if you leak references to the underlying data, you're asking for trouble, but that doesn't mean that it can't be used without leaking memory.

Unfortunately, I don't have any code snippets with me at the moment, so I can't give any concrete examples of usage, but any situation where you want to be able to operate on the array while building it needs the ability to get at the underlying array. Yes, in most cases, you're probably simply appending to the array, but at least once in a while, you need to operate on an array while building it.

- Jonathan M Davis
February 22, 2012
On 2012-02-22 15:01, Andrei Alexandrescu wrote:
> On 2/22/12 1:22 AM, Jacob Carlborg wrote:
>> Now I'm completely lost. According to what I've read this is thread this
>> is exactly what you want to do, put the formatting inside the exceptions.
>
> No, just have exceptions inform an external formatter.
>
> Andrei
>

Ok, I see.

-- 
/Jacob Carlborg
February 22, 2012
On Wednesday, February 22, 2012 14:12:07 Jonathan M Davis wrote:
> On Wednesday, February 22, 2012 12:16:43 Robert Jacques wrote:
> > There's a big difference between sealed and not accessible. .data's API requires exposing an array, and there's no way to do this without leaking memory like a sieve in one way or another. However, if all you need is to iterate the contents, that's easy to do. I'm currently adding backwards iteration. Even indexing is fairly efficient (not yet implemented), particularly relative indexing (i.e. n from back or front).
> > 
> > I haven't seen too many use cases yet where accessing the underlying array is important, nor has it come up on bugzilla. I've found one case in Phobos where appender was used as a stack. What's your example? What features does it have to support and how efficient does it have to be?
> 
> It's can be useful to just get at the underlying array and pass it to functions which are going to use it but not alter it (or at least not append to it). Iterating over it doesn't give you an array. And since appender's entire purpose is simply to make appending to an array more efficient, making it impossible to treat it as one until you're done appending is overly restrictive IMHO. Yes, if you leak references to the underlying data, you're asking for trouble, but that doesn't mean that it can't be used without leaking memory.
> 
> Unfortunately, I don't have any code snippets with me at the moment, so I can't give any concrete examples of usage, but any situation where you want to be able to operate on the array while building it needs the ability to get at the underlying array. Yes, in most cases, you're probably simply appending to the array, but at least once in a while, you need to operate on an array while building it.

Also, wouldn't it be less efficient if you _had_ to copy the array once you were done with the appender? That would seem to go against what appender is trying to do.

- Jonathan M Davis
February 22, 2012
On Wed, 22 Feb 2012 13:12:07 -0600, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> On Wednesday, February 22, 2012 12:16:43 Robert Jacques wrote:
>> There's a big difference between sealed and not accessible. .data's API
>> requires exposing an array, and there's no way to do this without leaking
>> memory like a sieve in one way or another. However, if all you need is to
>> iterate the contents, that's easy to do. I'm currently adding backwards
>> iteration. Even indexing is fairly efficient (not yet implemented),
>> particularly relative indexing (i.e. n from back or front).
>>
>> I haven't seen too many use cases yet where accessing the underlying array
>> is important, nor has it come up on bugzilla. I've found one case in
>> Phobos where appender was used as a stack. What's your example? What
>> features does it have to support and how efficient does it have to be?
>
> It's can be useful to just get at the underlying array and pass it to
> functions which are going to use it but not alter it (or at least not append
> to it). Iterating over it doesn't give you an array. And since appender's
> entire purpose is simply to make appending to an array more efficient, making it
> impossible to treat it as one until you're done appending is overly
> restrictive IMHO. Yes, if you leak references to the underlying data, you're
> asking for trouble, but that doesn't mean that it can't be used without
> leaking memory.
>
> Unfortunately, I don't have any code snippets with me at the moment, so I
> can't give any concrete examples of usage, but any situation where you want to
> be able to operate on the array while building it needs the ability to get at
> the underlying array. Yes, in most cases, you're probably simply appending to
> the array, but at least once in a while, you need to operate on an array while
> building it.
>
> - Jonathan M Davis

I view appender's purpose as array building, which is slightly different from simply speeding up array appending. Simply put, an array is a terrible data structure for building arrays. But, I can appreciate the need for mutation and if a particular array building algorithm can't be performed on appender, then appender has failed. Would exposing a bidirectional array be sufficient for your usages? A random access range?
February 22, 2012
Le 22/02/2012 18:50, H. S. Teoh a écrit :
> On Wed, Feb 22, 2012 at 11:53:39AM +0100, deadalnix wrote:
> [...]
>> Additionnaly, I would mention that the transient isn't a caracteristic
>> of the Exception, but of the recovery strategy.
>
> Technically correct. Though I'm playing with the idea of making recovery
> strategies a property of an exception - since a recovery strategy is
> meaningless without an associated exception (or problem). I need to
> think this through a bit more, though, as to how to correctly implement
> this.
>

I did though about this. This isn't the right way. Recovery strategy doesn't have any meaning at the catch point, so we shouldn't make it a property of Exception.

And sometime you don't care about the Exception. If you try to connect something that is know to fail for exemple, you really don't want to know what went wrong. You just want to try again with some backoff.

I do think you made a point with the handler getting an Exception and a recovery stratgy as parameter, but it is still unclear where all this goes to me.
February 22, 2012
On Wednesday, February 22, 2012 14:24:49 Robert Jacques wrote:
> I view appender's purpose as array building, which is slightly different
>  from simply speeding up array appending. Simply put, an array is a
> terrible data structure for building arrays. But, I can appreciate the
> need for mutation and if a particular array building algorithm can't be
> performed on appender, then appender has failed. Would exposing a
> bidirectional array be sufficient for your usages? A random access range?

Well, as long as you don't have access to the actual array, you're going to lose something. There are functions that you just won't be able to use because they take an array. However, a range with essentially the same properties as a range (bidirectional, random access, etc.) would cover a large number of the functions that you might want to call. And if you need to hide the array for speed for efficiency for some reason - especially if it results in a large increase in speed - then that could at least partially outweigh the cost of losing the array (especially if Appender has an API to at least use it as a range). But I'd definitely be against hiding it simply for safety, since speed is really its whole point of existance. I don't know why else you'd ever use appender.

So, I don't really like the idea of losing access to the underlying array, but if you can provide at least make it so that you could use it with range-based functions, and the changes provides a sigificant speed improvement, then the need for speed arguably outweighs the loss of being able to use the internal array directly.

- Jonathan M Davis
February 22, 2012
On Wed, 22 Feb 2012 14:17:09 -0600, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> On Wednesday, February 22, 2012 14:12:07 Jonathan M Davis wrote:
>> On Wednesday, February 22, 2012 12:16:43 Robert Jacques wrote:
>> > There's a big difference between sealed and not accessible. .data's  
>> API
>> > requires exposing an array, and there's no way to do this without  
>> leaking
>> > memory like a sieve in one way or another. However, if all you need  
>> is to
>> > iterate the contents, that's easy to do. I'm currently adding  
>> backwards
>> > iteration. Even indexing is fairly efficient (not yet implemented),
>> > particularly relative indexing (i.e. n from back or front).
>> >
>> > I haven't seen too many use cases yet where accessing the underlying  
>> array
>> > is important, nor has it come up on bugzilla. I've found one case in
>> > Phobos where appender was used as a stack. What's your example? What
>> > features does it have to support and how efficient does it have to be?
>>
>> It's can be useful to just get at the underlying array and pass it to
>> functions which are going to use it but not alter it (or at least not append
>> to it). Iterating over it doesn't give you an array. And since appender's
>> entire purpose is simply to make appending to an array more efficient,
>> making it impossible to treat it as one until you're done appending is
>> overly restrictive IMHO. Yes, if you leak references to the underlying
>> data, you're asking for trouble, but that doesn't mean that it can't be
>> used without leaking memory.
>>
>> Unfortunately, I don't have any code snippets with me at the moment, so I
>> can't give any concrete examples of usage, but any situation where you want
>> to be able to operate on the array while building it needs the ability to
>> get at the underlying array. Yes, in most cases, you're probably simply
>> appending to the array, but at least once in a while, you need to operate
>> on an array while building it.
>
> Also, wouldn't it be less efficient if you _had_ to copy the array once you were
> done with the appender? That would seem to go against what appender is trying
> to do.
>
> - Jonathan M Davis

No, because the array doesn't actually exist until appender makes copy. Internally, appender is using a list of arrays to store the data (similar to a VList and string builders from other languages). So it's little o(2N) for both memory and speed; the current appender is much worse than that. In terms of actual performance, on a clean machine I'm substantially faster for N < 4k (thanks to a free list), about the same for things of a few k in size, and then as things get bigger the current appender tanks.
February 22, 2012
> No, because the array doesn't actually exist until appender makes copy.

Will one be able to use the sort!()() algorithm directly on your appender,
that is, without accessing/creating the underlying array?

--jm


February 22, 2012
On Wednesday, 22 February 2012 at 20:59:15 UTC, Jonathan M Davis wrote:
> speed [...] is really its whole point of existance. I don't know why else you'd ever use appender.
[...]
> - Jonathan M Davis

A use case is to give identity to a built-in array.

Consider this:

     class MyClass {
         private MyData[] theData;

         public @property MyData[] data() {
             return theData;
         }
         ...
     }


     MyClass m = new MyClass();
     m.data ~= new MyData();
     //Nothing got appended:
     assert(m.data.length == 0);

For the 95% of the use cases, that is the desired
behaviour. You don't want anyone appending to
your private array. If you wanted to, you would
have defined MyClass.append(myData).

  But there are a few cases where you want to
give identity to the array, and let anyone who
has a "handle" to it, to be able to append it.
(another case is while porting code from languages
that don't represent arrays as ranges, and return
them as getters).

--jm