A recent bug report reminded me of how horrible D is at printing custom types.

Consider a container type that contains 1000 elements, such as a linked list.  If you print this type, you would expect to get a printout similar to an array, i.e.:

[ 1 2 3 4 5 ... 1000 ]

If you do this:

writeln(mylist);

then what happens is, writeln calls mylist.toString(), and prints that string.

But inside mylist.toString, it likely does things like elem[0].toString() and concatenates all these together.  This results in at least 1000 + 1 heap allocations, to go along with 1000 appends,  to create a string that will be sent to an output stream and *discarded*.

So the seemingly innocuous line writeln(mylist) is like attaching a boat anchor to your code performance.

There is a better way, as demonstrated by BigInt (whose author refuses to implement toString()):

void toString(scope void delegate(scope const(char)[] data), string format = null)

What does this do?  Well, now, writeln can define a delegate that takes a string and sends it to an output stream.  Now, there is no copying of data, no heap allocations, and no need to concatenate anything together!  Not only that, but it can be given an optional format specifier to control output when writefln is used.  Let's see how a linked list would implement this function (ignoring format for now):

void toString(scope void delegate(scope const(char)[] data) sink, string format = null)
{
   sink("[");
   foreach(elem; this)
   {
      sink(" ");
      elem.toString(sink);
   }
   sink(" ]");
}

It looks just about as simple as the equivalent function that would currently be necessary, except you have *no* heap allocations, there is a possibility for formatting, and D will be that much better performing.  Note that using a delegate allows much more natural code which requires recursion.

Should we create a DIP for this?  I'll volunteer to spearhead the effort if people are on board.

-Steve

"Steven Schveighoffer" <schveiguy@yahoo.com> wrote in message news:op.vmdglth8eav7ka@localhost.localdomain...
>A recent bug report reminded me of how horrible D is at printing custom types.
>
> Consider a container type that contains 1000 elements, such as a linked list.  If you print this type, you would expect to get a printout similar to an array, i.e.:
>
> [ 1 2 3 4 5 ... 1000 ]
>
> If you do this:
>
> writeln(mylist);
>
> then what happens is, writeln calls mylist.toString(), and prints that string.
>
> But inside mylist.toString, it likely does things like elem[0].toString() and concatenates all these together.  This results in at least 1000 + 1 heap allocations, to go along with 1000 appends,  to create a string that will be sent to an output stream and *discarded*.
>
> So the seemingly innocuous line writeln(mylist) is like attaching a boat anchor to your code performance.
>
> There is a better way, as demonstrated by BigInt (whose author refuses to implement toString()):
>
> void toString(scope void delegate(scope const(char)[] data), string format = null)
>
> What does this do?  Well, now, writeln can define a delegate that takes a string and sends it to an output stream.  Now, there is no copying of data, no heap allocations, and no need to concatenate anything together! Not only that, but it can be given an optional format specifier to control output when writefln is used.  Let's see how a linked list would implement this function (ignoring format for now):
>
> void toString(scope void delegate(scope const(char)[] data) sink, string
> format = null)
> {
>    sink("[");
>    foreach(elem; this)
>    {
>       sink(" ");
>       elem.toString(sink);
>    }
>    sink(" ]");
> }
>
> It looks just about as simple as the equivalent function that would currently be necessary, except you have *no* heap allocations, there is a possibility for formatting, and D will be that much better performing. Note that using a delegate allows much more natural code which requires recursion.
>
> Should we create a DIP for this?  I'll volunteer to spearhead the effort if people are on board.
>

I like it, *provided that* there's a quick-and-easy way to just get a string when that's all you want. At the very least there should be a standard sink function that's a default argument to toString that just simply builds a string. What we definitely *don't* want is for the user to ever have to write their own sink delegate just to get a string (which I've had to do with Tango on occasion).

On Thu, 18 Nov 2010 10:44:00 -0500, Nick Sabalausky <a@a.a> wrote:

> I like it, *provided that* there's a quick-and-easy way to just get a string
> when that's all you want. At the very least there should be a standard sink
> function that's a default argument to toString that just simply builds a
> string. What we definitely *don't* want is for the user to ever have to
> write their own sink delegate just to get a string (which I've had to do
> with Tango on occasion).

to!string(x);

(which will probably do the delegate/etc when x.toString is defined)

-Steve

On Thu, 18 Nov 2010 10:14:07 -0500, Steven Schveighoffer <schveiguy@yahoo.com> wrote:

>
> There is a better way, as demonstrated by BigInt (whose author refuses to implement toString()):
>
> void toString(scope void delegate(scope const(char)[] data), string format = null)

I should also note, I'm not fond of calling this 'toString', but that's how BigInt does it.  It seems inaccurate, when the delegate could do just about anything with the data, not just building strings.

commence bikeshedding :)

-Steve

On 18-nov-10, at 16:14, Steven Schveighoffer wrote:

> A recent bug report reminded me of how horrible D is at printing custom types.
>
> Consider a container type that contains 1000 elements, such as a linked list.  If you print this type, you would expect to get a printout similar to an array, i.e.:
>
> [ 1 2 3 4 5 ... 1000 ]
>
> If you do this:
>
> writeln(mylist);
>
> then what happens is, writeln calls mylist.toString(), and prints that string.
>
> But inside mylist.toString, it likely does things like elem[0].toString() and concatenates all these together.  This results in at least 1000 + 1 heap allocations, to go along with 1000 appends,  to create a string that will be sent to an output stream and *discarded*.
>
> So the seemingly innocuous line writeln(mylist) is like attaching a boat anchor to your code performance.
>
> There is a better way, as demonstrated by BigInt (whose author refuses to implement toString()):
>
> void toString(scope void delegate(scope const(char)[] data), string format = null)
>
> What does this do?  Well, now, writeln can define a delegate that takes a string and sends it to an output stream.  Now, there is no copying of data, no heap allocations, and no need to concatenate anything together!  Not only that, but it can be given an optional format specifier to control output when writefln is used.  Let's see how a linked list would implement this function (ignoring format for now):
>
> void toString(scope void delegate(scope const(char)[] data) sink, string format = null)
> {
>   sink("[");
>   foreach(elem; this)
>   {
>      sink(" ");
>      elem.toString(sink);
>   }
>   sink(" ]");
> }
>
> It looks just about as simple as the equivalent function that would currently be necessary, except you have *no* heap allocations, there is a possibility for formatting, and D will be that much better performing.  Note that using a delegate allows much more natural code which requires recursion.
>
> Should we create a DIP for this?  I'll volunteer to spearhead the effort if people are on board.

I agree wholeheartedly with this, I have always pushed in this direction every time the subject came up.
In tango for example exception uses this, also because I did not want memory allocations printing the stacktrace.

This is the way used in blip to output everything, I always felt bad in allocating things on the heap.

- in object I look for a void desc(void delegate(const(char)[] data) sink) method (well D1, so scope is implied ;)
  optionally with extra format arguments that don't have to be restricted to a simple string.

- i have implemented a writeOut templatized function to easily dump out all kinds of objects to sinks or similar objects
  with it you write writeOut(sink,object,possiblyExtraArgs); // see in blip.io.BasicIO

- I have defined a dumper object (just a struct) and a helper function for easy call chaining, so you can do
  dumper(sink)("bla:")(myObject)("\n");

- blip.container.GrowableArray completes the offer by giving an easy way to collect the results, and has two helper functions:

/// collects what is appended by the appender in a single array and returns it
/// it buf is provided the appender tries to use it (but allocates if extra space is needed)
T[] collectAppender(T)(void delegate(void delegate(T[])) appender,char[] buf=null){}

/// collects what is appended by the appender and adds it at once to the given sink
void sinkTogether(U,T)(U sink,void delegate(void delegate(T[])) appender,char[] buf=null){}

I find that such an approach works well, is not too intrusive, and is efficient.

Fawzi

If you take a look at blip.

On 18-nov-10, at 17:01, Fawzi Mohamed wrote:

> [...]
> If you take a look at blip.

ehm that was a leftover form my editing that I did not see because it was outside the visible area in my mail program... just ignore it.
well you *can* look at blip, but well you get the point...

On 18-nov-10, at 16:53, Steven Schveighoffer wrote:

> On Thu, 18 Nov 2010 10:44:00 -0500, Nick Sabalausky <a@a.a> wrote:
>
>> I like it, *provided that* there's a quick-and-easy way to just get a string
>> when that's all you want. At the very least there should be a standard sink
>> function that's a default argument to toString that just simply builds a
>> string. What we definitely *don't* want is for the user to ever have to
>> write their own sink delegate just to get a string (which I've had to do
>> with Tango on occasion).
>
> to!string(x);
>
> (which will probably do the delegate/etc when x.toString is defined)

I don't know I considered using the to!(T) conversion, but decided against it in blip, because I preferred having to for exact conversion, and use another set of methods for string conversion (that is special enough, and sometime used just for debugging, and not invertible).

by the way another nice effect of using a simple sink delegate is that you can easily redeclare it at low level and get rid of dependencies (well maybe you suffer a bit converting basic types, but it is doable)
Whereas using higher level streams is difficult in the runtime (you easily have object depending on them, forcing you to put them in object.d)

Fawzi

On Thu, 18 Nov 2010 10:14:07 -0500, Steven Schveighoffer wrote:

> A recent bug report reminded me of how horrible D is at printing custom types.
> 
> Consider a container type that contains 1000 elements, such as a linked list.  If you print this type, you would expect to get a printout similar to an array, i.e.:
> 
> [ 1 2 3 4 5 ... 1000 ]
> 
> If you do this:
> 
> writeln(mylist);
> 
> then what happens is, writeln calls mylist.toString(), and prints that
> string.
> 
> But inside mylist.toString, it likely does things like elem[0].toString() and concatenates all these together.  This results in at least 1000 + 1 heap allocations, to go along with 1000 appends,  to create a string that will be sent to an output stream and *discarded*.
> 
> So the seemingly innocuous line writeln(mylist) is like attaching a boat anchor to your code performance.
> 
> There is a better way, as demonstrated by BigInt (whose author refuses
> to implement toString()):
> 
> void toString(scope void delegate(scope const(char)[] data), string
> format = null)
> 
> What does this do?  Well, now, writeln can define a delegate that takes a string and sends it to an output stream.  Now, there is no copying of data, no heap allocations, and no need to concatenate anything together! Not only that, but it can be given an optional format specifier to control output when writefln is used.  Let's see how a linked list would implement this function (ignoring format for now):
> 
> void toString(scope void delegate(scope const(char)[] data) sink, string
> format = null)
> {
>     sink("[");
>     foreach(elem; this)
>     {
>        sink(" ");
>        elem.toString(sink);
>     }
>     sink(" ]");
> }
> 
> It looks just about as simple as the equivalent function that would currently be necessary, except you have *no* heap allocations, there is a possibility for formatting, and D will be that much better performing. Note that using a delegate allows much more natural code which requires recursion.
> 
> Should we create a DIP for this?  I'll volunteer to spearhead the effort if people are on board.

First of all, I think Andrei has already implemented this in the write*() functions.  I use this toString() style also for std.complex.Complex, and I can print complex numbers no problem.

That said, I also think toString is a bad name for this.  Especially considering it will be used as an imperative, i.e.

  obj.toString(sink);

instead of

  s = obj.toString();

I don't really have a good suggestion for an alternative name, though. Perhaps 'output'?

I would personally prefer a range-based solution:

  void output(R)(R sink, string fmt = null)
      if (isOutputRange!R)
  { ... }

Since not too long ago, a delegate taking T is considered an output range of T.  This will allow you to using both a sink delegate and a more conventional output range.

I do, however, realise that templates aren't everyone's cup of tea, and that there are situations where they can't be used, so perhaps the delegate solution is best after all.

-Lars

On 11/18/10 7:14 AM, Steven Schveighoffer wrote:
> A recent bug report reminded me of how horrible D is at printing custom
> types.
>
> Consider a container type that contains 1000 elements, such as a linked
> list. If you print this type, you would expect to get a printout similar
> to an array, i.e.:
>
> [ 1 2 3 4 5 ... 1000 ]
>
> If you do this:
>
> writeln(mylist);
>
> then what happens is, writeln calls mylist.toString(), and prints that
> string.
>
> But inside mylist.toString, it likely does things like
> elem[0].toString() and concatenates all these together. This results in
> at least 1000 + 1 heap allocations, to go along with 1000 appends, to
> create a string that will be sent to an output stream and *discarded*.
>
> So the seemingly innocuous line writeln(mylist) is like attaching a boat
> anchor to your code performance.
>
> There is a better way, as demonstrated by BigInt (whose author refuses
> to implement toString()):
>
> void toString(scope void delegate(scope const(char)[] data), string
> format = null)
>
> What does this do? Well, now, writeln can define a delegate that takes a
> string and sends it to an output stream. Now, there is no copying of
> data, no heap allocations, and no need to concatenate anything together!
> Not only that, but it can be given an optional format specifier to
> control output when writefln is used. Let's see how a linked list would
> implement this function (ignoring format for now):
>
> void toString(scope void delegate(scope const(char)[] data) sink, string
> format = null)
> {
> sink("[");
> foreach(elem; this)
> {
> sink(" ");
> elem.toString(sink);
> }
> sink(" ]");
> }
>
> It looks just about as simple as the equivalent function that would
> currently be necessary, except you have *no* heap allocations, there is
> a possibility for formatting, and D will be that much better performing.
> Note that using a delegate allows much more natural code which requires
> recursion.
>
> Should we create a DIP for this? I'll volunteer to spearhead the effort
> if people are on board.
>
> -Steve

Yes please.

Andrei

On Thu, 18 Nov 2010 10:53:20 -0500, Steven Schveighoffer wrote:

> On Thu, 18 Nov 2010 10:44:00 -0500, Nick Sabalausky <a@a.a> wrote:
> 
>> I like it, *provided that* there's a quick-and-easy way to just get a
>> string
>> when that's all you want. At the very least there should be a standard
>> sink
>> function that's a default argument to toString that just simply builds
>> a string. What we definitely *don't* want is for the user to ever have
>> to write their own sink delegate just to get a string (which I've had
>> to do with Tango on occasion).
> 
> to!string(x);
> 
> (which will probably do the delegate/etc when x.toString is defined)

On the Phobos list, Andrei suggested a unified toString signature, which I tried out for std.complex.Complex:

  // If sink is provided, output to it and return null,
  // otherwise return string.
  string toString(void delegate(const(char)[] ) sink = null,
      string fmt = null) { ... }

But I don't know, one might as well just define a separate toString().  I prefer the to!string() solution you suggest.

-Lars