On Thursday, 24 July 2014 at 23:40:56 UTC, Jakob Ovrum wrote:
> How often have you seen a formatted log message logged in a loop? I'd wager that happens quite often. Using `format` is a no-go as it completely thrashes the GC, it needs to use `formattedWrite` to write directly to the underlying buffer (such as a file).

To eloborate on this: using `format` like std.logger currently does, for short-lived strings that are consumed and discared almost immediately in a higher stack frame, is extremely inefficient as every string needs at least one heap memory allocation (though `format` can easily do *multiple* in one call). It's a good way to quickly thrash and fragment the GC heap, putting an extreme amount of stress on the collector.

Even C has fprintf. If we don't provide efficient formatted writing, people will not use our abstractions.

The solution is to use `formattedWrite` for custom allocation behaviour. When I've used `formattedWrite` for this kind of problem before, it has generally come down to the following patterns; let's assume the underlying sink is a file:

 * One solution is to use `formattedWrite` to write to the file directly. Of course, Logger doesn't provide an interface for writing partial log messages, or writing log messages in chunks, so this would require modifying the basic API. Also, `formattedWrite` doesn't guarantee any minimum chunk size, so in the worst case, it might result in one write operation per character, which is very inefficient.

* Another solution is to use `formattedWrite` to write to a stack-allocated character buffer, then subsequently pass it to `writeLogMsg`. This is a leaky abstraction because it puts an arbitrary upper limit on how long log entries can be. A practical compromise is to revert to a heap allocation for entries that don't fit in the stack buffer, but we can do better;

* The best solution is a hybrid one. Use `formattedWrite` to write to a stack-allocated buffer, then whenever it's full, write the contents of the buffer to the underlying sink (this is easily doable by passing a delegate to `formattedWrite` that sees the stack buffer as an upvalue). Yes, this does require modifying the basic logger API to support partial writes, but it gives us optimal performance.

The last solution gives us no dynamic memory allocations unless an exception is thrown, while still minimizing the number of writes to the underlying sink, which is important when writes can be expensive, such as writes to a file or a socket.

On 22/07/14 11:43, ponce wrote:

> NullLogger is there precisely because it's trivial and needed.

If it's so trivial then the users can implement that themselves. A standard library isn't about implementing what's trivial, it's about implementing what's most useful to most people.

-- 
/Jacob Carlborg

On Friday, 25 July 2014 at 07:11:06 UTC, Jacob Carlborg wrote:
> A standard library isn't about implementing what's trivial, it's about implementing what's most useful to most people.

On the other hand, not having to re-implement trivial functions that you need all the time is a form of usefulness.

Am Thu, 24 Jul 2014 11:50:54 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> 6. The current backend design requires use of classes and references, i.e. garbage collection. Amid the current tendency to make std work without requiring GC, I think a design based on RefCounted would be recommended here.

Classes could be backed by other allocators, but this requires some thought. For example a 'dispose' like method could be added. Or maybe RefCounted works with classes? But IIRC RefCounted actually uses the GC for some allocations, it only provides deterministic destruction.

I think the bigger problem is string formatting, which currently produces one garbage collected string for every log call. Has this been fixed in the mean time?

It can be fixed but it requires changes in the backend class interface. I even opened a pull request to fix it. As long as this is not fixed I can't vote for this proposal, I think it's a serious issue.

Am Fri, 25 Jul 2014 01:23:21 +0000
schrieb "Jakob Ovrum" <jakobovrum@gmail.com>:

> On Thursday, 24 July 2014 at 23:40:56 UTC, Jakob Ovrum wrote:
> > How often have you seen a formatted log message logged in a loop? I'd wager that happens quite often. Using `format` is a no-go as it completely thrashes the GC, it needs to use `formattedWrite` to write directly to the underlying buffer (such as a file).
> 
> To eloborate on this: using `format` like std.logger currently does, for short-lived strings that are consumed and discared almost immediately in a higher stack frame, is extremely inefficient as every string needs at least one heap memory allocation (though `format` can easily do *multiple* in one call). It's a good way to quickly thrash and fragment the GC heap, putting an extreme amount of stress on the collector.
> 
> Even C has fprintf. If we don't provide efficient formatted writing, people will not use our abstractions.
> 
> The solution is to use `formattedWrite` for custom allocation behaviour. When I've used `formattedWrite` for this kind of problem before, it has generally come down to the following patterns; let's assume the underlying sink is a file:
> 
>   * One solution is to use `formattedWrite` to write to the file
> directly. Of course, Logger doesn't provide an interface for
> writing partial log messages, or writing log messages in chunks,
> so this would require modifying the basic API. Also,
> `formattedWrite` doesn't guarantee any minimum chunk size, so in
> the worst case, it might result in one write operation per
> character, which is very inefficient.
> 
> * Another solution is to use `formattedWrite` to write to a stack-allocated character buffer, then subsequently pass it to `writeLogMsg`. This is a leaky abstraction because it puts an arbitrary upper limit on how long log entries can be. A practical compromise is to revert to a heap allocation for entries that don't fit in the stack buffer, but we can do better;
> 
> * The best solution is a hybrid one. Use `formattedWrite` to write to a stack-allocated buffer, then whenever it's full, write the contents of the buffer to the underlying sink (this is easily doable by passing a delegate to `formattedWrite` that sees the stack buffer as an upvalue). Yes, this does require modifying the basic logger API to support partial writes, but it gives us optimal performance.
> 
> The last solution gives us no dynamic memory allocations unless an exception is thrown, while still minimizing the number of writes to the underlying sink, which is important when writes can be expensive, such as writes to a file or a socket.

https://github.com/burner/logger/pull/9

On 25/07/14 09:37, Brian Schott wrote:

> On the other hand, not having to re-implement trivial functions that you
> need all the time is a form of usefulness.

I'm arguing that NullLogger is not something that's needed all the time.

-- 
/Jacob Carlborg

On Thursday, 24 July 2014 at 23:40:56 UTC, Jakob Ovrum wrote:
> On Thursday, 24 July 2014 at 23:01:56 UTC, Robert burner Schadek wrote:
>> I do this lazily in a function, because having it global froze std.concurrency and std.process unittest. I couldn't figure out why.
>
> It could still be initialized lazily, using `emplace`, ala
>
> ---
> private static __gshared Logger _defaultLogger;
>
> Logger defaultLogger() @safe @nogc
> {
>     static __gshared ubyte[__traits(classInstanceSize, StdoutLogger)] buffer;
>
>     if(!_defaultLogger) // TODO: thread safety
>         _defaultLogger = () @trusted { return emplace!StdoutLogger(buffer); }();
>
>     return _defaultLogger;
> }
>
> void defaultLogger(Logger newDefaultLogger) @safe @nogc
> {
>     _defaultLogger = newDefaultLogger;
> }
> ---
>
>> As said earlier, I think GC and Logger is a none issue. I mean how often has anyone seen a Logger created in a loop over and over again.
>
> Some programs want to forego having a GC-heap entirely. That means any GC allocation is a no-go.
>
> Class instances don't have to be GC-allocated, so it's not an issue to use classes.
>
>> nothrow will be hard as std.logger uses format, same for nogc
>
> How often have you seen a formatted log message logged in a loop? I'd wager that happens quite often. Using `format` is a no-go as it completely thrashes the GC, it needs to use `formattedWrite` to write directly to the underlying buffer (such as a file).
>
> Even using `formattedWrite` though, `nothrow` is still a problem, and since exceptions are still GC-allocated, it doesn't help with @nogc either. The latter is something we can fix in the future though.
>
>> So you're thinking of a stack array?
>
> No, MultiLogger could manage a non-GC yet still heap-allocated array using std.container.array.Array. It uses the C heap internally, i.e. malloc, realloc and free. Sortedness can be used for searching by name in logarithmic time if desired.

Andrei asked for a simple array like multilogger. If it uses an array sorting can be toggled by an template parameter

>
>> What about the log functions and there implementation as well as the Logger specific LogLevel and name?
>
> The log functions don't need to be virtual, only the `writeLogMsg` function does, so these implementations can either be final member functions of the interface, or UFCS functions as I suggested in the corresponding line comment.

all log functions are templates.

>
> The Logger-specific LogLevel and the name do not have to be implemented by Logger. Leave it to concrete classes to implement those. As an internal aid, an abstract GenericLogger base class that manages these properties as member variables (as Logger currently does) can help.

that seams manageable

On Friday, 25 July 2014 at 07:11:06 UTC, Jacob Carlborg wrote:
> On 22/07/14 11:43, ponce wrote:
>
>> NullLogger is there precisely because it's trivial and needed.
>
> If it's so trivial then the users can implement that themselves. A standard library isn't about implementing what's trivial, it's about implementing what's most useful to most people.

That is interesting clash of attitude to standard library :) In my opinion it is quite the opposite - standard library is here to ensure primarily that all trivial things are done in a same way in all projects. Anything more complicated can be packaged as a separate library but trivialities are exactly the things that set up common ground and ensure good library interoperation.

On Friday, 25 July 2014 at 07:11:06 UTC, Jacob Carlborg wrote:
> On 22/07/14 11:43, ponce wrote:
>
>> NullLogger is there precisely because it's trivial and needed.
>
> If it's so trivial then the users can implement that themselves. A standard library isn't about implementing what's trivial, it's about implementing what's most useful to most people.

http://dlang.org/phobos/std_math.html#.fmin

"fmin" is trivial as well, is not used all the time, yet is in the standard library. Sometime trivial stuff is still good to have.

On Friday, 25 July 2014 at 13:01:29 UTC, francesco cattoglio wrote:
> On Friday, 25 July 2014 at 07:11:06 UTC, Jacob Carlborg wrote:
>> On 22/07/14 11:43, ponce wrote:

> "fmin" is trivial as well, is not used all the time, yet is in the standard library. Sometime trivial stuff is still good to have.

I think one good reason to put some things into the standard library is to allow and enforce them to become a... standard.

That way, probably everybody would do that thing in a unique way, the... standard way.

Optimization of standard features also propagates everywhere.