On Monday, 1 September 2014 at 04:32:42 UTC, Kevin Lamonte wrote:
> Does this logger already exist, could I take a look at it?

Not in D AFAIK, circular in-memory logging is a common technique for fixing servers though.

> If not, if someone writes an appender for writing to the database, you could accomplish this goal with log4d using a buffer appender that triggers on fatal.

I guess the most robust solution is to use shared memory and fork, when the child dies you soup up the log and upload it to a logging-server.

I am also interested in lazy formatting, meaning you log a reference to the immutable formatting string and the parameters, but wait with the formatting until the result is needed.

On Sunday, 31 August 2014 at 01:09:33 UTC, Ola Fosheim Grøstad wrote:
> I've got some questions:
>
> How does logging interact with pure? You need to be able to log in pure functions.

Weakly pure function can take logger as an argument (so I doubt it is useful in practice). Strongly pure functions can't be logged in non-debug statement pretty much by D definition of purity. This may be or may not be an issue but is definitely goes out of the scope of this Phobos proposal. If you have any specific ideas how to address it, please create a separate thread.

> Does the logger implementation flush() in the case of a crash? (Does it trap all crashes in a way that ensures that  logging buffers are written to disk?)

Current implementations use basic std.stdio facilities and those flush upon writing a newline symbol -> flush happens after each log call. More efficient buffered implementation can be provided later, this shouldn't affect the API.

> Is logf() needed? Can't you somehow detect that the string is an immutable string literal with string formatting characters?

Unreliable, not KISS. I think it is a bad idea.

On Monday, 1 September 2014 at 16:30:46 UTC, Dicebot wrote:
> purity. This may be or may not be an issue but is definitely goes out of the scope of this Phobos proposal. If you have any

[…]

> Current implementations use basic std.stdio facilities and those flush upon writing a newline symbol -> flush happens after each log call. More efficient buffered implementation can be provided later, this shouldn't affect the API.

[…]

> Unreliable, not KISS. I think it is a bad idea.

If the standard library does not provide the following from the get go:

1. general usefulness
2. performance
3. KISS in terms of interface (not in language semantics)

then people will be better off rolling their own. Creating a simple logger is not difficult, the challenge is in creating a performant, generally useful one with legible syntax and full CT type safety.

On Monday, 1 September 2014 at 16:45:29 UTC, Ola Fosheim Grøstad wrote:
> If the standard library does not provide the following from the get go:
>
> 1. general usefulness
> 2. performance
> 3. KISS in terms of interface (not in language semantics)
>
> then people will be better off rolling their own. Creating a simple logger is not difficult, the challenge is in creating a performant, generally useful one with legible syntax and full CT type safety.

You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here only to provide standard API for those to integrate with each other and few trivial common implementation as examples. Rest is dub business.

And no, magic identification of format string is neither language KISS nor interface KISS.

On Monday, 1 September 2014 at 16:52:16 UTC, Dicebot wrote:
> You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here

*are expected

On Monday, 1 September 2014 at 16:52:16 UTC, Dicebot wrote:
> You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here only to provide standard API for those to integrate with each other and few trivial common implementation as examples. Rest is dub business.

Oh, I understand the intentions perfectly well, but the default should be performant, multithreaded, and cover the most common use scenario.

Without a performant reference implementation that can be used for performance testing it isn't  convincing. You also need to account for issues such as multi-threading, throttling etc. Meaning, you might need to have queues and a merger-thread, defer formatting etc etc.

> And no, magic identification of format string is neither language KISS nor interface KISS.

Formatting should be typesafe and preferably configurable with custom formatters, so it should resolve at compile-time one way or the other.

You don't want an exceptional path to have a runtime error triggered by a log() statement, i.e. you don't want the possibility that turning on logging can introduce bugs. That should be the most important requirement.

On Monday, 1 September 2014 at 10:43:34 UTC, Ola Fosheim Grøstad wrote:

> I guess the most robust solution is to use shared memory and fork, when the child dies you soup up the log and upload it to a logging-server.

I'm used to a centralized system taking logs on a continuous basis, with "normal, I'm happy" messages coming through in a regular interval.  When the application dies, it already has had its messages emitted and sucked up by the collection system, and if its heartbeat messages aren't seen then people are prompted to investigate anyway.  It's on the main server (e.g. syslog or LogStash) to handle storage space issues like log rotation.

> I am also interested in lazy formatting, meaning you log a reference to the immutable formatting string and the parameters, but wait with the formatting until the result is needed.

log4d does this, it saves the Logger+LogEntry references and only applies PatternLayouts at the end, BUT it does so with the risk that additional fields specified in the conversionPattern might be wrong since they were not generated/evaluated in the context of the original log call.  Thread ID (%t) is the main culprit (since it is not in LogEntry, PatternLayout has to get it), but also time between log calls and time since application startup (%r/%R).

But it sounds like you want std.logger to not apply formatting even in its infof/errorf/etc functions.  That might be a problem for things like the number of rows in a result set, the current time, or the remote system hostname.  For example, by the time the logging infrastructure evaluates the number of rows, the result set is long gone and could throw an exception.  I would argue that this kind of lazy evaluation would be fine if it was not enabled by default.

On Monday, 1 September 2014 at 18:57:25 UTC, Ola Fosheim Grøstad wrote:
> On Monday, 1 September 2014 at 16:52:16 UTC, Dicebot wrote:
>> You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here only to provide standard API for those to integrate with each other and few trivial common implementation as examples. Rest is dub business.
>
> Oh, I understand the intentions perfectly well, but the default should be performant, multithreaded, and cover the most common use scenario.

+1

While its useful for the standard library to provide stubs, these mean very little without a default implementation. And the latter shall be quite generic to cover most of the use cases.

Rolling one's own loggers shall be done only if the world is not enough.

On Tuesday, 2 September 2014 at 06:53:30 UTC, eles wrote:
>> Oh, I understand the intentions perfectly well, but the default should be performant, multithreaded, and cover the most common use scenario.
>
> +1
>
> While its useful for the standard library to provide stubs, these mean very little without a default implementation. And the latter shall be quite generic to cover most of the use cases.
>
> Rolling one's own loggers shall be done only if the world is not enough.

I disagree and it was declared among goals of this module from the very beginning that it won't go that way, won't even attempt to do it. If you have some good ideas about better default implementation - make pull request after it is merged into "std.experimental". Right now it is not in the scope of the review and I will simply ignore all comments that are related purely to implementation.

However, if there any API issues that will likely block the implementation you want - those are very important to hear about. This is #1 priority right now.

On Tuesday, 2 September 2014 at 06:24:45 UTC, Kevin Lamonte wrote:
> I'm used to a centralized system taking logs on a continuous basis, with "normal, I'm happy" messages coming through in a regular interval.  When the application dies, it already has had its messages emitted and sucked up by the collection system, and if its heartbeat messages aren't seen then people are prompted to investigate anyway.  It's on the main server (e.g. syslog or LogStash) to handle storage space issues like log rotation.

Yes, I think we are discussing many different things at the same time here and it would be a good idea to sort out the different use cases since that affects functionality. I have not thought about heartbeats/keep-alive etc as logging, but it is reasonable to view them as such.

I see a difference between signalling state, tracing execution and logging state. I guess one roughly can say that:

- signalling is for coordination of subsystems
- logging state is for tracking effects on the database
- tracing is for detecting logic failure after a crash

?

> log4d does this, it saves the Logger+LogEntry references and only applies PatternLayouts at the end, BUT it does so with the risk that additional fields specified in the conversionPattern might be wrong since they were not generated/evaluated in the context of the original log call.  Thread ID (%t) is the main culprit (since it is not in LogEntry, PatternLayout has to get it), but also time between log calls and time since application startup (%r/%R).

Sounds interesting. I'll have a look at log4d later. But it is not fully typesafe then? The way I see it you should log a tuple of values and a reference to a type-safe formatting expression, but only apply the formatting expression when you need to so you don't burden the performance thread with unnecessary work.

> But it sounds like you want std.logger to not apply formatting even in its infof/errorf/etc functions.  That might be a problem for things like the number of rows in a result set, the current time, or the remote system hostname.  For example, by the time the logging infrastructure evaluates the number of rows, the result set is long gone and could throw an exception.

I think you should log the values as a tuple, but not do the string-conversion, but it is more important for tracing execution than for logging. I guess formatting high level info/error is acceptable, but for tracing I am only interested in very terse value/type info along with an indicator of context.

Performance and trouble-free type safe logging is much more important than "nice formatting" IMO. Traced execution will primarily be used for log analysis after a crash. You can use this on game clients, game servers, web servers etc…

E.g.:

log configuration + trace last 50000 events -> crash -> compress and upload for analysis.