On 10/15/13 12:52 AM, Robert Schadek wrote:
> On 10/15/2013 04:06 AM, Eric Anderton wrote:
>> On Monday, 14 October 2013 at 11:39:52 UTC, Dicebot wrote:
>>> Lets unleash the forces of constructive destruction.
>>
>> So, not to be too heavy-handed with criticism on this library, but I
>> think this should come up to par with solutions like log4j, log4cpp,
>> or log4cxx, with respect to features and capabilities.  Libraries like
>> these have enjoyed a lot of very serious use, and once you have
>> something like that in your project, it's hard to not use most of what
>> they have to offer.  There's really not a lot of fluff in those
>> solutions.
> IMO these libraries are to heavy. Especially with phobos inclusion in mind.

I agree. A bunch of stuff at Facebook is heavily relying on logging for statistics and debugging, yet we're fine with the relatively scarce API of Google log. That said, I'm clearly biased because I've never used log4xxx.

One note - log4j, log4cxx, and log4cpp are not part of the respective languages' standards. That doesn't mean much (in fact it may be a competitive advantage to integrating log4d in std) but it is one factor to consider.

Eric, could you please enumerate a short list of features of log4j that you think would be really missed if absent?


Andrei

I think one increasingly important point for std.log is 'structured logging'.

Structured logging is basically not simply logging textual messages, but also logging additional KEY/VALUE pairs of data. The idea is that logs should not only be readable by humans but also easy to parse and analyze. Structured logging often also includes UUIDs to simplify finding similar errors/problems in a log file.

For example: Instead of logging
logf("Couldn't login as %s on server %s : Reason: %s", user, server,
      errorCode);

we'd do:
log(LOGIN_FAILED_UUID, "Couldn't log in", ["User": user, "Server":
    server, "ErrorCode": errorCode]);

The log can then be queried for all events with 'LOGIN_FAILED_UUID' uuid, Server="..." ErrorCode=42, etc.

As a nice benefit structured logging can also log the function, module and line of the source file that logged the message, Exceptions can be written to the log in a nice way, etc.

SystemDs journal [1] is a structured replacement for syslog and is already being used on some linux distributions (ArchLinux, Fedora). It's likely that the journal will replace syslog on linux so at least for server-side software structured logging support is becoming important. The GNOME guys are also working on a log viewer for systemd's journal [2].

std.log certainly doesn't need to provide a backend for the journal right now. But some questions regarding structured logging need to taken into account:

* Will structured logging be integrated with the normal logging at all?
* Will the public API somehow collide with the proposed std.log API?
* Should the API for log backends already support structured logging or
  is it possible to add this later on?

Most important:
* What to do about structured logging calls when logging to a simple
  text logger? Ignore the additional fields or serialize them? What
  about the UUID?

Even if we don't have a backend implementation it'd be nice to have the
public API available. Otherwise new projects cannot use structured
logging with std.log right now and the user code needs to be
rewritten once std.log does support structured logging.

[1]
http://0pointer.de/blog/projects/journalctl.html

[2] https://mail.gnome.org/archives/gnome-announce-list/2013-September/msg00097.html

On 10/15/2013 05:20 PM, Johannes Pfau wrote:
> I think one increasingly important point for std.log is 'structured logging'.
>
> Structured logging is basically not simply logging textual messages, but also logging additional KEY/VALUE pairs of data. The idea is that logs should not only be readable by humans but also easy to parse and analyze. Structured logging often also includes UUIDs to simplify finding similar errors/problems in a log file.
>
> For example: Instead of logging
> logf("Couldn't login as %s on server %s : Reason: %s", user, server,
>       errorCode);
>
> we'd do:
> log(LOGIN_FAILED_UUID, "Couldn't log in", ["User": user, "Server":
>     server, "ErrorCode": errorCode]);
>
> The log can then be queried for all events with 'LOGIN_FAILED_UUID' uuid, Server="..." ErrorCode=42, etc.
>
> As a nice benefit structured logging can also log the function, module and line of the source file that logged the message, Exceptions can be written to the log in a nice way, etc.
>
> SystemDs journal [1] is a structured replacement for syslog and is already being used on some linux distributions (ArchLinux, Fedora). It's likely that the journal will replace syslog on linux so at least for server-side software structured logging support is becoming important. The GNOME guys are also working on a log viewer for systemd's journal [2].
>
> std.log certainly doesn't need to provide a backend for the journal right now. But some questions regarding structured logging need to taken into account:
>
> * Will structured logging be integrated with the normal logging at all?
> * Will the public API somehow collide with the proposed std.log API?
> * Should the API for log backends already support structured logging or
>   is it possible to add this later on?
>
> Most important:
> * What to do about structured logging calls when logging to a simple
>   text logger? Ignore the additional fields or serialize them? What
>   about the UUID?
>
> Even if we don't have a backend implementation it'd be nice to have the
> public API available. Otherwise new projects cannot use structured
> logging with std.log right now and the user code needs to be
> rewritten once std.log does support structured logging.
>
> [1]
> http://0pointer.de/blog/projects/journalctl.html
>
> [2] https://mail.gnome.org/archives/gnome-announce-list/2013-September/msg00097.html
Currently there is no structured logging implemented in std.logger.

That been said, you can add it. The method Logger.logf is a variadic template. You can simple create your own Logger Class and overwrite that method and implemented your structured logging approach there. The only pitfall is that you need a ref of type WhateverYourLoggerClassIsCalled and call logf on that or you fall back to the default logf implementation provided by the Logger class. Another way would be to parse the message passed to logMessage and go from there.

Long story short. Global default Logger means structured logging
requires parsing. All other logger require correct reference type of logger.

On Tuesday, 15 October 2013 at 15:53:34 UTC, Robert Schadek wrote:
> That been said, you can add it. The method Logger.logf is a variadic
> template. You can simple create your own Logger Class and overwrite that
> method and implemented your structured logging approach there. The only
> pitfall is that you need a ref of type WhateverYourLoggerClassIsCalled
> and call logf on that or you fall back to the default logf
> implementation provided by the Logger class. Another way would be to
> parse the message passed to logMessage and go from there.
>
> Long story short. Global default Logger means structured logging
> requires parsing. All other logger require correct reference type of logger.

I don't think it is a good design to conflate logging mode with output mode in a single logger class. I'd prefer to see log() as variadic structured logger and logf() as it is now (making conditional logger `logIf()`)

On Tue, Oct 15, 2013 at 8:17 AM, Andrei Alexandrescu < SeeWebsiteForEmail@erdani.org> wrote:

> One note - log4j, log4cxx, and log4cpp are not part of the respective languages' standards. That doesn't mean much (in fact it may be a competitive advantage to integrating log4d in std) but it is one factor to consider.
>

It also gave rise to slf4j, to tie the various (java) logging solutions
together.

>From a core library standpoint, the slf4j model might be a good one to
emulate - provide a basic logging abstraction that can then be plumbed to whichever logging implementation is needed.  Logback is essentially the logging framework written by the slf4j guys, which is why I used it as an example.

And though I am not Eric, I do have a short list.  These are things that log4j/slf4j/etc provide that I'd consider required of any log framework before I use it in a production* environment:

Multiple log destinations (sinks/appenders), configurable.
  - required for logging to file, syslog, etc as appropriate
  - different running instances of same code may need different log
names/locations/appenders

Hierarchical logs, with inheritance of levels, configure at runtime.  Turn
on/off log level for part of hierarchy.
  - for debugging own code without being overwhelmed with log statements
from elsewhere
  - turn off extraneous logging in dependencies, or turn it on for deep
diving

Configurable log ouput with custom fields (time, thread, etc).
  - required for making log output match predefined formats
  - include needed metadata in the log line

Allow 'lazy' evaluation/formatting of log output (parameterized logging
equivalent).
  - no performance excuse not to log

Log rotation
  - if this isn't there out of the box, guarantee will be first
customization


* where 'production' is biased towards high availability services

On 2013-10-15 16:13, Dicebot wrote:

> For example, sending mail is clearly relying on external stuff and
> should never be in Phobos (again, std.net.curl was a terrible mistake)

I don't see why this couldn't be included in Phobos, if it doesn't have any external dependencies.

-- 
/Jacob Carlborg

On Tuesday, 15 October 2013 at 07:52:28 UTC, Robert Schadek wrote:
> On 10/15/2013 04:06 AM, Eric Anderton wrote:
>> Here's what I think is missing:
>> - System log support (as others have mentioned).  This would be syslog
>> or WEL, depending on environment.
> This is sort of the idea of the design, I can't anticipate your needs
> therefor I should not try. I should try to give you guidelines or a
> framework to work against.
>>
>> - Guarantees or options for working with log rotation (logrotate.d). It's nice to either know that you must restart your daemon once logs
>> are rotated, or can configure logging to re-open handles automatically
>> or when it detects rotation has occurred.
> See previous point

Disagree. We need a log rotation support.
As I can see, available options could be:

* rotating conditions
 - by date (rotate every hour, day (default), week, month, year)
 - by file size (rotate if file size more than ... Mb)
 - by count log lines (rotate if log contains more than ... log lines)
 - combination of previous conditions (for example, rotate every day or rotate if file size more than 100 Mb)

* file names after rotation
 - by numbers (my.log, my.log.0, my.log.1, ...)
 - by ISO date and time (my-2013-10-16-00-00-00.log)

* ability to use system log rotation utility

On 10/16/2013 01:34 AM, Jeremy Powers wrote:
> On Tue, Oct 15, 2013 at 8:17 AM, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org <mailto:SeeWebsiteForEmail@erdani.org>> wrote:
>
>     One note - log4j, log4cxx, and log4cpp are not part of the
>     respective languages' standards. That doesn't mean much (in fact
>     it may be a competitive advantage to integrating log4d in std) but
>     it is one factor to consider.
>
>
> It also gave rise to slf4j, to tie the various (java) logging
> solutions together.
>
> From a core library standpoint, the slf4j model might be a good one to emulate - provide a basic logging abstraction that can then be plumbed to whichever logging implementation is needed.  Logback is essentially the logging framework written by the slf4j guys, which is why I used it as an example.
>
> And though I am not Eric, I do have a short list.  These are things that log4j/slf4j/etc provide that I'd consider required of any log framework before I use it in a production* environment:
>
> Multiple log destinations (sinks/appenders), configurable.
>   - required for logging to file, syslog, etc as appropriate
>   - different running instances of same code may need different log
> names/locations/appenders
MultiLogger is planed.
>
> Hierarchical logs, with inheritance of levels, configure at runtime.
>  Turn on/off log level for part of hierarchy.
>   - for debugging own code without being overwhelmed with log
> statements from elsewhere
>   - turn off extraneous logging in dependencies, or turn it on for
> deep diving
no hierarchical logs, KISS just create a logger with different destination. new FileLogger("myDevLog"); $ tail -f myDevLog
>
> Configurable log ouput with custom fields (time, thread, etc).
>   - required for making log output match predefined formats
>   - include needed metadata in the log line
I think this has been discussed twice already, no configuration can anticipate all possible needs and will fail fast and hard. So why try, I rather write 7 lines, than wait for a patch to the configuration parser to appear in my production env.
>
> Allow 'lazy' evaluation/formatting of log output (parameterized
> logging equivalent).
>   - no performance excuse not to log
>
> Log rotation
>   - if this isn't there out of the box, guarantee will be first
> customization
see above argument
>
>
> * where 'production' is biased towards high availability services
>

Short version of below:
I want a powerful logging system.  Maybe std.logging should provide the
interface with some basic functionality, allow other solutions to fill in
gaps.  Should be able to always code against std.logging, complications
added as needed w/o code calling log() caring or changing.


On Wed, Oct 16, 2013 at 12:30 AM, Robert Schadek <realburner@gmx.de> wrote:

>  no hierarchical logs, KISS just create a logger with different
> destination. new FileLogger("myDevLog"); $ tail -f myDevLog
>
>
Without this provided out of the box, I'd have to create my own framework for such on top for any serious use.  This is arguably the most important feature of a logging framework for a large product.  Once you get multiple people/teams/companies/monkeys contributing to a system, you _need_ a way to distinguish and filter logging from each part.  Spitting to different log files is not a solution in most cases, the 'create my own' would have each module logger spitting to the same place as the root, with the root logger controlling what actually made it to the log.

Simple logging framework should be simple.  But it should also be powerful, without requiring custom boilerplate for more complex usage...  Earlier was mention of getting a module's log via import, this seems a good solution interface wise - basic implementation would just return basic logger, but would allow for a hierarchical solution to be plumbed in without the logging code knowing/caring.


 Configurable log ouput with custom fields (time, thread, etc).
>   - required for making log output match predefined formats
>    - include needed metadata in the log line
>
> I think this has been discussed twice already, no configuration can anticipate all possible needs and will fail fast and hard. So why try, I rather write 7 lines, than wait for a patch to the configuration parser to appear in my production env.
>

There are two parts to this: making sure the log output conforms to some format, and making sure required information is included.  You can never anticipate what everyone needs for either, but you can provide the tools to enable them.  Conceptually, this means separating the information being logged from the actual output - the basic logging framework doesn't need to even try to cover every case, so long as it provides hook points.

Once the output is separated from the information, custom output is as simple as a format string.  Important bit is that this is decided by the logger (or whoever configured it) not the code where it is logged.  Can provide basic information by default, or not, so long as there is a mechanism to include custom information.

MDC in slf4j/logback is one approach to the 'information included' problem: http://logback.qos.ch/manual/mdc.html

Not suggesting std.logging needs the same thing, but illustrates one way to solve.  Personal experience has shown this is useful.

On 10/16/2013 08:18 PM, Jeremy Powers wrote:
> Short version of below:
> I want a powerful logging system.  Maybe std.logging should provide
> the interface with some basic functionality, allow other solutions to
> fill in gaps.  Should be able to always code against std.logging,
> complications added as needed w/o code calling log() caring or changing.
>
>
> On Wed, Oct 16, 2013 at 12:30 AM, Robert Schadek <realburner@gmx.de <mailto:realburner@gmx.de>> wrote:
>
>     no hierarchical logs, KISS just create a logger with different
>     destination. new FileLogger("myDevLog"); $ tail -f myDevLog
>
>
> Without this provided out of the box, I'd have to create my own
> framework for such on top for any serious use.  This is arguably the
> most important feature of a logging framework for a large product.
>  Once you get multiple people/teams/companies/monkeys contributing to
> a system, you _need_ a way to distinguish and filter logging from each
> part.  Spitting to different log files is not a solution in most
> cases, the 'create my own' would have each module logger spitting to
> the same place as the root, with the root logger controlling what
> actually made it to the log.
>
> Simple logging framework should be simple.  But it should also be powerful, without requiring custom boilerplate for more complex usage...  Earlier was mention of getting a module's log via import, this seems a good solution interface wise - basic implementation would just return basic logger, but would allow for a hierarchical solution to be plumbed in without the logging code knowing/caring.
I still don't feel that hierarchy building should be part of std.logger, as it is way to heavyweight. But thinking about MultiLogger made me realize, that I need a way to identifier Logger to remove them. So currently, I think MultiLogger will have an AA and Logger will have names (string). You do the math ....
>
>
>>     Configurable log ouput with custom fields (time, thread, etc).
>>       - required for making log output match predefined formats
>>       - include needed metadata in the log line
>     I think this has been discussed twice already, no configuration
>     can anticipate all possible needs and will fail fast and hard. So
>     why try, I rather write 7 lines, than wait for a patch to the
>     configuration parser to appear in my production env.
>
>
> There are two parts to this: making sure the log output conforms to some format, and making sure required information is included.  You can never anticipate what everyone needs for either, but you can provide the tools to enable them.  Conceptually, this means separating the information being logged from the actual output - the basic logging framework doesn't need to even try to cover every case, so long as it provides hook points.
>
logMessage(LoggerPayload); is your all powerful hookpoint.