On Wednesday, 12 June 2013 at 16:34:08 UTC, Andrei Alexandrescu wrote:
> It's quite an old standard feature albeit obscure, I found it and forgot about it a couple of times already. Works everywhere, see http://www.cplusplus.com/reference/cstdio/ferror/ and http://msdn.microsoft.com/en-us/library/9yky46tz(v=vs.80).aspx
>
>> But yes, all open files should be flushed on termination. It's what C's
>> stdio does already.
>
> http://d.puremagic.com/issues/show_bug.cgi?id=10344
>

How come that in a first place, we have a file descriptor that isn't working ?

On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
> On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright <newshound2@digitalmars.com>
> wrote:
>
>> On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
>>> No, it does perform well.  You are still not understanding the proposal.
>>
>> Yes, I understand your proposal quite well.
>>
>> Your benchmark is seriously flawed. Most modern systems cache writes in
>> memory, and have a delayed write to the media. This hides the problem.
>
> Sorry, but this is just bunk.  One uneven flush does not mess up all future
> aligned writes.  I/O is not that fragile.
>
> If you can demonstrate your theory, I will concede.  Until you do, don't expect
> any more responses, it's no use arguing when you don't understand the problem.

May I present source code?
-----------------------------
int fflush(FILE *fp)
{       int length;
        int result= 0;

        /* if fflush(NULL) flush all buffers */
        if (fp == NULL)
        {
                if (flushall() >= 0)
                        result = 0;
        }
        else
        {
          /* don't flush buffer if we are not writing   */
        __fp_lock(fp);
        if ((fp->_flag & (_IOWRT | _IONBF | _IOERR)) == _IOWRT &&
            (fp->_base
#ifdef BIGBUF
                || fp->_seg
#endif
                        ))
        {       length = fp->_ptr - fp->_base;  /* # of bytes in buffer */
#ifdef BIGBUF
                if (length && _writex(fp->_file,fp->_base,length,fp->_seg)
                        != length)
                        fp->_flag |= _IOERR;
#else
                if (length)
                {   int nwritten = write(fp->_file,fp->_base,length);
                    /* The following check for isatty() is because:
                     *  #define WIN32_LEAN_AND_MEAN
                     *  #include <windows.h>
                     *  #include <stdio.h>
                     *  void main()
                     *  {
                     *      // Set console output to UTF-8 (one can use 'chcp 65001' instead)
                     *      SetConsoleOutputCP( 65001 );
                     *      // Latin small letter e with acute
                     *      fputs( "Output utf-8 accented char \xc3\xa9\n... and the rest is cut of
                     *  }
                     * fails because WriteFile() apparently treats UTF-8
                     * sequences as 1 byte when writing to the console.
                     */
                    if (!nwritten || (nwritten != length && !isatty(fp->_file)))
                        fp->_flag |= _IOERR;
                }
#endif
                fp->_cnt = 0;
                fp->_ptr = fp->_base;
        }
        else
                fp->_cnt = 0;
        result = (ferror(fp)) ? EOF : 0;
        __fp_unlock(fp);
        }
        return result;
}
-----------------------------------
This does exactly what I said it does. It's from the Digital Mars C runtime library. Now you could argue that this code sux, and you might even be right. But there are a lot of implementations of C fflush out there - are you sure that all of them will realign after a misaligned write?

And even if they do realign, you cannot write 10 bytes to a disk. You can only write a block or a sector. Which means the next write, even if aligned, has to write that block or sector again.

You are proposing that this repeated write of the first sector be done for all file output. You probably won't notice a speed difference if write caching is done, but that doesn't make it a good idea.

On 6/12/13 12:40 PM, Walter Bright wrote:
> You are proposing that this repeated write of the first sector be done
> for all file output.

No, he's proposing that only the very first write flushes. Not every write. I'd argue it's a bizarre thing to do, but not that it's necessarily slow.

Andrei

Denis Koroskin wrote:
> The best solution would be for writeln() to throw on use, and I think it's fairly easy to implement: just flush once after using the file descriptor for the first time, and throw if it fails.
> 
> While it doesn't cover a case where file descriptor becomes non-writable during the program lifetime, it covers the most common case of file descriptor not being writable at all.

	Actually, the best solution is to check if the file is writable
before writing and *without flushing*. On Posix systems, this can be
done with:

fcntl (fd, F_GETFL) & W_OK

		Jerome
-- 
mailto:jeberger@free.fr
http://jeberger.free.fr
Jabber: jeberger@jabber.fr

On 6/12/2013 10:07 AM, Andrei Alexandrescu wrote:
> On 6/12/13 12:40 PM, Walter Bright wrote:
>> You are proposing that this repeated write of the first sector be done
>> for all file output.
>
> No, he's proposing that only the very first write flushes. Not every write. I'd
> argue it's a bizarre thing to do, but not that it's necessarily slow.

I meant for every file being output.

And, again, it is forcing two writes of the first block, at a minimum. The perf cost can be hidden by write caching, but to rely on that is bad design and undermines how the I/O system is set up and tuned.

I predict a WTF bug report on this in the future, as someone with an unusual device attached is looking for the cause of why his D program writing to it behaves badly. Or he just writes it off as "D sux".

There's a *reason* why stdio distinguishes between block devices and character devices.

On 6/12/2013 10:36 AM, "Jérôme M. Berger" wrote:
> 	Actually, the best solution is to check if the file is writable
> before writing and *without flushing*. On Posix systems, this can be
> done with:
>
> fcntl (fd, F_GETFL) & W_OK

Sure, but that doesn't cover the case of a write failure because the disk is is near full and then is full.

The write has to be tested to see if it succeeded, not assumed to succeed because only one of the possible failure conditions is not present.

On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
> If you can demonstrate your theory, I will concede.  Until you do, don't expect
> any more responses, it's no use arguing when you don't understand the problem.

There's another reason it won't work. If there are 4 bytes of free space left on the drive:

void main() {
   writeln("foo");   // succeeds
   writeln("bar");   // silently fails
}

Here's a radical thought: should stdout even be global?

With any other file, this wouldn't be an issue because File's dtor calls close() which (I'm pretty sure) flushes the file, and thus would detect the error at the latest, when it goes out of scope.

I think if stdout wasn't global, I'd be among those complaining about the code it breaks and the hassle it brings, but it seems to me that this is the real problem here. It doesn't go out of scope until the program is terminating.

On Wed, 12 Jun 2013 12:40:53 -0400, Walter Bright <newshound2@digitalmars.com> wrote:

> On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
>> On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright <newshound2@digitalmars.com>
>> wrote:
>>
>>> On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
>>>> No, it does perform well.  You are still not understanding the proposal.
>>>
>>> Yes, I understand your proposal quite well.
>>>
>>> Your benchmark is seriously flawed. Most modern systems cache writes in
>>> memory, and have a delayed write to the media. This hides the problem.
>>
>> Sorry, but this is just bunk.  One uneven flush does not mess up all future
>> aligned writes.  I/O is not that fragile.
>>
>> If you can demonstrate your theory, I will concede.  Until you do, don't expect
>> any more responses, it's no use arguing when you don't understand the problem.
>
> May I present source code?
> -----------------------------
> int fflush(FILE *fp)
> {       int length;
>          int result= 0;
>
>          /* if fflush(NULL) flush all buffers */
>          if (fp == NULL)
>          {
>                  if (flushall() >= 0)
>                          result = 0;
>          }
>          else
>          {
>            /* don't flush buffer if we are not writing   */
>          __fp_lock(fp);
>          if ((fp->_flag & (_IOWRT | _IONBF | _IOERR)) == _IOWRT &&
>              (fp->_base
> #ifdef BIGBUF
>                  || fp->_seg
> #endif
>                          ))
>          {       length = fp->_ptr - fp->_base;  /* # of bytes in buffer */
> #ifdef BIGBUF
>                  if (length && _writex(fp->_file,fp->_base,length,fp->_seg)
>                          != length)
>                          fp->_flag |= _IOERR;
> #else
>                  if (length)
>                  {   int nwritten = write(fp->_file,fp->_base,length);
>                      /* The following check for isatty() is because:
>                       *  #define WIN32_LEAN_AND_MEAN
>                       *  #include <windows.h>
>                       *  #include <stdio.h>
>                       *  void main()
>                       *  {
>                       *      // Set console output to UTF-8 (one can use 'chcp 65001' instead)
>                       *      SetConsoleOutputCP( 65001 );
>                       *      // Latin small letter e with acute
>                       *      fputs( "Output utf-8 accented char \xc3\xa9\n... and the rest is cut of
>                       *  }
>                       * fails because WriteFile() apparently treats UTF-8
>                       * sequences as 1 byte when writing to the console.
>                       */
>                      if (!nwritten || (nwritten != length && !isatty(fp->_file)))
>                          fp->_flag |= _IOERR;
>                  }
> #endif
>                  fp->_cnt = 0;
>                  fp->_ptr = fp->_base;
>          }
>          else
>                  fp->_cnt = 0;
>          result = (ferror(fp)) ? EOF : 0;
>          __fp_unlock(fp);
>          }
>          return result;
> }
> -----------------------------------
> This does exactly what I said it does. It's from the Digital Mars C runtime library. Now you could argue that this code sux, and you might even be right. But there are a lot of implementations of C fflush out there - are you sure that all of them will realign after a misaligned write?

If I understand correctly, the buffer is flushed, and ptr is reset to the base, the count is reset to 0.  I don't see any code in there that does any kind of realignment.  Are you suggesting that other fflush code doesn't do this?

> And even if they do realign, you cannot write 10 bytes to a disk. You can only write a block or a sector. Which means the next write, even if aligned, has to write that block or sector again.

That is one block or sector.  The write cache of the OS or the drive will probably absorb this hit anyway.

The performance hit is extremely negligible.  Not only that, but the hardware can differ from file system to file system.  You are going through the file system driver, through the disk driver, through the disk.  All of those pieces are written to optimize writes that SPECIFIC hardware.  There isn't much you can do to make this perform poorly.

The only performance hit you can really affect is the system call penalty.  And that is done by allowing the normal flushing routine to continue for subsequent writes.

> You are proposing that this repeated write of the first sector be done for all file output. You probably won't notice a speed difference if write caching is done, but that doesn't make it a good idea.

No, this is incorrect.  fflush is not called on subsequent writeln.

Not only that, but we only have to do this on FILE * that were initialized with unknown file descriptors (such as stdin/stdout/stderr).  If we use fopen, you don't have to do this.

-Steve

On 6/12/13 1:36 PM, "Jérôme M. Berger" wrote:
> Denis Koroskin wrote:
>> The best solution would be for writeln() to throw on use, and I
>> think it's fairly easy to implement: just flush once after using
>> the file descriptor for the first time, and throw if it fails.
>>
>> While it doesn't cover a case where file descriptor becomes
>> non-writable during the program lifetime, it covers the most
>> common case of file descriptor not being writable at all.
>
> 	Actually, the best solution is to check if the file is writable
> before writing and *without flushing*. On Posix systems, this can be
> done with:
>
> fcntl (fd, F_GETFL)&  W_OK
>
> 		Jerome

When do you check and how frequently? I think the right thing to do is as the bug report indicates.

Andrei