On 6/13/2013 6:24 AM, Steven Schveighoffer wrote:
> If this was such a big deal, then FILE * would guarantee the alignment went
> back.  It's not that hard.

I know it isn't hard. But D's charter is not to go and patch the C runtime library, nor is that remotely practical across all the thousands of C implementations out there.


> And I have experience with turning them off -- some of my clients requested that
> to avoid losing data that was in the write cache.  It does not all of a sudden
> slow down the performance of the drive.

I ran some experiments a while back writing to a slow SD card. There was a big difference in speed when turning write caching on and off. I bet you'll see quite a whopper of a difference if you try that with a floppy drive!

In other words, the difference shows up when you've got a fairly slow device. Write caching would not have been invented if it had no effect.


> But this whole discussion is academic at this point, Jérôme identified that we
> can simply check to see if the file descriptor is valid, without flushing.

His fix is insufficient for other reasons as already pointed out.


> This isn't the problem that was presented.  The problem that was presented is
> that given an invalid file descriptor, writeln happily works (as long as you
> don't cause a flush) and does not throw.  This is unintuitive to someone who is
> expecting writeln to choke on an invalid descriptor.

That behavior is normal to anyone used to working with C stdio.

What is wrong and needs fixing is the program exiting with "success" indication when the output has actually failed.

On Thu, Jun 13, 2013 at 11:11:06AM -0700, Walter Bright wrote:
> On 6/13/2013 6:24 AM, Steven Schveighoffer wrote:
[...]
> >This isn't the problem that was presented.  The problem that was presented is that given an invalid file descriptor, writeln happily works (as long as you don't cause a flush) and does not throw.  This is unintuitive to someone who is expecting writeln to choke on an invalid descriptor.
> 
> That behavior is normal to anyone used to working with C stdio.
> 
> What is wrong and needs fixing is the program exiting with "success" indication when the output has actually failed.

Wouldn't an easy fix be for DMD to insert this into main()?

	scope(exit) if (fflush(stdout) != 0)
		throw new Exception(...);

	// ... rinse, repeat for whatever other fd's we care about

Of course, instead of hardcoding stdio into DMD, the scope(exit) can call some kind of library cleanup hook, maybe something like:

	scope(exit) _druntimeCleanup();

and the throw can happen inside _druntimeCleanup, which should cause the exit status to be non-zero.

Not sure how to make this exception user-catchable, though.


T

-- 
Ph.D. = Permanent head Damage

On 2013-06-13 20:11, Walter Bright wrote:

> I know it isn't hard. But D's charter is not to go and patch the C
> runtime library, nor is that remotely practical across all the thousands
> of C implementations out there.

Why not? Why shouldn't we try to be better than C? Perhaps we shouldn't rely on the C runtime library in this case, if possible.

> That behavior is normal to anyone used to working with C stdio.

But we're using D here, not C. You have to stop assume that everyone using D knows C or C++. Just because you do doesn't mean everyone else does.

-- 
/Jacob Carlborg

On 6/14/2013 12:10 AM, Jacob Carlborg wrote:
> On 2013-06-13 20:11, Walter Bright wrote:
>
>> I know it isn't hard. But D's charter is not to go and patch the C
>> runtime library, nor is that remotely practical across all the thousands
>> of C implementations out there.
>
> Why not? Why shouldn't we try to be better than C? Perhaps we shouldn't rely on
> the C runtime library in this case, if possible.
>
>> That behavior is normal to anyone used to working with C stdio.
>
> But we're using D here, not C. You have to stop assume that everyone using D
> knows C or C++. Just because you do doesn't mean everyone else does.


D is set up to work hand-in-glove with C. This isn't going to change, as such compatibility is a big feature of D.

Part of that is seamlessly working with existing C I/O. D code that does writeln's interleaved with C code that does printf's should work as expected.

Since this particular issue can be easily fixed without "fixing" the C runtime, there is no motivation to do so, either.

On Wednesday, 12 June 2013 at 06:48:48 UTC, Walter Bright wrote:
> On 6/11/2013 10:15 PM, deadalnix wrote:
>> On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
>>> I don't agree. Buffering is often done on page size boundaries - throwing out
>>> a random number of characters and then flushing will get it all wonky.
>>
>> You clearly missed something in the discussion here. The proposal is to flush
>> once at first use, so an Exception is thrown. Nothing change after that first
>> flush at initialization, other flushes stay where they are.
>
> Not at all. A flush forces a write to the disk - that's the point of it. Disks are not at all well tuned to writing a few bytes, they like to be written in aligned blocks of block sizes, and the I/O subsystem is designed for that.
>
> This is why stdout has a flag in it saying if it is a "block oriented" or "character oriented" device. It makes a big difference. This proposal attempts to treat a block device like a character device. It will work, but it will perform poorly.
>
> P.S. I've written device drivers for disks.
>
> P.P.S. The solution is simple, as I said earlier. Just do a flush after main() exits. It happens anyway - done by the C stdio subsystem - I just propose doing it in the D code before it hands things back to the C runtime. This will entail no performance degradation.

How about we test it before making any claims?

I wrote a simple test program that writes 4gigs of '!' to stdout, and measured it in 2 ways:

time ./a.out > /dev/null
time ./a.out > file.txt

Here are the test runs (a warmup followed by 3 runs of each):

Output to /dev/null W/O FLUSH
real	0m0.188s 0m0.186s 0m0.190s
user	0m0.166s 0m0.165s 0m0.168s
sys	0m0.021s 0m0.020s 0m0.021s

/dev/null WITH FLUSH
real	0m0.168s 0m0.166s 0m0.163s
user	0m0.145s 0m0.144s 0m0.142s
sys	0m0.022s 0m0.021s 0m0.021s

Output to file.txt W/O FLUSH
real	0m10.131s 0m11.766s 0m9.900s
user	0m0.444s 0m0.443s 0m0.451s
sys	0m5.352s 0m5.324s 0m5.308s

real	0m9.474s 0m9.971s 0m10.115s
user	0m0.233s 0m0.241s 0m0.233s
sys	0m5.588s 0m5.616s 0m5.591s

Does it look like it's slower? To me it looks like it's actually FASTER with a flush, although I don't know why. Try it yourself. The code that I tested with is below (apologies for using C).

#include <stdio.h>
#include <string.h>

int main()
{
    char buffer[4096];
    memset(buffer, '!', 4096);
    fwrite(buffer, 1, 100, stdout);
    //fflush(stdout);
    for (int i = 0; i < 1024*1024; ++i) {
      fwrite(buffer, 1, 4096, stdout);
    }
    return 0;
}

On Thursday, 13 June 2013 at 18:11:12 UTC, Walter Bright wrote:
>
> What is wrong and needs fixing is the program exiting with "success" indication when the output has actually failed.

What's wrong is a program existing with a status code *different* from one that was supplied by a programmer.

On 6/14/2013 1:11 AM, Denis Koroskin wrote:
> Does it look like it's slower? To me it looks like it's actually FASTER with a
> flush, although I don't know why.

Because of write caching, you can get very different results from one run to the next of the same program.

On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
> On 6/14/2013 1:11 AM, Denis Koroskin wrote:
>> Does it look like it's slower? To me it looks like it's actually FASTER with a
>> flush, although I don't know why.
>
> Because of write caching, you can get very different results from one run to the next of the same program.

I re-ran them a few times obviously. The results were consistent.

On Friday, 14 June 2013 at 08:24:54 UTC, Denis Koroskin wrote:
> On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
>> On 6/14/2013 1:11 AM, Denis Koroskin wrote:
>>> Does it look like it's slower? To me it looks like it's actually FASTER with a
>>> flush, although I don't know why.
>>
>> Because of write caching, you can get very different results from one run to the next of the same program.
>
> I re-ran them a few times obviously. The results were consistent.

That is super weird and unexpected to me, but I trust you.

On Fri, 14 Jun 2013 04:24:53 -0400, Denis Koroskin <2korden@gmail.com> wrote:

> On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
>> On 6/14/2013 1:11 AM, Denis Koroskin wrote:
>>> Does it look like it's slower? To me it looks like it's actually FASTER with a
>>> flush, although I don't know why.
>>
>> Because of write caching, you can get very different results from one run to the next of the same program.
>
> I re-ran them a few times obviously. The results were consistent.

Walter is right to a point -- caching can effect this behavior.

There are several levels of caching at play here, and all of them are working hard to optimize the writing to disk.  That likely is more of a cause than the flushing of the first element.

But in that sense, his argument is also broken -- you have to be in a very special situation (no caching, no drivers, etc.) in order for this to matter.

I had a similar test, and my results were not consistently one better than the other, but basically flushing had no discernible effect.

And quite honestly, if disk performance were dependent on user-land C buffering schemes, C runtime writers would pay more attention to the C buffering scheme and make sure it performs well!  i.e. if a single misaligned write can mess up gigabytes of output, then the next write would correct it.

-Steve