http://d.puremagic.com/issues/show_bug.cgi?id=3425

           Summary: std.stdio.File does not properly handle EOF from stdin
                    on Windows
           Product: D
           Version: 2.035
          Platform: Other
        OS/Version: Windows
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Phobos
        AssignedTo: nobody@puremagic.com
        ReportedBy: dsimcha@yahoo.com


--- Comment #0 from David Simcha <dsimcha@yahoo.com> 2009-10-20 11:55:53 PDT ---
Code:

import std.stdio, std.algorithm;

void main(string[] args) {
    auto foo = stdin.byLine();
    foreach(elem; foo) {
        writeln(elem);
    }
}

Input file (foo.txt) :
foo
bar

Result:

F:\>cat foo.txt | test2.exe
foo
bar
std.stdio.StdioException: Bad file descriptor

Works perfectly on Linux.  Haven't tested on Mac, BSD or whatever the heck else DMD runs on lately.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=3425


David Simcha <dsimcha@yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|std.stdio.File does not     |StdioException on end of
                   |properly handle EOF from    |stdin on Windows
                   |stdin on Windows            |
           Severity|normal                      |major


--- Comment #1 from David Simcha <dsimcha@yahoo.com> 2011-03-09 20:41:02 PST ---
I've managed to figure out where this exception is coming from, but I don't know how to fix it because I don't know why errno is getting set.  It's on line 2271 in stdio.d:

    if (fp._flag & _IONBF)
    {
        /* Use this for unbuffered I/O, when running
         * across buffer boundaries, or for any but the common
         * cases.
         */
      L1:
        auto app = appender(buf);
        app.clear();
        if(app.capacity == 0)
            app.reserve(128); // get at least 128 bytes available

        int c;
        while((c = FGETC(fp)) != -1) {
            app.put(cast(char) c);
            if(c == terminator) {
                buf = app.data;
                return buf.length;
            }

        }

        if (ferror(fps))
            StdioException();

Can we **PLEASE** fix this one ASAP?  It's been in Bugzilla for over a year and a half and makes it impossible to write simple text filter programs on Windows without ugly and/or unsafe workarounds.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=3425


Jay Norwood <jayn@prismnet.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jayn@prismnet.com


--- Comment #2 from Jay Norwood <jayn@prismnet.com> 2011-10-12 20:03:13 PDT ---
You can work around the issue by testing for eof  on the first line in the loop. This works with no error.

    foreach(elem; stdin.byLine()) {
        if (stdin.eof()) break;
        writeln(elem);
    }

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=3425


Max Vilimpoc <max@vilimpoc.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |max@vilimpoc.org


--- Comment #3 from Max Vilimpoc <max@vilimpoc.org> 2011-12-28 17:09:15 PST ---
(In reply to comment #2)
> You can work around the issue by testing for eof  on the first line in the loop. This works with no error.
> 
>     foreach(elem; stdin.byLine()) {
>         if (stdin.eof()) break;
>         writeln(elem);
>     }

This didn't work for me, when using the example program on the D homepage:

    import std.stdio;

    void main()
    {
        ulong lines = 0;
        double sumLength = 0;
        foreach(line; stdin.byLine())
        {
            if (stdin.eof()) break;

            ++lines;
            sumLength += line.length;
        }

        writeln("Average line length = ", lines ? sumLength / lines : 0);
    }

I still get:

    std.stdio.StdioException@std\stdio.d(2159): Bad file descriptor

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=3425



--- Comment #4 from Max Vilimpoc <max@vilimpoc.org> 2011-12-28 17:17:14 PST ---
However, I did find that if I modify stdio.d in Phobos to the following under the L1 label around line 2280, then the other form of piping, i.e. "type filename | executable", will work on Windows.

    if (ferror(fps) && EPIPE != ferror(fps))
        StdioException();

In the "|" pipe case, it could be that perhaps the "type" command already closed down its end by the time ferror() was called on the receiving side.

On Windows, "executable < filename" piping worked fine, but is not ideal for stringing filters together.

Here's the writeln-debugged block of stdio.d code that works:

// Private implementation of readln
version (DIGITAL_MARS_STDIO)
private size_t readlnImpl(FILE* fps, ref char[] buf, dchar terminator = '\n')
{
    FLOCK(fps);
    scope(exit) FUNLOCK(fps);

    writeln("Entered readlnImpl()");
    scope(exit) writeln("Exited readlnImpl()");

    /* Since fps is now locked, we can create an "unshared" version
     * of fp.
     */
    auto fp = cast(_iobuf*)fps;

    if (__fhnd_info[fp._file] & FHND_WCHAR)
    {
        /* Stream is in wide characters.
         * Read them and convert to chars.
         */
        writeln("Entered if (__fhnd_info[fp._file] & FHND_WCHAR)");

        static assert(wchar_t.sizeof == 2);
        auto app = appender(buf);
        app.clear();
        for (int c = void; (c = FGETWC(fp)) != -1; )
        {
            if ((c & ~0x7F) == 0)
            {   app.put(cast(char) c);
                if (c == terminator)
                    break;
            }
            else
            {
                if (c >= 0xD800 && c <= 0xDBFF)
                {
                    int c2 = void;
                    if ((c2 = FGETWC(fp)) != -1 ||
                            c2 < 0xDC00 && c2 > 0xDFFF)
                    {
                        StdioException("unpaired UTF-16 surrogate");
                    }
                    c = ((c - 0xD7C0) << 10) + (c2 - 0xDC00);
                }
                //std.utf.encode(buf, c);
                app.put(cast(dchar)c);
            }
        }
        if (ferror(fps))
            StdioException();
        buf = app.data;
        return buf.length;
    }

    auto sz = GC.sizeOf(buf.ptr);
    //auto sz = buf.length;
    buf = buf.ptr[0 .. sz];
    if (fp._flag & _IONBF)
    {
        writeln("Entered if (fp._flag & _IONBF)");

        /* Use this for unbuffered I/O, when running
         * across buffer boundaries, or for any but the common
         * cases.
         */
      L1:
        auto app = appender(buf);
        app.clear();
        if(app.capacity == 0)
            app.reserve(128); // get at least 128 bytes available

        int c;
        writeln("fp._cnt: ", fp._cnt);

        while((c = FGETC(fp)) != -1) {
            writeln("chars: ", cast(char) c);
            app.put(cast(char) c);
            if(c == terminator) {
                writeln("hit terminator");
                buf = app.data;
                return buf.length;
            }
        }

        writeln("feof(fps): ", feof(fps));
        writeln("ferror(fps): ", ferror(fps));

        // If EPIPE is seen then probably the other side has closed
        // already. This is the case when using, for example:
        //
        // "type filename | D-program" syntax on Windows.
        if (ferror(fps) && EPIPE != ferror(fps))
            StdioException();

        buf = app.data;
        return buf.length;
    }
    else
    {
        writeln("Entered if (!(fp._flag & _IONBF))");

        int u = fp._cnt;
        char* p = fp._ptr;
        int i;

        writeln("length of stdin fp: ", u);

        if (fp._flag & _IOTRAN)
        {   /* Translated mode ignores \r and treats ^Z as end-of-file
             */
            writeln("Entered if (fp._flag & _IOTRAN)");

            char c;
            while (1)
            {
                if (i == u)                // if end of buffer
                    goto L1;        // give up
                c = p[i];
                i++;
                if (c != '\r')
                {
                    if (c == terminator)
                        break;
                    if (c != 0x1A)
                        continue;
                    goto L1;
                }
                else
                {   if (i != u && p[i] == terminator)
                        break;
                    goto L1;
                }
            }
            if (i > sz)
            {
                buf = uninitializedArray!(char[])(i);
            }
            if (i - 1)
                memcpy(buf.ptr, p, i - 1);
            buf[i - 1] = cast(char)terminator;
            buf = buf[0 .. i];
            if (terminator == '\n' && c == '\r')
                i++;
        }
        else
        {
            writeln("Entered if !(fp._flag & _IOTRAN)");

            while (1)
            {
                if (i == u)                // if end of buffer
                    goto L1;        // give up
                auto c = p[i];
                i++;
                if (c == terminator)
                    break;
            }
            if (i > sz)
            {
                buf = uninitializedArray!(char[])(i);
            }
            memcpy(buf.ptr, p, i);
            buf = buf[0 .. i];
        }
        fp._cnt -= i;
        fp._ptr += i;
        return i;
    }
}

And here's what it outputs:

c:\>avg < index.html

Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
chars: <
chars: h
chars: t
chars: m
chars: l
chars: >
chars:

hit terminator
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 106
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 97
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 37
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 27
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 18
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 8
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
feof(fps): 16
ferror(fps): 0
Exited readlnImpl()
Average line length = 15.1429

c:\>type index.html | avg
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
chars: <
chars: h
chars: t
chars: m
chars: l
chars: >
chars:

hit terminator
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 106
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 97
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 37
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 27
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 18
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 8
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
feof(fps): 0
ferror(fps): 32
Exited readlnImpl()
Average line length = 15.1429

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------