On Mon, 2010-05-03 at 18:54 -0700, Steve Schveighoffer wrote:
> Everyone,
> 
> I worked on getting Lars' std.process compiled for Windows.  However, I've reached a very big blocker.
> 
> Basically, the digital mars C runtime library has no way to create pipes into file descriptors.  Therefore, you cannot redirect a child's output to the parent or vice versa.

This is not really related to your question, but your wording makes me curious:  Is D on Windows somehow tied to DMC?  Is that different to standard C?


> I can create pipes by directly calling the Windows system call, the problem is, the C runtime lacks a way of wrapping a Windows HANDLE type into a FILE *. [...]

You'd think that would be a common enough situation to warrant having a function in the Windows API doing just that -- like POSIX' fdopen().  I mean, isn't FILE* the de facto file handle standard in the C world?

-Lars

Oops.. sorry for the out-of-context e-mail, everyone.  I thought the message to which I was replying was sent to the Phobos list, but I see now it wasn't.

-Lars



On Tue, 2010-05-04 at 11:34 +0200, Lars Tandle Kyllingstad wrote:
> On Mon, 2010-05-03 at 18:54 -0700, Steve Schveighoffer wrote:
> > Everyone,
> > 
> > I worked on getting Lars' std.process compiled for Windows.  However, I've reached a very big blocker.
> > 
> > Basically, the digital mars C runtime library has no way to create pipes into file descriptors.  Therefore, you cannot redirect a child's output to the parent or vice versa.
> 
> This is not really related to your question, but your wording makes me curious:  Is D on Windows somehow tied to DMC?  Is that different to standard C?
> 
> 
> > I can create pipes by directly calling the Windows system call, the problem is, the C runtime lacks a way of wrapping a Windows HANDLE type into a FILE *. [...]
> 
> You'd think that would be a common enough situation to warrant having a function in the Windows API doing just that -- like POSIX' fdopen().  I mean, isn't FILE* the de facto file handle standard in the C world?
> 
> -Lars
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Lars Tandle Kyllingstad wrote:
> On Mon, 2010-05-03 at 18:54 -0700, Steve Schveighoffer wrote:
> 
>> Everyone,
>>
>> I worked on getting Lars' std.process compiled for Windows.  However, I've reached a very big blocker.
>>
>> Basically, the digital mars C runtime library has no way to create pipes into file descriptors.  Therefore, you cannot redirect a child's output to the parent or vice versa.
>> 
>
> This is not really related to your question, but your wording makes me curious:  Is D on Windows somehow tied to DMC?

Yes. It has to be tied to a C compiler.

> Is that different to standard C?
> 

No. Standard C does not cover an ABI.

> 
>> I can create pipes by directly calling the Windows system call, the problem is, the C runtime lacks a way of wrapping a Windows HANDLE type into a FILE *. [...]
>> 
>
> You'd think that would be a common enough situation to warrant having a function in the Windows API doing just that -- like POSIX' fdopen().  I mean, isn't FILE* the de facto file handle standard in the C world?
>
> 

Pipes aren't standard C <g>.

>
>  No. Standard C does not cover an ABI.


I realize that there is no *de jure* standard C ABI, but I thought on most operating systems/CPU architecture combinations there is a *de facto* one.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20100504/a8481433/attachment.html>

David Simcha wrote:
>
>     No. Standard C does not cover an ABI.
>
>
> I realize that there is no /de jure/ standard C ABI, but I thought on most operating systems/CPU architecture combinations there is a /de facto/ one.

There isn't one on Windows.

Even on the Linux/FreeBSD/OSX ports, you'll see a lot of variation. Heck, even the change from OSX 10.4 to 10.5 broke binary compatibility, then again from 10.5 to 10.6.

We try to hide a lot of this from D users of Phobos, but it's there in the implementation of Phobos and the code generated by the compiler.

On Tue, 2010-05-04 at 13:28 -0700, Walter Bright wrote:
> Lars Tandle Kyllingstad wrote:
> > On Mon, 2010-05-03 at 18:54 -0700, Steve Schveighoffer wrote:
> >> I can create pipes by directly calling the Windows system call, the problem is, the C runtime lacks a way of wrapping a Windows HANDLE type into a FILE *. [...]
> >> 
> >
> > You'd think that would be a common enough situation to warrant having a function in the Windows API doing just that -- like POSIX' fdopen().  I mean, isn't FILE* the de facto file handle standard in the C world?
> 
> Pipes aren't standard C <g>.

I didn't mean the pipe creation functions. :)

On POSIX you have fdopen() to wrap a FILE* handle around an OS file descriptor, and fileno() to extract an OS file descriptor from a FILE* handle.  These are part of the POSIX specification, and therefore available regardless of which runtime is being used.

If I've understood Steve correctly, the same isn't true on Windows. Rather, it seems Microsoft have tucked the corresponding functions away in their own C runtime. :(

-Lars

http://msdn.microsoft.com/en-us/library/ms235401%28VS.80%29.aspx http://msdn.microsoft.com/en-us/library/ms235351%28VS.80%29.aspx

Andrei

Lars Tandle Kyllingstad wrote:
> On Tue, 2010-05-04 at 13:28 -0700, Walter Bright wrote:
>> Lars Tandle Kyllingstad wrote:
>>> On Mon, 2010-05-03 at 18:54 -0700, Steve Schveighoffer wrote:
>>>> I can create pipes by directly calling the Windows system call, the problem is, the C runtime lacks a way of wrapping a Windows HANDLE type into a FILE *. [...]
>>>> 
>>> You'd think that would be a common enough situation to warrant having a function in the Windows API doing just that -- like POSIX' fdopen().  I mean, isn't FILE* the de facto file handle standard in the C world?
>> Pipes aren't standard C <g>.
> 
> I didn't mean the pipe creation functions. :)
> 
> On POSIX you have fdopen() to wrap a FILE* handle around an OS file descriptor, and fileno() to extract an OS file descriptor from a FILE* handle.  These are part of the POSIX specification, and therefore available regardless of which runtime is being used.
> 
> If I've understood Steve correctly, the same isn't true on Windows. Rather, it seems Microsoft have tucked the corresponding functions away in their own C runtime. :(
> 
> -Lars
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

First, those functions are unavailable to D programs, because that is the microsoft runtime, D uses the DMC runtime (DMC does have equivalent functions for those).  Second, getting the file descriptor, or creating a FILE * from a file descriptor, are not good enough.

To create a pipe, I need to use CreatePipe on Windows, or else a C-library wrapper (on Microsoft, it's _pipe).

The problem is, CreatePipe returns HANDLEs, which are not file descriptors.  So I need a way to transform HANDLEs into file descriptors.  With Microsoft's runtime there are two functions:

http://msdn.microsoft.com/en-us/library/ks2530z6%28VS.71%29.aspx http://msdn.microsoft.com/en-us/library/bdts1c9x%28VS.71%29.aspx

With DMC, there are no such functions (including a _pipe equivalent).  However, Walter has showed me some internal DMC runtime code, which I can port to D to get it to work properly.

But it does raise the concern -- we are depending on 20-25 year old code that may not allow us to write to newer OS APIs, and which leaves some aspects of I/O (arguably one of the most important performance-oriented parts of a standard lib) closed to innovation.  I think eventually, it would be best for phobos to ditch dependence on FILE *, and I would argue, the C runtime in general.  Of course, the C runtime should be *available* to people who want to use it, but Phobos shouldn't use it.

For a good example of why this absolutely sucks, see what the original author of std.process had to do to get shell() to work on Windows:

version (Windows) string shell(string cmd)
{
    // Generate a random filename
    Appender!string a;
    foreach (ref e; 0 .. 8)
    {
        formattedWrite(a, "%x", rndGen.front);
        rndGen.popFront;
    }
    auto filename = a.data;
    scope(exit) if (exists(filename)) remove(filename);
    errnoEnforce(system(cmd ~ "> " ~ filename) == 0);
    return readText(filename);
}

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com> http://msdn.microsoft.com/en-us/library/ms235401%28VS.80%29.aspx
http://msdn.microsoft.com/en-us/library/ms235351%28VS.80%29.aspx

Andrei

Lars
> Tandle Kyllingstad wrote:
> On Tue, 2010-05-04 at 13:28 -0700, Walter
> Bright wrote:
>> Lars Tandle Kyllingstad wrote:
>>> On Mon,
> 2010-05-03 at 18:54 -0700, Steve Schveighoffer wrote:
>>>> I can
> create pipes by directly calling the Windows system call, the problem is, the C runtime lacks a way of wrapping a Windows HANDLE type into a FILE *. [...]
>>>> 
>>> You'd think that would
> be a common enough situation to warrant having a
>>> function in the
> Windows API doing just that -- like POSIX' fdopen().  I
>>> 
> mean, isn't FILE* the de facto file handle standard in the C world?
>> 
> Pipes aren't standard C <g>.
> 
> I didn't mean the pipe creation functions. :)
> 
> On POSIX you have fdopen() to wrap a
> FILE* handle around an OS file
> descriptor, and fileno() to extract an OS
> file descriptor from a FILE*
> handle.  These are part of the POSIX
> specification, and therefore
> available regardless of which runtime is
> being used.
> 
> If I've understood Steve correctly, the same isn't
> true on Windows.
> Rather, it seems Microsoft have tucked the
> corresponding functions away
> in their own C runtime. :(
> 
> 
> -Lars
> 
> _______________________________________________
> 
> phobos mailing list
> 
> href="mailto:phobos at puremagic.com">phobos at puremagic.com
> 
> http://lists.puremagic.com/mailman/listinfo/phobos
_______________________________________________
phobos
> mailing list

> href="mailto:phobos at puremagic.com">phobos at puremagic.com

> href="http://lists.puremagic.com/mailman/listinfo/phobos" target=_blank
> >http://lists.puremagic.com/mailman/listinfo/phobos