On Mon, 26 Sep 2011 18:10:05 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On 9/26/2011 2:53 PM, Steven Schveighoffer wrote:
>> I think the reason Walter always gives is that dmd would then have to output COFF objects, whereas it currently outputs OMF.  OPTLINK also does not accept COFF objects.
> Wait, why does it have to output COFF? Can't you just use an OMF version of msvcrt.lib, like we're doing for all the other Windows libraries?
> I'm confused as to how this is an issue.

I don't know, maybe there's another reason.

>
>> I think a lot of people would very much welcome this change, but it's somewhat of an underwhelming change.  Instead of using DMC's runtime for C functions, you are using VC++'s runtime.  But who cares?  I want D's runtime to be better :)
> Well, sort of. It's not so much Visual C++'s runtime, but it's Microsoft's C runtime that's packaged with Windows (which Microsoft ironically doesn't want you to use, but which compilers like GCC use anyway... and it turns out better than anything bundled with Visual Studio). You don't actually need Visual Studio to use it (in fact, it's not even included) -- the Windows Driver Kit has the bundled stuff required.

It's the same thing.  MSVCRT.dll is included with every windows.  Just the version may be different from the runtime bundled with Visual Studio.  That's why you can install the visual studio redistributable package.

The lib file (which contains the linking information) is not included (with Windows).  Neither is the statically linked version.

But make no mistake, it's the Visual C++ runtime (MicroSoftVisualCRunTime).

>
> In all honesty I don't see a single thing about it that I would call "better". Do you have any examples of things that snn.lib does 'better' than msvcrt.lib?

I think it may conform to the standard better.  But I don't think any C runtime is better than what we could make in D :)

> The only incompatibility (which isn't "better" or "worse"... it's just an incompatibility) I can see is exception handling, but that's obviously an exception that can be left in snn.lib. Almost all the rest of the runtime, though, can be easily removed and redirected to msvcrt.dll... and I don't think there'd be any problems. (Are there?)

I don't know.  For certain, there are some pieces of phobos written specifically knowing the internals of Digital Mars' runtime (like std.stdio).

>> I think the best path is weaning ourselves off of C dependencies.  Starting with FILE *, which has a lot of limitations.  It's part of the reason I'm working on reworking std.stdio.
>> For example, with my version of std.stdio, you would never have this problem, because it deals with the HANDLE directly instead of through DMC's file descriptor layer.
> 100% agree, but we weren't talking about D in the first place. :) This is all regarding what's /already/ taking place in snn.lib /before/ D runs, which is obviously unrelated to what's going on in std.stdio.
>
> I'm really curious to know what specific problems are keeping us from doing the switch (i.e. what snn.lib does better than msvcrt.dll). :)

I'm not really qualified to answer that.  It seems like the right move to me, so there must be good reasons compiler-wise.

-Steve

On Mon, 26 Sep 2011 18:36:16 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On 9/26/2011 2:58 PM, Steven Schveighoffer wrote:
>> This is likely a DMC issue, and is probably best reported as a DMC bug.  I don't know how much more you want to pursue this, but the next steps I'd recommend are:
>>
>> 1. obtain the dmc compiler (it's free) if you haven't already.
>> 2. Compile an empty C program and run it using SciTE and Notepad++.  Verify the same error occurs
>> 3. report the failure using DMC's bugzilla.
>>
>> I'm guessing there's something in the way SciTE or Notepad++ sets up the pipes before executing the process which causes the problem.  I have a sinking feeling we'll see more of this issue when the new std.process is released :(
>>
>> -Steve
> OK, I think I found the bug, and I'm pretty sure it's what I was saying originally:
>
> A handle is being closed that isn't actually valid.
>
> Which handle? stderr.  stderr is being closed when in fact it is the same handle as stdout. So when _fcloseallp() is called, it goes through the structures in __iob and closes each one in turn -- which causes a duplicate close operation on stderr, because it already gets closed when stdout gets closed.
>
> So it's an snn.lib issue. Is it easy to fix? (I hope it is...)

Probably -- let the OS close the handles instead of manually closing them :)

-Steve

On 27.09.2011 2:33, Jonathan M Davis wrote:
> On Monday, September 26, 2011 14:53 Steven Schveighoffer wrote:
>> On Mon, 26 Sep 2011 17:35:20 -0400, Mehrdad<wfunction@hotmail.com>  wrote:
>>> On 9/26/2011 2:30 PM, Mehrdad wrote:
>>>> On 9/26/2011 1:51 PM, Steven Schveighoffer wrote:
>>>>> AHA!
>>>>>
>>>>> Yes, there is a bug in snn.lib regarding pipes. And I fixed it,
>>>>> waiting for Walter to incorporate it :) I needed it for the new
>>>>> std.process.
>>>>>
>>>>> What it comes down to is, DMC's FILE * implementation does not expect
>>>>> some of the quirks of pipe HANDLEs. For instance, if you open a FILE
>>>>> * around a pipe handle, it still tries to do a seek on that handle,
>>>>> and crashes. Also, when the write end of a pipe is closed, reading
>>>>> from the read end results in EPIPE from ReadFile, but this is
>>>>> translated to EBADF by the runtime. Therefore, FILE * sets an error
>>>>> instead of EOF.
>>>>>
>>>>> Is the email address you have for this message correct? If so, I can
>>>>> send you a new version of snn.lib to try linking your code against (if
>>>>> you are willing to go through these steps), to see if it fixes your
>>>>> problem.
>>>>>
>>>>> Using the command line dmd to build should be sufficient (I think).
>>>>>
>>>>> -Steve
>>>>
>>>> Thanks for the email.
>>>>
>>>> It seems like it still crashes from SciTE, though. :(
>>>> When I try debugging, I see it's doing so inside _close() which is
>>>> /immediately/ above kernel32.dll!@BaseThreadInitThunk@12() -- in other
>>>> words, it seems to be called directly from the entrypoint of the
>>>> program, assuming it hasn't undergone optimizations. The error is:
>>>> 0xC0000008: An invalid handle was specified.
>>>>
>>>> Is there any other place in which this can happen?
>>>
>>> Also, now that we're on the topic... this might or might not sound
>>> silly, but why not just use msvcrt.dll (if not the whole library, at
>>> least the I/O functions)? It's not like it introduces a new dependency
>>> (it's already in every version of Windows) and it's also proven to work.
>>> Not to mention that it reduces code size as well...
>>
>> I think the reason Walter always gives is that dmd would then have to
>> output COFF objects, whereas it currently outputs OMF. OPTLINK also does
>> not accept COFF objects.
>>
>> I think a lot of people would very much welcome this change, but it's
>> somewhat of an underwhelming change. Instead of using DMC's runtime for C
>> functions, you are using VC++'s runtime. But who cares? I want D's
>> runtime to be better :)
>
> The gain in switching to COFF would be _huge_, because then D code would be
> properly interoperable with most of the C code on Windows - and in particular
> C code compiled with Microsoft's compiler. The only thing saving the current
> situation from being a complete disaster is that you can still use dlls.
>

It's not that easy. I mean to gain this level of compatibility we'd have to do both: switch to MS runtime and use COFF object format. Linking together two C runtimes is *ehm* tricky.

But most importantly newer versions of VC++ use newer runtimes, I'm not sure how compatible they are, but judging by my previous experience - you still has to keep things separated like in DLLs and never mix allocation/FILE* stuff between them.


> - Jonathan M Davis


-- 
Dmitry Olshansky

On 9/26/2011 3:41 PM, Steven Schveighoffer wrote:
> Probably -- let the OS close the handles instead of manually closing them :)
>
> -Steve
I don't think that's a good idea because snn.lib likely does its own internal buffering of the data, which would be lost if the stream isn't closed.

On 9/26/2011 3:37 PM, Steven Schveighoffer wrote:
> On Mon, 26 Sep 2011 18:17:49 -0400, Mehrdad <wfunction@hotmail.com> wrote:
>
>> On 9/26/2011 2:58 PM, Steven Schveighoffer wrote:
>>> This is likely a DMC issue, and is probably best reported as a DMC bug.  I don't know how much more you want to pursue this, but the next steps I'd recommend are:
>>>
>>> 1. obtain the dmc compiler (it's free) if you haven't already.
>>> 2. Compile an empty C program and run it using SciTE and Notepad++.  Verify the same error occurs
>>> 3. report the failure using DMC's bugzilla.
>>>
>>> I'm guessing there's something in the way SciTE or Notepad++ sets up the pipes before executing the process which causes the problem.  I have a sinking feeling we'll see more of this issue when the new std.process is released :(
>>>
>>> -Steve
>> I'm suspecting it's /not/ a DMC issue (it's probably just an improperly used handle in snn.lib), because I don't see how it could affect my D program this way, if it was a DMC issue.
>> And yeah, I just tested it with a simple hello world C example using DMC as well, and got the same error. Pretty darn sure now that the handle is getting closed twice or something...
>
> My apologies, I meant DMC runtime issue (whose library is called snn.lib).
>
> At least with this test, we can rule out D being the culprit.  However, we still have to deal with this problem...
>
> -Steve
Ahhh ok I see. If I'm in the mood sometime I'll post it somewhere DMC-related as well. :)

On Monday, September 26, 2011 16:07 Dmitry Olshansky wrote:
> On 27.09.2011 2:33, Jonathan M Davis wrote:
> > On Monday, September 26, 2011 14:53 Steven Schveighoffer wrote:
> >> On Mon, 26 Sep 2011 17:35:20 -0400, Mehrdad<wfunction@hotmail.com>
wrote:
> >>> On 9/26/2011 2:30 PM, Mehrdad wrote:
> >>>> On 9/26/2011 1:51 PM, Steven Schveighoffer wrote:
> >>>>> AHA!
> >>>>> 
> >>>>> Yes, there is a bug in snn.lib regarding pipes. And I fixed it, waiting for Walter to incorporate it :) I needed it for the new std.process.
> >>>>> 
> >>>>> What it comes down to is, DMC's FILE * implementation does not expect some of the quirks of pipe HANDLEs. For instance, if you open a FILE * around a pipe handle, it still tries to do a seek on that handle, and crashes. Also, when the write end of a pipe is closed, reading from the read end results in EPIPE from ReadFile, but this is translated to EBADF by the runtime. Therefore, FILE * sets an error instead of EOF.
> >>>>> 
> >>>>> Is the email address you have for this message correct? If so, I can send you a new version of snn.lib to try linking your code against (if you are willing to go through these steps), to see if it fixes your problem.
> >>>>> 
> >>>>> Using the command line dmd to build should be sufficient (I think).
> >>>>> 
> >>>>> -Steve
> >>>> 
> >>>> Thanks for the email.
> >>>> 
> >>>> It seems like it still crashes from SciTE, though. :(
> >>>> When I try debugging, I see it's doing so inside _close() which is
> >>>> /immediately/ above kernel32.dll!@BaseThreadInitThunk@12() -- in other
> >>>> words, it seems to be called directly from the entrypoint of the
> >>>> program, assuming it hasn't undergone optimizations. The error is:
> >>>> 0xC0000008: An invalid handle was specified.
> >>>> 
> >>>> Is there any other place in which this can happen?
> >>> 
> >>> Also, now that we're on the topic... this might or might not sound silly, but why not just use msvcrt.dll (if not the whole library, at least the I/O functions)? It's not like it introduces a new dependency (it's already in every version of Windows) and it's also proven to work. Not to mention that it reduces code size as well...
> >> 
> >> I think the reason Walter always gives is that dmd would then have to output COFF objects, whereas it currently outputs OMF. OPTLINK also does not accept COFF objects.
> >> 
> >> I think a lot of people would very much welcome this change, but it's somewhat of an underwhelming change. Instead of using DMC's runtime for C functions, you are using VC++'s runtime. But who cares? I want D's runtime to be better :)
> > 
> > The gain in switching to COFF would be _huge_, because then D code would be properly interoperable with most of the C code on Windows - and in particular C code compiled with Microsoft's compiler. The only thing saving the current situation from being a complete disaster is that you can still use dlls.
> 
> It's not that easy. I mean to gain this level of compatibility we'd have to do both: switch to MS runtime and use COFF object format. Linking together two C runtimes is *ehm* tricky.
> 
> But most importantly newer versions of VC++ use newer runtimes, I'm not sure how compatible they are, but judging by my previous experience - you still has to keep things separated like in DLLs and never mix allocation/FILE* stuff between them.

That may be, but the fact that I can't just link C code built with Visual Studio with D code is a major problem in enviroments where Visual Studio is what is used on Windows (such as where I work). In comparison, in Linux it works, because dmd actually uses the compiler which is the de facto standard on Linux - gcc. But since dmd uses optlink on Windows, you just don't have that level of compatibility. dmc and optlink may very well be solid tools, but as a D programmer trying to interoperate with C code, compatability matters a lot, and dmd on Windows just doesn't have it.

- Jonathan M Davis

On 9/26/2011 3:38 PM, Steven Schveighoffer wrote:
> On Mon, 26 Sep 2011 18:10:05 -0400, Mehrdad <wfunction@hotmail.com> wrote:
>
>> On 9/26/2011 2:53 PM, Steven Schveighoffer wrote:
>>> I think the reason Walter always gives is that dmd would then have to output COFF objects, whereas it currently outputs OMF.  OPTLINK also does not accept COFF objects.
>> Wait, why does it have to output COFF? Can't you just use an OMF version of msvcrt.lib, like we're doing for all the other Windows libraries?
>> I'm confused as to how this is an issue.
>
> I don't know, maybe there's another reason.
>
>>
>>> I think a lot of people would very much welcome this change, but it's somewhat of an underwhelming change.  Instead of using DMC's runtime for C functions, you are using VC++'s runtime.  But who cares?  I want D's runtime to be better :)
>> Well, sort of. It's not so much Visual C++'s runtime, but it's Microsoft's C runtime that's packaged with Windows (which Microsoft ironically doesn't want you to use, but which compilers like GCC use anyway... and it turns out better than anything bundled with Visual Studio). You don't actually need Visual Studio to use it (in fact, it's not even included) -- the Windows Driver Kit has the bundled stuff required.
>
> It's the same thing.  MSVCRT.dll is included with every windows.  Just the version may be different from the runtime bundled with Visual Studio.  That's why you can install the visual studio redistributable package.
>
> The lib file (which contains the linking information) is not included (with Windows).  Neither is the statically linked version.
>
> But make no mistake, it's the Visual C++ runtime (MicroSoftVisualCRunTime).
>
>>
>> In all honesty I don't see a single thing about it that I would call "better". Do you have any examples of things that snn.lib does 'better' than msvcrt.lib?
>
> I think it may conform to the standard better.  But I don't think any C runtime is better than what we could make in D :)
>
>> The only incompatibility (which isn't "better" or "worse"... it's just an incompatibility) I can see is exception handling, but that's obviously an exception that can be left in snn.lib. Almost all the rest of the runtime, though, can be easily removed and redirected to msvcrt.dll... and I don't think there'd be any problems. (Are there?)
>
> I don't know.  For certain, there are some pieces of phobos written specifically knowing the internals of Digital Mars' runtime (like std.stdio).
>
>>> I think the best path is weaning ourselves off of C dependencies.  Starting with FILE *, which has a lot of limitations.  It's part of the reason I'm working on reworking std.stdio.
>>> For example, with my version of std.stdio, you would never have this problem, because it deals with the HANDLE directly instead of through DMC's file descriptor layer.
>> 100% agree, but we weren't talking about D in the first place. :) This is all regarding what's /already/ taking place in snn.lib /before/ D runs, which is obviously unrelated to what's going on in std.stdio.
>>
>> I'm really curious to know what specific problems are keeping us from doing the switch (i.e. what snn.lib does better than msvcrt.dll). :)
>
> I'm not really qualified to answer that.  It seems like the right move to me, so there must be good reasons compiler-wise.
>
> -Steve

I hope this isn't too huge of a request (Walter?), but would it be possible to PLEASE release an object or library file which would be helpful for implementing the use of msvcrt.lib?

Right now, most crucial thing I need is an object file which defines this symbol correctly (since it seems to be providing an anchor for the rest runtime and the GC):

phobos.lib(memory) Error 42: Symbol Undefined __xi_a

Having something that also implements the following would also be awesome, but I can't tell if it's as crucial as the above until I have the above and try it out first:

Test.obj(Test) Error 42: Symbol Undefined __except_list
phobos.lib(deh) Error 42: Symbol Undefined __tls_array
phobos.lib(deh) Error 42: Symbol Undefined __tls_index
phobos.lib(thread) Error 42: Symbol Undefined __tlsend
phobos.lib(thread) Error 42: Symbol Undefined __tlsstart
phobos.lib(memory) Error 42: Symbol Undefined __end

If I could _only_ have these, then I could very well try to see if I can integrate the C runtime with msvcrt.dll, which I think many people would appreciate.

I'm pretty sure that the definition of __xi_a itself must be incredibly small (maybe 2 lines of code total?) so I hope it's not too much trouble to release something about it... I have no idea how to define it myself. If a working definition was released then it'd be pretty awesome.

Thank you!

On 26.09.2011 16:55, Mehrdad wrote:
> DMD 2.055 seems to be crashing on my computer (Windows 7 x64).
>
> When I press Debug, I see it's doing so in CloseHandle, probably because
> I've turned on "Enable close exception" with Visual Studio's "Global
> Flags" tool. (It takes a reboot to take effect.)
>
> So this means there's an incorrect handle being used somewhere... but
> interestingly, by the time it crashes, the output file is already
> produced. I don't have the time to make a debug build right now, but I
> thought I'd mention this so that it could be looked into at some point.

That's interesting. I recently found another bug, which is also Windows7 specific.
http://d.puremagic.com/issues/show_bug.cgi?id=6660
Although I assumed that it's stack corruption, it could also be the same handle issue.

I think, http://d.puremagic.com/issues/show_bug.cgi?id=1550 is very much related.

IIRC, the handle for stdin, stdout and stderr are retreived by GetStdHandle(), which should not be closed at all. Definitely should be fixed in snn.lib.


On 27.09.2011 01:15, Mehrdad wrote:
> On 9/26/2011 3:41 PM, Steven Schveighoffer wrote:
>> Probably -- let the OS close the handles instead of manually closing
>> them :)
>>
>> -Steve
> I don't think that's a good idea because snn.lib likely does its own
> internal buffering of the data, which would be lost if the stream isn't
> closed.

Mehrdad Wrote:

> Also, now that we're on the topic... this might or might not sound silly, but why not just use msvcrt.dll (if not the whole library, at least the I/O functions)? It's not like it introduces a new dependency (it's already in every version of Windows) and it's also proven to work. Not to mention that it reduces code size as well...

gdc already does it.