On 9/26/2011 2:36 PM, Steven Schveighoffer wrote:
> Hm... its hard for me to say.  Why would it be calling _close before main() is called?  Very strange.  Is there any more stack information?
>
> Can you get it to crash by using a pipe from the command line?
>
> In other words:
>
> myprogram | more
>
> or echo hi | miprogram
>
> ?
>
> If this isn't killing the program in the same way, then it might be a different issue than the one I fixed.
>
> -Steve

Nope, pipes/redirections from the command prompt are all fine -- it just crashes when run from SciTE (and also Notepad++, which I just verified).

After somewhat painful debugging, it seems like the stack trace is a little off -- it actually happens when _exit() is called, which in turn calls _fcloseall() or something.

Not sure what else I can do about it, though -- it's pretty clear the handle was destroyed sometime before, but I don't know when.

On Mon, 26 Sep 2011 17:35:20 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On 9/26/2011 2:30 PM, Mehrdad wrote:
>> On 9/26/2011 1:51 PM, Steven Schveighoffer wrote:
>>> AHA!
>>>
>>> Yes, there is a bug in snn.lib regarding pipes.  And I fixed it, waiting for Walter to incorporate it :)  I needed it for the new std.process.
>>>
>>> What it comes down to is, DMC's FILE * implementation does not expect some of the quirks of pipe HANDLEs.  For instance, if you open a FILE * around a pipe handle, it still tries to do a seek on that handle, and crashes.  Also, when the write end of a pipe is closed, reading from the read end results in EPIPE from ReadFile, but this is translated to EBADF by the runtime.  Therefore, FILE * sets an error instead of EOF.
>>>
>>> Is the email address you have for this message correct?  If so, I can send you a new version of snn.lib to try linking your code against (if you are willing to go through these steps), to see if it fixes your problem.
>>>
>>> Using the command line dmd to build should be sufficient (I think).
>>>
>>> -Steve
>> Thanks for the email.
>>
>> It seems like it still crashes from SciTE, though. :(
>> When I try debugging, I see it's doing so inside _close() which is /immediately/ above kernel32.dll!@BaseThreadInitThunk@12()  -- in other words, it seems to be called directly from the entrypoint of the program, assuming it hasn't undergone optimizations. The error is: 0xC0000008: An invalid handle was specified.
>>
>> Is there any other place in which this can happen?
> Also, now that we're on the topic... this might or might not sound silly, but why not just use msvcrt.dll (if not the whole library, at least the I/O functions)? It's not like it introduces a new dependency (it's already in every version of Windows) and it's also proven to work. Not to mention that it reduces code size as well...

I think the reason Walter always gives is that dmd would then have to output COFF objects, whereas it currently outputs OMF.  OPTLINK also does not accept COFF objects.

I think a lot of people would very much welcome this change, but it's somewhat of an underwhelming change.  Instead of using DMC's runtime for C functions, you are using VC++'s runtime.  But who cares?  I want D's runtime to be better :)

I think the best path is weaning ourselves off of C dependencies.  Starting with FILE *, which has a lot of limitations.  It's part of the reason I'm working on reworking std.stdio.

For example, with my version of std.stdio, you would never have this problem, because it deals with the HANDLE directly instead of through DMC's file descriptor layer.

-Steve

On Mon, 26 Sep 2011 17:52:02 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On 9/26/2011 2:36 PM, Steven Schveighoffer wrote:
>> Hm... its hard for me to say.  Why would it be calling _close before main() is called?  Very strange.  Is there any more stack information?
>>
>> Can you get it to crash by using a pipe from the command line?
>>
>> In other words:
>>
>> myprogram | more
>>
>> or echo hi | miprogram
>>
>> ?
>>
>> If this isn't killing the program in the same way, then it might be a different issue than the one I fixed.
>>
>> -Steve
>
> Nope, pipes/redirections from the command prompt are all fine -- it just crashes when run from SciTE (and also Notepad++, which I just verified).
>
> After somewhat painful debugging, it seems like the stack trace is a little off -- it actually happens when _exit() is called, which in turn calls _fcloseall() or something.
>
> Not sure what else I can do about it, though -- it's pretty clear the handle was destroyed sometime before, but I don't know when.

This is likely a DMC issue, and is probably best reported as a DMC bug.  I don't know how much more you want to pursue this, but the next steps I'd recommend are:

1. obtain the dmc compiler (it's free) if you haven't already.
2. Compile an empty C program and run it using SciTE and Notepad++.  Verify the same error occurs
3. report the failure using DMC's bugzilla.

I'm guessing there's something in the way SciTE or Notepad++ sets up the pipes before executing the process which causes the problem.  I have a sinking feeling we'll see more of this issue when the new std.process is released :(

-Steve

SciTE (and probably N++ since it's based on scintilla) has a subsystem setting which can affect how it invokes a process. See the "Command subsystem" section here: http://www.scintilla.org/SciTEDoc.html

Usually I run with subsystem 0.

On 9/26/2011 2:53 PM, Steven Schveighoffer wrote:
> I think the reason Walter always gives is that dmd would then have to output COFF objects, whereas it currently outputs OMF.  OPTLINK also does not accept COFF objects.
Wait, why does it have to output COFF? Can't you just use an OMF version of msvcrt.lib, like we're doing for all the other Windows libraries?
I'm confused as to how this is an issue.

> I think a lot of people would very much welcome this change, but it's somewhat of an underwhelming change.  Instead of using DMC's runtime for C functions, you are using VC++'s runtime.  But who cares?  I want D's runtime to be better :)
Well, sort of. It's not so much Visual C++'s runtime, but it's Microsoft's C runtime that's packaged with Windows (which Microsoft ironically doesn't want you to use, but which compilers like GCC use anyway... and it turns out better than anything bundled with Visual Studio). You don't actually need Visual Studio to use it (in fact, it's not even included) -- the Windows Driver Kit has the bundled stuff required.

In all honesty I don't see a single thing about it that I would call "better". Do you have any examples of things that snn.lib does 'better' than msvcrt.lib?

The only incompatibility (which isn't "better" or "worse"... it's just an incompatibility) I can see is exception handling, but that's obviously an exception that can be left in snn.lib. Almost all the rest of the runtime, though, can be easily removed and redirected to msvcrt.dll... and I don't think there'd be any problems. (Are there?)

> I think the best path is weaning ourselves off of C dependencies.  Starting with FILE *, which has a lot of limitations.  It's part of the reason I'm working on reworking std.stdio.
> For example, with my version of std.stdio, you would never have this problem, because it deals with the HANDLE directly instead of through DMC's file descriptor layer.
100% agree, but we weren't talking about D in the first place. :) This is all regarding what's /already/ taking place in snn.lib /before/ D runs, which is obviously unrelated to what's going on in std.stdio.

I'm really curious to know what specific problems are keeping us from doing the switch (i.e. what snn.lib does better than msvcrt.dll). :)

On 9/26/2011 3:07 PM, Andrej Mitrovic wrote:
> SciTE (and probably N++ since it's based on scintilla) has a subsystem
> setting which can affect how it invokes a process. See the "Command
> subsystem" section here: http://www.scintilla.org/SciTEDoc.html
>
> Usually I run with subsystem 0.

(1) Scintilla is just the text editor. It doesn't have anything to do with the integrated command line.

(2) I tried both 0 and 1, and neither made a difference. In fact, while using 1 DID pop up an extra command-line window, in either case the I/O took place inside SciTE itself. And in both cases, both DMD and my program crashed.

You should be able to easily reproduce this if you set the "Enable close exception" and  "Enable bad handles detection" Global Flags, using the Global Flags utility that comes with Debugging Tools for Windows. (Just make sure you reboot before you expect to see the change.)

On 9/26/2011 2:58 PM, Steven Schveighoffer wrote:
> This is likely a DMC issue, and is probably best reported as a DMC bug.  I don't know how much more you want to pursue this, but the next steps I'd recommend are:
>
> 1. obtain the dmc compiler (it's free) if you haven't already.
> 2. Compile an empty C program and run it using SciTE and Notepad++.  Verify the same error occurs
> 3. report the failure using DMC's bugzilla.
>
> I'm guessing there's something in the way SciTE or Notepad++ sets up the pipes before executing the process which causes the problem.  I have a sinking feeling we'll see more of this issue when the new std.process is released :(
>
> -Steve
I'm suspecting it's /not/ a DMC issue (it's probably just an improperly used handle in snn.lib), because I don't see how it could affect my D program this way, if it was a DMC issue.
And yeah, I just tested it with a simple hello world C example using DMC as well, and got the same error. Pretty darn sure now that the handle is getting closed twice or something...

On Monday, September 26, 2011 14:53 Steven Schveighoffer wrote:
> On Mon, 26 Sep 2011 17:35:20 -0400, Mehrdad <wfunction@hotmail.com> wrote:
> > On 9/26/2011 2:30 PM, Mehrdad wrote:
> >> On 9/26/2011 1:51 PM, Steven Schveighoffer wrote:
> >>> AHA!
> >>> 
> >>> Yes, there is a bug in snn.lib regarding pipes. And I fixed it, waiting for Walter to incorporate it :) I needed it for the new std.process.
> >>> 
> >>> What it comes down to is, DMC's FILE * implementation does not expect some of the quirks of pipe HANDLEs. For instance, if you open a FILE * around a pipe handle, it still tries to do a seek on that handle, and crashes. Also, when the write end of a pipe is closed, reading from the read end results in EPIPE from ReadFile, but this is translated to EBADF by the runtime. Therefore, FILE * sets an error instead of EOF.
> >>> 
> >>> Is the email address you have for this message correct? If so, I can send you a new version of snn.lib to try linking your code against (if you are willing to go through these steps), to see if it fixes your problem.
> >>> 
> >>> Using the command line dmd to build should be sufficient (I think).
> >>> 
> >>> -Steve
> >> 
> >> Thanks for the email.
> >> 
> >> It seems like it still crashes from SciTE, though. :(
> >> When I try debugging, I see it's doing so inside _close() which is
> >> /immediately/ above kernel32.dll!@BaseThreadInitThunk@12() -- in other
> >> words, it seems to be called directly from the entrypoint of the
> >> program, assuming it hasn't undergone optimizations. The error is:
> >> 0xC0000008: An invalid handle was specified.
> >> 
> >> Is there any other place in which this can happen?
> > 
> > Also, now that we're on the topic... this might or might not sound silly, but why not just use msvcrt.dll (if not the whole library, at least the I/O functions)? It's not like it introduces a new dependency (it's already in every version of Windows) and it's also proven to work. Not to mention that it reduces code size as well...
> 
> I think the reason Walter always gives is that dmd would then have to output COFF objects, whereas it currently outputs OMF. OPTLINK also does not accept COFF objects.
> 
> I think a lot of people would very much welcome this change, but it's somewhat of an underwhelming change. Instead of using DMC's runtime for C functions, you are using VC++'s runtime. But who cares? I want D's runtime to be better :)

The gain in switching to COFF would be _huge_, because then D code would be properly interoperable with most of the C code on Windows - and in particular C code compiled with Microsoft's compiler. The only thing saving the current situation from being a complete disaster is that you can still use dlls.

- Jonathan M Davis

On 9/26/2011 2:58 PM, Steven Schveighoffer wrote:
> This is likely a DMC issue, and is probably best reported as a DMC bug.  I don't know how much more you want to pursue this, but the next steps I'd recommend are:
>
> 1. obtain the dmc compiler (it's free) if you haven't already.
> 2. Compile an empty C program and run it using SciTE and Notepad++.  Verify the same error occurs
> 3. report the failure using DMC's bugzilla.
>
> I'm guessing there's something in the way SciTE or Notepad++ sets up the pipes before executing the process which causes the problem.  I have a sinking feeling we'll see more of this issue when the new std.process is released :(
>
> -Steve
OK, I think I found the bug, and I'm pretty sure it's what I was saying originally:

A handle is being closed that isn't actually valid.

Which handle? stderr.  stderr is being closed when in fact it is the same handle as stdout. So when _fcloseallp() is called, it goes through the structures in __iob and closes each one in turn -- which causes a duplicate close operation on stderr, because it already gets closed when stdout gets closed.

So it's an snn.lib issue. Is it easy to fix? (I hope it is...)

Thanks!

On Mon, 26 Sep 2011 18:17:49 -0400, Mehrdad <wfunction@hotmail.com> wrote:

> On 9/26/2011 2:58 PM, Steven Schveighoffer wrote:
>> This is likely a DMC issue, and is probably best reported as a DMC bug.  I don't know how much more you want to pursue this, but the next steps I'd recommend are:
>>
>> 1. obtain the dmc compiler (it's free) if you haven't already.
>> 2. Compile an empty C program and run it using SciTE and Notepad++.  Verify the same error occurs
>> 3. report the failure using DMC's bugzilla.
>>
>> I'm guessing there's something in the way SciTE or Notepad++ sets up the pipes before executing the process which causes the problem.  I have a sinking feeling we'll see more of this issue when the new std.process is released :(
>>
>> -Steve
> I'm suspecting it's /not/ a DMC issue (it's probably just an improperly used handle in snn.lib), because I don't see how it could affect my D program this way, if it was a DMC issue.
> And yeah, I just tested it with a simple hello world C example using DMC as well, and got the same error. Pretty darn sure now that the handle is getting closed twice or something...

My apologies, I meant DMC runtime issue (whose library is called snn.lib).

At least with this test, we can rule out D being the culprit.  However, we still have to deal with this problem...

-Steve