I've finally finished the D port of GDMD and pushed everything including a detailed README to https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex combinations there could still be some unknown bugs.


I think the D port should already be much more useful than the perl script though. It supports gdmd -main, gdmd -run and gdmd -lib as well as most other options in DMD 2.066*. And it properly creates all output folders which dmd-script sometimes doesn't do...

So I'd like to ship this gdmd with the binaries starting with the next release. @Iain if you've got no objections I'd also like to merge this into the GDMD master branch ASAP. We have to keep dmd-script in master though for travis-ci.


* There are some DMD options which can't work the same way in GDC. For
  example treating deprecation as an error but all other warnings as
  warnings.

Am Sat, 26 Mar 2016 11:09:10 +0100
schrieb Johannes Pfau <nospam@example.com>:

> I've finally finished the D port of GDMD and pushed everything including a detailed README to https://github.com/D-Programming-GDC/GDMD/tree/dport
> 
> All parameters are tested for simple cases but in complex combinations there could still be some unknown bugs.
> 
> 
> I think the D port should already be much more useful than the perl script though. It supports gdmd -main, gdmd -run and gdmd -lib as well as most other options in DMD 2.066*. And it properly creates all output folders which dmd-script sometimes doesn't do...
> 
> So I'd like to ship this gdmd with the binaries starting with the next release. @Iain if you've got no objections I'd also like to merge this into the GDMD master branch ASAP. We have to keep dmd-script in master though for travis-ci.
> 
> 
> * There are some DMD options which can't work the same way in GDC. For
>   example treating deprecation as an error but all other warnings as
>   warnings.

I forgot to mention that the initial code was written by quickfur aka H. S. Teoh. Thanks for that!

On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
> I've finally finished the D port of GDMD and pushed everything including a detailed README to https://github.com/D-Programming-GDC/GDMD/tree/dport

Thanks for this.

I see that parsing response files isn't implemented yet. Response files are practically required when used with rdmd on Windows, because of the command line length limit.

std.process.escapeWindowsArgument is available on all platforms for this purpose:

http://dlang.org/phobos/std_process.html#.escapeWindowsArgument

(The response files use the Windows syntax, regardless of the host platform.)

On Saturday, 26 March 2016 at 10:35:44 UTC, Vladimir Panteleev wrote:
> On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
>> I've finally finished the D port of GDMD and pushed everything including a detailed README to https://github.com/D-Programming-GDC/GDMD/tree/dport
>
> Thanks for this.
>
> I see that parsing response files isn't implemented yet. Response files are practically required when used with rdmd on Windows, because of the command line length limit.
>
> std.process.escapeWindowsArgument is available on all platforms for this purpose:
>
> http://dlang.org/phobos/std_process.html#.escapeWindowsArgument
>
> (The response files use the Windows syntax, regardless of the host platform.)

Erm, sorry, of course gdmd needs to parse, not generate response files. Ignore the above.

Am Sat, 26 Mar 2016 10:35:44 +0000
schrieb Vladimir Panteleev <thecybershadow.lists@gmail.com>:

> On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
> > I've finally finished the D port of GDMD and pushed everything including a detailed README to https://github.com/D-Programming-GDC/GDMD/tree/dport
> 
> Thanks for this.
> 
> I see that parsing response files isn't implemented yet. Response files are practically required when used with rdmd on Windows, because of the command line length limit.
> 
> std.process.escapeWindowsArgument is available on all platforms for this purpose:
> 
> http://dlang.org/phobos/std_process.html#.escapeWindowsArgument
> 
> (The response files use the Windows syntax, regardless of the host platform.)
> 

I didn't realize response file parsing is that important for windows, thanks for letting me know ;-) I guess there's no proper documentation for response files or any test files?


There are some open questions:

* Are escape sequences allowed outside of quoted strings?
* Which characters end a comment?
* How does this parse: foo\"bar
* Or this: foo"bar


BTW:
The DMD implementation is using the backend license, so I
certainly couldn't copy it. It tried to follow clean-room design
principles when reimplementing it (writing a spec first, then using the
spec to implement the new parser) and choose a completely different
parsing approach (parsing/unescaping separated, range based) so I hope
it's different enough to avoid copyright or licensing issues....

https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
It'd be great if you could have a look at the unittests and tell me if
the parsing is correct ;-)

Now I still have to generate a response file for passing arguments to GDC. First have to check whether GCC even uses the windows escaping rules though. Do you know how many characters can be passed without response files?

On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
> I've finally finished the D port of GDMD and pushed everything including a detailed README to https://github.com/D-Programming-GDC/GDMD/tree/dport
>
> All parameters are tested for simple cases but in complex combinations there could still be some unknown bugs.

Good news. I'll follow the project.

Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after a renaming it was OK with a simple CE project. (linux x86_64 and latest stable gdc)

Thanks for working on this :)

On Saturday, 26 March 2016 at 22:37:53 UTC, Johannes Pfau wrote:
> I didn't realize response file parsing is that important for windows, thanks for letting me know ;-) I guess there's no proper documentation for response files or any test files?

It uses the CommandLineToArgvW syntax.

That's pretty much all there is to it.

> There are some open questions:
>
> * Are escape sequences allowed outside of quoted strings?

Apparently, though rdmd will never generate such response files.

> * Which characters end a comment?

There are no comments in response files (as far as I know).

> * How does this parse: foo\"bar

foo"bar

> * Or this: foo"bar

foobar

I suggest writing a small program to experiment, e.g.

void main(string[] args) { args.each!writeln(); }

> BTW:
> The DMD implementation is using the backend license, so I
> certainly couldn't copy it. It tried to follow clean-room design
> principles when reimplementing it (writing a spec first, then using the
> spec to implement the new parser) and choose a completely different
> parsing approach (parsing/unescaping separated, range based) so I hope
> it's different enough to avoid copyright or licensing issues....

I'm not sure this approach is the best - I think unescaping and splitting should be done in one pass. Essentially, I think arguments are separated by an unescaped space.

If you have a Windows machine, you can verify it yourself by comparing the results with CommandLineToArgvW.

> https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
> It'd be great if you could have a look at the unittests and tell me if
> the parsing is correct ;-)

The unit tests *look* OK, but I can't say for sure without having a way to verify them.

I suggest you have a look at the escaping code and tests from std.process. There should be no licensing conflict with Boost, I think. I went to great lengths to ensure that code is correct, there is even a test which brute-forces special character combinations and another that tries random strings to find bugs in the escaping code.

IIRC the simple regex that was in the Perl version of gdmd was also sufficient for parsing the response files generated by rdmd.

> Now I still have to generate a response file for passing arguments to GDC. First have to check whether GCC even uses the windows escaping rules though. Do you know how many characters can be passed without response files?

rdmd uses a limit of 1024, but I think that's lower than the OS limit.

Am Sun, 27 Mar 2016 08:35:00 +0000
schrieb Vladimir Panteleev <thecybershadow.lists@gmail.com>:

> Thanks for working on this :)
> 
> On Saturday, 26 March 2016 at 22:37:53 UTC, Johannes Pfau wrote:
> > I didn't realize response file parsing is that important for windows, thanks for letting me know ;-) I guess there's no proper documentation for response files or any test files?
> 
> It uses the CommandLineToArgvW syntax.
> 
> That's pretty much all there is to it.
> 
> > There are some open questions:
> >
> > * Are escape sequences allowed outside of quoted strings?
> 
> Apparently, though rdmd will never generate such response files.
> 
> > * Which characters end a comment?
> 
> There are no comments in response files (as far as I know).
> [...]

Well, that's the problem: There's no specification for response files. The DMD implementation certainly does support comments in response files. LDMD simply reuses the DMD code because of these issues...

The entries in the response file are formatted according to CommandLineToArgvW, this part is simple.

But the entries are separated by 'whitespace' which according to dmd is '\r' '\n' '\t' ' ' and '\0'! This leads to many questions such as does '\0' end a comment? Additionally dmd treats '#' as the start of a comment and 0x1a always ends the file...

DMD also only does the '\' escaping for double quotes, not for single quotes. I think the code I've pushed should be as DMD-compatible as possible.

> I'm not sure this approach is the best - I think unescaping and splitting should be done in one pass. Essentially, I think arguments are separated by an unescaped space.

Sure, escaping needs to be considered when splitting the string (to differentiate between whitespace in quoted strings / whitespace and to differentiate escaped quotes from normal quotes). But the code is nicer if the splitting pass simply returns a slice to the original content and does not assemble the unescaped strings on the fly.

> If you have a Windows machine, you can verify it yourself by comparing the results with CommandLineToArgvW.
> 
> > https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
> > It'd be great if you could have a look at the unittests and
> > tell me if
> > the parsing is correct ;-)
> 
> The unit tests *look* OK, but I can't say for sure without having a way to verify them.
> 
> I suggest you have a look at the escaping code and tests from std.process. There should be no licensing conflict with Boost, I think. I went to great lengths to ensure that code is correct, there is even a test which brute-forces special character combinations and another that tries random strings to find bugs in the escaping code.
> 
> IIRC the simple regex that was in the Perl version of gdmd was also sufficient for parsing the response files generated by rdmd.
> 

Yeah, probably supporting the sane subset of response files actually used is good enough. I tried to match the dmd implementation, so any weird special feature works, but maybe that's not required.

> > Now I still have to generate a response file for passing arguments to GDC. First have to check whether GCC even uses the windows escaping rules though. Do you know how many characters can be passed without response files?
> 
> rdmd uses a limit of 1024, but I think that's lower than the OS limit.
> 

OK, thanks!

Am Sun, 27 Mar 2016 04:25:27 +0000
schrieb Basile B. <b2.temp@gmx.com>:

> On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
> > I've finally finished the D port of GDMD and pushed everything including a detailed README to https://github.com/D-Programming-GDC/GDMD/tree/dport
> >
> > All parameters are tested for simple cases but in complex combinations there could still be some unknown bugs.
> 
> Good news. I'll follow the project.
> 
> Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after a renaming it was OK with a simple CE project. (linux x86_64 and latest stable gdc)

This is actually the first time I heard about gcc-ar...
But it seems using gcc-ar is indeed better:
http://manpages.ubuntu.com/manpages/trusty/man1/aarch64-linux-gnu-gcc-ar-4.7.1.html

Thanks for reporting this!

Am Sun, 27 Mar 2016 13:31:34 +0200
schrieb Johannes Pfau <nospam@example.com>:

> Am Sun, 27 Mar 2016 08:35:00 +0000
> schrieb Vladimir Panteleev <thecybershadow.lists@gmail.com>:
> 
> 
> > I'm not sure this approach is the best - I think unescaping and splitting should be done in one pass. Essentially, I think arguments are separated by an unescaped space.
> 
> Sure, escaping needs to be considered when splitting the string (to differentiate between whitespace in quoted strings / whitespace and to differentiate escaped quotes from normal quotes). But the code is nicer if the splitting pass simply returns a slice to the original content and does not assemble the unescaped strings on the fly.
> 

Thinking about this some more, it probably can't work in all cases. So I've changed it to do unescaping on the fly.

I've also integrated the phobos unittests:
* My parser doesn't report "" as an empty argument. I could fix this,
  but then we don't use empty arguments, so why bother?


However, MS is completely insane:

`"C:\abc\" def" foo` => `C:\abc" def`, `foo`
`"C:\abc\"def" foo` => `C:\abc"def`, `foo`
`"C:\abc\\" def" foo` => `C:\abc\`, `def foo`
`"C:\abc\\\" def" foo` => `C:\abc\" def`, `foo`

`"C:\abc\\"def" foo` => `C:\abc\def foo`
The last one is not supported in the parser cause I don't want to guess
any further what arbitrary rules MS has made up...