On 24/01/2013 13:25, Jacob Carlborg wrote:
> On 2013-01-24 14:12, Bruno Medeiros wrote:
>
>> That's a lot of continuing nice work coming out from there, impressive!
>>
>> BTW, something I've been meaning to ask. I see that in Mono-D you've
>> developed a hand-written D parser. How long did it take you to write
>> that? Did you test it extensively or not so much?
>>
>> I'm embarking on that same task in Java, for DDT, and wondering how long
>> it will take to build a quality parser. So far, it doesn't seem writing
>> the actual parser will take that long, but writing extensive tests for
>> it is seeming exceedingly complicated (or just lengthy and
>> time-consuming). I'm usually a big proponent of TDD, but when writing a
>> lot of tests starts to take a lot of time compared to the code being
>> tested (in this case, maybe 1.5 times the effort/time of the code being
>> tested), I'm not so certain it's the right call to spend so much time
>> writing tests...
>
> How about reusing the one in VisualD, it's written in D? Sure it's a bit
> more work since it's not written in a Java compatible language but it
> might be worth it.
>

If I was going with that approach I likely would rather port the MonoD parser since it looks just as good, if not better, and C# would be easier to port to Java than D.
But the descent.compiler experience (parser ported from DMD's parser) put me off that approach of porting from a parser in another language (although the VisualD parser might have less shortcomings than using the DMD parser since at least VisualD's parser is designed for IDE use). I want to have more control over the parser, and be able to effect my own changes in it (something tricky if you're porting - unless you give up the porting at some point, and just fork your own version and use ir from there)


-- 
Bruno Medeiros - Software Engineer

On 24/01/2013 15:11, Paulo Pinto wrote:
> On Thursday, 24 January 2013 at 13:12:31 UTC, Bruno Medeiros
> wrote:
>> On 21/01/2013 21:14, alex wrote:
>>> Hi everyone,
>>>
>>> No, the entire formatting engine is NOT finished yet. :P
>>> Anyway I've created a good compromise solution between releasing
>>> stuff early and having a fairly nice formatting result:
>>>
>>> I just let the indenting engine calculate the indents of all
>>> lines of the code - and fix all the incorrect indents afterwards.
>>> It works quite fast & reliable(?) - I hope it won't throw very
>>> often + you don't have to worry about resetting your code to an
>>> earlier state via [Ctrl+Shift][Z]
>>>
>>> Indenting only parts of the code is possible either! - So you
>>> don't have to worry about your entire code getting messed up
>>> probably ;)
>>>
>>>
>>> More info @ http://mono-d.alexanderbothe.com
>>>
>>> Issues: https://github.com/aBothe/Mono-D/issues
>>
>> That's a lot of continuing nice work coming out from there, impressive!
>>
>> BTW, something I've been meaning to ask. I see that in Mono-D you've
>> developed a hand-written D parser. How long did it take you to write
>> that? Did you test it extensively or not so much?
>>
>> I'm embarking on that same task in Java, for DDT, and wondering how
>> long it will take to build a quality parser. So far, it doesn't seem
>> writing the actual parser will take that long, but writing extensive
>> tests for it is seeming exceedingly complicated (or just lengthy and
>> time-consuming). I'm usually a big proponent of TDD, but when writing
>> a lot of tests starts to take a lot of time compared to the code being
>> tested (in this case, maybe 1.5 times the effort/time of the code
>> being tested), I'm not so certain it's the right call to spend so much
>> time writing tests...
>
> Are you making use of JavaCC or ANTLR?

No, I thought I was going to initially, but the more I explored it the less I was up for it. This is what I wrote in the DDT forum some time ago:
"
At the moment I'm still just on an exploratory phase: learning more about ANTLR, how it generates code, I looked a bit at Gyula's ANTLR grammar code, but I am also looking into the possibility of writing a handwritten parser. In particular, some interesting reads:
http://stackoverflow.com/questions/6319086/are-gcc-and-clang-parsers-really-handwritten
http://programmers.stackexchange.com/questions/17824/should-i-use-a-parser-generator-or-should-i-roll-my-own-custom-lexer-and-parser
It's a bit hard to get a good opinion on this though, because the shortcomings they mention for parser generators (bad error messages, difficulty to do error recovery), apply mainly to certain types of PGs, like LR ones, but not the LL or PEG ones (like ANTLR), which are good at error reporting&recovery. But still, can handwritten parsers be significantly better at error reporting&recovery than LL/PEG parser generators? Hum...
"
But the more I was trying ANTLR and reading about it, the more I got the impression it was a huge abstraction that added a lot of complexity (in learning and understanding), but didn't actually save you that much effort.
I mean, sure, if you just want to recognize a language, it saves a lot of effort vs. writting a custom parser. But to actually generate an AST, proper source range, DDoc comment annotations, parse trickier rules, handle error recovery properly, and make sure the parser is efficient, it seemed like a daunting task. I felt I would have to become an ANLTR expert (read the book, and the theory behind it) for something that seemed trivial and fairly easy to do without a parser generator.
This is by no means a fully-fledged, 100% assured opinion, but it's what I got so far. Comments and ideas are welcome. But from what I read from stackexchange a lot of other people seem to share this opinion (that it's better to write hand-written), although I don't know which kind of context and requirements they are comming from.


-- 
Bruno Medeiros - Software Engineer

On 2013-01-25 13:01, Bruno Medeiros wrote:

> If I was going with that approach I likely would rather port the MonoD
> parser since it looks just as good, if not better, and C# would be
> easier to port to Java than D.
> But the descent.compiler experience (parser ported from DMD's parser)
> put me off that approach of porting from a parser in another language
> (although the VisualD parser might have less shortcomings than using the
> DMD parser since at least VisualD's parser is designed for IDE use). I
> want to have more control over the parser, and be able to effect my own
> changes in it (something tricky if you're porting - unless you give up
> the porting at some point, and just fork your own version and use ir
> from there)

I didn't say anything about porting :) I was suggesting you integrate the VisualD parser without porting it. That's why I suggested the one in VisualD and not the one in Mono-D.

-- 
/Jacob Carlborg

On Friday, 25 January 2013 at 12:31:12 UTC, Bruno Medeiros wrote:
> On 24/01/2013 15:11, Paulo Pinto wrote:
>> On Thursday, 24 January 2013 at 13:12:31 UTC, Bruno Medeiros
>> wrote:
>>> On 21/01/2013 21:14, alex wrote:
>>>> Hi everyone,
>>>>
>>>> No, the entire formatting engine is NOT finished yet. :P
>>>> Anyway I've created a good compromise solution between releasing
>>>> stuff early and having a fairly nice formatting result:
>>>>
>>>> I just let the indenting engine calculate the indents of all
>>>> lines of the code - and fix all the incorrect indents afterwards.
>>>> It works quite fast & reliable(?) - I hope it won't throw very
>>>> often + you don't have to worry about resetting your code to an
>>>> earlier state via [Ctrl+Shift][Z]
>>>>
>>>> Indenting only parts of the code is possible either! - So you
>>>> don't have to worry about your entire code getting messed up
>>>> probably ;)
>>>>
>>>>
>>>> More info @ http://mono-d.alexanderbothe.com
>>>>
>>>> Issues: https://github.com/aBothe/Mono-D/issues
>>>
>>> That's a lot of continuing nice work coming out from there, impressive!
>>>
>>> BTW, something I've been meaning to ask. I see that in Mono-D you've
>>> developed a hand-written D parser. How long did it take you to write
>>> that? Did you test it extensively or not so much?
>>>
>>> I'm embarking on that same task in Java, for DDT, and wondering how
>>> long it will take to build a quality parser. So far, it doesn't seem
>>> writing the actual parser will take that long, but writing extensive
>>> tests for it is seeming exceedingly complicated (or just lengthy and
>>> time-consuming). I'm usually a big proponent of TDD, but when writing
>>> a lot of tests starts to take a lot of time compared to the code being
>>> tested (in this case, maybe 1.5 times the effort/time of the code
>>> being tested), I'm not so certain it's the right call to spend so much
>>> time writing tests...
>>
>> Are you making use of JavaCC or ANTLR?
>
> No, I thought I was going to initially, but the more I explored it the less I was up for it. This is what I wrote in the DDT forum some time ago:
> "
> At the moment I'm still just on an exploratory phase: learning more about ANTLR, how it generates code, I looked a bit at Gyula's ANTLR grammar code, but I am also looking into the possibility of writing a handwritten parser. In particular, some interesting reads:
> http://stackoverflow.com/questions/6319086/are-gcc-and-clang-parsers-really-handwritten
> http://programmers.stackexchange.com/questions/17824/should-i-use-a-parser-generator-or-should-i-roll-my-own-custom-lexer-and-parser
> It's a bit hard to get a good opinion on this though, because the shortcomings they mention for parser generators (bad error messages, difficulty to do error recovery), apply mainly to certain types of PGs, like LR ones, but not the LL or PEG ones (like ANTLR), which are good at error reporting&recovery. But still, can handwritten parsers be significantly better at error reporting&recovery than LL/PEG parser generators? Hum...
> "
> But the more I was trying ANTLR and reading about it, the more I got the impression it was a huge abstraction that added a lot of complexity (in learning and understanding), but didn't actually save you that much effort.
> I mean, sure, if you just want to recognize a language, it saves a lot of effort vs. writting a custom parser. But to actually generate an AST, proper source range, DDoc comment annotations, parse trickier rules, handle error recovery properly, and make sure the parser is efficient, it seemed like a daunting task. I felt I would have to become an ANLTR expert (read the book, and the theory behind it) for something that seemed trivial and fairly easy to do without a parser generator.
> This is by no means a fully-fledged, 100% assured opinion, but it's what I got so far. Comments and ideas are welcome. But from what I read from stackexchange a lot of other people seem to share this opinion (that it's better to write hand-written), although I don't know which kind of context and requirements they are comming from.

Thanks for the lengthy reply.

I was just curious.

Hand written parsers tend to better on error messages, that is
true.

Personally I tend to use parser generators, but since university
days I just do DSL kind of languages anyway.

The main problem is when you start parsing things that require a
lot of backtracking or do not fit well in a LL(k) grammar
description.

--
Paulo

On Friday, 25 January 2013 at 13:43:46 UTC, Jacob Carlborg wrote:
> I didn't say anything about porting :) I was suggesting you integrate the VisualD parser without porting it. That's why I suggested the one in VisualD and not the one in Mono-D.

I already suggested Rainer to make a native/non-native interface between VisualD and D_Parser - this will probably happen via COM or so.. and I dunno anything about that technique. I just can make sure that the parser library is fully stand-alone, only depending on .net internals. Let's see :)

On 2013-01-25 20:00, alex wrote:

> I already suggested Rainer to make a native/non-native interface between
> VisualD and D_Parser - this will probably happen via COM or so.. and I
> dunno anything about that technique. I just can make sure that the
> parser library is fully stand-alone, only depending on .net internals.
> Let's see :)

It should provide an C interface, then it can be connected to anything.

-- 
/Jacob Carlborg

On Friday, 25 January 2013 at 20:34:28 UTC, Jacob Carlborg wrote:
> On 2013-01-25 20:00, alex wrote:
>
>> I already suggested Rainer to make a native/non-native interface between
>> VisualD and D_Parser - this will probably happen via COM or so.. and I
>> dunno anything about that technique. I just can make sure that the
>> parser library is fully stand-alone, only depending on .net internals.
>> Let's see :)
>
> It should provide an C interface, then it can be connected to anything.

Why not wrap dmd's front-end?

On 25.01.2013 21:34, Jacob Carlborg wrote:
> On 2013-01-25 20:00, alex wrote:
>
>> I already suggested Rainer to make a native/non-native interface between
>> VisualD and D_Parser - this will probably happen via COM or so.. and I
>> dunno anything about that technique. I just can make sure that the
>> parser library is fully stand-alone, only depending on .net internals.
>> Let's see :)
>
> It should provide an C interface, then it can be connected to anything.
>

COM is the natural choice when interfacing native code on Windows with C#. On other platforms it might be different.

The semantic engine in Visual D is separated into another process and communicates with the IDE plugin through a number of "commands", just using this interface: https://github.com/rainers/visuald/blob/master/vdc/ivdserver.d . This can easily be mapped to C calls.

There's also an implementation of that interface using D_Parser (https://github.com/rainers/visuald/tree/master/vdc/abothe), but it isn't complete yet.

On 2013-01-26 01:23, alex wrote:

> Why not wrap dmd's front-end?

The problem with the DMD frontend is that it's not made to be used in on its own, like in an IDE.

-- 
/Jacob Carlborg

On Saturday, 26 January 2013 at 08:22:39 UTC, Rainer Schuetze wrote:
>
>
> On 25.01.2013 21:34, Jacob Carlborg wrote:
>> On 2013-01-25 20:00, alex wrote:
>>
>>> I already suggested Rainer to make a native/non-native interface between
>>> VisualD and D_Parser - this will probably happen via COM or so.. and I
>>> dunno anything about that technique. I just can make sure that the
>>> parser library is fully stand-alone, only depending on .net internals.
>>> Let's see :)
>>
>> It should provide an C interface, then it can be connected to anything.
>>
>
> COM is the natural choice when interfacing native code on Windows with C#. On other platforms it might be different.
>
> The semantic engine in Visual D is separated into another process and communicates with the IDE plugin through a number of "commands", just using this interface: https://github.com/rainers/visuald/blob/master/vdc/ivdserver.d . This can easily be mapped to C calls.
>
> There's also an implementation of that interface using D_Parser (https://github.com/rainers/visuald/tree/master/vdc/abothe), but it isn't complete yet.

Concerning completion server...why not a completion server? :D - I mean, it'll be launched as soon as VisualD launches..and then you can pipe-through commands etc. to interact like it's done the mspdbsrv already. That's imho even easier than using COM + can be driven even as a web server..which would be a real dream then!