On Sunday 31 July 2011 22:18:51 Andrei Alexandrescu wrote:
> On 07/31/2011 09:28 PM, Johann MacDonagh wrote:
> > Well, if Andrei wants to flex the power of D2's templates / etc... with a parser generator then maybe we should go down that route. Andrei, what do you think? Would the lexer/parser be generated at compile time or a a regular tool that would generate the appropriate D files?
> 
> Until we have fully integrated, EASY to use, and FAST lexer and parser generators, we haven't yet proven the power of CTFE technology. I don't have the time to embark on such a project for the time being.

We need to have all of the dmd and Phobos developers in triplicate. Then maybe we could actually get everything done that we'd like to get done...

Well, at least the change to github has better enabled commmunity support outside of the primary developers. So, even if still don't get it everything done that we'd like to, we're at least doing much better.

- Jonathan M Davis

On 7/31/2011 6:29 PM, Johann MacDonagh wrote:
> Basically, you give it some string (string, wstring, or dstring), and it gives
> you a range of tokens back. The token has the type, a slice of the input that
> corresponds to the token, line / column, and a value (e.g. an integer constant).

I suggest making the input a range.

Am 01.08.2011, 05:31 Uhr, schrieb Jonathan M Davis <jmdavisProg@gmx.com>:

> We need to have all of the dmd and Phobos developers in triplicate. Then maybe we could actually get everything done that we'd like to get done...

From reading what some C++ folks think is 'forced' on them in D (making common things easy vs a low level structure covering all exotic use cases) and my experience with the RedBlackTree (it is fast but, couldn't be created empty as I am used to from other language libraries) I think it is not just about getting things done, but about brainstorming in advance.
When we use containers, what use cases do we have? Programmer A wants to be able to easily switch the container implementation from an array list to a sorted tree, B requires pure speed without the overhead of virtual functions and a class hierarchy. Either A or B give in, there is room for both in Phobos or either A or B go and (re)write the containers. I believe this discussion has been here already, but I wasn't reading the list at that time. :)
So what I want to say is, before someone starts writing the code for a parser generator, try to make a list of use cases. I don't say every use case must be covered, but it would be great, if there were 'entry points' for others to implement what they need. A common example is a sorted set that sorts by the natural ordering of elements, but can be augmented with a comparator function. So far we have the
- general idea of a lexer/parser
- a D source code parser
I'd be very exited to see a CTFE solution to building a parser *g*, you know just for the awe of it. If that works out and it turns out that it is actually quite fast it *could* be used to write a D parser from definitions. But the point that you would want to keep close to the C source of DMD is strong.
I wanted to write a disk cleanup tool for Linux that uses a simple special script language basically using list operations (like list all java processes -> list all files in directory x containing any processes pid -> protect them from deletion). The format consists keywords that can open new blocks that again contain other keywords, so I ended up writing parse functions for each block type. I'd go with a complete AST here instead of the linear list of tokens that has been proposed. If I wrote a syntax highlighter then the AST would not be of much use. So this could be two use cases to cover. Comments?

On 2011-07-31 21:06, Jim Hewes wrote:
> On 7/31/2011 3:01 AM, Jacob Carlborg wrote:
>>
>> I personally would like to see DWT (GUI library,
>> http://dsource.org/projects/dwt) in a more finished state. These are
>> things that need work:
>>
>> * Finishing the Mac OS X port
>> * Update to later SWT versions
>> * Fixing bugs
>> * Port to 64bit
>>
>> Any help is much appreciated. Currently DWT is a low priority for me.
>>
>
> I agree with this one. I'll tell you why, hopefully without getting too
> long-winded. :-) I've just been a lurker here and although I like the D
> language haven't really used it seriously yet.
>
> Where I work, we use Visual Studio C++ for the main product. It's highly
> unlikely that would just get switch to using D. But we also write
> various small utilities and test programs that we use internally. These
> are almost all written in C++ with MFC. I have never been a fan of MFC.
> (Even from the beginning I preferred Borland's ObjectWindows.) But it
> has some advantages. It's a common, almost standard GUI that comes with
> VS and to some extent is integrated with VS. Using MFC I can produce
> single-file EXEs with a GUI that can be easily passed around the company
> internally. I can tell users to just grab one EXE, copy it to your
> computer and run it. Very easy for people. My point here is that these
> little applications are not critical and so they could be written in any
> language. In my company, this would be the best way for a new language
> to make inroads. I've written one or two utility apps in C# and haven't
> had any complaints.

That's understandable.

> For my job I work on Windows, though I try not to be an O/S fanboy and
> try to treat all platforms with equal credibility. But I guess one of my
> peeves about open source and Linux (which seem to go together) is that
> for any target goal there are a hundred half-finished, abandoned
> projects. I assume this is because people don't get paid to do it and so
> once they get tired of it they move on to other more interesting things.
> I can't depend on that or sell it to my colleagues. Take a look at the
> GUI page on the D wiki. There are a lot of projects, but almost all of
> them seem to be either in alpha or beta stage or are abandoned. Further,
> in every project, each developer usually chose to use some different
> array of tools that I need to download and install in my computer just
> to build his project. This gets kind of annoying on Windows.

I fully agree with that and it's annoying. The wiki page could probably use some cleanup. About the tools, I'm working on a package manager for D that will ease installation of D libraries, tools and application. I and at least one other person is working on a build tool with the hope that we can all agree on one standard built tool for D.

> I've recently downloaded the QtD binding to take a look because Qt seem
> to be a pretty popular cross-platform library. I was able to build it
> and then also build a couple of the sample projects, although I haven't
> really spent time with it beyond that. It seems that not only is a Hello
> World program a 1.7MB executable, but I think I would also need to
> distribute the libcpp_core.dll and libcpp_gui.dll files with it for a
> total of about 17MB. QtD looks like a lot of good work and I don't mean
> to knock it here. I just mean to say that in contrast what would be nice
> is a “standard” GUI written in D that could be statically linked and
> would be not too large. I confess I have not tried the existing DWT yet,
> mostly since I just didn't want to also download Tango.
>
> Jim

DWT works with D2 (Phobos) as well. It's statically linked so the hello world application gets quite large. But that's because it pulls in a lot of the library, meaning even if you start using other parts of the library it won't increase the size of the executable at the same rate.

-- 
/Jacob Carlborg

On 2011-08-01 03:56, Jonathan M Davis wrote:
> If we do a hand-written lexer of D for Phobos, it really should be a fairly
> direct port of the dmd front-end. It should be _somewhat_ D-ified as
> appropriate, (and the API should definitely be properly range-based and all
> that), but the implementation needs to be fairly close to dmd itself so that
> it's easy for someone to port changes and fixes back and forth between the two.
> Otherwise, they're going to get out of sync fairly easily. If we're not going
> to do a direct port, then we might as well just do the template-based lexer
> generator that Andrei and others would really like to see (which we should
> still do, but I think that the hand-written lexer is nowhere near as valuable
> if it's not a direct port of dmd's lexer).

In the long run I hope that this won't be necessary, I'm hoping that DMD will use the lexer from Phobos.

> Also, I see _zero_ value in making it support D1. If it's for D2's standard
> library, then what's the point of it lexing D1? That just complicates the
> lexer for what is essentially a legacy product. And given that the differences
> between D1 and D2 in dmd's lexer are covered with #ifdefs, it would be rather
> complicated to try and do a direct port which covers both D1 and D2. It would
> probably be easier if the two were completely separate.
>
> As for what I've done so far, I'd have to go look. I haven't touched it in a
> couple of months, I expect. There has been a lot of other stuff that I've
> needed to do, and Andrei was trying to discourage such an implementation the
> last time that I brought it up. So, I haven't exactly been in a rush to get it
> done. I'd like to do it, but I've been rather busy.

In this case I really don't think that you should listen to Andrei.

> So, if you really want to work on a potential D lexer for Phobos, that's fine,
> but I really think that it needs to be a rather direct port, and that doesn't
> sound like what you've been doing.
>
> - Jonathan M Davis


-- 
/Jacob Carlborg

streams and network programming is most desirable, this also impacts other libraries.

Another big one is async programming. Some kind of unified, higher level framework to deal with this could be a huge help - it's a hard problem.

It's interesting to see that in .NET a lot of new stuff concerning async programming is coming out, suggesting that the (many) existing patterns are considered inadequate. In C# 5, async operations will be a language level feature with the async/await keywords. There is also the reactive extensions library for async streams: http://msdn.microsoft.com/en-us/data/gg577609. Most interesting about this one is how easy it is to compose async operations.

On Monday 01 August 2011 10:12:17 Jacob Carlborg wrote:
> On 2011-08-01 03:56, Jonathan M Davis wrote:
> > If we do a hand-written lexer of D for Phobos, it really should be a fairly direct port of the dmd front-end. It should be _somewhat_ D-ified as appropriate, (and the API should definitely be properly range-based and all that), but the implementation needs to be fairly close to dmd itself so that it's easy for someone to port changes and fixes back and forth between the two. Otherwise, they're going to get out of sync fairly easily. If we're not going to do a direct port, then we might as well just do the template-based lexer generator that Andrei and others would really like to see (which we should still do, but I think that the hand-written lexer is nowhere near as valuable if it's not a direct port of dmd's lexer).
> 
> In the long run I hope that this won't be necessary, I'm hoping that DMD will use the lexer from Phobos.
> 
> > Also, I see _zero_ value in making it support D1. If it's for D2's standard library, then what's the point of it lexing D1? That just complicates the lexer for what is essentially a legacy product. And given that the differences between D1 and D2 in dmd's lexer are covered with #ifdefs, it would be rather complicated to try and do a direct port which covers both D1 and D2. It would probably be easier if the two were completely separate.
> > 
> > As for what I've done so far, I'd have to go look. I haven't touched it in a couple of months, I expect. There has been a lot of other stuff that I've needed to do, and Andrei was trying to discourage such an implementation the last time that I brought it up. So, I haven't exactly been in a rush to get it done. I'd like to do it, but I've been rather busy.
> 
> In this case I really don't think that you should listen to Andrei.

Well, when I have multiple things on my plate, and Andrei has a major disagreement with one, I'm less likely to focus on that one. It's also one which takes quite a bit of time and focus, whereas there is other stuff which I can do in smaller chunks of time without needing as much focus. So, I haven't worked on it as much. I should probably remedy that, but there's quite a lot that I've been trying to get done.

So, yes. I intend to do it in spite of Andrei's opinion on the matter, but lately I've been focusing on other stuff which needs doing.

- Jonathan M Davis

On 2011-08-01 04:34, Jonathan M Davis wrote:
> I can tell you right now that he wants a generic, template-based lexer /
> parser generator rather than a hand-written solution. He doesn't like the idea
> of the hand-written solution. And I definitely think that we should have a
> template-based lexer / parser generator. The question is whether it's also
> worth having a hand-written port of the dmd front-end's lexer and parser to
> Phobos so that we can have on official lexer and parser in Phobos which parse D
> the same way that the compiler does.
>
> - Jonathan M Davis

I think it's absolutly worth having a hand-written port of the DMD front end in Phobos.

-- 
/Jacob Carlborg

On 2011-08-01 04:38, Johann MacDonagh wrote:
> On 7/31/2011 10:34 PM, Jonathan M Davis wrote:
>> I can tell you right now that he wants a generic, template-based lexer /
>> parser generator rather than a hand-written solution. He doesn't like
>> the idea
>> of the hand-written solution. And I definitely think that we should
>> have a
>> template-based lexer / parser generator. The question is whether it's
>> also
>> worth having a hand-written port of the dmd front-end's lexer and
>> parser to
>> Phobos so that we can have on official lexer and parser in Phobos
>> which parse D
>> the same way that the compiler does.
>>
>> - Jonathan M Davis
>
> I see. In that case, my hybrid solution doesn't solve either problem ;)
>
> I'll keep my code in that branch and start on something new.

I think you should continue on your lexer, we need a hand-written lexer/parser in Phobos, preferably a port of the DMD front end.

-- 
/Jacob Carlborg

On 2011-08-01 10:21, Jonathan M Davis wrote:
> On Monday 01 August 2011 10:12:17 Jacob Carlborg wrote:
>> On 2011-08-01 03:56, Jonathan M Davis wrote:
>>> If we do a hand-written lexer of D for Phobos, it really should be a
>>> fairly direct port of the dmd front-end. It should be _somewhat_
>>> D-ified as appropriate, (and the API should definitely be properly
>>> range-based and all that), but the implementation needs to be fairly
>>> close to dmd itself so that it's easy for someone to port changes and
>>> fixes back and forth between the two. Otherwise, they're going to get
>>> out of sync fairly easily. If we're not going to do a direct port, then
>>> we might as well just do the template-based lexer generator that Andrei
>>> and others would really like to see (which we should still do, but I
>>> think that the hand-written lexer is nowhere near as valuable if it's
>>> not a direct port of dmd's lexer).
>>
>> In the long run I hope that this won't be necessary, I'm hoping that DMD
>> will use the lexer from Phobos.
>>
>>> Also, I see _zero_ value in making it support D1. If it's for D2's
>>> standard library, then what's the point of it lexing D1? That just
>>> complicates the lexer for what is essentially a legacy product. And
>>> given that the differences between D1 and D2 in dmd's lexer are covered
>>> with #ifdefs, it would be rather complicated to try and do a direct
>>> port which covers both D1 and D2. It would probably be easier if the
>>> two were completely separate.
>>>
>>> As for what I've done so far, I'd have to go look. I haven't touched it
>>> in a couple of months, I expect. There has been a lot of other stuff
>>> that I've needed to do, and Andrei was trying to discourage such an
>>> implementation the last time that I brought it up. So, I haven't
>>> exactly been in a rush to get it done. I'd like to do it, but I've been
>>> rather busy.
>>
>> In this case I really don't think that you should listen to Andrei.
>
> Well, when I have multiple things on my plate, and Andrei has a major
> disagreement with one, I'm less likely to focus on that one. It's also one
> which takes quite a bit of time and focus, whereas there is other stuff which I
> can do in smaller chunks of time without needing as much focus. So, I haven't
> worked on it as much. I should probably remedy that, but there's quite a lot
> that I've been trying to get done.
>
> So, yes. I intend to do it in spite of Andrei's opinion on the matter, but
> lately I've been focusing on other stuff which needs doing.
>
> - Jonathan M Davis

Fair enough.

-- 
/Jacob Carlborg