August 03, 2012
On Friday, August 03, 2012 06:14:08 Timon Gehr wrote:
> If it is optimal for D lexing and close-optimal or optimal for other languages then it is profoundly more useful than just a D lexer.

If fully concur that we should have a generic lexer, but unless the generic lexer can be just as fast as a hand-written one (which I seriously question), then I definitely think that we're going to want both, not just a generic one.

- Jonathan M Davis
August 03, 2012
On Fri, Aug 3, 2012 at 6:14 AM, Timon Gehr <timon.gehr@gmx.ch> wrote:

>> If someone wants to try and write a generic lexer for D and see if they
>> can
>> beat out any hand-written ones,
>
>
> I'll possibly give it a shot if I can find the time.

I propose we let him finish std.lexer, test it with Jonathan, benchmark it, etc.
Once it's out, we can create a small group interested in getting
generic and we can try and decorticate it at our leasure.
August 03, 2012
On Fri, Aug 3, 2012 at 5:59 AM, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> On Friday, August 03, 2012 05:36:05 Bernard Helyer wrote:
>> If the other guys think they've got it, then I can withdraw my efforts. I was just thinking I had a lexer just sitting around, may as well use it, but if the other guys have it, then I'm fine with withdrawing.
>
> I'm a fair ways along already. Making changes according to what Walter wants will slow me down some but not a lot. It's something that I've wanted to do for quite some time but just finally got around to starting a couple of weeks ago. So, I definitely want to complete it regardless of what anyone else is doing. It also gives me an opportunity to make sure that the spec and dmd match (and I've already found a few bugs with the spec and fixed them).
>
> Feel free to do whatever you want, but I'm definitely going to complete what I'm working on.

Look, many of us here were interested in your idea of having comments
and errors lexed as tokens.
Could it be possible to just add two static policies errorBehavior {
report, skip, ...} and comments { asToken, grouped }?
That way, a user just say;

// Walter
auto toks = Lexer!(comments.grouped)(code); // May be the default value
// Other:
auto toks = Lexer!(comments.asToken)(code);

It's just a small static if away...
August 03, 2012
On Friday, August 03, 2012 07:30:32 Philippe Sigaud wrote:
> Look, many of us here were interested in your idea of having comments
> and errors lexed as tokens.
> Could it be possible to just add two static policies errorBehavior {
> report, skip, ...} and comments { asToken, grouped }?
> That way, a user just say;
> 
> // Walter
> auto toks = Lexer!(comments.grouped)(code); // May be the default value
> // Other:
> auto toks = Lexer!(comments.asToken)(code);
> 
> It's just a small static if away...

I'm sure it's possible. I just don't know how messy it will be. It depends on how the code is affected by needing to not treat comments and errors as tokens for what Walter wants. I'll see what I can do though.

- Jonathan M Davis
August 03, 2012
On 2012-08-02 22:54, Andrej Mitrovic wrote:

> It can do that immediately for the text that's visible in the window
> because ~100 lines of text can be lexed pretty damn instantly. As soon
> as that's done the GUI should be responsive and the rest of the text
> buffer should be lexed in the background.

That should work if the lexer can handle it.

-- 
/Jacob Carlborg
August 03, 2012
On 2012-08-03 00:10, Walter Bright wrote:

> The rendering code should be in yet a third thread.

Most GUI systems are not thread safe. I know for sure that Cocoa on Mac OS X is not. All the changes to the GUI needs to happen in the same thread. But you can usually post a message from another thread to the GUI thread if you want to update the GUI from another thread.

> An editor I wrote years ago had the rendering code in a separate thread
> from user input. You never had to wait to type in commands, the
> rendering would catch up when it could. What was also effective was the
> rendering would abandon a render midstream and restart it if it detected
> that the underlying data had changed in the meantime. This meant that
> the display was never more than one render out of date.
>
> Although the code itself wasn't any faster, it certainly *felt* faster
> with this approach. It made for crisp editing even on a pig slow machine.

But if you type something and you don't see any new characters that will feel slow. Regardless of the application received the user input or not. The user doesn't care what's happening in the background of the application, it only cares about what it can see.

I've used many editors/IDEs where the GUI locks, for whatever reason, you continue typing and suddenly you get 5 characters at once. To me, that would indicate the the application do receive the user input but it cannot render it fast enough, for whatever reason.

-- 
/Jacob Carlborg
August 03, 2012
On 2012-08-03 00:25, Dmitry Olshansky wrote:

> OT:
> It never ceases to amaze me how people miss this very simple point:
> GUI runs on its own thread and shouldn't ever block on something (save
> for message pump itself, of course). Everything else (including possibly
> slow rendering) done on the side and then result (once ready) swiftly
> indicated on GUI.

I'm not entirely sure what you mean be "rendering" but most GUI systems are not thread safe and all GUI updates need to happen in the same thread.

-- 
/Jacob Carlborg
August 03, 2012
On 2012-08-03 00:01, Walter Bright wrote:

> But we do have the DMD lexer which is useful as a benchmark and a guide.
> I won't say it couldn't be made faster, but it does set a minimum bar
> for performance.

I'm not sure how easy it would be to just measure the lexing phase of DMD. If it's easy someone would probably already have extracted the lexer from DMD.

-- 
/Jacob Carlborg
August 03, 2012
Am 03.08.2012 08:37, schrieb Jacob Carlborg:
> On 2012-08-03 00:01, Walter Bright wrote:
>
>> But we do have the DMD lexer which is useful as a benchmark and a guide.
>> I won't say it couldn't be made faster, but it does set a minimum bar
>> for performance.
>
> I'm not sure how easy it would be to just measure the lexing phase of
> DMD. If it's easy someone would probably already have extracted the
> lexer from DMD.
>

wouldn't it be better to extract the lexer part of dmd into its own (hopefully small) library - that way the lexer is still useable by dmd AND benchmarkable from outside - it is then even possible to replace the dmd lexer by an D version due to the c linkage feature of D
August 03, 2012
On Friday, 3 August 2012 at 04:02:34 UTC, Bernard Helyer wrote:
> I'll let you get on with it then. I'll amuse myself with the
> thought of someone asking why SDC doesn't use std.d.lexer or
> a parser generator. I'll then hit them with my cane, and tell
> them to get off of my lawn. :P

I don't think you should give up on this adaption. SDC's lexer is already complete. I don't know if anyone else can claim that. Apart from that, any changes made will almost certainly benefit SDC as well.