This might sound silly, but how about if D stopped allowing   0..2  as a range, and instead just said "invalid floating-point number"?

Fixing it en masse would be pretty trivial... just run a regex to replace
	"\b(\d+)\.\."
with
	"\1 .. "
and you're good to go.

(Or if you want more accuracy, just take the compiler output and feed it back with a fix -- that would work too.)

The benefit, though, is that now you can do maximal munch without worrying about this edge case... which sure makes it easier to make a lexer.

Thoughts?

On 07-07-2012 23:39, Mehrdad wrote:
> This might sound silly, but how about if D stopped allowing 0..2  as a
> range, and instead just said "invalid floating-point number"?
>
> Fixing it en masse would be pretty trivial... just run a regex to replace
>      "\b(\d+)\.\."
> with
>      "\1 .. "
> and you're good to go.
>
> (Or if you want more accuracy, just take the compiler output and feed it
> back with a fix -- that would work too.)
>
> The benefit, though, is that now you can do maximal munch without
> worrying about this edge case... which sure makes it easier to make a
> lexer.
>
> Thoughts?

... why is this even done at the lexical stage? It should be done at the parsing stage if anything.

-- 
Alex Rønne Petersen
alex@lycus.org
http://lycus.org

On Saturday, 7 July 2012 at 21:41:44 UTC, Alex Rønne Petersen wrote:
>
> ... why is this even done at the lexical stage? It should be done at the parsing stage if anything.

Well, even better than -- it makes it easier to make a parser.

That said, what's wrong with doing it in the lexical stage?

On Sat, Jul 07, 2012 at 11:39:59PM +0200, Mehrdad wrote:
> This might sound silly, but how about if D stopped allowing   0..2 as a range, and instead just said "invalid floating-point number"?
[...]

I like writing 0..2 as a range. It's especially nice in array slice notation, where you _want_ to have it as concise as possible.

OTOH, having implemented a D lexer before (just for practice, not production quality), I do see how ambiguities with floating-point numbers can cause a lot of code convolutions.

But I'm gonna have to say no to this one; *I* think a better solution would be to prohibit things like 0. or 1. in a float literal. Either follow it with a digit, or don't write the dot. This will also save us a lot of pain in the UFCS department, where 4.sqrt is currently a pain to lex. Once this is done, 0..2 is no longer ambiguous, and any respectable DFA lexer should be able to handle it with ease.


T

-- 
If a person can't communicate, the very least he could do is to shut up. -- Tom Lehrer, on people who bemoan their communication woes with their loved ones.

On Sat, Jul 07, 2012 at 11:41:43PM +0200, Alex Rønne Petersen wrote:
> On 07-07-2012 23:39, Mehrdad wrote:
> >This might sound silly, but how about if D stopped allowing 0..2  as a range, and instead just said "invalid floating-point number"?
[...]
> ... why is this even done at the lexical stage? It should be done at the parsing stage if anything.
[...]

This is because the lexer can mistakenly identify it as "0." followed by ".2" instead of "0" followed by ".." followed by "2".

IMAO, this problem is caused by floating point notational stupidities like 0. and .1, especially the former. Get rid of the former (and optionally the latter) will fix a whole bunch of lexer pain in D.


T

-- 
They say that "guns don't kill people, people kill people." Well I think the gun helps. If you just stood there and yelled BANG, I don't think you'd kill too many people. -- Eddie Izzard, Dressed to Kill

On 07/07/2012 11:39 PM, Mehrdad wrote:
> This might sound silly,

+1.

> but how about if D stopped allowing 0..2 as a
> range, and instead just said "invalid floating-point number"?
>
> Fixing it en masse would be pretty trivial... just run a regex to replace
> "\b(\d+)\.\."
> with
> "\1 .. "
> and you're good to go.
>
> (Or if you want more accuracy, just take the compiler output and feed it
> back with a fix -- that would work too.)
>
> The benefit, though, is that now you can do maximal munch without
> worrying about this edge case... which sure makes it easier to make a
> lexer.
>
> Thoughts?

It does not make it easier to create a lexer, because this is not
actually an edge case worth explicitly testing for.

switch(input.front){
    case '0'..'9': ...
    case 'a'..'f', 'A'..'F': ...
    case '.': if('0'>input[1]||input[1]>'9') break;
        ...
}

On 07/08/2012 12:12 AM, Timon Gehr wrote:
> On 07/07/2012 11:39 PM, Mehrdad wrote:
>> This might sound silly,
>
> +1.
>
>> but how about if D stopped allowing 0..2 as a
>> range, and instead just said "invalid floating-point number"?
>>
>> Fixing it en masse would be pretty trivial... just run a regex to replace
>> "\b(\d+)\.\."
>> with
>> "\1 .. "
>> and you're good to go.
>>
>> (Or if you want more accuracy, just take the compiler output and feed it
>> back with a fix -- that would work too.)
>>
>> The benefit, though, is that now you can do maximal munch without
>> worrying about this edge case... which sure makes it easier to make a
>> lexer.
>>
>> Thoughts?
>
> It does not make it easier to create a lexer, because this is not
> actually an edge case worth explicitly testing for.
>
> switch(input.front){

I meant input[0]. No need for decoding.

>      case '0'..'9': ...
>      case 'a'..'f', 'A'..'F': ...
>      case '.': if('0'>input[1]||input[1]>'9') break;
>          ...
> }

On Saturday, July 07, 2012 15:01:50 H. S. Teoh wrote:
> On Sat, Jul 07, 2012 at 11:39:59PM +0200, Mehrdad wrote:
> > This might sound silly, but how about if D stopped allowing   0..2 as a range, and instead just said "invalid floating-point number"?
> 
> [...]
> 
> I like writing 0..2 as a range. It's especially nice in array slice notation, where you _want_ to have it as concise as possible.
> 
> OTOH, having implemented a D lexer before (just for practice, not production quality), I do see how ambiguities with floating-point numbers can cause a lot of code convolutions.
> 
> But I'm gonna have to say no to this one; *I* think a better solution would be to prohibit things like 0. or 1. in a float literal. Either follow it with a digit, or don't write the dot. This will also save us a lot of pain in the UFCS department, where 4.sqrt is currently a pain to lex. Once this is done, 0..2 is no longer ambiguous, and any respectable DFA lexer should be able to handle it with ease.

+1

I think that it's ridiculous that 1. and .1 are legal. 1.f was fixed, so I was shocked to find out recently that 1. and .1 weren't.

- Jonathan M Davis

On Saturday, 7 July 2012 at 22:00:43 UTC, H. S. Teoh wrote:
> On Sat, Jul 07, 2012 at 11:39:59PM +0200, Mehrdad wrote:
>> This might sound silly, but how about if D stopped allowing   0..2
>> as a range, and instead just said "invalid floating-point number"?
> [...]
>
> I like writing 0..2 as a range. It's especially nice in array slice notation, where you _want_ to have it as concise as possible.

Hmm... true..

> OTOH, having implemented a D lexer before (just for practice, not production quality), I do see how ambiguities with floating-point numbers can cause a lot of code convolutions.

Yeah that's exactly what happened to me lol.
(Mainly the problem I ran into was that I was REALLY trying to avoid extra lookaheads if possible, since I was sticking to the range interface of front/popFront, and trying not to consume more than I can handle... and this was the edge case that broke it.)

> But I'm gonna have to say no to this one; *I* think a better solution would be to prohibit things like 0. or 1. in a float literal. Either follow it with a digit, or don't write the dot. This will also save us a lot of pain in the UFCS department, where 4.sqrt is currently a pain to lex. Once this is done, 0..2 is no longer ambiguous, and any respectable DFA lexer should be able to handle it with ease.

Good idea, I like it too. How about just disallowing trailing decimal points then?

On Saturday, July 07, 2012 15:20:28 Jonathan M Davis wrote:
> On Saturday, July 07, 2012 15:01:50 H. S. Teoh wrote:
> > On Sat, Jul 07, 2012 at 11:39:59PM +0200, Mehrdad wrote:
> > > This might sound silly, but how about if D stopped allowing   0..2 as a range, and instead just said "invalid floating-point number"?
> > 
> > [...]
> > 
> > I like writing 0..2 as a range. It's especially nice in array slice notation, where you _want_ to have it as concise as possible.
> > 
> > OTOH, having implemented a D lexer before (just for practice, not production quality), I do see how ambiguities with floating-point numbers can cause a lot of code convolutions.
> > 
> > But I'm gonna have to say no to this one; *I* think a better solution would be to prohibit things like 0. or 1. in a float literal. Either follow it with a digit, or don't write the dot. This will also save us a lot of pain in the UFCS department, where 4.sqrt is currently a pain to lex. Once this is done, 0..2 is no longer ambiguous, and any respectable DFA lexer should be able to handle it with ease.
> 
> +1
> 
> I think that it's ridiculous that 1. and .1 are legal. 1.f was fixed, so I was shocked to find out recently that 1. and .1 weren't.

There's an existing enhancement request for it:

http://d.puremagic.com/issues/show_bug.cgi?id=6277

- Jonathan M Davis