d-bugmail@puremagic.com wrote:
>         // thinks it's [0 ... 1], no maximal munch taking place
>         assert (Foo[0... 1] == 0);
> }

this *is* maximal munch taking place. because of the ".." lexeme, float literals are not lexemes. they are context free production rules consisting of multiple lexemes. therefore "0." consists of two lexemes and "..." wins the max munch over ".".

Reply to Jascha,

> d-bugmail@puremagic.com wrote:
> 
>> // thinks it's [0 ... 1], no maximal munch taking place
>> assert (Foo[0... 1] == 0);
>> }
> this *is* maximal munch taking place. because of the ".." lexeme,
> float literals are not lexemes. they are context free production rules
> consisting of multiple lexemes. therefore "0." consists of two lexemes
> and "..." wins the max munch over ".".
> 

But is it the the correct way to do it? (Not is is doing what the spec says, but is it doing what it should be designed to do)

BCS wrote:
> Reply to Jascha,
> 
>> d-bugmail@puremagic.com wrote:
>>
>>> // thinks it's [0 ... 1], no maximal munch taking place
>>> assert (Foo[0... 1] == 0);
>>> }
>> this *is* maximal munch taking place. because of the ".." lexeme,
>> float literals are not lexemes. they are context free production rules
>> consisting of multiple lexemes. therefore "0." consists of two lexemes
>> and "..." wins the max munch over ".".
>>
> 
> But is it the the correct way to do it? (Not is is doing what the spec says, but is it doing what it should be designed to do)

i think "ptr ! is null" shouldn't be allowed, because it suggests that "!" and "is" are separate operators and "is null" is an unary expression.
if your lexer supports lookaheads (a single character is enough), you can match float literals as lexemes. this is also what DMD does.

Jascha Wetzel wrote:
> d-bugmail@puremagic.com wrote:
>>         // thinks it's [0 ... 1], no maximal munch taking place
>>         assert (Foo[0... 1] == 0);
>> }
> 
> this *is* maximal munch taking place. because of the ".." lexeme, float literals are not lexemes. they are context free production rules consisting of multiple lexemes. therefore "0." consists of two lexemes and "..." wins the max munch over ".".

this was formulated poorly. float literals *may* be considered context free to solve this problem...

Jascha Wetzel wrote:
> Jascha Wetzel wrote:
>> d-bugmail@puremagic.com wrote:
>>>         // thinks it's [0 ... 1], no maximal munch taking place
>>>         assert (Foo[0... 1] == 0);
>>> }
>>
>> this *is* maximal munch taking place. because of the ".." lexeme, float literals are not lexemes. they are context free production rules consisting of multiple lexemes. therefore "0." consists of two lexemes and "..." wins the max munch over ".".
> 
> this was formulated poorly. float literals *may* be considered context free to solve this problem...

Exactly. But the way I read the spec, float literals are considered tokens in and of themselves. Maybe I misunderstand, but it could use some clarification in that case: lex.html specifically says that the lexer splits the code into tokens, one of which is "0.", with maximal munch.

This isn't a /problem/ per se. In the extreme case, of course it is possible to parse D with maximal munch by considering every character a lexeme of its own and figuring everything else out thereafter. I'm just saying that the spec seems to contradict itself in saying that maximal munch should be used, and that it should match (among other things) "0." as one token. If you do things that way, it doesn't work the way DMD currently does it. If you match "0." as "0" and "." and construct a float literal from that later, it works.

-- 
E-mail address: matti.niemenmaa+news, domain is iki (DOT) fi

Matti Niemenmaa wrote:
> Jascha Wetzel wrote:
>> Jascha Wetzel wrote:
>>> d-bugmail@puremagic.com wrote:
>>>>         // thinks it's [0 ... 1], no maximal munch taking place
>>>>         assert (Foo[0... 1] == 0);
>>>> }
>>> this *is* maximal munch taking place. because of the ".." lexeme,
>>> float literals are not lexemes. they are context free production rules
>>> consisting of multiple lexemes. therefore "0." consists of two lexemes
>>> and "..." wins the max munch over ".".
>> this was formulated poorly. float literals *may* be considered context
>> free to solve this problem...
> 
> Exactly. But the way I read the spec, float literals are considered tokens in
> and of themselves. Maybe I misunderstand, but it could use some clarification in
> that case: lex.html specifically says that the lexer splits the code into
> tokens, one of which is "0.", with maximal munch.
> 
> This isn't a /problem/ per se. In the extreme case, of course it is possible to
> parse D with maximal munch by considering every character a lexeme of its own
> and figuring everything else out thereafter. I'm just saying that the spec seems
> to contradict itself in saying that maximal munch should be used, and that it
> should match (among other things) "0." as one token. If you do things that way,
> it doesn't work the way DMD currently does it. If you match "0." as "0" and "."
> and construct a float literal from that later, it works.

i agree that it's not clear. one can argue, though, that if you consider lookaheads as part of the lexeme specification, the maximal munch property remains intact. then "0." can only match if not followed by another "." - no contradiction to the max munch. but that should be stated in the specs, of course.

Jascha Wetzel wrote:
<snip>
> but that should be stated in the specs, of course.

Exactly, which is what the bug is about. :-)

-- 
E-mail address: matti.niemenmaa+news, domain is iki (DOT) fi

http://d.puremagic.com/issues/show_bug.cgi?id=1466


Walter Bright <bugzilla@digitalmars.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |bugzilla@digitalmars.com
         Resolution|                            |FIXED


--- Comment #9 from Walter Bright <bugzilla@digitalmars.com> 2010-11-09 19:43:04 PST ---
http://www.dsource.org/projects/phobos/changeset/2148

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------