View mode: basic / threaded / horizontal-split · Log in · Help
March 28, 2005
Unquoted regular expressions
IIRC, we left this issue at "they'd be followed by a left parenthesis."

I think that the lexer should try everyting else, and if nothing else is 
ok, then try if it is an unquoted regular expression. Without regard to 
parenthesis. Then we could write:

foo = /laksjdf/;
bar = /lskdf/sdf/;

Foo and bar would be functions taking a string, an int, and outputting 
an array, and returning bool.

This may not have to be slow. After all, we do already have the regex 
library, and the lexer could use only the "recognizing" part of it, i.e. 
do the parsing without generating any binary code.

If it does not parse as a regular expression, then flag error.

This would make parser writing much cleaner.

intR       = /lksjdf/;
floatR     = /fjfjf/;
stringR    = /yee\"lksjdlfks\"haw/;
hairyClass = /lksjdlfkjsdlfkjsldfklskdfjlskdjlfk/;

and also,

ok = /lksjdflks/ (string, pos, resultArray);

or

if (/lksjd/ (theArray, curPos, res))
{
    // do the doobydoo;
}

Since regular expressions are a natural part of parsing, that area 
should be the main target.

...
    if      (intR      (input, where, res) wehaveInt();
    else if (floatR    (input, where, res) wehaveFloat();
    else if (stringR   (input, where, res) wehaveString();
    else if (hairyClass(input, where, res) shootTheSheriff();
...
    else giveUp();

FTR, the whole idea here being that these regular expressions are 
compiled to executable at compile time.

And, that regexes encountered at runtime would be compiled at encounter, 
as in:

char[] getInput = askRegexDialog();  // string assignment
regexTypeFunction gi = getInput;     // compile regex

if (gi(ins, p, oa))
{
    // we found user's wishes in the input stream
}

------------

This is outline, so details are not accurate. (For example, how to 
mediate the end position of regex search, was not considered here.)
March 29, 2005
Re: Unquoted regular expressions
"Georg Wrede" <georg.wrede@nospam.org> wrote in message
news:4248919A.1010208@nospam.org...
> IIRC, we left this issue at "they'd be followed by a left parenthesis."
>
> I think that the lexer should try everyting else, and if nothing else is
> ok, then try if it is an unquoted regular expression. Without regard to
> parenthesis. Then we could write:

This is done in the DMDScript lexer. However, it's an ugly hack and requires
the lexer and parser to cooperate with each other. One of the design goals
of D is to have a lexer that is independent of parsing. This pretty much
means that regular expression literals using / is out. But check out some of
the new functions I added to www.digitalmars.com/d/std_regexp.html, I think
the syntax is reasonably clean and useful.
March 30, 2005
Re: Unquoted regular expressions
Walter wrote:
> "Georg Wrede" <georg.wrede@nospam.org> wrote in message
> news:4248919A.1010208@nospam.org...
> 
>>IIRC, we left this issue at "they'd be followed by a left parenthesis."
>>
>>I think that the lexer should try everyting else, and if nothing else is
>>ok, then try if it is an unquoted regular expression. Without regard to
>>parenthesis. Then we could write:
> 
> This is done in the DMDScript lexer. However, it's an ugly hack and requires
> the lexer and parser to cooperate with each other. One of the design goals
> of D is to have a lexer that is independent of parsing. This pretty much
> means that regular expression literals using / is out. But check out some of
> the new functions I added to www.digitalmars.com/d/std_regexp.html, I think
> the syntax is reasonably clean and useful.

It is! And I'm not forcing this issue here, it's just that "it would be 
nice", and "not difficult". (IMHO)

So, with the unquoted regexes + separate lexing, we have the following 
problem:

height = volume /width/ length;
halfHeight = volume /width/length/ 2;

(I've emphasized the problem above with stupid spacing.)

/width/length/ can either be a regex that substitutes every width with 
length, or it can be part of a math expression.

From the context it is painfully obvious which, but the lexer (who has 
aphasia because it is context independent), can't know if we here have 
unquoted regular expressions, or not.

Case closed. For good.

What else can we do? Oh, and why would we?

The existing regex library is nice! But as we see from other languages, 
having the regexes "in the code" and "obviously compiled at compile 
time, rather than runtime", does seem to have its merits.

"Obviously"? Well, reading source code of a compiled language, if a 
literal regex appears as such in the source code, then it is taken for 
granted that it is compiled at compile time. It's just something you do. 
OTOH, having regexes in quoted strings and used by library functions 
"feels" like they are runtime compiled.

Another thing, this discussion of Unquoted regexes has been a kind of 
misnomer (and I take most of the blame). A regex that looks like

/blkajsld/

is actually quoted already. (Look at substitute in ed, vi, vim, and the 
like, and you see that other characters can be used for quoting. That is 
usually convenient when the regex itself contains forward slashes.)

So, how would we have the lexer recognize a regex type? And the 
programmer immediately spot it too?

"/lkajsdl/" would seem natural. So "/ and /" would be two new tokens.

Very nice. Except that this precludes having /asdfadsf/ as the contents 
of a normal string. This is getting depressing.

Would reversing the tokens do it? Then we'd have /"lksjdlks"/ and 
/"lkasjdf/lkasjdf"/ for searching and substituting, respectively.

The lexer would not even have to know of the middle slash, since the 
function signatures for both are the same (at least as per the earlier 
thread).

Requiring backslash quoting of double quotes would take care of the 
situation where the regex contains a double quote immediately followed 
by a forward slash, as in /"lksjdlfkasd\"/lkasjdf"/.
March 31, 2005
Re: Unquoted regular expressions
/" isn't that good, it prevents you from having a normal opDiv(char[])...

how about RE"..." and RE`...`? we already have wysiwyg strings, so this 
wouldn't be something completely new (at least syntax-wise).. It doesn't 
use a slash, but who cares, I think it's still obvious that it's 
compile-time?


xs0


Georg Wrede wrote:
> Walter wrote:
> 
>> "Georg Wrede" <georg.wrede@nospam.org> wrote in message
>> news:4248919A.1010208@nospam.org...
>>
>>> IIRC, we left this issue at "they'd be followed by a left parenthesis."
>>>
>>> I think that the lexer should try everyting else, and if nothing else is
>>> ok, then try if it is an unquoted regular expression. Without regard to
>>> parenthesis. Then we could write:
>>
>>
>> This is done in the DMDScript lexer. However, it's an ugly hack and 
>> requires
>> the lexer and parser to cooperate with each other. One of the design 
>> goals
>> of D is to have a lexer that is independent of parsing. This pretty much
>> means that regular expression literals using / is out. But check out 
>> some of
>> the new functions I added to www.digitalmars.com/d/std_regexp.html, I 
>> think
>> the syntax is reasonably clean and useful.
> 
> 
> It is! And I'm not forcing this issue here, it's just that "it would be 
> nice", and "not difficult". (IMHO)
> 
> So, with the unquoted regexes + separate lexing, we have the following 
> problem:
> 
> height = volume /width/ length;
> halfHeight = volume /width/length/ 2;
> 
> (I've emphasized the problem above with stupid spacing.)
> 
> /width/length/ can either be a regex that substitutes every width with 
> length, or it can be part of a math expression.
> 
>  From the context it is painfully obvious which, but the lexer (who has 
> aphasia because it is context independent), can't know if we here have 
> unquoted regular expressions, or not.
> 
> Case closed. For good.
> 
> What else can we do? Oh, and why would we?
> 
> The existing regex library is nice! But as we see from other languages, 
> having the regexes "in the code" and "obviously compiled at compile 
> time, rather than runtime", does seem to have its merits.
> 
> "Obviously"? Well, reading source code of a compiled language, if a 
> literal regex appears as such in the source code, then it is taken for 
> granted that it is compiled at compile time. It's just something you do. 
> OTOH, having regexes in quoted strings and used by library functions 
> "feels" like they are runtime compiled.
> 
> Another thing, this discussion of Unquoted regexes has been a kind of 
> misnomer (and I take most of the blame). A regex that looks like
> 
> /blkajsld/
> 
> is actually quoted already. (Look at substitute in ed, vi, vim, and the 
> like, and you see that other characters can be used for quoting. That is 
> usually convenient when the regex itself contains forward slashes.)
> 
> So, how would we have the lexer recognize a regex type? And the 
> programmer immediately spot it too?
> 
> "/lkajsdl/" would seem natural. So "/ and /" would be two new tokens.
> 
> Very nice. Except that this precludes having /asdfadsf/ as the contents 
> of a normal string. This is getting depressing.
> 
> Would reversing the tokens do it? Then we'd have /"lksjdlks"/ and 
> /"lkasjdf/lkasjdf"/ for searching and substituting, respectively.
> 
> The lexer would not even have to know of the middle slash, since the 
> function signatures for both are the same (at least as per the earlier 
> thread).
> 
> Requiring backslash quoting of double quotes would take care of the 
> situation where the regex contains a double quote immediately followed 
> by a forward slash, as in /"lksjdlfkasd\"/lkasjdf"/.
March 31, 2005
Re: Unquoted regular expressions
xs0 wrote:
> 
> /" isn't that good, it prevents you from having a normal opDiv(char[])...

To what is a "normal opDiv(char[])" used?

> how about RE"..." and RE`...`? we already have wysiwyg strings, so this 
> wouldn't be something completely new (at least syntax-wise).. It doesn't 
> use a slash, but who cares, I think it's still obvious that it's 
> compile-time?

I could live with that.
April 01, 2005
Re: Unquoted regular expressions
Georg Wrede wrote:
> xs0 wrote:
> 
>>
>> /" isn't that good, it prevents you from having a normal opDiv(char[])...
> 
> 
> To what is a "normal opDiv(char[])" used?

I don't know (to split some string buffer with the parameter, perhaps?), 
 but the point was that /"RE"/ would clash with division by char[], 
which is not good..


xs0
April 01, 2005
Re: Unquoted regular expressions
xs0 wrote:
> Georg Wrede wrote:
>> xs0 wrote:
>>> /" isn't that good, it prevents you from having a normal 
>>> opDiv(char[])...
>> To what is a "normal opDiv(char[])" used?
> 
> I don't know (to split some string buffer with the parameter, perhaps?), 
>  but the point was that /"RE"/ would clash with division by char[], 
> which is not good..

Ehh, which might be more important? ;-)
April 01, 2005
Re: Unquoted regular expressions
Georg Wrede wrote:
> xs0 wrote:
> 
>> Georg Wrede wrote:
>>
>>> xs0 wrote:
>>>
>>>> /" isn't that good, it prevents you from having a normal 
>>>> opDiv(char[])...
>>>
>>> To what is a "normal opDiv(char[])" used?
>>
>>
>> I don't know (to split some string buffer with the parameter, 
>> perhaps?),  but the point was that /"RE"/ would clash with division by 
>> char[], which is not good..
> 
> 
> Ehh, which might be more important? ;-)

Well, one can argue both ways, but I think that the language should be 
without random exceptions like this.. It would be about the same s**t as 
requiring spaces between >> in template parameters in C++ ..


xs0
April 01, 2005
Re: Unquoted regular expressions
xs0 wrote:
> Georg Wrede wrote:
> 
>> xs0 wrote:
>>
>>> Georg Wrede wrote:
>>>
>>>> xs0 wrote:
>>>>
>>>>> /" isn't that good, it prevents you from having a normal 
>>>>> opDiv(char[])...
>>>>
>>>>
>>>> To what is a "normal opDiv(char[])" used?
>>>
>>>
>>>
>>> I don't know (to split some string buffer with the parameter, 
>>> perhaps?),  but the point was that /"RE"/ would clash with division 
>>> by char[], which is not good..
>>
>>
>>
>> Ehh, which might be more important? ;-)
> 
> 
> Well, one can argue both ways, but I think that the language should be 
> without random exceptions like this.. It would be about the same s**t as 
> requiring spaces between >> in template parameters in C++ ..

Good point. And I've personally argued for the same logic on several 
occasions here.

On this particular occasion, however, we'd either have to come up with 
something else as the token, or weigh principle against practice.

Suggestions, ideas?
Top | Discussion index | About this forum | D home