https://issues.dlang.org/show_bug.cgi?id=16457

greenify <greeenify@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |greeenify@gmail.com
            Summary|std.regex postprocesses at  |std.regex postprocesses
                   |ctRegex every time at       |ctRegex every time at
                   |runtime                     |runtime

--

https://issues.dlang.org/show_bug.cgi?id=16457

Dmitry Olshansky <dmitry.olsh@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmitry.olsh@gmail.com

--- Comment #1 from Dmitry Olshansky <dmitry.olsh@gmail.com> ---
(In reply to greensunny12 from comment #0)
> Consider the following program, it will crash as it allocates _a lot_ of memory before the garbage can be collected.
> 
> void main()
> {
>     import std.regex;
>     enum re = ctRegex!(`((c)(s)?)?ti`);
>     import core.memory : GC;
>     string text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit";
> 
>     foreach (i; 0..500_000_000)
>     {
>         if (auto m = matchAll(text, re)) {}
>         //if (i % 1_000_000)
>             //GC.collect();
>     }
> }
> 
> On my machine (16G) it crashes at about 5M iterations.
> The GC profile finds two hotspots (here 2M iterations):
> 
>      2048000000	        2000000	uint[] D main
> std/regex/internal/parser.d:1607
>       184000000	        2000000	std.regex.internal.ir.Bytecode[] D main
> std/array.d:852
> 
> (the latter is insertInPlace)
> 
> After looking at the code it seems pretty weird, because "postprocess" should be called while constructing the RegEx and once only.

The problem is enum re = ... line. Enum means ctRegex!`((c)(s)?)?ti` is
copy-pasted at the place of usage, basically reconstructing regex on each loop
iteration b/c compile-time version can't cache compiled patterns.

Replace enum with static and all should be good.

--

https://issues.dlang.org/show_bug.cgi?id=16457

--- Comment #2 from greenify <greeenify@gmail.com> ---
thanks a lot for looking at this so quickly :)

> The problem is enum re = ... line. Enum means ctRegex!`((c)(s)?)?ti` is copy-pasted at the place of usage, basically reconstructing regex on each loop iteration b/c compile-time version can't cache compiled patterns.

Okay that makes partially sense, but to give some background I found this problem because the D changelog generation crashed on my machine (16G) while parsing a 300kB log file. The dummy file is a dustmited & reduced version of the changelog generator:

https://github.com/dlang/tools/blob/master/changed.d#L77

And using `enum ... = ctRegex!...` seems to be a very common pattern:

https://github.com/search?q=enum+ctRegex&type=Code&utf8=%E2%9C%93

> Replace enum with static and all should be good.

Unfortunately I can still observe the same behavior, I also tried to move
ctRegex to the matchAll.
Funnily with regex I don't get this allocation / out of memory -problem.

FYI static immutable doens't work:

foo.d(10): Error: template std.regex.matchAll cannot deduce function from
argument types !()(string, immutable(StaticRegex!char)), candidates are:
/usr/include/dlang/dmd/std/regex/package.d(860):        std.regex.matchAll(R,
RegEx)(R input, RegEx re) if (isSomeString!R && is(RegEx ==
Regex!(BasicElementOf!R)))
/usr/include/dlang/dmd/std/regex/package.d(868):        std.regex.matchAll(R,
String)(R input, String re) if (isSomeString!R && isSomeString!String)
/usr/include/dlang/dmd/std/regex/package.d(875):        std.regex.matchAll(R,
RegEx)(R input, RegEx re) if (isSomeString!R && is(RegEx ==
StaticRegex!(BasicElementOf!R)))

--

https://issues.dlang.org/show_bug.cgi?id=16457

Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Hardware|x86_64                      |All
           See Also|                            |https://issues.dlang.org/sh
                   |                            |ow_bug.cgi?id=13532
                 OS|Linux                       |All

--- Comment #3 from Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> ---
I think the obvious reasonable improvement here is to improve the documentation, and mention to not use ctRegex with enum. Not sure much can be done in Phobos to work around an underlying language / compiler implementation issue.

--

https://issues.dlang.org/show_bug.cgi?id=16457

Dmitry Olshansky <dmitry.olsh@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|nobody@puremagic.com        |dmitry.olsh@gmail.com

--

https://issues.dlang.org/show_bug.cgi?id=16457

Iain Buclaw <ibuclaw@gdcproject.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P1                          |P4

--

https://issues.dlang.org/show_bug.cgi?id=16457

--- Comment #4 from dlangBugzillaToGithub <robert.schadek@posteo.de> ---
THIS ISSUE HAS BEEN MOVED TO GITHUB

https://github.com/dlang/phobos/issues/10198

DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB

--